Paper Summary. Dynet: The Dynamic Neural Network Toolkit

The programming model that underlies several pop toolkits such every bit TensorFlow uses a static annunciation approach: they split annunciation as well as execution of the network architecture.

Static annunciation has a number of advantages. After the computation graph is defined, it tin laissez passer the sack endure optimized inwards a number of ways so that the subsequent repeated executions of computation tin laissez passer the sack endure performed every bit chop-chop every bit possible. This also simplifies distribution of computation across multiple devices, every bit inwards TensorFlow. But static annunciation is inconvenient for the following:

• variably sized inputs
• variably structured inputs
• nontrivial inference algorithms
• variably structured outputs

Of course, it is possible to procedure variable sized inputs if the computation graphs tin laissez passer the sack stand upwards for objects whose size is unspecified at annunciation time. Flow command operations such every bit conditional execution as well as iteration tin laissez passer the sack endure added to the inventory of operations supported past times the computation graph. For example, to run an RNN over variable length sequences, Theano offers the scan operation, as well as TensorFlow offers the dynamic RNN operation.

While it is thence possible to bargain amongst variable architectures amongst static annunciation inwards theory, that yet poses about difficulties inwards practice:

• Difficulty inwards expressing complex flow-control logic
• Complexity of the computation graph implementation
• Difficulty inwards debugging

These are associated amongst about serious software technology scientific discipline risks. As an alternative, DyNet proposes reviving an option programming model: dynamic annunciation of computation graphs.

Dynamic declaration

The dynamic annunciation model inwards Dynet takes a single-step approach: the user defines the computation graph programmatically every bit if they were calculating the outputs of their network on a detail preparation instance. There are no split steps for Definition as well as execution: the necessary computation graph is created, on the fly, every bit the loss calculation is executed, as well as a novel graph is created for each preparation instance. (To avoid the overhead, DyNet strives to render real lightweight graph construction.)

Dynamic annunciation reduces the complexity of the computation graph implementation since it does non demand to comprise menses command operations or back upwards dynamically sized data. DyNet is designed to allow users to implement their models inwards their preferred programming linguistic communication (C++ or Python). A symbolic computation graph is yet constructed, only past times using the host linguistic communication (C++ or Python) rather than providing them separately at the computation graph level. Thus, dynamic annunciation facilitates the implementation of to a greater extent than complicated network architectures.

What is the excogitation inwards DyNet?

DyNet aims to minimize the computational toll of graph construction inwards social club to allow efficient dynamic computation. This agency DyNet aspires to take away barriers to rapid prototyping as well as implementation of to a greater extent than sophisticated applications of neural nets that are non slow to implement inwards the static computation paradigm.

DyNet's backend, which is written inwards C++, is optimized to take away overhead inwards computation graph construction, as well as back upwards efficient execution on both CPU as well as GPU. This is viable to do. Since menses command as well as facilities for dealing amongst variably sized inputs stay inwards the host linguistic communication (rather than inwards the computation graph, every bit is required past times static declaration), the computation graph needs to back upwards fewer performance types, as well as these tend to endure to a greater extent than completely specified (e.g., tensor sizes are ever known rather than inferred at execution time).

DyNet programs

DyNet programs follow the next template
1. Create a Model.
2. Add the necessary Parameters as well as LookupParameters to the model. Create a Trainer object as well as associate it amongst the Model.
3. For each input example:
(a) Create a novel ComputationGraph, as well as populate it past times edifice an Expression representing the desired computation for this example.
(b) Calculate the number of that computation frontwards through the graph past times calling the value() or npvalue() functions of the terminal Expression
(c) If training, calculate an Expression representing the loss function, as well as usage its backward() constituent to perform back-propagation
(d) Use the Trainer to update the parameters inwards the Model

In contrast to static annunciation libraries such every bit TensorFlow, inwards DyNet the "create a graph" pace falls inside the loop. This has the wages of allowing the user to flexibly do a novel graph construction for each representative as well as to usage menses command syntax (e.g., iteration) from their native programming language.

Here is an representative program.


This plan shows the procedure of performing maximum likelihood preparation for a uncomplicated classifier that calculates a vector of scores for each shape it volition endure expected to predict, as well as so returns the ID of the shape amongst the highest score.  Notice that, at describe 14: symbolic graph is defined dynamically, at describe 15: frontwards exceed is executed, as well as at describe 16: backward exceed automatic diff is executed. At describe 19, later the training, inference is done. To trouble organisation human relationship for dynamic input/graphs at inference, the graph is reconstructed for each serving input.


Dynet allows dynamic menses command at the inference fourth dimension easily. This tin laissez passer the sack allow the classifier to avoid wasting processing fourth dimension when the respond is clear. It is also possible to perform dynamic menses command at preparation time, as well as this supports to a greater extent than sophisticated preparation algorithms using reinforcement learning. These algorithms require interleaving model evaluation as well as conclusion making on the footing of that evaluation.

How do nosotros brand DyNet distributed

Dynet is currently centralized. There is also back upwards for automatic mini-batching to amend computational efficiency, taking the burden off of users who wishing to implement mini-batching inwards their models. For to a greater extent than complicated models that do non back upwards mini-batching, in that place is back upwards for data-parallel multi-processing, inwards which asynchronous parameter updates are performed across multiple threads, making it uncomplicated to parallelize (on a unmarried machine) whatever diversity of model at preparation time.

Petuum Inc. is working on extending this parallelism from unmarried machine to multiple machines data-parallel processing, past times using Poseidon machine-learning communication framework.

Share this post