Work Flexibly with Recurrent Nets

The Wolfram Language provides high-level recurrent layers that are able to seamlessly and efficiently deal with batches of inputs with different lengths without sacrificing performance. Moreover, tools like NetStateObject and NetFoldOperator provide easy access to recurrent evaluation and definition of custom high-level recurrent layers.

First, demonstrate how to create and use a simple recurrent layer. Define a recurrent layer acting on variable-length sequences of 2-vectors.

Evaluate the layer on a batch of variable-length sequences.

Define a net that takes variable-length sequences of characters and produces a number for each sequence.

Evaluate the layer on a batch of variable-length strings.

Recurrent layers or nets defined through NetChain or NetGraph are stateless by default. Their initial internal states are vectors of zeros and are reset after each evaluation, so that repeated evaluations on the same input will yield the same result.

Recurrent layers of nets can be turned into stateful objects with NetStateObject. A NetStateObject keeps its own states after evaluation, so that repeated evaluations on the same input will, in general, produce different results. This a way to keep the internal state of a recurrent layer and, for example, generate text efficiently.

Now demonstrate how to create a custom recurrent layer from an existing NetGraph using NetFoldOperator. Define the core unit of a multiplicative LSTM.

Create the multiplicative LSTM.

Evaluate the layer on a batch of variable-length sequences of length-64 vectors.