keras-tcn

Keras TCN

Keras TCN, short for Keras Temporal Convolutional Network, is a library designed to implement temporal convolutional networks (TCNs) using the Keras framework. This architecture is especially popular for sequence modeling tasks, outperforming traditional recurrent models like LSTMs and GRUs in many cases. The project cites a robust paper on TCNs, which can be accessed here.

Installation

Keras TCN is compatible with multiple versions of TensorFlow ranging from 2.9 to 2.17 as of July 2024. To install, simply run the following command:

pip install keras-tcn

For those who have TensorFlow and Numpy installed, they can opt to install without additional dependencies:

pip install keras-tcn --no-dependencies

For MacOS M1 users, there's a specific installation process to ensure proper compilation:

pip install --no-binary keras-tcn keras-tcn

Why Choose TCN over LSTM/GRU?

1. Extended Memory: TCNs have a longer memory span compared to recurrent architectures of the same size.
2. Performance: They outperform LSTM/GRU on various long time series tasks like Sequential MNIST and Word-level PTB.
3. Features: TCNs leverage parallelism through convolutional layers, boast flexible receptive field sizes, and maintain stable gradients.

TCN Layer Overview

TCN Class

The TCN class in Keras TCN offers numerous parameters for customized configuration:

• nb_filters: Determines the number of filters in the convolutional layers, akin to units in LSTM.
• kernel_size: Controls the spatial focus of convolutional operations.
• dilations: Defines the depth of the TCN layer using a list, e.g., [1, 2, 4, 8, 16, 32].
• padding: Options include causal for causal networks or same for non-causal setups.
• use_skip_connections: Allows for input connections to each residual block, aiding gradient flow.
• dropout_rate, activation, kernel_initializer: Various configurations to tailor the TCN to specific task needs.

Input and Output Shapes

The input shape for a TCN is a 3D tensor: (batch_size, timesteps, input_dim), while the output shape varies depending on whether full sequences are returned.

Configuration Tips

• Filters and Kernel Size: A larger number of filters or kernel size can enhance performance but may increase the risk of overfitting.
• Dilations: Strategically adjust dilations based on the sequence length and periodicity.
• Normalization: Utilize batch, layer, or weight normalization for larger networks or datasets.

Receptive Field

Understanding the receptive field is crucial for effectively utilizing TCNs. Simply put, it determines the window of time steps considered by each filter, impacting the TCN's ability to memorize sequences.

Example Tasks

Keras TCN provides example tasks which illustrate its capabilities:

• Word PTB: A language modeling task demonstrating TCN's edge over LSTMs.
• Adding and Copy Memory Tasks: Tasks that test the TCN's sequence modeling abilities.
• Sequential MNIST: A challenge that treats MNIST images as sequences for classification.

Implementation Results

The examples showcase reproducibility on GPUs and demonstrate impressive results across various tasks, emphasizing TCN’s efficiency and effectiveness.

Non-causal TCN

For scenarios where predictions can consider future data, non-causal TCNs may be used by altering the padding parameter.

Final Thoughts

TCNs offer a compelling alternative to traditional recurrent networks with unique features suitable for sequence modeling. Their flexible architecture allows them to excel in tasks that traditional models may struggle with. For those looking to explore or implement temporal convolutional networks using Keras, Keras TCN presents a robust and adaptable solution.

For detailed exploration and to contribute, check out the Keras TCN GitHub repository.