Liquid Time-Constant Networks (LTCs)
Liquid Time-Constant Networks (LTCs) represent a cutting-edge approach in the development of continuous-time models using backpropagation through time (BPTT). This approach is part of a research initiative aimed at advancing the understanding and capabilities of neural networks operating in continuous time, as detailed in a scholarly paper available on arXiv. For practical and educational purposes, there is also a Pytorch version complete with tutorials in a sister repository, which provides further hands-on resources for users.
Available Models
The repository enables training of several continuous-time models, including:
- Liquid Time-Constant Networks: As described in the paper on arXiv.
- Neural Ordinary Differential Equations (ODEs): Documented in the NeurIPS conference paper.
- Continuous-time Recurrent Neural Networks (RNNs).
- Continuous-time Gated Recurrent Units (GRUs): Foundational understanding detailed in relevant research papers.
System Requirements
The models are developed and tested to run smoothly using TensorFlow 1.14.0 and Python 3 on Ubuntu systems (specifically versions 16.04 and 18.04). It's important to follow steps as outlined within such an environment for optimal performance and compatibility.
Preparation Steps
To begin using the repository, users should first download all necessary datasets by executing a specific script:
source download_datasets.sh
This command creates a directory labeled data, where all the requisite datasets are stored, setting the groundwork for model training and evaluation.
Training and Evaluating Models
The repository provides distinct Python scripts tailored for each dataset, facilitating targeted training and evaluation:
- gesture.py for hand gesture segmentation
- occupancy.py for room occupancy detection
- har.py for human activity recognition
- traffic.py for traffic volume prediction
- ozone.py for ozone level forecasting
Each script can be modified with the following parameters:
--model:Choose from lstm, ctrnn, ltc, ltc_rk, ltc_ex.--epochs:Number of training epochs, with a default of 200.--size:Number of hidden RNN units, defaulting to 32.--log:Frequency of validation metric evaluations.
Scripts conduct training for the specified epoch count, evaluate validation performance periodically, and upon completion, restore the best-performing checkpoint for test set evaluation. Results are systematically saved in the results folder in a CSV format.
To illustrate, one might train and evaluate a CT-RNN using:
python3 har.py --model ctrnn
Upon completion, a CSV file encapsulating training and evaluation metrics appears, providing insights into epochs, losses, and accuracy on various datasets.
Hyperparameters
Key hyperparameters used in the model training include:
- Minibatch size: 16, indicating the number of samples per gradient descent update.
- Learning rate: Adjusted for model (LTC: 0.01-0.02, others: 0.001).
- Hidden units: 32 in each model.
- Optimizer: Adam, with parameters beta_1 (0.9), beta_2 (0.999), epsilon (1e-08).
- Number of epochs: Maximum set at 200.
- BPTT length: 32 time-steps.
- ODE solver steps: Consistent with the input sampling period.
- Validation interval: Every epoch.
Trajectory Length Analysis
For trajectory length analysis, the main.m file is available for running desired settings tuneable in the code. This tool allows a deep dive into specificity regarding the length and behavior of trajectories within your data set.
The project provides an in-depth exploration of continuous-time neural models and offers the community a robust framework for experimentation and learning.