Implementation of biologically inspired efficient deep learning models (available)

Starting Date: June 2024
Prerequisites: Having good knowledge of Python programming is essential. Familiarity with ML frameworks such as PyTorch/JAX is also important. Some knowledge of CUDA (or willingness to learn) is needed if interested in low-level (not all) implementations.
Will results be assigned to University: No

As deep learning models continue to grow in size and complexity to tackle increasingly difficult tasks, the need for efficient and scalable models becomes ever more important. Extremely large language models like GPT-4 require massive computational resources and expensive hardware to train and run. This makes them impractical to deploy at scale in many real-world applications. More efficient models that can achieve high performance with far fewer parameters and operations are needed to enable deep learning capabilities on resource-constrained devices like mobile phones, embedded systems, and Internet of Things sensors.

Spiking neural networks [1] are a type of artificial neural network that more closely mimic the behaviour of biological neurons in the brain. Instead of using continuous-valued activations like traditional neural networks, spiking neural networks use discrete spike events to transmit information between neurons, hence providing a more efficient alternative to the current deep learning models.

In this project, the goal is to create efficient implementations that can take advantage of activity and parameter sparsity in biologically inspired deep learning models such as spiking neural networks [1] or the EGRU [2] potentially using recent CUDA features such as sparsity support [3] or other high-level framework optimisations [4].

Students are welcome to email (anand.subramoney@rhul.ac.uk) for informal discussions.

Reading:

  1. Pfeiffer, Michael, and Thomas Pfeil. “Deep learning with spiking neurons: opportunities and challenges.” Frontiers in neuroscience (2018): 774. (link)
  2. Subramoney, A., Nazeer, K.K., Schöne, M., Mayr, C., Kappel, D., 2023. Efficient recurrent architectures through activity sparsity and sparse back-propagation through time. Presented at the International Conference on Learning Representations 2023. (link)
  3. Accelerating Inference with Sparsity Using the NVIDIA Ampere Architecture and NVIDIA TensorRT (link)
  4. Spyx: Spiking Neural Networks in JAX (link)