Efficient deep learning with sparsity (available)

Starting Date: June 2027
Prerequisites: Having good knowledge of Python programming is essential. Familiarity with ML frameworks such as PyTorch/JAX is also important. Some knowledge of CUDA or C++/Rust/Elixir/MPI (or willingness to learn) is needed if interested in low-level implementations.
Will results be assigned to University: No

As deep learning models continue to grow in size and complexity to tackle increasingly difficult tasks, the need for efficient and scalable models becomes ever more important. Extremely large language models require massive computational resources and expensive hardware to train and run. More efficient models that can achieve high performance with far fewer parameters and can enable deep learning capabilities on resource-constrained devices like mobile phones, embedded systems, and Internet of Things sensors.

Spiking neural networks [1] and event-based neural networks [2] are a type of artificial neural network that more closely mimic the behaviour of biological neurons in the brain. Instead of using continuous-valued activations like traditional neural networks, spiking neural networks use discrete spike events to transmit information between neurons, hence providing a more efficient alternative to the current deep learning models.

This project will explore creating efficient implementations that can take advantage of activity and parameter sparsity in deep learning models such as spiking neural networks [1] or the EGRU [2].

The implementations can be done in one of these three substrates:

  • Using CUDA features such as 2:4 or block sparsity support [3].
  • High performance implementations in C++, Rust, Elixir or MPI.
  • Implementations on FPGAs.

Students are welcome to email (anand.subramoney@rhul.ac.uk) for informal discussions.

Reading:

  1. Pfeiffer, Michael, and Thomas Pfeil. “Deep learning with spiking neurons: opportunities and challenges.” Frontiers in neuroscience (2018): 774. (link)
  2. Subramoney, A., Nazeer, K.K., Schöne, M., Mayr, C., Kappel, D., 2023. Efficient recurrent architectures through activity sparsity and sparse back-propagation through time. Presented at the International Conference on Learning Representations 2023. (link)
  3. Accelerating Inference with Sparsity Using the NVIDIA Ampere Architecture and NVIDIA TensorRT (link)