{Seminar} @ CDS: 11th January : “Communication Compression for Distributed System—A Venture from Implementation Perspective.”

When there are a lot of training data, or the deep neural network is too large, distributed parallel training becomes essential, which refers to either data or model parallelism. In both cases, parallelism introduces various overheads. Network communication is one such significant overhead in large-scale distributed deep learning. In the form of sparsification or quantization of stochastic gradients, many compressed communication schemes have been proposed to minimize the problem.

 However, there exist many significant discrepancies between theory and practice. Theoretical analysis of most existing compression methods assumes many artifacts that generally do not hold in practice. For example, theoreticians motivated by general practice design compressors with attractive theoretical properties and show communication gains. Nevertheless, researchers and practitioners face a daunting task when choosing an appropriate compression technique despite the potential theoretical gains. The reason is that training speed and model accuracy depend on multiple factors such as the basic framework used for the implementation, the communication library, the network bandwidth, and the characteristics of the model, to name a few.

This talk will provide an overview of gradient compression methods for distributed deep learning from the implementation perspective. We show that if the practical implementation aspects are better realized, they can provide strong theoretical foundations of compressed communication that are deployable in the real-world training of stateof-the-art deep neural network models. Understanding communication compression from the practical point of view would be advantageous for creating foundational aspects of scalable, parallel, and distributed algorithms for challenging paradigms such as federated learning.

BIOGRAPHY

Aritra Dutta is an Assistant Professor at the University of Southern Denmark (SDU), Department of Mathematics and Computer Science (IMADA). Additionally, he is affiliated with the Center for AI Science and Applications (CASA) at SDU, and Pioneer Center for AI (P1)— an interdisciplinary center at the forefront of fundamental AI research in Denmark. Aritra received the B.S. degree in Mathematics from Presidency College, Calcutta, India, in 2006, the M.S. degree in Mathematics and Computing from the Indian Institute of Technology (IIT), Dhanbad, in 2008, and the second M.S. and Ph.D. degrees from the University of Central Florida (UCF), Orlando, in 2011 and 2016, respectively, both in Applied Mathematics. Before joining SDU, Dr. Dutta was a Postdoctoral Fellow at the Extreme Computing Research Center at KAUST, hosted by Prof. Panagiotis Kalnis. Before that, he was a Postdoctoral Fellow at the Visual Computing Center at KAUST with Prof. Peter Richtárik.