ECE 5930/6930: Distributed Machine Learning

Primary Textbooks

None. Course materials will span largely from different sources (e.g., papers, book chapters). However, the following materials will be helpful:

• Amir Beck, “First-Order Methods in Optimization,” Society for Industrial and Applied Mathematics, 2017, ISBN978-1-61197-498-0.
• L´eon Bottou, Curtis E. Frank, and Nocedal Jorge, “Optimization methods for large-scale machine learning,” SIAM review 60.2 (2018) (Free online copy: https://epubs.siam.org/doi/epdf/10.1137/16M1080173 )
• Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar, “Foundations of Machine Learning,” second edition, MIT press, 2018. (Free online copy: https://cs.nyu.edu/ mohri/mlbook/)

Prerequisites:

• MAE 5370 or
• CS 4320 or
• CS 5640 or
• CS 5665 or
• MATH 2270/2275 and MATH 2210 and MATH 5710

Short Description

This course offers theory of machine learning (ML) and optimizations. Students will learn different topics of first-order optimization using various traditional and ML algorithms, followed by large-scale distributed ML algorithms, privacy-preserving federated learning (FL) algorithms, and communication- and computation-efficient distributed learning.

Course Overview

1. Introduction (≈ 3 lectures)
   • General topics (e.g., convex and non-convex function, smoothness, strong convexity, empirical risk function, gradients, subgradients, hessians, Jacobian, generalization errors, level sets, etc.)
2. First-order methods (≈ 8 lectures)
   • Gradient descent, mini-batch SGD, and convergence analysis
   • Momentum and variance reduction methods
   • Subgradient methods
   • Mirror descent
   • Proximal gradient methods
3. Distributed optimization a nd machine learning (≈ 16 lectures)
   • Introduction to distributed machine learning
     1. Distributed SGD
   • Introduction to communication-efficient learning
     1. Federated averaging (FedAvg)
     2. Data and computation heterogeneity
        • FedProx
        • FedNova
        • Scaffold
        • FedProto
     3. Communication-efficient learning - advanced topics
        • Client selections and partial device participation
        • Gradient compression/quantization
     4. Computation-efficient learning
        • Pruning
        • Split learning
     5. Information leakage and advanced privacy-preserving methods (differential privacy, secure aggregation)

Learning Objectives

This course provides foundations to ML, first-order optimizations, and distributed ML. The key objectives are

• To study various first-order optimization techniques
• Study various centralized and distributed ML algorithms
• Study large-scale ML in distributed networks
• Learn communication- and computation-efficient distributed learning
• Study advanced privacy-preserving techniques to prevent gradient leakage

Additional and Substantive Learning Objectives for 6000-level Students

In accordance with Utah System of Higher Education (USHE), the graduate students who are enrolled in the ECE 6930 section will have additional and substantive learning objectives as listed below.

• Learn joint optimization techniques for running distributed ML in practical networks (e.g., LTE, 5G, WiFi-7)
• Emulate distributed ML algorithms in practical wireless networks (e.g., LTE, 5G, WiFi-7)

There will be additional question(s) added to at least two homework assignments and additional requirements in at least one

project to achieve the two additional substantive learning objectives listed above.

Policies