第17卷第3期 智能系统学报 Vol.17 No.3 2022年5月 CAAI Transactions on Intelligent Systems May 2022 D0:10.11992/tis.202106029 网络出版地址：htps:/ns.cnki.net/kcms/detail/23.1538.TP.20211213.1548.002.html 基于隐式随机梯度下降优化的联邦学习 窦勇敢2，袁晓形 (1.南京信息工程大学自动化学院，江苏南京210044：2.江苏省大数据分析技术重点实验室，江苏南京 210044) 摘要：联邦学习是一种分布式机器学习范式，中央服务器通过协作大量远程设备训练一个最优的全局模 型。目前联邦学习主要存在系统异构性和数据异构性这两个关键挑战。本文主要针对异构性导致的全局模型 收敛慢甚至无法收敛的问题，提出基于隐式随机梯度下降优化的联邦学习算法。与传统联邦学习更新方式不 同，本文利用本地上传的模型参数近似求出平均全局梯度，同时避免求解一阶导数，通过梯度下降来更新全局 模型参数，使全局模型能够在较少的通信轮数下达到更快更稳定的收敛结果。在实验中，模拟了不同等级的异 构环境，本文提出的算法比FedProx和FedAvg均表现出更快更稳定的收敛结果。在相同收敛结果的前提下， 本文的方法在高度异构的合成数据集上比F©dProx通信轮数减少近50%，显著提升了联邦学习的稳定性和鲁 棒性。 关键词：联邦学习；分布式机器学习；中央服务器；全局模型：隐式随机梯度下降：数据异构：系统异构：优化算 法：快速收敛 中图分类号：TP8文献标志码：A文章编号：1673-4785(2022)03-0488-08 中文引用格式：窦勇敢，袁晓彤.基于隐式随机梯度下降优化的联邦学习.智能系统学报，2022,17(3)：488-495. 英文引用格式：DOU Yonggan,YUAN Xiaotong.Federated learning with implicit stochastic gradient descent optimization. CAAI transactions on intelligent systems,2022,17(3):488-495. Federated learning with implicit stochastic gradient descent optimization DOU Yonggan",YUAN Xiaotong'2 (1.School of Automation,Nanjing University of Information Science and Technology,Nanjing 210044,China;2.Jiangsu Key Laboratory of Big Data Analysis Technology,Nanjing 210044,China) Abstract:Federated learning is a distributed machine learning paradigm.The central server trains an optimal global model by collaborating with numerous remote devices.Presently,there are two key challenges faced by federated learn- ing:system and statistical heterogeneities.Herein,we mainly focus on the slow convergence of the global model or when it even fails to converge due to system and statistical heterogeneities.We propose a federated learning optimiza- tion algorithm based on implicit stochastic gradient descent optimization,which is different from the traditional method of updating in federated learning.We use the locally uploaded model parameters to approximate the average global gradient and to avoid solving the first-order and update the global model parameter via gradient descent.This is per- formed so that the global model can achieve faster and more stable convergence results with fewer communication rounds.In the experiment,different levels of heterogeneous settings were simulated.The proposed algorithm shows con- siderably faster and more stable convergence behavior than FedAvg and FedProx.In the premise of the same conver- gence results,the experimental results show that the proposed method reduces the number of communication rounds by approximately 50%compared with Fedprox in highly heterogeneous synthetic datasets.This considerably improves the stability and robustness of federated learning. Keywords:federated learning:distributed machine learning;central server;global model;implicit stochastic gradient descent;statistical heterogeneity;systems heterogeneity:optimization algorithm;faster convergence 收稿日期：2021-06-18.网络出版日期：2021-12-14. 近些年来，随着深度学习的兴起，人们看到了 基金项目：国家自然科学基金项目(61876090,61936005)：科技创 新2030-“新一代人工智能”重大项目(2018AAA0100400) 人工智能的巨大潜力，同时希望人工智能技术应 通信作者：袁晓彤.E-mail:xtyuanl980@gmail.com 用到更复杂和尖端的领域。而现实状况是数据分DOI: 10.11992/tis.202106029 网络出版地址: https://kns.cnki.net/kcms/detail/23.1538.TP.20211213.1548.002.html 基于隐式随机梯度下降优化的联邦学习 窦勇敢1,2，袁晓彤1,2 （1. 南京信息工程大学 自动化学院，江苏 南京 210044; 2. 江苏省大数据分析技术重点实验室，江苏 南京 210044） 摘 要 ：联邦学习是一种分布式机器学习范式，中央服务器通过协作大量远程设备训练一个最优的全局模 型。目前联邦学习主要存在系统异构性和数据异构性这两个关键挑战。本文主要针对异构性导致的全局模型 收敛慢甚至无法收敛的问题，提出基于隐式随机梯度下降优化的联邦学习算法。与传统联邦学习更新方式不 同，本文利用本地上传的模型参数近似求出平均全局梯度，同时避免求解一阶导数，通过梯度下降来更新全局 模型参数，使全局模型能够在较少的通信轮数下达到更快更稳定的收敛结果。在实验中，模拟了不同等级的异 构环境，本文提出的算法比 FedProx 和 FedAvg 均表现出更快更稳定的收敛结果。在相同收敛结果的前提下， 本文的方法在高度异构的合成数据集上比 FedProx 通信轮数减少近 50%，显著提升了联邦学习的稳定性和鲁 棒性。 关键词：联邦学习；分布式机器学习；中央服务器；全局模型；隐式随机梯度下降；数据异构；系统异构；优化算 法；快速收敛 中图分类号：TP8 文献标志码：A 文章编号：1673−4785(2022)03−0488−08 中文引用格式：窦勇敢, 袁晓彤. 基于隐式随机梯度下降优化的联邦学习 [J]. 智能系统学报, 2022, 17(3): 488–495. 英文引用格式：DOU Yonggan, YUAN Xiaotong. Federated learning with implicit stochastic gradient descent optimization[J]. CAAI transactions on intelligent systems, 2022, 17(3): 488–495. Federated learning with implicit stochastic gradient descent optimization DOU Yonggan1,2 ，YUAN Xiaotong1,2 (1. School of Automation, Nanjing University of Information Science and Technology, Nanjing 210044, China; 2. Jiangsu Key Laboratory of Big Data Analysis Technology, Nanjing 210044, China) Abstract: Federated learning is a distributed machine learning paradigm. The central server trains an optimal global model by collaborating with numerous remote devices. Presently, there are two key challenges faced by federated learn￾ing: system and statistical heterogeneities. Herein, we mainly focus on the slow convergence of the global model or when it even fails to converge due to system and statistical heterogeneities. We propose a federated learning optimiza￾tion algorithm based on implicit stochastic gradient descent optimization, which is different from the traditional method of updating in federated learning. We use the locally uploaded model parameters to approximate the average global gradient and to avoid solving the first-order and update the global model parameter via gradient descent. This is per￾formed so that the global model can achieve faster and more stable convergence results with fewer communication rounds. In the experiment, different levels of heterogeneous settings were simulated. The proposed algorithm shows con￾siderably faster and more stable convergence behavior than FedAvg and FedProx. In the premise of the same conver￾gence results, the experimental results show that the proposed method reduces the number of communication rounds by approximately 50% compared with Fedprox in highly heterogeneous synthetic datasets. This considerably improves the stability and robustness of federated learning. Keywords: federated learning; distributed machine learning; central server; global model; implicit stochastic gradient descent; statistical heterogeneity; systems heterogeneity; optimization algorithm; faster convergence 近些年来，随着深度学习的兴起，人们看到了 人工智能的巨大潜力，同时希望人工智能技术应 用到更复杂和尖端的领域。而现实状况是数据分 收稿日期：2021−06−18. 网络出版日期：2021−12−14. 基金项目：国家自然科学基金项目（61876090,61936005）；科技创 新 2030–“新一代人工智能”重大项目（2018AAA0100400）. 通信作者：袁晓彤. E-mail: xtyuan1980@gmail.com. 第 17 卷第 3 期 智 能 系 统 学 报 Vol.17 No.3 2022 年 5 月 CAAI Transactions on Intelligent Systems May 2022