84 research outputs found
How to collaborate: Towards maximizing the generalization performance in cross-silo federated learning
No full textOptimizing federated learning with heterogeneous and unreliable clients
No full textThe explosive development of artificial intelligence applications brings about massive data generated on edge devices. These data are valuable for training machine learning models but are also sensitive, raising privacy concerns. Federated learning (FL) emerges as a promising privacy-preserving solution for distributed training, which enables devices to collaboratively train machine learning models without sharing raw data. Despite its benefits, the training efficiency of FL faces several main obstacles, including data heterogeneity, communication bottleneck, and unreliable devices. In this thesis, we pro-pose novel algorithms to address these challenges with a particular focus on dealing with the heterogeneity and unreliability of clients in FL. First, we introduce a semi-decentralized federated edge learning (SD-FEEL) system to re-duce communication latency by leveraging efficient communications between edge servers. We theoretically show the convergence of SD-FEEL and investigate the impact of system parameters on its performance. Moreover, we propose an asynchronous algorithm with staleness-aware aggregation for heterogeneous devices, demonstrating enhanced convergence and model quality on image classification tasks. Next, to address the straggler issue in FL, we develop a stochastic coded FL (SCFL) frame-work that utilizes coded data to compensate for missing gradients. We analyze the privacy-performance tradeoff in SCFL and demonstrate its superior tradeoff compared to conventional coded FL algorithms. We further design an incentive mechanism to encourage clients to reduce noise in coded data, showing that the mechanism effectively boosts model performance. Finally, we identify the vulnerability of the classical FedAvg algorithm to client dropouts. To tackle this problem, we propose the MimiC algorithm, which adjusts local updates to mimic the central update. We prove the convergence of MimiC under various dropout scenarios and show its superior performance in simulations.</p
Asynchronous Semi-Decentralized Federated Edge Learning for Heterogeneous Clients
No full textFederated edge learning (FEEL) has drawn much attention as a privacy-preserving distributed learning framework for mobile edge networks. In this work, we investigate a novel semi-decentralized FEEL (SD-FEEL) architecture where multiple edge servers collaborate to incorporate more data from edge devices in training. Despite the low training latency enabled by fast edge aggregation, the device heterogeneity in computational resources deteriorates the efficiency. This paper proposes an asynchronous training algorithm to overcome this issue in SD-FEEL, where edge servers are allowed to independently set deadlines for the associated client nodes and trigger the model aggregation. To deal with different levels of model staleness, we design a staleness-aware aggregation scheme and analyze its convergence. Simulation results demonstrate the effectiveness of our proposed algorithm in achieving faster convergence and better learning performance than synchronous training.</p
MimiC: Combating Client Dropouts in Federated Learning by Mimicking Central Updates
Get PDFFederated learning (FL) is a promising framework for privacy-preserving
collaborative learning, where model training tasks are distributed to clients
and only the model updates need to be collected at a server. However, when
being deployed at mobile edge networks, clients may have unpredictable
availability and drop out of the training process, which hinders the
convergence of FL. This paper tackles such a critical challenge. Specifically,
we first investigate the convergence of the classical FedAvg algorithm with
arbitrary client dropouts. We find that with the common choice of a decaying
learning rate, FedAvg oscillates around a stationary point of the global loss
function, which is caused by the divergence between the aggregated and desired
central update. Motivated by this new observation, we then design a novel
training algorithm named MimiC, where the server modifies each received model
update based on the previous ones. The proposed modification of the received
model updates mimics the imaginary central update irrespective of dropout
clients. The theoretical analysis of MimiC shows that divergence between the
aggregated and central update diminishes with proper learning rates, leading to
its convergence. Simulation results further demonstrate that MimiC maintains
stable convergence performance and learns better models than the baseline
methods
How to Collaborate: Towards Maximizing the Generalization Performance in Cross-Silo Federated Learning
Get PDFThe work of J. Zhang was supported by the
Hong Kong Research Grants Council under the Areas of Excellence scheme
grant AoE/E-601/22-R and NSFC/RGC Collaborative Research Scheme grant
CRS HKUST603/22. The work of M. Kountouris was supported by the
European Research Council (ERC) under the European Union’s Horizon
2020 Research and Innovation Programme (Grant agreement No. 101003431).Federated learning (FL) has attracted vivid attention as a privacy-preserving distributed learning framework. In this work,
we focus on cross-silo FL, where clients become the model owners after training and are only concerned about the model’s
generalization performance on their local data. Due to the data heterogeneity issue, asking all the clients to join a single FL training
process may result in model performance degradation. To investigate the effectiveness of collaboration, we first derive a generalization
bound for each client when collaborating with others or when training independently. We show that the generalization performance of a
client can be improved by collaborating with other clients that have more training data and similar data distributions. Our analysis
allows us to formulate a client utility maximization problem by partitioning clients into multiple collaborating groups. A hierarchical
clustering-based collaborative training (HCCT) scheme is then proposed, which does not need to fix in advance the number of groups.
We further analyze the convergence of HCCT for general non-convex loss functions which unveils the effect of data similarity among
clients. Extensive simulations show that HCCT achieves better generalization performance than baseline schemes, whereas it
degenerates to independent training and conventional FL in specific scenarios.Hong Kong Research Grants Council AoE/E-601/22-RNSFC/RGC Collaborative Research Scheme CRS HKUST603/22European Research Council (ERC)Union’s Horizon 2020 No. 10100343
How to Collaborate: Towards Maximizing the Generalization Performance in Cross-Silo Federated Learning
Get PDFFederated learning (FL) has attracted vivid attention as a privacy-preserving distributed learning framework. In this work, we focus on cross-silo FL, where clients become the model owners after training and are only concerned about the model\u27s generalization performance on their local data. Due to the data heterogeneity issue, asking all the clients to join a single FL training process may result in model performance degradation. To investigate the effectiveness of collaboration, we first derive a generalization bound for each client when collaborating with others or when training independently. We show that the generalization performance of a client can be improved only by collaborating with other clients that have more training data and similar data distribution. Our analysis allows us to formulate a client utility maximization problem by partitioning clients into multiple collaborating groups. A hierarchical clustering-based collaborative training (HCCT) scheme is then proposed, which does not need to fix in advance the number of groups. We further analyze the convergence of HCCT for general non-convex loss functions which unveils the effect of data similarity among clients. Extensive simulations show that HCCT achieves better generalization performance than baseline schemes, whereas it degenerates to independent training and conventional FL in specific scenarios
How to Collaborate: Towards Maximizing the Generalization Performance in Cross-Silo Federated Learning
No full textFederated learning (FL) has attracted vivid attention as a privacy-preserving distributed learning framework. In this work, we focus on cross-silo FL, where clients become the model owners after training and are only concerned about the model's generalization performance on their local data. Due to the data heterogeneity issue, asking all the clients to join a single FL training process may result in model performance degradation. To investigate the effectiveness of collaboration, we first derive a generalization bound for each client when collaborating with others or when training independently. We show that the generalization performance of a client can be improved by collaborating with other clients that have more training data and similar data distributions. Our analysis allows us to formulate a client utility maximization problem by partitioning clients into multiple collaborating groups. A hierarchical clustering-based collaborative training (HCCT) scheme is then proposed, which does not need to fix in advance the number of groups. We further analyze the convergence of HCCT for general non-convex loss functions which unveils the effect of data similarity among clients. Extensive simulations show that HCCT achieves better generalization performance than baseline schemes, whereas it degenerates to independent training and conventional FL in specific scenarios. © 2002-2012 IEEE
- …
