| Author: | Zhang, Jie |
| Title: | Towards efficient and personalized federated learning on heterogeneous environment |
| Advisors: | Guo, Song (COMP) |
| Degree: | Ph.D. |
| Year: | 2022 |
| Subject: | Machine learning Federated database systems Hong Kong Polytechnic University -- Dissertations |
| Department: | Department of Computing |
| Pages: | xxiv, 163 pages : color illustrations |
| Language: | English |
| Abstract: | Federated Learning (FL) has emerged as a new paradigm that enables multiple clients to collaboratively train a shared model without exposing any privacy-sensitive raw data. In a typical FL framework, each client trains the model with its own data and then uploads the model parameter (or update) rather than the raw data to the FL server for aggregation. The server aggregates local models and distributes the updated model to all clients for the next round training. These steps are repeated until the desired accuracy is achieved. Though FL has great advantages in communication efficiency and privacy protection, it still faces several challenges on heterogeneous environment, which can be summarized as three types: data heterogeneity, resource heterogeneity and model heterogeneity. Both of these issues decrease the efficiency of stochastic gradient descent (SGD) based training process in FL. In this thesis, we explore effective ways to address the above heterogeneity challenges and design novel solutions to improve the training efficiency of FL. First, we consider to mitigate the baneful impacts incurred by heterogeneous data. Different from conventional FL training frameworks that either consider the optimization of server-side aggregation or focus on improving the client-side training efficiency, which only lead to sub-optimal performance, we aim to design a new training framework to optimize the local training process and global aggregation simultaneously. Specifically, we apply the deep reinforcement learning (DRL) to adaptively adjust the batch size in each client to control the set of participant clients and their corresponding local updates. Moreover, we also provide thorough theoretical analysis to demonstrate the impact of adaptive batch size on the model convergence, which corroborates the effectiveness of the proposed adaptive batch size based training algorithm. Second, we consider another FL scenario, vertical FL (VFL), where the training data across multiple parties are partitioned by features while the parties share the same sample space. In this case, data heterogeneity may become the heterogeneous feature distributions among different parties. However, existing work on VFL focus on uniform feature distribution, which ignore the effect of unbalanced features on training performance. Thus, we aim to propose a new learning framework to improve the training efficiency of VFL on unbalanced features. Specifically, we theoretically analyze the convergence properties of VFL on unbalanced features and exhibit that the number of local updates conducted by each participant has a great effect on the convergence rate and computational complexity. Based on our theoretical findings, we formulate an optimization problem and derive the optimal solution by selecting an adaptive number of local training rounds for each party. Third, we take the model heterogeneity into consideration and study a more challenging task, Federated Distillation (FD), that extends classic Federated Learning (FL) to a more general training framework and enables model-heterogeneous collaborative learning. However, existing algorithms usually require manually selecting a set of shared input samples for each local model to produce soft-prediction for distillation. Worse still, such a selection is accompanied by certain careful deliberations or prior information on clients' private data distribution, which is not in line with the privacy-preserving characteristic of classic FL. We aim to propose a novel FD training framework to break the data dependency by properly designing a distributed generative adversarial network (GAN) between the server and clients that can synthesize shared feature representations to facilitate the FD training. Last, we consider to achieve personalized federated learning (pFL) with heterogeneous data and models. Unlike most existing and our previous works that produce a single global model for all clients, we focus on training a personalized global model for each client to fully take advantages of the diversity and inherent relationship of local data. Specifically, we formulate the aggregation procedure in original pFL into a personalized group knowledge transfer training algorithm, which enables each client to maintain a personalized soft prediction at the server side to guide the others' local training. By the linear combination of all local soft predictions using a knowledge coefficient matrix, the personalized soft prediction can be updated in a parameterized manner. In summary, this thesis aims to design efficient and personalized training frameworks for FL to tackle the heterogeneity problems. Extensive experiments on various datasets and models demonstrate the effectiveness of our proposed methods. |
| Rights: | All rights reserved |
| Access: | open access |
Copyright Undertaking
As a bona fide Library user, I declare that:
- I will abide by the rules and legal ordinances governing copyright regarding the use of the Database.
- I will use the Database for the purpose of my research or private study only and not for circulation or further reproduction or any other purpose.
- I agree to indemnify and hold the University harmless from and against any loss, damage, cost, liability or expenses arising from copyright infringement or unauthorized usage.
By downloading any item(s) listed above, you acknowledge that you have read and understood the copyright undertaking as stated above, and agree to be bound by all of its terms.
Please use this identifier to cite or link to this item:
https://theses.lib.polyu.edu.hk/handle/200/12056