| Author: | Guo, Tao |
| Title: | Towards efficient and personalized collaborative edge learning on heterogeneous environment |
| Advisors: | Guo, Song (COMP) |
| Degree: | Ph.D. |
| Year: | 2024 |
| Subject: | Edge computing Artificial intelligence Machine learning Hong Kong Polytechnic University -- Dissertations |
| Department: | Department of Computing |
| Pages: | xxviii, 138 pages : color illustrations |
| Language: | English |
| Abstract: | With the development of artificial intelligence and the corresponding application by-products, Internet of Things (IoT) devices, e.g, smart watches, cell phones has engaged in people’s daily life. Such proliferation has resulted in the silos and isolation of local data. To make the full use of the long range personal data and ensure data privacy, a new computing paradigm arised, ,i.e, collaborative edge computing. Collaborative edge computing allows training a large neural model over a wide range based on their isolated datasets, e.g, Federated Learning (FL) and Split Learning (SL). As the diversified IoT edge devices prospered and thrived in recent years, several challenges present in today’s collaborative training process, including data heterogeneity, model heterogeneity and resource heterogeneity. Furthermore, as the model sizes become larger, foundation models, e.g, BERT, DALL-E, GPT-3, emerges as an assistant to the adaptation of of a wide range of downstream tasks. However, existing framework e.g, FL and SL, can not fully satisfy the heterogeneous requirements and keep up with the state-of-the-art models. Thus, there is a need to explore efficient collaborative edge learning frameworks for better performance. In this thesis, we explore novel frameworks and propose efficient and effective method to address the heterogeneous challenges above. First, we focus on tackling the model and resource heterogeneity across different IoT devices. Existing FL requires all users to store the entire model locally and employs an iterative update mechanism by exchanging model parameters repeatedly with the server. Such behavior has high demand for local computation and memory capabilities and can cause significant communication overhead. SL on the other hand address the resource by reallocating the most of the neural networks on the server, hence alleviate the majority of computation burden. However, the inherent training mechanism of SL, i.e, sequential training order between the edge and cloud, impede the optimal training efficiency. Thus, we come up with a novel collaborative learning framework, i.e, Tree Learning, to optimize the training efficiency across clients. Specifically, We allocate different layers for heterogeneous clients according to their different computation capacities and successfully facilitate all the participants to achieve the minimum synchronization overhead via a global level parallelism scheme. We further provide rigorous theoretical analysis of our framework and conduct extensive experiments across various datasets to validate the effectiveness. Second, we propose an innovate paradigm in FL to solve the existing challenges from a different perspective. Artificial intelligence (AI) nowadays has shown its success to train on broad data and produce large pretrained models (e.g., BERT, DALL-E, GPT-3) that can help human with timely and properly decisions. Recently, a paradigm shift arise when the pretrained models are utilized to adapt to the downstream tasks. And here we rename the aforementioned models the foundation models (FM). Inspired by the adaptation of FM in the centralized manners, we revisits the question of how FL mines the distributed data in iterative training rounds, and exploit the emerging foundation model (FM) to optimize the FL training. We propose PROMPTFL, other than training the whole model parameters, our framework works with the prompt vectors instead. Specifically, FL clients train prompts instead of a model, which can simultaneously exploit the insufficient local data and reduce the aggregation overhead. Experiments show the superiority of PROMPTFL from system feasibility, model performance and privacy preserving. Third, we consider to address the statistical heterogeneity in existing PROMPTFL to achieve a better personalization for local user modeling. Given the lightweight nature of prompt learning, researchers have migrated the paradigm from centralized to decentralized system to innovate the collaborative training framework of Federated Learning (FL), which we called PROMPTFL. However, current PROMPTFL mainly focuses on modeling user consensus and neglects the adaptation of local edge devices, leaving the personalization of PROMPTFL largely under-explored. Here we leverage the the unique advantage of multimodality in vision-language models by learning user consensus from linguistic space and adapting to user characteristics in visual space collaboratively. We also survey the personalization techniques in traditional pFL and reform them in current PROMPTFL scenario. Experiments show the superiority of our pFedPrompt against the alternative approaches with robust performance. Finally, we focus on the data utilization challenges on local client in existing PROMPTFL. Although PROMPTFL offers significant in benefiting computation, communication, and privacy over the existing frameworks, none of the researches analyze it from the data utilization manners. During the experiments, we found that federated prompting is a data-efficient but data-sensitive paradigm, and therefore, it is crucial to select data carefully for participation in the process. This work presents a local data selection strategy based on informative vectors that specify the most informative direction in the weight space of a vision-language model. Moving in this direction steers the behavior of pre-trained neurons precisely and improves performance on the local task. Experiments show that informative vectors offer promising robustness, making it a simple yet effective way to enhance the performance of federated prompting. In summary, this thesis aims to design efficient and personalized collaborative framework for edge devices on the heterogeneous environment. We identify challenges in the collaborative learning for edge devices and provide solutions from different perspectives to overcome the communication, computation, statistics and resource challenges. Extensive experiments show the effectiveness of our methods. |
| Rights: | All rights reserved |
| Access: | open access |
Copyright Undertaking
As a bona fide Library user, I declare that:
- I will abide by the rules and legal ordinances governing copyright regarding the use of the Database.
- I will use the Database for the purpose of my research or private study only and not for circulation or further reproduction or any other purpose.
- I agree to indemnify and hold the University harmless from and against any loss, damage, cost, liability or expenses arising from copyright infringement or unauthorized usage.
By downloading any item(s) listed above, you acknowledge that you have read and understood the copyright undertaking as stated above, and agree to be bound by all of its terms.
Please use this identifier to cite or link to this item:
https://theses.lib.polyu.edu.hk/handle/200/12906