| Author: | Wang, Yuxiang |
| Title: | Property inference attacks and defences in multimodal learning models |
| Advisors: | Hu, Haibo (EEE) |
| Degree: | M.Sc. |
| Year: | 2024 |
| Department: | Department of Electrical and Electronic Engineering |
| Pages: | 1 volume (unpaged) : color illustrations |
| Language: | English |
| Abstract: | The rapid advancement of artificial intelligence has positioned multi-modal learning as a pivotal research domain, focusing on the integration of diverse data modalities such as text, images, and audio to enhance model performance and generalization capabilities. This study delves into the research and application of property inference attacks within the context of multi-modal data models, emphasizing the significance of privacy protection in machine learning. We present a comprehensive analysis of multi-modal data fusion techniques, property inference attacks, and privacy protection strategies, offering a holistic framework that addresses the challenges and opportunities in this field. Our contributions include a deep learning-based multi-modal feature extraction method that leverages BERT and CNNs for text and image modalities, respectively, and an innovative multi-modal data fusion framework that adaptively adjusts weights to optimize attribute extraction. Furthermore, we propose a differential privacy-based approach to safeguard user privacy without compromising model performance, highlighting the balance between privacy and model accuracy in multi-modal learning applications. This research provides a thorough examination of the technical intricacies of multi-modal learning frameworks, particularly in the context of property inference attacks. We detail the construction and training of shadow models and meta-classifiers, which are instrumental in simulating target model behavior and inferring sensitive attributes from model predictions. Our work also extends to global attribute inference, which aims to reveal the joint distribution of multiple attributes within training data, thereby exacerbating the privacy risks associated with multi-modal models. The study concludes with a discussion on defensive mechanisms against privacy attacks, advocating for a strategic integration of these defenses to bolster the privacy and security of multi-modal systems in real-world deployments. Through these contributions, we aim to foster a deeper understanding of the interplay between multi-modal learning, privacy protection, and the evolving landscape of property inference attacks. |
| Rights: | All rights reserved |
| Access: | restricted access |
Files in This Item:
| File | Description | Size | Format | |
|---|---|---|---|---|
| 8305.pdf | For All Users (off-campus access for PolyU Staff & Students only) | 1.25 MB | Adobe PDF | View/Open |
Copyright Undertaking
As a bona fide Library user, I declare that:
- I will abide by the rules and legal ordinances governing copyright regarding the use of the Database.
- I will use the Database for the purpose of my research or private study only and not for circulation or further reproduction or any other purpose.
- I agree to indemnify and hold the University harmless from and against any loss, damage, cost, liability or expenses arising from copyright infringement or unauthorized usage.
By downloading any item(s) listed above, you acknowledge that you have read and understood the copyright undertaking as stated above, and agree to be bound by all of its terms.
Please use this identifier to cite or link to this item:
https://theses.lib.polyu.edu.hk/handle/200/13897