| Author: | Liu, Qijiong |
| Title: | Toward next-generation AI-powered recommender systems : exploration of pretraining and generative approaches |
| Advisors: | Wu, Xiao-ming (COMP) |
| Degree: | Ph.D. |
| Year: | 2024 |
| Subject: | Recommender systems (Information filtering) Machine learning Artificial intelligence Energy conversion Hong Kong Polytechnic University -- Dissertations |
| Department: | Department of Computing |
| Pages: | xvii, 152 pages : color illustrations |
| Language: | English |
| Abstract: | Recommender systems stand out as a prime example of machine learning’s success, designed to ease the decision-making process for users by automatically suggesting items that match their individual tastes. With the exponential increase in online data and the imperative to navigate the deluge of information, these systems have gained widespread application across various platforms. Examples include Google News for curating news content, Spotify for music suggestions, Goodreads for book discoveries, and Taobao for shopping recommendations. However, boosting recommender systems with external knowledge may introduce two primary challenges: efficiently using diverse data sources and improving the quality of weak data. To address these challenges, this doctoral thesis proposes several pretraining and generative solutions for developing practical, efficient, and robust recommender systems. Firstly, we address the inefficiency of end-to-end news recommendation by introducing a pretraining approach. Recent studies suggest using pretrained language models (PLMs) to enhance semantic understanding in news recommendations. However, deploying large PLMs for real-time tasks poses significant challenges. We propose a plug-and-play pre-trainer that optimizes user and news encoders without relying on complex PLMs for direct inference. Instead, we utilize cached representations to enhance the efficacy of traditional ID-based models in click-through rate prediction. Building on this, we introduce the first Green AI benchmarking framework for news recommendations, providing a metric to balance recommendation accuracy with efficiency. This framework evaluates 30 base models and their variants, including traditional training methods and our efficient, only-encode-once (OLEO) paradigm. Secondly, we explore pretraining for item representation using discrete tokens in click-through rate (CTR) prediction. Incorporating item content into CTR models is challenging due to industrial constraints on time and space. Traditional content-encoding methods, which integrate user and item encoders into CTR models, prioritize space efficiency. Meanwhile, embedding-based methods convert item and user information into latent embeddings, trading off space for faster processing. Here, we introduce a semantic-token paradigm with a discrete semantic tokenization approach. This method allows for quick training and inference while using minimal memory, effectively addressing both time and space constraints. Thirdly, we focus on accelerating sequence generation for list completion tasks, which predict the continuation of item lists. Deploying Transformer-based, autoregressive models in real-time applications is often impractical due to their slow generation process. To enhance both the efficiency and quality of inference, we adopt a non-autoregressive mechanism that decodes items simultaneously. We also implement a two-stage classifier to further quicken the decoding. Additionally, we incorporate curriculum learning to refine our training process Fourthly, we examine feature generation in ID-based recommendation scenarios, where category information significantly enhances system personalization and quality. However, consistent availability of item category information remains a challenge. We propose an automatic method for learning and generating entity category trees using a differentiable vector quantization framework, which is highly adaptable for integration into both sequential and non-sequential recommender systems. Finally, we utilize large language models (LLMs) to enhance the robustness of item and user information in recommender systems. Given the challenges in comprehending item content, LLMs with their extensive pretraining and deep semantic knowledge offer significant advantages. We investigate using both open- and closed-source LLMs to improve content-based recommendations. Open-source LLMs serve as content encoders to deepen content representation, while closed-source LLMs are used with prompting techniques to enrich training data. |
| Rights: | All rights reserved |
| Access: | open access |
Copyright Undertaking
As a bona fide Library user, I declare that:
- I will abide by the rules and legal ordinances governing copyright regarding the use of the Database.
- I will use the Database for the purpose of my research or private study only and not for circulation or further reproduction or any other purpose.
- I agree to indemnify and hold the University harmless from and against any loss, damage, cost, liability or expenses arising from copyright infringement or unauthorized usage.
By downloading any item(s) listed above, you acknowledge that you have read and understood the copyright undertaking as stated above, and agree to be bound by all of its terms.
Please use this identifier to cite or link to this item:
https://theses.lib.polyu.edu.hk/handle/200/13274