Author: Zhou, Qihua
Title: Towards efficient tiny machine learning systems for ubiquitous edge intelligence
Advisors: Guo, Song (COMP)
Degree: Ph.D.
Year: 2023
Subject: Machine learning
Artificial intelligence
Edge computing
Hong Kong Polytechnic University -- Dissertations
Department: Department of Computing
Pages: xxii, 189 pages : color illustrations
Language: English
Abstract: Modern machine learning (ML) applications are often deployed in the cloud en­vironment to exploit the computational power of clusters. However, traditional in-cloud computing schemes cannot satisfy the demands of emerging edge intelligence scenarios, including providing personalized models, protecting user privacy, adapt­ing to real-time tasks and saving resource costs. In order to conquer the limitations of conventional in-cloud computing, a new trend is to utilize the on-device learning paradigm, which makes the end-to-end ML procedure closer to edge devices. As a result, the promising advantages of on-device learning promote the rise of Tiny Machine Learning (TinyML) systems, a scope that focuses on developing ML algo­rithms and models on resource-constrained edge devices, e.g., microcontrollers, IoT sensors and embedded devices. The term “Tiny” highlights the limited processing capacity, memory volume, and energy resources on these devices. As discussed by the research background in §1.1, TinyML has become an important research topic due to the growth of edge intelligence applications, including smart homes, wear­ables, robotics, and healthcare services. By applying TinyML systems on ubiq­uitous edge devices, developers and researchers can effectively reduce inference latency, save resource costs, increase usage experience and protect user privacy.
However, implementing a high-performance TinyML system is not easy in practice. We need to dive into the fundamental architecture design and frame­work implementation, standing in the perspective of system implementation in a full stack, including reducing data scale, model complexity, computational over­head, and communication traffic. Aiming at building an efficient TinyML sys­tem, we summarize three core challenges of system design and implementation in §1.2. These challenges motivate the design principle of our methodologies, cor­responding to the major contributions of this thesis in §1.3. More precisely, by conducting a comprehensive background review of TinyML systems in Chap. 2, we intend to optimize the system design in three aspects: (1) leveraging the INT8 quantization-aware training to break computational resource constraints on edge devices in Chap. 3, (2) utilizing the hierarchical channel-spatial encoding to allevi­ate communication bottleneck during edge-cloud collaboration in Chap. 4 and (3) exploring the patch automatic skip scheme to improve on-device model execution efficiency in Chap. 5.
First, as will be discussed in Chap. 3, we focus on breaking the constraints of limited resources, alleviating computational overhead and discussing how to im­prove the computational speed of on-device learning. We show that employing the 8-bit fixed-point (INT8) quantization in both forward and backward passes over a deep model is a promising way to enable tiny on-device learning in practice. The key to an efficient quantization-aware training method is to exploit the hardware-level enabled acceleration while preserving the training quality in each layer. We implement our method in Octo, a lightweight cross-platform system for tiny on-device learning. Experiments on commercial AI chips show that Octo holds higher training efficiency over state-of-the-art quantization training methods, while achiev­ing adequate processing speedup and memory reduction over full-precision training.
Second, as will be discussed in Chap. 4, we also cover continuous data ana­lytics and video streaming applications. In this condition, improving communica­tion efficiency by reducing traffic size is one of the most crucial issues for realistic deployment. Existing systems mainly compress features at the pixel level and ig­nore the characteristics of feature structure, which could be further exploited for more efficient compression. In this work, we take new insights into implementing scalable CL systems through a hierarchical compression on features, termed Stripe-wise Group Quantization (SGQ). Different from previous unstructured quantization methods, SGQ captures both channel and spatial similarity in pixels, and simultane­ously encodes features in these two levels to gain a much higher compression ratio. Experiments show that SGQ can effectively alleviate the communication bottleneck with much less traffic, while still preserving the learning accuracy as the original full-precision version. This verifies that SGQ can be applied to a wide spectrum of edge intelligence applications.
Third, as will be discussed in Chap. 5, real-time video perception tasks are often challenging on resource-constrained edge devices due to the issues of accu­racy drop and hardware overhead, where saving computations is the key to per­formance improvement. Existing methods mainly rely on domain-specific neural chips or priorly searched models, which require specialized optimization accord­ing to different task properties. These limitations motivate us to design a general and task-independent methodology, called Patch Automatic Skip Scheme (PASS), which supports diverse video perception settings by decoupling acceleration and tasks. The gist is to capture inter-frame correlations and skip redundant computa­tions at the patch level, where the patch is a non-overlapping square block in visual. Experiments show that applying PASS can benefit the on-device video perception performance, including processing speedups, memory reduction, computation sav­ing, model quality, prediction stability and environmental adaptation. PASS can generalize to real-time video streams on commodity edge devices, e.g., NVIDIA Jetson Nano, with efficient performance in realistic deployment.
In summary, TinyML is an emerging technique to pave the last mile of enabling edge intelligence, which eliminates the limitations of conventional in-cloud comput­ing where dozens of computational capacities and memories are needed. Building an efficient TinyML system requires breaking the constraints of limited resources and alleviating computational overhead. Therefore, this thesis presents a software and hardware synergy for TinyML system implementation. Extensive evaluation on commercial edge devices shows the remarkable performance improvement of our proposed system over existing solutions.
Rights: All rights reserved
Access: open access

Files in This Item:
File Description SizeFormat 
7148.pdfFor All Users9.17 MBAdobe PDFView/Open

Copyright Undertaking

As a bona fide Library user, I declare that:

  1. I will abide by the rules and legal ordinances governing copyright regarding the use of the Database.
  2. I will use the Database for the purpose of my research or private study only and not for circulation or further reproduction or any other purpose.
  3. I agree to indemnify and hold the University harmless from and against any loss, damage, cost, liability or expenses arising from copyright infringement or unauthorized usage.

By downloading any item(s) listed above, you acknowledge that you have read and understood the copyright undertaking as stated above, and agree to be bound by all of its terms.

Show full item record

Please use this identifier to cite or link to this item: