| Author: | Mehmood, Imran |
| Title: | Automated and non-invasive mental fatigue assessment of construction equipment operators |
| Advisors: | Li, Heng (BRE) |
| Degree: | Ph.D. |
| Year: | 2024 |
| Subject: | Mental fatigue Construction equipment operators Construction workers -- Health and hygiene Hong Kong Polytechnic University -- Dissertations |
| Department: | Department of Building and Real Estate |
| Pages: | xix, 153 pages : color illustrations |
| Language: | English |
| Abstract: | The construction industry plays a vital role in enhancing and advancing the infrastructure, contributing significantly to its development. However, at the same time it is labour-intensive and famous for poor safety records around the globe. Despite the positive effects it brings, ensuring the safety of construction site workers remains an unresolved issue and a top priority. Construction workers have to perform repetitive and mentally demanding tasks. In construction industry, various heavy equipment, such as excavators, tower cranes, trucks, loaders, is used for construction tasks. The operations of these construction equipment are repetitive and mentally demanding. The equipment operators have to work for prolonged hours to complete ongoing equipment operations that require constant attention from the operator and can be mentally challenging. Extended periods of operating construction equipment can induce mental fatigue as it demands continuous attention from the operator. This can lead to an elevated risk of accidents caused by human errors and compromised health for operators due to lapses in attention. To mitigate the risk of accidents and safeguard the well-being of operators, it is crucial to consistently and promptly monitor their mental fatigue in real time. Mental fatigue poses a notable risk factor for on-site incidents and accidents, as it impairs operators' ability to sustain their focus during construction equipment operations. The published literature states that the mental fatigue can result in poor decision-making, human errors, or underperformance, potentially creating hazardous situations for the operators. Recognizing its high prevalence and profound effect on construction workers and machinery operators, extensive research has been conducted to detect and identify mental fatigue early. Recently, invasive technologies have been used to solve this problem. However, these methods require the use of physical sensors, which can irritate and annoy operators, thereby impeding normal construction site work. This study addresses these issues by introducing a non-invasive method for assessing mental fatigue using geometric measures of facial features, rather than having operators wear sensors on their bodies. The proposed method was further validated by comparing it with wearable electroencephalography (EEG) technology, establishing its ecological validity for construction equipment operators. The following are the primary objectives of this research study: (1) to study the non-invasive detection of mental fatigue in construction equipment operators through geometric measurements of facial features; (2) to examine the validity of facial features’ geometric measurements for a real-time assessment of mental fatigue in construction equipment operators; (3) to develop deep learning-based construction equipment operators’ mental fatigue classification using wearable EEG sensor data; (4) to examine multimodal integration for data-driven classification of mental fatigue during construction equipment operations: incorporating electroencephalography, electrodermal activity, and video signals. Although the application of facial features has been widespread in other domains, such as drivers and other occupation scenarios, its ecological validity for construction excavator operators remains a knowledge gap. Consequently, there is a dearth of knowledge about creating a contactless and non-intrusive system for detecting mental fatigue in construction equipment operators. To start with, a study was conducted to investigate whether there are variations in the geometric measurements of facial features owing to mental fatigue. An excavation experiment was conducted and simultaneously with the task, the operators were video recorded to collect the data on their facial features via mobile camera. Based on geometric measurements, facial features (eyebrow, mouth outer, mouth corners, head motion, eye area, and face area) were extracted. The results found that there was a significant difference in the measured metrics for high fatigue as compared to low fatigue. Specifically, the most noteworthy variation was for the eye and face area metrics, with respective mean differences of 45.88% and 26.9%. The results indicate that the use of geometric measurements of facial features is an effective and non-intrusive method for detecting mental fatigue in construction equipment operators. Secondly, the proposed method was further validated through investigations that involved a comparison with flexible headband-based wearable electroencephalography (EEG) technology. The aim was to establish the ecological validity of the proposed method for construction equipment operators. Ground truth data, including brain activity captured by wearable EEG, along with geometric measurements of facial features, were extracted and analysed at baseline and at 20-minute intervals over the course of one hour. The results revealed significant temporal variation in the measured metrics such as eye aspect ratio, eye distance, mouth aspect ratio, face area, and head motion. These metrics were also found to have a significant correlation with both the ground truth data and the EEG metrics. Additionally, the patterns observed in the brain visualizations obtained from EEG were associated with variations in the facial features. Overall, the findings of this study demonstrate that mental fatigue among construction equipment operators can be effectively monitored in a non-invasive manner using geometric measurements of facial features. Thirdly, a study was conducted to investigate mental fatigue in construction equipment operators as a multimodal problem. Previous studies classified mental fatigue using single modal data with acceptable accuracy. However, mental fatigue is a multimodal problem, and no single modality is superior. Moreover, none of the previous studies in construction industry have investigated the multimodal data fusion for classifying mental fatigue, and whether such an approach would improve mental fatigue detection. This study proposes a novel approach using three machine learning models and multimodal data fusion to classify mental fatigue states. Electroencephalography, electrodermal activity, and video signals were acquired during an excavation operation, and the decision tree model using multimodal sensor data fusion outperformed other models with 96.2% accuracy and 96.175% to 98.231% F1 score. Multimodal sensor data fusion can aid in developing a real-time system to classify mental fatigue, improving safety and health management on construction sites. Finally, a study was conducted to propose the feasibility of a construction site strategy that utilizes flexible headband-based sensors to capture raw EEG data, and deep learning networks to recognize operators' mental fatigue. Previous approaches, such as machine learning using EEG-based wearable sensing systems, have been proposed to detect mental fatigue accurately. However, implementing these strategies on actual construction sites remains a challenge. The limited mobility and systemic instability of EEG sensors restrict their application to laboratory settings rather than to real construction environments. In addition, machine learning classifiers relying solely on manually engineered EEG features may compromise their performance in practical construction scenarios. To address these issues, this study employed the NASA-TLX score as the ground truth for measuring mental fatigue. Brain activity patterns were recorded using a wearable EEG sensor, and raw EEG data were used to develop the deep learning-based classification models. The performances of different deep learning models, including long short-term memory (LSTM), bidirectional LSTM, and one-dimensional convolutional networks, were assessed using metrics such as accuracy, precision, recall, specificity, and F1-score. The findings revealed that the bidirectional LSTM (Bi-LSTM) model outperformed other deep learning models, achieving a remarkable accuracy of 99.941% and an F1-score ranging from 99.917% to 99.993%. These results demonstrate the feasibility of implementing the Bi-LSTM model and contribute to the recognition and classification of mental fatigue by using wearable sensors. Ultimately, this advancement enhances the health and safety of operations at construction sites. |
| Rights: | All rights reserved |
| Access: | open access |
Copyright Undertaking
As a bona fide Library user, I declare that:
- I will abide by the rules and legal ordinances governing copyright regarding the use of the Database.
- I will use the Database for the purpose of my research or private study only and not for circulation or further reproduction or any other purpose.
- I agree to indemnify and hold the University harmless from and against any loss, damage, cost, liability or expenses arising from copyright infringement or unauthorized usage.
By downloading any item(s) listed above, you acknowledge that you have read and understood the copyright undertaking as stated above, and agree to be bound by all of its terms.
Please use this identifier to cite or link to this item:
https://theses.lib.polyu.edu.hk/handle/200/12965