| Author: | Zhang, Tanhao |
| Title: | Seeing the invisible : development of a multi-modal vision system and its application |
| Advisors: | Navarro-Alarcon, David (ME) |
| Degree: | M.Sc. |
| Year: | 2022 |
| Subject: | Computer vision Image processing Hong Kong Polytechnic University -- Dissertations |
| Department: | Department of Mechanical Engineering |
| Pages: | xiv, 112 pages : color illustrations |
| Language: | English |
| Abstract: | Multi-spectrum vision system is a new sensor-based imaging system that obtain information of different part of spectrum. Unlike the vision system which has been widely used in our daily, multi-spectrum vision system has broader sensitive spectrum area, not only focus on the visible part. The Multi-spectrum vision system has been developed very fast in resent year and has been used in agriculture such as check crop growth and pests and in search and rescue areas for searching survivals. However, the existing system have their limitations, for example, most existing multi-spectrum vision systems are expensive and heavy, which limit the applied environment. In this thesis, our research are mainly about the new proposed low-cost multi-spectrum vision system and its application. We started by develop a sensing system with the capability to compute multispectral point clouds in real-time. The proposed multi-eye sensor system effectively registers information from the visible, (long-wave) infrared, and ultraviolet spectrum to its depth sensing frame, thus enabling to measure a wider range of surface features that are otherwise hidden to the naked eye. For that, we designed a new cross-calibration apparatus that produces consistent features which can be sensed by each of the cameras, therefore, acting as a multispectral “chessboard”. The performance of the sensor is evaluated with two different cases of studies, where we show that the proposed system can detect “hidden” features of a 3D environment. Then we explore the application of our proposed multi-spectrum system on Human-robot interactions. We apply the vision system on a robot arm by using image information guide the movement of robot arm. We process the image using artificial neural network to detect the objection target of the robot and how robot achieve the target. The information fusion of the vision system part and robot arm is based on Robot Operation System (ROS). The whole system can be used in dark environment which can be applied in industry and nursing. This is a practical application of our multi-spectrum vision system. Next, we explore the application on 3D reconstruction on dual-spectrum. Comparing with two-dimension (2D) thermal images, visualize three-dimension (3D) geometrical information with corresponding surface temperature provide a more intuitive way to perceive thermal information of objects. In this paper, we present a integrated system for large-scale and real-time 3D thermographic reconstruction through RGB-D camera and thermal camera fusion. The hardware is a dual spectrum imaging system consists of a RGB-D camera and a thermal camera with extrinsic calibrated. For software part, we propose thermal direct method based on feature of thermal imaging, which is thermal information robustness of illumination. Then we integrated our method into state-of-art location algorithm for generating reliable 3D thermal point cloud. The experimental result demonstrate that our proposed thermal direct method is accurate, and our whole system performed well on dual spectrum 3D reconstruction for small and large scale environment. Finally, we summarize the work, including shortcomings and future work. We believe multi-spectrum imaging will be a hot topic not only in engineering but also in research. |
| Rights: | All rights reserved |
| Access: | restricted access |
Files in This Item:
| File | Description | Size | Format | |
|---|---|---|---|---|
| 6883.pdf | For All Users (off-campus access for PolyU Staff & Students only) | 12.18 MB | Adobe PDF | View/Open |
Copyright Undertaking
As a bona fide Library user, I declare that:
- I will abide by the rules and legal ordinances governing copyright regarding the use of the Database.
- I will use the Database for the purpose of my research or private study only and not for circulation or further reproduction or any other purpose.
- I agree to indemnify and hold the University harmless from and against any loss, damage, cost, liability or expenses arising from copyright infringement or unauthorized usage.
By downloading any item(s) listed above, you acknowledge that you have read and understood the copyright undertaking as stated above, and agree to be bound by all of its terms.
Please use this identifier to cite or link to this item:
https://theses.lib.polyu.edu.hk/handle/200/12435