When artificial intelligence systems encounter scenes where objects are not fully visible, it has to make estimations based only on the visible parts of the objects. This partial information leads to detection errors, and large training data is required to correctly recognize such scenes. Now, researchers at the Gwangju Institute of Science and Technology have developed a framework that allows robot vision to detect such objects successfully in the same way that we perceive them.
Robotic vision has come a long way, reaching a level of sophistication with applications in complex and demanding tasks, such as autonomous driving and object manipulation. However, it still struggles to identify individual objects in cluttered scenes where some objects are partially or completely hidden behind others. Typically, when dealing with such scenes, robotic vision systems are trained to identify the occluded object based only on its visible parts. But such training requires large datasets of objects and can be pretty tedious.
Associate Professor Kyoobin Lee and Ph.D. student Seunghyeok Back from the Gwangju Institute of Science and Technology (GIST) in Korea found themselves facing this problem when they were developing an artificial intelligence system to identify and sort objects in cluttered scenes. “We expect a robot to recognize and manipulate objects they have not encountered before or been trained to recognize. In reality, however, we need to manually collect and label data one by one as the generalizability of deep neural networks depends highly on the quality and quantity of the training dataset,” says Mr. Back.
In a new study accepted at the 2022 IEEE International Conference on Robotics and Automation, a research team led by Prof. Lee and Mr. Back developed a model called “unseen object amodal instance segmentation” (UOAIS) for detecting occluded objects in cluttered scenes. To train the model in identifying object geometry, they developed a database containing 45,000 photorealistic synthetic images containing depth information. With this (limited) training data, the model was able to detect a variety of occluded objects. Upon encountering a cluttered scene, it first picked out the object of interest and then determines if the object is occluded by segmenting the object into a “visible mask” and an “amodal mask.”
The researchers were excited by the results. “Previous methods are limited to either detecting only specific types of objects or detecting only the visible regions without explicitly reasoning over occluded areas. By contrast, our method can infer the hidden regions of occluded objects like a human vision system. This enables a reduction in data collection efforts while improving performance in a complex environment,” comments Mr. Back.
To enable “occlusion reasoning” in their system, the researchers introduced a “hierarchical occlusion modeling” (HOM) scheme, which assigned a hierarchy to the combination of multiple extracted features and their prediction order. By testing their model against three benchmarks, they validated the effectiveness of the HOM scheme, which achieved state-of-the-art performance.
The researchers are hopeful about the future prospects of their method. “Perceiving unseen objects in a cluttered environment is essential for amodal robotic manipulation. Our UOAIS method could serve as a baseline on this front,” says Mr. Back.
It certainly looks like a giant leap for robotic vision!
Reference
Title of original paper: Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling
Journal: IEEE International Conference on Robotics and Automation
About the Gwangju Institute of Science and Technology (GIST)
The Gwangju Institute of Science and Technology (GIST) was founded in 1993 by the Korean government as a research-oriented graduate school to help ensure Korea’s continued economic growth and prosperity by developing advanced science and technology with an emphasis on collaboration with the international community. Since that time, GIST has pioneered a highly regarded undergraduate science curriculum in 2010 that has become a model for other science universities in Korea. To learn more about GIST and its exciting opportunities for researchers and students alike, please visit: http://www.gist.ac.kr/.
About the authors
Kyoobin Lee is an Associate Professor and Director of the AI laboratory at GIST. His group is developing AI-based robot vision and deep learning-based bio-medical analysis methods. Before joining GIST, he obtained a Ph.D. in Mechatronics from KAIST and completed a postdoctoral training program at the Korea Institute of Science and Technology (KIST). The author can be reached at kyoobinlee@gist.ac.kr
Seunghyeok Back is a Ph.D. Student at the AI laboratory at GIST. His research is on robot vision for unseen object manipulation using deep neural networks and Sim2Real transfer. He received a B.S. degree in mechanical engineering from GIST. The author can be reached at shback@gm.gist.ac.kr
