Newly accepted papers from our lab.
Human-object interaction prediction in videos through gaze following
Understanding the human–object interactions (HOIs) from a video is essential to fully comprehend a visual scene. The proposed approach leverages human gaze information, scene contexts, and visual appearances of human-object pairs through a spatio-temporal transformer to detect and anticipate HOIs in videos. The model outperforms state-of-the-art in HOI detection in the largest video HOI dataset, VidHOI, by large margins.
Ni, Zhifan., Esteve Valls Mascaró., Hyemin Ahn, & Dongheui Lee. Human-object interaction prediction in videos through gaze following, Computer Vision and Image Understanding, 2023. Webpage, opens an external URL in a new window
© Esteve Valls Mascaro
Long-Horizon Planning and Execution with Functional Object-Oriented Networks
Following work on joint object-action representations, functional object-oriented networks, opens an external URL in a new window (FOON) were introduced as a knowledge graph representation for robots. A FOON contains symbolic concepts useful to a robot's understanding of tasks and its environment for object-level planning, opens an external URL in a new window. We thereby introduce the idea of exploiting object-level knowledge as a FOON for task planning and execution.
David Paulius, Alejandro Agostini, Dongheui Lee, Long-Horizon Planning and Execution with Functional Object-Oriented Networks, IEEE Robotics and Automation Letters (RA-L), 2023. DOI: 10.1109/LRA.2023.3285510 . Webpage, opens an external URL in a new window
© Dongheui Lee
Vision-based Approximate Estimation of Muscle Activation Patterns for Tele-Impedance
Our goal is to transfer the human ability of end-point stiffness adaptation to the robot solely based on RGB images. For that, we introduce a model named muscle activation pattern estimator (MAPE). MAPE can infer:
- Whether the muscle is activated or inactivated (static states).
- Whether the degree of muscle activation is increasing, stable, or decreasing (temporal states).
Hyemin Ahn, Youssef Michel, Thomas Eiband and Dongheui Lee, Vision-based Approximate Estimation of Muscle Activation Patterns for Tele-Impedance, IEEE Robotics and Automation Letters (RA-L), pp. - , 2023
© Hyemin Ahn
Coordination tending towards an anti-phase relationship determines greater sway reduction during entrainment with a simulated partner
To develop effective balance support robotic devices for older adults at risk of falls, understanding the simultaneous occurrence of entrainment and sway reduction in human-human interaction is crucial. This study investigates the effects of different simulated sway-responsive interaction partners on sway entrainment, sway reduction, relative interpersonal coordination, and individual body schema accuracy. Our findings reveal that certain interaction modes, such as the Repulsor, lead to reduced body sway and improved interpersonal coordination, emphasizing the significance of accurate body schema and anti-phase relationships in facilitating sway reduction.
Youssef Michel, Katrin Schulleri, Leif Johannsen, and Dongheui Lee, Coordination tending towards an anti-phase relationship determines greater sway reduction during entrainment with a simulated partner, Human Movement Science, vol. 89, 103090, 2023. doi: 10.1016/j.humov.2023.103090
© Youssef Michel, Katrin Schulleri
Estimation of 6D pose of objects based on a variant adversarial autoencoder
This paper proposes a method for estimating the 6D pose of objects using a texture-less dataset, where the pose is obtained by rendering the 3D model and the orientation is represented by a latent space from RGB images, utilizing a variant Adversarial Autoencoder network. Experimental results demonstrate that the proposed method achieves better accuracy in object pose estimation compared to a state-of-the-art method.
Huang Dan, Hyemin Ahn, Shile Li, Yueming Hu, and Dongheui Lee, Estimation of 6D pose of objects based on a variant adversarial autoencoder, Neural Processing Letters, 2023. DOI: 10.1007/s11063-023-11215-2
© Dongheui Lee