The role of egocentric perception in assistive devices and clinical environments

Rubén Martínez-Cantín

Universidad de Zaragoza e Instituto de Investigación en Ingeniería de Aragón

Resumen/Abstract

Recent advancements in computer vision allow for extracting structural and functional information from first-person viewpoints. This talk explores this landscape through egocentric perception, focusing on wearable cameras for blind and visually impaired individuals, as well as real-time 3D reconstruction during endoscopic procedures. For assistive devices, we explore beyond the standard scene understanding to focus on  affordance representation, action anticipation and knowledge transfer. We combine self-supervised methods for affordance grounding, Neural Radiance Fields (NeRFs) for semantic knowledge transfer and a combination of Bayesian methods and attention mechanisms for action anticipation, allowing the system to understand how a human can interact with the environment. Additionally, we incorporate Bayesian deep learning to quantify predictive uncertainty for safe task assistance. Alongside these assistive methods, we present a new SLAM system for online 3D reconstruction of endoscopic images that combines classical geometric and optimization methods with physics-informed deep learning techniques that embed geometric constraints into neural networks for dense 3D reconstruction in clinical environments.

Curriculum ponente

Ruben Martinez-Cantin is an associate professor at the University of Zaragoza and the co-director of the Robotics, Computer Vision and Artificial Intelligence group at Instituto de Investigación en Ingeniería de Aragón (I3A) in Zaragoza. Previously, he worked at SigOpt, a SF startup focused on Bayesian optimization (acquired by Intel in 2020), at the Centro Universitario de la Defensa, in Zaragoza, at the Instituto Superior Técnico in Lisbon and at the University of British Columbia. He received a PhD and MSc in Computer Science and Electrical Engineering from the University of Zaragoza in 2008 and 2003, respectively. His research spans many fields: machine learning, computer vision, robotics, assistive technologies, optimization, etc.