Abstract: Robots are expected to execute increasingly complex tasks in increasingly complex environments, which requires understanding of the environment geometry, semantics, and dynamics in real time from streaming observations. The state of the art uses vision-language-action (VLA) models, which encode spatial and temporal context implicitly from image sequences. This talk presents an alternative explicit approach, which accumulates geometry, vision, and language information in a 3D multi-modal feature map to support autonomous robot navigation and manipulation. First, we focus on learning signed distance functions (SDF) from point cloud data and show that distance information accelerates motion planning and enables safety constraint specification. Next, we discuss learning multi-modal feature maps aligned with modern vision-language models and show how to generate map tokens to extend the spatial and temporal context for VLA models in long-horizon mobile manipulation.
Bio: Nikolay Atanasov is a Professor in the Department of Electrical and Computer Engineering at the University of California San Diego, La Jolla, CA, USA. He obtained a B.S. degree in Electrical Engineering from Trinity College, Hartford, CT, USA in 2008, and M.S. and Ph.D. degrees in Electrical and Systems Engineering from University of Pennsylvania, Philadelphia, PA, USA in 2012 and 2015, respectively. Dr. Atanasov's research focuses on robotics, control theory, and machine learning with emphasis on active perception problems for autonomous mobile robots. He works on probabilistic models and inference techniques for simultaneous localization and mapping (SLAM) and on optimal control and reinforcement learning techniques for autonomous robot navigation, manipulation, and uncertainty minimization. Dr. Atanasov's work has been recognized by the Joseph and Rosaline Wolf award for the best Ph.D. dissertation in Electrical and Systems Engineering at the University of Pennsylvania in 2015, the Best Conference Paper Award at the IEEE International Conference on Robotics and Automation (ICRA) in 2017, the NSF CAREER Award in 2021, and the IEEE RAS Early Academic Career Award in Robotics and Automation in 2023.
Abstract: Robots are expected to execute increasingly complex tasks in increasingly complex environments, which requires understanding of the environment geometry, semantics, and dynamics in real time from streaming observations. The state of the art uses vision-language-action (VLA) models, which encode spatial and temporal context implicitly from image sequences. This talk presents an alternative explicit approach, which accumulates geometry, vision, and language information in a 3D multi-modal feature map to support autonomous robot navigation and manipulation. First, we focus on learning signed distance functions (SDF) from point cloud data and show that distance information accelerates motion planning and enables safety constraint specification. Next, we discuss learning multi-modal feature maps aligned with modern vision-language models and show how to generate map tokens to extend the spatial and temporal context for VLA models in long-horizon mobile manipulation.
Bio: Nikolay Atanasov is a Professor in the Department of Electrical and Computer Engineering at the University of California San Diego, La Jolla, CA, USA. He obtained a B.S. degree in Electrical Engineering from Trinity College, Hartford, CT, USA in 2008, and M.S. and Ph.D. degrees in Electrical and Systems Engineering from University of Pennsylvania, Philadelphia, PA, USA in 2012 and 2015, respectively. Dr. Atanasov's research focuses on robotics, control theory, and machine learning with emphasis on active perception problems for autonomous mobile robots. He works on probabilistic models and inference techniques for simultaneous localization and mapping (SLAM) and on optimal control and reinforcement learning techniques for autonomous robot navigation, manipulation, and uncertainty minimization. Dr. Atanasov's work has been recognized by the Joseph and Rosaline Wolf award for the best Ph.D. dissertation in Electrical and Systems Engineering at the University of Pennsylvania in 2015, the Best Conference Paper Award at the IEEE International Conference on Robotics and Automation (ICRA) in 2017, the NSF CAREER Award in 2021, and the IEEE RAS Early Academic Career Award in Robotics and Automation in 2023.