MEM: Multi-Modal Elevation Mapping for Robotics and Learning
Gian Erni
†
, Jonas Frey
†
, Takahiro Miki
†
, Matias Mattamala
‡
, Marco Hutter
†
Fig. 1. Three different applications of our multi-modal elevation mapping framework. Left: PCA layer of visual features on a crosswalk using LiDAR,
Depth and RGB cameras. Center: semantic segmentation layer in an agricultural field using a RGB-D sensor. Right: semantic segmentation layer in a
garden using LiDAR, Depth and RGB cameras.
Abstract— Elevation maps are commonly used to represent
the environment of mobile robots and are instrumental for
locomotion and navigation tasks. However, pure geometric
information is insufficient for many field applications that
require appearance or semantic information, which limits their
applicability to other platforms or domains. In this work, we
extend a 2.5D robot-centric elevation mapping framework by
fusing multi-modal information from multiple sources into a
popular map representation. The framework allows inputting
data contained in point clouds or images in a unified manner.
To manage the different nature of the data, we also present
a set of fusion algorithms that can be selected based on the
information type and user requirements. Our system is designed
to run on the GPU, making it real-time capable for various
robotic and learning tasks. We demonstrate the capabilities of
our framework by deploying it on multiple robots with varying
sensor configurations and showcasing a range of applications
that utilize multi-modal layers, including line detection, human
detection, and colorization.
I. INTRODUCTION
Autonomous unmanned ground vehicles deployed in out-
door environments need to understand their surroundings to
† Authors are with the Department of Mechanical and Process Engineer-
ing, ETH Z
¨
urich, 8092 Z
¨
urich, Switzerland {gerni, jonfrey, tamiki,
mahutter}@ethz.ch. J. Frey is also associated with the Max Planck
Institute for Intelligent Systems, 72076 T
¨
ubingen, Germany. ‡ M. Mattamala
is with the Oxford Robotics Institute at the University of Oxford, UK
matias@robots.ox.ac.uk. This project was supported by the Max
Planck ETH Center for Learning Systems (Frey), Swiss National Science
Foundation (SNSF) through project 166232, 188596, the National Centre
of Competence in Research Robotics (NCCR Robotics), and the European
Union’s Horizon 2020 research and innovation program under grant agree-
ment No.780883 and ANID / Scholarship Program / DOCTORADO BECAS
CHILE / 2019-72200291 (Mattamala).
operate in a safe and reliable manner. Onboard range sensors,
such as LiDARs, stereo cameras, or RADARs can provide
the robot with a spatial understanding of its surroundings.
However, these different representations have varying nature
and noise, which motivates their aggregation into a single
unifying map.
Different types of maps have been developed for robotics.
The simplest are 2D maps [24], which store the occupancy
information in each cell. 3D volumetric maps are another
popular representation [15], [20] that better encode the ge-
ometry of the environment, though at a higher computational
and memory cost. 2.5D maps (also called elevation or height
maps) [5], [6], [18] are a compromise between both, and
well suited for ground platforms. All the aforementioned
maps only contain geometric information about the environ-
ment. To successfully capture different environments, these
representations need to be multi-modal, here multi-modal
refers to different information content types i.e., encode
semantic classes, friction, traversability, color, or other task-
specific information, which is an active area of research
nowadays [4], [10], [14], [23]. The multi-modal information
can significantly enhance the performance of a variety of
downstream tasks. For example, semantic information en-
ables robots to differentiate concrete from grass and mud,
which allows them to plan a path on the less cost-intensive
road surface.
In this work, we aim to contribute to the development
of multi-modal robotic perception by presenting a flexi-
ble, real-time capable Multi-Modal Elevation Map (MEM)
framework. It builds upon our previous work [18] to allow
the seamless integration of multi-modal information such as
arXiv:2309.16818v1 [cs.RO] 28 Sep 2023
资源评论
