SOFTMAP: Sim2Real Soft Robot Forward Modeling

SOFTMAP enables real-time 3D forward modeling of soft finger manipulators by combining simulation pretraining with lightweight residual correction, achieving accurate shape prediction for downstream trajectory tracking and teleoperation.

Overview. SOFTMAP combines simulation pretraining, ARAP-based topological alignment, and residual correction to achieve accurate real-time 3D shape prediction, trajectory tracking, and vision-based teleoperation of soft finger manipulators.

Interactive Demo

Control the virtual soft finger using the sliders or keyboard. The learned forward model predicts the full 3D shape in real time.

Click to activate keyboard controls

Fine ±100

↑ ← ↓ →

Coarse ±1000

W A S D

R Set zero

Space Go to zero

C Reset

S1 0 S2 0

Drag to rotate · Scroll to zoom

Abstract

While soft robot manipulators offer compelling advantages over rigid counterparts, including inherent compliance, safe human-robot interaction, and the ability to conform to complex geometries, accurate forward modeling from low-dimensional actuation commands remains an open challenge due to nonlinear material phenomena such as hysteresis and manufacturing variability. We present SOFTMAP, a sim-to-real learning framework for real-time 3D forward modeling of tendon-actuated soft finger manipulators. SOFTMAP combines four components: (1) As-Rigid-As-Possible (ARAP)-based topological alignment that projects simulated and real point clouds into a shared, topologically consistent vertex space; (2) a lightweight MLP forward model pretrained on simulation data to map servo commands to full 3D finger geometry; (3) a residual correction network trained on a small set of real observations to predict per-vertex displacement fields that compensate for sim-to-real discrepancies; and (4) a closed-form linear actuation calibration layer enabling real-time inference at 30 FPS. We evaluate SOFTMAP on both simulated and physical hardware, achieving state-of-the-art shape prediction accuracy with a Chamfer distance of 0.389 mm in simulation and 3.786 mm on hardware, millimeter-level fingertip trajectory tracking across multiple target paths, and a 36.5% improvement in teleoperation task success over the baseline.

Contributions

Sim-to-real framework for soft manipulator control. We propose SOFTMAP, a unified sim-to-real learning framework for real-time 3D forward modeling of tendon-actuated soft finger manipulators, bridging the simulation-to-reality gap to enable accurate shape prediction, trajectory tracking, and vision-based teleoperation from low-dimensional servo commands.
ARAP-based topological alignment. We propose using As-Rigid-As-Possible (ARAP) deformation to project both simulated meshes and real-world point clouds into a shared, topologically consistent vertex space, enabling direct vertex-level supervision across domains.
Sim-pretrained forward model. A lightweight MLP, pretrained entirely on simulation data, maps low-dimensional servo commands to full 3D finger geometry, providing a strong shape prior without requiring real-world training data.
Residual correction network. A small correction network, trained on a limited set of real observations, predicts per-vertex displacement fields that compensate for sim-to-real discrepancies such as material hysteresis and manufacturing variability.
Closed-form actuation calibration. A linear calibration layer aligns real servo commands to their simulation-equivalent inputs, enabling real-time inference at 30 FPS without iterative optimization.
State-of-the-art results. SOFTMAP achieves a Chamfer distance of 0.389 mm in simulation and 3.786 mm on real hardware, millimeter-level fingertip trajectory tracking, and a 36.5% improvement in teleoperation task success over the baseline.

Method

SOFTMAP combines ARAP-based alignment, simulation pretraining, residual correction, and actuation calibration into a unified pipeline for accurate sim-to-real forward modeling.

SOFTMAP pipeline. Simulation data and real multi-view images are independently processed into 3D representations, then encoded via ARAP into a shared topologically consistent vertex space. A learned MLP maps servo commands to 3D shape predictions, refined by a residual correction network for downstream trajectory generation and teleoperation.

Simulation. The soft finger is modeled in SOFA Framework using a Neo-Hookean hyperelastic material with four embedded tendons.

Real-world data collection. Two RGB cameras capture synchronized multi-view images of the soft finger for 3D reconstruction.

Results

Shape Prediction

Evaluation comparison. Left: Point cloud comparisons in simulation; Right: Point cloud comparisons in real data.

Model	Chamfer ↓ (mm)	Mean Vertex ↓ (mm)
Simulation
Laplacian-Encoded	8.739 ± 3.67	8.757 ± 4.811
DeepSoRo	4.520 ± 2.437	7.970 ± 4.127
Linear Model	2.726 ± 1.043	2.620 ± 1.143
XGBoost	1.28 ± 0.58	0.868 ± 0.563
ARAP-Encoded	0.867 ± 0.53	0.438 ± 0.274
SOFTMAP (Ours)	0.389 ± 0.188	0.196 ± 0.098
Real
DeepSoRo	6.386 ± 1.367	—
SOFTMAP (w/o residual)	5.681 ± 0.939	—
SOFTMAP (Ours)	3.786 ± 0.61	—

Trajectory Tracking

Qualitative trajectory tracking. Each column shows snapshots for trajectories S, H, O, and E, comparing SOFTMAP against DeepSoRo and simulation ground truth.

Trajectory	Method	Sim MSE (mm)	Sim Max (mm)	Real MSE (mm)	Real Max (mm)
Letter S	DeepSoRo	2.685 ± 1.109	3.646	5.917 ± 5.12	23.154
Letter S	SOFTMAP	1.124 ± 0.617	1.95	3.458 ± 2.976	15.38
Letter H	DeepSoRo	2.766 ± 0.999	4.319	5.854 ± 3.816	14.7
Letter H	SOFTMAP	1.214 ± 0.652	2.55	3.569 ± 2.343	10.3
Letter O	DeepSoRo	1.336 ± 0.918	1.582	3.164 ± 1.683	6.887
Letter O	SOFTMAP	1.184 ± 0.182	1.499	1.387 ± 0.831	3.839
Letter E	DeepSoRo	1.889 ± 0.92	3.015	4.791 ± 3.174	14.76
Letter E	SOFTMAP	1.174 ± 0.755	2.509	2.616 ± 1.777	8.05

Teleoperation

Task	Method	IoU ↑	Overlap ↑ (%)
Push-T	DeepSoRo	0.38 ± 0.26	46 ± 27
Push-T	SOFTMAP	0.72 ± 0.13	80 ± 11
Cube	DeepSoRo	0.63 ± 0.16	71 ± 15
Cube	SOFTMAP	0.80 ± 0.10	87 ± 8
Tri. Prism	DeepSoRo	0.55 ± 0.10	67 ± 6.5
Tri. Prism	SOFTMAP	0.61 ± 0.18	72 ± 15

Citation

@article{ma2026softmap,
  title     = {SOFTMAP: Sim2Real Soft Robot Forward Modeling via Topological Mesh Alignment and Physics Prior},
  author    = {Ma, Ziyong and Yoo, Uksang and Francis, Jonathan and Zhi, Weiming and Ichnowski, Jeffrey and Oh, Jean},
  journal   = {arXiv preprint arXiv:2603.19384},
  year      = {2026},
  eprint    = {2603.19384},
  archivePrefix = {arXiv},
  primaryClass = {cs.RO},
  url       = {https://arxiv.org/abs/2603.19384}
}

SOFTMAP

Sim2Real Soft Robot Forward Modeling viaTopological Mesh Alignment and Physics Prior

Under Review