Communication Dans Un Congrès Année : 2026

Automatic Physically-Based Sim2Real for Tactile Images through Differentiable Path-Tracing Rendering

Résumé

High-fidelity simulation of vision-based tactile sensors is essential for developing data-driven robotic manipulation algorithms. However, a significant sim-to-real gap persists due to the difficulty in modeling complex optical effects, such as refraction through protective glass layers, and in accurately estimating physical parameters like sensor pose and lighting. To bridge this gap, we introduce a novel, fully differentiable pipeline for visual tactile simulation. Leveraging a differentiable path tracer, our method optimizes critical parameters—including camera pose, lighting conditions, and object texture—directly from just three real images. This approach achieves highly realistic simulations with physically accurate light transport and glass refraction. We validate our method through a comprehensive benchmark against real-world data, demonstrating state-of-the-art sim-to-real accuracy. We also enable novel applications, such as mesh reconstruction from a single tactile image via inverse rendering. To overcome the computational cost of path tracing, we further use a image-to-image translation model. This model uses high-fidelity simulated data alongside Normalized Object Coordinate Space (NOCS) maps as input, preserving crucial deformation information while enabling rapid inference.

Fichier principal
Vignette du fichier
ICRAW_differentiable_rendering-4.pdf (6.33 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-05488623 , version 1 (06-02-2026)

Licence

Identifiants

  • HAL Id : hal-05488623 , version 1

Citer

Guillaume Duret, Anna Samsonenko, Florence Zara, Jan Peters, Liming Chen. Automatic Physically-Based Sim2Real for Tactile Images through Differentiable Path-Tracing Rendering. IEEE International Conference on Robotics and Automation (ICRA 2026), Jun 2026, Vienne, Austria. ⟨hal-05488623⟩
547 Consultations
388 Téléchargements

Partager

  • More