A team of researchers at the Max Planck Institute for Intelligent Systems (MPI-IS) present a powerful soft haptic sensor called “Insight” that utilizes computer vision and a deep neural network to accurately estimate where objects come into contact with the sensor and the size of the applied forces.
A thumb-shaped sensor with a camera hidden inside is trained to infer haptic contact information. Image Credit: Max Planck Institute for Intelligent Systems.
Details of the new sensor have been published in the February 23rd, 2022 issue of the journal Nature Machine Intelligence.
The research project is a major step to enabling robots to feel their environment as accurately as humans and animals. Similar to its natural counterpart, the fingertip sensor is robust, highly sensitive and high resolution.
The thumb-shaped sensor is composed of a soft shell constructed around a lightweight stiff skeleton. This skeleton holds up the structure just like bones steady the soft finger tissue. The shell is built from an elastomer blended with dark but reflective aluminum flakes, resulting in an opaque greyish color that stops any external light from entering in.
Concealed within this finger-sized cap is a miniature 160-degree fish-eye camera that captures colorful images irradiated by a ring of LEDs.
When any object makes contact with the sensor’s shell, the appearance of the color pattern within the sensor varies. The camera captures images a number of times a second and inputs a deep neural network with this data.
The algorithm senses even the smallest variation in light in each pixel. Within a fraction of a second, the trained machine-learning system can map out precisely where the finger is touching an object, establish how powerful the forces are, and specify the force direction. The system deduces what researchers term a force map: it offers a force vector for every point in the three-dimensional fingertip.
The image illustrates the sensor, the camera image produced inside the sensor, and the output of the deep neural network, which displays how precisely the position of contact, the force strength, and the force direction can be assessed across the contacting areas.
We achieved this excellent sensing performance through the innovative mechanical design of the shell, the tailored imaging system inside, automatic data collection, and cutting-edge deep learning.
Georg Martius, Research Group Leader and Head of Autonomous Learning Group, Max Planck Institute for Intelligent Systems
His Ph.D. student Huanbo Sun adds: “Our unique hybrid structure of a soft shell enclosing a stiff skeleton ensures high sensitivity and robustness. Our camera can detect even the slightest deformations of the surface from one single image.”
While testing the sensor, the scientists noticed it was sensitive enough to feel its own orientation relative to gravity.
The third member of the team is Katherine J. Kuchenbecker, the Director of the Haptic Intelligence Department at MPI-IS. She expresses that the new sensor will be beneficial.
Previous soft haptic sensors had only small sensing areas, were delicate and difficult to make, and often could not feel forces parallel to the skin, which are essential for robotic manipulation like holding a glass of water or sliding a coin along a table.
Katherine J. Kuchenbecker, Director of the Haptic Intelligence Department, Max Planck Institute for Intelligent Systems
Nevertheless, how does this sensor learn? Huanbo Sun engineered a testbed to produce the training data required for the machine-learning model to comprehend the correlation between the variation in raw image pixels and the forces applied.
The testbed probes the sensor all over its surface and captures the real contact force vector along with the camera image inside the sensor. In this manner, approximately 200,000 measurements were produced. It took almost three weeks to gather the data and one additional day to train the machine-learning model.
Enduring this lengthy experiment with so many diverse contact forces helped establish the sturdiness of Insight’s mechanical design, and tests with a larger probe demonstrated how well the sensing system simplifies.
Another unique feature of the thumb-shaped sensor is that it has a nail-shaped zone with a thinner elastomer layer. This tactile fovea is engineered to sense even minute forces and detailed object shapes. For this super-sensitive area, the researchers selected an elastomer thickness of 1.2 mm instead of the 4 mm they used on the rest of the finger sensor.
The hardware and software design we present in our work can be transferred to a wide variety of robot parts with different shapes and precision requirements. The machine-learning architecture, training, and inference process are all general and can be applied to many other sensor designs.
Huanbo Sun, Ph.D. Student, Max Planck Institute for Intelligent Systems
Journal Reference:
Sun, H., et al. (2022) A soft thumb-sized vision-based sensor with accurate all-round force perception. Nature Machine Intelligence. doi.org/10.48550/arXiv.2111.05934.