Participating institutions
Northwestern (lead)
Your logo here
Corporate partner
MIT
Carnegie Mellon
Texas A&M
Stanford
UC Berkeley
Georgia Tech
Hardware partner
Hardware & data partner
Northwestern HAND · NSF Engineering Research Center
Robot manipulation policies are trained on teleoperation data. But the teleoperators producing that data cannot feel what the robot feels. The result: demonstrations with wrong force profiles, operators working at a fraction of normal speed, and models that fail at the contact-rich tasks that matter most — insertion, threading, snap-fit assembly, in-hand manipulation.
Robot-side sensing is advancing rapidly. High-resolution tactile sensors now deliver thousands of pixels per fingertip. But sensing without feedback is a camera without a display: the information never reaches the human whose demonstrations shape the policy. And there is no shared dataset that pairs robot encoder and torque readings with ground-truth haptic expectation — the field has no ImageNet for touch.
As described in Ed Colgate's 2026 IEEE Haptics Symposium keynote, dexterity is a negotiation between operator, robot, and environment. Today's teleoperation stacks break that negotiation: hardware-in-the-middle introduces friction, hysteresis, and latency; the back-channel from environment to operator is too weak to convey what is actually happening at contact.
We propose a large-scale, multi-institution effort to capture what the robot should be feeling alongside encoder and torque readings across a standardized suite of touch-heavy manipulation tasks — and to release the resulting data, model, and benchmarks as a shared resource for the robotics community.
Concretely: build capture rigs that minimize hardware-in-the-middle (in the spirit of UMI and MIT Dexop), deploy them across partner labs, run operators through a common benchmark battery, and record kinematics, torques, and ground-truth tactile state simultaneously. Train a generative model that predicts expected tactile state from kinematics — usable as a training signal, a reward shaper, or a source of synthetic haptic supervision for future policies.
The outcome: an open dataset, an open generative model, and a standardized benchmark suite — the ImageNet moment for touch, owned by no single lab.
Touch-heavy manipulation tasks building on NIST benchmarks, ManipulationNET, Rice peg-in-hole, and inspection-to-100%-coverage protocols — tasks that don't just benefit from touch, but require it.
Rigs inspired by UMI and Dexop that avoid the hysteresis and friction of traditional teleop stacks — so recorded kinematics reflect real intent, not transmission artifacts.
Every episode pairs encoder and torque readings with high-resolution tactile state, so models can learn the mapping from motion to expected feel.
Dataset, model weights, and benchmark suite released to the community. Shared infrastructure, shared credit, cumulative progress.
Common rigs, common tasks, and common evaluation deployed across partner institutions so results transfer and the field's collective understanding compounds.
University labs — shared capture infrastructure, data access, co-authorship, ownership of specific benchmark tasks, reproducible evaluation.
Corporate partners — early access to dataset and model, influence over benchmark selection, integration support, research pipeline.
Students & researchers — training on open infrastructure, reproducible baselines, a standardized benchmark to publish against, a new subfield to contribute to.
Hardware contributors — reference deployments of sensors, actuators, and capture rigs; co-development of the rig standard the dataset runs on.
The capture rig combines high-resolution robot-side tactile sensing with operator-side haptic feedback — so the operator feels what the robot feels in real time, and every demonstration is annotated with ground-truth haptic state. Fluid Reality's 6 mm electroosmotic actuators contribute the operator feedback layer — 128 pressure levels per actuator, <15 ms response, 1 kHz update rate that renders both contact and vibration in a single silent micro-hydraulic element. Small enough to fit inside a glove fingertip; no compressors, no pneumatics. Hugging Face contributes the data layer: dataset hosting, versioning, and distribution on the Hub, plus model release alongside.
