The model learns representations of contact structure and changes over time by predicting in latent space. The article describes its intended use as connecting different tactile signals with dexterous robotic manipulation.
Magnetic input consists of three-axis tactile measurements. Piezoresistive measurements and decoded scalar visuotactile measurements are placed in the Z channel, with X and Y set to zero; raw RGB images are not model inputs.