SitSens - Situational Sensing for Learning

SitSens - Situational Sensing for Learning

A novel approach using on-body sensing to capture and digitize physical activities for training purposes. By utilizing on-body sensors, the system will enable the reconstruction of both observable and unobservable aspects of a trainer’s movements.

From Human Expertise to Immersive Digital Training.

Across industries such as healthcare, aerospace, manufacturing, and the culinary arts, mastering complex hand and body movements is essential for tasks involving precision, dexterity, and safety. However, training these skills, particularly when both hands are involved or when working in constrained environments, remains a significant challenge. Traditional training methods, such as direct observation, suffer from scalability, consistency, and safety limitations. In high-stakes or spatially restricted contexts, these shortcomings become even more pronounced, as trainers often cannot see their own hands or be easily observed by others. Remote methods further fall short in immersion and embodiment, essential for mastering nuanced motor tasks.

Digital training solutions such as video tutorials or virtual/augmented reality (VR/AR) present a scalable alternative, but they too face critical limitations. Standard video capture often misses essential details due to occlusions and fixed viewpoints, while VR/AR content creation is costly and limited in realism due to the difficulty of integrating real-world trainer movements. The cognitive load required to interpret 2D video from arbitrary angles further reduces the effectiveness of these solutions.

This project aims to overcome these limitations by developing a fully on-body sensing system to capture and digitize physical activities for training purposes. Through wearable technologies integrating body-worn cameras, inertial measurement units (IMU), and ultra-wideband (UWB) modules, the system will enable infrastructure-free capture of dynamic motion and 3D environments from the trainer’s perspective. A robust visual sensing framework will allow dynamic 3D reconstruction of scenes from first-person video, updating models in real time under occlusion and user movement, while reducing temporal inconsistencies and providing visually plausible reconstructions.

By combining cameras, IMUs and UWB in a minimal number of wearable nodes, the system will enable precise estimation of 3D position and orientation, even in enclosed or dynamic environments, with no need for manual calibration. To capture fine-grained hand motion, the project will advance a next-generation sensor glove using stretchable circuit technology. This glove will combine distributed IMUs with stretch sensors at each finger joint to overcome drift and ambiguity in motion data. It will also include small pressure sensors to quantify tactile readouts. A UWB antenna integrated into the glove will provide accurate absolute positioning of the hand in space.

For full-body tracking, the project will develop cleanroom-compatible sensor suits using textile ribbon interconnects that embed sensor nodes directly into the garment in a way that maintains wearer comfort and complies with contamination requirements. These suits will integrate flexible IMUs and UWB modules to achieve reliable motion capture in any industrial setting.

The final integrated system will enable seamless, calibration-free operation, providing real-time full-body pose estimation and 3D scene mapping without relying on external cameras or beacons. Setup will be rapid and unobtrusive, and the combination of body-mounted sensors and vision will support drift-resistant, accurate reconstruction of both hand motions and surrounding environments. The system’s fusion of visual and kinematic sensing will allow automatic generation of digital twins of training sessions, streamlining content creation for VR/AR environments while improving realism and scalability.

This infrastructure-free, immersive training platform lays the groundwork for safer, more effective, and more widely deployable digital skill transfer solutions across a broad range of professional domains.