Interactive Hand Pose Estimation using a Stretch-Sensing Soft Glove (SIGGRAPH 2019)

Interactive Hand Pose Estimation using a Stretch-Sensing Soft Glove (SIGGRAPH 2019)

We propose a glove for interact hand
pose capture it’s stretch sensing fully soft thin easy to put on and comfortable to wear it can capture complex hand poses with high accuracy and at an interactive framerate without requiring an external optical setup our glove successfully captures hand poses in situations where camera based trackers fail for example when occlusion occurs due to object manipulations or body occlusion or self occlusion or the hand moving out of field view also changing light conditions or motion blur due to fast translational movement are challenging for camera based systems but not our sensor based glove we demonstrate our glove can be fabricated at low cost using common tools available in modern fab labs the sensor consists of two conductive silicon patterns embedded into one single flat silicon sheet with a thickness of 1.25 millimeters in the shape of a hand wherever the patterns overlap a
capacitive sensor is formed measuring local stretch this matrix approach uses the available space efficiently requiring fewer leads than previous techniques and allows us to place as many as seven per finger for a total of 44 sensors other data gloves use at most one sensor per degree of freedom the sensor array is cast layer by layer the conductive patterns are etched with a
standard laser cutter the resulting silicone sensor on its own is not wearable therefore we design a cut pattern. The resulting elastic textile parts are first glued onto the silicone sensor and then closed up with textile
glue to form the actual glove since our sensor is thin and easily
adapts to the hand it is simple and efficient to capture training data with recent inexpensive off-the-shelf depth capture systems originally introduced to capture bare hands this allows us to employ a data-driven approach to model the mapping from the stretch centers to the hand pose. we propose a deep network architecture that exploits the spatial layout of the
sensor array and the hand itself a minimal per user calibration is performed on the fly using only the glove for each new user we only execute a per sensor min-max normalization in a comparison with two commercial state-of-the-art gloves ours achieves a 35% lower angular mean error in a leave-one-out experiment we demonstrate that a non personal general model trained with data from nine people and applied to the tenth person captures quite accurate poses if a more accurate prediction is desired it’s also possible to train a personal model on personal data only or fine-tune the non personal model with a few minutes of personal data it’s possible to take a model trained on data from one produced glove and predict poses with another glove here we show some live footage of our glove in action thank you

18 Replies to “Interactive Hand Pose Estimation using a Stretch-Sensing Soft Glove (SIGGRAPH 2019)”

  1. This Is amazing and still terrifying at the same time. Can't wait for this to be a consumer product

  2. great, now write an OpenVR driver for it so it works exactly like a knuckles controller without developer support for games and apps, and you'll instantly have funding. That's the one failing of these other products, not how good they are. None of these gloves get off the ground because they don't have native, plug and play openvr driver support for steam's hand pose system.

  3. Cool. If you could get 3D Space tracking resting on the wrist or back of the glove it would probably have lots of funding in VR. Or just as an attachable piece which is complementary to existing VR controllers.
    I would love to see a VR solution where gestures are (sometimes) usable over controller solutions.

  4. This seems like it would be extremely interesting in combination with DextrES, which was able to apply similarly compact and flexible force-feedback to a soft dataglove.

  5. Oh my god, this is brilliant! I may have to alter one of my projects to work with this instead of the other platform I chose.

  6. I used it in Siggraph 2019, but the accuracy of hand pose estimation was awful. This movie shows the just best case.

Leave a Reply

Your email address will not be published. Required fields are marked *