Hand Engine User Guide (DOC-5046)

This guide will provide an overview of how to use HandEngine to connect to a hand and create an animation.

Requirements


  1. Setting up the Glove Wired USB/Wireless Bluetooth Connection
    1. Connect your glove to the PC running Hand Engine using either a USB-MicroUSB wired connection or the StretchSense USB-Bluetooth Dongle.

      a. NOTE 1: If connecting via the USB-MicroUSB cable, you may need to do a one-time configuration of
      the COM port settings in Windows Device Manager. See Setting up COM Port for Wired USB Connection (DOC-5002) dongle for further instructions.

      b. NOTE 2: Windows will automatically assign a COM port number to the glove and/or USB Bluetooth dongle when they are connected to a USB port. You can find which COM port has been assigned to a particular glove/dongle by looking at the Ports (COM & LPT) category in Windows Device Manager. When you connect the glove/dongle, the associated COM port will appear in the Ports (COM & LPT) category, and unplugging the glove/dongle will make the associated COM port disappear from the Ports (COM & LPT) category.

    2. Power on the glove by pushing the button on top of the glove electronics once to enter Bluetooth mode. Once the glove is on (blinking blue) press the button twice to enter USB/wired mode. If you want to return to Bluetooth mode press the button twice again. The glove LED will blink Blue when in Bluetooth mode, and green when in USB mode.

      a. When starting in Bluetooth mode, the glove LED is set to a long blue blink to signify it is searching for an available StretchSense USB-Bluetooth Dongle. If it has previously been paired with a USB-Bluetooth Dongle and it is in range of that dongle it will automatically connect and the glove LED will switch to a short blue blink and the LED on the connected dongle will go solid blue. If this is the first time using the glove with an unpaired dongle (a dongle is unpaired if when plugged into a USB port the dongle LED is solid red), move the glove within <1 ft (<30cm) of the dongle until the red light on the dongle turns first green to indicate pairing has occurred, and then blue to indicate data is streaming from the glove.

2. Getting Started with Hand Engine
    1. Launch HandEngine using the Windows Start Menu or Desktop shortcuts or via the StretchSense Launcher (available on your account page on https://stretchsense.com/my-account/ )

       

       

       

    2. Once launched, Hand Engine will open for you to use.

      NOTE:  If at any point Hand Engine exhibits persistent unexpected behavior, you can save the state of the Hand Engine UI by clicking File→ Save Session to download a configuration *.session file, then you can reset Hand Engine. You can then click File → Load Session and navigate to your configuration *.session file to reload the previous state of the UI. This does not affect any calibration that you have done previously. If you have not restored the state of the Hand Engine UI, please note you will need to reselect the Performer and calibration profile and click on one of the output modes if you wish to reload a previous calibration.

       

    3. Make sure USB mode is selected to acquire live data from a glove.

    4. Under Performer, click on create and add the performer name.

      NOTE: If you have previously created a Performer and/or Profile on the same PC you are currently using, select them from the Performer and/or Profile drop-down boxes.

       


      NOTE: If you have previously created a Performer and/or Profile on another PC, you can transfer this data to your current PC by:

       

      1. Export Performer and/or Profiles from the other PC by selecting File → Export Performer or File → Export Calibration to download a JSON file containing the Performer and/or Profile data. 

        1. Exporting Performer exports all calibration profiles associated with that Performer

        2. Exporting a calibration Exports that Performer but with only the current calibration profile

      2. Copy the JSON file to a location your current PC can access.

      3. Import the Performer and/or Profile by selecting File → Import Performer or File → Import Calibration

         

       

    5. Under the boxes Left and Right, click on SCAN to populate the COM Port list and select the COM port of your USB dongle/Glove, then toggle the switch next to the COM PORT box from to On to open a connection.

       

    6. Inspect the single blue bar view at the top of the Performer tab that is moving. Curling your gloved fingers will make the bar grow taller, and vice versa.

       

    7. Follow our Quick Calibration (Express Calibration) guide to get hand data going.

      Express Calibration Mode - How to Setup (DOC-5034)

    8. Once you have completed the Express Calibration, you can start streaming data from Hand Engine. Please check our Streaming to Other Software session on our Knowledge Base home page to see the compatible third-party applications.

 

3. Advanced Calibration
    1. Click on Advanced Calibration Options under the Streaming box to expand the advanced options.

       

    2. The Captured Poses box will display all poses captured so far from each hand, they are all saved under the left and right profiles.

       

    3. You can start training hand poses to the calibration profile using the Pose Library Controls. You can select which poses you want to use depending on your particular application, we have provided several pose libraries to help you get started:

      Quick: A few simple poses for good general use.

      Performance: A more detailed pose set for more nuanced movement.

      Expert: An even more detailed pose set for expert hand data output.

      ASL: The complete American Sign Language set of poses.

      Game_Starter: A compilation of common hand poses in games.

      Extras: A compilation of common hand poses in general.

       

    4. For this example, we will use the Quick Pose set. First, ensure the switch in the hand viewport is set to Pose. Next, expand the Quick menu in the Pose Library Controls and select the first item on the list Paddle, ensure Auto Mode is On as this will automatically progress to the next pose on the list once the previous pose is captured.

       

      1. NOTE: If you wish to calibrate multiple gloves to the same pose at the same time then toggle Group Calibration to On. When Group Calibration is On, every glove that is connected to Hand Engine and Hand Engine is receiving data will be trained to the selected pose at the same time. Whether you are doing a pair of gloves on a single performer or calibrating multiple performers simultaneously, data is captured from every glove at the same time so all of the hands/gloves/performers must be in the correct pose at the same time during capturing.

      2. NOTE: Even if Auto Advance Pose is On, you can still manually select the next pose to be trained from the Pose Library Controls using your mouse, Auto Advance Pose will not interfere and simply jump to the next pose afterward.

      3. NOTE: If you’ve already calibrated before, make sure the switch in the hand viewport is set to Pose. before you try to add more poses from the library to the profile. 


         

    5. With your connected glove, get the performer to match the pose in the Viewport.

      IMPORTANT! For best results, have the performer initially copy the reference pose with their hand with the wrist straight. It is very important to make sure all of the joints on the performer’s hand match the reference hand pose from the pose library, if the performer cannot achieve the pose, they shouldn't train to that pose than to train it incorrectly. Furthermore, it is highly beneficial to ensure that if a particular joint is in the same position in more than one pose, for each of these poses the corresponding joint on the performer’s hand is also in the same position. This provides higher quality data to the hand solver that results in optimal calibration performance.  Once the performer is set in the hand pose, ask them to hold the hand pose steady and hold their forearm steady while gently rotating at their wrist (e.g. like the wrist would move when using a skipping rope), and to keep rotating their wrist in this circular motion until the capture is complete. Start the capture process by clicking the Capture button. The capture process takes <2 seconds for each captured pose.

       

       

    6. Once the capture is completed, the viewport will automatically progress to the next pose (for example in the case of the Quick library the second pose is the Thumbs Up pose). Repeat Step 4 above using this new pose. Once all poses in the library have been captured the viewport will stop updating, you will see the poses saved in the Captured Poses tree in the Performer 1 pane.

      1. NOTE: You can delete individual poses from the Captured Poses list by right-clicking on the pose and clicking Delete.

      2. NOTE: You can delete all of the poses from the Captured Poses list by clicking the RESET button underneath. This empties the calibration profile but does not delete the calibration profile.


         

    7. Next, you can select how to visualize the calibrated glove: 

      1. Blend Mode

        - Continuous movement that interpolates between the individual poses captured by you

        - Good for realistic hand movement

      2. Pose Detection Mode

        - Snaps between the individually captured poses selecting the most suitable one

        - Good for keyframing as the poses captured are the only ones that are output

      3. Hybrid Mode

        - Lifelike movements afforded by blend mode as well as being able to snap to key poses when handling props for example.

        - Check out our guide on how to set up hybrid mode.

      4. Express Mode

        - Continuous movement that exclusively uses your express calibration
        - Reinitialise the Express Mode output from the EXPRESS POSE in Captured poses.

        - This does NOT trigger a recalibration of Express Mode.

        - Check out our guide on how to Setup Express Calibration.

    8. To preview the hand animation from your calibrated glove, toggle the hand viewport is set to Live. The hand in the Hand Engine Viewport will now show the live output of the MoCap Pro glove hand capture in real-time.

      NOTE: If you are using Hand Engine version 1.1.0 or lower and you would like to add more poses from the pose library to your profile, you need to turn the Preview button Off.


       

    9. If you intend to stream hand data to your animation software, set Streaming to On. The guides on how to configure your animation software to receive this data stream using StretchSense plugins are provided here: 

      Streaming to Maya

      Streaming to UE4 (Unreal)

      Streaming to Unity

      Streaming to Mobu

       

      NOTE: Each glove requires a unique TCP port to stream data on. By default TCP port 9000 - 9011 are used for the 12 gloves respectively for Hand Engine Studio, or TCP port 9000-9003 for the 4 gloves respectively for Hand Engine Animator.

       

    10. Pose Detection Mode uses the same calibration data as Blend Mode. To enable Pose Detection Mode simply press the Use Pose Detection button. In this mode, you can specify a Debounce value measured in frames and transition animations using Animate Transition. This Debounce value is the number of consecutive frames that a pose output must be stable before the pose will be applied to the hand asset. The Animate Transition controls enable you to specify the type of animation (from left to right: Smooth in and out, linear, Explode out, Explode in) and the number of frames the animation
      takes.

       

       

    11. Repeat the steps above for each glove you are using.

4. Configuring Timecode in Hand Engine
    1. From the Master Timecode section, select the timecode source you wish to use for your glove data from the System TC dropdown box. The options available are:

      1. Local System Time (default): use the system clock of the PC running Hand Engine as the timecode source. Hand Engine converts the time from the system clock of your PC into a 30fps timecode value.

      2. Vicon - Network: Use this option to set up the Vicon timecode in Hand Engine
        (see Vicon DataStream Integration - Timecode (DOC-5038))

      3. OptiTrack - Network: Use this option to set up the OptiTrack timecode in Hand Engine
        (see OptiTrack DataStream Integration - Timecode (DOC-5037))

 

5. Setting the Local Recording Directory and Timer Based Calibration
    1. Changing the directory Hand Engine will save local raw data files to: 

      To change the root directory where files are saved:

      • First create that directory in Windows File Explorer, then copy the full pathname of that directory.

      • In Hand Engine, click Settings → Preferences, then paste the pathname to the Recording Location text box.

        NOTE: It is important to ensure a trailing backslash (“\”) character is at the end of the pathname.

      • The take name can also be formatted as preferred (e.g. device,actor,calibration,handtype,takename,time)


         

    2. Enabling Automatic Timer Pose Capture: You can automatically run through the poses in a pose library without requiring you to click Capture for each pose by using the Auto Capture Timer. Set the Auto Capture option to On, and specify the Time between capture in seconds to hold each pose before capture begins.
      NOTE: StretchSense recommends starting with an 8-second delay, to begin with, and reducing that time as appropriate.

      1. When Capture is clicked, the Capture button changes to Stop (Auto On) and: 

      2. A timer pops up on-screen counting down from the time specified in Time between capture and the Performer must adopt the displayed pose. 

      3. When the timer reaches seconds to go, the Performer will be asked to start rotating their wrist(s).

      4. At seconds the capture process begins. The capture process takes ~1 second to complete. 

      5. Once the capture is complete, if Auto Mode is On, the viewport will be updated with the next pose in the pose library and the timer will restart.

      6. Steps a-d will repeat until the last pose in the pose library is captured

        NOTE: The Auto Capture process can be stopped by clicking Stop (Auto On)

    3. Browse previous recordings: Select Settings → Browse Recordings to open a Windows File Explorer window at the directory where Hand Engine will save local raw data files to.



6. Saving and Restoring the State of Hand Engine

Once you have set up each glove and set up Hand Engine in the configuration you desire, you can save the state of Hand Engine by clicking File → Save Session to download a *.session file of the state of Hand Engine. To restore a previous state of Hand Engine, click File → Load Session and navigate to the previously saved *.session file. 


 

 

7. Recording MoCap Pro Glove Data using Hand Engine

This process assumes you have already connected each glove you are using to Hand Engine, created a calibration profile with the desired training poses for each glove, have selected Blend or Pose Detection mode, and have configured your Hand Engine timecode source (if applicable).

    1. In order to record raw capacitance data, joint angle data, and timecode data first define a Label for the recording, then toggle the Record Switch to Rec. To Stop the recording toggle the Record Switch to ‘Stop’.


       

    • NOTE: A subfolder is created for each take in C:\Users\<user_id>\Documents\StretchSense Hand  Engine with the naming convention <take name>_<date>_<time>

    • NOTE: within this subfolder, two files are created for each glove connected to Hand Engine

      1x Comma Separated Variable (CSV) text file with the naming convention Raw_GLOVE-<x>_<actor name>_<profile name>_Hand_<left/right>_<take name>_<date>_<time>.csv containing (from left to right):

      1. Raw capacitance data of sensors (16 columns)

      2. Raw IMU data (if applicable) (10 columns)

      3. Timecode (device) = timecode on the glove, sourced from the wireless jam sync from Hand Engine. This is a string of numbers representing hours mins seconds and frames, however, the colon is missing compared to Timecode (master) (1 column)

      4. Timer (device) = a millisecond counter of the device from startup should just increment in 8/9ms intervals (1 column)

      5. Timecode (master) = Hand Engine timecode (1 column)

      6. Solved joint angles (labeled) (60 columns)


        1x FBX File with the naming convention FBX_GLOVE-<x>_<actor name>_<profile name>_Hand_<left/right>_<take name>_<date>_<time>.fbx

        1x JSON file with the naming convention Cal_GLOVE-<x>_<actor name>_<profile name>_Hand_<left/right>_<take name>_<date>_<time>.json containing a snapshot of the calibration profile used at the time the data was recorded.

 8. Playing Back a Raw Data File Recorded in Hand Engine
    1. Select the Performer and Profile you wish to use to interpret the raw data file, and then click Train Blend, Pose Detection, Hybrid, or Express depending on your needs and the Profile you are using. 

    2. Select the file playback mode, then click Browse and navigate to a raw data CSV file previously recorded in Hand Engine. 


       

       

    3. Click Play to start the playback. Click Pause to pause the playback at the current frame and click Play again to resume from that point. Click the Stop button to reset the animation back to the start of the animation file.

      • NOTE: You can reprocess the raw data file using any Performer/Profile and any mode (Blend, Pose Detection, Hybrid, or Express). You do not need to use the same that was used for the original recording.

      • NOTE: If you wish to play several recorded raw data files simultaneously, repeat steps 1 and 2 in each glove in Hand Engine as required, then click the Play button in Global File Playback. This option is useful for triggering the streaming of multiple raw data files to your animation software at the same time.


         

         

 

Disclaimer, Intellectual Property, and Licensing

Revision Summary

Revision

Date

Notes

14

2022.05.25

Updated step 1 and reformatted the numbered steps.

13

2021.10.29

Updated Links

12

2021.10.01

Updated screenshots and workflow for Hand Engine 1.4

11

2021.08.21

Updated recording files and CSV column headers in Section 7 for 1.3.0

10

2021.07.27

Updated timecode information and extra features for Hand Engine 1.2.1

9

2021.06.03

Updated screenshots and extra features for Hand Engine 1.1.2

8

2021.05.10

Updated screenshots and extra features for Hand Engine 1.1.1

7

2021.03.16

Updated for Hand Engine 1.0.1

6

2020.12.07

Updated license information and Hand Engine screenshots

5

2020.10.02

Updated Hand Engine screenshots

4

2020.09.21

Updated for Beta3.5

3

2020.08.21

Updated for Beta3

2

2020.07.21

Updated related docs, descriptions, record function and where to find recorded files, note added that informs that user needs to turn off Preview off to add more poses, added disclaimer text

1

2020.07.10

Initial version