Legacy - User Guide - Hand Engine Version 1.3 (DOC-5010)

1. Connect your glove to the PC running Hand Engine using either a USB-MicroUSB wired connection or the StretchSense USB-Bluetooth Dongle.

   a. NOTE 1: If connecting via the USB-MicroUSB cable, you may need to do a one-time configuration of
   the COM port settings in Windows Device Manager. See Setting up COM Port for Wired USB Connection (DOC-5002) dongle for further instructions.

   b. NOTE 2: Windows will automatically assign a COM port number to the glove and/or USB Bluetooth dongle when they are connected to a USB port. You can find which COM port has been assigned to a particular glove/dongle by looking at the Ports (COM & LPT) category in Windows Device Manager. When you connect the glove/dongle, the associated COM port will appear in the Ports (COM & LPT) category, and unplugging the glove/dongle will make the associated COM port disappear from the Ports (COM & LPT) category.

2. Power on the glove by pushing the button on top of the glove electronics once to enter Bluetooth mode. Once the glove is on (blinking blue) press the button twice to enter USB/wired mode. If you want to return back to Bluetooth mode press the button twice again. The glove LED will blink Blue when in Bluetooth mode, and green when in USB mode.

   a. When starting in Bluetooth mode, the glove LED is set to a long blue blink to signify it is searching for an available StretchSense USB-Bluetooth Dongle. If it has previously been paired with a USB-Bluetooth Dongle and it is in range of that dongle it will automatically connect and the glove LED will switch to a short blue blink and the LED on the connected dongle will go solid blue. If this is the first time using the glove with an unpaired dongle (a dongle is unpaired if when plugged into a USB port the dongle LED is solid red), move the glove within <1 ft (<30cm) of the dongle until the red light on the dongle turns first green to indicate pairing has occurred, and then blue to indicate data is streaming from the glove.

1. Launch HandEngine using the Windows Start Menu or Desktop shortcuts.

    

   

    

    

2. Once launched, Hand Engine will open in a web browser tab for you to use.

   a. NOTE: StretchSense recommends running Hand Engine in a Google Chrome tab.

   b. NOTE: To avoid performance degradation, StretchSense recommends you do not have any other tabs open in the web browser window that is running Hand Engine.

   c. NOTE:  If at any point Hand Engine exhibits persistent unexpected behavior, you can save the state of the Hand Engine UI by clicking File→ Save Session to download a configuration *.session file, then you can reset Hand Engine by clicking the Refresh button on your browser tab which will set the Hand Engine UI back to its default state. You can then click File → Load Session and navigate to your configuration *.session file to reload the previous state of the UI. This does not affect any calibration that you have done previously. If you have not restored the state of the Hand Engine UI, please note you will need to reselect the actor and calibration profile and click Train Blend/Train Pose Detection if you wish to reload a previous calibration.

   

3. To connect the glove to Hand Engine, expand the Glove 1 dropdown by clicking on the dropdown arrow.

   a. NOTE: you can use the following process for the other Gloves to add more gloves as well.

   

    

4. Once expanded it will display the glove controls panel for Glove 1.

   

5. Make sure USB mode is selected to acquire live data from a glove.
   
   

   

    

6. Press Scan to populate the COM Port list and select the COM port of your USB dongle/Glove, then toggle the switch next to the COM PORT box from to On to open a connection.
   
   

   

    

7. Inspect the raw data plots in the graphical view at the bottom of StretchSense Device 1 to make sure all 15 (for MoCap Pro Original gloves)/16 (for MoCap Pro SuperSplay gloves) sensors are present and accounted for. Curling your gloved fingers will make the histograms grow taller, and vice versa.

   NOTE: MoCap Pro Original gloves do not have a wrist sensor so you should expect this histogram to be empty.
   
   

   

1. To calibrate a glove-first you will need to create an Actor and Profile which is used to save and recall calibration profiles. Press the Add button next to the Actor field and fill in the details when prompted. 

   NOTE: Calibration profiles relate to a specific hand. StretchSense recommends including the hand (i.e. “L. Hand”/”R. Hand”) in the Profile Description.

   NOTE: If you have previously created an Actor and/or Profile on the same PC you are currently using, select them from the Actor and/or Profile drop-down boxes.

    

   

    

    

   NOTE: If you have previously created an Actor and/or Profile on another PC, you can transfer this data to your current PC by:

    

   1. Export Actor and/or Profiles from the other PC by selecting File → Export Actor or File → Export Calibration to download a JSON file containing the Actor and/or Profile data. 

      1. Exporting Actor exports all calibration profiles associated with that actor

      2. Exporting a calibration Exports that Actor but with only the current calibration profile

   2. Copy the JSON file to a location your current PC can access.

   3. Import the Actor and/or Profile by selecting File → Import Actor or File → Import Calibration

      

       

2. Once an Actor and Profile have been created you can start adding training poses to the calibration profile using the Pose Library Controls. You can select which poses you want to use depending on your particular application, we have provided several pose libraries to help you get started:

   1. Quick: A few simple poses for good general use.

   2. Performance: A more detailed pose set for more nuanced movement.

   3. Expert: An even more detailed pose set for expert hand data output.

   4. ASL: The complete American Sign Language set of poses.

   5. Game_Starter: A compilation of common hand poses in games.

      

       

3. For this example, we will use the Quick Pose set. First, ensure that Preview in the Glove controls panel is set to Off. Next, expand the Quick menu in the Pose Library Controls and select the first item on the list Paddle, ensure Auto Mode is On as this will automatically progress to the next pose on the list once the previous pose is captured.

   • NOTE: If you wish to calibrate multiple gloves to the same pose at the same time then toggle Group Calibration to On. When Group Calibration is On, every glove that is connected to Hand Engine and Hand Engine is receiving data will be trained to the selected pose at the same time. Whether you are doing a pair of gloves on a single actor or calibrating multiple actors simultaneously, data is captured from every glove at the same time so all of the hands/gloves/actors must be in the correct pose at the same time during capturing.

   • NOTE: Even if Auto Advance Pose is On, you can still manually select the next pose to be trained from the Pose Library Controls using your mouse, Auto Advance Pose will not interfere and simply jump to the next pose afterward.

   • NOTE: If you’ve already calibrated before, make sure the Preview button is turned Off before you try to add more poses from the library to the profile. 

     

      

4. With your connected glove, get the actor to match the pose in the Viewport.

   IMPORTANT! For best results, have the actor initially copy the reference pose with their hand with the wrist straight. It is very important to make sure all of the joints on the actor’s hand match the reference hand pose from the pose library, if the actor cannot achieve the pose, it is better that they do not train to that pose than to train it incorrectly. Furthermore, it is highly beneficial to ensure that if a particular joint is in the same position in more than one pose, that for each of these poses the corresponding joint on the actor’s hand is also in the same position. This provides higher quality data to the hand solver that results in optimal calibration performance.  Once the actor is set in the hand pose, ask them to hold the hand pose steady and hold their forearm steady while gently rotating at their wrist (e.g. like the wrist would move when using a skipping rope), and to keep rotating their wrist in this circular motion until the capture is complete. Start the capture process by clicking the Capture button. The capture process takes <2 seconds for each captured pose.

    

   

    

5. Once the capture is completed, the viewport will automatically progress to the next pose (for example in the case of the Quick library the second pose is the Thumbs Up pose). Repeat Step 4 above using this new pose. Once all poses in the library have been captured the viewport will stop updating, you will see the poses saved in the Captured Poses tree in the Glove 1 pane.

   • NOTE: You can delete individual poses from the Captured Poses list by right-clicking on the pose and clicking Delete.

   • NOTE: You can delete all of the poses from the Captured Poses list by clicking the RESET button underneath. This empties the calibration profile but does not delete the calibration profile.

     

      

6. Next, you can select how to visualize the calibrated glove: 
   
   

   

    

   1. Blend Mode

      - Continuous movement that interpolates between the individual poses captured by you

      - Good for realistic hand movement

       

   2. Pose Detection Mode

      - Snaps between the individually captured poses selecting the one that is most suitable

      - Good for keyframing as the poses captured are the only ones that are output

       

   3. Hybrid Mode

      - Lifelike movements afforded by blend mode as well as being able to snap to key poses when handling props for example.

      - Check out our guide on how to set up hybrid mode.

   4. Express Mode

      - Continuous movement that exclusively uses your express calibration
      - Reinitialise the Express Mode output from the EXPRESS POSE in Captured poses.

      - This does NOT trigger a recalibration of Express Mode.

      - Check out our guide on how to Setup Express Calibration.

7. To preview the hand animation from your calibrated glove, toggle the Preview button to On. The hand in the Hand Engine Viewport will now show the live output of the MoCap Pro glove hand capture in real-time.

   NOTE: If you are using Hand Engine version 1.1.0 or lower and you would like to add more poses from the pose library to your profile, you need to turn the Preview button Off.
   
   

   

    

8. If you intend to stream hand data to your animation software, set Streaming to On. The guides on how to configure your animation software to receive this data stream using StretchSense plugins are provided here: Streaming to Other Software - StretchSense Knowledge Base.

   NOTE: Each glove requires a unique TCP port to stream data on. By default TCP port 9000 - 9011 are used for Glove 1 - Glove 12 respectively for Hand Engine Studio, or TCP port 9000-9003 for Glove 1 - Glove 4 respectively for Hand Engine Animator.

9. Pose Detection Mode uses the same calibration data as Blend Mode. To enable Pose Detection Mode simply press the Use Pose Detection button. In this mode, you are able to specify a Debounce value measured in frames and transition animations using Animate Transition. This Debounce value is the number of consecutive frames that a pose output must be stable before the pose will be applied to the hand asset. The Animate Transition controls enable you to specify the type of animation (from left to right: Smooth in and out, linear, Explode out, Explode in) and the number of frames the animation
   takes.

    

   

    

10. Repeat the steps above for each glove you are using.

1. From the Master Timecode section, select the timecode source you wish to use for your glove data from the System TC dropdown box. The options available are:

   1. Local System Time (default): use the system clock of the PC running Hand Engine as the timecode source. Hand Engine converts the time from the system clock of your PC into a 30fps timecode value.

   2. Vicon - Network: Use this option to set up the Vicon timecode in Hand Engine (see Vicon DataStream Integration - Timecode (DOC-5038))

   3. OptiTrack - Network: Use this option to set up the OptiTrack timecode in Hand Engine (see OptiTrack DataStream Integration - Timecode (DOC-5037))

1. Changing the directory Hand Engine will save local raw data files to: 
   
   To change the root directory where files are saved:

   • First create that directory in Windows File Explorer, then copy the full pathname of that directory.

   • In Hand Engine, click Settings → Preferences, then paste the pathname to the Recording Location text box.

     NOTE: It is important to ensure a trailing backslash (“\”) character is at the end of the pathname.

   • The take name can also be formatted as preferred (e.g. device,actor,calibration,handtype,takename,time)

     

      

2. Enabling Automatic Timer Pose Capture: You can automatically run through the poses in a pose library without requiring you to click Capture for each pose by using the Auto Capture Timer. Set the Auto Capture option to On, and specify the Time between capture in seconds to hold each pose before capture begins.
   NOTE: StretchSense recommends starting with an 8-second delay, to begin with, and reducing that time as appropriate.

   1. When Capture is clicked, the Capture button changes to Stop (Auto On) and: 

   2. A timer pops up on-screen counting down from the time specified in Time between capture and the actor must adopt the displayed pose. 

   3. When the timer reaches 3 seconds to go, the actor will be asked to start rotating their wrist(s).

   4. At 0 seconds the capture process begins. The capture process takes ~1 second to complete. 

   5. Once the capture is complete, if Auto Mode is On, the viewport will be updated with the next pose in the pose library and the timer will restart.

   6. Steps a-d will repeat until the last pose in the pose library is captured

      NOTE: The Auto Capture process can be stopped by clicking Stop (Auto On)

3. Browse previous recordings: Select Settings → Browse Recordings to open a Windows File Explorer window at the directory where Hand Engine will save local raw data files to.

   

    

Once you have set up each glove and set up Hand Engine in the configuration you desire, you can save the state of Hand Engine by clicking File → Save Session to download a *.session file of the state of Hand Engine. To restore a previous state of Hand Engine, click File → Load Session and navigate to the previously saved *.session file. 

This process assumes you have already connected each glove you are using to Hand Engine, created a calibration profile with the desired training poses for each glove, have selected Blend or Pose Detection mode, and have configured your Hand Engine timecode source (if applicable).

1. In order to record raw capacitance data, joint angle data, and timecode data first define a Label for the recording, then toggle the Record Switch to Rec. To Stop the recording toggle the Record Switch to ‘Stop’.

   

    

• NOTE: A subfolder is created for each take in C:\Users\<user_id>\Documents\StretchSense Hand  Engine with the naming convention <take name>_<date>_<time>

• NOTE: within this subfolder, two files are created for each glove connected to Hand Engine

  1x Comma Separated Variable (CSV) text file with the naming convention Raw_GLOVE-<x>_<actor name>_<profile name>_Hand_<left/right>_<take name>_<date>_<time>.csv containing (from left to right):

1. Raw capacitance data of sensors (16 columns)

2. Raw IMU data (if applicable) (10 columns)

3. Timecode (device) = timecode on the glove, sourced from the wireless jamsync from Hand Engine. This is a string of numbers representing hours mins seconds and frames, however the colon is missing compared to Timecode (master) (1 column)

4. Timer (device) = a millisecond counter of the device from start up should just increment in 8/9ms intervals (1 column)

5. Timecode (master) = Hand Engine timecode (1 column)

6. Solved joint angles (labelled) (60 columns)

   
   1x FBX File with the naming convention FBX_GLOVE-<x>_<actor name>_<profile name>_Hand_<left/right>_<take name>_<date>_<time>.fbx

   1x JSON file with the naming convention Cal_GLOVE-<x>_<actor name>_<profile name>_Hand_<left/right>_<take name>_<date>_<time>.json containing a snapshot of the calibration profile used at the time the data was recorded.
   
   

1. Select File mode, then click Browse and navigate to a raw data CSV file previously recorded in Hand Engine. 

   

    

2. Select the Actor and Profile you wish to use to interpret the raw data file, and then click Train Blend, Pose Detection, Hybrid or Express depending on your needs and the Profile you are using. 

3. Click Play to start the playback. Click Pause to pause the playback at the current frame and click Play again to resume from that point. Click the Stop button to reset the animation back to the start of the animation file.

   • NOTE: You can reprocess the raw data file using any Actor/Profile and any mode (Blend, Pose Detection, Hybrid or Express). You do not need to use the same that was used for the original recording.

   • NOTE: If you wish to play several recorded raw data files simultaneously, repeat steps 1 and 2 in each of Glove 1-12 in Hand Engine as required, then click the Play button in Global File Playback. This option is useful for triggering the streaming of multiple raw data files to your animation software at the same time.