HandWave

A wearable glove system for translating hand movement to audio queues

Team: Benjamin Meads and Val Clarke

Project

  • Capturing 9-axis data (accel, gyro, mag) from 6 IMUs placed on key hand joints
  • Translating human hand motion into digital data
  • Referencing and translating digital data to a recognized gesture
  • Sending the recognized gesture over Bluetooth to a wearable earpiece system that plays the associated audio signal

Applications in tactical situations, sign language translation, VR input, robotics, and motion capture

System

system flow-diagam

Methods

  • Hardware Input: 6 ICM-20948 IMUs connected via SPI to ESP32S3 microcontrollers
  • Six IMUs are sampled through Free-RTOS-based tasking
  • Gesture recognition is based on a combination of sensor fusion and sequential analysis
  • Communication between microcontrollers is over Bluetooth
  • Audio files are stored on RAM on audio microcontroller

Conclusion

  • Demonstrated end-to-end capture and transformation of hand motion to related audio output
  • Accurate sampling, translation, referencing, and transmission
  • Interfaced with 12 high-fidelity IMUs, reading 220,800 bits per second
  • Built libraries of gesture data in MATLAB to reference against using sequential analysis
  • Built a code library to communicate efficiently with the IMUs

Future Usage:

  • Expand registered data to include ASL
  • Integrate with various programs and hardware (Virtual Reality)