Interactive Virtual Game based on Gesture and Handwriting Recognition
Interaction - Processing - UX - Computer Vison
CONTEXT:This project was my master’s thesis, where I was part of a 3-student team aiming to tackle the following research proposal: “A user-centered technological intervention to motivate physical activity among patients in single-bed hospital rooms”. This project was done in collaboration with Herlev Hospital, which facilitated the access to patients and hospital resources.
HOW:
We followed the double diamond framework, and after doing an extensive research and diverging and converging the problem, the solution was designed and validated with patients. It takes the popular New York Time’s game Wordle and gives it a twist by incorporating physical activity to it. To do so, the solution has been powered with hand and optical character recognition. In order to input a letter, the user needs to draw the letter in a virtual canvas using their hand, and then the drawing is processed into a letter. This requires the user to be active as all the interaction takes place using the virtual interface. The graphics are done using p5.js, the hand recognition using mediapipe and the OCR using tesseract.
We followed the double diamond framework, and after doing an extensive research and diverging and converging the problem, the solution was designed and validated with patients. It takes the popular New York Time’s game Wordle and gives it a twist by incorporating physical activity to it. To do so, the solution has been powered with hand and optical character recognition. In order to input a letter, the user needs to draw the letter in a virtual canvas using their hand, and then the drawing is processed into a letter. This requires the user to be active as all the interaction takes place using the virtual interface. The graphics are done using p5.js, the hand recognition using mediapipe and the OCR using tesseract.
LEARNINGS:
This has been the biggest and more multidisciplinar project I’ve been part of. It has involved extensive research regarding the users and the hospital enviroment, the current market, and the technology. In terms of development, it initially became a challenge to combine CV with a user friendly interactive interface, but I’ve learned a lot on the process.
This has been the biggest and more multidisciplinar project I’ve been part of. It has involved extensive research regarding the users and the hospital enviroment, the current market, and the technology. In terms of development, it initially became a challenge to combine CV with a user friendly interactive interface, but I’ve learned a lot on the process.
