Gesture Controlled Game

The principle idea of this project, is to design an adventure/survival game on the Unity Game Engine, which uses the gesture controls of an arduino controller to interact with the game mechanics.


Marco H

Area of Interest

Computer Science/ Game Design


Northeastern University


Incoming College Freshman


First Milestone

I successfully developed the basic mechanics and design for the adventure/survival game. This includes running and fighting actions. I used the Animator Controller to manage the different animations based on my current computer input. This will be updated when the Arduino controller is completed. Additionally, the basic interface for Arduino-Unity communication has been established. Using the “Ardity” asset from the unity asset store, simple communication between Unity and Arduino was made possible. On the design side, the basic environmental design has been developed with trees, rocks, grass, rivers, and terrain all in place to allow for level design in the coming week. These models were created by me on Blender. All modeled characters were given skeletons to be able to develop animations.

Character Design+ Fighting Animation

Fighting Action

Flowing River

River Water

First Milestone Video

Second Milestone

For this milestone, I created the Arduino motion controller that can track the motion of the player’s arm through a six degrees of freedom motion-sensing accelerometer. The controller also includes a joystick to allow for motion, in the game. I soldered all components onto a perf board, which is then connected to the Arduino through multiple wires.

For the controller to properly work, there were two essential components that needed to be connected.

  1. The gyro/accelerometer
  2. The joystick

To attach the accelerometer, I had to make five connections. Firstly, a connection to power, as demonstrated by the red line. Secondly, a connection to ground (GND) to complete the power circuit (black line). Thirdly, two connections between Analog pin 4 and 5, and the SDA/SCL connections on the accelerometer. And finally a connection between INT and the 2nd digital pin.

To attach the joystick, I connected the GND pin with GND (black cable), and the 5V pin with 5 Volt on the Arduino. Next, I created a connection between VRX and Analog 0, for x-axis input and I connected VRY with Analog 1 for y-axis input.

The last connection I made was SW, with digital pin 4 to account for the pressing in of the joystick. With the last connection made, I soldered the joystick onto the board and soldered all the connections together to avoid connection problems between wires.

Second Milestone Video

Third Milestone

With this milestone, the main 3 step plan of the project is finished. Having built the controller, and developed the game the final part of this project was two integrate the two parts together. This was essentially made up of receiving the data being printed by the Arduino in Unity and then using the data in unity to enable certain actions.

To accomplish this I first organized how my data would be sent, into a more organized fashion: First sending rotation x,y, and z data. Then sending translation x, y, and z data. And finally sending joystick x, y, and button pressed data.


Once uploaded to the Arduino, I turned my attention to Unity to receive data from the controller. I primarily achieved this by using the “Ardity” asset on the Unity Asset Store. Once implemented, this asset allowed me to read all the data coming in from the Serial Monitor of the Arduino controller. Then I wrote a script that would break up the data into separate strings, and then I parsed the strings into actual numerical values that the game could read.

With the data finally being received by Unity, I began adjusting the values I received to be more useful for in-game mechanics. Values coming from the joystick for example, could be anywhere between 0, and 1023. Each representing the joystick being tilted on an extreme end of the respective axis. I normalized this range to be between -1 and 1 to be more compatible with my other input’s range of -1 and 1.

In regards to accelerometer data, I created a method in the Arduino code that would convert the raw data from the accelerometer, into readable values, or Euler Angles (degrees).


Once all the data was properly formatted I edited my movement script to allow for joystick controls and created a second to detect when your arm was swung. This second script simply paid attention to when the accelerometer would have a certain amount of acceleration, and how much its angle changed, to determine if a swing had occurred. If it did detect a swing, based on the acceleration and angle, it could determine which direction, and run the respective animation for it. Thus, the game’s controls were integrated with the Arduino controller.

Opening Menu

Finally, to wrap things up, I designed a user interface, with the player health bar, and the pillar’s health bar. If you didn’t defend the pillar it could be destroyed, and the game could be lost.

I also created a menu that would allow you to play again or exit. And I created an opening screen when you launch the game to allow the controller time to connect. It was here where I decided to call the game “Knight Attack!!!”

And with that, this is how the final game looks like!!!

Start typing and press Enter to search

Bluestamp Engineering