Piano for the Hearing Impaired

For my intensive project, I decided to create a Student Defined Project. I’m passionate about music and social good, so I knew I wanted to incorporate those concepts into my project somehow. I wrestled with a couple of ideas, and decided on building a piano for people who are Hard of Hearing. In a sense, I wanted to build a machine that would have more tactile stimulation and also translate music into a visual display.


Sira N.

Area of Interest

Jazz Composition/ Mechanical Engineering


Trinity School


Incoming Senior

Second Milestone: Visual Component

My second milestone is the visual component of the piano. My vision is that each key will have its own color, and the saturation of that color will be dependent on how hard the user presses the FSR. I also plan to have chords have some sort of visual indication of there quality. For example, a major chord produces a bright or happy sound, and that would be reflected in the brightness of the color displayed on screen. A minor chord produces a muddier, darker sound, and that again would be represented by a darker/duller color on screen. Because the piano has 12 keys, it’s hard to play 7th, 9th, or 13th chords effectively, but I hope to have a range of visuals respective to possible chord qualities achieved with three notes.

Arduino Code

In Arduino, I coded 12 keys, but for simplicity I’ll be describing the setup of just two keys. This code is pretty much identical to my Milestone 1 code, save for what’s being printed into the Serial monitor. In between each FSR value of the individual keys, I’m printing a comma. The final value I want Processing to receive, in this case it’s ‘valBb,’ must used a Serial.println() function, so that each set of values is printed on a new line while the loop runs. This will come into play later with the Processing code. I also added a delay function at the end of the loop as a method of not sending too much information at once to Processing.
I uploaded this code to my Arduino, checked the serial monitor, and closed it to begin my Processing code. It’s important to note that you can’t run the Arduino serial monitor and run a Processing program at the same time, as the port will be occupied by the Arduino program.

Processing Code

The Processing code was a little more complex, but eventually I gained a thorough understanding of sending Arduino data to Processing and displaying the data using Processing functions. I mostly followed this Sparkfun tutorial about Arduino to Processing connection, which explains most of the code in my setup.
Basically, I imported the Processing Serial library and created a Serial object that would allow the Processing program to receive the incoming data from the Arduino. In terms of variables, I have a float variable for each key, which would really be 12 but I’m using 2 for simplicity. It’s important to initialize these variables at a default value of 0. An error I ran into was a ‘NullPointerException’ error. There’s a thorough explanation of the error here, but basically when creating global variables in Processing with no definition they are automatically null until a value is applied to them.
In my setup, I defined the size of the background, which would be 1200 pixels long (12 ‘key’ on screen with widths of 100 pixels) and 500 pixels tall. I set the default background to black, and simply placed the ‘keys’, which were actually 100px by 500px rectangles, adjacent to each other on the screen. For prototyping, I wanted to color the rectangles blue when the user pressed on the FSR, so I made the final parameter of the RGB fill values dependent on the float values that the keys were receiving from the Arduino serial monitor. I also have a bufferUntil() function, which basically triggers the program to change the values of the float variables depending on the character parameter given. In this case I’m using \n, which indicates a new line. This is useful because I have a println function in my Arduino program, so Processing will register each new set of values printed on a new line of the Serial monitor.
There’s a tutorial here on the SerialEvent function, but essentially I’m telling Processing to read one set of values at a time, separated by a new line of data, and store that in a string variable. By storing this in a String variable, essentially each value is apart of a String array, with each ‘character’ having an index in the array. Therefore, the FSR values are converted to string values, but this will be fixed later. After trimming off the potential whitespace on either end of the string, I create an array that holds float values. I define this array by splitting the values in the string by each instance of a comma. This isolates individual values of the FSR readings. Then, I envelope all of this in a float(), which converts the numbers from String values into a numerical float value.

Visual Demonstration

First Milestone: Physical Components

Milestone 1 Video

My intensive project is piano designed for people with either complete or partial hearing loss. The piano has two components that appeal to touch and visual senses. The first component concerns vibration motors with Force Resisting Sensors (FSR). The second part will be an app that registers the keys the user presses and converts these readings into a visual display that uses colors and shapes to convey the visual meaning of the music being played.

My first milestone is the touch component of the piano. I used FSRs, micro vibration motors, and a lot of jumper cables to achieve the functionality I wanted.

Milestone 1

There are 12 FSRs used to represent the 12 notes in an octave, A through G#/Ab. The FSRs are connected to the analog pins of the Arduino Mega, allowing them to register varying levels of pressure. I connected each FSR to their respective clinchers, making it easier to connect to the breadboard using Male/Female jumper cables. I then connected each FSR to an individual analog pin, and placed a resistor right beneath where that connection was being made. I also connected each FSR to the positive well of the breadboard to pass a current through them. I arranged the FSRs to vaguely replicate a piano set up, and went on to work on the micro vibration motors.

I used a separate half-sized breadboard to connect the vibration motors to the Arduino. I used 12 to correspond with the 12 keys on the piano/the 12 FSRs. I connected the blue wires of the vibration motors to the ground well of the breadboard, and used jumper cables to connect the other end of the wire to the Arduino. I connected each motor to a digital pin on the Arduino, which would later receive an analog value to output in my Arduino code.

The code for this setup is a little busy, but that’s mostly by virtue of there being 12 keys to account for. I’ll walk through the setup of the ‘A’ key, but the setup for the other 11 keys is exactly the same.

Each key requires three variables unique to that key. One analog variable for the FSR pin, one digital pin for the vibration motor, and another to store the FSR reading. Three variables that are not unique to individual keys are the motorMap, forceMap, and scalar integer variables. The first two are used for the map function unique to each key, while the scalar is used to manipulate the FSR reading and increase sensitivity. This helps the vibration motors accurately communicate with the FSR in a way that’s realistic and parallels the actual force applied to the piano.

In my setup, I simply have the ‘A’ and ‘Av’ variables set as output pins.

In my loop, I first manipulate the FSR reading and scale it for the reasons mentioned above. I then created a variable named ‘valA,’ where I would store my mapped valued. The scales for an FSR and vibration motor are different, so I needed to use the map function to scale the FSR reading appropriately so they would correspond with values typically applied to a micro vibration motor.

I then used the analogWrite function to write the newly scaled value from the FSR to the ‘Av’ value, which is hooked up to the vibration motor. This makes it so that the force applied to the FSR is represented in how hard to motor vibrates. I then have some serial outputs for debugging purposes.


Starter Project

Video Summary of Useless Machine

The Useless Machine is a box with a motor controlled arm inside. When the user flips the switch, the arm comes out of the box and flips the switch back to the original position. The net gain is 0, making the functionality of the machine useless. This project solidified my soldering abilities, as well as how to go about assembling all the pieces in a logical way, both in terms of functionality and position.


How it works

The Useless Machine is powered by three triple 1.5V batteries. There is PCB circuit connecting all the parts of the machine, and it has two resistors, a flip switch, a snap switch, a 3 pin LED, and the wires of the motor attached. Attached to the motor is the acrylic arm, that will eventually flip the main switch that protrudes from the machine. When the user flips the switch, it closes the circuits in the PCB, allowing current to flow through all the components of the machine. Immediately, a green light from the LED is emitted, and the motor moves in one direction, allowing the arm to come out of the compartment and reach the switch. The motor is polar, so when the arm eventually flips the switch back into its original position, the motor starts to move in the opposite direction, bringing the arm back down. In addition, the LED emits a red light. Since the LED has two pins, it is able to emit two colors depending on where the current is being transferred to. When the switch is flipped back by the arm, the current is being transferred to the third LED pin, allowing red light to emit. When the arm goes back into the machine, a snap switch is directly beside it, and when the arm presses on it it signals the motor to stop moving, essentially turning the machine off.

These are the instructions

Useless Machine Schematic

Sira N.
Image Source: https://github.com/adafruit/Adafruit-MiniPOV4-Kit/tree/master/Hardware

Start typing and press Enter to search