Syng.

What? Named after the Greek word root syn meaning “with” or “together”, Syng is a synesthesia-inspired ear training web application designed to improve its users’ singing skills and note recognition by visualizing pitch.

How? Syng uses an open source machine learning model, called ml5.js pitch detection, to identify a pitch from microphone input. Ml5.js pitch detection identifies the frequency of sound in hertz and Syng is programmed to match the frequency to its corresponding musical notes (with a margin of error). The application guides singers through a three-part experience.

Part 1: Intro. The intro pairs each note, regardless of octave, to a particular colored circle. The objective of this introduction is to establish to the user that sounds have pitch frequency and therefore correspond to musical notes. These note-color pairings remain the same throughout the entire experience to create consistency. For example, the note C will always be red.

Part 2: Pitch Match. Pitch Match is the ear training mode. Users can play a tone and sing it back with a visual cue of whether they are sharp, flat, or perfectly matched. 

Part 3: Perform. This part is the final freestyle mode that provides a more elaborate and expressive visual while singing. The color changes to follow the pitch mapping used throughout the whole piece and the opacity changes with the vocalists volume.

Why? A common hurdle for novice singers and even advanced vocalists is learning to stay on pitch. An inexperienced singer may hear a note but will be unable to reproduce it precisely. However, if singers are not familiar with musical notes or even hearing themselves sing, how can they identify their mistakes? Syng was created to provide approachable music education that using the combination of sound and visualization to enhance learning.

The experience incorporates technical applications of dynamic web development, machine learning, JavaScript, and HTML/CSS.

See more about the research and design process:

RETURN TO PORTFOLIO

Music Interaction Design: Final

The time has come! After a whole semester developing this project, I’m proud to show how far it has come.

Post User testing, I decided in this last week to retain a lot of the existing functionality because it actually went over quite well. Why fix something that’s not broken. I put my attention towards aesthetics to highlight its simplicity. It is a visual piece and should look the part!

I learned so much about HTML and CSS. It was amazing to feel that in control of aesthetics of web design. I played with some different mappings in the performance mode, but felt it already looked the best that I could make it. I did successfully change the mapping on pitch match to be a bit smoother, which felt like the area that needed it the most. Lastly, I was able to combine the pages so that it truly feels like a navigable website.

I decided to emphasize the order and make it feel more like a progression. Each mode builds off the next.

In future iterations, I would love to add a harmony mode with color mixing and experiment with multiple mic input so it can be a collaborative experience.

Visual Style Guide:

Music Interaction Design: User Testing

Accurate pitch detection was my biggest challenge in this project journey. However, once I got a handle on ml5 and the pitch detection trained model, my concept got new life. I focused on creating 3 modes for the user test: an introduction to learn how the colors are associated with notes, a pitch match, and a free form performance mode.

I was really curious how people would interact with the project and if it would actually feel useful. My paranoia has been that the concept is too simple, but I was pleasantly surprised and received great feedback.

FUNCTIONALITY

My top priority was making this work. Are the mappings informative? Does it accomplish the goal (visualizing singing) in a meaningful way?

Color notes:

“The first one is confusing, I’m not sure what to do”

“I’m not sure what’s going on”

For the first mode people seemed a bit lost about what to do. There was a natural tendency to look for a pitch match scenario. Someone wanted less explanation of the task, another person wanted more, and most people didn’t really read the instructions.

Pitch Match:

“Pitch match is more effective for learning, maybe you need more explanation or a visual cue.”

“I’d like more feedback when I’ve gotten the right note”

“It’s satisfying seeing the circles match”

“i’m not that bad”

“I need this”

Pitch Match came across very straight forward and most people enjoyed it as a challenge. I liked watching people play the note many times to recenter themselves and try again. This is exactly how I intended it to be used.

Performance:

“there is good initial feedback in performance mode. It’s a bit fast”

“that initial fill is very surprising, maybe that can be used more”

“I can see the range of the song in the performance mode, saturation is mapped well to volume”

As expected this mode got a lot of feedback because it’s more expressive. People seemed to genuinely like the color to frequency mapping as well as the opacity changes based on volume. It was fun watching people get excited about singing a song when it was paired with the visuals, even country national anthems!

AESTHETICS

This project has a huge visual component so I took seriously any comments about improving its look.

“In the performance mode, maybe the graphics could be more feathery or painter-like”

“it should be as stylish as possible”

“circle buttons?”

“I like the color, its so beautiful”

My next step is to improve the navigation between these modes and make it resemble more of a website. I would love to add more modes, but will see how time allows. I feel better about it than I thought and am excited to make it polished.

Music Interaction Design: Post Midterm Progress

After presenting my midterm, I felt I was on the right track. I received positive feedback for the overall concept and aesthetic, but was encourage to take it further. In addition, I realized that the FFT() and getCentroid() functions I were using didn’t quite give me the accuracy I was looking for. Although it was suggested before to use ml5, I think I stayed away from it because I was too nervous to delve into another coding library. I stuck with what was familiar but I ended up applying the wrong tool to my task.

Since the midterm my main focus has been transferring my current progress over to ml5 for improved accuracy. However, it almost feels like a step back because it took me a while just to understand ml5 and get the pitchDetection examples running (after starting a server, I can only run the pitchDetection examples in FireFox, no other browsers! grrr why??).

Coding for me is a quite painstaking activity. It is kind of ironic that I’ve chosen a final project that is primarily coding-based, so it is slow going, but it is going.

I wish I had more to show, but I feel good about my current understanding of ml5, the example pitchDetection programs, and how to modify them for what I want to do.

Music Interaction Design: Midterm

Concept.

My project is a music learning tool for singers. Inspired by the phenomenon of synesthesia, the project combines color with notes as a way to provide a visual to sound. For an untrained ear, one may not be able to hear if she is singing incorrectly, but perhaps she could more easily adjust if she sees it.

I believe it is pretty entertaining. For the initial prototypes it certainly feels game like and participants become determined to see how their voice can change things. I think it is a good entry point to music learning because its simply exploring your own ability to listen to and produce sound.

Technology. Briefly, how does it work?

The project mainly uses the getCentroid function of FFT in p5 sound to identify pitch. A recording (with tuning tones) are analyzed and parsed for pitch. Live microphone input, to capture voice pitch, is analyzed the same way. Voice input is mapped in certain ways to give the user a better sense of what’s happening.

In action.

PROTOTYPE 1: This first attempted tried to map pitch to the width or height of the voice circle. Meaning that a flat note would squish the circle making is squat and stretched horizontally. A sharp note would become tall and narrow. Thus, the user is meant to aim for a perfectly round circle.

However, with all the ranges/bins of centroid values, it ends up producing more than 2 circles. This is confusing. My goal is to have one circle that’s static to represent the given pitch and an ever changing circle for voice input.

PROTOTYPE 2: Gone away with mapping pitch to width and height of the ellipse, and instead, this version asks the user to match the diameter of their voice circle to the diameter of the pitch circle. This is most similar to my initial modular prototypes. I also added a “confidence” mapping. This links volume to voice input. The louder the user sings, they create a bolder color.

PROTOTYPE 3: I still feel compelled to illustrate the nuances of pitch. So in this version, the circle travels vertically depending on whether the voice note is flat or sharp. This metaphor works well, I think, because you can think about producing sound as going below or above the target. Volume mapping is included in this version as well and as an exciting artifact, the color mixing for harmony is especially prominent.

Next steps.

• Although tones are associated with unique frequency values hertz…getting p5 to recognize them in the same way is almost impossible….in addition, something like the human voice, has way more idiosyncrasies than a computerized singular tone. Thus, my main battle is with noise. I tried different filtering techniques but still have a weird strobing effect that I don’t like. My next step is to really figure this out so that the aesthetics can be purer.

• I would like to continue building the interface to figure out the best way to help people learn to sing.

• title?

On another note, this project has really encouraged me to play with Ableton. I don’t think it relates to my project yet…but I wanted to also share my 2nd real attempt at a song:

Music Interaction Design: Mappings

This week we thought about mappings as a means to give the user more control. What aspects of music or music making could be controlled discreetly versus continuously? Should you map “one to one” or “one to many”?

I’m still working on this project very modularly. My project as of now only receives input, so its hard for me to think of multiple mappings. However, I thought I’d try breaking the voice down further and map components of voice sound.

Quick p5 sketch of the volume mapping. Singing more loudly produces bolder color!

Played around with a MIDI keyboard from the ER for the first time ever this week. Hooked it up to Ableton! Very fun.

Everything is really simple right now, partly because I’m learning as I go and I’m using what I know, but I also feel like the general design is quite simple. Is this a flaw? I want it to be super entry level, for now for the beginner user…but maybe this is boring.

Music Interaction Design: Thinking about the INTERACTION part

Verplank’s Design Framework:

Idea: Create a voice based music turn for learning and performance

metaphor: Synesthesia - a perceptual phenomenon where senses cross

model: web-based program (JavaScript.)….for now

display: projection or monitor

error: off pitch singing

scenario 1: beginner singers may not know they aren’t singing on key because they cannot not hear their errors

scenario 2: singers in an ensemble have trouble blending

scenario 3: singers want a live performance tool to add visuals to their sound

task 1: sing with a recording to match pitch

task 2: sing with recording or with others to create harmonies

task 3: sing freeform

control: voice controls visuals on the screen

7-degrees Axis Amount:

*I feel like these axis will move drastically depending on who uses my project, and I kind of like that. I’m just not sure how much of it is a natural outcome from the project or something I’m crafting. For example, a professional could make the sound quite expressive in the freeform version, but a beginner is using it to learn. I think it would be cool to play with more input somehow….perhaps volume or in a way “confidence” can be visualized as well.

Do, Feel, Know interaction 1: PITCH MATCH

User sings, and sees/feels the diameter of the change, they know pitch is matched if the diameter of the original circle lines up with the circle their voice controls

Prototype 1: https://editor.p5js.org/mpruitt/full/IOJ3ao-Wa

Video of user test - password: prototype1

Do, Feel, Know interaction 2: BLENDING

User sings, and sees color change, they know pitch has blended if the color mixes

Here is where I’m getting a little stuck. So far I’m looking at the project as quite modular, but it would be cool to give the user more control over the sounds they can make. I’m not sure how to implement this….do I do it by visualizing more nuance of voice to encourage them to sing in different ways? Should it be physical controls on the application that alter sound?

Music Interaction Design: Harmony & Chord Progressions

1. Music Teaching Tools: Harmony. Make some music using the following: chord progressions, arpeggios, chords. Write a short response discussing your experience. 

I attempted some chord progressions a la “The Four Chords Song”. I don’t have very good rhythm andIi’m not sure if it’s correct but it was fun. Though I’m not very good at playing the chords, I definitely see how the melody aligns on top of them.

Start thinking of interactions: what user actions will determine which musical outcomes? What aspects of the music will evolve independently of user actions (if any)?

I started playing around with visuals to sort of illustrate the concept in my head. The program below changes position and color on the canvas according to mic input and FFT analysis.

https://editor.p5js.org/mpruitt/full/O431kGklJ

Then there are a few animations below of how colors/size could change according to notes.

Pitch match

Moving forward: I really need to figure out if a computer is able to identify pitch from mic input.

Maybe this can help? https://tonescope.net/scope

2 note harmony

Music Interaction Design: Progress Report

Target Audience:

I have been thinking a lot about this project this week. I think the challenge of designing a user path, rendered me a bit of artist’s block. My original intention, I’m realizing is more SOUND interaction verses MUSIC interaction. I still need to flesh this, but going over all the music education tools in class was quite inspiring. Aaron made a comment in class, that these tools don’t often encourage singing which is for many people an easier entry point. So I think I would like to make my target audience singing learners. At least for now. I’d like to continue exploring the idea of vocal input + data visualization. A tool that helps singers learn pitch by following the user path. Then perhaps after this tutorial, they can be encouraged to use it to make their own sounds/music/performance, which then more experienced music makers could enjoy as well.

Aural Mood Board:

My vocalist friends declare this as a more introductory level choir piece. I think this could be a good place to start for easier harmonies and it is quite recognizable.

Another recommendation from my classically trained friend. We like the color and texture of the different voices in combination.

An example of stunning high notes, supported by the other vocalists. I like that it features an individual but still sounds whole only when there are many voices.

Harmonies with more of a synth/techno sound. This could be an interesting option. Perhaps a user can harmonize with software sounds as opposed to other voices.

A fun Jazz piece with vocal harmonies. I like the playfulness of the fast rhythm.

I love when you can hear how a harmony builds. The simplicity of adding each part over the original voice is very satisfying.

The human voice is a powerful instrument. I love how this completely instrumental piece with no lyrics of any kind transformed when sung by a group a cappella.

User Path:

Still working on this. I have never made music before and Ableton is quite overwhelming. I tried recording voice, but the volume is almost indiscernible!

But ideally the path is a few individual notes, with pause for the user to repeat them, move to chords, then melodies.

Music Interaction Design Prompts

A brief description of the concept that includes what it does, who it is for, and where it lives (not more than a couple of sentences)

• A drawn sketch (or sketches) that indicates form, materials, scale, and interaction

• The song you started from

• The oblique strategy you got

“My Boy” by Billie Eilish

Basic catalog of attributes:

• Key: B minor

• bpm: 90

• drums, female vocal’s (breathiness), “ghostly sound”, synth, harmony

Random oblique strategy: “Make something implied more definite (reinforce, duplicate)

Kemi and I imagined that we work for Billie Eilish and her team. We created an installation with a supplementary mobil application. The installation features a large touch screen that with squares that highlight different aspects of Eilish’s music, emphasizing the implications of her sound. Touching the center would showcase the original sound, and moving outward to the corners breaks it down into components of what created that original sound.

We also discussed having the user input their own sounds, either speaking or singing, and have our device modify it, almost like a filter, to sound like these parts of her music.

2. Write a project prompt for yourself. You will use it to frame subsequent assignments, but it can evolve/change later. Submit it here.

Build a tool that allows one person to sound like many: I am thinking of a machine of some kind that would allow the user to sing into it and have the output harmonize with them, kind of like a one-man a cappella group.

Design a visualization of sound: For this I imagine two possibilities. 1) In my collective play course we are using web sockets. I see this as an awesome opportunity to have multiple clients input to/modify a collective outcome. I imagine singing produces colorful fuzzball that combine and mix color based on harmony. 2) this could also serve as a teaching tool for singers perhaps displaying perfect pitch versus the users so they can learn how to match it.

Create an installation that plays with the principles of acoustics: Not exactly sure what to do with this, and I’m not sure what its called, but some places are designed so if you stand in one part of a room you can actually hear what’s happening in a completely different part of the room.