Music Interaction Design: Mappings

This week we thought about mappings as a means to give the user more control. What aspects of music or music making could be controlled discreetly versus continuously? Should you map “one to one” or “one to many”?

I’m still working on this project very modularly. My project as of now only receives input, so its hard for me to think of multiple mappings. However, I thought I’d try breaking the voice down further and map components of voice sound.

Quick p5 sketch of the volume mapping. Singing more loudly produces bolder color!

Played around with a MIDI keyboard from the ER for the first time ever this week. Hooked it up to Ableton! Very fun.

Everything is really simple right now, partly because I’m learning as I go and I’m using what I know, but I also feel like the general design is quite simple. Is this a flaw? I want it to be super entry level, for now for the beginner user…but maybe this is boring.

Music Interaction Design: Thinking about the INTERACTION part

Verplank’s Design Framework:

Idea: Create a voice based music turn for learning and performance

metaphor: Synesthesia - a perceptual phenomenon where senses cross

model: web-based program (JavaScript.)….for now

display: projection or monitor

error: off pitch singing

scenario 1: beginner singers may not know they aren’t singing on key because they cannot not hear their errors

scenario 2: singers in an ensemble have trouble blending

scenario 3: singers want a live performance tool to add visuals to their sound

task 1: sing with a recording to match pitch

task 2: sing with recording or with others to create harmonies

task 3: sing freeform

control: voice controls visuals on the screen

7-degrees Axis Amount:

*I feel like these axis will move drastically depending on who uses my project, and I kind of like that. I’m just not sure how much of it is a natural outcome from the project or something I’m crafting. For example, a professional could make the sound quite expressive in the freeform version, but a beginner is using it to learn. I think it would be cool to play with more input somehow….perhaps volume or in a way “confidence” can be visualized as well.

Do, Feel, Know interaction 1: PITCH MATCH

User sings, and sees/feels the diameter of the change, they know pitch is matched if the diameter of the original circle lines up with the circle their voice controls

Prototype 1: https://editor.p5js.org/mpruitt/full/IOJ3ao-Wa

Video of user test - password: prototype1

Do, Feel, Know interaction 2: BLENDING

User sings, and sees color change, they know pitch has blended if the color mixes

Here is where I’m getting a little stuck. So far I’m looking at the project as quite modular, but it would be cool to give the user more control over the sounds they can make. I’m not sure how to implement this….do I do it by visualizing more nuance of voice to encourage them to sing in different ways? Should it be physical controls on the application that alter sound?

Continuing Conceptual Package Development

We showed our first conceptual package to Brett, who gave us the following feedback. As expected our biggest critique was to take our pool of 10-12 ideas and focus it to 1-3 truly executable things.

FEEDBACK:

• Don’t need brainstorm section in the conceptual package

• **Add section with the research and the researcher (Keep pics of his work and add pic of him) 

• Think about the tech we’d like to work with (Projection mapping, AR, animation (2D or 3D))

• *digging through time & jumping rocks are strong ideas

• Sketches are awesome! - Don’t need to colorize them 

• Globe is hard to do physically → could make it flat or curved

• **can def fabricated the rock wall or the stackable time 

• THINK OF SCALE: Could do more than one of these ideas, but maybe metaphors can be turned into “cheapies”, Shrink down, do a panel, do an animation

• Physical volcano, can do dramatic lighting, could project to do the non-physical part

• Choose earth’s adventure could scale down → app, share your earth, compelling as small thing

• games:

  • Evolution has to be shown through time...show that time scale, look up interactives about evolution: Spore, diff choices lead to diff abilities 

  • Don’t press the button could be scaled, but education/payoff is smaller

• Describe what about the mood board relates to the project (magical lighting, walking through, tactile, experiencing ) Keep at the end to see how the sketches pair with the mood board part.

• Clarification

  • Message: overall message of the exhibit

  • goal/takeaway: talk about that for EACH concept...tie back the experiences to the learning goals

  • DO THIS then you KNOW THIS

MARCH 7TH WE PRESENT OUR CONCEPTUAL PACKAGE

• 20 min presentation, Make pdf version, send to him before the night before for feedback **Everyone must speak

Conceptual Package Version 1

In the making of museum exhibitions, we learned that after brain storming comes the narrowing of ideas. Brett asked us to prepare a conceptual package that would synthesize our research and propose our plan for production. For our first version we presented tons different ideas in order to get feedback on which would be the the most feasible to pursue.

Below is an outline of our conceptual package components, as well as an initial slide deck.

CONCEPTUAL PACKAGEV V.1

MAIN INTERACTIVE IDEAS:

1. Interactive rock layers (AR, stackable sandbox with projections)

2. Narrative (ancestor shrew → mammal descendants, volcano → dust → no light → dino → dead)

3. Dark room/Mass Extinction experience (projections, VR)

4. Geological timeline (jumping rocks, Globe - to show areas of geological evidence or illustrate different time periods)

5. Games: choose what happens to the earth, build an animal to survive extinction

What is the research?

1. Geology & rock layer analysis

2. Mass Extinction → Dust clouds and volcanic eruption

3. Space: astrophysical Galaxy

4. Connection between dark matter and volcanoes and asteroids

Message

1. What is our focus? 

   • Scale of “deep time” & “deep space”

   • The experience of a mass extinction: what does it feel like?

   • We could include some of Dr. Rampino’s more speculative theories, but we will specifically focus on geological evidence and how it relates to specific events that are agreed upon (i.e. mass extinction via comet, dust cloud)

2. What is the takeaway for others?

   • Mass extinctions are cyclical due to earth’s orbit in space that triggers both asteroid impacts and volcanic eruptions

   • Without the mass extinction of the dinosaurs, we would not exist. “We are here by chance” - Rampino

   • A more holistic understanding of our earth requires the consilience of science. Where separate disciplines intersect is where we discover a better picture of how the world works.

Audience

1. Who? 

   • Families 

   • All ages

   • Anyone who is interested in science and history

2. What type of museum?

   • Science/Natural History Museum

Goals

1. How awesome and necessary it is to show this

   • This exhibit is necessary because it is a tool to learn about and reflect upon why the earth is the way it is. Through our interactive, we will capture abstract concepts, such as the universe’s timeline, mass extinctions, and how we humans came to exist. 

   • Although the next extinction will not occur for millions of years, we hope to convey the idea that Earth is a special place to live and worth taking care of. While things like dark matter, asteroids, and volcanoes are out of our control, there are daily things we can do to protect the earth we have.

Sketches/Visualizations (can be shown as Renderings of interactive design form, digital illustrations, 3D models, can include notes on interactive components & materials)

Written Description 

Written version of interactive design

Prior Art Examples/references

• OpenSpace

• mammal infographic

• Interactive style

• How to understand how long 65 million years is:

Goal: What we hope users get out of the experience

In honor of the shrew that survived mass extinction, we deemed our team name: Ancestor Shrew

Ancestor Shrew doodle by Dylan.

Music Interaction Design: Harmony & Chord Progressions

1. Music Teaching Tools: Harmony. Make some music using the following: chord progressions, arpeggios, chords. Write a short response discussing your experience. 

I attempted some chord progressions a la “The Four Chords Song”. I don’t have very good rhythm andIi’m not sure if it’s correct but it was fun. Though I’m not very good at playing the chords, I definitely see how the melody aligns on top of them.

Start thinking of interactions: what user actions will determine which musical outcomes? What aspects of the music will evolve independently of user actions (if any)?

I started playing around with visuals to sort of illustrate the concept in my head. The program below changes position and color on the canvas according to mic input and FFT analysis.

https://editor.p5js.org/mpruitt/full/O431kGklJ

Then there are a few animations below of how colors/size could change according to notes.

Pitch match

Moving forward: I really need to figure out if a computer is able to identify pitch from mic input.

Maybe this can help? https://tonescope.net/scope

2 note harmony

Meeting with Dr. Mike Rampino

We arranged a time to meet with Dr. Rampino and were invited to visit his office. He was a delight to talk with and explained his research very clearly. He also showed us different examples of geological materials. It was extremely helpful in our early idea forming process.

Below are some images from Dr. Rampino’s office:

Link to my notes.

We asked Dr. Rampino what he would suggest we focus on, which was really helpful in framing the important aspects of his research. One of my other big takeaways was that although mass extinction has a dark undertone it also allows for evolution and rebirth. Without the extinction of the dinosaurs, humans wouldn’t exist.

Aspects to possibly focus on:

1. Cycle connection between mass extinction & dark matter (the disturbance) 

2. The ways mass extinction takes place (internal - volcanoes vs. external - asteroid impact)

3. Geological time scale (deep time, deep space)

Tilt Brush Drawing

For this assignment, we dove right into VR, drawing in 3D based on a description written by one of our classmates.

These were the words of MaryAnn Talavera:

This January I went back to the Dominican Republic and I got the chance to visit my mother’s hometown, Enriquillo.  It’s a small town in the southern region of the island. I hadn’t been back to Enriquillo since I was a little girl and it was wonderful to go back as an adult because I think I was able to appreciate it’s natural beauty even more.  My mother has always told me stories about her childhood in Enriquillo. I felt like I was able to connect to those stories by physically being there. 

Her stories were often related to the ocean, which makes sense because the blue color of the beach in that area is so intense.  

One memory was of her as a small child sitting on the rocks below the cliffs that meet the shore.  Although, they scared her, she would be mesmerized by the power of the waves. One day she waited too long to go back to the shore and the tide had risen up so much that she couldn’t swim back.  So she had to scream out for help. Thankfully, she was heard by a neighbor who was able to help her reach the shore. That story always stuck out to me because it illustrated how the ocean is both majestic yet dangerous; it’s important to respect nature’s power.

This was an awesome experience, though quite disorienting at times! It was most difficult for me to establish a ground, in both my drawing and for myself. I had to draw a tree to help me get a better sense of the space. It started to come together once Gilad showed me how I could change perspective and shrink/grow the canvas. Game changer.

I wanted to capture both the serene calmness of the ocean that MaryAnn described, as well as its powerful brutality.

Explore on Poly

Music Interaction Design: Progress Report

Target Audience:

I have been thinking a lot about this project this week. I think the challenge of designing a user path, rendered me a bit of artist’s block. My original intention, I’m realizing is more SOUND interaction verses MUSIC interaction. I still need to flesh this, but going over all the music education tools in class was quite inspiring. Aaron made a comment in class, that these tools don’t often encourage singing which is for many people an easier entry point. So I think I would like to make my target audience singing learners. At least for now. I’d like to continue exploring the idea of vocal input + data visualization. A tool that helps singers learn pitch by following the user path. Then perhaps after this tutorial, they can be encouraged to use it to make their own sounds/music/performance, which then more experienced music makers could enjoy as well.

Aural Mood Board:

My vocalist friends declare this as a more introductory level choir piece. I think this could be a good place to start for easier harmonies and it is quite recognizable.

Another recommendation from my classically trained friend. We like the color and texture of the different voices in combination.

An example of stunning high notes, supported by the other vocalists. I like that it features an individual but still sounds whole only when there are many voices.

Harmonies with more of a synth/techno sound. This could be an interesting option. Perhaps a user can harmonize with software sounds as opposed to other voices.

A fun Jazz piece with vocal harmonies. I like the playfulness of the fast rhythm.

I love when you can hear how a harmony builds. The simplicity of adding each part over the original voice is very satisfying.

The human voice is a powerful instrument. I love how this completely instrumental piece with no lyrics of any kind transformed when sung by a group a cappella.

User Path:

Still working on this. I have never made music before and Ableton is quite overwhelming. I tried recording voice, but the volume is almost indiscernible!

But ideally the path is a few individual notes, with pause for the user to repeat them, move to chords, then melodies.

Playful Communication of Serious Research begins...

OFFICIAL TEAM: Dylan Dawkins, Mingna Li, Emily Lin

After forming our teams, we began emailing potential researchers (a total of 9 to increases our chances). While waiting for replies, we started brainstorming possible interactives for the ones we were most interested in.

Organized chart for contacting potential research collaborators.

We were particularly excited about Dr. Mike Rampino and his studies of volcanoes and mass extinction! We expanded our brainstorming for his research just for fun.

Only a couple hours after reaching out to Dr. Rampino, he replied and said he would be willing to work with us. What luck!

Scent Tasting: a Study of Presence by Jacky Chen & Maya Pruitt

Since VR is often highly visual, we wanted to pair it down on what other elements or sense modalities make us feel present or immersed. The experiment invited participants to smell 6 different scent and listened to an accompanying sound, while blindfolded, inspired by a sort of wine tasting.

These were the pairings:

1. Apple Cider + Informational narration about Cider making

2. Brazilian Coffee + Personal story from coffee growers

3. Lavender + Sound of buzzing bee

4. Smoked Cedar Whiskey + “Tennessee Whiskey” by Chris Stapleton

5. Chocolate Chai + Ambient sound of an Indian market

6. Rum + A cappella pirate song

Some pairings had more obvious correlation, others were purposefully contrasting.