Student work Sound of Shape

Sound of Shape was a group project developed during Tangible Interfaces class subject at MIT Media Lab. It consists of a strand or rope-like object with rigid segments connected by flexible joints, which the user can manipulate to create the outline of different shapes. Each new shape becomes a new musical instrument that the user can play with intuitive gestures.

Tools and materials: Styrene, fabric, flex sensors, accelerometer, microcontroller.
Group: Carol Rozendo, I. Wicaksono, R. Ye, J. Trapp.

 

HOW IT WAS MADE

In the ideation process, our group took the challenge of imagining an interface that could create a connection between senses. We discussed sculpture, sound, dance and three dimensional shapes, and decided on an interface that would allow users to explore the materiality of sound in an intuitive way.

We chose the form of a sensitive strand that would have affordances similar to a drawing tool, but with limitations so that people with little skill for drawing would still be able to use. With this strand, the user would be able to make the shape of instruments and use them to generate sounds.

The strand-like object is formed by a succession of rigid segments connected by flexible joints. An array of short styrene segments on each side of the object provides structural support for twenty flex sensors and their wirings.

A long brass strip between these layers helps retain the shape after a hinge is bent. At the ends of the strip, in a final segment which also functions as a handle, an accelerometer, proximity sensor and magnets are attached.

While manipulating the object to the desired shapes, the user may shake it or tap it and hear which sound was generated. This allows the user to naturally explore the possibilities of instruments by transforming the object's shape. A round shape may be mapped to a drum, a bell figure to a cowbell and a star to a high-frequency bongo, for example. Once the desired shape is achieved, the user may play it intuitively, like a tamborine or other percussion instrument.

 

SOUND-SHAPE CORRESPONDENCE

The mapping of shapes into different sounds or instruments takes inspiration from a number of articles and research experiments, which suggest that the way how we connect auditory and visual-tactile perception is not completely arbitrary.

In the "Kiki-Bouba" experiment, conducted for the first time by Kohler in 1929 and replicated a few times afterwards, people were asked to match a spiky angular shape and a round, smooth shape to the nonsense words "kiki" and "bouba". There was a strong preference to associate "bouba" to the round, smooth shape, and "kiki" to the angular shape, independent of the language spoken by the individual. In a replication of this experiment conducted in 2001 by Vilayanur S. Ramachandran and Edward Hubbard, this preference accounted for at least 95% of the subjects.

In an article published by Nature in 2016, Chen and Huang investigate how this correspondence manifests across languages and contexts, and analyze the association of frequency or amplitude to shapes.

Sound of Shape extrapolates on the idea of visual-auditory correspondence present in these studies, applying the concept to a direct connection between the general appearance, the acute or open angles in figures, and the perception of sounds generated by certain instruments.

 

My main role on this project was to think of how the initial idea would materialize in a real object, fabricate such object, and figure out limitations and potentialities of the physical interface. For example: how can we make the association between the shapes and their correspondent sounds more intuitive to the user? After the first prototype, we discovered some technical problems, especially with weight and feel of the object and with the fragility of the sensors with the action of bending. I am currently working on a new iteration of this interface.