Just Listen: Prototyping with TinyML to Augment Everyday Sound

This thesis presents an investigation into everyday sound through a series of speculative experiments and one final refined interactive prototype. The everyday sound here refers to non-verbal sounds, such as the sound of a door opening or someone sneezing. Based on the findings of a series of speculative experiments, the final prototype considers the ubiquitous everyday sound as a material for interaction design. The intention behind the final prototype is to help people become more aware of the prevalence of everyday sound and rethink what it might offer to us in designed interactions. 

Sound can be thought of as an intangible material. Although it is invisible, it can be captured with the help of electronic sensors and sensory organs, then it can be modified and utilized as any other physical material. It is everywhere and permeable, whether you actively listen to it or not, it is there. Sound conveys certain information, but only through listening do people decode the information and understand the context it provides. The ability to decode and process sound is gained and polished as time passes; the more you have heard and experienced, the more you can learn from sounds. 

Interestingly, as media technology develops, people seldom question the source of the sound they hear from the media. They might take it for granted, even if the sound is a fabricated sound effect. They decode the information from the sound in the way that is expected by the sound creator. People are less likely to ask the question “Why does it sound like that?” and “What information can I get from that sound?” Gradually, people tend to accept a “standardized” sound system in everyday life through all kinds of interfaces and regard other sounds as outliers. People certainly benefit from the “standardized” sound system, either because of its efficiency or sweet-sounding. With such a system, less attention will be paid to the everyday sound and less familiarity/intimacy people will relate themselves to the object (the sound source). 

If we can become more aware of our relationships to sound/everyday sound, we, as a consequence, might be able to become more aware of the environment in which we dwell and realize the influence that we may pose on the people with whom we share the same space through the sound of our body movements. 

The final project presented in the later chapters explores how new forms of intelligence might creatively respond to everyday sound/interactions. In this, I am particularly interested in non-verbal communication and expression as a design material for interaction design. A prototype illustrating this opportunity is presented: a TinyML-enabled helper that augments the Google Home and enables it to recognize and respond to everyday phenomena such as a sneeze, a door opening, or rapping on a table. The helper offers a creative toolkit for designers and end-users to quickly explore, understand, and imagine the affordances of everyday sounds..