I never considered the use of sound for monitoring processes and events in real time a matter of interest. Efficient, yes. Effective, yes, too. The Geiger counter, which emits a very simple sound to signify the level of radiation in the immediate vicinity of the user (the faster the sound, the higher the radiation’s level); the sonar, which sends back from an underwater obstacle a sound to signify its presence - again, the faster the sooner, the closer the object; they are definitely among the most successful and enduring cases of data sonification, as mentioned in the very first chapter of The Data Sonification Handbook (Hermann et al., 2011). They are part of our collective memory, so much so that they did not present, to me, any challenge, be it from the design perspective or the theoretical one. The real challenge in sonification was, for me, to find the Graal of using sound to represent large “offline” (not real time) data sets for scientific exploration or public communication, as an alternative to data visualization, and prove the world it can work!
Fast forward one year, and I find myself mainly working on the use of sound to communicate data coming from monitoring processes - in particular, cybersecurity and medical processes - in real time. What changed?
First of all, process monitoring is more and more performed leveraging Artificial Intelligence algorithms trained to detect anomalous behaviour in the flow of data that come, at every instant, from - say - an Internet network. Process monitoring has thus become an extremely innovative frontier of the development and application of AI to the everyday tasks of the operation centres all over the world. What these algorithms allow us to do is not only have a better understanding of historical data. More importantly, based on their model of the ideal status of the system, they learn to detect any anomalous behaviour, making it possible for the human operators to understand in real time wether - for example - a cyberattack is in course. And even more importantly, they allow human beings to make predictions on the evolution of the attack or the anomalies.
This consideration led me to think of the difference - sonically and conceptually - between an alarm and an alert. An alarm is something we hear after that the incident occurred (Hildebrandt et al., 2016), i.e., hearing the fire alarms signify that there already is a fire in the building, and we should go - nothing we can do there, better leave it to the professionals. An alert should be something that gives us information on how seriously is the possibility of a problem being happening right now, right there, in order for us to act while we still can. I.e., we are stil in the position of doing something, avoiding the missile coming towards us or stepping backwards from where the level of radiations seem to be increasing. Geiger counters and sonars are alerts, not alarms.
This simple epiphanic consideration led me to seeing how interesting and stimulating (sound) designing an alert can be.
Here just in a random order, some reasons in support of my claim:
AI as described here, is used today to monitor digital or physical/digital processes such as network operations in light of cybersecurity threats, industry production, medical examinations. In this environments, human beings are exposed to continuous information overload.
Sound can relieve some information from the visual channel providing a peripheral (Bakker, 2018) monitoring system that holistically keep the operators updated on what’s going on when they do not to intervene. The behaviour of this sound in case of anomaly can trigger a response in the user and bring the information at the center of attention for further action. This sounded to me like a really good use case for sonification - being the lack of compelling use cases one of the main problems of data sonification.
Geiger counter represents a single variable of a specific data set (the level of radiation mapped to the frequency/pace of the sound played). But what about multi-variate, complex data sets (which are the majority in AI) where we have multiple flows of data organized hierarchically and more than one variable per data, to represent?
Well, sound composition is multi-variate in its own nature: a series of acoustic parameters are organized in a time based sequence where every unit (the single sound) handles multiple characteristics: pitch, amplitude, rhythm, timbre. And the human ear can quite well distinguish multiple single unit with their individual characteristics when played at the same time. Think of music, where we distinguish a voice from a guitar or a piano; or a natural soundscape where we distinguish birds from trees or water. It could be a goldmine for representing data when they flow in time!
And then, and more specifically on the design challenges:
how do you design a viable sonification that has to be listened to by a human being 8 hours a day, every day? Is this recommendable at all, or could we use sound to send out a, hourly “summary” of activity?
how should sound be designed in order to be at the same time audible, but not aggressive or annoying? Or disturbing other people who are not concerned with the monitoring? and so on and so forth.
again, and something I noticed only recently thanks to a user’s feedback: what if the operator likes to listen to her/his own music during the work day? can the two things co-exist? what if they leave for coffee break, or lunch break, or a meeting…or?
should sounds that have to represent something potentially dangerous sound…scary? or alarming?
what is an alarming sound after all? If we are walking in a silent forest, even the faintest sound of a little animal moving in a bush is alarming. Can we leverage this human experience to design compelling alert systems?
and, if we use music (“tuned” sounds), are they cross-cultural or do we give for granted stereotypes that are not shared by all human beings?
All these are specific design problems that make for potentially very interesting real world use case, that could be tested as any other design product - with a prototype, and some research through design experiments. Who would have thought that the good old Geiger counter was so much more than a vintage piece of scientific equipment?
Image Credit Jörg Schubert