From stock trends to changes in Earth’s temperature, data that vary over time are usually represented in graphs and charts. How boring! What if, instead, one could listen to the sudden drop of a stock price or the steady increase of global temperatures? Enter sonification, the process of transforming flat data into mellow soundwaves.
Mark Ballora, an expert on music technology at Pennsylvania State University in State College, uses sonification to create symphonies out of scientific data. Raised in the 1960s just outside San Francisco, California, by an architect and a pianist, Ballora lived and breathed music, playing piano, listening to Beatles records, and attending Grateful Dead gigs. For the past 20 years, he has collaborated with dozens of scientists to turn all kinds of data into music, from the energy emitted by a neutron star to the body temperature cycle of arctic squirrels. In June, Ballora received two $50,000 grants from the National Academies Keck Futures Initiative to help marine biologists translate data from the deep ocean into sound. He chatted withScience to discuss how—and why—he turns data into music. This interview has been edited for clarity and length.
Q: How did you start transforming data into sound?
A: It was very serendipitous. During the first year of my Ph.D., a physiologist asked me if I had any interest in making sound out of heart rate variability data. I didn't know what he was talking about but it sounded great. [Here’s the sound of a healthy heart. Here’s the sound of a heart that doesn't pump blood as well as it should.]
Q: How do you decide the type of sound to use for a specific data set?
A: That's the artistic part of my job. The first step is to get familiar with the data and understand what it means, which is actually very interesting: I get to meet all these scientists and learn what they do. The second step is to figure out what sound fits the data. When I was working on solar wind[charged particles shed from the sun that create aurora borealis when they hit Earth’s atmosphere], I created a shifting, shimmery sound. [Listen below to solar wind.] For tropical storms, I needed a swirling noise that would sound like a tornado.
Q: Which project was the most fun to work on?
A: It was great working with [former Grateful Dead percussionist] Mickey Hart on Rhythms of the Universe, the film he did with George Smoot, who is a Nobel Prize–winning cosmologist. I got to turn into music all kinds of data for them: solar wind, sunquakes, Earth’s electromagnetic resonances [electromagnetic waves that form between the Earth’s surface and its upper atmosphere]. There was also a representation of the motion of the planets in the solar system, the signal from a rotating neutron star, gravitational waves stretching the space-time. It was just a wonderful variety of things to work with, a very colorful data set. [Listen to the sound of a rotating neutron star.]
Q: What is the most interesting part of your job?
A: When I play sonifications for people, they might say, "Wow, this gives me a completely different idea of these data sets.” It's like when you add soundtrack to a movie: It takes you into that world and it makes it a much more visceral phenomenon. It shouldn't be an unusual thing for us to listen to data. I think we should try to put educational programs together so that young people grow up considering science to be something that you listen to as well as something you look at. As humans, we all respond to music: If we can leverage that with science, there's a real chance of giving students a much more intuitive understanding of the material than they would get from a visual presentation alone.
Music technology expert Mark Ballora translates numbers into sounds.
Patrick Mansell/Penn State
Q: What will you use the Keck Futures grants for?
A: Next year I'll be working with Karen Wishner, a marine biologist at the University of Rhode Island [in Narragansett], who has some data about oxygen and temperature in environmentally sensitive areas of the ocean. One group of her students will get the visuals only, and another group will get both the audio and the visuals. Then science education experts will assess the differences in what the students are able to get out of it. Under water, there's so much that we don't know, and the quality of our oceans is such a pressing issue for the health of the planet. If the public has a good understanding of this issue, they will support conscientious policymaking.
Q: Do scientists use sonification to get an insight into their data?
A: It's not a general practice yet, and that's what I'm working on. I talk a lot about turning data into music for outreach purposes, but there's more to that. For example, Wanda Diaz Merced is an astrophysicist at the South African Astronomical Observatory in Cape Town who, after losing her sight, worked with programmers to come up with a way to listen to her data. She is now able to work at the same level that she could work when she was sighted, and she actually discovered the presence of electromagnetic waves produced by the exchange of energy between particles during high-energy stellar explosions, something that nobody had detected by looking only at graphs. It turns out that a lot of her sighted colleagues now use [sonification] software because there are often patterns that they can more readily hear than they can see.
Q: Do you give concerts featuring your sonifications?
A: Not yet, but someday I'll probably do that. First I’d like to show that students learn well with sonification and that scientists work well with it. Once I make this case, I’ll be able to think more like a composer and do concerts. I’ve got so much music out of the data I’ve worked with. So maybe in my 60s and 70s I’ll work on that.