What does a sunset look like? Or ascending shades of red? Now you can listen through the Sonification Toolkit, a polished and working prototype born of a year-long initiative fueled by the combined imagination and expertise of two dozen MIT undergraduate students.
The latest big endeavor for MIT’s Digital Humanities Lab (DH Lab) began last spring with an initial idea from Evan Ziporyn, lab faculty member and Kenan Sahin Professor Emeritus of Music. Ziporyn thinks the DH Lab fellowship was a serendipitous transition as he considered how to expand his previous sonification project, which involved turning intricate spider webs into immersive audio installations.
The recently released Sonification Prototype, a work in progress with state-of-the-art capabilities, is a robust exploratory foray into the possibilities of sonification. The lab has many other ambitions for the toolkit: among the most exciting is a web application that will turn almost anything digital – from digital data to drawings – into sound.
Sonification is the process of translating an object or the structure of a set of data into sound, using computers. To take a simple example, think of a set of stairs (like those in the Boston Science Museum) with each step assigned a note on the major scale. Movement along the stairs produces respective notes. But is it really a sonification? Ziporyn makes an important distinction: he seeks to faithfully produce a sound based on the attributes of the object itself, not to impose a sound on an object.
He explains: “The staircase is a human object, as is a major scale. The staircase is not literally analogous to a major scale – it’s just that this major scale sounds good to many people. The staircase itself is not really sonicated, only values are assigned to it which give a pleasant and digestible result. “What we were trying to do by using the pure numerical relationships between [material and sound] is to move away from that…to ensure that you get a result that actually reflects the structure of the object rather than a humanized, palatable version of it.
The toolkit includes five sonification paths that users can experiment with. Teams of student-researchers have composed tools to sonify temporal data, polygons, colors, gestures and text shapes – creating accessible software to analyze the digital object and allow untrained users to listen and to play.
The imaginative scope of the project was a major draw for Jessica Boye-Doe, a computer science and cognition student. “I had already thought about the idea of turning digital elements into sounds,” she says, “but I was unaware of the extent of sonification research. I saw this UROP as a way to explore this application of technology in music.
The life cycle of the sonification project took place in the DH lab during Ziporyn’s fellowship. Forty students from the Undergraduate Research Opportunity Program (UROP), along with the lab’s teaching staff, gathered for the fall semester, following the competitive selection process for faculty fellows in the spring and initial stages of the project over the summer.
Using the new sonification toolkit, senior Moises Trejo converted the annual sunrise and sunset times into music in one place. The lowest tone corresponds to sunrise time and the highest tone to sunset time. Sunrise and sunset gradually move away from each other as the days get longer, then move closer together as the days get shorter.
Peihua Huang, a second-year computer science and engineering student, joined the project early on, having learned about the DH Lab at the lab’s first-year orientation event. She helped build the initial infrastructure for the project over the summer and stayed on for the fall semester to build and refine the “Gestures to Sound” app, an interface that allows users to gesture with their slider which are then transmuted into music.
“The lab’s intersection with the humanities meant that, while developing, we thought about the project through the prism of the discipline of the professors we work with, as well as our target audience,” says Huang. “As an IT specialist, I really appreciate these opportunities to practice my skills, work in small teams and collaborate to create a global project.”
Ryaan Ahmed, associate director and senior research engineer at DH Lab, connects all the many moving pieces of the lab, from brainstorming with professors to organizing dozens of UROP students. Projects like the Sonification Toolkit reveal the rich and unique range of DH Lab programming. During the first five years of the lab, he tackled emerging technologies for language learners, improving visual archive analyses, simulations around topics affecting democratic development in Africa, and computational analysis of language. gendered in novels, among other projects. The level of creativity and ability demanded by these projects speaks to the multidisciplinary strengths—and endless curiosity—of MIT undergraduates.
“There just aren’t many people in the world who have this kind of really deep cross-disciplinary expertise,” says Ahmed. “The lab students have the level of engineering excellence you find at MIT, and they also have a deep seriousness and investment in the humanistic side of things. I hope we’re showing our students here that you don’t have to choose between these areas: you can actually get into them.
The mix of programming and arts is what initially attracted first-year computer science and engineering major Grace Jau to the DH Lab. Jau worked as a research student on the sonification toolkit last fall.
“As someone who loves making art and music,” says Jau, “I appreciate that working at the DH Lab has given me the opportunity to gain experience in my technical field while connecting me to my interests in the humanities.”
A love of computers and music also drew sophomore Emeka Echezona to the lab. A graduate in electrical engineering and computer science, Echezona also plays trumpet in the MIT Festival Jazz Ensemble. Reflecting on the Sonification Toolkit, he says, “A lot of the work we put into designing and coding the instruments in the Toolkit was only possible because of knowledge from different fields. Even outside of the toolkit itself, we learned musical concepts like dynamics, pitch, and timbre from the perspective of areas like physics.
The imaginative connection to the arts and humanities through technology explored in the Sonification Toolkit has a number of other applications. UROPs thought about how to sonify paintings, maps and photography. What could the Mona Lisa look like? With recent MIT sonification projects reaching the level of protein and particle energy, the limits of this art form seem to be only the limits of technology and imagination.
This breed of technology shows another side of MIT’s innovation: the sonification toolkit is about what can be imagined and accomplished with the perspective of the arts and humanities.
“We will have a generation of technologists better equipped to think about these complex subtle and useful areas that the humanities are good at approaching,” says Ahmed, “thinking about how the technologies they work on affect people and bring the nuances arts, social sciences and humanities to their technical work.