Audio descriptors, also referred to as sound descriptors, are numerical representations of specific characteristics of a sound signal. These descriptors have been systematically formalized with a shared terminology since the late 1990s, driven by developments such as the MPEG-7 standard and significant advancements in the field of music information retrieval.
Audio features play a crucial role in music analysis by capturing diverse characteristics of sound, such as timbre, rhythm, pitch, and dynamics, enabling their application across various domains. In music information retrieval (MIR), these features are used to classify genres, identify songs, or recommend music based on similarity. In musicology, they provide quantitative insights into the structural and expressive qualities of musical works, aiding scholars in understanding compositional techniques and stylistic evolution. Furthermore, sound descriptors can be employed in big (audio) data analysis. Beyond these applications, audio features may also be used as a bridge between auditory and visual domains, facilitating the creation of artistic visual representations that are synchronized with sound, such as in multimedia installations or data-driven music visualizations.
Source code | Binaries
https://github.com/valeriorlandini/theinformer
Plugins | Rack Module | Standalone: GPLv3
Libraries | Max/MSP externals: MIT
The computed sound descriptors can be usedto modulate parameters in any software capable of receiving OSC messages, either natively or via appropriate plugins. For instance, visual shapes can be generated and dynamically altered in response to various features of the audio stream, enabling tight and non-trivial synchronization between sound and visuals.
The descriptors also offer a sophisticated way of modulating effects and synthesizer parameters. For example, you could dynamically adjust a filter’s cutoff frequency in response to the spectral centroid of a concurrent sound, or modulate the noise level in a synthesizer to maintain a consistent level of spectral entropy.
Idea
How sound descriptors can be usedin artistic and scientific practices? How to build an integrated framework that allows the computation and broadcasting of sound descriptors, also in real time, to cover a wide range of use cases using the same base interface?
Roadmap
- Definition of a set of sound descriptors
- Implementation of a framework to compute sound descriptors in different environments
- Digital audio workstations
- VCV Rack
- Max/MSP
- C++
- Python
- JavaScript
- Software and audiovisual works to demonstrate the potential of the framework
Real-time visualization of audio properties Integration into platforms like Jitter, TouchDesigner, or any other OSC-capable software. For artists and developers working with live audiovisual performances, this feature provides immediate visual feedback on sound characteristics.
Dynamic control of plugin parameters Audio descriptors can serve as modulators to adjust parameters in other plugins. For instance, its messages can be used to automate the cutoff frequency of a filter, the thresholds of a multiband compressor, or even spatialization effects based on real-time audio analysis.
Real-time analysis of audio streams Analysis of live audio streams and output data in various formats. This data can be used directly within compatible software for immediate visualization or recorded into text files, CSVs, or other text-based formats for later analysis.
Interactive installations In interactive art installations, The Informer can serve as a bridge between audio inputs and visual outputs, enabling responsive environments where sound influences visuals in real time.
Educational tools Use in educational settings to teach students about audio descriptors and their sonic meaning.
Sound design and post-production Sound designers and post-production engineers can use the plugin to analyze and refine audio elements in film, video games, or music production.
Biofeedback and experimental research Researchers exploring biofeedback or experimental psychology could use this system to map physiological signals (converted into audio streams) into visual or auditory feedback systems.
Upon the implementation of the descriptors computation different pieces of software were built: a plugin for DAWs and a standalone app (developed with JUCE), a module for VCV Rack and a set of Max/MSP externals.
Moreover, a Max for Live receiver device, that grabs the information sent through OSC by the plugin or by the Rack module, allows for an instant integration of the tool within Live, with each descriptor that can be used to modulate any parameter in the DAW.
