A camera that sees vibrations can record the sounds of several instruments separately

Although the sensitivity and performance of microphones have improved dramatically since Alexander Graham Bell first patented them, they still have one big drawback that researchers of Carnegie Mellon University maybe eventually overcome by using a pair of ordinary video cameras.

Put a microphone in a room with a group of musicians, and while you’ll capture every last note and nuance of their individual performances, you’ll end up with a single recording with everything mixed together. But to make that performance even better, you ideally want each instrument and musician captured separately, so that each performance can be remixed by a sound engineer with an expert ear.

Software tools have been developed to extract individual sounds from an audio recording, but the results just aren’t as good as what you would get from capturing a sound source with a microphone. This is why mixing desks are often so gigantic and elaborate: countless microphones with limited mic patterns must be configured to properly capture every component of a musical performance, from vocals to instruments, which is a lot of equipment to do things right.

There’s really no way to redesign microphones to differentiate between captured sound vibrations traveling through the air, which is why Carnegie Mellon University researchers School of Computing Robotics Institute instead turned to video cameras. Play the strings of a guitar, and not only will it produce sound waves vibrating through the air, but it will also vibrate the guitar itself. With the right equipment, these vibrations can be viewed and analyzed to recreate the sounds produced, even if no sound is recorded.

Optical microphones, as these camera systems are called, are not a new idea, but what CMU researchers proposed and shared in a recently published paper, “Dual Shutter Optical Vibration Detection,’ is a way to make them work using low-end, more affordable camera gear.

The new system shines a bright laser light source onto a vibrating surface, such as the body of a guitar, and captures the movements of the resulting light-flecked pattern. Since the range of human hearing can detect sounds oscillating as quickly as 20,000 times per second, optical microphones have typically relied on expensive high-speed cameras to capture physical vibrations oscillating just as rapidly. But the new CMU system uses cameras operating at just 63 frames per second, which would apparently miss the high-speed movements of a vibration occurring 20,000 times per second.

The clever breakthrough here is the simultaneous use of two different types of cameras: one with a global shutter that captures entire frames of video, resulting in distinct speckled patterns, and another with a rolling shutter that captures images line by line. from top to bottom of the sensor, resulting in distorted speckled patterns that actually contain more information about how they move through time.

Using a custom algorithm, images captured from each camera can be compared to each other to more accurately determine the movements of vibrating speckled laser patterns up to 63,000 times per second – or as fast as a camera. expensive high speed might do it.

The approach allows audio to be individually extracted from multiple sources into a single video, such as multiple musicians each playing their own guitar, or even multiple speakers all playing different music.

The extracted audio is not as clear or high fidelity as what a traditional microphone can capture, but the optical microphone could provide mixing engineers with easy monitoring of individual instruments during a live performance, and over time , there is no doubt that the quality of the extracted audio will continue to be improved. The system has other interesting applications apart from music. A video camera monitoring all the machinery in a factory, or pointed at the engine of a running car, could determine when individual parts or components are making an abnormal sound, indicating that maintenance may be needed before a problem actually becomes a problem.