Scientists decode imagined melodies from human brain waves

Researchers have successfully decoded the basic structure of imagined melodies from human brain waves by focusing on how notes relate to one another rather than their exact pitch. Published in the journal eNeuro, the small study demonstrates that computer models can reconstruct the general rising and falling contours of a silent song directly from neural recordings. These results show that the brain’s internal auditory experiences can be accessed and translated into recognizable musical sequences.

When we recognize a familiar song, we do not usually rely on the exact acoustic frequencies of the notes. We rely on relative pitch, which is the interval or distance between successive tones. This is why a song like “Twinkle, Twinkle, Little Star” sounds identical whether it is played on a high-pitched flute or a low-pitched tuba. The human brain naturally encodes these relational patterns, allowing people to carry tunes across different keys and instruments.

Imagining music engages many of the same neural networks as actually listening to it. When a person hears a song in their head, their brain generates an internal auditory experience. Extracting this hidden musical content from brain activity is highly difficult. Past attempts have mostly detected the presence of musical thought or basic rhythmic beats.

Predicting the actual melody a person is imagining requires capturing rapid, fine-grained changes in brain activity. Absolute pitch is tied to specific sound wave frequencies. Because most human brains process melodies relatively, attempting to decode exact absolute pitches often fails.

To address this challenge, researchers Jii Kwon and Chun Kee Chung from Seoul National University led a team to investigate whether neural decoding models could identify imagined melodies using a relative pitch framework. They wanted to see if they could predict individual imagined notes by their scale degree, like “Do” or “Re,” regardless of the starting key.

The researchers conducted a small study involving 10 patients with medically intractable epilepsy. These patients already had arrays of electrodes surgically implanted directly on the surface of their brains for clinical evaluation. This recording technique, called electrocorticography, provides a highly detailed look at electrical activity across the cerebral cortex. The direct contact with the brain tissue allows sensors to pick up high-frequency signals that external scalp monitors often miss.

During the experiment, the patients listened to short segments of familiar children’s songs, such as “Yankee Doodle.” The songs were played in three different musical keys. The researchers instructed the patients to listen to the first two measures of a song and then silently imagine the next two measures. To ensure they were actually following the tune, the patients hummed the melody aloud after the silent imagination period ended.

As the patients imagined the music, the scientists recorded their brain activity. They focused on specific electrical patterns known as high-gamma activity. These rapid brain waves oscillate between 60 and 150 times per second and are a reliable indicator of active sensory processing. The team isolated the signals coming from regions like the superior temporal gyrus, an area on the side of the brain heavily involved in processing sounds.

The researchers took this neural data and trained a machine learning program to recognize patterns associated with specific relative pitches. They broke the imagination period into half-second segments, matching each segment to a specific note in the song. The model learned to categorize the brain waves into six relative pitch classes. These classes correspond to the solfège syllables Do, Re, Mi, Fa, Sol, and La.

The research team then tested the computer models on a fresh set of brain recordings that the models had not seen before. The computer tried to guess which relative note the patient was imagining during each half-second window. The accuracy for these single-note predictions was modest, sitting around 26 percent on average. While this number might seem low, a random guess would only be correct about 16 percent of the time, meaning the models were successfully extracting real musical information.

The team also tried to decode the exact acoustic frequencies, or absolute pitches, from the same brain data. In this test, the models failed to perform better than a random guess. The results were not statistically significant for absolute pitch, supporting the view that the brain primarily represents the imagined melody through relative pitch relationships.

Although predicting individual notes was difficult, the researchers found success when they linked the individual predictions together over time. By arranging the decoded relative pitches in sequence, they reconstructed the overall melodic contour of the imagined songs. These reconstructed sequences generally matched the ascending and descending patterns of the original melodies. Even when a specific note prediction was slightly off, the overall shape of the song remained intact.

By looking closer at the data, the researchers identified which brain areas provided the most useful information for the computer models. Activity in the superior temporal gyrus on both sides of the brain was a major contributor. They also found helpful signals in the left precentral gyrus, a brain region often linked to motor planning. This suggests that internally imagining a melody might involve a mental rehearsal of the physical actions required to sing it.

While these findings present a novel way to access internal musical thoughts, the approach has several practical limitations. The accuracy of predicting individual notes remains relatively low. Reconstructing a perfectly accurate melody note-for-note is not yet possible with current methods. The reconstructed melodies rely on the general shape of the song rather than flawless individual predictions.

The study also relied exclusively on highly familiar children’s songs. Familiar melodies are deeply ingrained in memory, which likely helps patients produce stable and consistent internal imagery. Because of this, it is unknown if the models could decode novel or spontaneously composed melodies. Spontaneous imagination might not generate the same robust neural patterns as recalling a well-learned tune.

The need for surgically implanted electrodes restricts this technology to clinical settings. The invasive nature of electrocorticography means this specific method cannot be adapted into a wearable device for the general public. Future studies will likely explore whether non-invasive techniques can capture similar relative pitch information. Expanding the computer models to decode a wider range of notes across multiple octaves will also be a priority for subsequent research.

The study, “Reconstruction of Imagined Melody with Relative Pitch Decoding in Electrocorticography,” was authored by Jii Kwon, Youmin Shin, June sic Kim, Eunju Jeong, Sung-Phil Kim, Eun Jung Lee, and Chun Kee Chung.