So how would one go about recording a visual and auditory neural experience and converting it into playback video? One possible solution would be to “listen in” on the chatter between neurons and glia in the representational parts of the brain, namely the visual and auditory cortices, and try to make sense of what the cellular conversation is about. Think of it as watching a stage production with many actors talking in a cryptic language and trying to figure out the premise of the play. This is where the field of brain decoding comes in. Given a spatial and temporal pattern of brain activity (an encoded message) while a subject is performing a certain task, how can one meaningfully decode the information contained in that pattern of activity? As one might guess, the current version of brain decoding technology involves the techniques of fMRI and computational-statistical modeling. Using these techniques, one may literally “read” the mind.
Reported in a 2011 paper, Jack Gallant’s team at UC Berkeley pioneered a way to decode brain activity while a subject was watching a movie. First, the researchers started off performing fMRI on subjects as they were watching short movie clips. Next, the researchers tried to come up with an algorithm that could reconstruct an image given the BOLD signal associated with it. The better the algorithm, the closer the reconstructed image would resemble the actual image from the movie clip. To match the slower speed of fMRI data acquisition with the faster speed of seamless visual experience, the researchers incorporated mathematical filters into their algorithm. Finally, to test the accuracy of this decoding algorithm, BOLD signals from unknown test movie clips were fed into the decoding algorithm. Using a non-overlapping, random palette of YouTube videos, the model reported its predicted video reconstructions. Think of it as trying to reproduce a painting with a different set of watercolors than what the original painting was made from.
Since the model was constructed using fMRI data from a small region of the visual cortex, things like faces are well reconstructed, but certain other categories like abstract designs are not.
Address: 5636 Lemon Ave.
Dallas TX 75209
Phone: +1 214 5203694