Before we get ahead of ourselves, I wanted to write this post outlining a vague project description and motivation to serve as a starting point for people who want to follow along with this blog.
Information Processing in the Brain
Similarly to how a robot has to somehow store the output it receives from its sensors, the brain needs to encode information from the sense organs in order to process it. This is called neural encoding.
Nature.com defines it as:
... the study of how neurons represent information with electrical activity (action potentials) at the level of individual cells or in networks of neurons.
The flip side to neural encoding is neural decoding. While both aim to find relationships between sensory stimuli and neural signals, neural decoding focuses on reconstructing sensory stimuli from neuroimaging.
Measuring Electrical Activity
Intuitively, it makes sense that improvements in neuroimaging techniques should result in more accurate decoding. While fMRI imaging can be safely used on live humans, newer techniques like calcium imaging achieve far greater resolution, both temporally and spatially. With the help of viruses or gene manipulation, calcium indicators can be distributed in the brain. Neurons with calcium indicators will then emit fluorescent light depending on their calcium status when they absorb photons from a special microscope. Calcium imaging works by capturing these light emissions. Electrical activity can then be inferred from the recorded videos.
Calcium Imaging
Source
fMRI
Source
Decoding Biological Neural Networks with Artificial Neural Networks
Machine learning is a set of methods for finding information that is somehow encoded in data. An example would be determining whether an image is of a cat or a dog: it is not obvious how that information is encoded in the image file, but most humans would agree, that the information is present. Decoding visual stimuli from neural imaging data is another example. The challenge is finding the right method. Artificial neural networks seem very promising in this respect.
Somehow the visual stimuli is present in the fMRI data, but it is not obvious how.
Source
Artificial neural networks (ANNs) are algorithms inspired by biological neural networks. Through a process called training an ANN is fitted to the data; it adapts to get better and better at decoding the information by comparing its predictions with labels defining the information that should have been decoded. ANNs work by simulating a networks of nodes arranged in layers. The first layer represents the data with the encoded information, the last layer represents the information to be decoded and all layer in between are called hidden layers. Each layer is connected to the preceding and succeeding layer through certain parameters which are adjusted during training. ANNs with multiple hidden layers are said to be deep. Deep ANNs are part of deep learning, a family of machine learning methods.
Depending on the demand, I might make a post for beginners on how to start implementing deep learning models.
The Dream
Through python libraries like Tensorflow, deep learning is becoming more accessible and easier to implement. At the same time, modern techniques like calcium imaging coupled with two-photon microscopy are achieving incredible resolution. These developments make me hopeful, that new insights in brain decoding are right around the corner.
Next Steps
As a starting point I implemented an ANN for decoding fMRI data. Eventually, once the calcium data is ready, I want to shift my attention to trying to decode that. Until then I hope to gain some more insight from working with the fMRI data. I want to try different deep learning models and dimensionality reduction tools for example. Depending on the results it might even be possible to increase the resolution of the reconstructions. At the same time, I need to learn a lot more about deep learning and data science in general. In my next posts I plan to give some updates on the model for fMRI data as well as summarize the studies and textbooks I'll be reading. In the near future, I plan on making a post with some learning resources.
This is the next deep learning model I want to try out. (See yesterdays post)
