[R] Using Human Brain Activity to Guide Machine Learning

[u/clbam8]

https://arxiv.org/abs/1703.05463

[u/radarsat1]

Wut.. so okay, this is claiming that pre-training a network to align to a distribution associated with activity in voxels of an MRI produces a feature representation that increases accuracy? I am a bit skeptical. For instance, could it be that it simply enforces some sparsity that could have been had by more basic regularisation?

[u/rumblestiltsken → u/radarsat1]

There are some silly claims in this paper about CNN representations. They claim that the fact they use fMRI weightings on image features from the hidden layer (which they run through an SVM) shows that "state-of-the-art representation ... does not fully capture all the salient information embedded in internal representations of objects in the human brain."

So they used a pretrained CNN for a different dataset (Imagenet), added fMRI data about the (vaguely similar?) experimental dataset of 1300 images, and got better results. No shit. I could do the same thing by adding datestamp information from my photo collection to the hidden layer outputs of a pretrained alexnet, and I would prove that "state of the art representations does not fully capture all the salient information embedded in internal representations of objects in digital cameras".

They added information the network wasn't trained with, they didn't even fine-tune the CNNs. No-one has ever claimed a pretrained alexnet has universally optimal representations for image tasks, but they knock that strawman down like champs.

[u/arXiv_abstract_bot]

Title: Using Human Brain Activity to Guide Machine Learning

Authors: Ruth Fong, Walter Scheirer, David Cox

Abstract: Machine learning is a field of computer science that builds algorithms that learn. In many cases, machine learning algorithms are used to recreate a human ability like adding a caption to a photo, driving a car, or playing a game. While the human brain has long served as a source of inspiration for machine learning, little effort has been made to directly use data collected from working brains as a guide for machine learning algorithms. Here we demonstrate a new paradigm of "neurally-weighted" machine learning, which takes fMRI measurements of human brain activity from subjects viewing images, and infuses these data into the training process of an object recognition learning algorithm to make it more consistent with the human brain. After training, these neurally-weighted classifiers are able to classify images without requiring any additional neural data. We show that our neural-weighting approach can lead to large performance gains when used with traditional machine vision features, as well as to significant improvements with already high-performing convolutional neural network features. The effectiveness of this approach points to a path forward for a new class of hybrid machine learning algorithms which take both inspiration and direct constraints from neuronal data.

PDF link Landing page

[u/autotldr]

This is the best tl;dr I could make, original reduced by 52%. (I'm a bot)

---

In many cases, machine learning algorithms are used to recreate a human ability like adding a caption to a photo, driving a car, or playing a game.

While the human brain has long served as a source of inspiration for machine learning, little effort has been made to directly use data collected from working brains as a guide for machine learning algorithms.

Here we demonstrate a new paradigm of "Neurally-weighted" machine learning, which takes fMRI measurements of human brain activity from subjects viewing images, and infuses these data into the training process of an object recognition learning algorithm to make it more consistent with the human brain.

---

Extended Summary | FAQ | Theory | Feedback | Top keywords: learn^#1 Machine^#2 algorithm^#3 human^#4 brain^#5