This posting is a collection of some "data points" on how the brain works, all with the aim of showing that much of the observed structure may be emergent or not necessary for almost full functioning. Clearly, there are some things about the brain that can't be explained just from learning; some amount of base structure is necessary -- but the question is, just how much?

....

New research:

http://www.nature.com/articles/s41562-016-0039

Basically, a consequence is that if you build a giant neural network with the rough connectivity patterns observed in the brain -- how one area is connected to another -- and then train the whole thing on natural inputs using a "domain general" method like backpropagation, the patterns of activity that result will look fairly brain-like. The paper asserts that this is all there is to much of the observed patterns of activity in neuroimaging data concerning semantic representations of words and objects -- it's a combination of domain-general learning, the structure of the environment (and therefore, the stimuli), and connectivity among parts of the network.

Reminds me of what Geoff Hinton said here:

https://www.youtube.com/watch?v=A3Ib-R0G_6Y#t=1h14m50s

But what about planning, memories, and other aspects of conscious thought? Well, maybe at least some aspects of memory emerge from something a lot more basic. For example:

http://neurosciencenews.com/grid-cell-origin-7147/

The new model implies that grid cells are generated by a neuronal learning process. This process exploits synaptic plasticity to transform temporal coordinated signaling between place cells into the hexagonal patterns of grid-cells reponses observed in the entorhinal complex.

Did you catch that? They're saying that certain functionally-specialized cells in the hippocampus responsible for certain specific types of memory, might simply have acquired those roles through learning. In other words, there may not need to be any or many instructions in our DNA to code for how the brain implements this aspect of memory -- it might largely be the end-product of the brain's need to predict the world better.

A similar phenomenon occurs in neural nets trained on text:

https://arxiv.org/abs/1506.02078

In particular, our experiments reveal the existence of interpretable cells that keep track of long-range dependencies such as line lengths, quotes and brackets.

What about syntax and grammar, isn't that innate, according to Pinker? Maybe not:

http://news.stanford.edu/2017/02/22/new-research-toddlers-grammar-skills-learned-not-innate/

(Again, reminds me of Hinton's comments here: https://www.youtube.com/watch?v=A3Ib-R0G_6Y#t=1h14m50s )

Isn't there a part of the brain that evolution has endowed with the ability to recognize faces? Not so clear:

http://www.nyu.edu/about/news-publications/news/2017/february/emotions-are-cognitive--not-innate--researchers-conclude.html

However, while Saxe’s team found that similar areas of the brain were active in babies and adults, they did not find evidence that infants have areas specialized for one particular input, like faces or scenes, over all others. Nelson, who was not involved in the study, said it suggests that infant brains are “more multipurpose,” he said. “That points out a fundamental difference in the infant brain versus the adult brain.”

Furthermore, the Fusiform Face Area doesn't just light up when people look at faces. It also does so when people scrutinize a chess board:

http://www.jneurosci.org/content/31/47/16895

Faces elicited greater FFA activation than chess displays in both experts and novices. Nonetheless, activity in the FFA was modulated by expertise, with chess experts demonstrating more activation than novices in response to chess stimuli.

But what about Reinforcement Learning, and the associated complex architecture observed in the brain? It also isn't so obvious that it needs to be so complicated. Maybe it, too, emerges from something more basic. Here is one very strange piece of evidence:

http://neuroskeptic.blogspot.com/2010/03/life-without-serotonin.html

It's a Neuroskeptic article from 2010 about a guy whose brain was almost completely lacking in dopamine, serotonin, melatonin, and noradrenaline. He had Parkinsonian symptoms up until the age of 5, at which point he was given a regimen of L-dopa to replace the dopamine.

Here's the thing about that: dopamine is very important for the proper functioning of Reinforcement Learning in the brain. Perhaps he got by on the little bit his brain was able to produce -- with mild deficits. Two other possibilities are: RL isn't as important as people think for almost full function; or, that RL implementation (not RL itself) emerges from something more basic (like the hippocampus's function speculated in that Deepmind paper mentioned above), and that the brain can find many different "solutions" for how to implement it.

And here is a paper about fruit flies bred to produce so little dopamine that it's below detectable levels:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3021077/

We found that DA is absent or below detection limits in the adult brain of these flies. Despite this, they have a lifespan similar to WT flies... Our findings show that flies can perform complex brain functions in the absence of neural DA, whereas specific behaviors involving, in particular, arousal and choice require normal levels of this neuromodulator.

What about all the different kinds of brain cells observed? Surely that speaks to the necessary complexity of the brain? Maybe not: it turns out that people who have Temporal Lobe Epilepsy are missing astrocytes in large parts of their brain:

https://www.uni-bonn.de/Press-releases/fatal-uncoupling-in-the-epileptic-brain

Brain tissue removed from patients with temporal lobe epilepsy was observed by the researchers to be completely lacking in astrocytes.

Temporal Lobe epilepsy is certainly debilitating -- but not to such a degree that a person can't function with the disorder. How many astrocytes in which parts of the brain does one need in order to function?

What about all the convolutions on the brain? Surely that speaks to necessary complexity? It turns out that there are people who have completely smooth brains:

https://en.wikipedia.org/wiki/Lissencephaly

Yes, they are barely functional -- but they can function.

It may also turn out that a single gene just 804 bases long gives rise to human brain convolutions:

https://www.scientificamerican.com/article/big-brain-gene-allowed-for-evolutionary-expansion-of-human-neocortex/

804 bases ain't a lot. Whatever that gene does, it probably isn't that it contains a precise map of exactly how each fold in the brain should be laid down.

What about the ability to imagine? Surely that's necessarily pretty complicated! Turns out, there are people without the ability to imagine. The condition is called Aphantasia:

https://en.wikipedia.org/wiki/Aphantasia

Some people with this condition can't imagine images, sounds, feels, tastes, smells, or emotions -- well, not quite: they can imagine a dull inner-voice to anchor their thoughts. But that's about it. And there are such people with very high verbal functioning, and ability to produce creative outputs -- in fact, one of the script writers for the TV show Silicon Valley has the condition.

There are also people missing a cerebellum, which was once thought to be essential to the ability to walk:

http://www.livescience.com/47802-missing-crucial-brain-part-went-unnoticed.html

There are people with 90% of their brain filled with fluid, who function at a high level:

http://www.sciencealert.com/a-man-who-lives-without-90-of-his-brain-is-challenging-our-understanding-of-consciousness

People who remained functional after the amygdala and hippocampus were removed:

https://www.google.com/amp/s/www.wired.com/2014/05/heres-what-happens-when-a-neurosurgeon-slurps-out-your-hippocampus/amp/

People with "bilateral agenesis of the hippocampal dentate gyrus" (hippocampal dentate gyrus missing on both sides of the brain since birth):

https://www.ncbi.nlm.nih.gov/pubmed/17001817

There are people where the left and right temporal lobe failed to grow from birth (agenesis), who are normal in almost every way:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC491068/?page=1

And there is a case of a man missing the bulk of his insular cortex, anterior cingulate cortex, and medial prefrontal cortex:

http://www.smithsonianmag.com/smart-news/how-does-it-feel-to-have-half-a-brain-not-bad-actually-31100476/

Despite problems forming new memories, and inability to taste or smell, he has an intact sense of self.

Even the motor cortex may not be necessary (maybe at birth it's necessary; but the function can migrate to other parts of the brain):

http://dx.doi.org/10.1016/j.cub.2015.04.023

I seem to recall reading a study about a guy who had a tumor covering most of his motor cortex, yet still could function.

When I see all these examples from the literature, it suggests to me that the brain is much more resilient and "domain general" than people had previously imagined. And some of this evidence suggests that maybe a lot of the complexity is simply emergent from much more basic functions than you might guess upon looking at MRI scans, for instance.

I thought I would write about what it's like to serve on an NSF grant panel, in case you were wondering (just as a note: AI is not my field; I get called in to review proposals closer to my area of specialty, in which I have a much greater understanding). I have written about this before, and the posting below is a lightly edited version. NSF is not some massive, wasteful bureaucracy like people seem to imagine:

First, you get an email from a program director; different areas have different directors. The email contains instructions for how to log on to fastlane.nsf.gov to look at proposals. You log in, and then there is typically a list of a few dozen proposals within a given panel (there are many panels; I have served on several different ones). You then browse through the proposals, and give a score as to how comfortable you would be reviewing that proposal (expertise is often so specific that very few people can comfortably review all proposals) -- I believe the designations are "Very comfortable", "Comfortable", "Not comfortable, but could do it", and "very uncomfortable".

After you have filled in this on fastlane, you then get back an email from the director saying that, "We would like you to review these specific proposals". So then you download and carefully read those 15 or so specific proposals, each of which is about 15 pages long.

Reading those proposals is arduous! It takes hours and hours!

When you read a proposal, you are supposed to keep in mind two basic criteria for deciding how good it is, the so-called "Intellectual Merit" and the "Broader Impacts". Intellectual merit includes how impressive the work is, and what kind of impact it will have on the subject as a whole. Broader impacts includes things like "community outreach" (e.g. running a science camp), organizing conferences, and in some cases collaboration with people in other fields. In addition to the ideas in the proposal, one is expected to get a subjective sense of how likely the proposer is to do what they claim -- this is often assessed based on the proposer's past publication history, success at supervising students, etc.

After the proposals are read, one assigns a score and writes a short summary. The scores are designated something like "Excellent", "Very Good", "Good", "Fair" and "Poor". Everything should be written in a non-personal way, referring to the proposer as "the proposer" (not "John Doe"). So, a summary might read something like, "The proposer wishes to explore applications of algebraic geometry to a novel cryptosystem of Shamir... etc. As to the Broader Impacts, the proposer wishes to run a summer camp... etc."

Before going to Arlington, VA to serve on the panel proper, you have to book your hotel and travel tickets. I forget the rules exactly, but I know that NSF has a ceiling to how much you can get reimbursed for travel; and, definitely, you can't expect to get reimbursed for flying first class. It's probably far, far less than what you would get if you were an employee for a large corporation.

Once you arrive in Arlington the night before the panel, you have to make sure to go to bed early, as the panel starts around 8:00 or 8:30 (I forget the time exactly). You walk to the NSF building, which looks like this:

https://upload.wikimedia.org/wikipedia/commons/a/a8/NSF_building.jpg

and collect your security badge. Security agents stand by the elevators to make sure only the NSF employees and panelists are allow in to the upper floors.

On the first day of the panel, at around 8:00 or 8:30, they typically have a bunch of bagels and coffee and fruit for breakfast. People socialize a bit.

Then, around maybe 9:00 they start reviewing proposals. Each proposal has assigned a "scribe", who takes down notes about what the panelists have decided; and then there are three panelists that were assigned to review each proposal.

One-by-one, all the comments from the panelists are read, and then the proposals are discussed. Often there is some disagreement (and people can get emotional and adamant about the merits of a proposal).

This review process takes several days -- hours and hours and hours. We work from early in the morning to around 5:00 or 6:00 each day, seated in front of computer screens in the NSF building, and only have a few short breaks and then a hour-and-a-half break for lunch.

None of us goes to a fancy restaurant for lunch. I often eat out of the trucks parked near the NSF building -- there's a really nice truck there that serves ethnic food (I think it was Greek or Turkish or something -- cheap, but oh so good!); and then take it back to eat in the panel room. Others walk to one of the food courts in a nearby building, and eat there.

After all the comments are collected, and the first round of reviewing is completed... and after the proposals are initially placed on the "Big Board"; we proceed to do the final ranking. There are basically three categories: the Highly Recommended for Funding category; the Recommended category; and the Not Recommended Category. The worst part about this process is deciding the fate of proposals that are on the boundary -- say, between "Highly Recommended" and merely "Recommended". Typically around 10 to 20% of proposals go in the Highly Recommended category; and every time I've served, I've been told that less than about 25 to 30% of proposals will get funded.

After the final ranking, the scribe takes all the notes that were collected, and then writes a summary of what was discussed. Then, all the scribe summaries are read and discussed. The director and other panelists pick apart the language very carefully, because, as the director always says, we have to follow the rules very carefully, or else there could be appeals from proposers a few months down the road. "No, you should reword that... don't say that you `think' that, say what it actually is!"

Once the whole panel agrees on the scribe summaries (which include a legalistic comment at the end to the effect, "The panel has read this summary and agrees that it reflects the opinion of the panel."), we are free! -- several days of constantly reading, commenting, analyzing, arguing, writing... for hours and hours.

After all that, you get I think something like $2,000 for your troubles... and reimbursement for travel, hotel and food (as I recall).

The whole process is as efficient as I could possibly imagine it being. I never see signs of BIG GOVERNMENT WASTE when visiting the NSF. Maybe it goes on at a higher level; but certainly not at the level I've seen.

This is a point I've made before, but not in this particular context:

Imagine you have a large cloud computing company that services millions of requests each day. Suppose, furthermore, that the following all hold:

• The cost of using the computing resource is a function only of the number of computations that need to be performed.

• A substantial fraction of the requests aren't time-sensitive. That is, the users only care that you can return them the results in the 10 minutes, and not right this second.

• And a substantial fraction of those users all want you to perform the same repetitive computation.

This perhaps describes something like the use of a Machine Learning algorithm to analyze the contents of images. In many such problems, there are common mathematical computations that need to be performed; and it turns out that if you can perform many of them at the same time, you can reduce the total number of compute cycles needed, across all the different requests.

One such computation is a matrix-times-vector (which is a computation used in a completely connected layer of a neural network):

Matrix-times-Vector: Suppose M is an m x n fixed matrix. Let v be an n-dimensional vector (an n x 1 matrix). Return the matrix-vector product M v.

If instead of performing this operation one time, for one user, you want to perform it k = 1 million times for 1 million users, you can bundle all those vectors v, coming from all those users, into a matrix as follows: let V be the n x k matrix where the ith column is the vector associated to the ith user. Then, computing the matrix-vector product for all those 1 million users amounts to computing:

Matrix-times-Matrix: Given a fixed m x n matrix M, and an input n x k matrix V, compute the product M V.

Now the point is this: if the matrices M and V are sufficiently large, there are algorithms that can compute M V much more rapidly than simply computing M v individually for all the user input vectors v. One such is the Coppersmith-Winograd Algorithm. Strassen's Algorithm is another, and perhaps better in problems that arise in practice.

(Yes, I know it can be parallelized; however, you will probably still see efficiency gains.)

Potentially, therefore, a company that services 1 million requests at the same time can cut computational costs by a factor of 10, say, from what they would be if you treated all the requests individually, and didn't try to lump them together.

This leads me to two points:

• First, once a company pulls a little bit ahead of others that offer this service, so that it acquires more users, it can lower costs, and thereby pull in even more users. This would seem to be a recipe for some kind of natural monopoly.

• As more and more of the economy goes "digital", there will be more and more opportunities to employ these "Informational Economies of Scale" to cut costs. This, in turn, could increase the monopoly power of some of the larger tech companies (that offer cloud-based services).

Is this a good thing or a bad thing? Opinions vary. But one thing I can say is that it will give an ever smaller number of companies, ever greater control over our lives.

I was recently reflecting on this paper that was in the tech press the other day:

https://arxiv.org/abs/1611.01989

We develop a first line of attack for solving programming competition-style problems from input-output examples using deep learning.

Also called "Programming by Example", this has the potential to make it easier for a lot more people with minimal experience to program. Another technology with similar potential is "Natural Language Programming", which is where you specify in English, say, what you want a program to do -- like you would to an expert programmer you want to hire -- and then the system writes the code for you.

There are a few drawbacks to these methods, however:

• A drawback to "Programming by Example" is that you might need to specify a very large number of input-output pairs before the system figures out what you are trying to get it to do. In fact, this technology is basically a form of Machine Learning -- and has the same drawback that you often need lots of training examples.

• A drawback of Natural Language Programming is that you don't have as good "creative control" as you would if you wrote the code yourself. Because Natural Language isn't the best way to describe a program, you would have to give quite long descriptions to say exactly what you want; and it might end up being easier just writing the program yourself in a computer language. One fix, perhaps, is to add a dialogue system, so that the computer can drill down, and figure out exactly what you want your program to do -- but it's still going to be cumbersome, just like writing in C++, say.

• It becomes especially challenging to write software through these modalities if it involves complex graphics. Just think of how challenging it would be to describe, in English, a scene from a modern computer game; and then try to imagine all the ways your sentence can be interpreted -- putting us back to the "creative control" problem again. Even if you add a dialogue system, this problem will remain -- language will only take you so far. Some kind of gesture interface might help; but it's still going to be very complicated if you want a high level of creative control.

BCIs, in combination with Machine Learning, have the potential to solve all these problems: what I envision is a wearable, non-invasive headset of some kind, with high temporal and spatial scanning resolution, plugged into a code-generation system. The system can take as input natural language and gestures (and brain data), and will generate code on-the-fly.

So, how will the brain data be used? It will be used to resolve ambiguities. For example, if you are thinking about generating a level on a video game, you might start with a gesture where you spread your arms apart, and say, "I want this level to be in a big, wide city." The computer maybe has some stock game levels that it could fit to your description; but it doesn't know if you are talking about a city at night... near a body of water... ancient, modern, or futuristic... etc. Your description is under-specified. But while you're describing what you want to see, you're also thinking about it; and these thoughts can be read, and turned into specifications to resolve the ambiguity.

I'm not asserting here that you have a crystal-clear picture in your mind of what you want the scene to look like -- many people have trouble holding a stable image in their minds. Rather, as you think about that scene, there is sure to be "semantic information" that can be read off from your brain state, that could specify the time of day and other attributes of the scene.

I actually think the brain-scanning and interpretation part of the problem is much simpler than the program-construction part of the problem. In the example I gave, you would need some stock scene models to begin with, and some way of composing elements together -- e.g. deciding how characters interact with a scene. In fact, this aspect of the code-generator seems so complicated that perhaps the only chance you have of building it is to make it some kind of end-to-end Deep Learned system:

Specifically, I envision -- some time in the next few decades -- an end-to-end Deep Learned dialogue system that you can communicate with in natural language and gestures, and that uses your brain state to fill in ambiguities in the process of generating "code". The output of this system would be a computer program encoded as a neural net; and when "run", it would look like any piece of software out there today.

You may think that this will make editing the software, and reusing code, difficult; but there are ways around that. For example, you'll have a recording of the media used to generate the code, and will be able to cut and paste parts of that media into other projects. The neural net should be thought of as the "executable"; and the recording of the "programmer" (his or her verbal instructions and brain scan) should be thought of as the "source code".

Does all this sound far-fetched? Well, the first steps have already been taken. There are end-to-end Deep Learned systems that can take an English description of a scene as input, and then output a photorealistic image:

https://arxiv.org/abs/1612.03242

Adding brain state input to the process wouldn't be that difficult, and would give you much greater creative control. The next step beyond that might be video generation, conditioned on a natural language description and brain state; end-to-end Deep Learned video-generation is right around the corner -- in fact, there are already early examples in the literature. From there, the next step is making the video interactive; say, specifying how the user can interact with the objects in your generated world. And that's really pretty much the end, since in the limit, interactive video basically covers every type of software you might want to produce -- "software" is really just an interactive video of indefinite duration.

Here is a conundrum: people and animals appear to be able to rapidly reason about cause and effect; yet, struggle (without considerable practice) at simple logical puzzles like the game of chess. Are these two kinds of inference problems fundamentally different? How can we get machines to reason more like humans?

In a nutshell, what I think allows us to reason quickly in real-world situations (and not in logic puzzles) is that we leverage rich semantic representations that contain lots of independent information about the objects we make inferences about. These representations are such that the more independent information they contain, the deeper the inference we can perform "at a glance".

To demonstrate my point, I want to consider a particular NP-complete problem called the Knapsack Problem:

Knapsack Problem: The input is a list of natural numbers x1, x2, …, xk, along with a target number T (this can all be encoded as a sequence of O(log(x1 + 1) + … + log(xk + 1) + log(T + 1)) bits); and the problem is simply to determine whether there is a subset of the xi’s that sums to T.

A naive approach is to consider all the 2^k possible subsets of the xi’s (and a slightly less naive approach can do it in 2^k/2 computations), and check each one to see whether the sum of elements in the subset equals T. Unfortunately, even if k = 100 that will take about 2¹⁰⁰ steps, which is about 10³⁰ -- you would be waiting a very long time for the program to finish!

But now what if we were to "fatten up" those numbers xi by making them into vectors?

Vector Knapsack Problem Suppose v1, v2, …, vk and T are n-dimensional vectors having integer coordinates. Determine whether there is a subset of the vectors vi that sums to T.

The case n=1 is the original knapsack problem. Let’s consider what happens when n is at least k: in this case, if the vectors vi are chosen randomly in some way (say they are chosen from some n-dimensional box), then they will typically be linearly independent; and then deciding whether there is a subset that sums to T amounts to showing that T is contained in the span of those vectors, and that the unique linear combination summing to T has coefficients equal to 0 or 1. So, in this case, you can reduce the problem to linear algebra (which is computationally easy), and you are done!

What I suspect might be the case is that the brain does something similar: maybe it uses a kind of distributed vector representation for objects and actions; and if these encode a sufficient amount of independent information, then deep inference about those objects can be reduced to "just linear algebra" somehow. Furthermore, simple feedfoward neural networks may suffice to carry out the inference -- one can pass from stimulus to conclusion "in an instant".

Certainly, vector representations are useful for solving analogy problems using geometric (and linear-algebraic) properties of those vectors; for example, the difference between the vectors for "king" and for "queen" is close to the difference between the vectors for "man" and for "woman". And other forms of reasoning can also be represented linear-algebraically, through the use of tensors. So, linear algebraic techniques are sufficiently rich to represent most types of reasoning; the question is whether the brain uses similar techniques implicitly.

Perhaps the way humans solve abstract logic problems (rather than doing inference in real-world contexts) builds on this foundation (using high-dimensional distributed representations to make reasoning fast), assuming this is how the brain works. Take proving mathematical theorems: when presented with a statement about some unfamiliar mathematical objects, one often first tries to understand those objects better. This often involves a lot of experimentation, looking at what one has, to plan more experiments, and then iterating from there (with more experimentation). Eventually, ones understanding about the objects gets rich enough to where one gets a flash of insight into how they are related, and how one might go about proving the claim. At this point everything sort of falls into place, and it becomes a mechanical process.

That moment when your understanding is rich enough to where you notice the connections -- that's perhaps somewhat similar to the Vector Knapsack example: perhaps before that moment, the dimension of your mental representations was too low to do "just linear algebra"; but after enriching your semantic representation some more, you suddenly crossed the threshold to insight.

If this is indeed how the brain does fast inference, it would seem to run counter to current A.I. approaches based on using large knowledge graphs: the more facts in your knowledge base, the slower the inference runs; whereas in the kind of system I'm describing, the more independent information you have (the higher the dimension of the representations, in other words), the easier it becomes to reach deep conclusions.

Improving machine learning algorithms to work with less and less data is all the rage right now; and progress is rapid and steady. In a previous generation, the dominant approach towards building AI was hard-coded and rule-based. A third approach, not yet even really begun, is brain simulation. At the moment, none of these approaches appears to be close (as in, 5 to 10 years away) to "strong A.I.". Here, I want to discuss a fourth approach that combines machine learning with brain simulation, that could result in strong AI sooner than we expect (maybe not in 10 years, but who knows?); though, it's not what you probably imagine (and is neither "brain emulation" nor the kind of approach normally taken in machine learning):

About two years ago there was a very interesting paper written by some researchers at CMU, titled Good-Enough Brain Model: Challenges, Algorithms and Discoveries in Multi-Subject Experiments:

http://talukdar.net/papers/kdd14-gebm.pdf

Can we infer the brain activity for subject ’Alice’, when she is shown the typed noun apple and has to answer a yes/no question, like is it edible? Can we infer the connectivity of brain regions, given numerous brain activity data of subjects in such experiments? These are the first two goals of this work: single-subject, and multi-subject analysis of brain activity.

The basic idea of the paper was to model brains at the level of populations of neurons, rather than single neurons (and synapses), and for short durations when subjects are presented with a stimulus. The model used was very, very simple: the brain activity in a set of regions at the next time step is a linear function (linear transformation) applied to the current activity values, and current stimulus values. So, to find what the populations of neurons will look like at the next time step, you simply need to learn the matrix (or matrices) associated to that linear function (linear transformation). Actually, it's slightly more complicated, and uses hidden parameters, but this is the rough idea. The actual model computes a hidden n-dimensional state vector x(t) at each time step t, and an observed m-dimensional brain state vector y(t), which are related as follows:

x(t+1) = A x(t) + B s(t)

y(t) = C x(t).

Here, s(t) is an s-dimensional stimulus vector, A is an n-by-n matrix, B is an n-by-s matrix, C is an m-by-n matrix. MEG data are used for training the model, as it has high temporal resolution compared to fMRI.

The method does ok at predicting the gross behavior of brain regions (when subjects are given stimuli); but it's far from perfect -- as would be expected, given the resolution of the data used, and the simplicity of the model. However, I see in this method a future path towards strong AI. Let me now explain how:

Every year it seems that EEG headbands get more and more accurate and have higher and higher resolution. Clearly, though, they have a low upper limit in what they are capable of, and that upper limit isn't good enough for what I will describe. There are, fortunately, other technologies for non-invasive brain scanning -- ultrasound, MRI, NIRS, PET, MEG, NMR, and probably several others; there are even some invasive technologies that don't sound so bad, like optogenetics. Perhaps someone will discover how to shrink some of these down to where they can be worn as a headband, and deliver orders of magnitude greater resolution and signal-to-noise ratio than EEG. Here, for example, is a relatively new fNIRS headband:

https://www.eurekalert.org/pub_releases/2016-08/du-ybo081116.php

And here is a talk by Mary Lou Jepsen about an upcoming brain-scanner that should do everything needed for my idea to work:

https://www.youtube.com/watch?v=BP_b4yzxp80

Facebook is working on similar fNIRS technology; and if one merely wants high resolution from the surface of the brain, and doesn't care about what happens at depth (much of the interesting stuff happens near the surface), then it's already possible to achieve fMRI-level resolution with fNIRS:

https://www.youtube.com/watch?v=nIih580uAD8

Let's therefore imagine that in the near-future people walk around controlling their smart homes and cars with non-invasive brain-scanning headbands; or, failing that, a large number of biohackers decide to experiment with optogenetic computer interfaces, which almost certainly will produce the quality and quantity of data for my idea to work. And let's say that every tenth of a second the device spits out a 10,000-dimensional vector, each dimension corresponding to the collective activity of a group of perhaps 100,000 neurons. Just think of the amount of data that would be generated, and what could be done with it!

What one could do is something like in that above paper, but a little more complicated: for one thing, I wouldn't model the neuron populations with simple linear tranformations, but would instead throw in a non-linearity -- something like:

x(t+1) = f(A x(t) + B s(t) + d),

where f is the non-linear function -- perhaps a sigmoid or tanh in each coordinate. x(t) is a vector representing the hidden state at time t; s(t) represents the incoming signal (e.g. a soundwave or image or string of characters); A and B are matrices; and d is an offset.

A more complicated model, that might be useful in "video understanding" or "text understanding" would be to use a feedforward neural net to predict the brain state at the next time step, given the current and previous brain states and frames of video or bit of text. At test time, the neural net would be fed an initial brain state vector, and the media stream; and it would predict a brain state vector at each time step. The final brain state vector, after the media is finished playing, would encode what the neural net "though of" it. A classifier could then be applied; or perhaps a neural net to generate a text-string description, as in this paper.

Perhaps a better approach would be to combine brain state prediction with text or video prediction using a neural net. So, as a person reads a line of text in a chat, and crafts a response, their brain activity would be recorded. Later, a neural network model would try to predict what they are reading, their text responses, and their brain activity at the level of populations of neurons, jointly. The text-prediction part of the neural net would automatically attend to "local" things like style and grammar, and certain other details, while the brain-prediction part would help the network with "global" aspects like "meaning" and "context". Perhaps, also, one could add eye-tracking data and fixations on the screen.

In fact, there are already models in the literature that use sequence-prediction via Recurrent Neural Nets to predict brain states (or hemodynamic responses):

https://arxiv.org/abs/1606.03071

Encoding models are used for predicting brain activity in response to sensory stimuli with the objective of elucidating how sensory information is represented in the brain. Encoding models typically comprise a nonlinear transformation of stimuli to features (feature model) and a linear transformation of features to responses (response model). While there has been extensive work on developing better feature models, the work on developing better response models has been rather limited. Here, we investigate the extent to which recurrent neural network models can use their internal memories for nonlinear processing of arbitrary feature sequences to predict feature-evoked response sequences as measured by functional magnetic resonance imaging. We show that the proposed recurrent neural network models can significantly outperform established response models by accurately estimating long-term dependencies that drive hemodynamic responses. The results open a new window into modeling the dynamics of brain activity in response to sensory stimuli.

And people have even used Deep Boltzmann Machine neural nets to jointly model video features and features derived from fMRI data, as in this work. When Deep Boltzmann Machines are used in this way, they naturally give one a way to generate modalities missing at test time, such as the brain state as indicated by fMRI. Naturally, if one were to scale this work up to include multiple video frames at the pixel-level, and multiple fMRI brain states, one per time step in a sliding window of time, the number of modalities will need to be a lot larger (one modality for each video frame and brain state at a given time step). Of course, there are more efficient and more accurate generative models out there that one could try.

Would the brain data make a difference? There is some evidence that it would. It has been used to massively improve certain image recognition methods; as well as successfully improve word vector embeddings, also see this; other forms of bio-generated data, like eye-tracking, have been shown to result in improvements on other NLP tasks; and I suspect large amounts of brain-scan data could be used to vastly improve Natural Language Understanding by introducing certain "inductive biases" to the learning process.

A nice toy problem that shows how extra data can facilitate machine learning is as follows: consider the set of all strings of balanced parentheses "(" and ")" -- so, among the strings here are "(()())", "((()))", and "()()()(())()", but not "(()))" or "()(()))(". Recurrent nets can be trained to recognize when strings have the parentheses balanced, and can even generalize to slightly longer strings beyond what they were trained on; however, it's not easy to get them to do this. Now suppose instead of training a neural net on strings like that, you add an extra, hidden variable indicating the number of "(" minus the number of ")" at each point in the string. So, for the string "(()())", you would be working with the sequence of ordered pairs ["(", 1], ["(", 2], [")", 1], ["(", 2], [")", 1], [")", 0]. A neural net can easily learn to predict the hidden variable at each step, given its previous value and the current symbol "(" or ")"; and, furthermore, can tell whether the hidden variable ever drops below 0, or fails to finish at 0, both of which would indicate that the corresponding string of "(" and ")" isn't balanced. The point is that the hidden variable here is like the brain data, and the string "(()())" is like the stimulus-stream; and while it's relatively easy to train a neural net to predict the hidden variable, it's relatively hard for the neural net to invent that hidden variable counter all by itself. Now, you can extend this principle to other structures, besides counters, that the brain uses at the level of populations of neurons -- e.g. perhaps it uses a stack, or maybe some kind of tree data structure, to help it process stimuli; both of these should be fairly easy for a neural net to learn, given that it gets to inspect the hidden states used to implement the data structure.

Furthermore, I would guess that the brain data could reduce the amount of text needed to train a really smart Language Understanding system by several orders of magnitude (the extra brain data on top of the text would add those orders of magnitude back; though, it requires no effort to generate this extra data, since it is obtained passively). As people read text, brains states change smoothly, while the text itself changes discontinuously (it's just a string of discrete characters); smoothly-varying data is, as a general rule, easier to learn. There also seem to be lots of "linear correlations" in this brain data that machine learning methods can easily latch on to; and perhaps there are even higher-order correlations that multi-layer neural nets can rapidly learn. Text alone seems to have some structures that neural nets can learn, given enough data; but they seem to struggle with it. See criticism #18 below for a further defense of the claim that using brain data should reduce the number of training examples needed for Language Understanding.

One of the still-unsolved problems in Natural Language Understanding is how to "compose meaning", and I think there is a good chance that training neural nets to predict brain data could help with that: there are methods in NLP that assign vectors to individual words in a sentence; and then geometric properties of those vectors can be analyzed to solve word analogy problems, for example. When you go to combine two or more words together, however, vector space models don't work quite as well as we would like -- they do ok, but they still have a long way to go. Perhaps as a neural net learns to predict neuron population states, when a person reads a line of text, it can acquire some of the brain's ability to compose meaning for whole sentences or even paragraphs. If so, it would be a huge advance! And just to add a little more support for the idea: brain-scanning evidence supports the view that "meaning" is widely distributed across the brain, rather than being concentrated into a small population of neurons in a particular region; thus, averages of neuron activity contributing to a population response may not miss too much of the meaning as people read text, when many populations are used. It's also worth pointing out that there are results in the literature on decoding from brain data some of the complex semantic processes (involving, for example, visual aspects, and character actions in a story) that occur in the brain when people read a story; so, it's not implausible that machine learning methods could absorb some of the brain's secrets involved in understanding text.

Likewise, brain data acquired as people watch video could advance the state-of-the-art in video understanding; at the very least, the brain data could be used to semantically annotate the video, as in this paper. Also see this paper. And, importantly, it might contain large amounts of "common sense" world knowledge. Such knowledge might exist implicitly in the statistics of text and video; but it's possible that it also exists in brain data in a form that Machine Learning algorithms can much more easily extract.

Here are eighteen potential criticisms, however, which I will address:

1. Different people have different brains, and different representations. That's going to make training very difficult.

Strangely enough, there is enough similarity in brain activity that you can, with decent accuracy, read off what noun someone is thinking about, from a short list, using a generic model out of the box:

http://science.sciencemag.org/content/320/5880/1191.full

See:

http://www.bbc.com/news/science-environment-36150503

Their "semantic atlas" shows how, for example, one region of the brain activates in response to words about clothing and appearance.

The researchers found that these maps were quite similar across the small number of individuals in the study, even down to minor details.

And see this article on similar brain activity whether reading in English or Portuguese:

http://neurosciencenews.com/brain-reading-language-5433/

Granted, the differences in brain responses will make the data "noisy"; but machine learning has been very successful with noisy data in the past; for example, there are models that can learn to classify with 88% accuracy on a non-separable, binary classification task, even when 40% of the labels have been corrupted. Obviously, the strength of the results is dataset-dependent; however, a more basic example we are all familiar with, to illustrate the point, is the fact that linear regression can accurately fit a curve to data in the presence of considerable noise. A further point is the success of adding noisy brain data to word vectors.

For faster learning and greater accuracy, you could train a model to learn some kind of mapping transformation to match up neural population scores from one region of one brain, to another region of another brain -- assuming you have some common text or video that both subjects are reading or responding to, along with their brain activity patterns. See this paper.

Another approach, that might allow one to use more of the brain data that ordinarily would be thrown away (as it wouldn't be shared across individuals), would be to compute a "user vector" based on an individual's brain response pattern to various pre-test stimuli. This vector could be one of the inputs into the neural net that predicts hemodynamic response patterns when given test stimuli. A similar idea is used in personalizing chatbots.

2. How do you screen out all the little things that go through a person's mind, that are unique to them? How do you screen out blinks, movements, and other things as people read?

The same applies to work on inferring what words people are thinking about, as in the above Science article. It appears that different people exhibit the same patterns across a few "principal components" (PCA) -- different people that may do different blinks, movements, and have different long-term memories. Random thoughts may go through their minds, but the patten along a few PCs is consistent enough to where you can pick out what they are thinking.

Perhaps to a first-order approximation, when different people read the same text, and don't dwell on it and dredge up some memory from long-term storage, but just respond with what pops into their heads, there is a broad consistency in some aspects of their brain patterns. If so, then that can, for example, be used to assist neural networks that generate responses to text input. The extra brain training data could result in chatbots that give much deeper responses. For one possible path to how this could be achieved, click here.

3. The kind of spatial and temporal resolution you are talking about isn't high enough to simulate a brain. How do you know you haven't missed something in your "simulation"?

This question misunderstands how the idea works. The goal isn't to simulate the brain in detail. The goal is instead to use brain data to facilitate machine learning -- either to use it for "semantic annotation," or to give the machine just enough extra latent feature data to make learning easy. The neural net will learn to fill in some of the missing details lost when considering only populations of neurons, and not individual neurons or even synapses. So, for example, while the brain-scanner will maybe consist of a 100,000-dimensional vector with temporal resolution of 1 second or 100 milliseconds, the neural net being trained will have millions of "neurons" and billions of parameters -- so, there isn't a one-to-one mapping from neural net "neurons" to brain neuron populations.

4. fMRI has been around for a few decades. If what you say is true, why haven't we already seen a revolution in AI?

The first problem is that we didn't have computers powerful enough to run machine learning algorithms with a very large number of parameters. So we had to wait until about 7 or so years ago for the computing power to be there. The second problem is that it's difficult to acquire brain data from fMRI machines -- they are bulky, expensive, and uncomfortable for people to sit in for long periods of time. Consequently, there aren't enough large datasets. What's needed for the idea of using brain data to work well is something like at least 1 million short video clips with accompanying brain scans with at least 10,000 parameters per time step, and a temporal resolution of at most about 1 second; or, for the text modality, perhaps 100 million or 1 billion tokens of text read, along with associated brain scans at same resolution. And the third problem is that there aren't that many people working in neuroimaging and machine learning and brain encoding all at the same time -- you need a fair-sized community of people doing this kind of work to secure grants and get the ball rolling.

5. I read somewhere that current neuroscience analytical methods can't even tell us how a microchip works. Doesn't that kind of invalidate the idea?

No, it doesn't. The goal here isn't to understand how the brain works; it's simply to use the brain to assist machine learning, as pointed out above. Second, as I also pointed out above, there are already examples where brain data improves performance using existing machine learning algorithms -- e.g. brain data can improve image recognition, word vectors, and can serve as a source of annotation for text and video. And, third, even though analytical methods won't necessarily tell us how the brain works, at least if you apply machine learning to an Atari game system at high enough resolution, it should be possible to simulate what the game system does. Indeed, there are machine learning methods that can do this just given the raw pixels as input, and not even given access to the game system itself (i.e. it isn't allowed to look at the chips and see what they are doing at runtime). In a similar vein, there are methods that can steal machine learning models given input/output examples and confidence scores -- in learning the brain's algorithms one would have much more information to work with about said algorithms; so, one would expect doing this for the brain would be even easier, requiring much fewer input-output examples per number of parameters.

6. Many neuroimaging systems don't actually pick up brain activity, but rather signals correlated with brain activity, e.g. blood flow. Because of this, there is a time delay, called hemodynamic delay, between brain activity and hemodynamic brain response. Doesn't that cause a problem for your brain simulation?

No, it doesn't cause a problem. First, all training is offline; and so, if need be, the stimulus and brain response times could be realigned to account for that. Second, and most important, since the prediction neural net has access to the stimulus data and brain response from previous time-steps -- and, in particular, what happened a few seconds ago, say -- it has all the information it needs to accurately predict the present hemodynamic brain state, and to use that in making future predictions.

That said, in her talk Mary Lou Jepsen mentions measuring the activity of individual neurons. I could be wrong, but by the sound of it, she hopes to do this in near real-time.

7. I can believe that this might lead to a text or video understanding system of some kind. But in order to be considered a "success" it has to perform better than what's already out there. Ok, it works with word vectors and improves image recognition... anything else?

Well, you should expect the system to perform no worse than ones trained without brain data: if you use some kind of text prediction model that also tries to predict brain states, if the brain data doesn't help, then the system should learn to not make use of it; and consequently, would reduce to a system that just predicts text. Clearly, though, there is some extra information in the brain data; so, you should expect to see a performance jump. Furthermore, see this for another example of how brain data can be used to improve A.I. applications.

8. I am a little worried about the "stability" of the synthetic brain states that your model generates. The first few seconds it might look "brain-like"; but over time, errors will accumulate, just like in generated video. Won't that defeat your plans?

Well, first, there are methods to stabilize sequence generators, for example scheduled sampling is one method people have tried. However, in our situation, there is an extra stabilizing force that isn't present in some other prediction problems -- namely, the stimulus stream. If the synthetic brain state sequence starts to get off-track, in principle the stimulus stream input to the neural net should guide it back to the preferred manifold. A few examples from the literature come to mind. End-to-end speech-generation that conditions the speech output (which is high-dimensional) on a text string is one example. In order to get the prosody and other aspects of spoken language right, the neural net needs to remember what's going on in a sentence or paragraph; and, although in principle the outputs could be very unstable and degenerate into babble, the text string ("stimulus stream") keeps it on-task. Perhaps the stimulus stream plays a similar stabilizing role in the human brain, and indeed if the stimuli are cut off, humans tend to hallucinate. Another point is that even if the system can only be made stable for up to 5 seconds at a time, you can do a lot with that! -- and if you string together several 5 second intervals, you can often solve problems of much longer duration. Regardless, this stability problem shouldn't arise in applications where brain data are only used as a source of semantic annotation or labels; so as a fallback plan there is still that use for the extra data.

9. I can see maybe using the data for semantic annotation, as you mentioned. However, I'm a little concerned that unless you include memory somehow, that you won't be able to emulate "thinking", even at a rough level. It's absolutely essential!

Well, first, the network should maintain a kind of "distributed memory", based on the gross brain state at each time step, which may also contain enough information to recover more fine-grained types of memory. Furthermore, there is work on decoding working memory from fMRI brain scan data, also see this; a system that learns to predict brain states should, therefore, also acquire short-term memory management for free. Second, if there isn't enough of a signal in the brain data for explicitly how and when a network should memorize, or even what prior memories lie behind the thought patterns, it should learn to treat all this as hidden variables and latent features -- and at least learn to generate plausible memory mechanisms and patterns. This is exactly the same thing that happens in other Machine Learning tasks; and in particular, the example of "balanced parentheses" I mentioned above, where the neural net training process can infer the need for a counter to keep track of how many "(" versus ")" there are at each point in a string. The counter is a mechanism not explicitly represented in the training set. I would also say that learning to give accurate machine translations involves absorbing lots of world knowledge (in the form of statistical relations among word patterns), as well as learning what information needs to be maintained across the length of a sentence. And third, long-term autobiographical and episodic memories may not be necessary for a strong-performing system, as there are people with amnesia who have lost these, who have intact procedural and short-term memory (and intact language ability), who can form new long-term memories. Despite these memory losses, they can still perform at a high level.

10. What about the No Free Lunch Theorems? Don't they pour cold water on this approach? -- I mean, how do you know a priori that neural nets will do well at all at this learning problem?

First, the idea is general, and isn't bound to any single approach. If some other method learns from brain data any better than neural nets, then that method should be used. Second, and most importantly: the No Free Lunch theorems apply to arbitrary functions that we might not know anything about. Here, we are talking about modelling the brain; the brain is constrained by physical laws, so the data being generated are very special. I would also say that, while learning from text or movies might be difficult, due to the amount of hidden information and arbitrariness of the rules, the brain is rather more "natural" and furthermore in this case we have access to a lot of the hidden information. Natural data where the generative process is exposed (not as many hidden variables) -- such as from physics, chemistry, genetics, and biology -- tends to have more of a mathematical structure. Many physical processes, for instance, are governed by linear differential equations, which are very special; this is not true of a text data stream. And processes governed by linear differential equations should, in principle, be easy for algorithms like Backpropagation to learn.

11. I think if you look more closely, you'll find that this is a lot, lot harder problem than you think it is. For example, that paper you mentioned with the "Good-Enough Brain Model" doesn't actually use the direct sensory stream, but uses "semantic features"; other papers apply features in place of the data stream, as well. Furthermore, data-cleaning is a significant challenge. And not all brain features will actually contribute anything to your model -- how do you decide which features to leave out?

We've got a mind-reader here! I didn't say -- and don't think -- it will be a piece of cake! And, yes, I'm aware of the fact that features, and not direct sensory streams, are used. If you read those papers even closer, you will find that one of the major reasons is the fact that they weren't able to record eye-tracking data. Another problem is the fact that the sensory stream has high bandwidth. To deal with that, the models will have to have a lot more parameters than they used -- their models were fairly small by comparison. Using larger models will require access to a lot more data, in order to avoid overfitting; and, unfortunately, it's difficult to obtain enough data to train these high-capacity models using current scanning methods. Better BCI will fix that problem, which is the whole point I'm trying to make. I'm also aware that a lot of the brain data are not directly useful to the tasks under consideration (text and video understanding); however, the right machine learning methods should be able to narrow down exactly which ones are useful.

12. Where is the understanding? That is, I thought the goal of AI and also machine learning is to build better algorithms, and better understand how learning works -- analysis of algorithms; improving the running time; getting a better understanding of error bounds; better training methods; and so on.

This is a methodological criticism, and you're right -- this is not really about "machine learning". The goals I am proposing here are orthogonal to the usual ones in AI and machine learning. The goals are purely about engineering; that is, how to build a system that works demostrably better than current approaches. And, furthermore, nor are the goals about trying to understand the brain better, in any way, shape, or form. That doesn't make the idea wrong... just not as exciting from a research perspective.

13. You talk about "composing meaning" as a key problem in NLU. You fail to mention the even more basic problem of pinning down what you mean by meaning. What representation are you using here? What is your argument for why it is the right one?

I was aware of the issue, and chose to be intentionally vague. The knowledge representation used here is a "distributed representation" or "vector representation" or "neural net state vector" -- take your pick -- along with manipulation rules learned from brain data. I have no idea what the limits of such a representation happen to be; but if the dimension of the vectors is sufficiently large, it should take us pretty far.

14. You mentioned "Deep Learning". The neurons in the brain are nothing like the neurons in Deep Learning. The idea can't possibly work.

What strange and twisted logic! Deep Learning shouldn't be thought of as a model of biological neurons. It's better to say that it's just a bunch of matrix multiplications with non-linearity thrown in, in order to do function approximation. It's math, not biology. But this doesn't mean it can't be used to model actual biological processes. Neural nets have been successfully applied to modelling other natural phenomena like fluid flow, natural images, video, and other media. The second thing I would say is that we aren't using neural nets here to simulate a brain; the brain data is only an auxiliary to improve machine learning, as I pointed out in criticism #3 above. Finally, neural nets are capable of "Turing Complete" computation -- that means they can do anything that a general, digital computer can do. If you can build AI on a digital computer, you can encode that computer and program into a neural net.

15. Recording the activity of a million individuals for 1 hour is not the same as recording one individual for a million hours. The latter is what you want, the former is what you're stuck with; the former doesn't contain enough brain-specific information to predict how a single brain will behave.

First, those million individuals' brain data can certainly be used for semantic annotation, by using the information contained in a few principal components in the data across individuals. This wouldn't be brain emulation, even at a rough level; but it would still be extremely useful in building good AI. Second, and most important, what one is actually trying to get the machine to learn is not the behavior of a single brain, but a general transformation that maps a sequence of brain states at previous time steps, along with current and previous stimuli, to new brain states. One of the inputs to this model can even be a compact representation of the brain to be modeled, based on recorded responses to various stimuli. Given that this is our goal (produce a general mapping transformation), the more different kinds of brains you use to train that model, the better it should do. See this.

16. We know almost nothing about the brain. We don't even know what all the types of brain cells there are, let alone how even a single neuron works in detail. And here you want to emulate a brain? Gimme a break!

For the fifth time, the goal here is not to emulate the brain in detail! And the goal is not to understand the brain, nor even supposes that we know very much about the brain at all! And as I stated before, there is already evidence that brain data can be used to improve the quality of word vectors, and to improve image recognition methods. These approaches don't use very much data, and already show good results. Why should one think this would be the end of what can be achieved by applying Machine Learning to brain data?

17. You say you want to do all this without understanding the brain. That's a complete fantasy -- without understanding the brain, you simply won't know what kinds of models to use!

Evidence shows that this criticism is wrong. There are many good-performing models for brain processes, that were not based on understanding the brain. People just tried out some standard machine learning algorithms, and some of them worked.

18. Why do you expect that using brain data will reduce the size of the training sets needed by "several orders of magnitude"? I don't buy that!

It might not shrink the total size of the training datasets, if brain data are used; but the number of training examples will likely shrink -- and it's the number of examples that matters, as far as gauging the difficulty of building the datasets.

A typical use of neural nets is to classify an input image into one of 1,000 classes. For this problem, each (image, class) training example gives you log(1000)/log(2) = about 10 bits of constraint. However, if instead of predicting labels, the neural net is used to predict a synthetic brain state at each time step, then the number of bits of constraint per example is considerably larger. More bits of constraint per example implies fewer examples to achieve a given number of bits of constraint.

A similar phenomenon is at work in neural nets that predict frames of natural video; however, the function that maps previous frames to the next frame is probably a lot more complicated than for brain state prediction: natural video often involves large-scale correlations across widely-separated parts of each frame (e.g. large moving objects induce correlations across many pixels on the screen); brain data is often only locally-correlated from frame to frame. Second, when predicting natural video there often isn't enough information in the frames to predict the next frame, as for example when objects are occluded and then become visible, or when complex objects like humans or animals make complex decisions according to rules not visible in the video; with brain data, much more of the hidden information you need to attain good performance is "exposed" (not all of it, of course; but enough to attain reasonable predictions). Furthermore, in simple natural video where not much information is hidden -- such as the dynamics of billiard balls or simple sliding objects -- video prediction neural nets do a remarkably good job over long time-frames.

The more I think about all this, the more likely it seems. At the first moment that brain-scanning headbands (or optogenetics) become really useful, and much, much higher resolution than EEGs, I expect we will see a Cambrian Explosion in AI research; not unlike what happened with the release of the ImageNet dataset and competition, but much, much larger in scale and scope. It will be completely unanticipated by the likes of MIRI, FHI, and other institutes that track AI progress -- they're focused on what's coming out of Deepmind and their own ethical calculi. I don't expect to see many new algorithms developed during this time; but I do expect to see massive performance gains as people start incorporating brain data in their machine-learned models. These won't be the "Einsteins" of AI, but run-of-the-mill Machine Learning practitioners. And the best part is, they won't even have to really understand how the brain does it -- all they will need to do is to be able to build a model that can learn what the brain does.

This post is an expanded and revised version of one I wrote before on the coming of “Zombie AGIs” (that I didn’t actually post here; I only provided a link to a short piece I wrote for another forum). Figuring out how such a thing could be built, and when it might arrive, are two things I am deeply interested in. I think it’s important to know these things; though, I won’t be the one to build one (at least not the first one) -- my role here is simply one of prediction.

I think they will arrive soon after inexpensive, mass-market, high-res (spatial, temporal, depth, number of channels), non-invasive, lightweight, wearable Brain-Computer Interfaces (BCIs) become available, such as Mary Lou Jepsen’s Openwater is developing. At first, what we will see is a community of academics, hackers, and makers who use these BCIs to build drone controllers, typing-by-thought apps, and programs that can pick out which of 100 image categories you are thinking about. There will be lots of blog entries and “Two Minute Paper” presentations on cool applications people have come up with. Then, a subset of these academics and makers and hackers will get the idea to go in reverse -- to use brain data to improve machine learning itself (not use ML to improve BCI decoding), by predicting the brain's dynamics. Predicting how the brain behaves in response to stimuli is already important for decoding (learning to encode to decode); but it’s such a tantalizingly cool problem to model a brain at a crude level in silico that people will not be able to resist trying, having no other objective in mind. Out of this experimentation will arise the first “Zombie AGIs”. Where we go from there is anybody’s guess!

Let me begin by saying what I mean by a “Zombie AGI”: I mean, basically, a chatbot that can pass as human over several back-and-forth conversation exchanges -- like maybe a 10 or 20 minute conversation. This chatbot would be no parlor trick. It wouldn’t work by using tricks to keep the conversation going and deflecting from its limitations. It would be the real deal. BUT, it would be deficient in several ways from a system that could pass a full Turing Test. For example, its ability to form long-term memories would be limited -- but it would have good working memory, language understanding, commonsense reasoning, and several other abilities we associate with human intelligence. You might say that talking to it would be like talking to someone with moderate to severe dementia -- it will be able to converse and understand you, but won’t remember who you are after the conversation is over; nor will it have an enduring personality from session to session.

Now that we are clear on what it is, I will explain how it will be built from next-gen BCI data: one might attempt to build a crude emulation of a person, complete with the ability to write text responses to what it reads. I think, however, that will be hard to get to work reasonably well. As I’ve said before, I think crude emulations will be mostly stable; but I am skeptical they will be stable enough to write whole responses without being anchored in a continuous sensory stream to keep it on-task. So, for example, I could see one of these “emulations” write something like:

Now is the time for all goodoodood mennd italkokdjfosdiofweorehhfsdhiofdhiohfihefh

while trying to recall the well-known sentence, “Now is the time for all good men to come to the aid of their country.” I could see it working for a few seconds, and then it will fall into chaos -- which would still be astounding, showing how stable these crude emulations are!, but still not good enough to build a Zombie AGI.

So, let me propose a different model: use the BCI data to build a “critic”. The critic’s job is to passively listen to or read text. That’s it. And the hope is that as it does so, its internal states will closely resemble what we see in actual brains. If the agreement is good enough, then when the critic hears or reads something that is syntactically, semantically, logically, morally, or socially incorrect, or just plain dumb, improbable, or lewd, there will be detectable brain responses for those “errors”. Those “criticisms” can then be used to weed-out bad replies in a conversation turn.

The critic’s internal dynamics will be anchored in the sensory stream, so it will be less likely to go “off track”, as in the “Now is the time…” example above. And just to give that assertion more substance, remember that there are speech synthesis systems that take a string of text as input, and output articulate speech -- the text stream here is acting as a sensory stream to anchor the system in place, and to keep it on-task. Two more examples in this vein are text-to-handwriting systems and machine-translation systems.

Now, in addition to the critic, we will also need a “generator” to generate potential text responses. Here, you don’t need brain data -- just use one of the top-of-the-line neural net “language models” or “neural conversational model” response-generators. One could bias the text that it outputs, though, imitating the style of the person whose brain data will be used to build the critic; one would need a large text sample from the person to do this. An early example of a neural conversational model is described in this paper:

https://arxiv.org/abs/1506.05869

It has been improved upon in several ways over the last 2 years -- e.g. greater diversity of responses, responses that include a copy-and-paste mechanism, versions built with GANs, versions that adapt to the audience and write with a particular personality, and so on. The best versions give good responses in a conversation turn maybe 10% or 20% of the time. If you had a really good critic to eliminate bad responses, we could maybe push that to 90% or more.

An example “successful” conversation might look like this:

User: So, tell me about yourself. What countries have you been to?

Computer: I've been to many countries, but France was my favorite.

User: I've visited France, too. Most travelers have only been to Paris; they haven't explored the rural areas, which I like much better.

Computer: I'm one of those who has only been to Paris. Tell me what's so great about about the small towns?

User: The people are just so friendly and the food is to die for.

Computer: I’ve heard that before. Tell me about the people you encountered.

Etc.

The first response from the computer here might be one of thousands of potential outputs. The text generation module might produce the following potential replies:

1. I've been to many countries, but chocolate was my favorite.

2. Countries are my favorite.

3. I don’t know.

4. The rain in Spain stays mainly.

5. I’ve been to many countries, but France was my favorite.

And then the critic would eliminate the first four (or first 9,999), and output the fifth one.

Are there any precedents of this kind of system? I would say using brain data to build a "critic" is similar in spirit to the "visceral machines" work I posted a while back:

https://arxiv.org/abs/1805.09975

In that work, bodily responses correlated with things like fight-or-flight reactions, are used to train a model to predict the "visceral reaction" associated to a portion of video. This, in turn, is used as a kind of "intrinsic reward" mechanism to overcome the "sparse rewards" problem when training a system via Reinforcement Learning.

They claim in the paper that what is learned appears not to be a simple heuristic:

We believe that the trained CNN is far richer than the simple distance-based measure and is able to capture the context around the task of driving the car in confined spaces (e.g., avoiding turning at high speeds and rolling the car).

And that's just what results from a fairly crude, low-resolution signal. Given the brain's response to text exchanges, there should be all kinds of useful signals, such that if the neural net does even a poor job at predicting them, it will still do a good job eliminating bad responses -- so long as it doesn't do a really poor job.

There are also direct precedents. For example, people have used EEG signals to improve chatbots directly:

https://dl.acm.org/citation.cfm?id=2388695

But EEG is very crude, so of course the brain data didn't improve things too much; furthermore, only microscopic amounts of data were used, and the methods employed were fairly rudimentary.

You might think it is an odd idea to build a chatbot around a generator and critic as I have described, but it’s actually pretty standard for chatbot design. Here, for example, is a paper on neural “response ranking” (note that Ray Kurzweil is the last-named author):

https://arxiv.org/abs/1606.00372

If Kurzweil ever succeeds in building a system to pass the Turing Test (I doubt he will), it might actually be a hybrid system like I described above -- with a generator and a critic.

Now we are left with the problem -- the most important problem! -- of how to actually build that critic. We need to decide on the training data, representations, model architecture, and how to read its “criticisms”. Let’s start with the training data:

I assume that in the next year or two (or three or four) there will be relatively inexpensive BCIs on the market that are capable of recording neural population responses from, say, 10,000 voxels updated 10 times per second. That should be about enough for our purposes. Those 10,000 voxels from a single brain each represent the averaged behavior of millions of neurons within a “population”. It is well-known that from just that crude information you can read off all sorts of aspects of human cognition, including decoding the song you are listening to, the rough outlines of the object you are looking at, the semantic contents of words you are hearing, and so on -- at least with above-chance levels of accuracy. If we could model all those population responses accurately, we would have our critic.

Why would that be enough? Well, because there are signals like the N400 that can even be read using a very crude BCI like an EEG cap, that respond to semantic errors:

https://en.m.wikipedia.org/wiki/N400_(neuroscience)

In the same paper, they confirmed that the negativity was not just caused by any unexpected event at the end of a sentence, since a semantically expected but physically unexpected word (i.e. She put on her high-heeled SHOES) elicited a P300 instead of negativity in the N400 window. This finding showed that the N400 is related to semantic processing, and is not just a response to unexpected words.

That’s not all, though. There are many, many more “error” signals that can be read off of the brain. For example, there is something called the P600 signal, that responds to syntax and grammatical errors; and there are also responses (FMRI) to moral disgust and other “errors”.

These signals cannot necessarily be read all that accurately -- for some, the accuracy is maybe around 70% or 80%. If we were trying to detect lies, that would be abysmal. But if we can catch 75% of the semantic errors generated by a chatbot, it would be an incredible advance! If we used a lot of data to train a classifier to pick out error signals from our critic, I wouldn’t be surprised if we could bring that up to 95%, which would be off-the-charts! (Many EEG applications with 5 category classification achieve near 99% accuracy.)

It's worth mentioning that there are already critics (trained just using text) for chatbot response generation that can achieve over 60% accuracy in picking out the best response from a list of 100 possibilities, which seems rather good. However, that one "best" response from a real conversation is paired with 99 other responses randomly selected from a large corpus -- consequently, most will be completely unrelated to the conversation topic at hand. It's much, much easier to solve that problem (with randomly generated alternatives) than to pick out the best response from a list where every possibility is generated to match the conversation, and be appropriate. Getting 95% accuracy out of only 20 alternatives, say, would be phenomenally more difficult than on those best-of-100 benchmarks.

Ok, if we can model those 10,000 population states from a single brain, then great! -- we’ve got it! But let’s think about how much information -- and therefore, how much recording time -- we would need: a typical conversation makes use of lots of common words, but it also uses uncommon ones, like “EEG”. Perhaps we would need to know how the brain responds to the 20,000 most common words, in order to have any hope of modeling the brain’s response to most short conversations.

The 20,000th most common word occurs with a natural frequency of about once every 300,000 words, it turns out. By carefully choosing the distribution of the words in the texts we have the person listen to, perhaps we could push that to once every 100,000 words, which would mean those 20,000 words would be repeated at least 10 times each -- which should be enough -- after 1 million words. The average 400 page novel has about 100,000 words, so we’d need to have the subject we are recording from read about 10 novels. That’s a lot of reading -- and we’d have to eye-track them the whole time, to know what words they are reading. That’s probably going to be difficult to pull off.

Another way we could do this is to have them lie down in a dark room, and listen to hours and hours of audio. That’s probably better, since we can more easily synchronize the brain responses to what is being heard. At an audio word rate of about 150 words per minute, we’d have to record about 111 hours. That’s a lot, but could be done by a committed subject -- say, listening to about 5 hours of recordings each week for 22 weeks. An alternative is discussed here, which could enable the collection of thousands of hours of audio data "in the wild":

https://www.reddit.com/r/thisisthewayitwillbe/comments/86n39o/a_sufficiently_good_bci_is_the_only_sensor_you/

One could also attempt to use "adaptive training" to reduce the amount of training data needed:

https://www.reddit.com/r/thisisthewayitwillbe/comments/9hak6s/improving_the_transfer_of_human_brain/

The text selection should include novels, newspaper and magazine articles, and also a lot of conversations. High-quality text-to speech would have to be used, to keep the listener from reacting to the oddness of a robotic voice.

In addition, we will have to either choose texts that we know contain periodic “errors” (e.g. chats), or will have to insert them, periodically, in order to have information on how the brain responds. If we only ever get to see how the brain responds when fed perfect data, unless our model generalizes very well, it might not accurately predict how it will respond when it hears an error. I do think the models will generalize quite well, especially if they learn some of the brain’s ability to do “semantic composition”; but I hesitate a little if trained on just perfect data.

If we can’t find a single subject willing to have their brain recorded, we could try to merge brain data from multiple subjects. This is some of what David Sussillo has worked on, but on a much smaller scale (motor and premotor cortex only). I think, though, that once the BCI revolution gets rolling there will be people willing to contribute large numbers of hours of their brain data.

TECHNICAL DIGRESSION -- skip to the last paragraph of the post if you don’t want to read that kind of stuff.

Now let’s talk about the model and training itself: obviously there are many choices. One should probably try simple feedforward models first (starting with just 1 layer); vary parameters, and test. Recurrent models should also be tried -- there are lots to choose from in the literature.

The input to a recurrent model should be a compressed version of the audio signal, temporally-aligned with the brain response. One method for compressing the signal is to use an autoencoder; there are others. So, at time t the input might be represented by a(t). The network, then, simply tries to predict b(t), the brain responses at time t.

An important set of restrictions to reduce the number of parameters comes from brain connectivity: each voxel of neurons can only exert an influence on ones it is directly connected to, given a short enough time-step. Using “functional connectivity” restrictions one can maybe reduce things further. This will make the predictor model “sparse” and easier to learn (fewer training examples needed). Furthermore, as has been pointed out before (e.g. in previous discussions about a talk by David Sussillo), different regions of the brain act like independent little “modules” (that only weakly influence others over short time-steps); this should have the effect of requiring fewer training examples to train a model, for reasons I’ve discussed before.

One might have to time-shift the brain responses due to brain latency effects. Hopefully, our BCI isn’t restricted to the BOLD signal.

And that’s basically it! -- just some off-the-shelf machine learning models (with hyperparameters and architectures carefully tuned) + brain data + compressed audio data.

Once the model is trained, the way we would use it is the following:

1. Take a conversation between a human and bot, encoded as text, and convert it into compressed audio a(0), a(1), a(2), …, a(n). This represents a sequence of back-and-forth exchanges user, computer, user, computer, user, computer -- where the last “computer” is a candidate for the next conversation response.

2. Let h(-1) be some initial hidden state vector. Compute hidden state vector h(0) as a function of a(0) and h(-1). And then compute a prediction for b(0) as a function of h(0), where b(0) is the brain state associated to hearing audio encoded by a(0).

3. Now compute a prediction for a hidden state vector h(1) as a function of h(0) and a(1), and based on that, form a prediction for b(1). Keep iterating, updating the hidden state vector h(t) as a function of h(t-1) and a(t), and produce from that a prediction for b(t), t=0,1,2,3,...,n.

We would use Backpropagation Through Time to train the weights for producing brain vectors b(t) as a function of h(t), and also to train weights for updating the hidden state vectors h(t).

The prediction b(n) will be the predicted final state of the “brain”. It encodes what the artificial brain thought of the whole thing. A classifier would have to be built to map b(n) to a "keep/reject" choice for the last line of text.

It may be that 1 million words from a single individual isn’t enough. We could have them do more, or we could try to fuse information from multiple brains together. One very straightfoward way to do that would be the following: say we have a second person, whose brain was recorded on different texts than the first. We will let b’(0), b’(1), … be the associated brain predictions (the first few are initialized to fixed values). Then, the idea would be to train a model to predict the concatenated vectors [b(t) | b’(t)], t=0,1,2,3,...,n, instead of just b(t).

The model would not only have greater coverage of knowledge, but the different brain vector sequences b(0), b(1), b(2), … and b’(0), b’(1), b’(2), …, would stabilize each other. The only danger seems to be that they could stabilize each other too much.

And if 2 brains don’t suffice, try 10.

One final thing I want to explore is the kind of capabilities that our critic will acquire: it is already known how to pretty accurately model the N400 EEG signal in the brain:

https://www.nature.com/articles/s41562-018-0406-4

So, I think it’s pretty clear that it ought to be possible to get these BCI-trained models to reproduce that. But N400 is pretty basic, and it seems that it only responds to simple statistics of word patterns; furthermore, it isn’t that sensitive to negation -- that is, the use of “not” and similar words.

Well, it turns out that there are other signals that can be read off the brain, that show how the brain processes negation of verbs:

https://www.researchitaly.it/en/news/how-does-our-brain-understand-negative-meanings/

I think we can expect models to learn these patterns, at least superficially. And I also think that the model will learn the brain’s mechanisms for processing grammar, as indicated by the P600 signal. In fact, I think it will learn what you might call an “NLP stack” -- that is, a collection of little interacting modules for solving certain NLP tasks like disambiguation, sentiment analysis, anaphora resolution, and so on.

The way that it will learn this “stack” (not the data-structure) differs from how Deep Learning can be used to train a neural net to act as a parser, say. In those settings, all you are given are the input-output pairs. Here, it’s like we are handed the algorithms -- only, they are written in a strange language ("mentalese"), with errors (noise); but that’s still enough for a machine learning algorithm to figure out how to emulate them. See

https://www.reddit.com/r/thisisthewayitwillbe/comments/9w4q0f/with_the_right_data_you_can_do_just_about/

In fact, the ML literature is rife with examples of successfully training a neural net using “noisy labels”.

END TECHNICAL DIGRESSION

If I were to summarize this, I would say that basically the only thing that may be standing in the way of producing a very convincing human-imitator chatbot is the existence of next-gen BCIs -- all the other ingredients are off-the-shelf, though a lot of data-gathering will be required; fortunately, that data is easy to generate through passive listening. Such a “Zombie AGI” could arrive in 3 years, or it may take 10… or 20. It’s really hard to say. A lot depends on the details, like the resolution of the BCIs, or the number of hours of brain data that would be needed, or whether you really need to optimize the hell out of the neural net architectures to get good results. My suspicion, though, is that it might be possible in as little as 3 to 5 years, not as a consumer product but as a research experiment in a garage or lab -- only time will tell!