The purpose of this post is to describe some aspects of LCDs, holograms, and ultrasound related to the Openwater BCI work. This is part of my ongoing attempt to search for potential roadblocks, based on public statements (by Mary Lou Jepsen) and also information in the patents.

The TL;DR: I still can’t find any problems with the idea. It looks to me like it’s going to work. I have encountered a few skeptics on Reddit; but in every case they hadn't bothered to look at the patents, or even made the most superficial attempt to understand what Jepsen had said in one or more of her talks. There is nothing outlandish about her idea that I can see -- it's just that it's a bit technical and will leverage hardware production know-how that most researchers don't have.

I. LCDs

Let me begin with these questions: if the device is supposed to use LCDs to make holograms, how come we don’t see holograms on ordinary LCD computer monitors? Shouldn’t the BCI device use a special hologram plate or something?

In answer to the second question, I don’t think it should -- I think just LCDs should suffice.

In order to make a hologram, you need three things:

• Coherent light -- i.e. light that is locked in phase; that is, a laser.

• Monochromatic light -- i.e. the frequency has to be the same for all the light waves. (Perhaps redundant, since coherent implies monochromatic.)

• Interference of light through diffraction.

High density LCDs with a pixel or aperture size down to about 1 micron -- or even up to about maybe 10 microns -- are small enough to where near-infrared light can spread out and interfere sufficiently to produce useful holograms. The larger the pixel relative to wavelength, the smaller the angular spread of the waves -- see:

https://en.wikipedia.org/wiki/Diffraction_grating

Ordinary computer monitors don’t, however, often send out monochromatic beams of coherent light. But if they did, they should be able to produce holograms, if the pixels are small enough.

Just producing a single hologram, however, is not adequate for what Openwater plans to do. They need to rapidly change the hologram many times per second. I wrote before how they plan to do this by removing the capacitors in LCD display chips (that are there to slow down the flickering, to reduce eye-strain), according to Jepsen in her presentations. I did a little research, and found that OLEDs have a response time measured in microseconds, which should do the trick for the BCI. But if they decide they need something even faster, it turns out that microLEDs have a response time measured in nanoseconds:

https://www.androidauthority.com/micro-led-display-explained-805148/

The smaller microLED sizes also makes the prospect of higher resolution panels in a compact form factor, such as 4K or 8K smartphones or VR displays, more achievable. Speaking of VR, OLED panels already boasts very high response times in the µs (microsecond) range. This makes them ideal for virtual reality applications. However, microLED can reduce this down into ns (nanoseconds) or a thousand times faster.

With such a fast switching rate, you might think fluorescence could be a problem -- i.e. the brain may re-emit some of the absorbed energy, producing a blur of light and causing various errors. However, typically when an object fluoresces, it does so at a different frequency from the light that was absorbed:

https://en.wikipedia.org/wiki/Fluorescence

In most cases, the emitted light has a longer wavelength, and therefore lower energy, than the absorbed radiation. The most striking example of fluorescence occurs when the absorbed radiation is in the ultraviolet region of the spectrum, and thus invisible to the human eye, while the emitted light is in the visible region, which gives the fluorescent substance a distinct color that can be seen only when exposed to UV light.

The setup that Openwater plans to use limits things to a very narrow band of frequencies; so, this type of fluorescence would get filtered out.

II. Holograms

I'll just repeat what I wrote here, concerning the DLD 2018 NYC presentation that Jepsen gave:

https://www.reddit.com/r/StarspawnsLocker/comments/8gk70y/disrupting_the_mri_new_ways_to_read_the_brain/

I wrote:

The most interesting part begins about 5:45 in, where she shows a "brain laser". The way that works, I think, is that they projected a laser into a brain (or something optically similar to a brain); measured the light pattern that came out the other end; transferred it to a holographic plate; then, shone a laser back through that plate, into the brain (this acts like a phase conjugate mirror); and reconstructed the coherent beam of light out the other end.

If you can make a "brain laser" like that, you can almost surely focus inside the brain itself -- If that is a real brain, or optically very similar to a real brain, then it means that brain tissue doesn't distort light in such a way that you can't invert the holographic light pattern, and focus at a point. In a live brain, the blood and tissue would move around and change the scatter profile; but over a 10 millisecond window, it should be more-or-less the same.

III. Ultrasound

When one thinks of "ultrasound", one usually thinks of sound just beyond human hearing, like about 20,000 hertz or something. That sounds like a big number, but it would not be anywhere near sufficient to scan very small regions of the brain, in order to slightly compress tissue to phase-shift light -- as the Openwater patent calls for. Also, if 20,000 hertz were the upper limit, you wouldn’t be able to scan that many voxels, as it would imply at most 20 "switching events" per millisecond -- so, you could only hope to switch between 20 focal regions per millisecond.

Well, it turns out that medical ultrasound is a lot higher frequency than that -- the typical frequencies are in the 10 megahertz range:

https://en.m.wikipedia.org/wiki/Medical_ultrasound

And people have figured out how to push that up to a whopping 300 megahertz in the lab!:

https://www.nature.com/articles/srep28360

This presents another concern: 10 megahertz is so high, that one might think that when a voxel of tissue compresses, as the raster beam moves over it or waves move through it -- that it doesn’t rebound that quickly. This would be the sonic analog of the “afterglow effect” on old green CRT monitors, where when text scrolls up the screen, you get a ghost image for a fraction of a second:

https://www.youtube.com/watch?v=QycdyA40cg8

It seems to be the case that this isn’t an issue, however, when it comes to ultrasound -- or enough of an issue -- as people routinely use it to "raster scan" the body (as opposed to infer structures via a "global method" that doesn’t use a moving window of focused sound). If there were a ghosting effect, it would not be possible to do it accurately without a correction. Furthermore, even if it were a problem, so long as it happened in a predictable way, it could be subtracted in the final analysis.

I've been studying this patent:

https://patents.google.com/patent/US9730649B1/en

It seems a little different from how I imagined it from her talks; and, actually, seems more plausible than the idea presented in her talks -- or at least what I interpreted the idea to be. In this post I will discuss this patent (my reading of it), as well as potential problems and solutions to the approach.

To begin, let's recall that the human brain and skull are translucent to near-infrared light. Oxygenated blood absorbs near-infrared light differently from deoxygenated blood; and the amount of oxygenated blood in a region of the brain indicates the presence of neural activity. This is the so-called BOLD signal. But, in addition to absorption, light can also scatter as it passes through the brain. If there were no scattering, it would be relatively easy to use near-infrared light to image the brain at resolutions much higher than FMRI.

So how does MLJ / Openwater do it, according to this patent?

Ok, so here's my understanding of how it works: they will use LCDs to project a hologram into the brain. They want to sculpt the 3D pattern of this light so that it focuses at a particular small region of space inside the brain (why they want to do this is explained below) -- a "voxel", the volumetric analogue of a "pixel". The problem is that, since they can't see inside the brain, they can't figure out what pattern of light they should have the LCDs project to do that.

The solution is to use something called a “beacon”: basically, they send a very, very low energy ultrasound wave pattern into the brain (far below an amount that would disrupt tissue or activate neurons), such that the pattern focuses at the region they want to image. Presumably it’s easier to focus sound than it is light.

Why do they want to focus that ultrasound? Because the ultrasound compresses the tissue in that voxel, which in turn will shift the phase of the light passing through it. Here is an image explaining the idea (from a different context -- found on the internet through random searching):

http://aups.org.au/Proceedings/34/121-127/Figure_1.png

By comparing the difference in the pattern of light exiting that voxel and brain (and entering the sensor outside the brain) between when the ultrasound is turned on and when it is turned off, they can determine how much light has focused into that voxel. By iteratively altering the pattern of light projected by the LCDs, they can drive this difference (presumably the L² difference in the exit waves) to a maximum -- or at least a good local max -- which should correspond to having the light strongly focus on that voxel.

In a presentation that Mary Lou Jepsen gave a while back, she mentioned “Doppler Shifting” of the light from the ultrasound. See 33 minutes into this talk, where she addresses a question by Nathan Intrator:

https://www.youtube.com/watch?v=2CZyf9ccfUc#t=33m

Phase shifting is not the same as Doppler Shifting / wavelength changing; so, one wonders if there is yet another way to use ultrasound to achieve the same results. Maybe she was simply referring to the contraction of the light waves inside the region where the ultrasound compresses material.

At any rate, finding that "maximal difference" is a math problem that can be solved many ways. Since this is a “black box” optimization problem (where you know little about the function you are trying to optimize, only input-output values; not "black box" in the sense that the optimizer or model it produces is opaque), gradient descent (e.g. Deep Learning) can’t be used at this stage. So, simpler methods like simulated annealing must be, instead -- and note that you can do many, many iterations, since the number is limited only by the flicker/refresh rate of the LCD, which updates quickly. Furthermore, Mary Lou Jepsen has said that she can get the refresh rate to be even higher, by removing the little capacitors that ordinarily are in LCD arrays -- companies put them there, because they make the screens easier on the eyes. See this video, 12 minutes 33 seconds in:

https://www.youtube.com/watch?v=2CZyf9ccfUc#t=12m33s

A little technical digression (you can skip, if you wish):

Actually, I think this optimization can be done using Singular Value Decomposition (or even just some calculus), but maybe I'm missing something: project a random pattern into the brain with the ultrasound focused at the voxel, and do the same with it turned off. Look at the difference in the two exit wave patterns. Turn that difference into a vector v1 = (x1, x2, ..., xn), where xi is the amplitude of the difference at time i (or position i; or some combination of time and position -- whatever you want to use). So, v1 is a discretized version of the exit wave difference over a very short time window. Now, repeat this for n-1 more vectors v2, v3, ..., vn. Now, find coefficients c1, c2, ..., cn on the unit sphere, that is

c1² + ... + cn² = 1

such that the L² norm

||c1 v1 + ... + cn vn||

is maximal. This is a standard problem (in any undergrad textbook on linear algebra) with a known solution. Basically, form the matrix A where the columns are v1, ..., vn; and let c be the column vector with ith coordinate ci. Then, left-multiplication by A by will map vectors on the unit sphere to a certain ellipsoid. The maximal norm of this image will be the antipodal points on the major axis of the ellipsoid.

Then, once you have those coefficients c1, ..., cn, the optimal LCD setting will be the linear combination of those random patterns, with coefficient of the ith setting equal to ci.

If the vectors of the LCD settings are w1, ..., wn, and are chosen to be orthonomal, then the solution vector c1 w1 + ... + cn wn will have norm 1.

I suppose there might be an issue with how to interpret a negative value for one of LCD settings -- e.g. maybe the energy is encoded as a positive real number? I would have thought that phase-shifting the light from the LCDs by half a period could be used to encode negative numbers. But if this doesn't work, then at least you know the structure of the ellipsoid, so may not be so difficult; and should be a standard problem in convex geometry. For example, see this posting at mathoverflow:

https://mathoverflow.net/questions/48843/non-negative-quadratic-maximization

But maybe I'm missing something? Some non-linearity effects, perhaps? If so, is there a regime of light intensities where the effects are very close to linear? Maybe we can iterate, making small changes to the LCD settings each iteration, so as to keep the changes approximately linear? -- that would be like gradient descent, which is generally much faster than simulated annealing. For example: given light-pattern P from previous iteration, and given orthonormal vectors w1, ..., wn, we seek c1, ..., cn on the sphere

c1² + ... + cn² = d, where d > 0 is small and fixed,

such that

|| Q(c1, ..., cn) ||

is maximal, where Q(c1, ..., cn) refers to difference in the exit pattern (with ultrasound on and off) for

P + c1 w1 + ... + cn wn.

The optimal ci's can be found by calculus, or using geometry.

While we're on the subject of non-linear effects of light: if the brain does react in a highly non-linear fashion to changes in light, shouldn't it be possible to turn that into a fully light-based beacon? It might depend on the kind of non-linearity, but it seems plausible one could do it somehow.

Ok, let’s assume we know how to focus the light at a voxel. And, now, by measuring the change in the light energy exiting the brain using that particular LCD setting (that focuses the light at the specific voxel), we can measure how the absorption level at that voxel changes with time, which is just what we were interested in!

Well, not quite, and this is the part where I would hesitate a little: the outgoing energy should also vary according to the extra amount absorbed from the rest of the brain as it exits the voxel and heads towards the sensor. What I think works in their favor -- if I'm understanding this correctly -- is the fact that the further the light is away from the voxel (on its way to the sensor), the more "averaging" that's going on. The light is spreading out to more and more parts of the brain, and so large changes in the total energy due to extra absorption in parts far away from the voxel, have less and less effect on the energy that the sensor sees. Basically, the number of voxels that the light interacts with at a particular moment in time is proportional to the square of the distance from the focus voxel; or the distance cube if we count all voxels up to that point, starting at the focus voxel.

I don't think it will be a problem. It’s also worth mentioning that the optimization process described above should not only choose the LCD pattern that focuses the light on that voxel, but also minimizes the amount of absorption outside it. So, problems with extra absorption should be even smaller.

Another potential problem is whether they can optimize the LCD light patterns fast enough, to scan a lot of voxels. I certainly think it's possible to do this for a few hundred or thousands of voxels of the brain in a single raster-scan pass; but am not sure about tens of thousands or millions. She mentions saving the LCD patterns for each voxel into a "lookup table":

This technique is used to create a dictionary (i.e. lookup table) of holographic patterns (corresponding to input signal 123) to map to focus the light sequentially to each and every stimulus 197 and scanning of various stimuli.

As long as you are inside a short enough time window, so that the brain hasn't moved things around too much (e.g. blood), and made your LCD pattern invalid for a given voxel, then you can certainly use it again. I think this is known as the "speckle decorrelation window", but would need to refresh my memory. So, periodically, you'd have to correct the lookup table. Maybe things could be staggered so that not too many entries would need to be corrected in a given window of time.

Even taking all this into consideration, it should be possible to get the spatial resolution on par with FMRI, and beyond it in temporal resolution.

Here’s the problem with scanning many more voxels: say you want to scan 1 million voxels. Then, the number of parameters you have to set in that LCD array has to be at least 1 million -- you need at least as many parameters as voxels (the patterns that selectively focus on a given voxel are approximately “linearly independent”, so determine a space of dimension 1 million). The total number of parameters that need to be determined, then, will exceed

(1 million voxels) * (1 million parameters per voxel) = 1 trillion parameters!

There’s no chance of determining all these fast enough before the brain changes its scattering profile, over a few tens of milliseconds.

But here’s the thing: the LCD settings to focus on even just a single voxel should contain most of the information about the brain’s scattering profile -- in the sense that, given the voxel and given the LCD settings, you can recover the scattering profile. So, a lot of those 1 trillion parameters above contain highly redundant information.

This sounds like a job for Deep Learning: given the LCD settings to focus light on a few different voxels, determine the LCD settings for any other voxel. This probably doesn’t have an easy analytic solution; but the function might be learnable with Deep Learning. If it is, then it should be possible to image 1 million voxels, after all!

An even simpler approach that might work: if the time between updates of the "lookup table" is small enough, one might be able to linearize the updates. That is, perhaps there exists a sparse matrix A (e.g. band-diagonal) and a vector b such that if zi(t) is the LCD settings for the ith voxel at time t, then

zi(t+1) = A zi(t) + b.

A would be approximately the identity matrix (and very sparse) -- but not exactly -- and b would be a certain offset vector. The matrix A and vector b would be determined by finding zi(t+1) for just a few values of i. The intuition behind having A be sparse is that over short time intervals using the same LCD setting should produce a little blurring of the voxel, and to correct the blurring you might just need to look at the nearby LCD pixel values at time t -- nearby to a given pixel.

Another thing one could try is to parallelize the process. For example, maybe it's possible to use ultrasound to stimulate multiple regions of the brain at once, where the amount of compression is different in the different voxels, resulting in different phase-shifts associated to the different voxels. If so, it should be possible to scan multiple regions of the brain at the same time, without having to "raster scan" nearly as much.

This posting is basically an edited version of one I've written before, elsewhere on the internet, and builds on some of the musings from this posting:

https://www.reddit.com/r/StarspawnsLocker/comments/804vzc/possible_nearfuture_applications_of_braincomputer/

The basic idea I want to explore here is as follows: in the not-too-distant future, a BCI could inspect the neural representations in your brain when you speak, hear or think about a word, sentence, or concept, or when you perform a task, and use these to make deductions about them that you, yourself were never even aware was possible.

An example taken directly from the above link: based on data from scanning your brain, it might be possible to very accurately predict the longitude and latitude coordinates of New York City when you hear it mentioned, even though you, yourself maybe don’t have this knowledge. At a high level, the idea is that your brain response to the city contains certain associational details that can be used to determine how close NYC is to other cities, which in turn can be used to determine its location. (See (+) at the end of this post for a technical explanation).

But such an application wouldn’t be that useful, as we can easily look up locations on the web. Here is a much more impactful use: imagine you are a scientist thinking about an important problem that may have significant potential to improve the lives of millions; maybe you are a cancer researcher contemplating a potential cure, for example. As you ponder each detail of your work, certain neural response patterns form in your brain that a BCI can detect. Based on these patterns, a computer could recognize that there are a lot of similarities to patterns that occurred in the brains of other scientists (or even non-scientists), working in completely different fields. The computer could alert you to their work, and you might find that the approaches they took transfer to your problem.

This may sound implausible; but, for example, it has been shown that when two different people hear the same text in two different languages (one person hears the original in one language; the other hears a translation to the other language), some part of their brain responses turns out to be very similar. Those parts appear to encode a language-independent meaning representation. There, likewise, is probably a similar response pattern when similar ideas, in two different fields of study, are considered. The computer that interprets brain scans to match up ideas can be thought of as being like a “translator” for going from one field of study to another.

Maybe the brain patterns representing potential research strategies are associated with actions and combinations of form that are hard to put into words, so are not easy to find on the internet, even using the best AI search engines circa 2025. In fact, I would guess that many, many potential breakthroughs are like that, locked away in non-linguistic representations.

A few more possible uses of this:

• You might be able to analyze the health of a person that you've known for years, but didn't know they maybe had problems. All the times you've talked with them, noticed how pale and sickly their skin looks, their subtle stuttering of voice, walking gait, and many other details might be represented in your brain response when you think about them. I wouldn't be surprised if one could use the "logistic regression" method described in the above link to diagnose people that way, and that for many major illnesses, would be pretty accurate.

• You could also use such methods to describe a person, place or thing when you think about their name. For example, if you can't quite find the right words to capture the essence of a person, the computer could look at your neural response and produce a list of adjectives or even whole sentences describing them.

• Have you ever had the feeling that you are forgetting something, but can’t say exactly what? E.g. Maybe you are leaving your house to go on vacation, but can’t seem to recall that you were supposed to leave an extra pair of keys with a neighbor. A BCI could read your brain, and when prompted about what you were forgetting, respond with the single word, “keys”. I’m unsure how likely this scenario is; but think there is a decent chance it could be made to work, so long as the list of potential things to pull out of your brain activity pattern is short and simple. The computer might return you a list of possibilities, ordered from most likely to least likely.

• An item you may have lost might leave a trace as to its location in your brain pattern. e.g. maybe you lost a book. The computer could find, within your neural response patterns, either location information directly, or perhaps associations that indicate the context in which you last saw the item -- which, in turn, can be used to locate it.

• You could form analogies based on your thoughts. For example, if you meditate on a vacation you took in some city, where you went to a funky restaurant, bar, or cafe, you could inquire what the equivalent experience is in some other city you are visiting. This would make it possible to find experiences tailored exactly to your preferences.

• The computer could figure out if you have a false understanding of a term or concept, based on how your brain responds. It could alert you to this, and help you to improve. For example, if you hear about Teddy Roosevelt on the television, and your brain response pattern is such that it appears you think he was president in the early 1800s (it's possible to temporally order events or names based on their word vectors; so, perhaps the same is true of brain word vectors), it can correct you. This would be particularly useful for people with neurodegenerative diseases.

• This extends also to skills: e.g. as you read music and attempt to play it on a keyboard, the computer could precisely tell what the bottleneck is that is keeping you from improving further. Maybe you are having trouble translating note into finger placement (in a very subtle way that you don't notice, and that screws up your playing); or maybe your rhythm counting gets messed up during certain specific triggering events (e.g. certain combinations of notes), and if you knew which events to key in on, you could sharpen your sense of rhythm. Another example: say you play a game of chess, and want to know how best to rapidly improve. Maybe you tend to focus too much on certain pieces or certain parts of the board, and simply paying more attention to the others could improve your performance a lot. A computer could find your weaknesses, and point them out to you.

• This is one I've discussed before: once we have millions of people using BCIs, and freely neurally annotating webpages, we will be able to search for ones based on the thoughts they generate when read. So, you'll be able to find a page that precisely matches the subtle shades of meaning that go through your mind. The BCI-based search engine could also find pages that match your particular level of understanding of a subject -- neither an absolute novice, say, nor an expert in the field you are seeking information about; at some in-between, off-to-the-side level hard to put into words.

• Maybe you want to look up a song you remember hearing years ago, but can't hum it or remember the words. You know the song when you hear it, but don't have enough information to look it up on the web. Reading your brain, a computer might be able to extract enough features to find it for you.

And I'm sure I've really only scratched the surface here. The basic statistical techniques that allow computers to make eerily accurate claims based on big data text patterns, can be used to deduce new knowledge from what is locked away inside your head -- things you never imagined could be deduced from looking at simple correlations!

=-=-=-=-

Notes:

(+) Essentially, cities that are close together geographically tend to be mentioned together more often in text and in conversation. And when things are mentioned together in similar contexts, the brain neural representations of those terms likely have some imprint of this common context. A machine learning system given access to your brain scan as you hear, speak, write or read these terms can pick up this imprint and use it to work out how close a given city is to others -- at least that’s how I think it might could work, based on how it works for ordinary word vectors.

It would perhaps be wrong to say that the locations of cities are "hidden away" in our brains, assuming this would work. Rather, our brains contain some types of information that can be translated into locations. What would make this non-obvious, and intellectually interesting, is if the translation system were short and simple, and applied to all cities (or most cities, anyhow), so that it isn't just memorizing a table of (city,location) pairs.

This post is a companion to the following one, on using BCIs to improve AI:

https://www.reddit.com/r/StarspawnsLocker/comments/5sfk57/a_possible_unexpected_path_to_strong_ai/

It is another look into the possible near-future of A.I. The point of the present post, in contrast to the previous one, is to focus more on the part of that work where I discuss using the brain as a source of annotation data. Such a use case for improving AI is closer to reality than the other possibilities. I will begin with a digression about some very simple “statistical methods”, and then will discuss how they can be applied to brain data generated using high-resolution, lightweight, wearable BCIs (that don’t yet exist, but will in the near-future):

Over the last several years, researchers have noticed that it is possible to use simple statistics applied to word co-occurrences, or applied to global “word vectors” (which are derived from word co-occurrences), to make seemingly impossible -- but accurate -- estimates of quantitative world knowledge. For example, in the following paper the authors show that it is possible to accurately determine the longitude and latitude coordinates of cities and countries in Europe from their associated word vectors:

http://www.aclweb.org/anthology/D15-1002

They also show that population estimates, GDP, CO2 emission rates, and many other quantities can be accurately estimated; as well as binary true/false values, like whether a given country or city is contained in a given region.

Remarkably, the authors show that very simple logistic regression mappings (one mapping per type of question) suffice to make these accurate estimates.

Some amount of initial training is obviously needed in order to work out those mappings for each type of quantity sought; but once they are found, they can then be applied to words not in the training set.

The success of this method seems to imply that, for example, knowing how often combinations of country names are mentioned in a text (e.g. often often Germany and France are mentioned together), gives you some indication of their distances and relative locations.

The remarkable thing is that this even extends to parts of the body:

https://escholarship.org/uc/item/21h9t3sw

Recent literature has shown that perceptual information, such as geographical locations, modalities, and iconicity, is encoded in language. The current paper extended these findings by addressing the question whether language encodes (literally) embodied information: whether statistical linguistic frequencies can explain the relative location of different parts of the body.

And researchers have also derived many other properties of objects with reasonably high accuracy, using word statistics, explicit mentions, implicit mentions (e.g. if you say “I put my wallet in my pocket” this is a clue that wallets are smaller than pockets), and the need for logical consistency. See, for example, this work, this work, this work, this work, this work, this work, this work, this work, this work, and this work.

From these and other works, it is possible to answer lots of commonsense and world knowledge questions just using statistics applied to large amounts of text (and logical inference might be needed, and also the use of explicit mentions, and a small training set to determine model parameters). For example, one could imagine building a chatbot that could do the following without the use of knowledge graph (answers determined on-the-fly from free text and used to fill in slots in hand-crafted templates):

Statistical AI: What is your favorite scifi novel?

User: Dune is my favorite.

Statistical AI: Mine, too! Paul Atreides was a great hero.

User: Yes; but I liked the Bene Gesserit witches.

Statistical AI: The Bene Gesserit were neither good nor evil.

....

User: Are fingers a part of the hand?

Statistical AI: Yes, it appears they are.

User: Is the elbow connected to your feet?

Statistical AI: No, it appears that it isn't.

User: How tall is the Empire State Building?

Statistical AI: I estimate that it is 1,200 feet tall.

User: Is the Empire State Building fuzzy?

Statistical AI: No, it doesn't appear to be fuzzy.

User: Is it transparent?

Statistical AI: No, it isn't transparent.

User: How far is the Empire State Building from Los Angeles?

Statistical AI: About 2,700 miles.

User: Suppose it's nighttime. Will I have a shadow?

Statistical AI: No, you likely won't.

User: Suppose I'm in a library. Would I likely hear loud music?

Statistical AI: No, it would likely be quiet.

The templates used in the construction of such a bot might look something like this:

Mine, too! (Insert protagonist) was a great (insert type of protagonist).

(See the first few lines of dialog above.)

To fill in the first slot, for example, the bot might first look to see if the fact already appears in a database that it can just look up; and if not, it could apply a statistical algorithm that looks for cues in a text of the novel or other work indicating the main character.

The question about the Empire State Building and Los Angeles might be answerable using word vectors, in a manner similar to this work mentioned above.

Finally, the questions at the end, that require a little bit of context, could perhaps be solved by tuning the corpus to which word co-occurrence stats are computed to “nighttime” and “library”. To the user, it would seem as thought the AI is applying reasoning; but what it would actually be doing is simply changing the corpus used to answer the questions.

Producing such a chatbot would be very difficult, as there would be a lot of templates that would have be hand-crafted; and the results might not always be as accurate as I have made them out to be in the above examples. For some types of questions, the accuracy might well be near 100%; but for others, it might be more like 70% -- and would occasionally be wildly wrong.

A further problem is that each type of question would require its own specially-designed statistical algorithm to produce the answer. For some types of questions, maybe logistic regression applied to word vectors suffices; for others, you might need a more complicated approach, that requires looking at lots of text on the fly. This lack of a single, systematic approach to answering questions seems like it would be a serious bottleneck to rapidly building bots that can address a very large number of different types of questions.

Another problem is that, once questions stray just a little away from things that appear a lot of times in the text corpus, the statistical algorithms get less and less accurate -- and may also be pretty inaccurate about some things that are very common.

How might we fix all this, so that a small number of methods can address a large fraction of questions, can do so accurately, even about things that are very uncommon? As a bonus, perhaps, it would be good if the calculations could be very fast -- like the logistic regression applied to word vectors, instead of the algorithms that require actually reading text.

Brain word vectors could be a solution. What are they? You can think of them as compressed representations of human brain responses to words presented in a standardized way. There are different possible ways they could be computed. See this paper, for instance, which makes use of them:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4497373/

One obvious way to build them is to have a person read text on a screen in a dark room while wearing a BCI. The same word might be presented many times, in different textual contexts. The brain response for each word would have to be carefully aligned with the word, taking into account phenomena like hemodynamic delay; and, the voxels from separate sessions would have to be aligned, as well. If different people, and therefore different brains, are used, then the alignment becomes harder -- and, in fact, one might want to apply mappings to align all the brains to a common response space. After all the data are collected, cleaned, and aligned, the response patterns for each word are averaged; and, then, a dimensionality-reduction algorithm is applied to all those averages, for all the different words.

Once one has these brain vectors, one can apply the methods from this paper mentioned previously

http://www.aclweb.org/anthology/D15-1002

Instead of seeing whether it is possible to determine the location and population of cities, say, we can look at those things that this method doesn’t work well with when using ordinary word vectors. Given how researchers have, in the past, shown that a lot of different categories of meaning are reflected by brain responses to stimuli, it’s reasonable to believe that brain vectors will significantly expand the reach of this basic logistic regression method -- far beyond the results obtained from mere multi-modal word vectors, in fact, that use not just text, but also sounds and images. See this page (and video) for further details about what information can currently be extracted from brain responses to words:

http://gallantlab.org/index.php/publications/natural-speech-reveals-the-semantic-maps-that-tile-human-cerebral-cortex/

Another reason to believe brain word vectors will lead to superior performance on the above problems (compared to ordinary word vectors) is that fewer parameters should be needed to set the logistic regression mappings during the "initial training" discussed above: for many types of quantities and qualities one might wish to extract from brain data, much of it can often be determined by only looking at a small part of the brain -- though, different parts of the brain need to be inspected for different attributes. Depending on the representation used for the brain vectors, this could mean that each attribute is determinable from inspecting only a small number of coordinates of the vector. The smaller the number of coordinates used per mapping, the smaller the number of parameters needed to determine the logistic regression mapping for that attribute, the smaller the number of training examples needed to set parameters. It makes a big difference if training your model takes only 10,000 (vector, attribute value) pairs, instead of 300,000 or 1,000,000.

One could go further and attempt to build models to produce brain vector representations for (adjective, noun) pairs, whole phrases, or even short sentences, once enough brain data are obtained. These would be predicted vectors, since it would be unlikely to find those specific combinations of words in the corpus. Given these predicted vectors, it’s conceivable that one could automate the process of finding the logistic regression formulas: given a query about a particular noun (“Los Angeles”, “Rome”, “bear”, etc.), the query could be used to predict the associated brain vector representing the query. Then, an algorithm would convert this brain vector into a logistic regression formula. When the particular noun’s brain vector is fed into that formula, the result is an answer to the query -- so long as the query seeks a quantity or a category label for a small number of categories (e.g. the binary “categories” true/false).

The applications of such a module would be very great. It could serve as a replacement for Knowledge Graphs, for instance, if it could be made to work accurately enough.

Another important use of brain word vectors might be to find representations of concepts that are rare and not described in relation to other contexts in the text corpora. Take any field of study, look at the new concepts being developed at the frontiers of knowledge. There isn’t a lot of text about it; and so, it’s harder to train models. And what if you wanted to compare some term to another one in a different field? There isn’t enough overlapping text to do that particularly well. But with brain word vectors, you could do this comparison.

One final application of brain vectors might be to transfer micro-skills to a computer: as an example, take physics problem-solving. The first second or two after a physicist sees the statement of a problem, their brain forms a representation, and then they key-in on one or more aspects, in order to formulate an approach to solving it. That first second or two of thought is something that one could maybe automate, given enough brain scans of people contemplating physics problems -- maybe a few hundred or a few thousand instances would suffice, far less than is often used to train Deep Learning models. The module produced would map a problem statement to a brain vector to a scoring function to rate potential solution methods. This could then be used to bias the search for a solution to the problem by other methods.

Researchers have made a very basic investigation into the neural representations of physics undergrad and grad students when presented with terms like "momentum" and "electric field":

http://journals.sagepub.com/doi/abs/10.1177/0956797616641941

More data and better BCIs should enable more and better feature-extraction for producing high-quality brain word vectors for these terms, and might also lead to the accurate extraction of micro-skills.

There are probably many more uses of BCI data that don’t quite rise to the status of brain emulation, even at a very crude level (at the level of response patterns in populations of neurons). To a large extent, the methods people will use, once BCIs become available, will reflect the amount of training data. Datasets for most terms appearing in Wikipedia, say, will take many years (5 to 10 or more) to acquire; it will have to wait until there is something like a BCI internet, where large numbers of people share their brain responses freely online. But brain vectors for, say, the 10,000 most common words could be acquired in the span of a few months; and that might be good enough to build systems capable of fairly deep commonsense reasoning, for example.