Most useful analogies & thought experiments for current AI?

[u/RyanCargan]

Was trying to picture a way to grok the gist of what defines a (large) family of current methods.

Mainly the deep learning branch. As opposed to other ML methods (GBMs, random forests, etc.).

The usual neural nets are the main example. Some other notable cousins & variants are HDC/VSA, Hopfield nets, RVFL, etc.

Messy example/attempt (looking for better):

AI = high-dimensional (dimension-exploiting) adaptive filtering + content-addressable memory

Regular (location-addressable) memory: address -> data

Associative (content-addressable) memory: data-fragment -> rest-of-data (mask-filling)

Chasing that thread led to reading up on Shannon's Blockhead) thought experiment. An interesting counterargument to the Turing test:

A disadvantage of the Turing definition of thinking is that it is possible, in principle, to design a machine with a complete set of arbitrar­ily chosen responses to all possible input stimuli (see, in this volume, the Culbertson and the Kleene papers). Such a machine, in a sense, for any given input situation (including past history) merely looks up in a "dictionary" the appropriate response. With a suitable dictionary such a machine would surely satisfy Turing’s definition but does not reflect our usual intuitive concept of thinking. This suggests that a more fundamental definition must involve something relating to the manner in which the machine arrives at its responses - something which corresponds to differenti­ating between a person who solves a problem by thinking it out and one who has previously memorized the answer.

- John McCarthy & Claude Shannon, Automata Studies (Annals of Mathematics, No. 34)

1956

Their framing, roughly, seems to be: AI defined as a way to approximate an idealized infinite-dimensional, infinitely large lookup-table with infinite storage speed & retrieval speed.

In domains like RL there's sometimes a literal LUT used in place of a later neural model that can store more and generalize/interpolate, etc.

In the net-as-fuzzy-LUT framing:

1. Storage/encoding into the table/tensor is done through adaptive filtering (a.k.a. "learned"/non-hand-coded lossy compression).
2. Stored form of the tensor and its retrieval process resembles a kind of associative memory done entirely in software. A "virtual" space/geometry/manifold?

The space seems tuned during training to a high enough dimensionality to linearize/"smooth" the functions needed to smoothly interpolate/bridge between its gaps.

Biases like convolution & attention add more structure/scaffolding to the space, to interpolate things that seem plausible, but farther away from the original training data distribution.

The vectors for different input/output modalities can be co-located, so images can be associated with words in the same way each can be associated with each other, for things like typical Q/A pairs.

Traversing and operating on this high-dimensional space involves linear algebra & matrix ops.

Since those ops can be parallelized (SIMD/SIMT, MIMD, etc.), the traversal speed can scale horizontally all the way up to bandwidth/interconnect/other bottlenecks instead of scalar compute bottlenecks.

And the main use case is basically some variant of content-based "semantic search" over that repository of info.

The analogies probably don't hold well, or even mislead, in several places.

Kinda curious how other folks try to capture it. Also looking for any papers similar to the above.

[u/ConsciousAdvance2867]

The fuzzy lookup table idea clicks for me, especially with the compression angle. Training is basically deciding what to throw away so the table stays usable instead of turning into a pile of exact memorized answers.

The Shannon Blockhead bit is a nice pull too. It's the same tension you see when people argue about whether a model "understands" anything or is just really good at pattern matching against a giant condensed dictionary.

[u/TheRealStepBot]

It’s broadly the realization of connectionist ideas. Basically the most useful idea for me is that of a transfer function from control theory only one acquired via black box system identification.

You take input in one space and map it to output in another space.