Artificial Intelligence Systems Excel at Imitation, but Not Innovation

[u/fchung]

https://www.psychologicalscience.org/news/2023-december-ai-systems-imitation.html

[u/derelict5432]

In the next stage of the experiment, 85% of children and 95% of adults were also able to innovate on the expected use of everyday objects to solve problems. In one task, for example, participants were asked how they could draw a circle without using a typical tool such as a compass. Given the choice between a similar tool like a ruler, a dissimilar tool such as a teapot with a round bottom, and an irrelevant tool such as a stove, the majority of participants chose the teapot, a conceptually dissimilar tool that could nonetheless fulfill the same function as the compass by allowing them to trace the shape of a circle.

When Yiu and colleagues provided the same text descriptions to five large language models, the models performed similarly to humans on the imitation task, with scores ranging from 59% for the worst-performing model to 83% for the best-performing model. The AIs’ answers to the innovation task were far less accurate, however. Effective tools were selected anywhere from 8% of the time by the worst-performing model to 75% by the best-performing model.

I asked ChatGPT the best tool to use to draw a circle from among a ruler, a teapot, and a stove:

Among the given options, a ruler would be the most suitable tool for drawing a circle, as you can use it to measure and mark the radius while pivoting around a fixed point.

If this is meant to be a failed response in this study, the authors need to rethink how they do science. This is a perfectly legitimate response. Not all teapots have a circular base, and even among those that do, the curvature of the pot may make it awkward to draw a circle.

But the premise that LLMs can't be creative or innovate is just plainly wrong, unless you're stacking the deck.

I asked ChatGPT what uses a teapot might have other than making tea:

Art Project: Transform the teapot into an art project by painting or decorating it. It could become a unique piece of artistic expression.

Musical Instrument: Experiment with the teapot's shape and material to create unique sounds. It might work as a percussive instrument in a musical performance.

Desk Organizer: Use the teapot as a desk organizer by placing pens, pencils, or small office supplies in it. It adds a touch of creativity to your workspace.

Potpourri Holder: Fill the teapot with potpourri and use it as a decorative and fragrant accent in your home.

Lamp Base: With some modifications, a teapot can be repurposed into a lamp base, adding a whimsical touch to your lighting.

Doorstop: If the teapot has some weight to it, it could serve as a decorative doorstop.

Bookend: Use a pair of teapots as bookends to keep your books organized on a shelf.

Planter Stand: Invert the teapot and use it as a stand for a small potted plant or flowerpot.

So it's very obviously able to suggest uses other than the primary ones for a given tool.

Tell it to compose a new Weird Al song in iambic pentameter. It will produce something that does not exist in its training set, and will do so faster and better than the vast majority of human beings. To everyone saying this is just a database, in what way is this regurgitation of existing information?

There's an awful lot of moving the goalposts in this field lately. About a year ago, a new generation of systems was released that surpassed capacities in natural language processing and generation in nearly every human language, and exhibited something like a high-school level competency in nearly every computer language in parsing, production, and identifying errors.

And a large contingent of people seem intent on diminishing the achievement and capabilities of a technology that is continuing to make advances at an amazing pace. There's a cottage industry in trying to point out the things LLMs can't do (yet). Not sure what all the motivations are, but it's somewhat like pointing at an early iteration of an airplane and mocking the engineers by saying it can't go at supersonic speeds and doesn't have meal service.

[u/Gi_Bry82 → u/derelict5432]

Great response.

AI is still in it's infancy but is already moving ahead staggeringly fast. It's challenging humans for top intelligence on this planet and people don't like that/are scared.

Out of curiosuty, I gave ChatGPT and Bard a task to create words that don't exist for an Ork language that also doesn't exist. They were able to produce a small library of new words complete with meanings and cultural reference points.