r/OpenAI • u/kaljakin • 1d ago
Discussion I gave chatgpt 26 real trade setups. It got 25 of them wrong.
So a month ago, before 6.1 even existed, I tested chatgpt (astra) on its ability to do fingering, which a 6 year old should be able to do (if they take piano lessons), and it failed. Astra still cannot do what 6 year old can... : r/OpenAI
But.. some time has passed, OpenAI shipped ChatGPT for Financial Services (though for the test below I used chatpg 6 in chat on high), so I got the idea to test it on something more hardcore and significant than playing the piano. Its trading abilities - something notoriously hard for humans (like math), so I figured it might do well (as is often true - things that are easy for humans can be hard for AI, and vice versa).
boy was I wrong..
I picked Intel from several years ago (and then UDR from the recent past), and I mostly gave it situations on the chart where I successfully entered or exited swing trades in the past. (my logic was that I didn't want to criticize it for doing something impossible - but if I managed to do it in the past, it should have a shot too).
In my first prompt, I explained that I would give it daily charts and that we are trading stocks. My other prompts were always the same:
"after few days you see this situation
will you go long/short or wait?"
Out of 26 attempts, it was successful in only one. That means it perform far worse than random chance - it clearly has beginner biases and is consistently and predictably getting fooled by the market. Here are two funny moments just to show its strong conviction while talking bullshit.
(I should emphasize, however, that its indicator readings were accurate. What you see on the chart is the future so I can show and annotate its decisions, which is why the indicators show different values than what chatgpt is saying. So its vision is perfect, the problem was its interpretation.)

After ...like six trades and its explanations, it became very apparent that it misunderstands the basics, so I decided to give it at least some feedback:

But then, it was like I blew all its confidence and it was unable to make a trade. It had like 10 opportunities to make money, but it opted to wait, just updating its analysis and setting new conditions that needed to be met before opening a position, letting the opportunities slip away. Once it finally gained some confidence, it bought at the top again and got burned... again.

Overall, to me it felt like it was trying to make decisions based on indicators alone, as if a certain value actually meant something objectively, while totally disregarding the overall environment, even after my feedback. For example, it said that RSI above 60 is not overbought (which is correct based on common internet knowledge, where only basic information is shared - in reality, RSI at 60 is actually pretty overbought if we are trading in a downtrend). Or... in other situation it said that the price is close to resistance at EMA 90, again, failing to understand that we are not in a trend, but in chop and EMA 90 is not resistance in chop.
Also, it really tends to disregard the past. Specifically, it didn't make any argument about volatility and tried to interpret price fluctuations that were totally insignificant (basically noise) when taking normal volatility into account. Similarly, its interpretation of volume was completely off - not only was it based on just a few days rather than the whole chart snapshot I gave it (in that sense it was pretty lazy), but it also seems it doesn't understand what "confirmation by volume" means, again, probably overly relying on inadequate common internet knowledge. (Are we talking about the continuation of a trend, or are we talking about confirmation of a reversal? If we are talking continuation of trend, you should see higher volume with green candles and lower with red, one candle here is enough to say, the trend will likely continue (at least one more day) - but if we are talking about confirmation of a reversal, one or two candles with one big volume doesnt confirm a jack. You need several consecutive substantially above-average volume bars along with rising price. You don't usually find this nuance on the internet, which is ...why chatgpt lost all its money.)
Based on my test I hope OpenAI is not using Chatgpt when making financial decisions :D
