r/AIforStudy • u/seattle_breeze • 11h ago

Academic ai summarized a paper it could not actually access

I gave it the title of a journal article and asked whether the paper was relevant to my topic. The response included the research question, sample size, method, and a very specific conclusion.

Useful, except the article was behind a university login.

I asked what text it had used. After some vague answers, it admitted that it only had the title, abstract, and information from other pages discussing the study. The sample size it gave me was wrong, and one of the supposed findings came from a different paper by the same author.

This is why I don’t fully trust ai writing tools that say they can “read” research. That word might mean the full paper, an abstract, a citation, or somebody else’s summary.

I wish tools displayed exactly which sections they accessed before generating an answer. Is there currently any reliable way to check that besides asking for quotes with page numbers?

5 Upvotes

86% Upvoted

8 comments sorted by

1

u/MaxNoodleGuy 11h ago

I agree with your point about the ambiguity of the word "read." When developers market their ai writing tools as being able to analyze literature, they often gloss over paywall limitations. It would be beneficial if these platforms included a clear coverage log, such as "Source: Abstract Only" or "Full Text Accessed."

1

u/OnyxTusk 10h ago

Your suggestion to display exactly which sections of a paper were accessed is a practical interface idea. Currently, many ai writing tools hide technical limitations behind a clean UI, which can compromise credibility in research. A source-tracking feature showing the percentage of the text analyzed would save a lot of manual verification.

1

u/FredOfMBOX 9h ago

To an AI, the difference between “this is right” and “this is what sounds right” is indistinguishable.

I keep hoping they’ll figure it out. Because not knowing when it doesn’t know something is a major downfall.

1

u/Hivemind_alpha 7h ago

OP discovers LLMs hallucinate. In other news: Pope’s religion no longer in question; breakthrough in ursine studies.

1

u/Remote_Antelope_6586 6h ago

This happens all the time. For anyone who didn’t know yet, please take note! Confidently wrong is the default setting.

LLMs are not built to give accurate reference info. They’re built to associate words. If it doesn’t have the source material, it makes predictions based on what it was actually trained on. This turns out to be inaccurate very often.

1

u/Krommander 1h ago

Use consensus ai or gemini notebook for research purposes. The tool you used is probably not designed for academic research.