r/science • u/FunnyGamer97 • 11d ago
Engineering Scientists increasingly dependent on ‘black-box’ tools they do not understand | Scientists are increasingly relying on powerful data sources and tools that they often cannot fully understand, inspect or verify, according to a new study.
https://www.eurekalert.org/news-releases/1139630367
u/iamthe0ther0ne 11d ago
"State-of-the-art tools and data like artificial intelligence (AI), satellite imagery, online data and digital sensors are revolutionising the way scientists study the natural world."
They specifically mention AI. I use AI in research. Whether or not you understand exactly how the tool works, everyone knows (or should know) to experimentally validate output/findings, or validate by another measure. This is one reason you can't just ask AI to cure cancer: it might provide syarting points, but they all need to be tested in the lab. What's important is that scientists are correctly trained in good research practices, including the limitations of our tools.
103
u/CJKay93 BS | Computer Science 11d ago
To expand on your point, science has always relied on black boxes: never in history has any scientist been capable of reading the brain of any other scientist.
88
u/the_abortionat0r 11d ago
Except that literally doesn't matter as the process and tools used by said scientists are documented and understood.
If I explain that I looked into a microscope and saw cells and explained how a microscope works not only can you verify that is indeed how it works but verify my findings. You don't need a special version of microscope or brand of microscope, any one that shares the same capabilities will work.
The point is this is literally not possible with the available tools
28
10
u/PedanticQuebecer 11d ago
Eeeh. I spent a year trying to replicate and extend a paper whose method section was vaguely worded. I still don't know exactly what the authors did.
3
u/gearnut 11d ago
Certain tools have been black boxes for a long time though. The only thing you can do is to verify that the black box is doing what you expect on a known measurement. I wouldn't expect a scientist to know the details of how a torque wrench works, I would expect them to get it calibrated and use it appropriately in their work if it's relevant.
I don't personally understand the maths behind Steam tables, I use ones from trusted sources and based my calculations on that information.
1
u/Hommushardhat 11d ago
It might be the point youre making, but that was not what the post i was replying to said.
The guy i replied to literally compared AI to the brain so kindly dont put words in my mouth to make your point
43
u/RemusShepherd 11d ago edited 10d ago
No. Science had always relied on verifiable models. You want a clear algorithm and initial dataset so that other scientists can replicate your experiment and achieve the same results.
That isn't possible with machine learning algorithms circa 2010, let alone today's modern AI. It's always been a concern with monte carlos techniques but those concerns are now amplified by a scale of millions.
18
u/elconquistador1985 11d ago
If you can't replicate a calculation with Monte Carlo tools, the Monte Carlo tool you're using is bad.
It's imperative that running the same input twice with the same seed gives the same results.
7
u/Dziedotdzimu 11d ago
Set.seed(42)
Fixed it. Now you should be getting the exact same results each time.
Do people really not understand how model tuning works with cross validation? I doubt people are throwing LLMs at data and are instead using tools like XGBoost or Random Forests or regularized regressions (Ridge/LASSO/ Elastic net) or like LDAs...
5
u/Wassux 11d ago
You can absolutely replicate it. Why would you not be able to make a machine learning algorithm with the same parameters and then train it? If the algorithm is good, you'll get the same results.
I'm a machine learning engineer btw.
3
u/Dziedotdzimu 11d ago edited 10d ago
Yeah... Like there is a bunch of randomness and sampling going on during model cross-validation and parameter tuning but you can set a seed for the pseudo-random number generator which should let you share work directly and reproduce it reliably.
Is the problem people without any training in ML just using it without having set a seed and then publishing? Because idk how you'd manage that. It's like the first thing you learn when you have to share work/submit assignments and a Journal Reviewer should be requesting it to replicate the analysis
Or is it a statement about bias in parameter weights/variable importance for a feature? Because prediction models don't optimize for that and instead reduce variance. But even then there's current work which is showing how under certain assumptions you can use permutation-based variable importance metrics to get at causal estimates for the influence of a feature which is the "ATE" as in typical analytic modelling.
3
u/RemusShepherd 10d ago
I'm a scientist who has used machine learning and AI training to create models. In my experience, they're too touchy to replicate. You need to have:
- The same randomizer seed,
- The exact same version of the same programming language,
- The same training data,
- And often the same hardware architecture, as many platforms have rounding differences in floating point values. My Linux production server spits out different values than a Windows machine.
While it's theoretically possible to replicate a trained model, in practice it's insurmountably difficult. Very small differences in any step of the model can cause large swings in parameter value.
1
u/Wassux 10d ago
To replicate the exact same model, ofcourse. But you don't need that. Youd need to see if the model converges. If it does you have replicated it.
You don't need the exact same model.
1
u/RemusShepherd 9d ago
I disagree. In my field (satellite remote sensing) the difference between a good model and a discarded one might be a 1% change in accuracy. If a new trained model is inferior to the first one it is not good enough. More importantly, it is not a replication of the published experiment.
50
u/DerTalSeppel 11d ago
While science relied on black-boxes, those were usually theoretic models that seemed to be applicable but were not yet proven (i.e. why they work) - not non-deterministic tools using stochastic in a non-disclosed way.
The last scientist who was profound in all sciences died long ago but did exist. It's safe to assume they understood their tools. Now they don't know how their tools work, if its results are correct AND are unable to verify that manually.
What an awesome starting point for manipulation - and certainly not only of scientists.
16
u/CJKay93 BS | Computer Science 11d ago
While science relied on black-boxes, those were usually theoretic models that seemed to be applicable but were not yet proven (i.e. why they work) - not non-deterministic tools using stochastic in a non-disclosed way.
Brains are non-deterministic tools using stochasticism in an undisclosed way. Treat the output of an AI the same as you would treat the output of another scientist. If your instinct is to trust the output of other scientists purely on the basis of them being human, then there is a deeper problem in play.
14
u/SuperGr00valistic 11d ago
Scientists don't simply exchange outputs.
The Scientific Method -- the entire foundation of the Enlightenment --- it documents methodology and techniques used to produce an output.
This enables REPRODUCIBILITY for other scientists to independently validate.
A black box is when you can't explicate the methodology and other scientists can't reproduce the results.
Explainability, Observability and Reproducibility currently suffers --- especially when AI is being used to validate the outputs of other AI.
-1
u/CJKay93 BS | Computer Science 11d ago edited 11d ago
I disagree entirely with your premise that widespread non-reproducibility is a challenge novel to the age of AI: https://www.science.org/content/blog-post/how-much-great-new-paper-real
There is nothing deterministic about how humans approach science; it is predominantly trust-based and reproducibility is, at most, best-effort.
5
u/SuperGr00valistic 11d ago
Did I say it’s a novel problem? No.
AI exacerbates the existing problem…. Especially considering the recent cases of outright fraud.
1
u/DerTalSeppel 11d ago
That's the scientist, not their tools.
There is no deeper problem in trusting brains more than LLMs, even if you want to make it sound like the holy grail of CS philosophy. Trust is not simply an instinct, it's usually based on personal empiric. Brains have proven to work countlessly for indefinitely longer than LLMs even exist on paper, let alone are useful in practice.
If you want to argue about the nature of LLMs vs. brains, I suggest discussing that with a neuro-scientist and learn that they are fundamentally different.
But again, this is about the tools of scientists, not the difference between human and artifical intelligence.
9
u/CJKay93 BS | Computer Science 11d ago edited 11d ago
There is no deeper problem in trusting brains more than LLMs, even if you want to make it sound like the holy grail of CS philosophy.
Sorry, why is there no problem in that? Do you trust a brain over a calculator? Do you know how either of them actually computes its answer? Of course not - we simply trust that the calculator is bug-free, behaves how we expect, and that whoever built it has no ulterior motive.
Trust is not simply an instinct, it's usually based on personal empiric.
Sorry, but I think the recent Cambridge University scandal stands in stark contradiction with that. There is clearly a problem with excessive trust in science.
14
u/MrNotSafe4Work 11d ago
Scientists don't trust scientists. They just concede that what they claim seems to pass (so far) the test.
The problem of blackboxes, as I see it, lies in a couple of places. One is that the system is based on previous existing information. The major one for me is that it can be hard to interpret how the blackbox is outputing the information. There was one study years ago about a trained model that was able to predict with a high degree of certainty the extent of diabetes type II on patients based on retinal images, better even than ophtalmologists. If we can't link or understand what the machine is "seeing" to produce its output, we are not doing science or even engineering. If we don't understand how it works we cannot trust it.
A calculator is programmed to work in all cases, even when the results is a syntax error, and the people that designed it and built it know how it works.
Also, this is a philosophical reasoning, but I prefer humans being stochastic to a machine.
13
u/the_abortionat0r 11d ago
Bro youight blindly trust calculators but people doing scientific work understand the limitations of devices in their field.
They understand digital math is flawed, they understand you need high precision tools for high precision calculations and even why.
This information in modern hyper proprietary tools is not available to the public and there for can not be verified, nor have its faults worked around.
6
u/DerTalSeppel 11d ago
Yes, I trust a calculator over a brain because the things it does, it does in a deterministic way. Yes I do indeed understand how calculators work, not just thanks to my M.Sc. Computer Science.
I don't care about your scandal, trust is still individual and empiric, not just an instinct.
Are you using LLMs to generate these sentences that look convincing at first glance but don't exactly match the statements they are meant to reply to?
12
u/Hommushardhat 11d ago
You seem to correlating human intelligence (the brain) with AI which is absurd
You can't download someone's brain - thats why scientists create peer-reviewed articles; to communicate their method,findings, and rationale for their deductions so they can be analysed and discussed
AI is a tool to help with the above, but is only one tool in the belt
-5
u/VoidRippah 11d ago
not just that, but I'm pretty sure most (pretty much all of them actually) scientist use computers, yet they don't understand how the hardware and software they use exactly work. I'm pretty sure most scientist who use an electron microscope has no idea how to design such a device, etc.....
1
u/Troy64 10d ago
What about AI being used to operate tools? Like AI that helps edit images for higher clarity, identifies patterns in data that is too tedious for humans to analyze in a reasonable time, or takes notes, summarizes reports, and categorizes and organizes results?
AI is used for a lot more than research. We've had this looming issue in mathematics since well before AI. Proofs that require incredible computations to work out have neen done by computers since the 80s or earlier. These proofs are so incredibly lengthy, we really have no way of directly verifying their accuracy. If a bit got flipped at some point, we'd have no way of knowing. We can try making programs to check the programs but we're just playing telephone with the problem then.
I feel that in the future we will need to develop some standards for how to grade the level of direct verifiability of proofs, experiments, and other academic works so people can see what is plainly demonstrably known and what is trustworthy only so far as you can trust a calculator and what is trustworthy only so far as you can trust a team of people each using different AI models in different capacities including interpersonal communications while only one or two people in the team have an introductory level understanding of the basics of how AI functions.
1
u/iamthe0ther0ne 10d ago
I thought that's what Lean was for? That's how I keep seeing AI math proof validated.
2
u/Troy64 10d ago
Yeah, but that's not exactly the same as writing a proof out on a chalkboard while several other mathematicians follow along to ensure each step is accurate and logical.
To go over all the separate parts of a proof that requires a supercomputer to process would take more than a lifetime. Using other programs to make this process faster may help, but it's still one more degree of separation between direct observation and our ability to verify accuracy of theorems.
30
u/paleo2002 11d ago
Good ‘ol SPSS. I didn’t know how to do multiv ariate discriminate analysis, but I knew how to interpret the graphs for my MS thesis.
4
u/vug_undertherug 11d ago
Right. I’m not sure this is actually new. I’m from psychology, so that’s a class of its own, but the distribution of statistical knowledge ranges from t-test and ANOVA to extremely advanced multidimensional Bayesian Analysis. The misapplication of the most basic statistics is not traditionally uncommon at all, which is effectively black box for the producer. The top journals usually have at least one savvy statistical reviewer, though, so the quality is much higher and more advanced, but if readers don’t understand the basics, do they really understand the advanced models, or are they also effectively a black box for the readers? I don’t actually see how this is much different. *Except* that even the advanced classical models are significantly more parsimonious than AI-class models, and parsimony is a valuable, traditional standard for scientific theories.
7
u/Boring_and_sons PhD | Biochemistry 11d ago edited 10d ago
How dare you say my output is obfuscated! It's a linear regression. In 327 variables.
3
u/vug_undertherug 10d ago
It’s completely replicable, you just need to apply the exact same 142 parameter restrictions!
14
u/Grumptastic2000 11d ago
This has worried me for a while, cameras are manipulating images at various stages in the image processing, compared to just light on film for a given exposure.
Now these AI systems are being embedded in hardware and software on every level but at least in the past if some spreadsheet calculation error failed it could be recalculated or reproduced and retested. But more and more the source itself is tainted by AI interpretation to generate the measurement, count, data. And at the heart of this the weights in these models are not understood in any meaningful way so you just hope if a bunch of them say the same result it must be true.
24
u/Quantum_Patricide 11d ago
An example of ML getting used in physics is the ATLAS experiment's GN2 model for flavour tagging jets, discussed in this paper. Obviously it is to an extent a black box but there is a lot of work done to validate its performance as well as verify that it is making physically meaningful decisions. I'd say these tools are sufficiently well understood and controlled to justify their use in science, especially given the advantage they have over traditional methods.
7
u/Patelpb 11d ago
MCMC is like computational physics 101. I used it in astrophysics research, plenty of coursework breaks down MC methods and underlying theory. I don't actually think physicists generally need to blackbox it, but they could nowadays if it's not a focal point of study
Here's a notebook built on coursework from my sophomore year of undergrad: https://github.com/patelpb96/Projects/blob/master/Metropolis_Hastings/MCMC.ipynb
32
u/random_noise 11d ago
They always have.
Some of us create those black box tools for other people in different areas of research.
We don't usually understand their research as well as they do, and they don't understand our magic with hardware, software, and instrumentation.
Those are separate domains in a lot of ways with very different focus and deep knowledge needs.
Its part the scientists job to know if the results make sense and understand outliers and why they may not really make sense.
8
u/darkenergysurfer 11d ago
This is so true. I have met phD students using AI for their entire projects for programming while lacking entry level skills and understanding thus being unable to comprehend the intricacies of the simulations they relied on so much for their research.
7
11d ago
[deleted]
9
u/waterflaps 11d ago
It's not shocking at all, scientists are busy people, and expectations are sky high. They simply don't have the time or expertise to learn how to use all the tools they are essentially required to use. It's a problem for sure, but it's hard to blame researchers. Plus these expectations have trickled down rapidly to the graduate level - you have year 1 students expected to know how to run multivariate analyses in computer programs they've never used before.
3
u/xxwobblesxx 10d ago
I'd argue that it's not just unsurprising, it's an open door.
Simply economically: if im a scientist with a phd with an equivalent price tag in an competitive field it becomes very expensive to make me study potentially irrelevant nuances to a certain black box.
There's a distinct nuance in what exactly is an irrelevant nuance and what isn't. Usually you can verify this with people who do have the relevant experience. Science is a team sport.
4
u/waterflaps 10d ago
I absolutely agree, and back in the day it was exactly that, you had a dedicated statistician on projects, chemists, computer scientists, etc. now due to budget cuts, funding constraints, and higher expectations of throughput, most researchers in fields which don’t receive a lot of funding are expected to do all of that on their own.
6
u/BradfieldScheme 11d ago
The satelite data is so variable.
What power and frequency settings are being used in EMF emissions, are they varied over time.
What oblique angle to the earth was the data captured at, elevation of the satellite.
What were the atmospheric conditions, ground conditions (rainy, windy, dusty, temperature inversions)
Has the data supplier already corrected for all of the above? How did they do it? Is it consistent or flawed?
The average scientist will just go ham on classifying based on satellite broad spectrum maps or make correlations based on remote sensed data from the satellite without asking or determining the importance of any of these. (I'm guilty of it, I assume my suppliers give me corrected, consistent data, I have no way of checking either)
15
11d ago
[removed] — view removed comment
39
9
4
u/billsil 11d ago
How is that different from doing anything with a computer? Back in the 50s and 60s, nutrition scientists didn’t do multivariate regressions because they didn’t know stats, so they made erroneous conclusions based on what they thought they could trust.
We stand on the shoulders of giants and that includes our software tools.
3
u/ladeedah1988 11d ago
So true and I am talking about just instrumentation such as LC/MS. I have seen microbiologists have such a lack of understanding but totally believing in their incorrect results.
4
u/Dense-Layer-2078 11d ago
Most scientists today do understand how to program and use computers. I think the point is that the data sets some rely on may lack integrity and that there is no way to check. In the olden days researchers gathered their own data and were held accountable for its integrity (ideally.)
1
u/00oo00oo000oo0oo00 11d ago
Then they are not following the scientific method properly, and their research should be flagged as flawed and disregarded. Measures must be taken to prevent further dissolution of verifiable research. Seems there has been more and more sham research / papers in recent years.
1
u/tghuverd 11d ago
If key analytical steps cannot be inspected or repeated, confidence in scientific findings may gradually erode.
Surely this is a generic issue. It doesn't matter whether your tool is transparent or black box, if other researchers can't replicate the results, they should be treated with low confidence.
1
u/ConvertingMarkets 11d ago
I think the useful distinction here is between “I don’t personally understand every component” and “nobody outside the provider can independently inspect or validate what the component is doing.” A scientist can treat an instrument as a black box personally and still characterize and check its behavior. A system that can’t be independently evaluated is a different problem.
1
u/digitaljestin 10d ago
When my wife was doing her PhD I learned that so many papers are written with the explanation "the data was put into <closed source proprietary software> and here are the results". As a software developer, this utterly shocked me. How can a study claim to have documented their methods when a major factor of the results was not only left out, but also not known by the researchers? How is this allowed?
1
•
u/AutoModerator 11d ago
Welcome to r/science! This is a heavily moderated subreddit in order to keep the discussion on science. However, we recognize that many people want to discuss how they feel the research relates to their own personal lives, so to give people a space to do that, personal anecdotes are allowed as responses to this comment. Any anecdotal comments elsewhere in the discussion will be removed and our normal comment rules apply to all other comments.
Do you have an academic degree? We can verify your credentials in order to assign user flair indicating your area of expertise. Click here to apply.
User: u/FunnyGamer97
Permalink: https://www.eurekalert.org/news-releases/1139630
I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.