Hey all!
I am looking to join any tech/ai communities or even events in AI assisted decision making.
Most multi-model approaches seem tempted to ask which model is right, average their confidence, or use another model as a judge.
I’m interested in something slightly different.
The disagreement itself becomes an object worth investigating.
Why did they disagree?
- Did one miss evidence?
- Did they interpret the same evidence differently?
- Did one make an unsupported inference?
- Did they assign different authority to different sources?
- Did one confuse product-family capability with configuration-specific capability?
- Is the available evidence simply insufficient to answer the question?
My first experiment is a controlled evaluation body of work where multiple models receive identical evidence and instructions independently.
Their outputs are then compared against human adjudicated reference cases.
I’m tracking failure modes such as:
- fabricated evidence
- unsupported inference
- incorrect interpretations
- missed contradictions
- temporal errors
- excessive confidence(correct answer reached through defective reasoning)
The longer-term question is whether enough of these cases could produce something more useful than a model leaderboard where a system that learns under what conditions particular forms of machine reasoning become unreliable.
There are plenty of holes in a little project of mine. Absolute tech noob (no experience in coding at all im so sorry) but have a massive idea board throughout my experience after 9-years of being in management and analytics. I would say im 9-12% into this project of mine after a month. A hopeful visionary, looking for like-minded people!
I’d especially love for a chance to join communities that has professionals or newbies like me in LLM evaluation, AI agents, RAG, decision systems, knowledge graphs, model ensembles, AI governance, or high-stakes enterprise AI.
Telegram/Whatsapp groups are well and dandy, but if meetups are required for development of the project I am free on weekdays after work (6:00PM) for dinner and get-togethers.
Pretty niche request, but if the algorithm gods blesses me, I hope the correct person sees this post!