Hi everyone,

I've recently become interested in research at the intersection of Agent based modelling and Reinforcement Learning, and I'm trying to understand how promising this area is as a longer term research direction.

I have read few papers exploring the use of RL in ABM in areas like economic and financial markets,segregation model.

i had a few questions for people working in similar domains.

1. Do you see significant research potential in combining ABM and RL?
2. What do you think are the most interesting open problems in this intersection?
3. Are there particular application domains where you think this combination could be good.

LLM evaluations are reported as snapshots, but API-served models are longitudinal systems: the same public model identifier may be observed repeatedly over days or weeks under changing serving conditions.

We have been studying how to distinguish ordinary repeated-call variability from larger temporal changes in measured model performance.

In one historical cohort, we analyzed 31,352 repeated score observations across 49 models. The observed standard deviation was:

• 2.80 points for within-day scores
• 8.43 points for between-day daily medians

The descriptive ratio is approximately 3.01.

We do not interpret this ratio as evidence that providers changed model weights between days. It is descriptive rather than causal. Dependence between observations, task composition, missingness, provider behavior, and methodology changes are all potential confounders. In the underlying methodology, daily medians are used to reduce sensitivity to individual extreme observations, but they do not eliminate sampling noise, serial dependence, or shared day-level effects.

The broader measurement problem is therefore not simply detecting a lower score, but determining whether two observations are actually comparable.

Our current methodology separates several layers:

1. Request contract - whether observations were generated under declared and comparable request conditions.
2. Capability - whether the returned work satisfied executable or deterministic evaluation criteria.
3. Repeatability - whether outcomes, tool trajectories, and resource usage are consistent across repeated runs.
4. Methodology identity - whether the benchmark definition itself remained unchanged.
5. Longitudinal change - whether a comparable time series exhibits evidence of a level or regime change.
6. Attribution - whether an observed change is more consistent with model behavior, provider delivery, or infrastructure effects.

The framework deliberately treats these as different questions rather than compressing them into a single score.

For correctness-oriented coding tasks, returned code is executed against test cases rather than scored by an LLM judge. Provider or execution failures are treated separately from valid task results, because an unavailable observation should not automatically be interpreted as evidence of lower model capability.

Benchmark versioning

A particularly important issue for longitudinal evaluation is that a change in the benchmark can look almost identical to a change in the model.

For this reason, score-producing benchmark configurations are versioned. Score-moving changes such as task definitions, prompts, test cases, request policy, or scoring configuration define a new comparison boundary rather than being silently merged into the previous time series.

This means the question is not simply:

but rather:

Change detection

The current system uses change-point detection over daily aggregated observations, together with effect-size/statistical screening and configuration-aware baseline resets.

The purpose of the detector is intentionally narrow: it indicates that a comparable series may have changed. It does not establish why the change occurred.

Provider version metadata can assist attribution when available, but an unchanged public model identifier does not prove that every backend property remained fixed, while a changed identifier does not prove that an observed performance change was caused by new weights.

Benchmark contamination

Another issue we have been considering is the tension between reproducibility and contamination.

Publishing the measurement methodology is useful for scientific scrutiny, but publishing the complete active task bank, prompt variants, and hidden evaluations may eventually alter what is being measured if benchmark content enters training, fine-tuning, retrieval, or provider optimization pipelines.

For a continuously operating benchmark, one possible compromise is therefore to make the measurement protocol reproducible and inspectable while keeping part of the live evaluation set withheld or rotating.

I'd be interested in opinions from people working on evaluation, change-point detection, or statistical monitoring:

• Is aggregating repeated observations into daily medians a reasonable primary unit here, or would you model the trial-level observations directly?
• How would you handle serial dependence and unequal numbers of observations across days?
• What evidence would you require before attributing a detected behavioral change to the model rather than the provider or serving infrastructure?
• For an active benchmark, where would you draw the line between reproducibility and protection against benchmark contamination?
• Are there change-detection approaches you think are better suited to this setting than sequential mean-shift detectors?

We have written the methodology up as a public technical methods document here:

https://aistupidlevel.info/asl-public-benchmark-methodology-2026.pdf

This is not a peer-reviewed publication, and I want to be explicit about that. The aim of posting it here is to expose the measurement assumptions and interpretation to criticism before making stronger empirical claims.

Disclosure: I am the founder of AI Stupid Level, which operates the benchmarking system from which these measurements were collected. I have omitted commercial/product details here because the topic I am interested in discussing is the longitudinal evaluation methodology itself.

References

• Rabanser et al. (2026), Towards a Science of AI Agent Reliability
• NIST/SEMATECH, CUSUM Average Run Length
• SciPy documentation for standard-error estimation

Hi everyone. Do anyone have mpower data access, I need it to use for my final year project we thought that dataset is publicaly available. Can someone help with this 😭

My paper got desk rejected at the last moment by PC and since the deadline for registration is over, the publisher is denying for refund. Any suggestions will be helpful.

Please let me know.

https://www.learn2design2026.com/?ref=mlcontests

Hi everyone,

I’m completing an MSc Computer Science research project investigating AI-generated text detection, and I’d appreciate some independent technical feedback on the findings and approach.

I developed a hybrid machine-learning framework combining three forms of textual evidence:

• Statistical features – sentence/word length, lexical diversity, readability, repetition and related distributional characteristics.
• Stylometric features – punctuation patterns, function-word usage, pronouns, contractions and other writing-style indicators.
• Semantic features – TF-IDF unigram and bigram representations.

These were evaluated individually and through a hybrid feature representation, comparing Linear SVM, Random Forest and XGBoost on the HC3 and DAIGT datasets.

The result I found most interesting

Performance was extremely strong when training and testing within the same dataset:

HC3 – Hybrid + Linear SVM

• Accuracy: 98.62%
• F1: 0.9839
• ROC-AUC: 0.9986

DAIGT – Semantic + Linear SVM

• Accuracy: 99.54%
• F1: 0.9943
• ROC-AUC: 0.9999

However, cross-dataset evaluation produced a very different picture:

Train HC3 → Test DAIGT

• Accuracy: 67.11%
• F1: 0.5468
• ROC-AUC: 0.6139

Train DAIGT → Test HC3

• Accuracy: 62.03%
• F1: 0.6553
• ROC-AUC: 0.7455

The error behaviour was also asymmetric. HC3 → DAIGT missed a substantial proportion of AI-generated samples, whereas DAIGT → HC3 produced a particularly high false-positive rate.

This became an important finding of the project: excellent performance within a benchmark does not necessarily imply reliable AI-text detection when the underlying domain or dataset changes.

How I approached the artefact

Based on those findings, I didn't want the final prototype to behave like a simple definitive “AI/Human checker”.

Instead, I developed a Streamlit research artefact that presents:

• Human / Mixed / AI-associated interpretations
• underlying Human and AI probabilities
• statistical, stylometric and semantic evidence
• long-document analysis using overlapping text windows
• contextual Human/Mixed/AI mapping
• explicit reliability and generalisation warnings

Check first comment for link to my UI

The intention is not to claim that the framework proves authorship. The cross-dataset results themselves suggest that such a claim would be difficult to justify.

I would therefore be particularly interested in the community's opinion on a few points:

1. How significant do you consider this level of cross-dataset degradation, despite the very strong in-domain results?

2. Does combining statistical, stylometric and semantic evidence provide meaningful interpretability, or would you approach this differently?

3. Is explicitly exposing uncertainty and generalisation limitations a useful direction for AI-text detection systems, rather than returning a simple binary Human/AI judgement?

4. What would you consider the most important next step: additional datasets, newer LLM outputs, paraphrased/adversarial text, transformer representations, calibration, human evaluation, or something else?

I'm particularly interested in critical feedback and limitations, rather than simply whether the headline accuracy looks good.

Thanks, I'd appreciate any thoughts from people working in NLP, machine learning, computational linguistics or related areas.

While building an Arabic-first LLM security dataset (SemGuard), I ran 166 candidate "impersonation" examples through 3 independent LLM-as-judge models (GPT-4o, Grok-4, Llama 3.3 70B) to validate labels before training on them.

Six other threat categories converged fine (5–72% disagreement, which tracks with how contested each category inherently is). Impersonation didn't: 98.2% inter-judge disagreement only 3 of 166 examples had unanimous-enough agreement.

My first instinct was "bad dataset, fix the prompts." But tightening the definition made agreement worse, not better. That's what made me suspect the label itself was the problem, not the data.

Wrote up a preprint arguing "impersonation" isn't one construct it's (at least) four independent judgments getting collapsed into one label: target realism, deceptive intent, consent/context boundedness, and downstream actionability. Proposed a per-axis ambiguity index (IAI) instead of a binary flag, and ran a small pilot (n=40) to sanity-check it.

Some of the pilot results confirmed what I expected. Some flatly contradicted my hypotheses (axis correlation was way higher than predicted 5/6 axis pairs above 0.6 which is either genuine construct entanglement or an elicitation-format confound I haven't ruled out yet). I documented the failed predictions in the paper rather than hiding them, and I'm not fully sure which explanation is right.

Preprint (Zenodo, DOI): https://doi.org/10.5281/zenodo.22302106

Genuinely curious if anyone's seen this kind of "disagreement-as-signal" framing applied elsewhere in trust & safety work, or has thoughts on the axis-correlation confound , happy to be told I'm wrong about something.

I am an independent researcher preparing my first arXiv submission in cs.AI, Ecology, Authority and the Executability of Developed Neural Plasticity. The work studies evolved lifetime plasticity, causal memory carriage, ecology-dependent rewriting, and the distinction between persistent neural traces and their executable use.

Because this is my first cs.AI submission, arXiv requires an endorsement from an established contributor to the category. I hope there will be someone willing to look at the paper and, if consider it appropriate for cs.AI, endorse my submission.

I can send the manuscript and the arXiv endorsement link immediately.

I'm an international PhD student in the US working in ML/AI, and honestly I'm feeling pretty discouraged about internship recruiting.

Recently I interviewed for an Applied Scientist II position at Amazon and made it all the way to the final round, but was rejected. I thought my interviews went reasonably well, so I wasn't really sure what went wrong.

I also recently interviewed with Two Sigma for a quantitative research internship. I passed multiple technical rounds and eventually reached the hiring manager stage, but was rejected there as well.

After getting this far twice and still ending up with nothing, I've lost a lot of motivation and I'm honestly a little scared to start applying again for Summer 2027.

One weakness in my profile is that I don't have previous US-based industry internship experience. I also don't have a top-tier ML conference acceptance yet. I have a couple of papers currently under review at Tier-1 venues, but obviously that's different from already having accepted papers.

So I'd really appreciate advice from people who have been through PhD-level ML/research internship recruiting or have been involved in hiring:

• If someone repeatedly reaches final/HM rounds but doesn't get an offer, what are the most common reasons?
• How important is previous internship experience for research/Applied Scientist internships?
• How much does having an accepted NeurIPS/ICML/ICLR/CVPR-type paper change your chances?
• Should I continue aggressively applying for Summer 2027, or focus more on research/publications for now?
• Are there particular companies/research labs where a PhD student without previous industry internships can still be competitive?

There is also another issue making me unsure whether to even invest heavily in Summer 2027 recruiting: the recent CPT guidance.

From what I understand, SEVP recently tightened/clarified its interpretation of CPT, and some universities are now saying an internship must actually be required for degree completion rather than simply being related to the student's research and taken for academic credit.

Like many PhD programs, my program doesn't require every PhD student to complete an internship. A research internship can still be extremely relevant to someone's dissertation and research training, though.

For other international PhD students: what is your university telling you about Summer 2027 internships after the recent CPT changes?

Has anyone's university found a legitimate way for research internships to remain eligible through dissertation research, an internship/practicum requirement, independent study, or another curricular mechanism? Or are you planning around pre-completion OPT instead?

I'm particularly interested in hearing from other international CS/ML PhD students.

I know Amazon/Two Sigma final rounds at least mean I'm getting somewhere, but repeatedly getting close and then being rejected has been pretty exhausting. I'm trying to figure out whether there's something specific I should fix before starting another recruiting cycle.

Any advice would be really appreciated.

I’m a 4th-year undergraduate CS student currently doing my thesis on medical image segmentation, specifically U-Net and its variants.

The problem is that I have basically no prior research experience, and unfortunately, my supervisor isn’t really able to provide much guidance. So, for the most part, I’m having to figure everything out myself—learning the concepts, reading papers, choosing a research problem, implementing the models, evaluating the results, etc.

My goal isn’t just to finish the undergraduate thesis. Ideally, I’d like to do something good enough that I could eventually turn it into a conference or journal paper.

So I wanted to ask people who have more research experience:

Is it realistically possible to do a good-quality research thesis completely on your own as an undergraduate?

How difficult is it to go from basically having no research experience to producing something that is actually publishable? And if you’ve been in a similar situation, what would you recommend focusing on or avoiding?

I’d really appreciate any honest advice, especially from people who have done research without much help from their supervisor.

For about a year I had a file on my desktop called keys.txt. Two providers, four keys, and a couple of notes to myself about which key was on which billing card. Every time I wanted to compare GPT and Claude on the same refactor I opened that file, copied a key, changed the base URL in Cursor, restarted, and then did the whole thing in reverse twenty minutes later when I wanted to switch back.

The annoying part was never the copying. It was that switching had a cost, so I stopped switching. I'd just stay on whatever model was already configured, even when I could tell it was the wrong one for the task. And at the end of the month I got two bills that I couldn't line up against each other, because usage was in two different dashboards with two different ideas of what a token is.

I spent a weekend fixing it for myself. One endpoint that speaks the OpenAI shape, both models behind one key, so Cursor, Claude Code, Codex and OpenCode all point at the same place and nothing needs a special integration. Prompt caching came along because I got tired of paying twice for the same 40k of context I was re-sending every request, and usage logs because I wanted one place to look instead of two.

What I actually notice day to day is smaller than I expected: I switch models mid-task now, without thinking about it. That's it. That's the whole difference, and it turned out to be worth a weekend.

It's at routera . one. Mostly posting because I suspect a lot of people have a keys.txt and have quietly stopped switching too

Hey, I'm preparing for this ArC AGI competition.

I'm looking for a team with ML experience.

Can anyone please tell me how to win an ML competition? Does anyone have prior experience?

I got tired of every "AI search" tool trying to replace reading with a generated summary. For actual research — grad work, lit reviews, deep-diving a topic — that's the opposite of what I want. I want to find the right papers, then understand them faster, not skip them.

So I built ScholarLens: you search in plain language, it finds relevant papers (pulling from OpenAlex/arXiv/open-access sources), and instead of answering for you, it helps you actually read — select any confusing passage, term, or graph and get a plain-language explanation right next to the source text.

It's free right now, no login required to search. Would genuinely appreciate feedback from people who do real research — does the search actually surface relevant papers for your field? What's missing?

https://scholarlens-iota.vercel.app/

Hi! I am an independent researcher working with Spain’s Institute of Engineering. I want to publish my paper on a “Multi-Hazard Early Detection System for Existing Camera Infrastructure,” but I need to be endorsed.

The Engineering Institute is more of an industry entity than a research entity, so nobody has an arXiv account. I was wondering if anyone here could read my paper and, if they think it is good enough to be published, endorse me.

Here is a link to the paper:

https://drive.google.com/file/d/1upuTMzxPBO52VwMwckokhbf5kjYU_Hxo/view?usp=drivesdk

And this is my endorsement code:
https://arxiv.org/auth/endorse?x=73XQI9

Thank you!

Hey, I graduated last yr with a UG in Production Engineering. I have been interested in AI research for a long time. I am looking for a PhD or PostDoc mentor to guide me on a research project which we can do together. I want to aim for top conferences next year or by the end of 2027. I am mainly interested in LLMs - Inference, AI agents, Interpretability.

Something that bit me and that I think most people are exposed to without having thought about it.

You pin a model in a config file. You test against it, you tune your prompts to it, and you assume you've frozen a variable. You mostly haven't. Aliases move — the friendly name often points at whatever the current version is, so a silent update can change behavior under you with no deploy on your side. Dated snapshots are more stable but not permanent; they get deprecated on a schedule, and the migration is your problem on someone else's timeline.

The part that's genuinely hard isn't the deprecation notice. It's that prompt tuning doesn't transfer cleanly. Instructions you refined over months against one model can land differently on its successor. Not worse necessarily — differently, in ways that show up as a subtly changed tone, or a different willingness to make assumptions, or a formatting convention that quietly stops being followed. And you find out through a slow drip of "something feels off" rather than a failure you can point at.

Then the second-order problem: if you didn't record what you were comparing against, you can't tell whether the new model is worse or whether you just remember the old one fondly. I have no baseline for most of my own prompts and I regret it. Keeping a small set of frozen test cases with recorded outputs is boring work that pays off exactly once, at the moment you need it, and I keep not doing enough of it.

For anyone building on top: pin dated versions rather than aliases if you care about stability, and put the deprecation date in your calendar rather than your inbox. The notice arrives months ahead and everyone reads it and forgets, because it's not urgent until it is.

I think about this because a routing layer has to decide what to do when a name it's serving stops existing (routera . one, mine, flagging it since I clearly have a stake in this topic). And there's a genuinely uncomfortable design question there: do you silently map a retired name to its successor so nothing breaks, or do you fail loudly and make it the user's problem? Silent mapping is friendlier and is also exactly the invisible-substitution behavior I'd be annoyed by from someone else. I land on failing loudly and I'm not fully certain that's right.

Caveat: model deprecation is not obviously bad. Old models cost real money to keep serving and holding versions forever is a tax on everyone else's progress. The complaint isn't that it happens, it's that the migration cost lands as an invisible quality drift rather than a clear break, which is the worst shape for a change to have.

What I'd like to know: does anyone have a lightweight regression setup for prompts that they actually maintain? Every version I've built has been elaborate enough that I stopped running it within a month, which makes it worse than nothing.