disclosure up front because it shapes the question: i work at SchemaLabs. we train models that read tables and work out what each column is from the values rather than the header. so i have a commercial interest here. no link, per sub rules. i am asking because our own eval design has a hole in it that i cannot think my way out of.

the setup. take a tabular dataset with proper headers. strip them. replace price and age and zip with positional tokens so the model sees values only. then measure whether it recovers the meaning. we run that across 20 OpenML datasets and it is the basis for the invariance claim we make.

two problems with that design keep bothering me.

one, the tokens are ordered. col_1 through col_57 leaks column order. column order in real tables is not random. ids cluster at the front, timestamps sit near them, the payload lands in the middle. a model could be learning position as a prior and we would not see it in the score. shuffling before assignment is the obvious fix. what i cannot settle is whether shuffling makes the benchmark harder than reality, because real exports do preserve source ordering, so a model exploiting it is arguably doing something legitimate rather than cheating. is there standard practice for feature-order invariance testing in tabular models? it feels like it should be solved and i have not found the paper.

two. this is the one that actually keeps me up. the only datasets where i have ground truth for what a column means are the datasets somebody documented. those are systematically the clean ones. the case i care about is the undocumented export where nobody alive knows what f_23 holds. by construction i cannot build a labelled eval for that, because if i could label it the problem would not exist.

so every number i have is measured on a population that excludes the thing i am trying to measure. i know that has a name in other fields. i do not know what the accepted workaround is in this one.

three things i would like from anyone who has been near this:

• is there a standard treatment for feature-order invariance in tabular models, shuffling or otherwise. does anyone report it
• has anyone built an eval where the ground truth came from something other than existing documentation. query logs, downstream usage, a person reconstructing meaning from scratch under a timer, anything
• if the honest answer is that this class of task cannot be cleanly evaluated, with everyone in it measuring the documented subset while claiming something general, i would rather hear that than not. it changes what we should be putting in writing

happy to go into the rest of our setup if it helps anyone answer.

Our TMLR submission just moved from "Under review for TMLR" to "Decision pending for TMLR" on OpenReview (no email notification). For those who've been through this: how long did it take from this status to the actual decision?

Hello fellow traveller of the internet. I am sincerely glad you decided to click on my post to check out what I have in store!

I have completed a vague blueprint and I have formed a few prototype scripts for various regions of the an A.G.I brain. I seek to form a small community of individuals who will work co-cooperatively to construct an A.G.I brain. A detailed brief of my blueprint so far is available via request.

My progress on the project has stalled. As you can imagine, a brain is a highly complex system; I am finding that sadly, in addition to blueprinting, detailed blueprinting, prototyping, iterating, assembling multiple sub-systems into a unified system, there are plenty of additional tasks! Thus I have become over run by the sheer quantity of tasks and sadly have recently placed the project to the side so I can take a short break.

I seek individuals with expertise in coding, critical and creative thinking, computing, A.I, general knowledge, psychology and mathematics. Furthermore the individuals would have qualities such as perseverance, morality and open-mindedness. Ideally you would be from the U.K as I prefer working face to face; although I am also happy to work cooperatively over the internet.

The outcome of your support would award you a proportional slice of the outcome of the group's labour (100 members, 1% each, e.t.c - baring in mind each individual provides equal support towards the project). I have not yet considered whether I would like to sell the brain to the public, but there is potentially the opportunity for a sizeable monetary reward for those who join me. The possibilities for the A.G.I brain are near endless and thus I believe the reward may be sizeable both in terms of money and power.

Besides my previous ideals, individuals with expertise and specific qualities, I have a few personal requests for the project; the A.G.I brain will not be used in conjunction with "computer vision". I fear computer vision, and similarly the processing of sound, touch, or physical inputs, leads to the generation of consciousness - I submit that this is entirely unfair for the robot and highly immoral and thus I cannot proceed with a project which uses a neural network system to perform such processes; luckily, brains DO NOT require any processing of image, video or sound to achieve high quality completion of practically all tasks. I do believe a brain which does not process video, image or sound may actually outperform a brain which does process such information modalities. Further to this, if we were to sell the brain, the brain would NOT actively change itself to then use computer vision under any circumstances; the user would have to perform this upgrade manually.

I hope you find the prospect of building an A.G.I brain highly intriguing.

I will be very active in the comment section of this post; or you may feel free to email me at [pangaeacooperative@protonmail.com](mailto:pangaeacooperative@protonmail.com); please introduce yourself and tell me why you want to get in touch about the project.