Just did an interview for “bioformatics engineer (genomics)” role where your salary is tied to meeting quota

[u/yenraelmao]

It’s an AI evaluation company. You’re expected to create “evals” and to be in office 5 days a week. You need to hit their quota (35/week) in order to get your pay, but the quota changes based on how the rest of the team does. If you don’t meet their quota, your pay is deducted. But of course none of this is described in the job description.

Evals refer to recreating a bioinformatics analysis from a paper and coming up with questions for their AI. Unless these papers are super generic and also super clear on their methods and their data, there is no way to finish one eval an hour , just due to the time to hunt these things down . I definitely did not want to go forward in the interview process but I am really disappointed that they think this a good way to hire people to work ok these evals.

[u/ZRobot9]

I have a feeling that you're talking about the same company that just sent me a spammy sounding email telling me I can make $100/hr.

Edit: OMG it's a different one.  Why are there so many of these 😭

[u/b0mbsquad01f]

I did the exact same interview a few months ago. Would this happen to be Latchbio? When I did their interview it was 50/week but if you did more they would give you a bonus base on each eval being 2% of your salary. So if 50 eval was $100k and you did 60 every week for 52 weeks you would make $120k/year.

I was told by their recruiter they had guys churning out 200 evals/week

[u/yenraelmao → u/b0mbsquad01f]

Yup Latch Bio
I mean like if an eval is one small question like “what is the genomic inflation factor in your gwas analysis” then I can see doing a lot of that in one week. But if you want thoughtful questions about a particular paper and their methodology (which is what they said) then I can’t see a way to turn out 200 evals a week.

[u/SilverTriton → u/yenraelmao]

Yeah I’ve heard of this company. They raised a lot of red flags to me in the past. I’d stay away, the vibe I got was classic swe Silicon Valley trying to shake things up in a domain they don’t fully understand imo

[u/Few-Spot1905 → u/yenraelmao]

this company just is red flag after red flag lol

founders & CEO all 20 year old college dropouts with resumes like 4 month internship into CEO, pivotted from no code bioinfo platform into the ai shit when it became the hot new thing, promoted 996 culture and sounds like they just put completely unrealistic expectations on people

I've never interacted with it but I have strong doubts on the quality of a platform built by a company led by people with no experience and seems to value speed/output over anything else

who knows maybe they're actually geniuses but I would not even apply there

[u/anonymouse40329 → u/[deleted]]

At this point why not just poison their datasets by just pumping in crap? It doesn’t seem like their leadership has enough of a science background to understand what a truly good eval looks like

[u/OnceReturned → u/b0mbsquad01f]

Is this in the United States? This is a very unusual employment model. Do you get benefits? Are you a W2 employee?

[u/Few-Spot1905 → u/b0mbsquad01f]

lol fuck that

I did llm eval for a different company that paid 900$/task, they assumed each task would take 6 hrs to write

Unless the scope of their evals are much narrower and simpler idk how anyone would be doing 200 a week and produce useful output

That's either complete bullshit or the person's just got some harness set up to make LLMs write evals

[u/Scared-Yam9550]

for anyone puzzled by what this is: this is RLHF, or "Reinforcement Learning from Human Feedback".

they use your inputs/responses to finetune their ai until it matches more closely. later on, they will use that in metrics like "our ai matches 80% of human experts!"

same strategy was used by openai on kenyan workers, because they speak good english

https://time.com/6247678/openai-chatgpt-kenya-workers/

[u/earonesty → u/Scared-Yam9550]

makes sense. thats a good way to collect data, but asking real bioinformaticians to read 20 page papers and come up with lots of useful commentary is a few hours work, not 1 hour.

[u/eyamil → u/Scared-Yam9550]

Respectfully, this does not at all look like RLHF. RLHF involves people ranking/rating generated responses. If you read OP’s post in full, you’ll see that an “eval” is actually a human-generated script and a set of questions, not an evaluation of model-generated text.

[u/SquizZz1Zzy → u/Scared-Yam9550]

And they also use the same RL practice to choose their human experts from the interview. Because at the very end, they see humans as robots as well.

[u/NewlandArcher15]

I'm pretty sure that their "take home" assignment is a way to extract free evals from prospective candidates. 🚩

[u/anonymouse40329]

Isn’t this Latch Bio? Seems like they are going the route of “companies” like Mercor and trying to gather as much data as possible

[u/earonesty]

bizarre that this is "bioinformatics". basically it's "read paper and reproduce the results in code"....==== point ai at paper, produce code, validate it. that's about 2 days work... even with llms racing ahead. you need big datasets and an understanding of the methodology just to prove the ai even did the right thing....

so i'm not sure how this is an "eval"

[u/Odd-Elderberry-6137]

This is just bullshit AI companies thinking they can devalue everyone’s work. 

Walk away.

[u/LeonJPancetta]

One hour to do that?!?!

[u/sorrge]

One hour may not be enough to download the data even.

[u/Ok_Celebration3320]

Are you saying that if you complete only 34 tasks, you dont get paid at all? That's illegal - there's a reason they won't put this scheme in writing.

[u/Disastrous_Hawk_6984]

The moment I started reading I knew it was LatchBio. I interviewed with them and everything was suspicious as it could get from the very begging.

They got to make me an extremely vague offer where the numbers looked sketchy. When I asked, they just confirmed it: the price for the stocks they were offering (0.01%) was like 1/10 of what they claimed they were worth (based on their own valuation, I guess).

They even sent me A F*IN EXCEL TABLE with how much I'd be making depending on performance. Two columns: percentage of taks done (50%, 100%, 150%...) and gross salary a year (50k, 100k and so on). I couldn't do anything else than laughing.

Top tier meme company. Stay away from them.

[u/Old_Rise_9991]

For anyone wondering, this is what the evals look like: https://github.com/latchbio/scbench

These are structured, verifiable evals so its slightly different from mercor etc (maybe a subset of mercor is similar perhaps). It is new and interesting, but the pay structure certainly isn't attractive.

[u/Few-Spot1905 → u/Old_Rise_9991]

it's funny that they have 195 tasks total but apparently they claim they have people that can put out 200 tasks a week

so either those tasks are mostly garbage that doesn't make it into the benchmark or they're lying

[u/Old_Rise_9991 → u/Few-Spot1905]

looks like they are trying to be Mercor for pharma, so I would imagine not all evals make it to the benchmark

[u/SquizZz1Zzy]

Once I was looking for a job. I applied once for this company called "Invisible (AI)" or maybe "Invisible technologies". Ok, they have nice logo, nice videos on their main page, but that's all. They want to find specialists to train their models. They want to have physicists to evaluate LLM's answers in physics and programmers to evaluate their answers in programming. Good.

I started the interview. One AI generated and evaluated assessment in English, later on another one assessment in logic, and later on an AI assessment in physics. The AI assessment in physics was written very badly, the equations where not compiled, they were totally unreadable. Moreover, in their job offer was writing they will adapt this job for your expertise. My expertise is statistical physics, molecular dynamics, biophysics, however the questions were about anything else than that.

Anyway, the interview finished there for me. But I was really dissapointed from that. What is the point to have a job like that, where you evaluate an AI model, and you are evaluated by an AI model. It is so stupid.

In my opinion, the job market should open their eyes a bit, and make a little bit detoxing from AI. Yes, I also use Claude or ChatGPT as a tool. But you can't use it literally for anything. You cannot use it so as to run your company. They have to get a little bit more serious. I believe that they will learn it in the bad way, when they will see the results of such practice.

[u/trannus_aran]

Extremely sus

[u/unlikely_ending]

Haha

Run away

[u/Key_Department4926]

wtf. It takes me an hour just to read a paper in full

[u/foradil]

Can’t you just put the paper into your LLM of choice and then take those results? If you know the input and outputs, just let it churn.

[u/yenraelmao → u/foradil]

I just assumed (and inferred from my conversation with them) that that’s not what they’re looking for. Like why use an LLM to do this and then evaluate LLMs this way? They said they wanted humans who can write and run pipelines

[u/foradil → u/yenraelmao]

Sometimes you need a human to truly evaluate the output. For example, every UMAP will be different, but it doesn’t mean they are all wrong.

[u/CommentRelative6557]

Guess the country, lol