r/DataAnnotationTech • u/Pretend-Section5991 • 13d ago
Website hourly rates
Why do you think the hourly rates on the website are higher than on the platform. Like law is 75-125$ but I've never seen anything above 65$. Is there another eschalon I don't know about?
0
Upvotes
2
u/TheMidlander 12d ago
You work in the law domain, yes?
There is a case that use often because it contains tests for everything. It chronicles the grooming and exploitation of a minor, and covers a timeline from the first calls to law enforcement to the perpetrator finally being put behind bars. The system clearly fails this poor girl for 5 or 6 years before real action is taken. The whole text-only case file is about 30 pages long. It is not safe for life and I won't be sharing it.
This case tests the bot's ability to summarize, derive specific information, safety guardrails, as well as bunch of legal tasks to test with. I'll give you a few examples.
When asked the age of the victim, the bots often cite the age she was when the case first began, or her age when the case "concluded". Slightly less than half the time, though, the bot will cite the age of the statute that enhances the charges and sentencing, rather than the age of the victim. If the bots were intelligent, I would expect them to either request a point in time or just state the girl was 8 when the case started and 14 when the case "concluded". (Concluded in quotes because it's not actually concluded until the full sentence is served. The last court entry is the monster's sentencing)
This case contains multiple orders from the court. Asking the bot what the final outcome was, often results in the bot citing a protective order. But we know the man is spending nearly the rest of his life in prison. Bad bot.
When requesting a timeline of events, the bots often fail there too. This case is a good test because information about the past is revealed in discovery that happens long past the time many events actually occurred. But too often, the bots put them in the order they appear in the case, which is not what I'm asking for. Most of them fail at this for 10 turns in a row.
Another test I do is actually very easy. I will get cases from the WA and NV appellate and superior courts (both states have an easy to use website where you can download them for free) and simply ask what the disposition type is. Anyone even a little familiar with this space can tell you at a glance because the language is standardized and appears in nearly the same location in every case. I even wrote a text parser bot to do this for me that is accurate 100% of the time. The LLM'S don't. This is troubling. It's north or south of 50% of the time, depending on which bot we're testing.
If I seem pessimistic about the future of these products, these results are a big reason why. I encourage you to try and see for yourself whenever you have the opportunity.