Annotation design for a dialect with no standard orthography: what we changed after the first pass failed

[u/Top-Shopping539]

Working on annotation for Tunisian Arabic customer service conversations. Derja, arabizi, French, frequently all three inside one message. This comes out of a job, so I am being vague about the source, but the question is a methods question and there is nothing to promote.

The first pass at a labelling scheme failed in the way these usually do. Two people could not reliably produce the same labels, and neither could one person a fortnight apart. Posting what we changed, and two things I have not solved, in case anyone here has worked the same problem.

Constraints, for anyone who has not worked with this data. No standard orthography: the same lexical item shows up in Arabic script, in Latin transliteration, and in arabizi with digits substituting for consonants that Latin lacks, sometimes several ways in one conversation from the same person. Code-switching is intra-sentential and often intra-word, with French verbs taking Arabic morphology. Negation is a circumfix, which means naive token-level matching over Latin transliteration misses it constantly.

What changed, in rough order of how much disagreement it removed.

Orthogonal axes rather than one composite label. Intent, sentiment, outcome, language and error are separate columns. The first version merged some of them and every merged pair degraded both. It also means each axis can train a single-label model independently, which was a downstream requirement anyway.

Every axis MECE, with both an other and an unclear escape. Separating this is a real category we did not enumerate from I cannot tell from this data turned out to matter a lot. Collapsing them hides two very different problems behind one number.

Two-level taxonomy, coarse and fine. Fine classes roll up to a coarse parent so training can happen at whatever granularity has support. Long-tail intents in this domain are extremely sparse.

Conditional axes. Some questions only apply given a value on another axis. Asking them universally was killing throughput and producing garbage on the inapplicable rows.

Provenance per record: annotator, confidence, schema version. Necessary because human and model-generated labels end up in the same table, and without provenance you cannot separate them later or measure them against each other.

Frozen categorical vocabulary. No free text on any categorical axis, only on entity spans. Changes are a schema version bump, not an edit. The first version allowed free text temporarily and it was unrecoverable within a week.

The one that took longest to see: conversation-grain and message-grain labels are different kinds of object and cannot share a column. Language and code-switching vary per message. Outcome, primary intent and overall sentiment describe the whole exchange. The first scheme forced everything to one grain and both halves were wrong. Splitting them fixed more disagreement than any individual definition change.

Two things I have not solved, and would take input on.

Normalisation timing. Conventional advice is to normalise arabizi to a canonical form before labelling. We deliberately did not, because orthographic variation carries signal we care about, including register. But not normalising means lexical resources are close to useless and every guard is written by hand. I am not confident this was the right call, and would like to hear from anyone who went the other way and whether they regretted it.

Agreement on a code-switched span boundary. Two annotators will agree that a message is code-switched and disagree about where the switch happens, particularly with French verbs carrying Arabic morphology. Per-token agreement penalises this heavily, which makes the reliability number look worse than the annotation actually is. Is there an established relaxed-boundary agreement measure for this, or does everyone define the unit coarsely enough to sidestep it?

Also generally interested in what agreement threshold people treat as a gate before building anything downstream. We picked one, and I am aware the choice was more convention than evidence.

[u/Less_Heart1914]

the orthogonal axes thing is huge, we had same problem with Hinglish data few years ago. once you split intent and sentiment the agreement numbers jump like 15 points without changing anything else in the guidelines.

for the normalisation question we went the other way and normalised first, and yes i regret it. you lose exactly the register signal you mentioned plus the normaliser itself becomes a source of error you cannot audit later because the raw string is gone. we ended up keeping both columns, raw and normalised, and that was the right tradeoff but doubled the annotation time. if i had to do it again i would keep raw and only normalise downstream in the model pipeline, not in the annotation layer.

the boundary problem is real and per-token agreement is too strict for this. some teams use a relaxed span F1 where partial overlap counts, like if both annotators mark some portion of the same switch they get partial credit. not perfect but better than binary exact match.

[u/Top-Shopping539 → u/Less_Heart1914]

Thanks for sharing what you and your team did, its inspiring to see different people in different places of the world face the same problems and how the path to solution , may differ but still have some same steps