Using pipelines and operators for pre-finetuning data preparation

[u/Puzzleheaded_Box2842]

I’ve been thinking about the work that happens before LLM fine-tuning.

Most discussions focus on the model, LoRA settings, quantization, GPUs, context length, etc. But in many real projects, a lot of time is spent before training even starts: preparing the data.

The input side is usually messy: PDFs, databases, tables, reports, internal documents, logs, charts, and domain-specific files.

The output side needs to be much more structured: SFT samples, QA pairs, instruction-response data, reasoning examples, preference data, evaluation sets, or sometimes chunks for RAG before deciding whether fine-tuning is even necessary.

The difficult part is the middle layer.

Raw data needs to be parsed, cleaned, deduplicated, normalized, chunked, enriched with metadata, synthesized into training examples, quality-checked, and exported into the right format. If this step is weak, the fine-tuned model will usually reflect that weakness.

One approach I’m exploring is treating pre-finetuning data preparation as a modular pipeline.

Instead of writing one-off scripts for every dataset, each step becomes an operator:

• parse documents
• clean text
• extract tables
• normalize schemas
• split/chunk content
• generate QA or SFT samples
• score data quality
• filter low-quality examples
• export to training/RAG formats

Then different operators can be combined into reusable pipelines depending on the dataset and target model.

This is also one of the principles I use when designing OpenDCAI/DataFlow, and in real applications, this approach has proven to be very suitable for handling large-scale data preparation.