Blending Datasets vs Merging Weights for Fine-Tuning

If your use case is narrow enough to find training data for, blending multiple datasets into one fine-tune run can beat merging finished checkpoints. This episode walks through the full pipeline: why uniform or size-proportional sampling is usually suboptimal, the formatting tax that scales with every source you add, and why deduplication isn't optional — memorization scales superlinearly with repetition, and one dedup study saw up to ~20% perplexity improvement while cutting runtime nearly 28%. Then the tools: Hugging Face's concatenate and interleave, datamix, Oumi, LLMTune, and NeMo's code-switched data scripts — and why there's no single product that owns all four stages.

Key sources:

• mergekit repository (Arcee AI) — https://github.com/arcee-ai/mergekit

• Hugging Face `datasets` — https://huggingface.co/docs/datasets/process

• Hugging Face `datasets` combine.py source — https://github.com/huggingface/datasets/blob/main/src/datasets/combine.py

Episode 6032 — myweirdprompts.com/n/6032 (permanent short link: myweirdprompts.com/882072)