blogmeta-synthetic-data-kit
meta synthetic data kit
Fine-tuning is only as good as the data you can afford to make.
the idea
the problem
Every fine-tuning project meets the same wall: a handful of documents on one side, thousands of clean instruction pairs needed on the other. Writing them by hand does not scale, and naive generation gives you repetitive, unusable data.
the approach
Meta's synthetic-data-kit turns source documents into training datasets through extraction, generation and curation. My work sits upstream: making the generator behave on real inputs, and making the output survive review.
what's done
Contributions merged into meta-llama/synthetic-data-kit.
Work on generation quality and output formatting for downstream fine-tuning.
Used in my own PEFT and LoRA experiments.
what's missing
Broader document-format coverage upstream. It is tracked in the project's issues, and not mine to close alone.
Tighter curation tooling. The generator is ahead of the reviewer.
how to try it
git clone https://github.com/meta-llama/synthetic-data-kit
cd synthetic-data-kit
pip install -e .Then point it at a folder of documents. Upstream docs: meta-llama/synthetic-data-kit.