dataproc4: Notes on Reuse and Ruins

“we built this before Lepton, before R2, before anyone stopped us.” > view it on github > 中文版

dataproc4: Notes on Reuse and Ruins 📍 Why This Exists 🧱 Architecture Recap ✅ What’s Still Usable Namespacing and Per-Source Merging Deterministic Pipelines Execution Model: Checkpointing + Resumability Prototyping Flow ❌ What Didn’t Age Well Memory-Conscious Chunking WebDataset Retrofitting (Incomplete) Beneath the Surface 🏁 Final Notes Appendix📚 Bonus: Further ReadingsGeneral Articles on Data Pipelines and MLOpsFrameworks ComparedOther Useful Tools

📍 Why This Exists

You want to know what dataproc4 solved—and what still matters. This won’t walk you through every node. It’s a summary of core ideas, re-usable patterns, and what didn’t age well.
You can plug the full codebase into an LLM now. This isn’t competing with that. Think of it as a map of intentions: what the system meant to do, not just what it does.

🧱 Architecture Recap

完整细节请见 Lark文档

The pipeline has several stages 1. We’ll revisit the design rationale in later sections.

  1. Merge meta: Crawler → S3 → metadata parquet (*.todo.parquet)

  2. Prefiltering: Drop low-quality rows (e.g. 85% of Pixiv is amateur art)

  3. Calculate metrics: Via SageMaker → *.[metric].parquet

  4. Merge metrics: One file per dataset 2

  5. Merge datasets: Align images across datasets

  6. Assemble prompts: Metric-based filtering + prompt building

  7. Export dataset: Push to versioned S3 bucket

Merge‑Meta (factory)

clip_metrics

base/pixiv/dbr
mergers

mld_tags

pixiv_meta

dbr_meta

data‑source config

Sources

Pixiv raw

Danbooru raw

Twitter raw

ArtStation raw

Update Todos
(inventory diff → todo.parquet)

SageMaker Jobs
(clip scores, softmax, taggers)

*_merged.parquet → S3

Prefilter Data
(clip ≥ 1.2, softmax, general rows)

Augment & Tag
(pixels, year, quality)

Dedup (intra‑dataset)

Dedup (cross‑dataset)

Assemble Prompt‑dict
(caption + tags)

Sanity Check
(AI flags, artist, banned terms)

Encode Latent
(Lepton)

Training Pipeline

Gradio / Comfy Demo

✅ What’s Still Usable

Platform constraints changed (e.g. we now use WebDataset + H100s, not small RAM nodes). Many old constraints are gone. But the design logic still holds.

Namespacing and Per-Source Merging

Dataproc4 processed anime-style images. Certain assumptions shaped our strategy:

We responded by:

This let us prioritize more reliable tags, and helped prompt structure: key info up front, uncertain info later, better for CLIP-style models.

Code to Review: pipelines/merge_meta/config.py → See pipeline configs and how merge_function performs per-source merging. source

Deterministic Pipelines

We used Kedro for development. Its philosophy aligned well with our goals:

The result is reproducibility, sanity, and easier onboarding.

image

*(Variable pipeline behaviours are encoded into config files).

Execution Model: Checkpointing + Resumability

Each stage in dataproc4 was built to be restartable. Checkpoints allowed us to pick up work midstream:

Trade-offs considered:

MethodProsCons
S3 versioningFast, stableNo preview / UI
HF DatasetsBetter UX, public viewerSlower, but sane
Local parquetFast prototypingBrittle, hard to share

Checkpoint logic matters less as infra improves, but stage-separation is still worth maintaining—for stability, reuse, and iteration.

An example of caching intermediate results can be found in the caption-generating pipeline, and the actual dataset is defined in the data catalog.

Prototyping Flow

Early iteration followed predictable steps:

  1. Sketch stages in a flowchart (I prefer paper a lot)

  2. Prototype transforms in notebooks 6

  3. Extract stable code → pipeline nodes

  4. Build full DAGs with clear inputs/outputs

This let us move fast without degrading structure. Pipelines became readable, testable, and reproducible.

Don’t handcraft datasets, don’t mutate data manually, don’t sneak side-effects into notebooks. The pipeline is the source of truth. Let it stay that way.

Note:
Cursor may not handle notebooks well; for AI-assisted dev, notebooks are still useful for prototyping, but integrate gradually. Avoid agent-mode modifications to the pipeline. Small corrections are brittle—compose, test, then merge. Developing in dedicated test files could be an cursor-friendly alternative to notebooks.

❌ What Didn’t Age Well

Memory-Conscious Chunking

Now we’re on 8xH100 nodes with 2TB RAM. Chunking adds overhead. The retry logic was elegant, but probably overkill now.

WebDataset Retrofitting (Incomplete)

Structural shift happened midway:

Gathering meta may need rewrites. The latter parts might be re-used, but only with adaptation.

Beneath the Surface

This document focused on reuse. Some deeper topics weren’t covered:

These belong in a separate doc—Episode 2, if you’re brave.

🏁 Final Notes

Dataproc4 worked.

It scaled, handled contributor load, and withstood hostility. Its fixes to data chaos—naming discipline, probabilistic merging, modular transform logic—still matter.

Reuse what fights entropy. Let the rest rot.

If you go deeper, I’ll see you in Episode 2.




Appendix

  1. Instead of using Kedro's included datasets, I wrote unibox as a faster and more flexible alternative.

  2. To increase efficiency, metrics are calculated only on a subset of data.

  3. Refer to yandere data processing doc for deduplication setups.

  4. Kedro does not allow passing in variables unless logged inside a config file. Still, we did several bypasses in the actual pipeline.

  5. Lore wise, Kedro was developed and open-sourced by McKinsey, a consulting firm.

  6. Refer to Kedro's official documentation for experimenting with notebooks.

📚 Bonus: Further Readings

Before settling on Kedro, I surveyed a wide field of frameworks and philosophies. These links may be useful if you're thinking about pipeline strategy—or just want to see how deep the rabbit hole goes:

General Articles on Data Pipelines and MLOps

Frameworks Compared

Other Useful Tools