A small MLX model can turn a sentence into a flow

I've been turning over an idea for a while: if a flow is just structured JSON (triggers, modules, connections, parameters), then generating one from a natural-language sentence is exactly the kind of task a small model can do well, given the right examples.

Big foundation models already do this with prompting; that's not the question. The question is whether a small, locally-runnable, fine-tuned model can do it better than a generic large model on a specific flow grammar, and fast enough to feel instant inside an authoring tool. Apple's MLX framework is the obvious place to try this on a Mac, so this post is the plan, written down before any runs. I'll come back with numbers.

(Why MLX, since people will ask: llama.cpp is great for inference but less ergonomic for fine-tuning, PyTorch on Apple silicon works but runs slower and rougher, and cloud GPUs defeat the privacy point. MLX keeps fine-tuning and inference on the same machine with the least friction. That's the whole rationale.)

If you're not on the engineering side: this is how flow authoring stops being a UI exercise and becomes a sentence, the way you stopped writing formulas to filter a spreadsheet and started describing the filter. A small local model is what could make that feel instant, and private enough to actually use.

Why fine-tuned and small, not GPT-class prompting

Three reasons, in order of how much they actually matter:

1. Latency. A 0.5B–3B model on MLX can plausibly hit sub-second generation for a typical flow on an M-series chip, and sub-second is the difference between a feature people use and a spinner they avoid.
2. Determinism on a fixed schema. A model fine-tuned on one flow JSON schema should make fewer schema mistakes than a generic model that has seen a million unrelated schemas. The grammar gets baked into the weights.
3. Privacy and runtime location. The flow being generated will usually end up touching private data. Generating it on the same hardware the flow will run on is much easier to govern than routing it through a third-party API. This fits how flow8 is positioned generally (AI running where the data lives), so a local flow-generation model is the right shape for that worldview.

The target task

Input: a natural-language description of a business process. Output: valid flow-schema JSON (modules, parameters, connections, triggers).

A toy example:

INPUT:
"Every morning at 7am, pull yesterday's invoices from QuickBooks,
generate a one-page summary PDF, save it to S3, and email it to the
finance group."

EXPECTED OUTPUT:
{
  "trigger": {"type": "schedule", "cron": "0 7 * * *"},
  "modules": [
    {"id": "qb_fetch", "type": "quickbooks.invoices.list",
     "params": {"date": "yesterday"}},
    {"id": "summarize", "type": "ai.summarize",
     "params": {"format": "one-page", "input": "$qb_fetch.items"}},
    {"id": "pdf", "type": "pdf.generate",
     "params": {"content": "$summarize.text"}},
    {"id": "s3", "type": "aws.s3.upload",
     "params": {"bucket": "fin-reports", "body": "$pdf.bytes"}},
    {"id": "email", "type": "smtp.send",
     "params": {"to": "[email protected]",
                "attachments": ["$pdf.bytes"]}}
  ],
  "edges": [
    ["qb_fetch", "summarize"], ["summarize", "pdf"],
    ["pdf", "s3"], ["pdf", "email"]
  ]
}

A small model trained on a few thousand of these should learn the structure cold. That's the hypothesis, anyway.

Candidate base models

The shortlist for an initial sweep, all available in MLX-compatible format:

• Qwen 2.5: 0.5B, 1.5B, 3B (instruction-tuned variants)
• Llama 3.2: 1B, 3B
• Phi-3.5 mini: 3.8B
• Gemma 2: 2B
• Mistral 7B v0.3 (as the "is bigger meaningfully better?" anchor)

All small enough to fit comfortably on an M-series laptop, all supported by mlx-lm.

Training plan

Dataset.

• ~5,000 (prompt → flow JSON) pairs
• sources: synthesized from real flow definitions plus variations, plus paraphrased duplicates to teach robustness to wording
• 80/10/10 split: train / validation / held-out test
• a hard-case test set: ambiguous prompts, multi-trigger flows, flows requiring sub-flows

Method.

• LoRA fine-tuning via mlx-lm.lora
• rank 16, alpha 32, dropout 0.05 as a starting point
• ~50k tokens per epoch, 5 epochs initially
• mixed-task training: some "explain this flow" examples included, so the model stays coherent for inverse questions later

Validation.

• cross-entropy on the JSON output (standard), plus structured checks:
• JSON parseability rate
• schema validity rate (against a JSON Schema for the execution layer)
• module-name validity (only modules that actually exist in the module set)
• exact-match rate vs. the gold flow
• functional equivalence: different shape, same behaviour, judged by an evaluator model

Benchmark plan

Being honest about what hasn't been run yet: everything below is a placeholder until it isn't.

Hardware. M3 Max, 64 GB unified memory (later an M4 Pro for comparison).

Compared systems.

• base Qwen 2.5 0.5B / 1.5B / 3B: fine-tuned vs. zero-shot prompted
• base Llama 3.2 1B / 3B: same
• Phi-3.5 mini: fine-tuned vs. zero-shot
• GPT-4o-class via API: zero-shot, with a tight system prompt + JSON Schema
• Claude Sonnet: zero-shot, same prompt strategy

Targets per system.

Results table, to be filled in.

What I expect (written down now, so the follow-up has to be honest):

• the best fine-tuned small model (Qwen 2.5 3B or Llama 3.2 3B) beats zero-shot GPT-4o on schema validity and module-name validity
• zero-shot GPT-4o still wins on functional equivalence for weird, underspecified prompts
• the 0.5B–1.5B fine-tuned models are fast enough for live in-editor generation; the 3B ones are borderline
• p50 latency on the M3 Max for the 1.5B model lands somewhere in the 200–400 ms range

How this lands in a product

If the numbers come out the way I expect, the path is fairly clean: ship the fine-tuned model as an optional flow-authoring assistant that runs locally next to the execution layer. For customers running the platform on their own infrastructure (the default), the same model runs on the same box: no cloud round-trip, no egress. Every new module in the catalog becomes a new "verb" the model can compose with. And for enterprises, the approach repeats per customer: fine-tune on their own flow history, so the assistant learns their conventions.

The bigger point: fine-tuned small models are the right shape for narrow, structured, frequent tasks like this one. The reflex to reach for the biggest available model is usually wrong when the problem rewards specialization.

Numbers in a follow-up. And if they come out worse than the expectations above, I'll say so plainly. If you've fine-tuned small models on MLX for structured output, curious what you saw, especially on latency!