On 1 September 2026, Shopify CEO Tobi Lütke published a result that compresses a large engineering idea into one sentence: a fine-tuned 0.8-billion-parameter model beat GPT-5.6-sol at xhigh reasoning effort on one specialised task.
The attached Buyer Profile slide makes the scope clearer. Shopify reports an internal judge score of 84.6 for Qwen3.5-0.8B, compared with 83.0 for the frontier teacher and 77.2 for an earlier Qwen3.5-2B production model. The score rose across three data iterations: 29,000, 42,000 and 54,000 samples. The slide also reports a system prompt compressed from 9,100 to 1,100 tokens and throughput rising from two million to 72 million profiles per day on 100 H100 GPUs.
Those are striking numbers. They are also first-party results for one internal task. Shopify has not published the Buyer Profile dataset, judge prompt, test split, confidence intervals, checkpoint or complete training recipe. Nobody outside the company can currently reproduce the comparison.
The valuable conclusion is narrower than the headline: when a task has a stable boundary, enough examples and a trustworthy evaluation loop, a small model can become better at that measured task than a much larger general model. It gives up breadth to gain density.
That trade is relevant to products such as PLAYGRND. It is not a reason to fine-tune a model before defining the job.
Model size and training are different decisions
Four terms are often collapsed into “training a custom model.” They describe different operations.
Fine-tuning continues training an existing base model on task-specific data. It changes its behaviour by changing weights.
Supervised fine-tuning (SFT) shows the model a request and a desired response or tool trajectory. The model learns to imitate those examples. This is normally the first useful step for a structured task.
Distillation transfers behaviour from a stronger teacher into a smaller student. In sequence-level distillation, the teacher produces or repairs training examples and the student learns them through SFT. Logit distillation is more direct, but it requires access to the teacher’s output distributions, which a closed API usually does not provide.
LoRA changes a comparatively small set of adapter parameters instead of updating the whole model. It makes experiments cheaper and easier to roll back. A full-parameter fine-tune updates all weights and can capture more, but it needs more memory, more compute and stronger protection against forgetting.
Preference or reinforcement optimisation, including GRPO, comes later. The model generates several candidates, a reward function scores them and training favours the better ones. If that reward is weak, the model learns to exploit the judge instead of improving the product.
Model size is only the container for these choices. The real system is data, contract, evaluation, training, serving and feedback.
How Shopify’s flywheel works
Shopify has published more detail about the surrounding method in its Sidekick continual-learning report. The sequence is more instructive than any single benchmark:
- Define quality as a scored rubric covering task completion, execution, response quality and safety.
- Have domain experts label random production samples and measure whether they agree.
- Calibrate focused automated judges against that human ground truth.
- Improve the existing prompts, tool definitions and harness before changing model weights.
- Mine de-identified production failures and other hard negatives.
- Ask a panel of stronger models to critique each failure, merge the critiques into a repair and replay the trajectory.
- Keep repaired trajectories that pass the judge; send unresolved cases to people.
- Distil the accepted trajectories into a smaller model with SFT.
- Optionally apply GRPO using the calibrated judge as the reward.
- Retrain on old and new examples so the model learns new failures without forgetting previous behaviour.
Shopify also uses learned gist tokens to compress long, stable prompt instructions. That is prompt compression, not another name for making the model smaller. The model learns a short embedding sequence that reproduces the effect of a much longer fixed prefix.
Here, “recursive” describes an engineered pipeline rather than a model secretly rewriting itself in production: collect, judge, repair, review, train, evaluate, release. Each cycle starts with a stronger dataset and, if the release gate passes, a stronger student.
The warning hidden in Shopify’s other case study
Shopify’s separate Flow agent report is more useful than a success story because it includes a failed release signal.
The team fine-tuned Qwen3-32B to turn natural-language requests into Shopify Flow automations. The model looked competitive on a 300-example offline benchmark. At a one-percent production rollout, however, the rate at which merchants activated its generated workflows was 35% lower than for the previous agent.
Real requests included work that the synthetic benchmark missed. Small differences between training and production also mattered: tool names, tool ordering, JSON key ordering, response fields and system-prompt revisions all moved quality.
Shopify used production examples, a human-calibrated judge and weekly retraining to close the reported gap. The useful warning is that an offline judge can approve a model that users reject; Shopify’s recovery does not imply that every flywheel wins in two weeks.
This also prevents us from mixing two different Shopify stories. Buyer Profile uses the 0.8B result from Tobi’s slide. Flow used a 32B model and a different dataset, metric, workload and training system. One cannot validate the other.
A PLAYGRND task narrow enough to test
PLAYGRND has many possible AI surfaces. Most are too broad for a first specialised model. Saying that an assistant “knows football” leaves the product contract unresolved.
One candidate is narrow enough: a bilingual, read-only statistics tool router.
A user could ask in Croatian or English:
- “Who scored most often for this team last season?”
- “Koja je momčad primila najmanje golova u ovoj ligi?”
- “Show the last five meetings between these teams.”
- “Je li ovo pitanje o igraču ili ekipi? Treba mi pojašnjenje.”
The model would not write SQL and would not change an official record. Its output would be one approved read-only tool call with typed arguments, or an explicit request for clarification, refusal or fallback. The existing backend would resolve identities, enforce the schema and execute the query against canonical data.
This is a proposed experiment, not a feature we claim is live today.
It is a good first candidate for four reasons:
- the available tools and arguments can be enumerated;
- the result can be executed and checked deterministically;
- Croatian wording, inflection and local football vocabulary create real specialisation value;
- a failure can fall back to a larger model or normal search without altering the record.
Administrative writes would remain outside the first model. If a later system proposes a score correction or competition change, permission checks, source evidence and human confirmation would still run in deterministic application code.
The implementation plan
We would build this in stages. The first release decision happens before the first training run.
1. Freeze the task contract
Define a small versioned tool vocabulary and a single structured output. A request must produce a tool name, typed arguments, language, confidence category and one of execute, clarify, refuse or fallback.
The output schema should reject unknown tools, malformed identifiers and unsupported date ranges. Tool responses should be deliberately small and stable. If the serving tool is called player_season_summary, the same name and response shape must appear in every training trajectory.
2. Build the evaluation set first
Before generating training data, create a held-out set that represents the difficult product boundary:
- Croatian and English versions of the same intent;
- declension, dialect, typos and abbreviated club names;
- players and teams with similar names;
- missing season or competition context;
- questions that need two tools rather than one;
- unsupported predictions and unknowable facts;
- requests for private or unauthorised data;
- prompt-injection and malformed tool-output cases.
Split examples by competition, time and entity pattern, not by random near-duplicate sentence. Otherwise the test set measures memorisation of the same fixture in different words.
The score should remain decomposed: valid schema, correct tool, exact arguments, execution success, answer grounded in the returned record, appropriate clarification or refusal, Croatian/English parity, latency and cost. One blended judge score is useful for ranking, but not for diagnosis.
3. Establish three baselines
Run the frozen evaluation against:
- the unmodified small base model;
- the current frontier model and full harness;
- a deterministic keyword or rules baseline.
The third baseline matters. If a simple router handles 95% of the task, training a model may only add operational risk.
Qwen3.5-0.8B is an obvious candidate because it is the model in Tobi’s example and its published weights use the Apache 2.0 licence. We would still make Croatian performance a measured go/no-go condition rather than assume it from an English-language result. A 1.7B or 4B model may be a better quality-cost point.
4. Construct traceable training data
Start with product-owned examples and public or appropriately licensed text. Where production interactions are used, document their purpose and lawful basis, minimise and de-identify them, exclude sensitive free text and set retention rules.
A stronger teacher can generate several candidate tool trajectories under the exact production schema. Each candidate runs against an isolated, read-only database snapshot. Deterministic checks reject invalid tools, arguments and ungrounded answers. Domain reviewers inspect ambiguous and high-risk slices.
Every accepted example should retain provenance: source category, teacher and prompt version, tool-schema version, deterministic check result, human-review status and dataset split. A flywheel without lineage is an automated way to lose track of why a model behaves as it does.
Teacher use also has a contractual boundary. Before using outputs to train another model, the team must review the provider’s current terms, the base-model and dataset licences, privacy commitments and any third-party rights. Hidden model chain-of-thought should not be treated as training material. A compact, product-owned action trace and reviewed explanation are easier to govern.
5. Run LoRA SFT first
The first experiment should fine-tune adapters on accepted examples, while masking user and tool-result tokens that the model is not expected to generate. Keep a replay sample of ordinary and refusal cases in every run.
We would vary only a small set of parameters at first: model size, LoRA rank, learning rate, number of epochs and whether concise reasoning/action traces help. Each run evaluates the same frozen split. The winning checkpoint is the smallest model that clears every hard gate, not the one with the highest average score.
Tobi’s public QMD fine-tuning example shows how modest this phase can be. A Qwen3-1.7B LoRA trained on roughly 2,290 query-expansion examples in about 45 minutes on one A10G for approximately $1.50. That is a useful scale reference, not a runtime or quality forecast for PLAYGRND.
6. Distil failures, then consider RL
Once the SFT model runs in shadow mode, mine failures that deterministic checks or reviewers can explain. A stronger teacher can repair those trajectories; verified repairs return to the next SFT dataset.
GRPO becomes interesting only when most of the reward is executable: correct tool, valid arguments, successful query, grounded answer and correct abstention. If an LLM judge supplies the entire reward, reward hacking and judge drift become harder to detect.
Full-parameter fine-tuning is also a later experiment. It is justified when adapter training has clearly plateaued and the measured gain is worth a more expensive training and rollback path. Shopify runs daily full-weight training at enormous scale. That is not the starting point for a smaller product.
7. Release through shadow traffic
The model first receives copied, de-identified requests without answering users. Compare it with the existing path, inspect disagreement slices and measure tail latency. Then expose it to a small reversible percentage with automatic fallback.
Release gates should include zero unauthorised tool calls, high schema validity, a tested clarification rate, no statistically credible regression in grounded answers, bounded p95 latency and a lower total cost per successful request. A kill switch must return traffic to the previous path without a data migration.
Which GPU we would rent
For a 0.8B or 1.7B LoRA experiment, an H100 is usually the wrong first rental. A single 24GB A10G or L4-class GPU is enough to establish the pipeline. A 48GB L40S gives more room for longer context, larger batches or a 4B–8B candidate. An A100 80GB is useful when memory, not prestige, is the constraint.
Public rates on 3 September 2026 provide a planning envelope:
- Hugging Face lists A10G at $1.50/hour, L40S at $1.80/hour, A100 80GB at $2.50/hour and H200 at $5/hour.
- Runpod Secure Cloud lists L40S at $0.99/hour, A100 80GB at $1.39–$1.59/hour, H100 at $2.89–$3.29/hour and H200 at $4.59/hour.
- Lambda lists A10 at $1.29/hour, A6000 48GB at $1.09/hour and H100 PCIe at $3.29/hour.
At those rates, reserving two to eight hours for an initial experiment is roughly $3–$12 on a Hugging Face A10G or $3.60–$14.40 on its L40S, compute only. This is a budget box, not an estimate that training will finish in that time. Sequence length, dataset size, batch, validation frequency, checkpointing, failed runs, storage, egress, availability and tax change the total.
The expensive part may not be the student GPU at all. Teacher inference, repeated data generation, domain review and building a trustworthy evaluation set can dominate the budget.
H100 or H200 becomes rational for a large teacher, high-throughput generation, a much larger full-weight model or multi-GPU training. Shopify’s Flow team reports two nodes of H200 GPUs and a 12-hour run for Qwen3-32B. That is a description of their scale, not a hardware prescription for ours.
Mac Studio Ultra as the deployment baseline
Apple has just announced the M5 Ultra Mac Studio, with availability beginning 22 September 2026. The US base configuration starts at $5,499 with 96GB of unified memory. The larger chip configuration reaches 512GB and 1.2TB/s of memory bandwidth.
For one 0.8B model, that is a lot of computer. The raw weights occupy roughly 1.6GB at 16 bits or 0.4GB at 4 bits, before runtime and KV-cache overhead. Even the 96GB base M5 Ultra leaves substantial room for batching, several processes or larger fallback models.
That makes the Ultra a useful deployment baseline, not an automatic purchase recommendation.
It makes sense when PLAYGRND wants:
- predictable always-on inference without per-token billing;
- local control over prompts, model weights and transient inputs;
- several small specialised models on one machine;
- enough unified memory to test larger quantised fallbacks;
- low-latency access from a colocated application environment.
It makes less sense when traffic is low or irregular, the machine sits in an office without redundant power and network, or the team needs CUDA-specific serving and monitoring tools. A single Mac Studio is also a single point of failure.
We would train on rented NVIDIA hardware, merge the adapter, export and quantise the model, then benchmark both MLX and GGUF/llama.cpp-compatible paths on the real Mac. The benchmark must use PLAYGRND request lengths and concurrency, not a one-line token-speed demo.
Apple’s MLX-LM project supports LoRA, QLoRA, quantisation and local inference. Its own documentation says the included basic HTTP server is not recommended for production because it implements only basic security checks. A production deployment therefore needs a hardened internal service in front of the runtime: authentication, request limits, queueing, timeouts, metrics, structured logs, model/version pinning and health checks.
The minimum reliable shape is one primary inference service plus a cloud or frontier-model fallback. A more serious self-hosted setup uses a second node, remote power and recovery, tested model rollbacks and enough observability to distinguish queueing, prompt processing and generation latency.
The purchase calculation should compare the $5,499 US starting price, local taxes and operations against measured cloud inference cost. Without steady load or a data-control reason, renting remains more flexible. With steady load, several models and an existing operational environment, local inference can become the simpler bill.
What would make us stop
We would not continue because the fine-tune is technically interesting. We would stop or stay with the frontier path if:
- Croatian accuracy fails the held-out slices;
- the rules baseline is already good enough;
- production requests change faster than the dataset can be governed;
- the small model requires so many fallbacks that it does not reduce cost or latency;
- reviewers cannot agree on what a correct answer is;
- the data or model licence does not fit the intended use;
- operating local inference costs more attention than it returns.
The recursive flywheel is an advantage only when every turn produces trustworthy signal. Feeding uncertain labels back into weights compounds uncertainty just as efficiently.
Our view
Tobi’s result matters because it shows where model competition is going next. Frontier models remain the best way to explore an uncertain task and generate difficult candidate solutions. Once the task stabilises, their own output and production failures can help train a much smaller worker.
Qwen’s base weights are public and GPU time is rentable, so the 0.8B checkpoint alone provides little defensibility. Durable advantage comes from the product-specific contract, representative data, a judge aligned with domain experts, the deterministic execution environment and the speed at which a team can turn a real failure into a safe release.
For PLAYGRND, we would begin with one read-only bilingual statistics skill, not a general football brain. We would keep the canonical record and permissions in application code, let the small model do only the fuzzy language-to-tool step and require evidence before replacing the frontier baseline.
Small models win when the problem has become precise. Making the problem precise is still the engineering work.
Hardware specifications, availability and public prices in this article reflect sources available on 3 September 2026 and can change. Shopify’s figures are its own reported results and are not guarantees for another workload. The PLAYGRND section is a technical proposal, not a statement of a deployed capability. This article is not legal, privacy or procurement advice. If you are evaluating a specialised model and production feedback loop, talk to HILLS Lab.