02AI & Automation
A deterministic rules engine with an LLM allowed only to explain it
A direct-to-consumer apparel brand reviewed paid media weekly across three funnel stages. The work was mechanical — open each stage, read the metrics per ad, decide keep or pause — but the judgement was inconsistent between reviews, because the caps were recalled rather than written down.
There was also a failure mode nobody was systematically checking: pausing an underperforming ad can leave a product with no live ad at all. That is a merchandising problem disguised as a media-buying decision.
A staged pipeline: connector → cache → transform → metrics → rules → coverage → report. The threshold file is the pivot — it feeds the rules engine, the spreadsheet column templates and the LLM prompt from one place, which is what keeps the three from drifting apart.
This is the load-bearing decision of the whole project. A pluggable rules engine loads pause thresholds from a single JSON config, applies them per funnel stage and sub-stage, and enforces a seven-day grace period so a new ad is never flagged. That engine — and only that engine — decides.
The LLM is a narrative analyst layered on top. It receives the batch summary, the ads the engine flagged, the ads paused recently, each ad's product and remaining coverage count, and the engine's one-line verdict. It writes the review. It cannot set a threshold.
The system prompt is generated at import time by rendering the threshold table directly out of the same JSON the engine reads. The model is structurally incapable of citing a cap that differs from the one enforced, because both come from the same file.
Output is constrained to a fixed verdict/action/stance matrix: exactly three bullets per ad, the breach bullet and the reasoning bullet must cite two different metrics, hedging language and invented numbers are banned.
For ads that are already paused, the recommended action is computed deterministically in Python from rules status, age and spend — and any model bullet containing an action marker is stripped. The LLM cannot reintroduce a recommendation that contradicts the engine, even if it tries.
A missing API key or a failed call degrades to an empty analysis section rather than breaking the report. The deterministic review still ships.
Prompt versions v4 through v7 are retained in-tree as source files. v7 adds a holiday override that may only convert a pause into a keep when three conditions hold at once: a curated shopping holiday falls inside a 30-day window, the breach is at most 10% over cap, and the model can quote a theme token from the ad name tying the product to that holiday.
That is what a prompt looks like when it is treated as production logic rather than a text box — bounded, versioned, and reviewable in a diff.
A product-coverage index parses the base product out of each ad name and counts how many other live ads still serve it. A pause recommendation carries that count, so the review can never silently recommend taking a product's last ad offline.
An LLM asked to justify a pause will justify a pause — convincingly, and sometimes with a number it invented. The fix was structural rather than prompt-tuning: render the caps into the prompt from the engine's own config, compute the action for paused ads in Python, and strip any action marker the model emits. Constrain the surface, not the wording.
The thresholds feed the rules engine, the spreadsheet column templates and the prompt. Any one of those drifting makes the review quietly wrong. Making all three read the same JSON at import time was the cheapest possible guarantee that they agree.
An ad paused two days before the review disappears from the default insights window, taking its context with it. A separate two-step fetch recovers those ads so the report can say what happened to them rather than silently dropping them.
The model is gpt-4o-mini in JSON mode, called once per ad-set bucket. It never sees creative imagery — thumbnails are downloaded for the report, not sent to a model, and there is no vision call anywhere in the repo. Its entire job is to turn a deterministic verdict into three cited bullets a media buyer will actually read.
Replaces the manual weekly pass: pulling per-stage performance, comparing each ad against remembered thresholds, deciding keep or pause, and re-typing the result into a document. The pipeline does fetch, evaluation, coverage check and write-up in one run, and emits the review as Markdown, JSON, self-contained HTML and PDF, plus an optional formatted Google Sheet with creative thumbnails rendered in-cell.
Four features each shipped with a dated design document stating locked decisions and component boundaries, paired with a separate step-by-step implementation plan — roughly 5,000 lines of design writing against an 11,700-line codebase. Work landed through feature branches merged by pull request with Conventional Commit subjects; the Sheets export was decomposed into a dozen commits that each ship one component rather than one large drop. Several code comments cite stakeholder review as the reason for a behavioural change.
Objective size signals taken from the repository. No impact metrics are claimed that the source does not prove.
Internal reporting pipeline. Reports contain live account performance and are not published.