Interactive demo · synthetic dataEvery record on this page is fabricated.

Master File Report Pipeline

Run the upsert the docstring described and the date-overlap replace the code does, over the same rows.

The system behind this demo was engineered by Prada Dipa — LinkedIn profile, opens in a new tab and Luthfi Aditya — LinkedIn profile, opens in a new tab. My part — manager and technical lead, technical & product direction, set the connector / transform / merge rule that later became the data platform's layout contract, required a written decision record wherever behaviour is surprising — including the merge whose docstring is wrong, directed the move from notebooks to an interface, and the containerised deployment that followed.

Read about the system

The merge, running in your browser

Pick a policy and a window. The third frame is what the orders tab would hold afterwards.

MERGE POLICY
THE WINDOW THIS RUN WAS ASKED FOR

WHERE THE TWO POLICIES DISAGREE

One row. Everything else is identical under both policies.

  • deleted by the date-overlap replace, kept by the upsert — it is inside the window and the pull did not return it.

ROWS AFTER

9

UPDATED

1

ADDED

2

DELETED

1

STALE

0

DUPLICATED

0

ORDERS TAB, BEFORE

8 rows, written by yesterday's run

  1. MER-0998$268

    2026-02-26 · Halvard Ness · Shopify

    OUTSIDE WINDOW

  2. MER-0999$742

    2026-02-28 · Ileana Cotrus · Salesforce

    OUTSIDE WINDOW

  3. MER-1001$149

    2026-03-02 · Junus Abelard · Shopify

    IN WINDOW

  4. MER-1002$402

    2026-03-03 · Perpetua Vance · Shopify

    IN WINDOW

  5. MER-1003$516

    2026-03-05 · Osian Tremaine · Salesforce

    IN WINDOW

  6. MER-1004$288

    2026-03-07 · Renske Bouwmans · Shopify

    IN WINDOW

  7. MER-1005$655

    2026-03-09 · Taddeo Marchetti · Shopify

    IN WINDOW

  8. MER-1006$131

    2026-03-10 · Vanja Kolarov · Salesforce

    IN WINDOW

NEW PULL

7 rows returned by the connector

  1. MER-1001$149

    2026-03-02 · Junus Abelard · Shopify

  2. MER-1002$361

    2026-03-03 · Perpetua Vance · Shopify

  3. MER-1003$516

    2026-03-05 · Osian Tremaine · Salesforce

  4. MER-1005$655

    2026-03-09 · Taddeo Marchetti · Shopify

  5. MER-1006$131

    2026-03-10 · Vanja Kolarov · Salesforce

  6. MER-1007$294

    2026-03-11 · Wieslaw Dombrow · Shopify

  7. MER-1008$1,032

    2026-03-12 · Yesenia Alcaraz · Salesforce

ORDERS TAB, AFTER

date-overlap replace

  1. MER-0998$268

    2026-02-26 · Halvard Ness · Shopify

    CARRIED

  2. MER-0999$742

    2026-02-28 · Ileana Cotrus · Salesforce

    CARRIED

  3. MER-1001$149

    2026-03-02 · Junus Abelard · Shopify

    UNCHANGED

  4. MER-1002$361

    2026-03-03 · Perpetua Vance · Shopify

    UPDATED · was $402

  5. MER-1003$516

    2026-03-05 · Osian Tremaine · Salesforce

    UNCHANGED

  6. MER-1005$655

    2026-03-09 · Taddeo Marchetti · Shopify

    UNCHANGED

  7. MER-1006$131

    2026-03-10 · Vanja Kolarov · Salesforce

    UNCHANGED

  8. MER-1007$294

    2026-03-11 · Wieslaw Dombrow · Shopify

    ADDED

  9. MER-1008$1,032

    2026-03-12 · Yesenia Alcaraz · Salesforce

    ADDED

  10. MER-1004$288

    2026-03-07 · Renske Bouwmans · Shopify

    DELETED · the pull did not return it

DECISION RECORD · THE MERGE

The merge is a date-overlap replace, and its docstring says upsert

CONTEXT

The merge is the third of the pipeline’s three layers — a connector that paginates and returns flat rows, a pure transform, and this. Its docstring described a per-order upsert keyed on the order id. It has never been one. It deletes every previous row whose order date falls inside the window the run was asked for, and concatenates the pull.

DECISION

Record the behaviour rather than quietly change it. These tabs are read directly by people doing their jobs, and a merge that starts behaving differently is a change to what everyone downstream sees. The record names the policy, the two ways it goes wrong, and the fact that the docstring is a trap — so the next person meets the trap in a document instead of in a report.

CONSEQUENCES

  • An order genuinely removed upstream disappears from the tab, which is usually what was wanted and is the reason nobody noticed.
  • An order the pull merely failed to return — cancelled, or excluded by a status filter — also disappears. That is MER-1004 above.
  • Correctness depends on the declared window and the pull agreeing about their edges. Where they drift, the same order is written twice.

WHY THIS IS ON THE PAGE AT ALL

This is the smallest project in the portfolio and it is here for one reason: a function whose name and whose behaviour disagree is a trap until somebody writes it down, and the widget above is that record made executable. Thirty tests assert it. You can change the window yourself and watch the claim hold or break.

Restated in my own words. The original records are in a private repository and name internal systems; nothing quoted from them appears here, and neither do any of their examples.

DECISION RECORD · THE TOKEN REFRESH — STATIC EXPLAINER

The refresh mutates the shared header dictionary in place

The storefront token is short-lived and a full run outlives it. Several modules import the header dictionary once, at import time. So the refresh has to update the object those modules are already holding — not rebind a name.

WHAT IT DOES

def refresh(headers):
    token = exchange_client_credentials()
    # mutate the object every importer
    # is already holding a reference to
    headers["X-Access-Token"] = token

Every module that imported HEADERS from the auth module sees the new token immediately. No restart, no re-import, no plumbing a token through forty call sites.

THE TIDY-UP THAT BREAKS IT

def refresh():
    global HEADERS
    # rebinds the name in THIS module
    HEADERS = {
        "X-Access-Token":
            exchange_client_credentials()
    }

Importers bound their own name to the old dictionary at import time. Rebinding the module global leaves them pointing at it, still sending an expired token — and only after the token's lifetime, part-way through a long run.

It is worth a record because the correctness lives in a property of Python’s import system rather than in anything visible at the call site, and because the version that breaks it looks tidier.

There is no interactive version of this, deliberately. Demonstrating it would mean reproducing module-level name binding in a browser, and the visitor would be watching my imitation of Python rather than the thing itself. A widget that convincing about a claim it cannot support is worse than a paragraph.

What is real, and what is simulated

Runs in your browser

  • The merge step as a pure function — delete every previous row whose order date falls inside the run's window, then concatenate the pull
  • The per-order upsert the docstring described, beside it, so the disagreement is a computation and not an assertion
  • The comparison itself: which order ids the two policies differ on, counted rather than eyeballed

Not a line-for-line port, and not claimed as one: the production merge is Python in a private repository, so unlike some demos here there is no digest to compare against. What runs is the policy the decision record names, rebuilt as a pure function, with 30 tests asserting each sentence that record makes — including that the two policies differ on exactly one row when the window matches the pull, and on two when it does not.

Simulated

  • Shopify GraphQL, Salesforce, the Redo Returns API, gspread and the four destination tabs
  • The Streamlit multipage interface, the containerised deployment and the run lock that stops two people starting the same job
  • The connector's status filter. The reason MER-1004 is missing from the pull is stated on this page, not executed by it
  • Every order, name, email and figure — all fabricated, all emails on the reserved .invalid domain