Measurement Framework

This is the page to send to your analytics or data science team before your first JustAI send. It covers how users are assigned to control and treatment, what holdout designs are available, how events are attributed, what identifier-level data you can get, and how to analyze it.

If you’re the marketer setting the campaign up, the Decision checklist at the bottom is the short version: lock those six choices before the first send and the analysis works.

The short answer

Question	Answer
Can JustAI produce a list of treatment vs. control identifiers per send?	Yes. Every request is logged with the recipient identifier, template, variant, tracking ID, timestamp, and a control flag.
Is the control group persistent across sends?	Only if the template uses persistent bucketing. Persistence is per template, not per program.
Is there a true no-treatment holdout?	Yes — a control arm can be configured to send nothing at all.
Does optimization use last-click?	No. Conversions are attributed at the user level within a time window, whether or not the user clicked.
How does my team get the data?	Consult with the JustAI team on the path — the right one depends on your ESP, your warehouse, and what your analysts need.

How users are assigned

Assignment happens on every /api/generate request, before any copy is selected.

Where the split happens

Split location	How it works	Trade-off
Internal (recommended)	JustAI receives all traffic and assigns each recipient to the control arm or a treatment variant.	JustAI sees both arms, so lift can be broken down by segment on the control side as well as the treatment side.
External	Your ESP holds part of the audience back and only sends the rest to JustAI.	Answers “did JustAI beat control overall?” but not “within which segment?”, because JustAI never sees the control-side attributes. Whether the same people stay in the holdout across sends depends entirely on how that audience is built in your ESP.

Unit of diversion

With an internal split, the unit of diversion decides whether a recipient’s arm is stable over time:

• Random — each request draws independently. A recipient can be control on Monday and treatment on Tuesday.
• Persistent — the arm is a deterministic function of the recipient. The same person lands in the same arm on every send of that template, indefinitely.

Persistent bucketing hashes org · user_id · template_id · salt with MurmurHash3 and normalizes the result to [0, 1), then walks the weighted buckets. Three consequences your analysts should know:

• It requires a stable user_id. If you send a per-message identifier rather than a durable customer ID, “persistent” degrades to random. Send the same ID you’ll join on in your warehouse.
• Persistence is scoped to a single template. The template ID is part of the hash, so the same person can be control in one campaign and treatment in another. That is fine for per-campaign lift, but it is not a program-level holdout.
• Saving a split change re-randomizes the cohorts. The salt is not something you have to edit deliberately to lose continuity: while the split stays internal and persistent, saving the Configure screen with any difference in the A/B split — the control ratio, the bucket weights, the split method — mints a fresh salt and reshuffles who sits in which arm. The console warns you and offers to log the change as a reversion, but it re-randomizes either way, so choosing to save without one leaves no marker in the template’s event history. Treat every such save as an analysis boundary and don’t pool data across it.

Note

If you want to measure whether ongoing personalization produces cumulative lift, you need a group that is held out everywhere, not just on one template. That is usually built as an audience in your ESP and excluded from every JustAI-integrated campaign, so it stays consistent across the whole program. It costs you reach on every campaign, so size it as the smallest group that still reads significant on your primary KPI.

Split ratio

The control share is configured per template (inheriting an org default). Common settings are 10%, 20%, or 50% control. The treatment share is then divided among active variants by the bandit, which shifts traffic toward better performers over time.

Caution

Because the bandit reallocates treatment traffic continuously, variant-vs-variant comparisons are not a fixed-allocation A/B test and their traffic shares change daily. Control-vs-treatment is a clean randomized comparison and is the one to use for lift claims.

Holdout designs

There are three designs, and they answer different questions.

Design	Where the split lives	Holdout receives	Question it answers	Control-side segmentation visible to JustAI	Stable across sends
BAU control in JustAI	JustAI	Your existing creative	Does JustAI copy beat our copy?	Yes	Yes, with persistent bucketing
BAU control in your ESP	Your ESP	Your existing creative	Does JustAI beat control overall?	No	Depends on how the ESP audience is built
No-send holdout	JustAI	Nothing at all	What is the incremental effect of sending at all?	N/A	Yes, with persistent bucketing

BAU control is the common choice. It isolates the lift from the copy, holding send volume, timing, and audience constant.

No-send holdout is configured by making the control arm a send-skip marker: the recipient is assigned, logged, and then nothing is dispatched.

Caution

A no-send holdout is fully tracked at the assignment level, but by definition it generates no ESP engagement events — there is no open or click on an email that was never sent. Its outcomes are only measurable against your own conversion data, joined on user_id. Plan the warehouse join before you launch, or the arm will look empty rather than null.

Running both at once is reasonable and is often the right call for a true incrementality read: a small pure no-send holdout sized for significance on the primary KPI, and a BAU control/treatment split across the remaining audience. Budget the volume deliberately — every arm you add costs reach, and the no-send arm costs revenue you would otherwise have earned.

What is recorded per send

Each generate request writes one record. These are the fields available for identifier-level export:

Field	Description
event_timestamp	Unix timestamp of the request
org_slug	Your JustAI org
user_id	The recipient identifier you sent us — customer ID or email, whichever you pass
template_id	The JustAI template (campaign touchpoint)
tracking_id	The message/send identifier, used to join to ESP engagement events
copy_id	The specific variant served
bucket_is_control	The arm flag: true for control, false for treatment
bucket_name	The named bucket, if your split defines more than two arms
request.features	The recipient’s normalized personalization attributes as of the request, e.g. lifecycle stage or locale. This is the field to segment on — see Attributes vs Fields
features	The targeting attributes of the variant that was served. Frequently empty, and not a substitute for request.features
ESP template ID	The ESP-side template served, where the integration uses one

Engagement events (opens, clicks, unsubscribes) arrive separately from your ESP and carry the identifiers of the specific message, so they join back on tracking_id. Conversion events you forward to JustAI join on user_id.

Tip

user_id is the single most important field to get right. It is the bucketing key, the conversion join key, and the key your analysts will use in the warehouse. Pick one durable identifier and use it consistently across the JustAI request, your conversion events, and your warehouse tables.

Attribution

Attribution answers which send gets credit for this event. JustAI uses two different methods depending on the event type, and the difference matters for reconciliation.

Engagement events: direct

Opens, clicks, and unsubscribes are generated by your ESP, which knows exactly which message triggered them. These tag back to that exact send, variant, and recipient. There is no ambiguity and no window.

Conversion events: user-level, windowed

Conversions happen in your product, not in the email, so there is no identifier tying them to a message. JustAI attributes them at the user level:

1. A conversion event arrives for a user.
2. JustAI looks up the sends that user received within the attribution window.
3. Those sends are credited.

This is not last-click. A user who never opened the email but converted within the window is counted. If your CRM reporting has historically been last-click, expect JustAI’s conversion counts to be materially higher than what you’re used to, and don’t try to reconcile the two line-for-line — they are measuring different things. The comparison that stays valid either way is control vs. treatment computed the same way on both sides.

Two parameters are configured per integration during onboarding:

• Window length — commonly 24 or 48 hours. It should match how your team defines the KPI. If your purchase cycle is a week, a 24-hour window will undercount.
• Crediting rule — either all qualifying sends in the window share credit, or only the most recent qualifying send is credited (last-touch). Ask your JustAI contact which applies to your account; both are in production for different customers.

Note

Under an all-sends rule, one conversion can be credited to more than one template, so summing conversions across templates double counts. Under a last-touch rule it cannot, but a campaign that sends shortly after another will absorb credit from it. Either way, the same rule applies to every variant and every arm, so control-vs-treatment lift stays unbiased — the distortion is in absolute totals, not in the comparison.

Scoring windows are a separate thing

Two more parameters govern which sends feed the optimizer — the scoring the bandit allocates traffic on — and they are frequently mistaken for the attribution window:

• lookback_days (default 14) — how far back sends are aggregated when scoring variants.
• offset_days (default 3) — a maturity period before a send’s metrics count, so recent sends aren’t judged on conversions that haven’t landed yet.

The dashboard does not use these. A dashboard read is a date range: the range you select, or — with nothing selected — the template’s full history through three days ago, narrowed to the most recent reversion only when you’re looking at a specific milestone. So the dashboard normally spans the whole experiment rather than a rolling 14-day window.

That leaves three distinct windows, and reproducing any number means knowing which one produced it: the attribution window (which conversions belong to a send), the optimizer’s lookback (which sends the bandit scores), and the dashboard’s date range (which sends you are looking at). Analysts recomputing from raw exports should match the dashboard’s date range, not the optimizer’s lookback — matching the wrong one is the usual reason the numbers disagree. Agree on which surface is the source of truth before the first readout.

Getting the data into your warehouse

Three paths, in increasing order of analyst control:

1. In-app analytics — the template Overview and Analytics tabs, with control/treatment lift, per-variant breakdowns, segment analysis, and significance. Good for the marketer, not sufficient for a custom incrementality study.
2. API and MCP — programmatic access to template metrics, for dashboards or scheduled jobs on your side.
3. Identifier-level data delivery — the per-send records above, landed somewhere your analysts can query them. This is what a cohort lift or incrementality study needs, and it is the one to consult with the JustAI team on the path: the right mechanism depends on your ESP, your warehouse, and which fields you actually need. Bring your data team to that conversation.

Have the conversation alongside campaign setup rather than after the first send. Data delivery can usually be sorted out after the fact; a decision about the unit of diversion cannot — once sends have gone out under the wrong bucketing, that period is not recoverable.

Analyzing the results

Once the data is in your warehouse:

• Pick the unit of analysis deliberately. Per-send rates answer “did this message perform better”; per-user rates answer “did this program work”. With persistent bucketing, a user appears in many sends, so per-send rows are not independent — cluster standard errors by user, or aggregate to one row per user, before testing significance. Treating repeat sends as independent will overstate confidence.
• Compute lift as a relative difference on the same metric definition for both arms: (treatment_rate − control_rate) / control_rate.
• Segment on pre-treatment attributes only. Slicing by anything determined after assignment — opened, clicked, converted — reintroduces the selection bias the randomization removed.
• Filter the date range at analysis boundaries. New variants, a milestone/ship event, or any saved change to a persistent split all reset the baseline. Mixing periods across those boundaries blends different experiments.
• Check the arms are the size you expect — and read a mismatch carefully. Assignment happens inside JustAI, on the request, so filtering the audience upstream changes who is in the study without making the split within the traffic we receive any less random. A small deviation from the configured ratio is ordinary sampling variation. A large one points at a config change mid-flight (a new salt or ratio), assignments that aren’t being logged, or something dropping one arm after assignment — check those before concluding the randomization failed.
• Power the no-send arm on the primary KPI, not on opens. A holdout sized for a click-rate read will usually be far too small to detect a conversion difference.

Decision checklist

Lock these before the first send:

1. Identifier — which user_id you’ll pass, and that it matches your warehouse and your conversion events.
2. Unit of diversion — random or persistent. Choose persistent for anything multi-touch or cumulative.
3. Holdout design — BAU control, no-send holdout, or both; and the split percentages.
4. Scope of persistence — per template, or a program-wide holdout audience in your ESP.
5. Conversion event and window — which event counts, over what window, under which crediting rule.
6. Data delivery — the path agreed with the JustAI team, and who on your side owns the analysis.

Related resources

• Conversion Events — setting up and verifying event forwarding
• Reading Results — interpreting the in-app dashboard
• Template Configuration — where split method, ratio, and significance thresholds are set
• How the Bandit Works — how treatment traffic is allocated across variants
• AWS Integration — shared S3 bucket and IAM setup for data exchange