A variant label in an event stream is only as trustworthy as the path that wrote it. Before celebrating a lift, we run a short set of instrumentation checks.
Five checks we keep repeating
- Does every exposed user carry exactly one assignment for the experiment key?
- Do assignment timestamps precede the outcome events you care about?
- Are holdout or control users excluded from treatment-only surfaces?
- Do timezone cuts match the warehouse table and the product clock?
- Can you reproduce the primary metric from raw events without hand edits?
When a check fails
Stop arguing about lift. Fix the measurement path first, then decide whether the experiment window can continue. That sequence is the core of an Experiment Measurement Audit.