Measuring Impact When You Can’t A/B Test — a Quick and Practical Guide to Causal Inference
Last Updated on September 22, 2026 by Editorial Team
Author(s): Jonty Haberfield
Originally published on Towards AI.
A tour of methods including Propensity Score Matching, Double ML, Instrumental Variables, and Two Way Fixed Effects
What do you do if you want to separate correlation from causation — if you want to know the actual, causal, impact of…something? A new drug, a re-designed app, a loyalty scheme.
After introducing why causal impact is hard without random assignment, the article builds a running “loyalty program” example and shows how naive comparisons overestimate uplift due to selection bias from confounders. It then walks through methods under the key assumption of no unobserved confounders: OLS regression (works only with a correctly specified linear world), Propensity Score Matching (pairing treated and control “twins” via propensity overlap, yielding an ATT-style estimate), IPTW (reweighting to emulate an RCT but sensitive to accurate propensity scores and producing an ATE), and Double ML (residualizing out confounders using cross-fitting and estimating a treatment effect close to the true value). The author clarifies what each estimand means (ATE vs ATT vs IPTW’s ATE vs IV’s LATE) and discusses when to choose each approach. In Part 2, the focus shifts to unobserved confounders (e.g., “intrinsic loyalty”), exploring mitigation via proxies and their attenuation-bias limitations, and then using encouragement designs with instrumental variables (random email nudges) to estimate a local average treatment effect, while highlighting assumptions like monotonicity and wide confidence intervals when only compliers drive the signal. Finally, Part 3 covers panel data, explaining two-way fixed effects as a difference-in-differences-with-fixed-effects approach, what it can solve (controlling time-invariant customer traits), and the remaining pitfalls (parallel trends, time-varying confounders, reverse causality, and dynamic/heterogeneous treatment effects), closing with practical takeaways about avoiding naive comparisons and matching the method to the estimand and data constraints.
Read the full blog for free on Medium.
Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.
Published via Towards AI
Towards AI Academy
We Build Enterprise-Grade AI. We'll Teach You to Master It Too.
15 engineers. 100,000+ students. Towards AI Academy teaches what actually survives production.
Start free — no commitment:
→ 6-Day Agentic AI Engineering Email Guide — one practical lesson per day
→ Agents Architecture Cheatsheet — 3 years of architecture decisions in 6 pages
Our courses:
→ AI Engineering Certification — 90+ lessons from project selection to deployed product. The most comprehensive practical LLM course out there.
→ Agent Engineering Course — Hands on with production agent architectures, memory, routing, and eval frameworks — built from real enterprise engagements.
→ AI for Work — Understand, evaluate, and apply AI for complex work tasks.
Note: Article content contains the views of the contributing authors and not Towards AI.
Related posts
Recent Posts
