Imagine Facebook introduces a new storefront feature allowing UK creators to sell products directly from their profiles. You want to measure whether this new feature actually helps creators earn more money. To understand this, you'd ideally compare:

What actually happened (with storefronts).
What would have happened if storefronts had never been introduced.

The second scenario—called the counterfactual—is impossible to directly observe. That’s precisely the problem Difference-in-Differences (DiD) is designed to solve.

What is Difference-in-Differences?

DiD estimates a causal effect by combining two intuitive comparisons:

Before vs. After:
How did creators' revenue in the UK change after introducing the storefront compared to before?
Treated vs. Control:
How does the revenue change in the UK (the "treated" group) compare to similar markets (like Canada or Australia) that never received the feature (the "control" group)?

Why Randomised Controlled Trial (RCT) Can't Work?

A Randomized Controlled Trial (RCT)—often considered the gold standard for causal inference—works by randomly assigning participants or units into treatment and control groups. This randomization ensures, in theory, that the groups are comparable, allowing a clean measurement of the treatment’s causal impact.

However, RCTs aren’t always practical or ethical, as in the Creator Storefront case. Here are three reasons that explain why RCT isn't feasible:

Ethical Concerns:
Randomly withholding monetization tools from eligible creators could be viewed as unfair, harming their livelihood and trust in Facebook.
Operational Complexity:
Randomization at a creator level would complicate backend implementation and user experience, especially if creators interact or share audiences.
Network Effects & Spillovers:
Creators and their audiences overlap; treating some but not others risks contamination, violating the assumption of independent treatment effects.

The Math Behind DiD (Step-by-Step)

Notations

Let’s define the setup for Difference-in-Differences:

\(Y_{it}\): Outcome for unit \(ii\) at time \(tt\)
\(D_i \in \{0, 1\} \): Indicator for treatment group
\(D_i = 1\) if unit \(ii\) belongs to the treatment group; 0 otherwise
\(T_t \in \{0, 1\} \): Time indicator
\(T_t = 1\) if time tt is after the treatment; 0 otherwise

We observe outcomes across two groups (treated and control) and two time periods (before and after treatment).

Basic DiD Estimator

Consider the following outcomes.

Group	Before ( $T=0$ )	After ( $T=1$ )
Treated	$Y_{10}$	$Y_{11}$
Control	$Y_{00}$	$Y_{01}$

The Difference-in-Differences (DiD) estimate is computed as:

$$\text{DiD} = (Y_{11} - Y_{10}) - (Y_{01} - Y_{00})$$

Where:

\(Y_{11}=E[Y∣D=1,T=1]Y_{11} = E[Y \mid D=1, T=1] \): Treated group, after treatment
\(Y10=E[Y∣D=1,T=0]Y_{10} = E[Y \mid D=1, T=0] \): Treated group, before treatment
\(Y01=E[Y∣D=0,T=1]Y_{01} = E[Y \mid D=0, T=1] \): Control group, after treatment
\(Y00=E[Y∣D=0,T=0]Y_{00} = E[Y \mid D=0, T=0] \): Control group, before treatment

This formula compares the change in the treated group to the change in the control group.

Regression Framework

The DiD estimator can also be derived from the following linear regression model:

$$Y_{it} = \alpha + \beta D_i + \gamma T_t + \delta (D_i \cdot T_t) + \varepsilon_{it}$$

Where:

α: Baseline outcome for the control group in the pre-treatment period
β: Group fixed effect (captures time-invariant differences between groups)
γ: Time fixed effect (captures time-specific shocks)
δ: Treatment effect — the DiD estimator
\(ε_{it} \): Error term

The coefficient \(\delta\) on the interaction term \(D_i \cdot T_t \) captures the causal effect of the treatment under the parallel trends assumption.

Applying DiD to the Facebook Storefront Example

Problem Statement

Facebook launched the storefront feature exclusively in the UK market in January 2024, with other comparable markets such as Canada and Australia serving as natural controls where the feature was not available during the study period.

Business Question:

Did the introduction of the storefront feature increase average revenue per creator in the UK relative to markets that did not receive the feature?

We use a Difference-in-Differences (DiD) framework to estimate the treatment effect of the storefront launch, comparing pre-post changes in the UK against those in control markets over the same period.

Framing the Problem

Let's put real numbers on this to clarify further:

Market	Before Revenue	After Revenue	Change
UK (treated)	$126	$136	+$10
Canada/Australia (control)	$110	$105	-$5

We then compute the DiD estimate:

$$\text{DiD Effect} = (Y_{11} - Y_{10}) - (Y_{01} - Y_{00})$$

$$\text{DiD Effect} = (+\$10) - (-45) = +\$15$$

The DiD analysis suggests that UK creators earned, on average, $15 more per month than they would have if the UK had followed the same trajectory as Canada and Australia, indicating the storefront feature likely increased creator revenue. But, is this result statistically significant?

Regression Analysis

Apply Difference-in-Differences (DiD) regression to estimate the impact of the UK storefront rollout on average creator revenue.

The model becomes:
$$\text{AvgRevenue}_{ct} = \alpha + \beta \cdot \text{UK}_c + \gamma \cdot \text{Post}_t + \delta \cdot ( \text{UK}_c \times \text{Post}_c) + \varepsilon_{ct}$$

Where \( \delta \cdot ( \text{UK}_c \times \text{Post}_c) \) is the interaction that determines the storefront effect.

# Interaction model equation
model_formula = 'AvgRevenuePerCreator ~ UK + Post + UK*Post'

# Fit the regression model
model = smf.ols(formula=model_formula, data=df).fit()

# Output the summary results of the model
print(model.summary())

Here are the key findings from the OLS regression results in bullet form:

Significant positive treatment effect:
The interaction term UK:Post is statistically significant (coefficient = 14.92, p = 0.030), indicating that the storefront rollout increased average revenue per UK creator by approximately 15 units relative to the control markets.
Higher baseline revenue in the UK:
The main effect for UK is significant (coefficient = 16.02, p = 0.002), suggesting that UK creators already earned more than their counterparts in control markets before the treatment. This underscores the importance of using a Difference-in-Differences framework.
No significant platform-wide trend:
The Post coefficient is negative but not statistically significant (coefficient = -5.75, p = 0.139), suggesting there were no global changes in revenue across all markets that would bias the treatment effect.
Good model fit for a DiD analysis:
The R-squared value is 0.638, meaning approximately 64% of the variance in average revenue per creator is explained by the model. The overall F-statistic is also highly significant (p < 0.001), supporting the validity of the model.

Parallel Trend Assumption

A critical assumption behind DiD is the parallel trends assumption. This means that without the storefront, UK creator revenue would have evolved similarly to control markets. Before introducing the feature, both treated and control markets should have similar revenue trajectories.

To ensure this assumption holds, we typically:

Look at historical revenue trends before the storefront launch.
Visually or statistically verify that the trends between treated and control markets were roughly parallel.

If this assumption doesn’t hold, the DiD estimate might not accurately capture the causal effect.

# --- keep only pre-treatment observations ---
pre_df = df[df["Post"] == 0].copy()

# create a numeric month index 0,1,2,… in chronological order
unique_months = sorted(pre_df["Month_dt"].unique())
month_to_idx   = {m: i for i, m in enumerate(unique_months)}
pre_df["TimeIndex"] = pre_df["Month_dt"].map(month_to_idx)

# DiD slope-check: does UK have a different *trend* before treatment?
model = smf.ols(
    "AvgRevenuePerCreator ~ UK + TimeIndex + UK:TimeIndex",
    data=pre_df
).fit(cov_type="HC1")          # robust s.e.

print(model.summary())

Key Findings:

The non-significant UK:TimeIndex interaction (p = 0.300) suggests no evidence of divergent pre-treatment trends.
This supports the validity of the Difference-in-Differences design.
R-squared of 0.823 indicates strong model fit for a pre-trend check, but significance of the interaction term is what matters here.

Summary

In short, DiD provides a simple yet robust way to estimate causal effects when direct experimentation isn’t feasible:

Clearly define treated and control groups.
Identify before-and-after time periods.
Calculate how each group's outcomes change across these periods.
Use the DiD formula to isolate the true treatment effect.

When used thoughtfully and accompanied by careful assumption checking, DiD is an invaluable tool for causal inference, clearly illustrated by the Facebook storefront example above.