Why Machine Learning Demands Causal Inference

[Series] Causal Machine Learning - 1. Rethinking the Predictive Paradigm

By Kuriko IWAI

Table of Contents

Standard machine learning (ML) models operate on observational data, making them suitable to answer what will happen under the current data-generating distribution, but failing to predict what would happen if an operational policy actively alters that distribution.

Causal inference tackles this challenge by shifting the objective from passive observation to structural intervention.

This article dives deep into its mathematical foundations, hidden structural traps, and practical Python implementations of integrating causal inference into production machine learning pipelines.

Causal inference is the formal statistical and mathematical framework dedicated to uncovering, quantifying, and predicting the independent, directional effects of specific variables (treatments) on targeted outcomes.

The below diagram illustrates how causal inference works, compared to standard, predictive machine learning:

Figure 1. Symmetrical Statistical Associations in Predictive ML vs. Directional Causal Inference Mechanics

Rather than treating data as a collection of symmetrical statistical associations (Left, Figure 1), causal inference models the underlying physical mechanisms and data-generating processes of the system (Right, Figure 1).

◼ Structural Causal Model (SCM) Components

To differentiate standard ML (correlation) from causal inference, we define a chronologically ordered directed acyclic graph (DAG) across four variables:

1. Baseline Confounder X

2. Intervention A_0 ∈ {0, 1}

3. Mediator M ∈ {0, 1}

4. Outcome Y ∈ {0, 1}

To anchor these concepts, we map each variable to a real-world hiring scenario.

▫ 1. Baseline Confounder X

Baseline confounder (confounder) refers to the pre-existing context or characteristics before any action is taken:

where:

• f_X is a deterministic, non-parametric structural mapping function.

• ε_X is mutually independent, unobserved background noise terms (exogenous variables) drawn from an arbitrary joint probability space.

Hiring context: The jobseeker’s background (e.g., years of experience, tech stack proficiency).

Causal mechanic: Acts as a confounder by simultaneously biasing the assignment choice (X → A_0) and inherently increasing candidate market mobility(X → Y), making them more likely to churn regardless of the intervention.

▫ 2. Intervention A_0 ∈ {0, 1}

The intervention refers to the initial action taken by human or other events:

Where:

• f_A is a deterministic, non-parametric structural mapping function.

• ε_A is mutually independent, unobserved background noise terms (exogenous variables) drawn from an arbitrary joint probability space.

Hiring context: The hiring manager's decision to hire (1) or reject (0).

Causal Mechanic: The decision is driven by the confounder X (the jobseeker's background) and unobserved management heuristics ε_A such as the vibe check, impromptu deep-dive, and even mood and fatigue.

For example, the manager might think, "They have the technical skills, but their communication felt a bit hesitant during the system design round."

That hesitation isn't a standardized metric; it's a subjective interpretation (ε_A).

Or, if the manager is interviewing the candidate at 4:30 PM on a Friday after a brutal production outage, their decision threshold might be entirely different than it would be at 10:00 AM on a Tuesday.

▫ 3. Mediator M ∈ {0, 1}

The meditator refers to an incident happened after the intervention:

Where:

• f_M is a deterministic, non-parametric structural mapping function.

• ε_M is mutually independent, unobserved background noise terms (exogenous variables) drawn from an arbitrary joint probability space.

Hiring context: The specific project assignment given to the employee after they join the company.

Causal mechanic: The intervention (the hiring decision) triggers a behavioral shift (A_0 → M), capturing the indirect effect of the treatment.

▫ 4. Outcome Y ∈ {0, 1}

The outcome is the consequence we'd observe after the intervention and mediator:

Where:

• f_Y is a deterministic, non-parametric structural mapping function.

• ε_Y is mutually independent, unobserved background noise terms (exogenous variables) drawn from an arbitrary joint probability space.

Hiring context: Promotion within the first year.

Causal mechanic: The terminal node endogenous to the system, structurally determined by the direct effects of the baseline context, the intervention, and the mediator.

The below diagram illustrate standard ML approach (top) and three causal inference approaches (bottom):

Figure 2. Structural comparison displaying a standard flattened machine learning input layer versus causal directed acyclic graph frameworks. (Created by Kuriko IWAI)

◼ Statistical View of Standard ML

As shown in Figure 2, standard ML treats confounders X, intervention A_0, and mediator M equally as a flat layer of independent input features.

These features all point directly to the outcome Y, even when the true casual reality (left, Figure 2) has an open backdoor path between the intervention A_0 and the outcome Y.

This makes the optimizer shift weight to whatever features yield the highest immediate impact to the outcome, instead of isolating the true impact of the intervention.

Mathematically, standard ML throws observed features into the conditioning set of the conditional probabilities such that:

Where:

• Y, A_0, M, and X is the outcome, intervention, mediator, confounder, respectively.

• P(Y | ...) denotes the conditional probability distribution of the outcome Y given the conditioning set: A_0, M, and X.

And because the mediator M occurs in close temporal proximity to the outcome (Y = 1, indicating the contract termination), the observational data exhibits a strong conditional dependency such that:

Where:

• Y = 1 indicates the final outcome, the contract termination.

• M = Drop is the mediator, indicating a severe decline in communication response rate.

• ≈ 1.0 indicates the empirical conditional probability approaching unity due to tight chronological and statistical correlation between M and Y.

Consequently, the optimizer assigns maximum weight to the mediator M, concluding that "Communication degradation is the primary driver of turnover".

The challenge here is that an operational policy would misallocate capital toward the symptom (communication degradation) rather than the root cause.

For instance, leadership might decide to automate HR alerts to improve response rates, while leaving the true mechanism, the project mismatch entirely unaddressed.

◼ The Feature Flattening Trap

This fundamental failure of a standard ML in decision-making contexts stems from the feature flattening trap.

Eq. 1.1 denotes that the model completely ignores the structural dependencies and directional relationships between features by flattening them.

This induces two primary structural failures in production:

1. Mediator conditioning, and

2. Weight stealing.

▫ Challenge 1. Mediator Conditioning - Confusing Symptoms with Causes

By flattening the feature space, the model optimizes solely for predictive accuracy on the observational manifold.

It cannot distinguish between an upstream cause, the intervention A_0 and a downstream symptom, the mediator M.

Because M shares a high mutual information metric with Y due to its temporal proximity, the model treats the symptom as the lever.

▫ Challenge 2. Weight Stealing - The Statistical Masking

During back propagation, the optimizer attempts to identify the path of least resistance.

Because the mediator M sits closer to Y on the causal path, it absorbs the gradient updates during training.

Concurrently, because the model cannot map the directional flow X → A_0, the baseline confounder X leaks spurious correlation.

And the resulting parameters β's show a catastrophic distortion such that:

Where:

• β_{M} is the estimated coefficient assigned by the model to the mediator M (communication drop).

• β_{X} is the estimated coefficient assigned by the model to the confounder X.

• β_{A_0} is the estimated coefficient assigned by the model to the primary action lever A_0 (project assignment).

Eq. 1.3. shows that the true cause - intervention A_0 - is masked entirely by a downstream symptom, the meditator M and an upstream context, the confounder X.

Causal inference transitions the paradigm from passive observational conditioning to active, counterfactual intervention.

Unlike standard ML, the causal inference approach maps out the exact structural mechanics of how variables interact using Directed Acyclic Graphs (DAGs).

In this section, I'll explore three frameworks as shown in Figure 2:

• Standard DAG.

• Controlled Direct Effect (CDE).

• Total Causal Effect (TCE).

Standard DAG establishes the foundational network of cause and effect where confounders X influence both the intervention A_0 and the mediator M.

The framework can acknowledge the baseline confounder X such as the human manager's bias affect who gets the treatment, and that the treatment flows through a downstream mediator M before hitting the outcome Y.

Mathematically, standard DAG is denoted as observational joint distribution factorization:

Where:

• P(X, A_0, M, Y): The joint observational probability density function over all variables in the system.

• P(X): The marginal distribution of the baseline confounder X such as a candidate's background context.

• P(A_0 | X): Human bias. The conditional probability representing historical human policy. This models how heavily a manager's assignment action A_0 is influenced by the candidate's background context X.

• P(M | A_0): The conditional probability tracking the downstream transition from action to mediator (e.g., the likelihood of a communication drop given a specific project match style).

• P(Y | A_0, M, X): The conditional probability mapping the final outcome based on all preceding factors combined.

Eq. 2.1 represents the joint probability before any intervention, reflecting the historical data-generating process including human biases exactly as it exists in the data logs.

◼ Pros and Cons - Use Cases

• Can serve as the structural hypothesis, before mapping out any domain knowledge.

• Purely describes the status quo historical system. No mechanism to predict what happens if the network is disrupted by changing corporate policies.

Controlled Direct Effect (CDE) tackles this limitation of standard DAGs by isolating the direct impact of the intervention on the outcome by stripping away any influence from the mediator.

Mathematically, the Controlled Direct Effect (CDE) is defined as:

Where:

• CDE(m) is the Controlled Direct Effect evaluated at a fixed mediator state m.

• E[Y | ...] is the mathematical expectation operator under the designated interventional distributions.

• do(A_0 = 1) and do(A_0 = 0) represent Judea Pearl’s interventional operators which set the treatment to active and baseline control states respectively.

• do(M = m) denotes forcing the mediator to a static value m via a joint, secondary intervention.

Eq. 2.2 performs graph surgery on standard DAGs where the mediator M is held as a constant m using do-operator do(M=m).

▫ The Career Progress Scenario

In our tech pipeline, we want to isolate the Controlled Direct Effect (CDE) of the initial hiring decision (A_0) on a first-year promotion (Y), independent of the subsequent project assignment (M).

Imagine leadership wants to know: Does the raw signaling value of a high-bar hiring decision (A_0) inherently accelerate an employee's promotion velocity (Y), even if they are assigned to a low-visibility, maintenance-heavy project (M)?

To measure this via CDE, we enforce two simultaneous interventions:

1. Change the assignment policy: do(A_0 = 1) vs. do(A_0 = 0)

2. Hold M as constant m by invoking do(M = no_drop).

The second intervention can be a situation (m) where the company introduces a strict corporate policy that every new hire is uniformly assigned to the exact same baseline project, regardless of their background (X) or how highly they were rated during the hiring loop (A_0).

By mathematically locking M in place via do(M = m), we effectively sever the causal link from the hiring decision through the project assignment (from A_0 → M → Y to A_0 → Y).

◼ Pros and Cons

• Best to assess if certain action has an independent path to the outcome.

• Forces an artificial environment where the human behavior which acts as the mediator M is frozen while the policy A_0 is shifting, which is unrealistic in most real-world cases.

▫

Total Causal Effect (TCE) can tackle the challenge of CDE by measuring the entire, unfiltered impact of the intervention on the final outcome Y.

In TCE, the mediator is allowed to flow naturally with the causal signal, instead of being blocked.

Mathematically, TCE can be denoted as the net change in outcome expectation generated across all downstream pathways:

Where:

• E[Y | do(A_0 = 1)] is the expected value of the outcome Y in a counterfactual world where the entire population is actively forced to receive the intervention (A_0 = 1 or A_0 = 0)

• do(A=a) represents Judea Pearl’s interventional operator.

For example, A_0 = 1 refers to the situation where every candidate is hired, whereas in A_0 = 0, every candidate is not hired.

This includes the meditator M block which directs to the outcome Y.

▫ Use Cases

• Best for policy optimization and strategic ROI planning.

• Can simulate the true, real-world impact of policy updates using historical data logs (No need to run an expensive A/B test).

▫ The Unconfoundedness Trap

Unconfoundedness trap is the situation where TCE assumes that all confounders are observed and adjusted.

This works against TCE especially when some unobserved feature influences both the policy and the outcome because TCE cannot capture this hidden feature in the first place.

For example, in the hiring / career progress example, a major hidden confounder could be an employee's hidden motivation or personal resilience level.

Highly motivated employees naturally seek out and volunteer for the most complex project because they want to prove themselves (hence, A_0 = 1).

Those candidates are inherently more likely to promote (hence, Y = 1) because of their strong work ethic and motivation.

This unobserved trait influences both the policy assignment and the outcome.

But here's the challenge.

Because the personal motivation metric isn't captured in the confounder, the causal model cannot adjust for it.

As a result, the TCE calculation falls into the unconfoundedness trap: it will absorb this hidden bias and incorrectly credit the low turnover rate entirely to the project assignment policy, creating a skewed ROI metric for leadership.

Causal inference isolates the interventional distribution such that:

Where:

• Y is the random variable representing the terminal target outcome.

• A_0 is the actionable treatment variable (e.g., project assignment policy).

• a ∈ {0, 1} is the specific counterfactual assignment value enforced by the intervention.

• do(A=a) is Judea Pearl’s interventional operator.

◼ Judea Pearl’s Do-operator

Judea Pearl’s do-operator (Judea Pearl, et. al, 2009) provides the mathematical syntax required to formalize this distinction by separating "seeing" (passive observation) from "doing" (active intervention).

Standard ML operates on the observational distribution denoted in Eq. 1.1, which conditions on the subpopulation that happened to receive treatment A = a under the historical, non-randomized policy.

The do-operator, denoted as do(A = a), represents a hypothetical, physical intervention that overrides the natural data-generating mechanism.

It forces the variable A to take the exact value a for the entire population, irrespective of historical tendencies.

For example, suppose we decide to programmatically overwrite a human manager.

We instantiate a strict routing policy that forces every single employee in the pipeline into a high-pressure project, completely ignoring their resume prestige, their historical background, or the managers' personal preferences.

Mathematically, we evaluate the interventional distribution:

Where do(A_0 = 1) represents the programmatic mutation of the system where we exogenously force the assignment A_0 to 1 for the entire incoming population (employees).

▫ The Enterprise Unlock: Off-Policy Evaluation (OPE)

Shifting from P(Y | A_0) to P(Y | do(A_0)) enables counterfactual Off-Policy Evaluation (OPE).

Instead of deploying an unverified algorithm to production and risking live revenue for instance, a causally identified model allows engineers to leverage biased historical logs to answer the counterfactual optimization problem:

What would our retention curve look like if we had executed policy A1 instead of the legacy human-driven policy A_old?

◼ Identification via the Backdoor Criterion

To recover the true interventional distribution from purely observational logs, we must locate an adjustment set Z that satisfies Pearl's Backdoor Criterion relative to the causal pair (A_0, Y).

Pearl’s Backdoor Criterion is the structural graph theory used to determine exactly which variables must be controlled for to isolate a true causal effect.

It provides the algorithmic rules to select a conditioning set of variables Z that blocks all spurious correlations without accidentally destroying or masking the true causal signal.

▫ The Two Rules of the Backdoor Criterion

A set of variables Z satisfies the backdoor criterion relative to an intervention A and an outcome Y that it passes two strict topological conditions:

• Rule 1: No Downstream Mediators. No variable in Z can be a descendant of A. If a variable M is caused by A, meaning it sits downstream on the causal path like A → M → Y, M is a mediator or a symptom. If you control for it, you block the very effect you are trying to measure, which is what triggers the feature flattening trap.

• Rule 2: Block All Upstream Leaks (The Backdoor Paths) Z must block every path between A and Y that contains an arrow pointing into A. An arrow pointing into A such as N → A → Y represents an upstream cause or a confounder (like human bias or background environment) impacting A. These paths allow statistical information to flow backwards from A, through the confounder, and into Y, creating an illusion of causality.

▫ The Backdoor Adjustment Formula

Once an adjustment set Z is mathematically validated via the Backdoor Criterion, we can safely rewrite the do-operator using standard, observable conditional probabilities:

By applying Eq. 4.1 to our specific staffing pipeline topology, M is structurally excluded, and X represents the support space of our baseline confounders—the continuous and discrete backdoor adjustments yield:

Where:

• X is the complete support space of the baseline confounder X (e.g., the distribution of talent and prestige strata across the candidate pool).

• P(Y | A_0 = a, X = x) is the standard conditional observational probability of the outcome given specific real-world realizations of the treatment action and background confounder.

• P(X = x) or dP(X = x) is the marginal observational probability weight of each confounder stratum, used to re-weight the conditional estimates uniformly across the entire population.

▫ The Enterprise Unlock

Mathematically, Eq. 4.2.2 forces the historical data to simulate a counterfactual world where the assignment action A_0 was distributed completely at random across the background environment X (e.g., all candidates are assigned to complex projects).

This eliminates selection bias and enables engineers to execute a clean, synthetic A/B test directly out of old data logs, completely neutralizing the feature flattening trap.

Let us simulate an enterprise dataset characterized by our structural causal topology.

After defining a confounded dataset, the script runs both standard ML and causal inference, and visualizes the profound estimation bias that occurs in production frameworks:

1import numpy as np
2import pandas as pd
3import statsmodels.api as sm
4import statsmodels.formula.api as smf
5
6
7## 1. define a confounded dataset
8np.random.seed(42)
9N = 10000
10
11# X: baseline context confounder (e.g., talent score)
12X = np.random.normal(0, 1, N)
13
14# A0: match action (historical human managers over-index on X when making matches)
15prob_A0 = 1 / (1 + np.exp(- (1.5 * X)))
16A0 = np.random.binomial(1, prob_A0)
17
18# M: downstream mediator (communication dropout - a bad match A0=1 causes communication dropouts)
19prob_M = 1 / (1 + np.exp(- (-2.0 + 2.5 * A0)))
20M = np.random.binomial(1, prob_M)
21
22# Y: true project exit - driven directly by the bad match AND the communication dropout. True total effect on Y is a combination of direct and mediated pathways
23prob_Y = 1 / (1 + np.exp(- (-3.0 + 1.2 * A0 + 1.8 * M + 0.5 * X)))
24Y = np.random.binomial(1, prob_Y)
25
26# create a dataframe
27df = pd.DataFrame({'X': X, 'A0': A0, 'M': M, 'Y': Y})
28
29
30# 2. standard ML approach (feature flattening)
31## throw the mediator M into the feature matrix.
32standard_model = smf.logit("Y ~ A0 + M + X", data=df).fit(disp=0)
33
34
35# 3. causal inference
36## structurally leave mediator M out of the regression
37causal_model = smf.logit("Y ~ A0 + X", data=df).fit(disp=0)
38
39
40# 4. visualize estimation bias
41models_data = {
42    'Model Type': [
43        'Standard ML (Flattened)', 
44        'Causal Backdoor Adjustment'
45    ],
46    'Estimated Beta (A0)': [
47        standard_model.params['A0'],
48        causal_model.params['A0']
49    ],
50    'CI_lower': [
51        standard_model.conf_int().loc['A0', 0],
52        causal_model.conf_int().loc['A0', 0]
53    ],
54    'CI_upper': [
55        standard_model.conf_int().loc['A0', 1],
56        causal_model.conf_int().loc['A0', 1]
57    ]
58}
59
60results_df = pd.DataFrame(models_data)
61

◼ The Simulation Results: Proof of the Trap

By running the Python script, we generate a dataset where the initial match decision (A_0) has both a direct impact on project success and an indirect impact by causing a downstream communication dropout (M).

When we run both a standard machine learning and causal inference, the resulting coefficients expose a massive blind spot.

▫ The Results - Empirical Breakdown of the Coefficients

Table 1. Empirical Parameter Comparison: Structural Ground Truth vs. Observational Feature-Flattened Frameworks

The Model Illusion

The Standard ML model will report a highly deceptive feature importance profile.

It will convince humans that a drop in communication (M = 1.81) is nearly twice as critical to fix than the actual matching process (A_0 = 1.12).

The Downstream Value Drain

If leadership uses the standard model's outputs to plan investments, the company will pour capital into downstream band-aids such as automated Slack nudges or communication alerts.

These interventions will ultimately fail because they leave the true, high-leverage driver—the broken matching habit—completely unaddressed.

The Causal Unlock

By utilizing Pearl's Backdoor Criterion and blocking only the baseline confounder (X), the Causal model uncovers the true structural coefficient of 2.00.

This gives leadership an accurate, non-confounded ROI metric to justify rewriting the entire onboarding and assignment policy.

While the Backdoor Criterion provides the structural blueprint to escape the Feature Flattening Trap, implementing it on real-world enterprise architectures introduces an immediate engineering bottleneck.

In our Python simulation, I utilized a clean, low-dimensional confounder X.

But in production, the baseline confounder X is high-dimensional.

Attempting a naive, exact stratification on such an expansive space triggers the curse of dimensionality where the data splits into empty strata, and math collapses.

Worse yet, real-world historical data is plagued by selection bias and institutional behavioral cloning.

Because legacy human managers systematically over-indexed on specific shortcuts (like assigning all Ivy League graduates exclusively to premier accounts), there are vast regions of the feature space where alternative choices were never attempted.

To run a true offline evaluation, we must find a way to re-weight our dirty observational data to create a balanced, equitable foundation.

◼ Up Next — Part 2: De-Biasing Historic Logs

▫ Synthetic Randomization via Inverse Propensity Scoring (IPS)

In the next installment of this series, we will transition from pure structural graph theory to scalable algorithmic execution.

I will explore how to use Inverse Propensity Scoring (IPS) to isolate and neutralize day-one human selection shortcuts.

◼ References

• Judea Pearl, Causal inference in statistics: An overview

Written by Kuriko IWAI. All images, unless otherwise noted, are by the author. All experimentations on this blog utilize synthetic or licensed data.

FAQ

Share What You Learned

Continue Your Learning

If you enjoyed this blog, these related entries will complete the picture:

• Fixing Selection Bias in Enterprise Data with Inverse Propensity Scoring (IPS)

  

  Fix selection bias in observational data with IPS. Learn propensity math, positivity limits, and Python implementation.

Related Books for Further Understanding

These books cover the wide range of theories and practices; from fundamentals to PhD level.

Linear Algebra Done Right

Foundations of Machine Learning, second edition (Adaptive Computation and Machine Learning series)

Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems

Machine Learning Design Patterns: Solutions to Common Challenges in Data Preparation, Model Building, and MLOps

Hands-On Large Language Models: Language Understanding and Generation