PhD Committee

Embedding medical pathways for causal inference on healthcare supply shocks and patient outcomes

Meilame Tayebjee

8 October 2025

Table of contents

• 1 Introduction
  • About me
  • Broader view of the Graph4Health ANR
  • My project: embedding patient pathways for causal inference
• 2 Related work
  • Transformer-based models for medical pathway embedding
  • ML/DL for causal inference
  • Causal impact of PCP loss on health outcomes
• 3 Work done so far
  • Data management
  • Notations
  • Pre-training task
  • First trained models
  • Plot of the embedding space
• 4 Avenues of work
  • Causal inference
  • Benchmarking
  • Improving pre-training task
  • Scaling
• References

1 Introduction

About me

• Graduated from HEC (19-24), X (X20), ENSAE and MVA (administrateur de l’Insee, 2024)
• Research internships:
  • 2023 (3A X): MIT (Operations Research Center), Microtransit system design and optimization
  • 2024 (ENSAE-MVA): Inria HeKA, Generative models for longitudinal images
• Currently, first position in the Corps since sept. 2024:
  • Data Scientist at SSP Lab (MLOps, model trustworthiness, NLP (text classification, RAG), satellite imagery…)
  • PhD in CS Candidate (part-time, journées recherche)

Broader view of the Graph4Health ANR

• The general objective is to
  • infer the causal impact of the healthcare supply (and its geographic distribution) on health outcomes
    • using classic econometric tools
    • using deep learning methods for causal inference
  • study policies that contribute to the efficiency of the French healthcare system
• 11 years of exhaustive SNDS
  • 70M patients
  • consultations, procedures, hospitalizations, medications, deaths etc.

My project: embedding patient pathways for causal inference

• Our data: patient (medical) pathways
  • unstructured (just as in NLP)
  • temporally irregular (not like NLP!)
• To feed classical causal ML estimators, we need to represent those pathways as vectors (embeddings)

2 Related work

Transformer-based models for medical pathway embedding

Model name	Model archi.	Pre-train. Task(s)	Fine-tun. task(s)	# Params	Training data size	Vocab size (type)	Generative	Trajectory1
Delphi (Shmatko et al. (2025))	NanoGPT	Next disease event + Time to event	None	2M	400k patients	1,258 (ICD10)	✅	✅
MOTOR (Many Outcome Time Oriented Representation) (Steinberg et al. (2024))	Transformer (causal masking and local attention)	Time to event (TTE)	TTE for part. disease	143M	55M patients, 9B events	8,192 (ICD10 and other specific codes)	≅	≅
Life2Vec (Savcisens et al. (2023))	BERT	MLM + SOP	Mortality Prediction	8M	3M patients	~700 (labour + ICD10)	❌	❌
TransformEHR (Yang et al. (2023))	Encoder-Decoder	Visit masking: Given visits 1-3, predict all codes in visit 4	Disease (pancreatic cancer) / Outcome (self harm) pred.	?	255M visits from 7M patients	ICD10	✅	❌
BEHRT (Li et al. (2020))	BERT	MLM	Disease prediction	n.r. (~6L, 12H, h=288)	1.6M patients	301 disease codes	❌	❌

1. Efficient (i.e. in one forward pass) computation of embeddings at any time

ML/DL for causal inference

Several methodological papers:

• Abécassis et al. (2025), Doutreligne and Varoquaux (2025)
  • The best predictive model is not the best suited for causal inference
  • Model selection using risk scores
  • Designing a causal inference framework (Directed Acyclic Graphs (DAGs), Population-Intervention-Control-Outcome-Time (PICOT))
• Wager (2024):
  • Definitions and assumptions
  • Meta-learners
• Curth and Schaar (2021):
  • Comparison of different meta-learners in different settings

Causal impact of PCP loss on health outcomes

From T. Le Calvé, PhD Student, CREST

3 Work done so far

Data management

• Validation and EDA of the first delivered simplified tables: prestations and medications
• Jointure, cleaning and subsampling
  • So that any subsample can fit in vRAM
• Tokenization

Notations

Notations: dataset

Dataset of medical pathways: For i = 1, \dots, n, X_i \coloneqq (e_i, t_i, f_i), where:

• e_i \in \mathbb{R}^{cs} are the tokenized events, containing integers between 0 and the vocabulary size \lvert \mathcal{V} \rvert
• t_i \in \mathbb{R}^{cs} is the temporal vector, containing the dates
• f_i \in \mathbb{R}^{2} corresponds to the patient general features, for now age and gender

Notations: model

• Embedding matrices of size \mathbb{R}^{\lvert \mathcal{V} \rvert, d_{embed}} (vocab), \mathbb{R}^{2, d_{embed}} (gender) and \mathbb{R}^{8, d_{embed}} (age class)
  • We use t_i to have a time-aware positional encoding using a sinusoidal encoding
  • A symbolic [START] token, composed of the summed embeddings of age and gender.
• Causal self-attention layers so that the Transformer-based model outputs causally contextualized embeddings: GPT_{\theta}(X_i) \in \mathbb{R}^{cs, d_{embed}}

where \theta parametrizes the neural network.

Pre-training task

Objectives

• Having a time-aware Transformers (\neq NLP)
• Keeping in mind causal inference
  • We want to be able to efficiently compute the embedding of the pathway at any time (the trajectory of the patient)
• Problems with the next token task:
  • a lot of simultaneous events, especially for medications
  • different types of events: does it make sense to have a competition among them ?

Description

We predict for each token, if it will appear within the next t days.

We consider, simulatenously, several t \in \mathcal{T}, \mathcal{T} being the sorted set of short term horizons, typically 14, 30, 90, 180 days.

First trained models

We trained a “large” model (40M params) and a “small” one (5M).

ROC curve of the pre-training task for a given specialty on a test set of 10,000 patients.

Still, the models converge quickly, and even small models achieve good convergence. This raises the (open) question of complexifying the task at some point.

Plot of the embedding space

4 Avenues of work

Causal inference

• Goal: Estimate the impact of losing one’s Médecin Traitant (Primary Care Provider, PCP), and looping back with Titouan’s work

• Up to a given time t, we have:

  • The covariate X_{<t} \coloneqq (e_{<t}, t_{<t}, f_{<t}), the medical history of the patient
  • W_t is the (binary) treatment at time t, say: the patient has lost its PCP (death or retirement) before t
  • Y_t (W_t), the binary outcome, indicating whether death has occured within a given time window [t, t+ \delta_d]

• We want to estimate the Conditional Average Treatment Effect (CATE): \tau(x, t) \coloneqq \mathbf{E}[Y_t(1) - Y_t(0) | X_{<t} = x]

Assumptions

• SUTVA : Y_t(w) = Y_t \quad \textit{ if } W_t = w

  • The others not having a PCP for a given amount of time does not influence my probability to die
  • Only one version of treatment ?

• Conditional ignorability - Unconfoundedness \forall w \in \{0,1\}, Y_t(w) \perp W_t \mid X_{<t}

• Overlap 0 < \mathbb{P}(W_t = w | X_{<t} = x) < 1, \forall (w,x) \in ( \{0,1\}, \mathbb{R}^{d_{embed}})

Estimators of the CATE: S-learner

For a given time t, we define {pos}_t as being the last position in the patient’s pathway before t, and we define the embedding of the pathway: GPT^{\text{path}}(X_{<t}) \coloneqq GPT_{\theta^{*}}(X_{<t})_{{pos}_t} \in \mathbb{R}^{d_{embed}}

We train on the calibration set a head g_{\phi}: \mathbb{R}^{d_{embed}} \times \mathbb{R}_{+} \to [0,1], minimizing the BCE loss on death prediction - to obtain \phi^{*}. \hat{\tau}_{\text{SLearner}}(x) \coloneqq g_{\phi^{*}} (GPT^{\text{path}}(x), 1) - g_{\phi^{*}} (GPT^{\text{path}}(x), 0)

Estimators of the CATE: R-learner

• Another approach consists in trying to model the propensity score, the probability of being treated given X.

• A first head g_{\phi_{m}}: \mathbb{R}^{d_{embed}} \to [0,1] is trained on the death prediction task, so that for a given pathway x, g_{\phi_{m}^{*}}(GPT^{\text{path}}(x)) is close to the conditional mean outcome m(x) = \mathbb{E}[Y_t | x].

• Another head g_{\phi_{e}}: \mathbb{R}^{d_{embed}} \to [0,1] is trained to predict the treatment W_t of a given individual (minimizing a Mean Squared Error).

• Finally, we train the CATE estimator training a head g_{\phi_{\tau}}: \mathbb{R}^{d_{embed}} \to [0,1] by minimizing the R-loss: \begin{align*} \phi_{\tau}^{*} \in \arg\!\min_{\phi_{\tau}} \; & Y_t - g_{\phi_{m}^{*}}\left( GPT^{\text{path}}(X_{<t}) \right) \\ & - \left( W_t - g_{\phi_{e}^{*}}\left( GPT^{\text{path}}(X_{<t}) \right) \cdot g_{\phi_{\tau}}\left( GPT^{\text{path}}(X_{<t}) \right) \right) \end{align*}

Estimators of the CATE: at the end of the day

We have a lot of other meta-learners:

• T-learner
• X-learner
• DR-learner
• …

Target pipeline - from Abécassis et al. (2025)

Benchmarking

• How to compare embedding models ? What is a good embedding ?
  1. Death or other outcome prediction
  2. R-loss (Doutreligne and Varoquaux (2025))
  3. Visualization of the embedding space and semantic understanding (metric t.b.d.)
• Competitors:
  • SVD-PPMI (static embeddings - SVD of a cooccurrence matrix with a given window size)
  • BERT (pure embeddings, no trajectories)
    • MLM pre-training task (Savcisens et al. (2023))
    • Temporal window masking task
  • GPT with next-token + time-to-event task (Shmatko et al. (2025))

Improving pre-training task

• Exploring Posson loss - ground truth being number of events for each token, for each time window
• Exponential loss for time-to-event task (as in MOTOR (Steinberg et al. (2024)))

We want ideally to have a generative model.

Scaling

• Hoffmann et al. (2022) propose a scaling law for LLMs

  • Trade off between number of training tokens (train dataset size) and number of parameters in the model (model size)

  • Estimated optimal ratio: \frac{\text{\# training tokens}}{\text{\# parameters}} = 20

    • is it the same for medical pathways ?

From Shmatko et al. (2025)

From Hoffmann et al. (2022)

References

Abécassis, Judith, Élise Dumas, Julie Alberge, and Gaël Varoquaux. 2025. ‘From prediction to prescription: Machine learning and Causal Inference’. Annual Review of Biomedical Data Science, ahead of print, April. https://doi.org/10.1146/annurev-biodatasci-103123-095750.

Curth, Alicia, and Mihaela van der Schaar. 2021. Nonparametric Estimation of Heterogeneous Treatment Effects: From Theory to Learning Algorithms. https://arxiv.org/abs/2101.10943.

Doutreligne, Matthieu, and Gaël Varoquaux. 2025. ‘How to Select Predictive Models for Decision-Making or Causal Inference’. GigaScience 14 (January): giaf016. https://doi.org/10.1093/gigascience/giaf016.

Hoffmann, Jordan, Sebastian Borgeaud, Arthur Mensch, et al. 2022. Training Compute-Optimal Large Language Models. https://arxiv.org/abs/2203.15556.

Li, Yikuan, Shishir Rao, José Roberto Ayala Solares, et al. 2020. ‘BEHRT: Transformer for Electronic Health Records’. Scientific Reports 10 (1): 7155. https://doi.org/10.1038/s41598-020-62922-y.

Savcisens, Germans, Tina Eliassi-Rad, Lars Kai Hansen, et al. 2023. ‘Using Sequences of Life-Events to Predict Human Lives’. Nature Computational Science 4 (1): 43–56. https://doi.org/10.1038/s43588-023-00573-5.

Shmatko, Artem, Alexander Wolfgang Jung, Kumar Gaurav, et al. 2025. ‘Learning the Natural History of Human Disease with Generative Transformers’. Nature, ahead of print, September 17. https://doi.org/10.1038/s41586-025-09529-3.

Steinberg, Ethan, Jason Alan Fries, Yizhe Xu, and Nigam Shah. 2024. ‘MOTOR: A Time-to-Event Foundation Model for Structured Medical Records’. The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=NialiwI2V6.

Wager, Stefan. 2024. Causal Inference: A Statistical Learning Approach.

Yang, Zhichao, Avijit Mitra, Weisong Liu, Dan Berlowitz, and Hong Yu. 2023. ‘TransformEHR: Transformer-Based Encoder-Decoder Generative Model to Enhance Prediction of Disease Outcomes Using Electronic Health Records’. Nature Communications 14 (1): 7857. https://doi.org/10.1038/s41467-023-43715-z.