student projects

The accuracy of machine learning methods often drops when they are trained on one domain and deployed on another. This is a finding which has been observed over and over empirically. However, it is less clear which actions can be undertaken to mitigate this failure, if any. One intuitive approach, distributionally robust optimization (DRO), aims to find a model which performs well on all test datasets “close” to the training data in some probability distance. However, this approach results in overly pessimistic models which are robust against “unrealistic” distribution shifts. Instead, a number of methods have been proposed which aim to identify and exclusively use stable (“causal”) relationships in the data. Corresponding theory states that such methods are guaranteed to perform better than standard empirical risk minimization (ERM) under worst-case distribution shifts. When put to the test empirically, these findings do not seem to hold up well: on real-world datasets, causality- (or invariance-) based generalization methods are very often outperformed by ERM, and seem to generalize worse both in- and out-of-distribution (OOD) [Nastl & Hardt, 2024, Salaudeen et al., 2025]. This seems consistent with the “accuracy-on-the-line” hypothesis, which postulates that the ranking of models is often preserved across distributions. In recent work, it has been argued that this mismatch between theoretical and empirical findings is an “artifact” of misspecified OOD datasets, which do not contain sufficiently adversarial shifts.

The goal is to resolve the mismatch between theoretical and empirical findings in multiple ways:

• by verifying whether invariance-based OOD methods rank better if the distribution shift is constructed to be worst-case;
• by constructing benchmarks with varying strength and complexity of the distribution shift which could help evaluate a “spectrum” of OOD generalization of models;
• by providing a theoretical justification of recent empirical findings through analysis of the mismatch between benchmarks and assumptions.

Goals of the project:

• Create novel evaluation schemes for OOD machine learning methods;
• Construct novel benchmarks which more accurately measure OOD generalization;
• Test a large variety of large-scale models via the novel schemes and benchmarks.

Key Skills & Qualifications:

• Strong background in machine learning, familiarity with reading and understanding research papers.
• Solid Python and PyTorch skills, plus basic experience running experiments on GPUs.
• Background in statistics, basic machine learning theory, training and evaluation of machine learning models.
• Interest in out-of-distribution generalization, design of safe and robust models, and causality

Modern LLM systems often choose between multiple ways of answering a prompt: a cheap model, a stronger model, a retrieval-augmented pipeline, a cascade, or a verifier-based workflow. This project studies this choice as a contextual mediator-selection problem: given a prompt, select the most appropriate answer-generating procedure while balancing quality, cost, and latency. Building on the well-understood theoretical framework on mediator-feedback bandits, where a learner selects a mediator/policy that produces an outcome, the goal is to develop a useful method for efficient LLM orchestration (with a potential extension of the theoretical framework too). The expected contribution is a compact empirical study showing when mediator-based routing improves over standard model selection, and whether treating LLM procedures as mediators is useful for practical agentic systems.

Key Skills & Qualifications:

• Strong background in machine learning, familiarity with reading and understanding research papers also from a theoretical lens.
• Solid Python and PyTorch skills, and experience running experiments on GPUs.
• Experience working with LLMs is a plus.

Semaglutide and dulaglutide are GLP-1 receptor agonists used to treat type 2 diabetes. The SUSTAIN 7 trial compared the two drugs under controlled conditions in patients receiving metformin monotherapy and measured changes in HbA1c and bodyweight after 40 weeks. In real-world UK primary care, patients often take several glucose-lowering drugs, may change or stop treatment, have other comorbidities, and differ in many other aspects from the trial population. Many would not have qualified for SUSTAIN 7. We would like to understand what the comparative treatment effect is in this broader clinical setting and how causal effects estimated in the trial population can be generalized to this real-world setting. This question matters for clinical decisions, reimbursement, and regulation.

Standard methods for generalizing or transporting trial results use reweighting, outcome modeling, or doubly robust combinations. They rely on assumptions about which patient characteristics change the treatment effect and on enough overlap between the trial and target populations. These assumptions may be difficult to justify for patients who differ strongly from the trial participants.

The main focus of this project is invariance. A starting point is to review the medical and statistical literature, including work on causal graphs for type 2 diabetes, and look for relationships that may remain stable across trial and real-world settings. Disease severity and proxy variables used to measure it are one possible starting point. Based on this review, possible invariance assumptions can be stated and assessed in the available data. The next step is to study whether these assumptions can help estimate treatment effects for parts of the target population that are not well represented in the trial. Standard transport methods can provide baselines.

The main data source will be CPRD Aurum, a large UK primary-care database. The exact target population, treatments, outcomes, and methodological approach will be refined after the literature review and the first data checks. HbA1c and bodyweight are likely outcomes. The project description and scope is preliminary and may change as we learn more about the data.

Goals of the project (to be further specified):

• Read into mechanisms behind type 2 diabetes, GLP-1 drugs, associated causal graphs, and the literature on generalizability, transportability, and invariance.
• Compare the trial and real-world populations and implement suitable standard transport methods as baselines, if applicable.
• Identify plausible invariance assumptions, assess them in the available data, and use or adapt them for treatment effect estimation.
• Apply the resulting methods to the UK data and study how the conclusions depend on the assumptions.

Key Skills & Qualifications:

• Highly motivated, with a strong background in statistics or machine learning and an interest in causal inference.
• Genuine interest in clinical applications (this is an applied project).
• Willing to read both statistical and medical papers.
• Solid programming skills, preferably in Python, and willing to spend substantial time handling real-world data.
• Ability to work carefully and independently.