I recently finished BlueDot’s Technical AI Safety course. The reading list is long. Read one unit at a time, it is easy to lose the shape of the whole.
So I did what one does with a pile of components. I laid all 99 readings out along the system they belong to: the lifecycle of a model, from the data it learns from to the point where it is served and retrained. What came out of it is fairly simple.
The readings crowd the middle of the lifecycle, where a model is shaped and examined, and thin out at both ends. Most of the open problems, wherever they sit, are problems of observation.
To be clear, that last claim (and the title) is my rough estimate. It rests on the readings of one course: BlueDot’s Technical AI Safety course, which I recently completed and which is very professionally run. A different reading list might well draw a different map.
The numbers in brackets point to the references at the end. Each course reading keeps its number from my reading spreadsheet, from 1 to 99, so [4] always means the same METR post. Anything from outside the course is numbered from 100 onwards.
Reading list - a brief EDA
Figure 1. The 99 course readings by author, form, year and lifecycle phase. The last panel counts only the 81 technical readings.
Fifty-one of the readings are mandatory. Eighty-one are about the technical work, and the remaining 18 are about careers and research directions. The list is very recent. Sixty readings are from 2025 or 2026, and only five predate 2022. It also leans on the labs. A quarter of it was written by a frontier lab, including the longest documents on the list, the safety frameworks and system cards [44, 45, 46, 50, 51, 52, 53, 54].
The last panel is the one this post is about. Evaluation alone holds 33 readings. Pretraining holds none.
Reading list - the landscape
Each reading in my spreadsheet carries a few scores. Two of them are useful here: how important the problem it addresses is, and how clearly it points to something still unsolved. Figure 2 plots the 81 technical readings on those two axes. Each dot is a reading, numbered as in the references, and coloured by where it sits on the lifecycle. The scores are a first pass, so the exact positions should be taken loosely.
Figure 2. The technical readings by importance and open-problem signal. Each dot links to its reference.
Only four readings make it to the top-right corner. They are on chain-of-thought monitorability, anti-scheming training, alignment faking and the Mythos Preview system card [66, 24, 36, 46]. All four ask, in one way or another, whether we can trust what we see of a model.
Figure 3 cuts the same list by the risk each reading addresses.
Figure 3. Lifecycle phase against risk. A reading is counted once for each risk it addresses.
Evaluation spreads across almost every risk. Pretraining has an empty row, and monitoring has only a handful of entries.
Ninety-nine readings on one loop
Let us draw the lifecycle of a model as a loop, similar to Chip Huyen’s picture of a machine learning system [100]. Data flows forward to deployment, and monitoring feeds new data back in. Her later book adds the stages that foundation models bring with them, such as post-training, prompting, retrieval and agents [101]. I filled in the training stages from the Llama 3 and Tülu 3 reports [102, 103]. That gives eight phases and 37 stages.
Figure 4. The model lifecycle drawn as a loop. A filled square marks a stage that at least one course reading is mainly about.
I then tagged every reading with the one stage it is mainly about. The tags are rough. A reading on evaluation often says something about deployment too. But the overall picture does not change much. Interpretability alone gets 15 readings [56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 69, 70, 71, 72, 73]. Thirteen of the 37 stages get no reading of their own. Six are not touched by any reading at all: data provenance, mid-training, downstream fine-tuning, inference optimisation, user feedback and model updates.
The middle is well read. On post-training, the course covers the limits of RLHF [17], Constitutional AI [19], and two routes to scalable oversight, debate [27, 28] and weak-to-strong generalisation [30]. The failure these guard against is already here, with recent models patching tests and graders while saying they knew it was unintended [4]. Evaluation gets the most pages, from alignment faking [36] and blackmail in a simulated office [38] to a re-run of the shutdown experiments that found clearer instructions removed most of the resistance [34]. Interpretability has its findings [71, 72, 69] and its critics [59, 62, 63] side by side.
The two ends are thin. At the front, there are three readings on filtering training data [11, 12, 13] and one on poisoning it, where about 250 documents planted a backdoor at every model size tested [14]. Nothing covers where the data comes from [104], and pretraining has no reading of its own. At the back, the course covers guardrails [55, 77] and AI control [74, 75], but says little about the rest of serving. A little fine-tuning can strip safety training [106, 107], retrieved documents can carry instructions an agent will follow [108], and a model can turn harmful once quantised [109]. The loop also closes through users. GPT-4o became sycophantic after an update leaned on thumbs-up feedback [110], and GPT-4 drifted on one simple task from 84% to 51% accuracy between two releases [111]. Huyen spends two whole chapters on what happens after deployment [100]. The course has no reading on it.
A map of open problems
For each reading I wrote down the open problem it points to most clearly. Many of these repeat. Merged, they come to twenty. Figure 5 places them on the same loop, with the readings behind each one.
Figure 5. Twenty open problems placed on the model lifecycle, with their references. Filled tags mark the ones I would call observation problems: the hard part is seeing what the model is or does. Each tag links to its entry below.
Framing and governance
OP1. Threshold calls under uncertain evals. Every frontier framework commits to act when a capability threshold is crossed, but none says much about how confident the measurement must be [44, 50, 52, 31]. Outside assessments of what the labs actually do are not encouraging. The most recent one puts no company above C+ on basic control practices [121, 122].
OP2. Safeguards that survive open weights. Once weights are released, most safeguards can be fine-tuned away [54, 12].
Data
OP3. What filtering misses. We do not know what a model can rebuild from the facts that remain, or retrieve in context [11, 12, 13]. Traits can also travel through data that looks clean. A teacher model’s preferences, and its misalignment, pass to a student through plain number sequences [114], and sycophancy transfers through preference data in which no single example is sycophantic [115].
OP4. Finding a few poisoned documents. If a near-constant number of documents suffices, detection has to work at the level of a few hundred samples in billions [14]. Narrow secret loyalties have already been built into open-weight models in ways that evade black-box audits [117].
OP5. Provenance of the corpus. Much of the training data has no reliable record of where it came from [104].
Pretraining
OP6. Reasoning that stays legible. Chain-of-thought monitoring works only while models think in readable text, and architectures that reason in latent space would remove it [66, 105]. Chain-of-thought was built for capability in the first place, and whatever a model computes before its first token never appears in it [113].
Post-training
OP7. Removing reward hacking without teaching concealment. Penalising detected hacks may train models to hide them [4, 24].
OP8. Oversight by a much weaker judge. Debate and weak-to-strong results are promising, but nothing has been shown to work when the gap is large [21, 27, 28, 30]. Using AI to help with oversight is an obvious bet. It only works if the overseeing models cannot themselves evade evaluation [118, 120].
OP9. Safety that survives fine-tuning. Safety training is shallow enough to be undone by a little downstream tuning [106, 107].
Evaluation
OP10. Alignment versus evaluation awareness. Better behaviour on evals may reflect a model that knows it is being tested [24, 36, 37, 46]. The original alignment-faking setup can be reproduced outside a lab [127].
OP11. Evals with stated confidence. Elicitation and statistics are rarely standardised, and benchmark scores are rarely linked to real-world risk [31, 41, 42]. Measurement is harder still when we extrapolate whatever is easiest to measure [126].
OP12. Interpretability as assurance. Feature and circuit methods explain only part of the computation, and nobody yet knows how to turn several imperfect detectors into a guarantee [63, 60, 69, 72]. The theories of impact have been around for a while [123]. Measurable gains from them are harder to point to. Outside language models, even ranking attribution methods against each other is unsettled [124].
Deployment and serving
OP13. Guardrails with guarantees under new attacks. Streaming classifiers need robustness to attacks that were not in their training set [55, 76, 77].
OP14. Control for long-horizon agents. Control protocols are tested on short tasks, and monitor quality is the bottleneck [74, 75, 78]. At the labs themselves, control is only partly practised so far [121].
OP15. Injected instructions and quantised weights. The serving stack itself opens new failure modes [108, 109].
Monitoring and iteration
OP16. A measure of monitorability. There is no agreed metric for how much of a model’s reasoning a monitor can see [66, 68, 45], and good arguments for watching internal states as well as the trace [113, 118].
OP17. Misuse and incidents in the wild. Agentic misuse can be split across many sessions so that none of them looks harmful [47]. There are no norms yet for reporting incidents [5], and independent reports of agents probing the public web have only just begun [119].
OP18. Feedback and updates that do not regress. User feedback can train in sycophancy, and silent updates can break what worked [110, 111]. Probing the assumptions a model makes about its user is one handle on it [116]. The transfer of sycophancy through preference data is another warning [115].
Around the model
OP19. Stability and collusion among agents. Safety evals test one agent at a time, and populations of agents have no stability analysis [6].
OP20. Measuring gradual disempowerment. Slow loss of human control has no measurement while it happens [79, 80, 82]. Secret loyalties are a sharper version of the same worry. Whether they can be found, or removed, after training is open [117].
Most of them are observation problems
By my rough count, fifteen of the twenty are about seeing what the model is or does. The other five are about making training or safeguards hold up.
The evidence is spread across the list. Anti-scheming training cut covert actions in o3 from 13% to 0.4%, but the models also talked more about being evaluated, and some of the drop may come from that [24]. An early Mythos Preview privately considered that it was being tested in about 29% of runs, and behaved worse when steered to believe a scenario was real [46, 112]. Opus 4.6 got better at finishing suspicious side tasks without tripping automated monitors [45]. And the reasoning trace those monitors read may not survive future training [66].
In control terms, we have decent tools for identifying a system before we close the loop, and weak ones for observing it while the loop runs. The state we care about now includes whether the model believes it is being watched, and our measurements change that state. That is an observer design problem with a plant that adapts to the observer. I find this framing useful mostly because control theory has decades of results on observers and robustness to borrow from. I tried a small piece of it for safety classifiers.
The course is very good at the middle of the loop. Perhaps the most useful thing it did for me was to make the rest of the loop visible. We have learnt to examine a model before it ships. Watching it afterwards is still largely open.
Afterword: where non-frontier labs come in
Almost every reading on the list is either written by a frontier lab or about a frontier model. So what is left for the rest of us? The course does not ask this directly, so what follows is my own reading of its resources. Most of the answer, I think, sits at the thin ends of the loop.
- Evaluating from outside. Much of the list is already third-party work. METR writes on reward hacking and pre-deployment risk [4, 32], Apollo on a science of evals [31], and Epoch and RAND on bio evals [41, 42]. SecureBio reviewed a lab’s own chem and bio risk report, agreed with its risk levels, and still raised concerns about expert jailbreakers and users exempt from the classifiers [49]. That is what a useful outside review looks like. Others now grade the labs’ practices [121, 122], and the most recent assessment finds detection and third-party assessment the strongest areas, with prevention and containment the weakest [121]. Transluce reports agent incidents it finds on the public web [119]. None of this needs a frontier model. Much of OP11 and OP17 is an outside job.
- Domain evaluation. The course has one reading on critical infrastructure [82], and its cyber benchmarks score exploits against ordinary software [39, 40]. Operational technology and industrial control systems have almost no evals of their own. That matters, because a failure in one system can cascade through power and water, and on to everything that depends on them [82]. The labs are unlikely to fill this gap. Industrial AI groups hold the plants and the data, and know how those systems fail. Much of that data is time series, where even evaluating interpretability methods is unsettled [124].
- The back of the loop. Whoever deploys a model owns its monitoring, which is where the list is thinnest. Control protocols assume a trusted, weaker model watching an untrusted one [74, 75], and the deployer is the one who has to run them. Guardrail classifiers [55] and probes [65, 68] are small models that need no frontier compute. Hallucination probes, for instance, flag made-up entities token by token and scale to 70B models [68], which is cheap enough to run beside a deployed system. Incident reporting [5] and misuse detection [47] are as much operational practice as research. On the map, this covers OP13, OP14, OP16 and OP17.
- The front of the loop, with open models. Filtering the pretraining data of 6.9B-parameter models gave resistance to 10,000 steps of adversarial fine-tuning [12]. That is pretraining safety research at a scale a smaller lab can afford, and it speaks to OP2 and OP3. The same paper notes that filtering does not stop a model using hazardous information supplied in context [12], so the problem is far from closed. Anyone distilling one open model into another should also read the subliminal learning work, since traits pass most easily between models that share a base [114]. The alignment-faking setup comes with a replication guide [127], and physics-minded researchers have proposed quantitative bounds on model behaviour [125].
The research agendas on the list ask for outside collaborators, mostly on monitoring and evaluation [87, 88, 89]. Research leads say the scarcest people are empirical iterators [86].
For someone coming from control, the back of the loop is perhaps the most natural place to start. Watching a system you do not fully trust, with sensors that are themselves imperfect, is an old problem there.
The middle of the loop belongs to the labs.
The two ends are open to the rest of us.
References
Numbers 1 to 99 are the course readings, grouped by the lifecycle stage they are mainly about. Numbers from 100 are outside the course; 113 onwards are papers and pages I read after it.
Framing and governance (13)
[1] Adam Jones (2024). What is AI alignment?.
[2] Rational Animations (2024). Specification Gaming: How AI Can Turn Your Wishes Against You.
[3] Ajeya Cotra (2021). Why alignment could be hard with modern deep learning.
[7] Richard Ngo (2024). Reframing AGI Threat Models.
[8] Paul Christiano (2019). What failure looks like.
[9] Holden Karnofsky (2022). AI Could Defeat All Of Us Combined.
[10] Holden Karnofsky (2022). Why Would AI “Aim” To Defeat Humanity?.
[20] Google DeepMind Safety Research (2025). An Approach to Technical AGI Safety and Security.
[44] Anthropic (2026). Anthropic’s Responsible Scaling Policy.
[48] Anthropic (n.d.). The Long-Term Benefit Trust.
[50] OpenAI (2025). OpenAI’s Preparedness Framework.
[52] Google DeepMind (2025). Google DeepMind’s Frontier Safety Framework.
[54] Meta (2025). Meta’s Advanced AI Scaling Framework.
Data (4)
[11] Sarah Hastings-Woodhouse (2025). What is input data filtration in AI safety?.
[12] O’Brien et al. (2025). Deep Ignorance.
[13] Chen et al. (2025). Enhancing Model Safety through Pretraining Data Filtering.
[14] Souly et al. (2025). A small number of samples can poison LLMs of any size.
Post-training (15)
[4] METR (2025). Recent Frontier Models Are Reward Hacking.
[15] Rational Animations (2024). The True Story of How GPT-2 Became Maximally Lewd.
[16] Li-Lian Ang (2024). A simple technical explanation of RLH(AI)F.
[17] Sarah Hastings-Woodhouse (2024). Problems with Reinforcement Learning from Human Feedback (RLHF) for AI safety.
[18] Lambert, Castricato, von Werra et al. (2022). Illustrating Reinforcement Learning from Human Feedback (RLHF).
[19] Bai, Kaplan et al. (2022). Constitutional AI: Harmlessness from AI Feedback.
[21] Adam Jones (2024). Can we scale human feedback for complex AI tasks?.
[23] Sarah Hastings-Woodhouse (2025). What is deliberative alignment?.
[24] OpenAI (2025). Detecting and reducing scheming in AI models.
[25] Sarah Hastings-Woodhouse (2025). What is AI debate, and can it make systems safer?.
[26] Julian Michael (2024). Empirical Progress on Debate.
[27] Khan, Hughes, Valentine et al. (2024). Debating with More Persuasive LLMs Leads to More Truthful Answers.
[28] Kenton et al. (2024). On scalable oversight with weak LLMs judging strong LLMs.
[29] Sarah Hastings-Woodhouse (2025). What is Weak-to-Strong Generalisation?.
[30] Burns, Izmailov, Kirchner et al. (2023). Weak-to-strong generalization: Eliciting Strong Capabilities With Weak Supervision.
Evaluation and assurance (33)
[31] Marius Hobbhahn (2024). We need a Science of Evals.
[33] CivAI (2025). When AI Chooses Harm Over Failure.
[34] Rajamanoharan & Nanda (2025). Self-preservation or Instruction Ambiguity? Examining the Causes of Shutdown Resistance.
[35] Schoen et al. (2025). Chain-of-Thought Snippets from Deceptive AIs.
[36] Greenblatt et al. (2024). Alignment faking in large language models.
[37] Anthropic (2025). System Card: Claude Sonnet 4.5.
[38] Lynch et al. (2025). Agentic Misalignment: How LLMs could be insider threats.
[39] Jonathan Ng (2024). ‘3cb’: The Catastrophic Cyber Capabilities Benchmark.
[40] Ristea & Mavroudis (2025). HonestCyberEval: An AI Cyber Risk Benchmark for Automated Software Exploitation.
[41] Anson Ho & Arden Berg (2025). Do the biorisk evaluations of AI labs actually measure the risk of developing bioweapons?.
[42] Dev et al. (RAND) (2025). Toward Comprehensive Benchmarking of the Biological Knowledge of Frontier Large Language Models.
[43] Hackenburg et al. (2025). The levers of political persuasion with conversational AI.
[45] Anthropic (2026). System Card: Claude Opus 4.6.
[46] Anthropic (2026). System Card: Mythos Preview.
[49] Andrew Liu (SecureBio) (2026). Review of Anthropic’s Unredacted Chemical and Biological Risk Report.
[51] OpenAI (2025). GPT-5 System Card.
[53] Google DeepMind (2025). Gemini 3 Pro Model Card.
[56] Rational Animations (2024). What Do Neural Networks Really Learn? Exploring the Brain of an AI Model.
[57] Sarah Hastings-Woodhouse (2024). Introduction to Mechanistic Interpretability.
[58] Robert Wiblin (80,000 Hours) (2025). Neel Nanda on the race to read AI minds.
[59] Dan Hendrycks & Laura Hiscott (2025). The Misguided Quest for Mechanistic AI Interpretability.
[60] Farr et al. (2025). MoSSAIC: AI Safety After Mechanism.
[61] Scott Alexander (2023). Let’s Try To Understand AI Monosemanticity.
[62] Charbel-Raphael Segerie (2023). Against Almost Every Theory of Impact of Interpretability.
[63] Neel Nanda (2025). Interpretability Will Not Reliably Find Deceptive AI.
[64] Connor Leahy (2023). Barriers to Mechanistic Interpretability for AGI Safety.
[65] Sarah Hastings-Woodhouse (2025). What are probing classifiers and can they help us understand what’s happening inside AI models?.
[67] Hubinger et al. (2023). Model Organisms of Misalignment.
[69] Marks et al. (2025). Auditing language models for hidden objectives.
[70] Callum McDougall (2024). ARENA Curriculum: Transformer Interpretability.
[71] Templeton, Conerly et al. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet.
[72] Lindsey et al. (2025). On the Biology of a Large Language Model.
[73] Bogdan et al. (2025). Thought Anchors: Which LLM Reasoning Steps Matter?.
Deployment and serving (7)
[32] METR (2025). AI models can be dangerous before public deployment.
[55] Meta (2025). Llama Guard 4 Model Card.
[74] Robert Miles (2024). Using Dangerous AI, But Safely?.
[75] Sarah Hastings-Woodhouse (2025). Introduction to AI Control.
[76] Sarah Hastings-Woodhouse (2025). What is input/output filtering in AI safety?.
[77] Sharma et al. (2025). Constitutional Classifiers: Defending against universal jailbreaks.
[78] Robert Wiblin (80,000 Hours) (2025). Buck Shlegeris on controlling AI that wants to take over, so we can use it anyway.
Monitoring and iteration (4)
[5] AI In Context (2025). If you remember one AI disaster, make it this one.
[47] Anthropic (2025). Anthropic’s Threat Intelligence Report.
[66] Korbak et al. (2025). Chain of Thought Monitorability.
[68] Obeso et al. (2025). Hallucination Probes.
Around the model (5)
[6] Lewis Hammond (2025). Multi-Agent Risks from Advanced AI.
[79] Davidson, Finnveden, Hadshar (2025). AI-Enabled Coups: How a Small Group Could Use AI to Seize Power.
[80] Aniket Chakravorty, Dewi Erwan (2025). Gradual Disempowerment Summary.
[81] Will Saunter (2025). How AI could enable catastrophic pandemics.
[82] Li-Lian Ang (2025). How AI could enable critical infrastructure collapse.
Careers and research directions (18)
[22] Anthropic Alignment Science (2025). Recommendations for Technical AI Safety Research Directions.
[83] Neel Nanda (2022). Become a person who Actually Does Things.
[84] Li-Lian Ang (2025). Should you do an AI safety research / engineering project?.
[85] Li-Lian Ang (2025). The software engineer’s guide to making your first AI safety contribution in <1 week.
[86] yams & Carson Jones (2024). Talent Needs of Technical AI Safety Teams.
[87] Open Philanthropy (2025). Open Philanthropy’s Technical AI Safety RFP: Research Areas.
[88] Hilton et al. (2025). UK AISI’s Alignment Team: Research Agenda.
[89] Greenblatt et al. (2025). Recent Redwood Research project proposals.
[90] Li-Lian Ang (2025). I’m an experienced software engineer. How can I contribute to AI safety?.
[91] BlueDot Impact (2026). 1-Pager Template.
[92] Charlie Rogers-Smith (2024). Alignment careers guide.
[93] Marius Hobbhahn (2022). Some advice on independent research.
[94] 80,000 Hours (2023). Career Review: Technical AI Safety.
[95] Gabe Mukobi (2022). Levelling up in AI safety Research Engineering.
[96] John Wentworth (2019). Being the (Pareto) Best in the World.
[97] Toby Shevlane (2021). How to succeed as an early-stage researcher: the “lean startup” approach.
[98] John Wentworth (2021). How To Get Into Independent Research On Alignment/Agency.
[99] Neel Nanda (2025). How To Become A Mechanistic Interpretability Researcher.
Outside the course (28)
[100] Chip Huyen (2022). Designing Machine Learning Systems.
[101] Chip Huyen (2025). AI Engineering.
[102] Llama Team, Meta (2024). The Llama 3 Herd of Models.
[103] Lambert et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training.
[104] Longpre et al. (2023). The Data Provenance Initiative.
[105] Hao et al. (2024). Training Large Language Models to Reason in a Continuous Latent Space.
[106] Qi et al. (2023). Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!.
[107] Betley et al. (2025). Emergent Misalignment.
[108] Greshake et al. (2023). Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection.
[109] Egashira et al. (2024). Exploiting LLM Quantization.
[110] OpenAI (2025). Sycophancy in GPT-4o.
[111] Chen, Zaharia and Zou (2023). How Is ChatGPT’s Behavior Changing over Time?.
[112] 80,000 Hours (2026). Claude Mythos: hacking and alignment (summary).
[113] Christopher Potts (2026). The fragile foundations of CoT monitoring.
[114] Cloud, Le et al. (2025). Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data.
[115] Blank, Ying, Potts, Hase and Huang (2026). Sycophantic Agreement Transfers with Neutral Data via Contrastive Preference Optimization.
[116] Cheng et al. (2026). Verbalizing LLMs’ assumptions to explain and control sycophancy.
[117] Kwon, Lamerton et al. (2026). AIs with Secret Loyalties are a Serious but Addressable Threat.
[118] Jacob Steinhardt (Transluce) (2026). Foundation Models for Oversight.
[119] Transluce (2026). Incident Reports.
[120] Joe Carlsmith (2025). AI for AI safety.
[121] Guidelight AI Standards (2026). Control Assessment of Frontier AI Companies.
[122] AI Lab Watch (2025). Lab safety ratings (archived September 2025).
[123] Neel Nanda (2022). A Longlist of Theories of Impact for Interpretability.
[124] Turbé, Bjelogrlic, Lovis and Mengaldo (2023). Evaluation of post-hoc interpretability methods in time-series classification.
[125] Brill and Greenspan (Apart Research) (2025). Problem Areas in Physics and AI Safety.
[126] Denain, Kwon and Ho (Epoch AI) (2026). Toward an O*NET for AI R&D.
[127] Anthropic Alignment Science (2024). How to replicate and extend our alignment faking demo.