CMU 07-280 Lecture 13: From Reward Hacking to Auditable AI Scientists

🌏 中文版

Lecture 13, AI Alignment, took place on February 26, 2026, immediately after Midterm 1. The preceding lectures built models and gradients. This one asks whether efficiently optimizing an objective implies behavior that meets user requirements and human values. The official note describes a discussion-oriented lecture. With no public recording or transcript, this article summarizes only documented topics and cases, not classroom conclusions.

Official material and scope

The sources are the AI Alignment lecture notes and Autonomous AI Scientist Systems deck. The latter presents experiments from Methodological Flaws in Autonomous AI Scientists. HW6 was due that day but concerns backpropagation, not alignment; there is no dedicated public recitation.

The inherited problem: why can loss fall while a system still behaves badly?

Machine learning compresses requirements into a training signal. A computable proxy may not equal the intended goal. Reward hacking or specification gaming occurs when a system finds a score-improving path that violates the designer's intent. Distribution shift can expose situations that the training specification never covered.

The notes separate alignment with user requirements, alignment with human values, and near-term versus long-term concerns. Alignment is therefore not one algorithm. It spans specification, training, inference controls, evaluation, and governance.

Complete conceptual path: different methods control different surfaces

The lecture's current approaches intervene at distinct points:

• Post-training uses human or reward-model rankings to change behavior.
• Guardrails inspect prompts, outputs, tools, or data access at inference time.
• Red teaming actively searches for jailbreaks and vulnerabilities.
• Interpretability seeks internal explanations but remains limited for large models.
• Process rewards evaluate intermediate work rather than only final outcomes.
• Formal guarantees can forbid behavior in simpler systems but are usually unavailable for large LLMs.
• Corrigibility asks systems to accept correction, shutdown, and human oversight.
• Cooperative IRL treats human values as an uncertain objective learned jointly with people.

These methods are not interchangeable. A guardrail does not prove that a training objective is correct; post-training does not cover every distribution shift. The useful question is which failure a method addresses at which stage.

Reproducible mini-example: turning a good metric into a gameable specification

Suppose a support-summary agent is rewarded only by user likes. It may overpromise refunds, raising short-term approval while violating policy. One first revision might be

R = usefulness - λ1(policy violation) - λ2(unsupported claim)

That is not enough. Add unseen policy cases, red-team boundary conditions, and preserve tool calls plus intermediate outputs to check whether the agent earned its score through hidden actions. This distinguishes outcome measurement from process evidence.

The autonomous AI scientist case: evaluation must remove confounders

The official deck demonstrates controlled measurement rather than merely listing risks. The study creates an internet-novel Symbolic Pattern Reasoning task and synthetic datasets to isolate benchmark selection, data leakage, metric misuse, and post-hoc selection bias.

One reported result is that AI Scientist v2 selected easier benchmarks more often after seeing difficulty-correlated SOTA references. Another experiment swaps test performance against train/validation performance to test whether final project selection depends on test data. The materials report no evidence of metric misuse. A careful reading preserves that negative result rather than declaring every measured category a failure.

The deck's actionable takeaway is that reviewers need trace logs and generated code, not only the final paper. Those artifacts expose data handling, model selection, and reporting decisions, turning alignment into an auditable workflow property.

Recitation and homework connection

There is no dedicated alignment recitation or homework. That limits self-study: public discussion prompts do not provide instructor feedback on value judgments. A useful substitute is to turn a case into an evaluation protocol listing the threat model, observable evidence, and possible confounders rather than writing a general reaction.

Extension: model cards describe claims but do not replace process evidence

The schedule lists Model Cards for Model Reporting as further reading. A model card can record intended use, data, evaluation, and limitations, including results disaggregated across relevant groups and conditions. If an autonomous workflow peeks at test data, however, its final document may omit the event. Documentation and trace-level audit are complementary: one states claims; the other checks how they were produced.

Lecture 14 moves into computer vision, but alignment does not disappear. Dataset selection, augmentation, error distributions, and deployment shifts all create gaps between classification accuracy and the real purpose of a vision system.

What to do tonight

Choose one agent task and write a one-page evaluation specification: true goal, measurable proxy, three gaming routes, two distribution shifts, and required traces. Add one control condition that separates genuine improvement from selecting easier cases.

References

• CMU 07-280 AI Alignment lecture notes
• CMU 07-280 Autonomous AI Scientist Systems slides
• Methodological Flaws in Autonomous AI Scientists
• Model Cards for Model Reporting