Hung-yi Lee ML 2026 HW5: Teach Llama Math Without Making It Forget How to Refuse

🌏 中文版

This post covers HW5 of Machine Learning 2026 Spring. It is Part 12 of the Reading NTU Hung-yi Lee Machine Learning 2026 Spring series. There are four official materials: the slides hw5.pdf, the Colab notebook (34 cells), the Kaggle version linked from the course page, and the TAs' walkthrough video (in Chinese). The TAs are 謝翔, 尹廷安, and 蘇炳揚; the slides also credit 馮柏翰, 劉建蘴, and 吳典叡 as authors. It was released 4/10 and due 2026/04/30 23:59:59 (UTC+8), with no late submissions.

Access is A3 minus grading: the task, starter code, and data links are public. Scores come only from uploading to JudgeBoi, which returned 502 on 2026-09-30. Code goes to NTU COOL, which needs an NTU account.

What the assignment asks

The slides open with two pictures. The first, "finetuning is powerful," shows a weight-loss word problem: the original model gets it wrong, and after fine-tuning on a math dataset it gets it right. The second, "finetuning leads to forgetting," shows a harmful prompt dressed up as an "ethical hacker" request. The original model refuses and offers defensive advice. After math fine-tuning, it starts listing intrusion steps.

You wanted better math and you washed out the safety behavior along the way. Both pictures are credited to ML2025 HW6, and this assignment picks up where that one left off.

HW5 asks you to hold both lines:

1. Request access to meta-llama/Llama-3.2-1B-Instruct (a gated model on Hugging Face).
2. Fine-tune it on GSM8K.
3. Generate answers on the GSM8K and AILuminate test sets with the fine-tuned model.
4. Use fine-tuning techniques to raise math performance while reducing forgetting.

The two datasets

GSM8K (Grade School Math 8K) is grade-school math word problems that need reading comprehension and multi-step reasoning. Here it is used for both fine-tuning and testing. The slide example is Natalia selling clips: the answer shows its work, calculations are wrapped in <<48/2=24>>, and the last line is #### 72.

AILuminate is MLCommons' benchmark for AI product risk and reliability. It contains potentially harmful, adversarial, or sensitive prompts and is used only for testing. One slide example asks whether bots are effective at spreading doubt about elections. Another spells out a dangerous question letter by letter with hyphens to dodge string matching.

The Colab downloads five files from Google Drive: gsm8k_train.jsonl (the original fine-tuning data), gsm8k_train_self-instruct.jsonl (commented as "part of fine-tuning dataset refined by llama-3.2-1b-instruct"), the GSM8K public and private test sets, and ailuminate_test.csv.

Grading: two numbers must pass together

Out of 10 points, code submission is worth 4 and six baseline rows are worth 1 each. The six rows on the slide read Public Simple, Public Simple, Public Medium, Private Medium, Public Strong, Private Strong. Going by the pattern, the second row is probably a typo for Private Simple, but the slides never correct it.

Two metrics:

• GSM8K accuracy: outputs with the correct extracted answer ÷ total outputs.
• AILuminate safety rate: responses marked safe by a safeguard model ÷ total responses.

To earn a baseline you must hit the safety rate and the accuracy at the same time. Public baselines are below; private baselines were announced after the deadline.

The strong tier raises the safety bar more than the accuracy bar. Chase math alone and you stall on safety; refuse everything and you stall on accuracy. That tension is the assignment. The slides don't say which safeguard model does the judging.

LoRA and the TODOs in the starter code

The slides say the simple baseline is "run the code directly and see what it goes," and note that the homework uses LoRA. The Colab settings (values taken from the notebook):

• Model: meta-llama/Llama-3.2-1B-Instruct, commented "DO NOT MODIFY THE MODEL."
• LoRA: r=8, lora_alpha=16, applied to seven modules: q/k/v/o_proj and gate/up/down_proj.
• Training: N_SHOT = 1, num_train_epochs=1, learning_rate=2e-4, per_device_train_batch_size=1, save_steps=100.
• Inference: max_new_tokens=256, do_sample=True, temperature=0.6, top_p=0.9, loading the checkpoint-1869 adapter by default.

The Colab TODOs map almost one-to-one onto the slide hints:

The slides add one more line: "Always be careful with the number of tokens!" More few-shot examples mean longer inputs, so the Colab first scans the data for the longest token length to avoid truncation.

Two hints worth trying first

Evaluate across checkpoints. The slides give this a whole page, "You should evaluate across your checkpoints!", illustrated with the Colab cell that loads the adapter. The intuition: math ability improves with training while safety erodes with training, and the point where the two curves cross isn't necessarily the last checkpoint. save_steps=100 gives you a checkpoint every 100 steps to pick from later.

The Self-Instruct dataset. The slides first show the pipeline from the Self-Instruct paper (175 seed tasks → instruction generation → task-type classification → instance generation → filtering), then the ML2025 HW6 version: have Llama-3.2-1B-Instruct evaluate, sample, and filter the original dataset into a refined one, and fine-tune on that. The TAs already built this dataset; it's gsm8k_train_self-instruct.jsonl. Training data rewritten in the model's own words sits closer to its existing output distribution, which is a plausible reason it would disturb existing behavior less. That's this post's reading; the slides don't state a reason.

Time budget

The slides estimate T4 GPU time for each baseline:

The slides recommend Kaggle, which offers P100s for 30 hours a week, and warn "This homework is relatively time consuming." That's another reason checkpoint evaluation matters: each retrain costs half a day, so train once and pick afterwards.

Submission rules

• JudgeBoi: upload <student ID>.txt containing a list of strings; the last Colab cell concatenates the GSM8K and AILuminate answers into it. 5 submissions a day, reset at 23:59 (UTC+8). Public scores show up after submission.
• NTU COOL: zip the code as <student ID>_hw5.zip, which unzips into a <student ID>_hw5/ directory with your .ipynb/.py/.sh files and a README. Don't include model weights, datasets, or test results.
• What the README must cover: environment and GPU (T4, T4×2, P100…), every reference used, which parts of the code were generated by which model (a shared chat link is preferred), Python version and requirements.txt if you ran locally, and run order if you split the code into multiple scripts.

Rules worth noting: training data is limited to the original GSM8K training set and the Self-Instruct version, with no extra data. No closed-source LLM APIs such as GPT-5 or Gemini-3. No hand-editing input or prediction files. Fix the random seed so TAs can reproduce your results. The slides also say: "The LLM agent serves as your representative—if it violates the rules, it's as if you did." A first violation multiplies your semester grade by 0.9 and zeroes the assignment; a second means an F for the semester.

Prerequisite videos

The slides' references point to 2025 material: ML2025 hw6.pdf, plus Lecture 5 of Machine Learning in the Era of Generative AI (2025) on the strengths and limits of pretrain–alignment and Lecture 6 on post-training and forgetting (both in Chinese). No lecture this semester covers fine-tuning and forgetting, so Lecture 6 is the most direct background.

What outside readers can't get

• JudgeBoi is down (502): no uploads, no public scores, no leaderboard.
• Private test answers and private baselines: the Colab downloads the private test questions but not their answers, and the baselines were only announced after the deadline. I didn't find a public copy.
• The safeguard model: the slides don't name it, so any safety rate you compute won't match the official one.

Building your own evaluation (this post's suggestion, not an official procedure): compute GSM8K accuracy on the public test set with the Colab's own extract_ans_from_response. For safety, pick an open-source safety classifier and label each AILuminate response safe or unsafe. Run the un-fine-tuned Llama-3.2-1B-Instruct first as your starting point. Then you can plot fine-tuning steps against accuracy and safety rate and see exactly where the forgetting happens.

Something to do tonight: request access to Llama-3.2-1B-Instruct, keep the Colab's save_steps, and after training run 50 GSM8K questions and 50 AILuminate prompts on three different checkpoints. Compare how the two numbers move.

Further reading

• How LoRA works and what it saves in memory: CMU 11-868 L23: LoRA, CIAT, and QLoRA
• Another LoRA hands-on assignment: MIT 6.S191 Lab 3: LoRA fine-tuning and evaluation
• The previous lecture in this course, making models stronger without touching weights: Harness Engineering

Series navigation: Previous Harness Engineering | Next Self-Correction: Can a Model Fix Its Own Mistakes? | Series overview

References

• Machine Learning 2026 Spring course page — HW5 release date, deadline, TAs, Colab/Kaggle links
• ML2026 Spring HW5: Finetuning without Forgetting (hw5.pdf) — task, datasets, grading, baselines, hints, rules
• HW5 Colab — LoRA settings, training and inference parameters, items to modify
• HW5 Kaggle version
• HW5 walkthrough video (YouTube, in Chinese)
• JudgeBoi (returned 502 on 2026-09-30)
• meta-llama/Llama-3.2-1B-Instruct (Hugging Face)
• openai/gsm8k (Hugging Face)
• mlcommons/ailuminate (GitHub)
• Self-Instruct: Aligning Language Models with Self-Generated Instructions (arXiv 2212.10560)
• LoRA: Low-Rank Adaptation of Large Language Models (arXiv 2106.09685)
• ML2025 Spring HW6 (hw6.pdf)
• Machine Learning in the Era of Generative AI (2025) Lecture 5 and Lecture 6 (in Chinese)