This version replaces the one published on 12 October. Download the notebook and the JSON template again and use this version. The top of the notebook shows its version date.
What changed:
autoencoders and diffusion_models (Section 2).In one sentence: take a question that Qwen3-1.7B gets wrong, build a repair that makes it answer correctly, and test whether the repair also works on questions it has not seen.
In Assignment 1 you found questions that the model gets wrong. In this assignment you fix one of those mistakes. The fix must be reusable: it has to work on new questions, not just the one you started from. You will test it the way an ML method is tested: against a baseline and a control, on new questions it has not seen, and on questions the model already gets right.
Files: cs3264_assignment2.ipynb (notebook) · submission_template.json (JSON template) · Open in Google Colab ↗
Open the notebook in Colab with the link above, or upload the downloaded file with File → Upload notebook. Read the recorded examples first. You need a GPU runtime only to run the model yourself (we suggest a T4, see Section 6).
The rest of this page is the detailed instructions for the assignment. Please read it carefully.
Your task is to choose one ML question that the unmodified Qwen3-1.7B gets wrong, explain the mistake it makes, and build one repair that fixes it. Then compare three things on the same questions: the original model (the baseline), the model with your repair, and a control that lets you tell what your repair actually contributes.
Write up an explanation that is supported by your experimental results. Describe what fails, why your repair might help, and how the results support your reasoning. A careful, well-supported negative result earns full marks.
Training the model is optional. Prompting, worked examples, retrieval, tools, verification, and parameter-efficient finetuning are all acceptable repairs.
You submit one ZIP file through Canvas. It contains a short PDF report, your notebook with saved results, a JSON file of your experiment records, and a brief README (Section 7). Start small: read the notebook examples and test your candidate question first. You do not need all six evaluation questions before you begin.
Start from one of your A1 questions, or write a new one. The question must be within the course scope, have one unambiguous numeric answer, and come with your own derivation. Put one of the A1 topic names in the question's topic field: regression_prob, classification, pac_learning, nn_backprop, kernels_svm, unsupervised, autoencoders, or diffusion_models. Section 2 of the Assignment 1 page says which one to use. Give the topic in your report as well.
The two examples in the notebook are for illustration only. Do not submit them, or numerical or wording variants of them, as your own questions, and do not use their recorded responses as evidence. You may work on the same topics and adapt the repair ideas.
Then, in this order:
An ambiguous question, an unfinished response, or a parsing failure is not a mathematical mistake. Fix an invalid question before you test any repair, and say so if you changed your A1 wording. Report what the baseline did, including the cases where it got the answer right.
A1 and A2 use the same model and the same inference settings. Even so, check the actual baseline response. If the model answers your A1 question correctly here, choose another question.
The notebook contains two complete questions with checked solutions and the model's actual responses. They were recorded on an NVIDIA GeForce RTX 4060 with PyTorch 2.12.0+cu126, before the package versions were fixed. The notebook keeps their original settings. Responses can differ across software versions and GPUs, so look at the results from your own setup.
| Example | Original answer | Correct answer | Observable mistake |
|---|---|---|---|
| Gaussian mean with unequal known variances | 3 | 57/31 ≈ 1.83871 | Uses the ordinary mean instead of weighting by inverse variance. |
| Bernoulli naive Bayes with observed zero features | 0.2727 | 9/65 ≈ 0.13846 | Does not take the complement for the observed zero X3 = 0 in class 0. |
The example repair appends a reference card of likelihood formulas to each question. The same model, with unchanged weights, then answers again. The card contains formulas, not the answers to the example questions. The full question texts, solutions, and both responses are in the notebook.
In the recording, the card fixes the Gaussian question (57/31, scored correct) but not the Bernoulli question (0.6, still wrong). The Gaussian response gives a boxed fraction after Final Answer:, which the extractor accepts. Report this kind of result too: giving the model the right formula does not guarantee that it applies the formula correctly.
These examples show the workflow. Do your own investigation, use a control, and evaluate variants of your own question.
Choose one main method, or a combination you can justify. Some options:
This list is not exhaustive.
What your repair may use. While developing it, you may use the anchor question, the development variant, and their solutions, for training or as fixed examples. Keep the held-out and regression questions and their solutions out of training data, example collections, reference material, and tuning. Scores on the anchor and the development variant measure performance on material the repair has seen. Scores on the held-out questions measure transfer.
What your repair must do. It must run on a new question with no help from you. It may pick a rule or a reference based on the question text. It must not receive the question's answer, grading feedback, or hints you wrote for that question. Returning stored answers, or looking up question IDs, is not a repair.
Keep the target model fixed. The repair may train the weights of Qwen3-1.7B, but the baseline you compare against stays unmodified. Do not switch to a stronger model for evaluation. If a tool computes the final answer, record the raw model response and the final system response separately. Cite any references and data your method uses.
Include one control or ablation. A control tells you what your repair contributes. For example, compare a targeted mathematical reminder with a generic "check your work" prompt, or compare a verifier with a method that uses the same number of calls. An ablation removes one component of your repair, such as the reference card. If your card both gives formulas and asks for concise reasoning, a single run cannot tell you which part helped. Choose one comparison that suits your method. You do not need to try all of these.
You evaluate on six questions. Each group answers a different question about your repair:
| Group | Number | Purpose |
|---|---|---|
| Original anchor | 1 | The baseline gets this wrong. It establishes the mistake you are fixing. |
| Development variant | 1 | Use it to improve your repair before the final test. |
| Held-out variants | 2 | Test whether the frozen repair works beyond the questions you developed it on. One numerical/wording variant and one condition/structural variant of your anchor. |
| Regression checks | 2 | Test that the repair does not break answers the baseline already gets right. |
Why these groups? A repair that fixes only the original question may just be tailored to that one example. The development variant lets you improve the method. The held-out variants test transfer. The regression checks reveal side effects. Six questions is a small experiment, and your conclusions should say so. You write the three variants and pick the two regression checks. The anchor can be your A1 question.
Variants. A numerical/wording variant keeps the mathematics the same and changes the numbers or the wording. A condition/structural variant changes an assumption or a relationship so that the solution has to be reworked. For example, replace a linear activation with a named nonlinear one in a gradient question. Say which anchor each variant comes from and solve every variant yourself. It does not matter if the baseline gets a held-out variant right. Evaluate all six questions anyway.
Regression checks are valid ML questions outside your failure case that the unmodified model answers correctly. A regression is a previously correct answer that becomes wrong. For example, a likelihood hint might fix a probability question but break a gradient calculation that was right before. The notebook supplies four candidate questions, and you may also write your own. Verify their solutions, then run the baseline on them and pick two that it gets right. Do this before you test the repair on them. A candidate that the baseline gets wrong cannot be a regression check. Do not use the two regression checks while you develop your repair. Once you have chosen them, keep them, even if your repair gets them wrong.
Freeze before you test. Write and verify all six questions and solutions before you run the final evaluation. "Held-out" means that you never use those questions' model outputs to choose or adjust your repair. Freeze your prompt, code, reference collection, adapter, and settings, then evaluate all six questions. If you change the method after seeing the held-out results, keep the original records, label that run as development, and write fresh held-out questions with new IDs for the final test.
Calls. Run the baseline, your repair, and the control on all six questions. For single-call methods that is 18 calls. Then repeat the baseline and the repair on the anchor in a fresh session, which adds two. Development and screening need more. You may use at most 100 calls to the target model in total. This includes retries, optional reruns of the examples, and every call inside a multi-call method. Reading the recorded examples uses none. Each call may generate at most 8192 tokens. If you train, report the training compute separately and keep to the same evaluation budget.
Reserve at least 20 calls for the final comparison and the anchor repeats. If only your repair changes during development, rerun only the repair and keep the earlier baseline and control results. Notebook Section 5 shows a 31-call plan, with two calls per extra repair-only iteration. Count interrupted calls and any calls made outside the runner, and record them in the notebook.
For a fresh repeat, restart the kernel, reload the frozen method, and use a new run name. Reusing a run name resumes its saved results instead of running again. Before you leave a temporary runtime, download the log backup described in the notebook, and restore it in the next session before you continue. The frozen final run is your main comparison. Report development trials and repeats separately.
Scoring. The notebook's scorer accepts an answer if it is within 1e-5 absolute or 1% relative of yours, whichever is larger. Keep the four outcomes separate: correct, incorrect, unparsed, and truncated. Look at any extraction that seems off, and read the reasoning. A response may give a clear boxed answer without the required prefix. Keep its automatic status, record your own review separately, and apply the same rule to every method. Never take a provisional number from unfinished thinking as the answer.
The extractor reads the same formats as the A1 extractor: Markdown bold, a Unicode minus sign, x = 0.2, trailing units, decimals, fractions (including signed LaTeX fractions and boxed answers), scientific notation, and exponentials such as e^{-2}. It ignores a percent sign, so 20% is read as the number 20. It does not read radicals such as sqrt(2), so review those by hand. Give your answers enough decimal places for the tolerance. Do not change the baseline's instruction.
What to report. For each group, give accuracy as a fraction (correct out of total), the wrong-to-right and right-to-wrong changes, how many responses completed and how many parsed, and the calls, tokens, time, and hardware used. Decide which run is your final run before you look at its results, and report the anchor repeats separately. Do not pick the best run afterwards.
Automatic accuracy for a method is the number of its six final responses scored correct, divided by 6. Completed-response mathematical accuracy is the number of finished responses whose final answer is mathematically correct, including unparsed ones that you reviewed, divided by the number of finished responses. Report it next to the automatic accuracy, not instead of it. If no response finished, report N/A. For a single group, divide by that group's size instead of 6. A response that hit the token limit counts as not delivered. It is not evidence of a mathematical mistake.
These metrics score the delivered response. For ordinary prompting that is the raw model response. If a tool or a selector changes the delivered answer, use the template's final_system_response, system_predicted, system_status, and system_truncated fields for the main comparison, keep both texts, apply the same scorer to both, and report the raw model's scores separately.
Download cs3264_assignment2.ipynb. Open it in Google Colab directly, or open Google Colab, choose File → Upload notebook, and select the file.
MODE="live" and install the packages as described below, then run the cells in order. The model download is about 4 GB, and the rerun uses four calls.MY_QUESTIONS. Connect a GPU runtime, install the packages as described below, and keep MODE="recorded" so the examples are not rerun. run_my_questions always calls the model. Follow the staged calls in Section 5 of the notebook: baseline check, regression screening, development, frozen evaluation, repeat. Enter your repair in PROMPTS["my_repair"] before development, and add the remaining questions before the final evaluation. If you use another kind of method, adapt the runner, and use a new run name whenever an experiment changes.Install once per runtime. In notebook Section 1, set INSTALL_INFERENCE_PACKAGES=True and run the setup cell. After installation, select Runtime → Restart session, set INSTALL_INFERENCE_PACKAGES=False, and run the cells from the top. Live model loading checks the installed versions. Recorded mode needs no inference packages.
The unmodified baseline is the pretrained Qwen3-1.7B (Qwen/Qwen3-1.7B) with its original weights and nothing added: no repair prompt, examples, retrieval, tools, or extra calls. It sees only the fixed system prompt, the question text, and the final-answer instruction printed in the notebook. The settings are the same as the A1 notebook: BF16, PyTorch 2.11.0 with CUDA 12.8, Transformers 5.16.1, Tokenizers 0.23.3, SDPA attention, batch size 1, greedy decoding, thinking enabled, and at most 8192 output tokens. Use the same runtime for the baseline, the control, and the repair, and report any decoding change that is part of your repair. The notebook saves the settings and every completed response, with start and finish timestamps. An unchanged run resumes its saved results. The exported JSON keeps these records so a marker can check the order of development and final evaluation.
A response that reaches the token limit is marked incomplete. Do not raise the baseline's token budget, and do not treat a partial answer as final. A prompt that makes the model more concise can be investigated as a repair under the same budget. Report any improvement in completion separately from the repair of mathematical errors.
A long response can take several minutes, depending on your GPU and the length of the response. The runner announces each new call. If your method changes the decoding settings, record the changes in the method_runtime_overrides field described in the JSON template.
Arrange your own GPU access and start early. Free GPU time has limits and is sometimes unavailable, and accounts can take time to set up. Do not leave the experiments until the last few days. Save completed results, avoid reruns you do not need, and disconnect GPU runtimes when you are done.
The teaching team does not provide GPUs or Colab credits, does not run experiments for you, and does not arrange access. Send account, quota, and infrastructure questions to the provider. If you use another platform, follow its setup instructions and keep the assignment's model and baseline settings. Training is optional. Choose a repair method that fits the resources you have.
When: by Friday 13 November 2026, 23:59 (Singapore time), the same deadline as Assignment 1.
Upload one ZIP file named <student_id>_A2.zip through Canvas → Assignments → Assignment 2. The four required files go at the top level of the archive and use the same student ID. Add folders only if your method needs them.
A0123456X_A2.zip ├── README.md ├── A0123456X_A2_report.pdf ├── A0123456X_A2.ipynb ├── A0123456X_A2.json ├── code/ # Optional └── assets/ # Optional
| File | Purpose and contents |
|---|---|
README.md | A short guide to the package: your student ID, your repair method, where the files are, what to install, and how to view the saved results and run the methods. Explain how to load any extra files. Do not repeat the report's analysis. |
<student_id>_A2_report.pdf | Your analysis and conclusions: the mistakes you observed, why you chose your repair and control, the results by group, and the costs. At most four pages, not counting references. Use any layout. Refer to questions by their IDs and give each question's topic. |
<student_id>_A2.ipynb | Runnable code with saved outputs. Organise it under Questions and solutions, Method and reproduction, Results, and Experiment evidence. Include your independent checks, the actual prompts or code, any references or training data, and short run instructions. This is the entry point, and it may call scripts in the ZIP. |
<student_id>_A2.json | The structured records for marking: questions and solutions, the settings of each run, complete model responses with scores, costs, and any manual reviews. The notebook helper exports it. |
Download submission_template.json for an explanation of every field and a filled example. The example has two questions and six responses to show the structure, and its responses are invented. Your final comparison must contain all six questions and 18 question–method results. Also include your screening, development, and repeat runs, labelled as such.
The JSON file is the record. The notebook holds the code and the readable analysis. If you use the supplied single-call runner, the notebook generates the JSON for you. For other methods, adapt the implementation and the exporter and keep the required fields. Use the same question IDs, run names, and method names everywhere. Keep the automatic scores and put manual reviews in their own records. Exporting needs no new model calls. Do not retype responses, and do not submit the unchanged template. For multi-call or tool methods, keep the intermediate calls and the final system answer, as the template explains.
Follow the notebook's submission steps to export your records and save the notebook with its outputs. A marker must be able to open your results from the JSON alone, without a GPU, a model download, or your log folder. Keep the JSON next to the notebook and turn off live experiment and training calls in the saved-results workflow. The export checks only test the file format.
Use relative paths inside the ZIP. Include what your method needs. Anything already in the notebook does not need a second copy.
| Method | Additional material, where applicable |
|---|---|
| Prompting, verification, or multi-call methods | The actual prompts and execution code, with the intermediate calls kept in the JSON as the template explains. |
| Retrieval or tools | Your code, the reference collection you used, and instructions for loading or rebuilding any index. |
| LoRA or other parameter-efficient finetuning | The final adapter weights, the configuration and loading code, and the training code, data, and settings. |
| Full finetuning | The final model weights, the configuration and loading code, and the training code, data, and settings. |
If you trained a model or an adapter, submit the trained weights you used in the final comparison. This includes a trained control. Include any tokenizer files you changed. A marker must be able to load the weights without training them again. Training code alone is not enough. Describe the training steps and compute in the notebook.
Do not include separate question files, raw log folders, or screenshots. Do not include base-model downloads, caches, virtual environments, optimizer states, or intermediate checkpoints.
Large model files: Canvas limits the size of uploads. If your ZIP is too large, leave the model files out and put a download link in README.md. Name the files and say how to load them. Do not change them after the deadline, and keep the link working until marking is finished. The report, notebook, JSON file, and code always stay in the ZIP.
Before you upload, unzip the file into a new folder and check that the notebook opens the saved results and that every file it refers to is there. After uploading, check that Canvas shows the file name and a submission confirmation. If you resubmit, upload a complete replacement ZIP.
| Component | Marks | What we look for |
|---|---|---|
| Mathematical understanding and failure analysis | 30 | Valid questions, correct derivations, and a correct account of where the model goes wrong. |
| Repair design and justification | 25 | A well-motivated, reusable repair, with clear limits on what information and tools it uses. |
| Evaluation and generalisation analysis | 30 | A meaningful control, held-out tests, regression checks, repeated runs, and conclusions that the evidence supports. |
| Reproducibility and cost reporting | 15 | A submission that runs, complete records, and honest accounting of calls, tokens, and time. |
A careful, well-supported negative result earns full marks.
The frontier-model bonus from Assignment 1 is added to your A2 mark separately, and the A2 mark is capped at 100.
Check that your submission has: