This version replaces the one published on 9 September. Download the notebook again and use this version. The top of the notebook shows its version date.
What changed:
autoencoders and diffusion_models. All earlier topic names are still accepted (Section 2).If you checked a question with an earlier version, run it again. The verdict can change.
In one sentence: write six math questions that a specific AI model gets wrong, and explain why you think it will get them wrong.
This is not about writing hard or tricky questions for its own sake. It is about finding a real weak spot in how a model reasons — the kind of weak spot you can only find if you actually understand the material yourself. Simply asking an AI tool to "make a hard question" will not work reliably, because you also need to know the model well enough to predict, in advance, why it will fail. That is the actual skill being tested here.
Write 6 original questions. Each question must:
Across your 6 questions, you must use at least 4 different topics from the list in Section 2, and no more than 2 questions on the same topic. This stops everyone from writing all 6 questions about the one topic that happens to be easiest to break.
Pick one topic name from the list below for each question and use it in the category field. All previously allowed topics remain valid, including kernels and SVMs, which are no longer taught this semester. The list also covers the new topics in the updated course schedule.
| Topic name (use this exact word) | What it covers |
|---|---|
| regression_prob | Regression, probability, MLE/MAP (Weeks 1–2) |
| classification | Classification, naive Bayes, overfitting, bias-variance (Weeks 3–5) |
| pac_learning | Learning theory / PAC bounds (Week 6) |
| nn_backprop | Neural networks, backpropagation, CNNs, transformers, SGD (Weeks 8–9) |
| kernels_svm | Kernels, support vector machines (still accepted) |
| unsupervised | k-means, PCA, Gaussian mixtures, EM, latent variable models, variational inference, VAEs, ELBO, KL divergence (Weeks 10–11) |
| autoencoders | Autoencoders: encoding and reconstruction (Week 10) |
| diffusion_models | Diffusion models, forward noising and denoising (Week 12) |
Choose the topic that best matches the main concept tested. Use unsupervised for clustering, PCA, Gaussian mixtures, EM, latent variable models, variational inference, VAEs, the ELBO and KL divergence. Use autoencoders only for questions about autoencoders themselves, such as encoding and reconstruction. Do not assign different tags to similar questions just to meet the topic coverage requirement.
You do not need to replace questions that already meet the assignment requirements, and you do not have to use the new topics. The requirement remains 6 questions, at least 4 different topics, and at most 2 questions per topic.
Here is one real, complete question, written the way yours should look:
Question: A hard-margin SVM has weight vector w = [6, 8]. The two supporting hyperplanes are w·x + b = 1 and w·x + b = −1. Report the perpendicular distance between these two hyperplanes.
Your solution: ‖w‖ = √(6² + 8²) = √100 = 10. The full margin width is 2/‖w‖ = 2/10 = 0.2.
Answer: 0.2 (accept anything within 1% of this)
Why you think the model will fail: Models sometimes report 1/‖w‖ instead — the distance from just one hyperplane to the decision boundary — because the word "margin" is used loosely in different textbooks to mean either quantity. The full distance between the two hyperplanes is 2/‖w‖, not 1/‖w‖.
This question is well formed: it is short, self-contained, has exactly one number as the answer, and the written reason is specific to this exact question rather than a generic guess.
This example is here to show the format. It is not an actual stump: Qwen3-1.7B answers it correctly with 0.2. It is also the sample question in the notebook, so you can see this for yourself. A well-formed question with a plausible failure reason is the starting point; only running it against the model, as in Section 5, tells you whether the model actually falls for it.
Submit one JSONL file (a plain text file with one JSON object per line — see the worked example below). It should contain your 6 questions, one per line. Each question needs these nine pieces of information, plus one optional field if you are claiming the bonus in Section 7:
| Field | What goes here |
|---|---|
| student_id | Your matriculation number (e.g. A0123456X) — the same on all 6 lines |
| question_number | Which question this is: 1 through 6 |
| category | One of the topic names listed in Section 2 |
| failure_mode | A short label for why you expect the model to fail — one of: formula_confusion, sign_or_direction, multi_step_compounding, edge_case_or_degenerate, arithmetic_precision, definition_ambiguity, or other |
| question_text | The question itself — nothing else. Do not add your own "give me the final answer" instruction; we add this automatically so every student's question looks the same to the model. |
| derivation | Your full written solution, step by step |
| final_answer | The correct number |
| answer_tolerance | How close a model's answer needs to be to count as correct — an object with a type ("relative" or "absolute") and a value. Use relative (e.g. 1% = 0.01) unless your answer is 0 or very close to 0, in which case use absolute instead. |
| failure_prediction | 2–3 sentences, in your own words, on why you expect the model to get this specific question wrong |
| bonus_note | Optional — include it only if you are claiming the frontier-model bonus from Section 7 for this question. Name the exact model and version you tested, the date you tested it, and what it answered. Leave the field out otherwise. |
This is the same worked example from Section 3, written exactly as it should appear in your file. In your actual file, this whole thing is one single line — it's only broken across multiple lines here to make it easier to read.
{
"student_id": "A0123456X",
"question_number": 1,
"category": "kernels_svm",
"failure_mode": "definition_ambiguity",
"question_text": "A hard-margin SVM has weight vector w = [6, 8]. The two supporting hyperplanes are w.x + b = 1 and w.x + b = -1. Report the perpendicular distance between these two hyperplanes.",
"derivation": "||w|| = sqrt(6^2 + 8^2) = sqrt(100) = 10. The full margin width is 2/||w|| = 2/10 = 0.2.",
"final_answer": 0.2,
"answer_tolerance": {"type": "relative", "value": 0.01},
"failure_prediction": "Models sometimes report 1/||w|| instead of the full 2/||w|| width between both hyperplanes, because the word margin is used ambiguously to mean either quantity."
}
Your finished file has 6 lines like this one, back to back — one per question, each a complete, valid JSON object on its own line.
If you are claiming the bonus for a question, add "bonus_note": "..." as a tenth field on that question's line. Section 7 of the notebook checks your finished file against all of these rules.
When: by Friday 13 November 2026, 23:59 (Singapore time) — the last Friday of lectures, in Week 13.
Where to submit: through Canvas, as a single file upload to the Assignment 1 submission point.
File name: name your file <your_student_id>_submission.jsonl — for example, A0123456X_submission.jsonl.
We use one fixed AI model for the whole term, named exactly below. It is picked so it is neither too weak (everyone would beat it easily) nor too strong (nobody could beat it) — it sits in a range where a genuinely good question can succeed and a weak one cannot.
Qwen3-1.7B (Hugging Face id: Qwen/Qwen3-1.7B), run with these fixed settings:
A ready-to-run notebook, cs3264_assignment1_test_your_question.ipynb, is provided together with this document. Open it directly in Google Colab (free, needs only a Google account), go to Runtime → Change runtime type and select a T4 GPU, then paste in your question text, your final answer, and your tolerance. It downloads and runs the exact same model with the exact same settings used for grading — greedy decoding, thinking mode on, the same system prompt and the same fixed instruction appended to your question — and tells you directly whether the model matches your answer.
Section 1 of the notebook installs the reference package versions, including PyTorch 2.11.0 with CUDA 12.8 and Transformers 5.16.1. After installation, restart the session and continue with the import cell, which checks the versions. Grading uses an NVIDIA T4 with these pinned package versions. The notebook also repeats your question to check stability in your current runtime. Matching runs do not guarantee identical answers across different software versions or GPUs. If you checked a question with an earlier version of the notebook, run it again with this version.
The notebook reports one of four verdicts: NOT STUMPED, STUMPED, STUMPED (no readable number), or NO RESULT. The last one means the model ran out of its 8192-token budget before finishing, usually because it fell into a repetition loop. Such a run earns no stump marks in grading (see Section 6), so if you see it, rewrite the question until the model commits to a definite answer.
You can also get a rough first impression of how today's AI models handle your question on Qwen Chat. This is faster, but less reliable: Qwen Chat currently only offers newer Qwen versions than the one used for grading, and it does not let you control decoding settings, so it cannot confirm reproducibility the way the notebook can. Use it as an early filter, not as your final check — the notebook above is what actually tells you whether a question works.
After the deadline, every student's 6 questions are sent to the model, one at a time, using the exact same setup for everyone:
Separately, a teaching assistant checks that your written solution is actually correct and that the question really belongs to the topic you chose. A question with a wrong or unclear solution does not earn stump marks, even if the model also happened to get it wrong — the point is to find genuine weaknesses, not to submit broken questions.
| Part | Marks | How it's marked |
|---|---|---|
| Valid questions with correct, well-written solutions | 54 | 9 marks per question, marked by a TA |
| Questions that actually stump the model | 32 | 8 marks per stumped question, up to a maximum of 4 (you do not need to stump all 6) |
| Your written reasons for why the model would fail | 14 | Marked leniently — about how specific and thought-through your reason is, not whether it turned out to be right |
Valid questions and solutions (54 marks, 9 per question):
A question with little or no marks here also cannot earn stump marks — a broken question "beating" the model doesn't count.
Stumps (32 marks, 8 each, capped at 4): this part is decided automatically by running your questions through the model, as described in Section 6. Only questions that pass the check above are eligible.
Failure reasons (14 marks total):
A good reason still earns full marks even if that particular question does not end up stumping the model — you are marked on the quality of your thinking, not on the outcome.
Example: 6 valid, well-written questions gives the full 54; 2 real stumps gives 2 × 8 = 16. That's 70 from those two parts alone, plus up to 14 more depending on how specific your failure-reason writing is — solid, honest work like this lands roughly in the low-to-mid 80s out of 100.
This bonus is not added to Assignment 1's own score. If one of your questions also stumps a frontier-level model — GPT-5.5 or later, or Claude 4.7 or later — you earn up to 10 percentage points added to your Assignment 2 score, capped so Assignment 2 cannot go above 100%. It only helps if your Assignment 2 mark was below full marks.
To claim it, add the optional bonus_note field to that question's line in your file (see Section 4): name the exact model and version you tested, the date, and what it answered. Keep a screenshot or a shareable link to the conversation in case a TA asks to see it.
We compare every student's questions against every other student's, using an automatic similarity check. Two very similar questions from different students get flagged for a TA to look at by hand. This is only a flag for review, never an automatic penalty — and we never reward a question for being "more different," so there is no reason to deliberately make your questions strange just to avoid a flag.
STUMPED on both runs — not NO RESULTbonus_note naming the exact model and version you tested