DRAFT — under teacher review.

Designing an Evaluation Matrix

The Hamilton and Alexandra College · Year 12 · 2026

Most students learn to fill in an evaluation matrix. Far fewer learn to design one — and that is the harder, more valuable skill. A good matrix is not a neutral measuring device that fell from the sky; it is a series of justified design decisions. Change who you are evaluating for, and a sensible matrix changes with it.

This matters directly for Unit 4 Outcome 1, where you evaluate your own solution against criteria you chose and you must justify. The single most common failure is writing criteria after the solution is built, to flatter what you happened to make. Designing the matrix first — and defending it — is how you avoid that.

Start by taking a great matrix apart

Before you build one, reverse-engineer one that professionals already trust. CHOICE magazine's May 2026 headphones test ("Hi-fly way" — the best headphones for travel) is an excellent model. Look past the scores and ask why the matrix is shaped the way it is:

It produces a single composite CHOICE Expert Rating from six weighted criteria: sound quality 45%, active noise cancellation 15%, ease of use 17.5%, battery 10%, comfort 7.5%, durability 5%.
The weights are not equal, and they are not arbitrary — they encode a point of view. Sound at 45% says, plainly, that it matters nearly as much as everything else combined; ANC earns its own 15% precisely because this is a travel test (planes, trains, noisy hostels).
Two things are measured and shown in the table but deliberately excluded from the rating: microphone quality and cleaning. CHOICE reports them because some readers care, but decided they should not move the headline score.
Criteria use different measurement methods: lab measurement for sound and ANC (measured with ANC active at maximum reduction), a subjective panel for comfort, and claimed-versus-measured figures for battery.
It sets a recommendation cut-off — CHOICE only recommends headphones scoring 75% or above. A matrix can decide not just ranking but what counts as good enough.
It handles missing criteria gracefully: for headphones with no ANC or no battery, those weights are redistributed evenly across the remaining criteria, so different product types are still judged fairly.
The weights themselves were revised (new as of April 2025) — proof that even a trusted matrix is a set of decisions its owners revisit, not a law of nature.

Every one of those is a decision someone made and could defend. That is what you are learning to do.

The seven design decisions

Designing a matrix means making — and justifying — these seven choices, roughly in order:

Define the user. Criteria flow from who you are evaluating for. "Best headphones" is meaningless; "best headphones for a daily train commuter" tells you what to measure.
Choose criteria that matter to that user. Not everything that can be measured should be. Pick the handful that decide the outcome for this person.
Decide how each criterion is measured. Objective lab test, subjective rating, feature checklist, or spec sheet — and say which, because it changes how much to trust the number.
Assign weights, and justify them. Weights are where your point of view lives. Equal weights are themselves a claim (that everything matters the same) — usually a weak one.
Choose a scale. 0–5, 0–100, or a traffic-light system. Be consistent so scores are comparable across options.
Decide what to exclude — on purpose. Naming what you leave out, and why, is a mark of a considered matrix (CHOICE excluding microphone quality). Silent omissions look like oversights; stated exclusions look like judgement.
Stress-test it. Swap the user, try to game it, and hunt for the criterion you forgot. A matrix that survives this is one you can defend.

Watch: how weighting works (5 min)

🎯 Watch for: how the weights — not the scores — are where "what matters most to this user" actually gets encoded.

The Weighted Scoring Decision Matrix — airfocus. (It's framed for product managers; the method is exactly the same for software evaluation.)

Worked example — choosing a Python IDE

Let's make all seven decisions explicit, using three tools you might actually meet: Thonny, VS Code, and IDLE.

User: a Year 11 student in their first programming class, on a school laptop, who needs to get a program running quickly without being overwhelmed.
Criteria that matter to them: how fast you get started, how simple the interface is, how helpful it is for a beginner's errors, whether it runs on school hardware, and how much room there is to grow.
Measurement method: getting started = time to first running program (objective); simple interface and beginner debugging = a subjective rating against a checklist; runs on school laptop = does it install and run within the SOE; room to grow = a feature checklist.
Weights, justified: a first-week beginner is won or lost on getting started and not being overwhelmed, so those carry the most; "room to grow" barely matters yet, so it carries the least.
Scale: 0–5.
Deliberate exclusion: extension ecosystems and Git integration are reported but not scored — they matter enormously to a professional, and not at all to this student in week one.

Criterion (weight)	Thonny	VS Code	IDLE
Getting started — time to first run (30%)	5	2	4
Simple, uncluttered interface (25%)	5	2	4
Helpful for beginner errors / step-debugging (25%)	5	3	2
Runs on the school laptop (12%)	4	3	5
Room to grow (8%)	3	5	2
Weighted total / 5	4.7	2.6	3.5

(Scores are illustrative judgements, not benchmarks.) For this user, Thonny wins — which makes sense, because it was designed for exactly this person.

Stress-test your matrix

The result above is only as good as the persona behind it. Pressure-test it three ways:

Swap the persona. Re-run it for a Year 12 student mid-SAT, building a large program. Now "room to grow", debugging power and extensions matter far more, getting-started far less — re-weight, and VS Code climbs to the top. Same options, different user, opposite answer. That is the whole lesson.
Try to game it. Could someone inflate one weight to force their favourite to win? If a single criterion at 60% decides everything, your matrix is really a one-criterion decision wearing a disguise.
Hunt the missing criterion. What did you forget? Cost? Offline use? Accessibility? A matrix is only as honest as its least-considered gap.

VCE names software quality factors — effectiveness, efficiency, usability, reliability, maintainability, robustness, security. Treat that list as a menu to select from, never a ready-made matrix. Your job is to select the few that matter to your user, weight them, and justify both choices. Tipping the whole menu into a table with equal weights is the opposite of evaluation. For the efficiency-versus-effectiveness distinction specifically, see Efficiency vs Effectiveness.

Connection to your SAT

In C4-2 you apply this exact skill to your own design ideas — building your evaluation matrix to decide which design proceeds. In Unit 4 Outcome 1 you do it again to evaluate your finished solution. The discipline is the same every time: decide and defend the criteria before you judge anything against them. Criteria written to flatter a finished product fool no examiner.

Practise

A Year 12 student wants to choose note-taking software for VCE — OneNote, Notion, or Obsidian. Design an evaluation matrix for this user: define the persona precisely, choose and weight four or five criteria (with a measurement method for each), name one thing you deliberately exclude and why, then state which tool you would recommend (explain) and what you trade away by not choosing the runner-up (justify).