Designing a Metric for Ranked, Multi-Label Classification: Why It's a Business Decision

I'm building data pipelines that parse financial documents and, with an LLM doing the judgment-heavy step, classify them against a taxonomy. The work never stops: new pipeline versions, new prompts, new taxonomies, models to test against each other. And every experiment ends at the same place: "Is the new version better than the last one?"

The question wants a single number. The trouble is that what the pipeline produces isn't a single decision, so most of the work of answering honestly goes into deciding what information is allowed to survive when we collapse a rich, structured output down to one comparable score. This post is about that collapse: the problem that forces it, why the obvious metrics fall short, and the scoring scheme we landed on that finally makes "better" mean something.

The problem: the output is a structured object, not a label

For each document, the pipeline doesn't pick one class. The classification is multilabel: a document can belong to several categories of the taxonomy at once, so the output is a set of labels, with no fixed size. And it goes one level deeper: within that set, each label also carries a rank, primary or secondary. A primary label is a core one, central to what the document is fundamentally about. A secondary label is a supporting one: relevant and worth capturing, but not what the document is mainly about. The distinction is about importance, not just presence, and that's what makes getting the rank right as consequential as getting the label right.

So a single prediction isn't a label but a small structured object: a set of (label, rank) pairs, something like

{ (A, primary), (C, secondary), (F, secondary) }

That's one primary category and two secondary ones, drawn from a taxonomy of many possible labels. (The taxonomies themselves are domain-specific and we swap them in and out; what stays constant is this two-level shape, so that's what the evaluation has to handle.)

That shape is the whole difficulty. There are really two tasks stacked on top of each other, and a good pipeline has to get both right:

1. Multilabel classification: which labels apply? (a set, with no fixed size)
2. Ranking: among the labels that apply, which ones are core to the document and which merely support it? (primary vs secondary, with no fixed number of either)

A pipeline can ace one of these and fail the other. It can find exactly the right set of labels and then rank them backwards. It can rank impeccably over a set that's missing half the labels it should contain. Any honest evaluation has to hold both axes in view at once, and then somehow still produce the single number the question demands. So the first job was just to measure each axis without letting it hide the other. Only once we could see both clearly did it make sense to talk about folding them into one score.

First moves: F1 over sets, read at two granularities

The natural starting point for scoring a set is precision, recall, and F1. For each document we compare the predicted set against a human-labeled gold set and count true positives, false positives, and false negatives.

The subtlety is in what counts as one unit. We compute F1 two ways, and the distance between them is itself the first diagnostic the evaluation gives us for free.

Label-only F1 treats each label as the unit and ignores rank entirely: did we even find the right labels? Pair-keyed F1 treats each (label, rank) pair as the unit: (A, primary) earns credit only if the truth says exactly (A, primary), with no partial credit for getting the label right and the rank wrong.

Pair-keyed F1 is the stricter, more honest headline, because in our setting what a label is and whether it's the dominant one aren't independent dimensions; downstream, they're consumed as a single unit. But reading the two side by side is what isolates the failure mode:

• pair-keyed much lower than label-only → the pipeline knows the labels but botches the ranking.
• both equally low → the pipeline is missing labels outright, a coverage problem rather than a ranking one.

We report each at macro and micro granularity. Macro averages the per-document F1, so every document counts equally. Micro pools all the TP/FP/FN counts first, so large, label-heavy documents dominate. Rare labels vanish inside micro averages, so macro plus a per-class table is what catches a pipeline that quietly fails on a minority class while looking fine overall.

This already answers a lot. But it doesn't yet answer the question, because F1 has a blind spot that no amount of slicing fixes.

Why F1 still isn't "the number": all errors are not equal

F1 treats every error as equally bad. A false positive is a false positive; a false negative is a false negative. Count them up, balance them, done.

But our errors are nowhere near equal in what they cost. Compare two mistakes:

• The pipeline misses a document's primary label entirely. The single most important thing about the document is simply gone, with no way for a downstream consumer to recover it.
• The pipeline adds one extra secondary label that doesn't belong. A reader glances past it. Mildly annoying.

F1 scores both as "one error." Worse, it can actively mislead: a run that nails every primary but over-lists a couple of cheap secondaries can post a lower F1 than a run that quietly drops a primary, which is exactly backwards from what we care about. A metric that ranks a careful run below a careless one isn't measuring quality; it's measuring something correlated with quality and calling it quality.

The fix isn't a better-tuned F1. It's to stop counting errors and start pricing them.

The unification: price every error by what it costs

The move that makes a single number trustworthy is to recognize that every mistake the pipeline can make falls into one of a small number of error classes, and that those classes can be ranked by how much harm they do.

The ranking axis is what makes this possible. Because each label carries a primary/secondary rank in both the truth and the prediction, we can tell a total loss apart from a recoverable one: a primary that got demoted to secondary is still somewhere in the list, so a consumer scanning the secondaries can still find it. That distinction is invisible to set-overlap metrics, and it's exactly the distinction that determines cost.

Every error sorts cleanly into one of six classes, and that completeness isn't luck. Each label, in any document, has three possible states in the truth, absent, primary, or secondary, and the same three in the prediction. That's a 3×3 grid of nine outcomes: the three on the diagonal are correct, absent stays absent, primary matches primary, secondary matches secondary, and the six off the diagonal are exactly the ways a single label can go wrong. The taxonomy isn't a list we brainstormed and hoped was exhaustive; it's every off-diagonal cell of that confusion matrix, so nothing can fall outside it. From most to least harmful:

1. Missing primary — a true primary appears nowhere in the prediction. The dominant signal is gone, with no recovery path. This is the catastrophe.
2. Primary demoted to secondary — the true primary was found, but tagged secondary. The rank is lost, but the label is still in the list and recoverable.
3. Spurious primary — a predicted primary isn't in the truth at all. It pollutes the most important view with a false signal.
4. Secondary promoted to primary — a true secondary was over-ranked to primary. Wrong, but close: the label is real, just over-weighted.
5. Missing secondary — a true secondary is absent. A minor signal lost; it never touches the primary read.
6. Spurious secondary — a predicted secondary isn't in the truth. A reader can simply ignore it.

That ordering is the judgment call, and it's the heart of the whole scheme. Everything about the primary tier (classes 1–3) outranks everything about the secondary tier, because the primary read is what decisions get made on. And within each tier, a missing signal hurts more than a misranked one, which hurts more than a spurious one. Encoding that ordering explicitly is what lets the final number reflect cost rather than count.

Making the price tiered, not just ordered

Once the six classes are ranked, each gets a weight, and a run's penalty is the weighted sum of how many errors of each class it made:

penalty = w1·c1 + w2·c2 + w3·c3 + w4·c4 + w5·c5 + w6·c6

The only real design decision left is the shape of those weights, and it's less a math choice than a business one. The weights are where we encode how much we actually care about each step in severity, so picking them is picking a policy about which mistakes we're willing to tolerate to avoid which others.

The obvious choice is linear weights: (6, 5, 4, 3, 2, 1). It respects the ordering and reads intuitively. But it says something about the business that isn't true: that the gap between every pair of adjacent severities is the same size. In our world it isn't: the jump from "spurious secondary a reader ignores" to "missing a core label entirely" is not five equal steps, it's a cliff. The linear scheme flattens that cliff into a gentle ramp. A run that drops a primary can bury that catastrophe under a tidy list of cheap mistakes and come out even with a run that did real damage. The metric stops reflecting what the business feels.

The fix is to make the weights grow faster than linearly as severity climbs, so that the gap between a primary-tier error and a secondary-tier one is itself large, not just one more equal step. Exactly how much faster is the real design work, and it isn't a math question. It's a business question: how many of a lesser error are you willing to accept to avoid one of the error directly above it? Two? Ten? Never any number at all?

Each answer encodes a different stance, and they're all legitimate depending on what the business actually feels. A gentle super-linear curve says "higher tiers hurt more, but a lot of small mistakes can still add up to something serious." A steep one says "a higher-tier error is in a different class entirely, and it takes a great many lesser ones to rival it." At the limit, a strict lexicographic ordering says "no quantity of lesser errors ever equals a single greater one." That's the absolute wall. Picking a point on that spectrum is the act of writing down, in numbers, how harmful each error is relative to the ones beneath it. That's not something the metric can decide for you; it's something the business decides and the metric records.

So the weight shape is the most concentrated expression of the whole post's thesis. The ordering of the error classes says which mistakes are worse; the steepness of the weights says by how much, and "by how much" is a judgment about your domain, your users, and what a wrong answer actually costs them. We calibrated ours to match how the people across the table feel about each mistake, validated that the weights stay strictly decreasing so they can't contradict the severity ordering, and moved on. The specific numbers matter far less than the principle: the curve is a business artifact, and it should be set by whoever owns the consequences, not defaulted into.

From penalty to a comparable 0–1 score

A raw penalty isn't comparable across documents: a document with five true labels has far more to get wrong than one with two. So we normalize each document's penalty against its own worst possible penalty:

penalty_max = (#true primaries · w1) + (#true secondaries · w4)
score       = max(0, 1 − penalty / penalty_max)

One honest caveat about that denominator: it's the worst you can do on the labels that should be there, not the worst you can do period. Spurious errors (classes 3 and 6) are unbounded, a pipeline can always invent more labels, so a document's raw penalty can in principle climb past its own penalty_max. That's what the max(0, …) is quietly absorbing: a prediction noisy enough to blow through the worst-case budget bottoms out at 0 rather than going negative. The trade-off is that gradation disappears at the floor, a document whose primary was perfect but buried under spurious junk can score the same 0 as one that missed the primary outright. We accept that because by the time a prediction is that polluted the exact depth of the hole stops mattering, but it's a deliberate choice, not a property the formula hands you for free.

That gives every document a clean scale: 0 = worst possible, 1 = perfect. How we aggregate those per-document scores is the same decision that's been running underneath everything. Our unit of value is the document, not the individual classification. What the business wants to maximize is the number of documents we get right, not the number of individual labels we get right across the corpus. A document with one true label and a document with seven are each worth exactly one correctly-classified item.

That choice dictates the aggregation directly. We normalize per document so that each one lands on the same 0–1 scale regardless of how many labels it carries; otherwise label-heavy documents would silently count for more, which is precisely backwards from "each document is worth one." And we headline the macro average, the mean of the per-document scores, because that treats every document equally, which is just the document-as-unit principle applied to averaging. We still compute micro (1 − Σpenalty / Σmax across the corpus) as a secondary read, but it answers a different question, "how many labels did we get right?", and that's deliberately not the question we optimize. A run passes when the macro score clears a threshold. That single percentage, "across our documents, how much of the worst-case harm did we avoid?", is the number we finally hand across the table.

Because the score is built from priced error classes rather than raw counts, the final number carries meaning the way F1 never could: it's denominated in harm avoided, so a run that protects the things that matter scores higher than one that's merely tidy on average. Two runs with identical F1 can land far apart here, and the gap is exactly the business-relevant difference F1 was blind to.

Closing

The arc is worth restating, because it's reusable well beyond our particular task. A set-valued, ranked output breaks ordinary accuracy, so you reach for F1 over sets, read at two granularities to separate a coverage failure from a ranking failure. But F1 prices every error the same, which inverts the ranking you actually care about, so you stop counting errors and start pricing them: a small taxonomy of error classes ordered by harm, weighted so each step up in severity counts for more than the one below, normalized per item into a clean 0–1 score.

An evaluation metric is a business decision wearing a mathematical costume.

"Is the new version better?" finally has an answer that means something, not because we found a cleverer average, but because we were explicit about which information deserved to survive the collapse into a single number, and priced the rest by what it actually costs.

That's the real lesson, and it outlasts any particular weight vector. Every choice we made (pair-keyed over label-only, six classes instead of one error count, a steep severity curve rather than a flat ramp, promotion as the worst case for a secondary) looks like a technical knob, but each one is really an answer to a business question about which mistakes we can live with and which we can't. The math doesn't decide those answers; it only records them faithfully once we've decided. When a metric and the business disagree about which version is better, it's almost never the business that's wrong; it's that the metric is quietly encoding a value nobody actually holds. The work isn't finding the objectively correct number. It's making sure the number you ship is denominated in the things you actually care about, so that when it goes up, something real got better.