Ordinal Metrics Node

O

AI/ML/Ordinal

Evaluate predictions for an ordered target with distance-aware metrics. Plain accuracy is inadequate here: it treats "predicted high when the truth was medium" exactly as harshly as "predicted low", so a model that is reliably one level off scores like one that guesses. Quadratic weighted kappa is the standard headline metric because it weights every miss by how far off it was and corrects for chance agreement, but it answers only one of three questions: the linear kappa and the macro-averaged error say how far off the model is under a different cost structure and on the rare levels, while Kendall's tau-b and the Spearman correlation say whether it orders the rows correctly at all.

ml_ordinal_metricsml

Inputs5

Outputs12

Security exposure8/10

Packageml

Ratings

Scores range from 0 to 10. Higher values mean more impact, exposure, or operational weight.

SecurityAttack surface and exposure impact.

8/10High

PrivacyPotential sensitivity of processed data.

8/10High

PerformanceRuntime or resource pressure.

7/10High

GovernancePolicy, audit, or compliance impact.

8/10High

ReliabilityOperational stability considerations.

9/10High

CostExternal or compute cost impact.

9/10High

Input Pins

5

Input

Execution

exec_in

Execution trigger that begins the ordinal evaluation

Database

Struct

database

Database connection containing the predicted levels and the true levels

SchemaNodeDBConnectionNodeDBConnectionNodeDBConnection1 fields

cache_keystringrequired

Schema enforced

Predictions Column

String

predictions_col

Column holding the predicted level of each row. The labels must be the same ones the actuals column uses, since both columns are ranked against one shared level order.

Default prediction

Actuals Column

String

actuals_col

Column holding the true level of each row. When no Class Order is given, the level order is inferred from this column, and a predicted level that never occurs here is an error rather than a silent extra rank.

Default target

Class Order

String

class_order

Comma-separated level labels from LOWEST to HIGHEST, e.g. `low, medium, high`. Leave empty when the levels are numeric and their numeric order is the order you want. Non-numeric labels have no inferable order (sorting them alphabetically would rank high < low < medium), so they have to be listed here.

Output Pins

12

Done

Execution

exec_out

Activated once the ordinal evaluation completes

Quadratic Weighted Kappa

Float

quadratic_weighted_kappa

Headline ordinal metric: chance-corrected agreement weighted by the squared level distance. 1.0 perfect, 0.0 chance, negative worse than chance.

Linear Weighted Kappa

Float

linear_weighted_kappa

The same chance-corrected agreement with every level of distance costing the same. Read this one instead of the quadratic kappa when a level is a level — grading scales, severity tiers, anything where two steps off is exactly twice as bad as one. Quadratic weighting charges a near miss only a quarter of a two-level miss, so it flatters a model that merely hovers next to the truth; where that discount is not real, this is the honest number and it will be the lower of the two.

Mean Absolute Rank Error

Float

mean_absolute_rank_error

Average miss in levels. 0.0 is perfect, 1.0 means being off by one level on average.

Macro Mean Absolute Error

Float

macro_mean_absolute_error

The mean absolute rank error computed per true level and averaged with one vote per level. Look here whenever the levels are imbalanced: the plain error averages over rows, so the majority level speaks for the model and a predictor that collapses onto it still scores well while missing every rare level. This metric gives the rare levels equal weight, so it is the one that moves when that happens. Levels absent from the actuals are skipped rather than counted as perfect.

Exact Accuracy

Float

accuracy_exact

Share of predictions hitting the exact level. Reported for reference; it ignores how far the misses are off.

Accuracy Within One

Float

accuracy_within_one

Share of predictions landing on the true level or one of its direct neighbours

Kendall Tau-b

Float

kendall_tau_b

Tie-corrected rank association: +1.0 orders the rows exactly as the truth does, 0.0 no association, -1.0 exactly backwards. This answers "does the model rank the rows correctly", which is a different question from "does it land on the right level" — a model whose every prediction is one level too high ranks perfectly and scores 1.0 here while the kappas drop. Consult it when the output feeds a sort, a triage queue or a threshold you can recalibrate, and read it against kappa to tell a miscalibrated model from a model that has learned nothing.

Spearman Rank Correlation

Float

spearman_rank_correlation

The same ordering question as tau-b, computed as a correlation on midranks. It is the less conservative of the two under the heavy ties ordinal data always has, so it reads higher than tau-b on the same predictions; prefer tau-b when you need a defensible figure and this one when comparing against Spearman values reported elsewhere. Like tau-b it ignores calibration entirely.

Samples

Integer

n_samples

Number of rows evaluated

Levels

Integer

n_levels

Number of distinct levels both columns were ranked against

Result

Struct

result

All ordinal metrics plus the resolved level order they were computed against

SchemaOrdinalMetricsResultOrdinalMetricsResultOrdinalMetricsResult12 fields

quadratic_weighted_kappanumber:doublerequired

Quadratic weighted kappa: agreement corrected for chance, with the penalty growing with the square of the level distance. 1.0 is perfect, 0.0 is chance, negative is worse than chance.

format double

linear_weighted_kappanumber:doublerequired

Linear weighted kappa: the same chance-corrected agreement with every level of distance costing the same. Lower than the quadratic figure whenever the misses are near misses.

format double

mean_absolute_rank_errornumber:doublerequired

Average distance between the predicted and the true level, measured in levels

format double

macro_mean_absolute_errornumber:doublerequired

Mean absolute rank error computed per true level and averaged with one vote per level, so the majority level cannot hide what the model does on the rare ones.

format double

accuracy_exactnumber:doublerequired

Share of predictions hitting the exact level (plain accuracy, for reference only)

format double

accuracy_within_onenumber:doublerequired

Share of predictions landing on the true level or one of its neighbours

format double

kendall_tau_bnumber:doublerequired

Kendall's tau-b: tie-corrected rank association. +1.0 orders the rows exactly as the truth does, 0.0 no association, -1.0 exactly backwards. Ignores calibration.

format double

spearman_rank_correlationnumber:doublerequired

Spearman rank correlation on midranks: the same ordering question as tau-b on a different scale, and less conservative under heavy ties.

format double

n_samplesinteger:uintrequired

Number of rows evaluated

format uintmin 0

n_levelsinteger:uintrequired

Number of distinct levels both columns were ranked against

format uintmin 0

levelsArray<string>required

The level labels from lowest to highest, in the rank order the metrics used

itemsstringarray item

orderingOrdinalOrderingrequired

Whether that order came from the Class Order pin or from parsing the levels as numbers

variant 1constvariant

Every distinct label parsed as a number, so the order is the numeric one.

const "Numeric"

variant 2constvariant

The caller supplied the order explicitly.

const "Explicit"

Node Info

Internal name

ml_ordinal_metrics

Category

AI/ML/Ordinal

Version

2