feat(eval): add dataset-level evaluator framework with precision/recall/f-score

[ajay-kesavan]

Summary

Adds a dataset-level evaluator framework that consumes the per-datapoint results of an existing classification evaluator and emits one run-level metric (precision / recall / F-beta) with a structured ClassificationDetails payload (per-class TP/TN/FP/FN/support/value, full confusion matrix, micro/macro, skip counts).

The dataset evaluators' config is embedded in the per-datapoint evaluator's JSON via an aggregators[] array. Each aggregator entry is self-contained — its own classes, averaging, and (for fscore only) fValue — and there is no separate dataset-evaluator entity or external pointer.

Shape

{
  "id": "intent_classifier",
  "evaluatorTypeId": "uipath-multiclass-classification",
  "evaluatorConfig": {
    "name": "intent_classifier",
    "targetOutputKey": "intent",
    "classes": ["book", "cancel", "reschedule"],
    "aggregators": [
      { "type": "precision", "classes": ["book","cancel","reschedule"], "averaging": "macro" },
      { "type": "recall",    "classes": ["book","cancel","reschedule"], "averaging": "macro" },
      { "type": "fscore",    "classes": ["book","cancel","reschedule"], "averaging": "macro", "fValue": 1.0 }
    ]
  }
}

What's in the PR

• _aggregator_specs.py — discriminated AggregatorSpec union with type field. Three concrete variants: PrecisionAggregatorSpec, RecallAggregatorSpec, FScoreAggregatorSpec. Pydantic catches unknown type values at parse time.
• base_dataset_evaluator.py — BaseDatasetEvaluator abstract class, constructed from (spec, source_evaluator_name). No source_evaluator pointer in the spec itself.
• classification_dataset_evaluators.py — single ClassificationDatasetEvaluator class dispatches on spec.type to choose precision / recall / F-beta math. Confusion matrix oriented [predicted_idx][expected_idx] (documented as differing from sklearn's [true][predicted]).
• dataset_evaluator_factory.py — build_dataset_evaluator(spec, source_evaluator_name) returns the right instance for the spec's type.
• binary_classification_evaluator.py / multiclass_classification_evaluator.py — gained an aggregators: list[AggregatorSpec] | None field on their config. Other evaluator configs are unchanged.
• runtime.py — compute_dataset_evaluator_results() walks each evaluator's evaluator_config.aggregators, builds the dataset evaluators on demand, runs them against the per-datapoint results, and returns a dict keyed "{source_evaluator_name}.{aggregator_type}" (e.g. intent_classifier.precision).
• Tests — 488 evaluator tests pass, including math verification (2-class, 3-class, macro vs micro, F-beta), confusion matrix orientation, out-of-vocab skipping, malformed-details skipping, factory dispatch, JSON round-trip with fValue camelCase alias.
• Demo — examples/dataset_evaluators_demo.py runs the framework end-to-end and prints the resulting ClassificationDetails via model_dump_json(indent=2, by_alias=True).

What's NOT in the PR (consciously)

• No EvaluationSet.dataset_evaluator_refs — aggregators live on per-datapoint evaluator configs, not as a separate top-level list.
• No EvalHelpers.load_dataset_evaluators / UiPathEvalContext.dataset_evaluators — the SDK runtime discovers aggregators directly from the per-datapoint evaluator configs.
• No separate PrecisionDatasetEvaluator / RecallDatasetEvaluator / FScoreDatasetEvaluator classes — collapsed into one ClassificationDatasetEvaluator that switches on spec.type.
• No EvaluatorType.DATASET_PRECISION/RECALL/F_SCORE enum entries — the discriminator field on the spec is the dispatch key.

How to test

cd packages/uipath
uv run pytest tests/evaluators -x          # 488 pass
uv run ruff check . && uv run ruff format --check .
uv run mypy src tests
uv run python examples/dataset_evaluators_demo.py

Companion PRs

Repo	Branch	What
uipath-python	feat/classification-evaluator-types (#1663)	Sample projects + e2e tests wiring the framework to Binary/Multiclass classification samples
Agents	feat/eval-dataset-evaluators-backend (#5307)	Backend storage + python-eval-worker DatasetEvaluatorsWorkflow that consumes the embedded aggregator specs

[ajay-kesavan]

Superseded by #1674 (ClassifierEvaluator pure-metadata aggregator). The earlier dataset-level evaluator framework was replaced by the cleaner Classifier design that delegates aggregation math to the C# layer.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ad32c22c64

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
98.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[github-actions]

🚨 Heads up: uipath-langchain cross-tests are FAILING 🚨

Your changes may break the uipath-langchain-python integration.

⚠️ These checks are NOT enforced by branch protection rules. Please review the failures before merging.

🔍 Inspect the failed run →