maestro: add JSON validation reports and --fail-on-mismatch flag#275
Merged
Conversation
Add machine-readable JSON output to kci-dev maestro validate builds and boots so automation can verify Maestro and Dashboard consistency without parsing human-readable CLI output. This supports kernelci/kernelci-pipeline#1125, Validate maestro build results consistency. KernelCI users need stable build and boot counts with no silent result loss, missing records, or mismapped statuses on the dashboard. The structured summary lets daily validation jobs compare Dashboard API data,Maestro API data, and recent checkout expectations, then decide policy from summary.ok. The JSON mode emits a single JSON object on stdout, redirects incidental progress output away from stdout, and exits 0 for successful validation runs even when mismatches are reported. Runtime/API errors still exit nonzero. An optional --fail-on-mismatch flag is also added for workflows that want the command itself to fail when validation finds inconsistencies. Tests cover JSON output for builds, boots, and build history mismatch cases; stdout cleanliness; missing ID and status mismatch summaries; mismatch exit behavior; and runtime failure handling. Signed-off-by: Denys Fedoryshchenko <denys.f@collabora.com>
nuclearcat
force-pushed
the
maestro-validate-json
branch
from
065e67f to
18768a6
Compare
Member
|
please fix tox errors |
Member
Author
|
ci error is not relevant to the PR, kcidev/libs/dashboard.py retry wrapper calls time.sleep(...) but does not import time. |
Member
|
thanks |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add machine-readable JSON output to kci-dev maestro validate builds and boots so automation can verify Maestro and Dashboard consistency without parsing human-readable CLI output.
This supports kernelci/kernelci-pipeline#1125, Validate maestro build results consistency. KernelCI users need stable build and boot counts with no silent result loss, missing records, or mismapped statuses on the dashboard. The structured summary lets daily validation jobs compare Dashboard API data,Maestro API data, and recent checkout expectations, then decide policy from summary.ok.
The JSON mode emits a single JSON object on stdout, redirects incidental progress output away from stdout, and exits 0 for successful validation runs even when mismatches are reported.
Runtime/API errors still exit nonzero. An optional --fail-on-mismatch flag is also added for workflows that want the command itself to fail when validation finds inconsistencies.
Tests cover JSON output for builds, boots, and build history mismatch cases; stdout cleanliness; missing ID and status mismatch summaries; mismatch exit behavior; and runtime failure handling.