[O2B-1564] Ccreate background GAQ summary worker#2175
Open
isaachilly wants to merge 54 commits into
Open
Conversation
Add GaqSummaryRepository and GaqSummaryInvalidationRepository and export them from the repositories index. Update migration to replace the previous invalidated_at field with created_at and add updated_at to mirror default Sequelize tables.
Rename and change mcReproducible to be the coverage float not the boolean.
Forgot to remove.
Add GAQ summary invalidation whenever underlying data affecting GAQ changes. These changes ensure GAQ summary caches/records are marked for recomputation whenever detectors, QC flags, or run QC times that influence GAQ summaries are modified.
Introduce background processing for GAQ summary invalidations and wire it into the app scheduler. Changes include:
- Add gaq config to services config.
- Move GaqService to lib/server/services/gaq and update imports across controllers/use-cases/tests.
- Extend GaqService to pop invalidations and recalculate summaries within a transaction.
- Add GaqWorker that guards concurrent runs and calls GaqService to process a batch of invalid summaries.
- Schedule the GaqWorker in application startup when GAQ recalculation is enabled.
- Add a soft-delete filter (where: { deleted: false }) when querying QC flags to map ids to run numbers as otherwise summary invalidations occur even on already deleted flags.
These changes enable periodic recalculation of GAQ summaries when invalidations are queued in the table.
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
Validate GAQ summary invalidation behaviour on QC flag create/verify/delete, deleteAllForDataPass, explicit/default GAQ detector changes, and run QC time updates.
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
Converted gaqWorker to a singleton. This allows adding of pause() and resume() methods to prevent the worker from processing invalidated summaries during test execution. Reduced the default GAQ recalculation period from 1 minute to 10 seconds to improve test suite performance.
Adds tests that verify the worker removes an invalidation and adds a summary, upserts for an already present summary, batch processes correctly, and doesn't run concurrent recalculations. Update resetDatabaseContent to pause/resume the GAQ worker to avoid worker failures when the invalidation table is dropped.
GaqService summary recalculation and invalidation processing functions tested directly circumventing worker. Tests verify correct computed fields, summary upsert, behaviour when no coverage exists, and batched invalidation processing.
Add a calculation_failed boolean column to gaq_summaries. This allows us to know whether a summary has been attempted to be calculated but unsuccessful due to limited data etc.
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
Remove not-null constraints from several GAQ summary columns in the migration to allow NULL when values are unavailable.
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
Stop returning early when summary is missing; always upsert a GaqSummary record and include a calculationFailed boolean. This ensures failed calculations are persisted for monitoring instead of being silently skipped.
Update test to expect that when there is no coverage data for a run, a GAQ summary is stored with calculationFailed set to true.
Store invalidation timestamp directly on gaq_summaries instead of a separate gaq_summary_invalidations table.
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
Replace usage of GaqSummaryInvalidationRepository with GaqSummaryRepository across services and tests. Upsert calls now set an invalidatedAt timestamp to mark GAQ summaries as invalidated.
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
Replace iterative invalidation pop with a single query that fetches up to batchSize invalidated GAQ summaries and processes them concurrently with a Promise.all.
Create an index on invalidated_at column of gaq_summaries table to improve query performance, as it is commonly used to fetch the invalidated summary queue.
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
…already soft deleted The simplest way of invalidating GAQ summaries affected by the deletion of a dataPass' Flags is to just take the list of deleted flags. But this is inefficient if some flags were already deleted and thus had already had their summaries at the time.
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
More details logged to help debugging the state of the GAQ worker.
…consistently exceeds batch size Also move any GAQ worker related logging to within the GAQ worker itself (separation of concerns).
Sequelize model omitted the primaryKey that migration file defines.
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
Extract invalidation into helpers. Guard GAQ updateRun reload so don't run reload unnecessarily. Add to notComputable field in model allowNull default params. Add more test coverage for invalidation on run patch.
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
Protects invalidatedAt from being set to null if the datapass and runNumber becomes invalid for whatever reason in the middle of computeSummary. ComputeSummary would overwrite the invalidatedAt column with null at the end. Now it performs a check to prevent this.
Add a helper that logs a warning when the env var is set to an invalid value instead of silently falling back to the default. If batchSize is not enough log the first warning after 5 consecutive overflow ticks, then every 30 ticks while it persists and recovery line when the streak ends so operators can grep for "backlog recovered". Log at debug level when the worker is paused or resumed.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #2175 +/- ##
==========================================
- Coverage 45.63% 42.39% -3.24%
==========================================
Files 1039 1044 +5
Lines 17324 17337 +13
Branches 3148 3169 +21
==========================================
- Hits 7905 7350 -555
- Misses 9419 9987 +568 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I have a JIRA ticket
Notable changes for users:
Notable changes for developers:
Changes made to the database:
A configurable background worker that runs a function that batch recalculates GAQ summaries that have an invalidatedAt column that is not null.
Dynamically adjusts batch size between a min and a max based on backlog.
Worker can be paused (useful for testing but maybe in prod for whatever reason).