Feat/dynamic dedup tcp#184
Open
praagyajain wants to merge 4 commits into
Open
Conversation
In k8s the dedup coverage collector runs in the k8s-proxy Deployment pod while
the app JVM is in a separate pod, so the shared-/tmp unix sockets can't reach it.
Add a TCP transport selected by KEPLOY_COVERAGE_ENDPOINT (host:port): the SDK
dials the collector and keeps one bidirectional connection for the whole replay.
Wire protocol (newline-delimited), matching the k8s-proxy collector:
collector -> SDK : "START <id>" | "END <id>"
SDK -> collector : "ACK" (after START reset)
"COV <compact-json>" + "ACK" (after END dump)
- CoverageTcpClient mirrors CommandServer's dispatch but inverts roles (SDK is the
client); reuses CoverageCollector reset/capture unchanged. COV precedes ACK so the
collector records the payload before the ACK releases the caller.
- Connect loop retries (collector listens only once replay begins). No read timeout
on the long-lived idle-between-tests connection.
- Unix transport stays the default for local/docker (unchanged); the worker field is
generalized to Closeable so stop() is transport-agnostic.
Validated end-to-end: real JVM (this SDK as -javaagent, TCP client) against the
k8s-proxy DedupCoverage TCP server — coverage flows correctly; the score>=90 pair
dedupes (identical line sets), the score=40 case differs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A fresh JVM runs each class's static initializer (<clinit>) on first use. JaCoCo charged those one-time lines to whichever test ran first, so a test could look different from its true duplicates by luck of timing -> the duplicate set flip-flopped run to run (e.g. 2 vs 4). On the first START command (app fully started), eagerly Class.forName(..., initialize=true) every indexed application class so all <clinit> lines run once, then the normal reset clears them. Every test window then captures only the lines its own request executes -> deterministic dedup. Best-effort per class; disable via KEPLOY_JAVA_DEDUP_WARMUP_DISABLED=true. Go is unaffected (AOT, init() runs once at startup before the first reset). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fingerprint each test by the set of executed JaCoCo probes per class
({className -> [probeIdx]}) instead of executed source lines. Each branch
is instrumented as a distinct probe, so the probe set distinguishes which
branch a test took (true vs false) on a shared line — line status and even
branch counts report identically for both paths. The probe set subsumes
line coverage and uses canonical VM class-name keys (no path normalization).
Only capture() changes; the wire payload, collector, store, and enterprise
compute stay generic over map[string][]int. Removed the now-dead line-decode
helpers (Analyzer/CoverageBuilder, executedLines, resolveSourcePath).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR updates the Java dynamic-dedup agent to support a TCP transport mode (intended for k8s / non-shared-/tmp setups) and changes how coverage fingerprints are produced/sent during replay.
Changes:
- Add dynamic transport selection: unix-socket
CommandServer(local/docker) vs TCP clientCoverageTcpClient(k8s) based onKEPLOY_COVERAGE_ENDPOINT. - Add one-time “warmup” of indexed application classes on first
STARTto reduce<clinit>noise in the first test window. - Change coverage capture to emit per-class executed JaCoCo probe indices (instead of source-file executed line numbers).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+713
to
722
| boolean[] probes = executionData.getProbes(); | ||
| Set<Integer> fired = new LinkedHashSet<>(); | ||
| for (int i = 0; i < probes.length; i++) { | ||
| if (probes[i]) { | ||
| fired.add(i); | ||
| } | ||
| } | ||
| if (!fired.isEmpty()) { | ||
| raw.put(executionData.getName(), fired); | ||
| } |
Comment on lines
+516
to
+524
| Map<String, List<Integer>> executedLinesByFile = collector.capture(); | ||
| if (executedLinesByFile.isEmpty()) { | ||
| log(Level.FINE, "No Java coverage lines collected for " + command.testId, null); | ||
| } else { | ||
| // COV must precede ACK: the collector reads lines sequentially, | ||
| // so the payload is recorded before the ACK releases the caller. | ||
| writeLine(out, "COV " + GSON.toJson( | ||
| new DedupPayload(command.testId, executedLinesByFile))); | ||
| } |
Comment on lines
+449
to
+452
| if (running.get()) { | ||
| log(Level.INFO, "Keploy dedup: TCP connect to " + host + ":" + port | ||
| + " failed (" + e.getClass().getSimpleName() + ": " + e.getMessage() + "), retrying", null); | ||
| } |
- TCP END always emits COV before ACK (even when empty), mirroring the unix
transport, so the line-oriented collector reads exactly one COV per END and
cannot desync (Copilot comment).
- Reconnect failures log at INFO only on the first failure, then FINE, to
avoid ~1/s log spam in k8s; reset on a successful connect (Copilot comment).
- smoke-javaagent.sh asserts the VM class-name key ("smoke/Work") instead of
the old source-file key ("Work.java"), matching the probe-based fingerprint
contract; fixes the JDK 8/17/21 smoke jobs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issue
Closes: #[issue number that will be closed through this PR]
Describe the changes you've made
A clear and concise description of what you have done to successfully close your assigned issue. Any new files? or anything you feel to let us know!
Type of change
How did you test your code changes?
Please describe the tests(if any). Provide instructions how its affecting the coverage.
Describe if there is any unusual behaviour of your code(Write
NAif there isn't)A clear and concise description of it.
Checklist:
Screenshots (if any)