diff --git a/.claude/skills/pr-description/SKILL.md b/.claude/skills/pr-description/SKILL.md
index ec57684287a..bd7a072f40e 100644
--- a/.claude/skills/pr-description/SKILL.md
+++ b/.claude/skills/pr-description/SKILL.md
@@ -13,7 +13,7 @@ Generate a pull request title and description for the current branch using the p
1. Determine the base branch:
- Use the argument if provided
- Otherwise, auto-detect: `git remote set-head origin --auto >/dev/null 2>&1 && git symbolic-ref refs/remotes/origin/HEAD 2>/dev/null | sed 's|refs/remotes/||'`
- - Fall back to `v3.1-dev` if the command above fails
+ - Fall back to `v4.0-dev` if the command above fails
2. Gather context by running these git commands:
- `git log --oneline $(git merge-base HEAD )..HEAD` — all commits on this branch
diff --git a/NIGHTLY_STATUS.md b/NIGHTLY_STATUS.md
index 43efc08683f..52b29bd2b36 100644
--- a/NIGHTLY_STATUS.md
+++ b/NIGHTLY_STATUS.md
@@ -2,9 +2,9 @@
[](https://github.com/dashpay/platform/actions/workflows/tests.yml?query=event%3Aschedule)
-> **Note:** This page is manually maintained. For live results, check the [latest nightly run](https://github.com/dashpay/platform/actions/workflows/tests.yml?query=event%3Aschedule+branch%3Av3.1-dev) directly.
+> **Note:** This page is manually maintained. For live results, check the [latest nightly run](https://github.com/dashpay/platform/actions/workflows/tests.yml?query=event%3Aschedule+branch%3Av4.0-dev) directly.
-Nightly tests run every day at **23:00 UTC** on the `v3.1-dev` branch via the [Tests workflow](https://github.com/dashpay/platform/actions/workflows/tests.yml?query=event%3Aschedule). They exercise the full CI pipeline including Docker image builds, E2E tests, and the platform test suite.
+Nightly tests run every day at **23:00 UTC** on the `v4.0-dev` branch via the [Tests workflow](https://github.com/dashpay/platform/actions/workflows/tests.yml?query=event%3Aschedule). They exercise the full CI pipeline including Docker image builds, E2E tests, and the platform test suite.
## Nightly Jobs
@@ -52,7 +52,7 @@ The functional tests have been intermittently failing for months. This is a know
## Links
- [All nightly runs](https://github.com/dashpay/platform/actions/workflows/tests.yml?query=event%3Aschedule)
-- [Latest nightly run](https://github.com/dashpay/platform/actions/workflows/tests.yml?query=event%3Aschedule+branch%3Av3.1-dev)
+- [Latest nightly run](https://github.com/dashpay/platform/actions/workflows/tests.yml?query=event%3Aschedule+branch%3Av4.0-dev)
- [Long-running Rust nightly](https://github.com/dashpay/platform/actions/workflows/tests-rs-nightly-long-running.yml)
- [Security audits (Rust)](https://github.com/dashpay/platform/actions/workflows/security-audit-rust.yml)
- [Security audits (JS - npm)](https://github.com/dashpay/platform/actions/workflows/security-audit-js-npm.yml)
diff --git a/README.md b/README.md
index 0335eceed4f..6663cf2bbae 100644
--- a/README.md
+++ b/README.md
@@ -11,8 +11,8 @@
- 
- 
+
+ 
@@ -24,10 +24,10 @@
| Crate | Lines | Coverage |
|-------|------:|----------|
-| [rs-dpp](./packages/rs-dpp) | 129k | [](https://codecov.io/gh/dashpay/platform/component/dpp) |
-| [rs-drive](./packages/rs-drive) | 171k | [](https://codecov.io/gh/dashpay/platform/component/drive) |
-| [rs-drive-abci](./packages/rs-drive-abci) | 125k | [](https://codecov.io/gh/dashpay/platform/component/drive-abci) |
-| [rs-sdk](./packages/rs-sdk) | 23k | [](https://codecov.io/gh/dashpay/platform/component/sdk) |
+| [rs-dpp](./packages/rs-dpp) | 129k | [](https://codecov.io/gh/dashpay/platform/component/dpp) |
+| [rs-drive](./packages/rs-drive) | 171k | [](https://codecov.io/gh/dashpay/platform/component/drive) |
+| [rs-drive-abci](./packages/rs-drive-abci) | 125k | [](https://codecov.io/gh/dashpay/platform/component/drive-abci) |
+| [rs-sdk](./packages/rs-sdk) | 23k | [](https://codecov.io/gh/dashpay/platform/component/sdk) |
diff --git a/book/src/drive/average-index-examples.md b/book/src/drive/average-index-examples.md
index 997ab58b8ab..c707b9ddb84 100644
--- a/book/src/drive/average-index-examples.md
+++ b/book/src/drive/average-index-examples.md
@@ -1,10 +1,10 @@
# Average Index Examples
-This chapter walks through a representative contract and shows how **average queries** work on Drive. Every example uses the same `grade` document type on the **grades contract** at [`packages/rs-drive/tests/supporting_files/contract/grades/grades-contract.json`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/tests/supporting_files/contract/grades/grades-contract.json).
+This chapter walks through a representative contract and shows how **average queries** work on Drive. Every example uses the same `grade` document type on the **grades contract** at [`packages/rs-drive/tests/supporting_files/contract/grades/grades-contract.json`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/tests/supporting_files/contract/grades/grades-contract.json).
The chapter assumes you've read [Document Count Trees](./document-count-trees.md) and [Document Sum Trees](./document-sum-trees.md) — averages are built directly on top of both, so understanding count + sum trees individually is the prerequisite. Here we take that machinery as given and look at the queries that need *both*.
-> **Status:** the [`document_average_worst_case`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/benches/document_average_worst_case.rs) bench lands the reproducible numbers below — same convention as the [Count](./count-index-examples.md) and [Sum](./sum-index-examples.md) chapters. All proof sizes are measured against a 31 620-grade fixture; verified `(count, sum)` values are the actual numbers the bench's matrix reports. The full surface — primary-key global average, point lookups, range aggregates, and both carrier variants — is wired through to grovedb PR #670's verifiers end-to-end.
+> **Status:** the [`document_average_worst_case`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/benches/document_average_worst_case.rs) bench lands the reproducible numbers below — same convention as the [Count](./count-index-examples.md) and [Sum](./sum-index-examples.md) chapters. All proof sizes are measured against a 31 620-grade fixture; verified `(count, sum)` values are the actual numbers the bench's matrix reports. The full surface — primary-key global average, point lookups, range aggregates, and both carrier variants — is wired through to grovedb PR #670's verifiers end-to-end.
## Why Averages Need a New Primitive
@@ -70,11 +70,11 @@ Five things to internalize before reading the queries:
2. **`byClass` / `byStudent` / `bySemester` are `countable: countable` + `summable: "score"`** (no range flags). Each per-key value-tree (one per class / student / semester) is a **`CountSumTree`** carrying both metrics at one merk lookup — point-lookup averages get the same shortcut count proofs and sum proofs do.
3. **`byClassSemester` and `byStudentSemester` set both range flags** (`rangeCountable: true` AND `rangeSummable: true`). The `semester` continuation under each (class | student) value-tree is a **`ProvableCountProvableSumTree`** (PCPS), the structurally-richest tree variant — every internal merk node carries both a per-node count *and* a per-node sum. This is what `AggregateCountAndSumOnRange` walks for "average for class X in semester range [a..b]" style queries.
4. **Every `summable` index here is also `countable`.** There's no `summable`-only index in this contract — averages need both axes, so a sum-only index would be unreachable from the average surface. (Pure-sum surfaces are covered by the [tip-jar contract](./sum-index-examples.md) in the previous chapter; the grades contract is deliberately the dual-axis counterpart.)
-5. **`countableAllowingOffset` on the PCPS indexes** — `rangeCountable: true` requires the count tier to be `countable` or `countableAllowingOffset` (per the rule documented in [`Index::range_countable`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-dpp/src/data_contract/document_type/index/mod.rs)). The offset-allowing tier upgrades the property-name tree to a `ProvableCountTree` at minimum; combined with `summable: "score"` and `rangeSummable: true` the dispatcher resolves it to PCPS.
+5. **`countableAllowingOffset` on the PCPS indexes** — `rangeCountable: true` requires the count tier to be `countable` or `countableAllowingOffset` (per the rule documented in [`Index::range_countable`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-dpp/src/data_contract/document_type/index/mod.rs)). The offset-allowing tier upgrades the property-name tree to a `ProvableCountTree` at minimum; combined with `summable: "score"` and `rangeSummable: true` the dispatcher resolves it to PCPS.
The bench populates **50 000 grades** under a deterministic, realistic-data-shaped schedule: 500 students × 10 classes × 10 semesters = 50 000 grade documents. The score model layers three deterministic axes:
-- **Per-class baseline + spread** (see [`class_profile` in the bench](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/benches/document_average_worst_case.rs)) — hard classes get low means and wide spreads; easy classes cluster near 85–90 with narrow spreads. The 10 classes have semantic names matching the chapter's references:
+- **Per-class baseline + spread** (see [`class_profile` in the bench](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/benches/document_average_worst_case.rs)) — hard classes get low means and wide spreads; easy classes cluster near 85–90 with narrow spreads. The 10 classes have semantic names matching the chapter's references:
| Class | Baseline mean | Spread | Profile |
|---|---|---|---|
@@ -94,7 +94,7 @@ The bench populates **50 000 grades** under a deterministic, realistic-data-shap
A skill score is *amplified by class spread* — `skill × spread / 8` — so a +10-skill student in PHYS101 (spread=12) gains +15 over baseline, while the same student in ARTS101 (spread=4) only gains +5. This produces the realistic spread real transcripts exhibit: a strong student stands out more in hard classes, struggling students fall further behind in hard classes, easy classes flatten the curve.
-**Realistic enrollment.** Not every student takes every class — that's not how transcripts work. The bench walks all 500 × 10 × 10 = 50 000 possible `(student, class, semester)` triples but only emits a grade when [`is_enrolled`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/benches/document_average_worst_case.rs) returns true, using a deterministic per-class popularity table:
+**Realistic enrollment.** Not every student takes every class — that's not how transcripts work. The bench walks all 500 × 10 × 10 = 50 000 possible `(student, class, semester)` triples but only emits a grade when [`is_enrolled`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/benches/document_average_worst_case.rs) returns true, using a deterministic per-class popularity table:
| Class | Popularity | Profile |
|---|---|---|
@@ -1699,7 +1699,7 @@ The split closely parallels the count and sum chapters — point lookups for Q1
## What's Next
-The chapter is grounded in the [`document_average_worst_case`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/benches/document_average_worst_case.rs) bench's measured numbers — Q1–Q7 verify cleanly end-to-end against the shared root hash `8b15f732…ffc7`.
+The chapter is grounded in the [`document_average_worst_case`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/benches/document_average_worst_case.rs) bench's measured numbers — Q1–Q7 verify cleanly end-to-end against the shared root hash `8b15f732…ffc7`.
A natural expansion follow-up (out of scope here): a worked example of "exact-precision" averages — for callers that need fractional averages (e.g. `avg = 50.7142857…` rather than `50.99`), the protocol-level approach is to return `(count, sum)` and let the client compute in its preferred numeric format (the chapter notes this in [Numerical Considerations](#numerical-considerations) above; a future expansion could walk through the fixed-point vs. floating-point trade-offs).
diff --git a/book/src/drive/count-index-examples.md b/book/src/drive/count-index-examples.md
index 2cf1de6145e..c1d086638ee 100644
--- a/book/src/drive/count-index-examples.md
+++ b/book/src/drive/count-index-examples.md
@@ -1,6 +1,6 @@
# Count Index Examples
-This chapter walks through a representative contract and shows what a count-query proof actually proves — both the path query the prover signs and the verified element the verifier extracts. Every example uses the same `widget` contract (the same one the count-query bench at [`packages/rs-drive/benches/document_count_worst_case.rs`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/benches/document_count_worst_case.rs) populates) so the proof bytes, verified elements, and diagrams can all be cross-referenced against the same data.
+This chapter walks through a representative contract and shows what a count-query proof actually proves — both the path query the prover signs and the verified element the verifier extracts. Every example uses the same `widget` contract (the same one the count-query bench at [`packages/rs-drive/benches/document_count_worst_case.rs`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/benches/document_count_worst_case.rs) populates) so the proof bytes, verified elements, and diagrams can all be cross-referenced against the same data.
The chapter assumes you've read [Document Count Trees](./document-count-trees.md) — that chapter explains the three tree variants (`NormalTree` / `CountTree` / `ProvableCountTree`), what `Element::NonCounted` does, and how the schema's `documentsCountable` / `rangeCountable` flags select between them. Here we take that machinery as given and trace what each query *sees*.
@@ -44,7 +44,7 @@ The widget document type carries three properties (`brand`, `color`, `serial`),
Three things to notice:
1. **`documentsCountable: true`** at the document-type level upgrades the doctype's primary-key subtree (at `widget/[0]`) from `NormalTree` to `CountTree`. The unfiltered total count is one read against this element's `count_value`.
-2. **`byBrand` is `countable: "countable"` only.** It doesn't opt into `rangeCountable`, so `brand > X` range counts aren't supported. But **every countable terminator's value tree is stored as a `CountTree`** regardless of `rangeCountable` (see [`add_indices_for_index_level_for_contract_operations/v0/mod.rs`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/drive/document/insert/add_indices_for_index_level_for_contract_operations/v0/mod.rs)), so point-lookup count proofs (e.g. `brand == "X"` or `brand IN [...]`) get the same compact value-tree-direct shape on byBrand that they do on rangeCountable indexes. `rangeCountable` is strictly an opt-in for `AggregateCountOnRange` support — orthogonal to proof-size shape.
+2. **`byBrand` is `countable: "countable"` only.** It doesn't opt into `rangeCountable`, so `brand > X` range counts aren't supported. But **every countable terminator's value tree is stored as a `CountTree`** regardless of `rangeCountable` (see [`add_indices_for_index_level_for_contract_operations/v0/mod.rs`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/drive/document/insert/add_indices_for_index_level_for_contract_operations/v0/mod.rs)), so point-lookup count proofs (e.g. `brand == "X"` or `brand IN [...]`) get the same compact value-tree-direct shape on byBrand that they do on rangeCountable indexes. `rangeCountable` is strictly an opt-in for `AggregateCountOnRange` support — orthogonal to proof-size shape.
3. **`byColor` and `byBrandColor` are `rangeCountable: true`.** Their property-name subtrees (e.g. `widget/color`) are stored as `ProvableCountTree` rather than `NormalTree`, which is what `AggregateCountOnRange` walks for `color > floor` style queries.
The bench populates 100 000 documents under a deterministic schedule — `row → (brand_(row % 100), color_(row / 100), serial=row)`. That gives exactly 1 000 docs per brand, exactly 100 docs per color, and exactly 1 doc per `(brand, color)` pair. Those numbers show up in every verified count below.
@@ -86,7 +86,7 @@ flowchart TB
Three layout facts to internalize before reading the queries:
-- **`brand_050` is a `CountTree` with `count_value = 1000`.** That's true *because* `byBrand` is countable; the rule applies uniformly to every countability tier (see [`add_indices_for_index_level_for_contract_operations/v0/mod.rs`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/drive/document/insert/add_indices_for_index_level_for_contract_operations/v0/mod.rs)). The `color` continuation that branches off this value tree is `NonCounted`-wrapped so the parent's count equals exactly the 1 000 refs in `[0]`.
+- **`brand_050` is a `CountTree` with `count_value = 1000`.** That's true *because* `byBrand` is countable; the rule applies uniformly to every countability tier (see [`add_indices_for_index_level_for_contract_operations/v0/mod.rs`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/drive/document/insert/add_indices_for_index_level_for_contract_operations/v0/mod.rs)). The `color` continuation that branches off this value tree is `NonCounted`-wrapped so the parent's count equals exactly the 1 000 refs in `[0]`.
- **`widget/color` is a `ProvableCountTree`**, not a regular `NormalTree`. The yellow class above marks that — each internal merk node carries its subtree's count, which is what makes `AggregateCountOnRange` a single-pass primitive.
- **`color_00000500` is a `CountTree` with `count_value = 100`** under either parent. The same element layout would result from a query against `byColor` or against `byBrandColor`'s second level; the path that gets there differs, but the destination is structurally the same.
@@ -99,7 +99,7 @@ Every example below has four sections:
3. **Proof display** — the proof bytes, decoded via `bincode` into the structured `GroveDBProof` AST and rendered through its `Display` impl. This is the same view [dash-evo-tool's Proof Log screen](https://github.com/dashpay/dash-evo-tool/blob/master/src/ui/tools/proof_log_screen.rs) shows when its display mode is set to "JSON" — each layer is a separate `LayerProof` carrying its merk-tree operations (`Push` / `Parent` / `Child` over `Hash` / `KVValueHash` / `KVHash`) plus a `lower_layers` map naming the children to descend into. Wrapped in a collapsible block per example because the merk path through 4-5 grovedb layers makes for long output.
4. **Diagram** — the path the proof walks through the layout. Blue arrows trace the descent; the cyan node is the verified element; faded gray nodes show context.
-All proof-size numbers come from running the bench against a 100 000-row fixture; see [`document_count_worst_case.rs`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/benches/document_count_worst_case.rs)'s `report_proof_sizes` / `display_proofs` / `report_group_by_matrix` helpers. The proof bytes are reproducible — run the bench, grep `[proof]` from stderr, and you'll get the same hashes shown here.
+All proof-size numbers come from running the bench against a 100 000-row fixture; see [`document_count_worst_case.rs`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/benches/document_count_worst_case.rs)'s `report_proof_sizes` / `display_proofs` / `report_group_by_matrix` helpers. The proof bytes are reproducible — run the bench, grep `[proof]` from stderr, and you'll get the same hashes shown here.
## Queries in this Chapter
@@ -1863,4 +1863,4 @@ Four takeaways:
- **Queries 7 and 8 use a fundamentally different verifier** (`verify_aggregate_count_query` vs `verify_query`). Queries 1–6 return an element list and read `count_value_or_default` per branch; Queries 7 and 8 return a pre-summed `u64`. Query 8 is just Query 7 one grove layer deeper — same primitive, applied to byBrandColor's terminator rather than byColor's.
- **Query 8 is the most expensive.** It pays for both the compound descent (Query 4's 4-extra-layer cost) *and* the range-aggregate boundary (Query 7's primitive). The verified count is far smaller than Query 7 (499 vs 49 900), but the proof bytes are 28 % larger because the merk-tree boundary at L8 has roughly the same shape regardless of how many keys the cut spans.
-The path-query builder these examples decode lives at [`packages/rs-drive/src/query/drive_document_count_query/path_query.rs`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/query/drive_document_count_query/path_query.rs); the verifier mirror sits in [`packages/rs-drive/src/verify/document_count/`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/verify/document_count/). Both the prover and the verifier reconstruct the exact same `PathQuery` via the shared builder — touching one without the other is a Merkle-root mismatch waiting to happen, and the byte-identical contract is what makes the proof bytes here reproducible against the bench fixture.
+The path-query builder these examples decode lives at [`packages/rs-drive/src/query/drive_document_count_query/path_query.rs`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/query/drive_document_count_query/path_query.rs); the verifier mirror sits in [`packages/rs-drive/src/verify/document_count/`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/verify/document_count/). Both the prover and the verifier reconstruct the exact same `PathQuery` via the shared builder — touching one without the other is a Merkle-root mismatch waiting to happen, and the byte-identical contract is what makes the proof bytes here reproducible against the bench fixture.
diff --git a/book/src/drive/count-index-group-by-examples.md b/book/src/drive/count-index-group-by-examples.md
index 29d16c80578..4d820775790 100644
--- a/book/src/drive/count-index-group-by-examples.md
+++ b/book/src/drive/count-index-group-by-examples.md
@@ -1,6 +1,6 @@
# Count Index Group By Examples
-This chapter is the `GROUP BY` companion to [Count Index Examples](./count-index-examples.md). It uses the same `widget` contract, the same 100 000-row fixture, and the same bench at [`packages/rs-drive/benches/document_count_worst_case.rs`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/benches/document_count_worst_case.rs). Read chapter 29 first — most of the mechanics (CountTree variants, the merk-proof reconstruction algorithm, `node_hash_with_count` and friends) carry over unchanged.
+This chapter is the `GROUP BY` companion to [Count Index Examples](./count-index-examples.md). It uses the same `widget` contract, the same 100 000-row fixture, and the same bench at [`packages/rs-drive/benches/document_count_worst_case.rs`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/benches/document_count_worst_case.rs). Read chapter 29 first — most of the mechanics (CountTree variants, the merk-proof reconstruction algorithm, `node_hash_with_count` and friends) carry over unchanged.
What's different here:
@@ -35,7 +35,7 @@ Range queries are different. `AggregateCountOnRange` (chapter 29's Q7) walks the
## Queries in this Chapter
-All proof-size and behaviour numbers below come from the same bench helper (`report_group_by_matrix`) as chapter 29's. The dispatcher's group_by surface validation lives in [`validate_count_query_groupby_against_index`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/query/drive_document_count_query/validate.rs); the per-mode path-query builders sit in [`packages/rs-drive/src/query/drive_document_count_query/path_query.rs`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/query/drive_document_count_query/path_query.rs)'s `group_by_*` family.
+All proof-size and behaviour numbers below come from the same bench helper (`report_group_by_matrix`) as chapter 29's. The dispatcher's group_by surface validation lives in [`validate_count_query_groupby_against_index`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/query/drive_document_count_query/validate.rs); the per-mode path-query builders sit in [`packages/rs-drive/src/query/drive_document_count_query/path_query.rs`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/query/drive_document_count_query/path_query.rs)'s `group_by_*` family.
| # | Query | Filter + group_by | Complexity | Avg time | Proof size | Verified shape | Notes |
|---|-------|-------------------|------------|----------|------------|----------------|-------|
@@ -194,7 +194,7 @@ Entries([
])
```
-This is the load-bearing behaviour to know about: grovedb's `verify_query` *without* `absence_proofs_for_non_existing_searched_keys: true` drops absent-Key branches from the elements stream. The drive-side verifier ([`verify_point_lookup_count_proof_v0`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/verify/document_count/verify_point_lookup_count_proof/v0/mod.rs)) uses the default (off) and so emits one entry per **present** In value, not one per **requested** In value. Test coverage: [`test_point_lookup_proof_omits_absent_in_branches_from_entries`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/query/drive_document_count_query/tests.rs).
+This is the load-bearing behaviour to know about: grovedb's `verify_query` *without* `absence_proofs_for_non_existing_searched_keys: true` drops absent-Key branches from the elements stream. The drive-side verifier ([`verify_point_lookup_count_proof_v0`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/verify/document_count/verify_point_lookup_count_proof/v0/mod.rs)) uses the default (off) and so emits one entry per **present** In value, not one per **requested** In value. Test coverage: [`test_point_lookup_proof_omits_absent_in_branches_from_entries`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/query/drive_document_count_query/tests.rs).
**Caller implication.** Callers MUST NOT assume `entries.len() == |In|`. To check whether a specific In value matched, demux entries by serialized key (the same `serialize_value_for_key(field, value)` the path-query builder uses for outer Keys) — see the test for the canonical pattern. A `0`-count vs absent-key distinction would require passing `absence_proofs_for_non_existing_searched_keys: true` end-to-end, which the platform doesn't expose today.
@@ -840,7 +840,7 @@ Entries(100 groups, sum = 10 000)
The 100 groups are color_00000501 through color_00000600 (the first 100 in-range colors in lex-asc order, capped by the limit). Each carries `count_value_or_default = 100` since the fixture's deterministic schedule gives each color exactly 100 documents.
-Wait — but [Q7](./count-index-examples.md#query-7--range-query-aggregatecountonrange) said there are 499 distinct in-range colors and `sum = 49 900` over the same `color > "color_00000500"` predicate. So why does G4 see only 100 groups summing to 10 000? Because `GroupByRange`'s `distinct_count_path_query` applies the 100-entry response cap (`Some(limit)` in [`execute_distinct_count_with_proof`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/query/drive_document_count_query/execute_range_count.rs)). Without that cap the proof would scale linearly with the full in-range distinct count (~5.5 KB for the full 499 colors at ~110 B per resolved CountTree branch). The cap is a response-size safety control — the verifier ceases the walk once it has 100 entries.
+Wait — but [Q7](./count-index-examples.md#query-7--range-query-aggregatecountonrange) said there are 499 distinct in-range colors and `sum = 49 900` over the same `color > "color_00000500"` predicate. So why does G4 see only 100 groups summing to 10 000? Because `GroupByRange`'s `distinct_count_path_query` applies the 100-entry response cap (`Some(limit)` in [`execute_distinct_count_with_proof`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/query/drive_document_count_query/execute_range_count.rs)). Without that cap the proof would scale linearly with the full in-range distinct count (~5.5 KB for the full 499 colors at ~110 B per resolved CountTree branch). The cap is a response-size safety control — the verifier ceases the walk once it has 100 entries.
**Proof size:** 10 992 B — ~5.3 × [Q7](./count-index-examples.md#query-7--range-query-aggregatecountonrange). The structural reason:
@@ -1538,7 +1538,7 @@ SizedQuery::limit: 10 // platform defa
outer Query.left_to_right: false // from order_by [(brand, desc)]
```
-**Same-field range merging.** The caller's wire shape carries *four* range clauses (`brand >`, `brand <`, `color >`, `color <`). The dispatcher merges each same-field pair into a single `BetweenExcludeBounds` clause via [`merge_same_field_range_pairs`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/query/drive_document_count_query/drive_dispatcher.rs) before mode detection runs. After merging, the structure is identical to G8's two-range shape; mode detection routes to `RangeAggregateCarrierProof` for the same reasons.
+**Same-field range merging.** The caller's wire shape carries *four* range clauses (`brand >`, `brand <`, `color >`, `color <`). The dispatcher merges each same-field pair into a single `BetweenExcludeBounds` clause via [`merge_same_field_range_pairs`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/query/drive_document_count_query/drive_dispatcher.rs) before mode detection runs. After merging, the structure is identical to G8's two-range shape; mode detection routes to `RangeAggregateCarrierProof` for the same reasons.
**Verified payload** (descending walk — outer keys come out from highest to lowest, capped at `L = 10`):
@@ -1684,7 +1684,7 @@ group_by = [brand, color]
prove = true
```
-**Outcome:** `Err(QuerySyntaxError::InvalidWhereClauseComponents("count query supports at most one range where-clause; combine two-sided ranges via `between*` instead of separate `>` / `<` clauses, or use `group_by = [outer_range_field]` with `prove = true` for the carrier-aggregate shape with one outer range and one inner ACOR range on a different field"))` — at [`detect_mode`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/query/drive_document_count_query/mode_detection.rs)'s `range_count > 1` short-circuit, before any index picking or path-query building.
+**Outcome:** `Err(QuerySyntaxError::InvalidWhereClauseComponents("count query supports at most one range where-clause; combine two-sided ranges via `between*` instead of separate `>` / `<` clauses, or use `group_by = [outer_range_field]` with `prove = true` for the carrier-aggregate shape with one outer range and one inner ACOR range on a different field"))` — at [`detect_mode`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/query/drive_document_count_query/mode_detection.rs)'s `range_count > 1` short-circuit, before any index picking or path-query building.
**Why.** The two-range carrier shape (`outer_range AND inner_range` on distinct fields) is opened by mode detection **only** when `mode == GroupByRange` *and* `group_by.len() == 1` *and* `prove = true`. G8b violates the first two: with `group_by = [brand, color]` the request maps to `CountMode::GroupByCompound`, which routes to `distinct_count_path_query` — a builder that knows how to walk an `In + range` fan-out but not a `range + range` cartesian product. Two design points:
@@ -1721,7 +1721,7 @@ This chapter now mirrors chapter 29's per-query structure: every section above c
Two pieces of infrastructure made this possible:
-- `query_g1_*` … `query_g6_*` criterion `bench_function` calls in [`document_count_worst_case.rs`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/benches/document_count_worst_case.rs) — produce the **Avg time** column in [Queries in this Chapter](#queries-in-this-chapter).
+- `query_g1_*` … `query_g6_*` criterion `bench_function` calls in [`document_count_worst_case.rs`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/benches/document_count_worst_case.rs) — produce the **Avg time** column in [Queries in this Chapter](#queries-in-this-chapter).
- `display_group_by_proofs` (a sibling of `display_proofs` in the same bench file) — emits each `group_by` shape's verbatim merk-proof structure via bincode decode + `GroveDBProof::Display`. Tagged with `[gproof]` prefix in stderr so reviewers can grep deterministically.
Open follow-ups:
diff --git a/book/src/drive/document-count-trees.md b/book/src/drive/document-count-trees.md
index e858971ef4d..6250753ee57 100644
--- a/book/src/drive/document-count-trees.md
+++ b/book/src/drive/document-count-trees.md
@@ -60,7 +60,7 @@ A document type opts in via two schema flags:
## How a Document Type Picks Its Tree Variant
-Selection lives in [`packages/rs-drive/src/drive/document/primary_key_tree_type.rs`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/drive/document/primary_key_tree_type.rs):
+Selection lives in [`packages/rs-drive/src/drive/document/primary_key_tree_type.rs`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/drive/document/primary_key_tree_type.rs):
```rust
impl DocumentTypePrimaryKeyTreeType for DocumentTypeRef<'_> {
@@ -122,9 +122,9 @@ A single unified gRPC endpoint exposes the feature: `GetDocumentsCount`. The res
### No-Prove (Server-Side O(1) or O(log n))
-When `prove=false`, drive-abci calls into `DriveDocumentCountQuery` (in [`packages/rs-drive/src/query/drive_document_count_query/mod.rs`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/query/drive_document_count_query/mod.rs)). The handler picks a path based on the where clauses:
+When `prove=false`, drive-abci calls into `DriveDocumentCountQuery` (in [`packages/rs-drive/src/query/drive_document_count_query/mod.rs`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/query/drive_document_count_query/mod.rs)). The handler picks a path based on the where clauses:
-**Unfiltered total (no where clauses) on a `documentsCountable: true` document type** ([`Drive::read_primary_key_count_tree`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/query/drive_document_count_query/drive_dispatcher.rs)):
+**Unfiltered total (no where clauses) on a `documentsCountable: true` document type** ([`Drive::read_primary_key_count_tree`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/query/drive_document_count_query/drive_dispatcher.rs)):
The doctype's primary-key tree at `[contract_doc, contract_id, 1, doctype, 0]` is itself a `CountTree`. One grovedb read gives `count_value` — the total document count. O(1).
@@ -153,17 +153,17 @@ When `prove=true`, the proof shape depends on whether the query carries a range
- **Aggregate (`return_distinct_counts_in_range = false`, default)**: drive-abci builds a grovedb [`AggregateCountOnRange`](https://docs.rs/grovedb/latest/grovedb/struct.GroveDb.html#method.verify_aggregate_count_query) path query against the property-name `ProvableCountTree`, and `get_proved_path_query` produces an aggregate-count proof. The client verifies via `GroveDb::verify_aggregate_count_query` and recovers `(root_hash, count)` directly — proof size is O(log n) regardless of how many keys match. No documents are ever materialized.
-- **Distinct (`return_distinct_counts_in_range = true`)**: drive-abci builds a *regular* range path query (no `AggregateCountOnRange` wrapper) against the same `ProvableCountTree`. Because the leaf is a `ProvableCountTree`, merk emits one `Node::KVCount(key, value, count)` op per matched in-range key, with each `count` cryptographically committed to the merk root via `node_hash_with_count(kv_hash, l_hash, r_hash, count)` — same forge-resistance as the aggregate path's `HashWithCount` collapse. The SDK's [`drive_proof_verifier::verify_distinct_count_proof`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive-proof-verifier/src/proof/document_count.rs) runs the standard hash-chain check, then walks the proof's op stream to extract the counts as a `BTreeMap, u64>`. Trade-off vs. the aggregate path: proof size is O(distinct values matched) rather than O(log n), because each distinct in-range key emits its own `KVCount` op instead of being collapsed into a boundary subtree. Acceptable for typical histograms (a few dozen distinct values in range); for "give me a single count" use the aggregate path instead.
+- **Distinct (`return_distinct_counts_in_range = true`)**: drive-abci builds a *regular* range path query (no `AggregateCountOnRange` wrapper) against the same `ProvableCountTree`. Because the leaf is a `ProvableCountTree`, merk emits one `Node::KVCount(key, value, count)` op per matched in-range key, with each `count` cryptographically committed to the merk root via `node_hash_with_count(kv_hash, l_hash, r_hash, count)` — same forge-resistance as the aggregate path's `HashWithCount` collapse. The SDK's [`drive_proof_verifier::verify_distinct_count_proof`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive-proof-verifier/src/proof/document_count.rs) runs the standard hash-chain check, then walks the proof's op stream to extract the counts as a `BTreeMap, u64>`. Trade-off vs. the aggregate path: proof size is O(distinct values matched) rather than O(log n), because each distinct in-range key emits its own `KVCount` op instead of being collapsed into a boundary subtree. Acceptable for typical histograms (a few dozen distinct values in range); for "give me a single count" use the aggregate path instead.
**Without a range clause** (point-lookup with prove): two sub-paths based on the request shape.
-- **Unfiltered total + `documentsCountable: true`**: drive-abci proves the doctype's primary-key `CountTree` element at `[contract_doc, contract_id, 1, doctype, 0]`. One merk path proof; the SDK's [`drive_proof_verifier::verify_primary_key_count_tree_proof`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive-proof-verifier/src/proof/document_count.rs) reads `count_value` off the verified element. O(log n) bytes.
+- **Unfiltered total + `documentsCountable: true`**: drive-abci proves the doctype's primary-key `CountTree` element at `[contract_doc, contract_id, 1, doctype, 0]`. One merk path proof; the SDK's [`drive_proof_verifier::verify_primary_key_count_tree_proof`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive-proof-verifier/src/proof/document_count.rs) reads `count_value` off the verified element. O(log n) bytes.
- **Equal/In against a fully-covering `countable: true` index**: drive-abci proves one `Element::CountTree` per covered branch. Two sub-shapes:
- **Equal-only fully-covered** → one element at `[..., last_field, last_value, 0]`.
- **`In` at any index position (with any number of trailing Equals)** → one element per In value, fetched via outer Query + a subquery whose `set_subquery_path` carries the post-In Equal segments (zero of them when In is on the last property; one or more when In sits earlier in the index). The subquery's `Key([0])` picks off the CountTree at `[..., in_field, in_value, , 0]` for each matched In branch.
- The In position rule for count queries is **more permissive than the regular document query path's `Index::matches`** rule (which restricts In to last-or-before-last because of a positional path-construction assumption — see `DriveDocumentQuery::get_non_primary_key_path_query` for the layout that forces it). The count path doesn't have that constraint: there's no document-key terminator descent, no `order_by` interpretation, and no `limit/offset` semantics — it's a pure CountTree-element lookup, so `set_subquery_path` with an arbitrary trailing tail works. Both no-proof and prove count executors route through a single `point_lookup_count_path_query` builder (no-proof runs the path query via `grove.query` and sums the emitted `CountTree` elements' counts; prove signs the same path query via `get_proved_path_query`), so they accept the same query shapes by construction. The SDK's [`drive_proof_verifier::verify_point_lookup_count_proof`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive-proof-verifier/src/proof/document_count.rs) verifies and extracts `count_value_or_default()` from each verified element.
+ The In position rule for count queries is **more permissive than the regular document query path's `Index::matches`** rule (which restricts In to last-or-before-last because of a positional path-construction assumption — see `DriveDocumentQuery::get_non_primary_key_path_query` for the layout that forces it). The count path doesn't have that constraint: there's no document-key terminator descent, no `order_by` interpretation, and no `limit/offset` semantics — it's a pure CountTree-element lookup, so `set_subquery_path` with an arbitrary trailing tail works. Both no-proof and prove count executors route through a single `point_lookup_count_path_query` builder (no-proof runs the path query via `grove.query` and sums the emitted `CountTree` elements' counts; prove signs the same path query via `get_proved_path_query`), so they accept the same query shapes by construction. The SDK's [`drive_proof_verifier::verify_point_lookup_count_proof`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive-proof-verifier/src/proof/document_count.rs) verifies and extracts `count_value_or_default()` from each verified element.
Both sub-paths share the proof shape: each CountTree element's `count_value` is cryptographically bound to the merk root via `node_hash_with_count(kv_hash, l_hash, r_hash, count)`, same forge-resistance guarantee the range-distinct path relies on. Neither materializes documents or runs per-key bookkeeping client-side.
@@ -172,9 +172,9 @@ Proof size: **O(k × log n)** where k is the number of covered branches (1 for t
**Symmetric rejection contract**: prove count requires a `countable: true` index whose properties exactly match the where clauses — same requirement as the no-proof `Total` / `PerInValue` modes. Partial coverage (where the where clauses are a strict prefix of the index, or the index has uncovered properties) rejects with a `WhereClauseOnNonIndexedProperty`-class error pointing the caller at the index-design fix. The `documents_countable: true` fast path handles unfiltered total counts in O(log n) proof bytes when set on the document type. No silent fallback to materializing matching documents — that path doesn't exist anymore.
Implementation reference:
-- Path query: [`DriveDocumentCountQuery::point_lookup_count_path_query`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/query/drive_document_count_query/path_query.rs) — shared by prover and verifier.
-- Server executor: [`DriveDocumentCountQuery::execute_point_lookup_count_with_proof`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/query/drive_document_count_query/execute_point_lookup.rs).
-- Verifier: [`DriveDocumentCountQuery::verify_point_lookup_count_proof`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/verify/document_count/verify_point_lookup_count_proof/mod.rs); SDK wrapper [`drive_proof_verifier::verify_point_lookup_count_proof`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive-proof-verifier/src/proof/document_count.rs) composes tenderdash signature verification on top.
+- Path query: [`DriveDocumentCountQuery::point_lookup_count_path_query`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/query/drive_document_count_query/path_query.rs) — shared by prover and verifier.
+- Server executor: [`DriveDocumentCountQuery::execute_point_lookup_count_with_proof`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/query/drive_document_count_query/execute_point_lookup.rs).
+- Verifier: [`DriveDocumentCountQuery::verify_point_lookup_count_proof`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/verify/document_count/verify_point_lookup_count_proof/mod.rs); SDK wrapper [`drive_proof_verifier::verify_point_lookup_count_proof`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive-proof-verifier/src/proof/document_count.rs) composes tenderdash signature verification on top.
### Supported Where Operators
@@ -386,7 +386,7 @@ A few notes about the index-level flag:
| O(1) filtered count: `count(*) WHERE col = X` | `countable: true` on an index whose properties are exactly `["col"]`. A composite index whose leading column is `col` (e.g. `["col", "other"]`) does NOT answer this query — partial coverage rejects with `WhereClauseOnNonIndexedProperty`. Define a separate `["col"]` countable index if you want this count. |
| Per-`In`-value sub-counts: one `CountEntry` per value in an `In` clause | `countable: true` on an index whose properties exactly match the query's `==` clauses plus the `In` field. **The `In` field may sit at any position in the index** — both the no-proof and prove count paths use `set_subquery_path` to descend through any trailing Equals after the In, which is strictly more permissive than the regular document query path's last-or-before-last rule. E.g. `WHERE color IN [...]` needs `["color"]`; `WHERE brand = X AND color IN [...]` needs `["brand", "color"]`; `WHERE brand IN [...] AND model = X AND year = 2024` needs `["brand", "model", "year"]` with In on `brand` (position 0 of 3). |
| O(log n) range count: `count(*) WHERE col BETWEEN A AND B` | `rangeCountable: true` on an index whose last property is `col` and whose other properties cover any equality predicates as a prefix. Implies `countable: true`. |
-| Per-distinct-value range histogram: one `CountEntry` per distinct value in a range | Same `rangeCountable: true` index as above, plus `return_distinct_counts_in_range = true` on the request. Available on both prove and no-prove paths; the prove path returns a regular range proof against the property-name `ProvableCountTree` and the SDK extracts per-key counts from the proof's `KVCount` ops via [`drive_proof_verifier::verify_distinct_count_proof`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive-proof-verifier/src/proof/document_count.rs). |
+| Per-distinct-value range histogram: one `CountEntry` per distinct value in a range | Same `rangeCountable: true` index as above, plus `return_distinct_counts_in_range = true` on the request. Available on both prove and no-prove paths; the prove path returns a regular range proof against the property-name `ProvableCountTree` and the SDK extracts per-key counts from the proof's `KVCount` ops via [`drive_proof_verifier::verify_distinct_count_proof`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive-proof-verifier/src/proof/document_count.rs). |
| Range count proof (`prove = true` + range clause) | Same `rangeCountable: true` index. The handler uses grovedb's `AggregateCountOnRange` proof primitive — proof is O(log n), no cap on matched docs. |
| Future offset-style range queries (not yet released — see above) | `rangeCountable: true` on the document type |
| Nothing count-aware (default) | Don't set any of these flags. Primary-key tree stays a `NormalTree`. |
diff --git a/book/src/drive/document-sum-trees.md b/book/src/drive/document-sum-trees.md
index a419fd2c595..932b240704c 100644
--- a/book/src/drive/document-sum-trees.md
+++ b/book/src/drive/document-sum-trees.md
@@ -2,7 +2,7 @@
Summing a numeric property across the documents that match a query used to mean fetching them all and adding values up client-side. The grovedb upgrade that landed alongside [Document Count Trees](./document-count-trees.md) adds **provable sum trees** and **references with sum item** as primitives — the building blocks Drive uses to turn `sum(amount)`-style queries into an O(log n) provable lookup. This chapter explains the three sum-tree variants, how a document type opts into one, the unified `GetDocumentsSum` endpoint that exposes the feature, and the parallels with the count-tree machinery.
-> **Status:** the grovedb-level sum-tree primitives (`SumTree`, `ProvableSumTree`, `BigSumTree`, and reference elements that carry a sum-item contribution) are in place. The Drive-level schema syntax, query handler, and SDK surfaces described below are the proposed design — the [Sum Index Examples](./sum-index-examples.md) chapter is the worked-example companion, and the tip-jar contract fixture at [`packages/rs-drive/tests/supporting_files/contract/tip-jar/tip-jar-contract.json`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/tests/supporting_files/contract/tip-jar/tip-jar-contract.json) is the schema this design targets.
+> **Status:** the grovedb-level sum-tree primitives (`SumTree`, `ProvableSumTree`, `BigSumTree`, and reference elements that carry a sum-item contribution) are in place. The Drive-level schema syntax, query handler, and SDK surfaces described below are the proposed design — the [Sum Index Examples](./sum-index-examples.md) chapter is the worked-example companion, and the tip-jar contract fixture at [`packages/rs-drive/tests/supporting_files/contract/tip-jar/tip-jar-contract.json`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/tests/supporting_files/contract/tip-jar/tip-jar-contract.json) is the schema this design targets.
## Why Sum Trees Exist
@@ -124,7 +124,7 @@ The two-path / two-shape split is identical to the count endpoint's, and for the
### No-Prove (Server-Side O(1) or O(log n))
-When `prove=false`, drive-abci calls into `DriveDocumentSumQuery` (the proposed analog of `DriveDocumentCountQuery` in [`packages/rs-drive/src/query/drive_document_count_query/mod.rs`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/query/drive_document_count_query/mod.rs)). The handler picks a path based on the where clauses:
+When `prove=false`, drive-abci calls into `DriveDocumentSumQuery` (the proposed analog of `DriveDocumentCountQuery` in [`packages/rs-drive/src/query/drive_document_count_query/mod.rs`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/query/drive_document_count_query/mod.rs)). The handler picks a path based on the where clauses:
**Unfiltered total (no where clauses) on a `documentsSummable: "amount"` document type**:
diff --git a/book/src/drive/indexes.md b/book/src/drive/indexes.md
index c3169045295..bc8e026ae4a 100644
--- a/book/src/drive/indexes.md
+++ b/book/src/drive/indexes.md
@@ -32,7 +32,7 @@ Concrete example. Given:
## The `Index` Struct
-The compiled-Rust shape — the JSON schema fields are deserialized into this — lives in [`packages/rs-dpp/src/data_contract/document_type/index/mod.rs`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-dpp/src/data_contract/document_type/index/mod.rs):
+The compiled-Rust shape — the JSON schema fields are deserialized into this — lives in [`packages/rs-dpp/src/data_contract/document_type/index/mod.rs`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-dpp/src/data_contract/document_type/index/mod.rs):
```rust
pub struct Index {
@@ -122,7 +122,7 @@ The schema accepts both the legacy boolean form (`true` → `Countable`, `false`
## How Drive Builds the IndexLevel Trie
-The flat list of `Index`es declared on a document type is compiled, at contract-load time, into an `IndexLevel` trie ([`packages/rs-dpp/src/data_contract/document_type/index_level/mod.rs`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-dpp/src/data_contract/document_type/index_level/mod.rs)):
+The flat list of `Index`es declared on a document type is compiled, at contract-load time, into an `IndexLevel` trie ([`packages/rs-dpp/src/data_contract/document_type/index_level/mod.rs`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-dpp/src/data_contract/document_type/index_level/mod.rs)):
```rust
pub struct IndexLevel {
@@ -468,7 +468,7 @@ Count trees automatically count *every* child element. A `NormalTree`, an `Item`
Sum trees behave the opposite way. Only sum-bearing element variants — `SumItem`, `ItemWithSumItem`, `ReferenceWithSumItem`, and the sum-bearing tree variants themselves — contribute to a parent `SumTree`'s running sum. `Item`, `Reference`, plain `NormalTree`, `CountTree` — all contribute **0** by default. That has two consequences:
-1. **Per-document contributions don't appear automatically.** A plain `Element::Reference` under a `SumTree` does not propagate any sum. We need a different reference element — `Element::ReferenceWithSumItem(path, max_hops, sum_value, flags)` — that carries an explicit `i64` sum contribution (the document's value at the `summable` property, frozen at insert time) alongside the usual reference-path bytes. Grovedb PR 670 adds this variant; Drive's index walker constructs it via [`make_document_reference_with_sum_item`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/drive/document/mod.rs) under any index path with `summable.is_some()`.
+1. **Per-document contributions don't appear automatically.** A plain `Element::Reference` under a `SumTree` does not propagate any sum. We need a different reference element — `Element::ReferenceWithSumItem(path, max_hops, sum_value, flags)` — that carries an explicit `i64` sum contribution (the document's value at the `summable` property, frozen at insert time) alongside the usual reference-path bytes. Grovedb PR 670 adds this variant; Drive's index walker constructs it via [`make_document_reference_with_sum_item`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/drive/document/mod.rs) under any index path with `summable.is_some()`.
2. **Sibling continuations usually don't need a wrapper.** A `NormalTree` continuation under a sum-bearing value tree contributes 0 by default — exactly what we want. No `NotSummed` wrap required. The exception is when the continuation is *itself* sum-bearing (e.g. a deeper compound index that's also `range_summable`); in that case wrap the continuation in `Element::NotSummed<*>` to keep its sum from leaking into the outer index's aggregate. Compare with `range_countable`, where **every** continuation needs `NonCounted` because every non-count-aware element auto-contributes 1.
#### Layout
@@ -509,7 +509,7 @@ Extend the widget contract with a numeric `price` property and promote both inde
Both indexes name **the same** sum property — `summable: "price"` in both. The DPP validator requires this: grovedb's sum trees aggregate `i64` per merk node with no per-tree property tag, so a contract that mixed `summable: "price"` and `summable: "fee"` on the same doctype would feed inconsistent contributions into the same merk hierarchy. `price` is `type: integer` and listed in `required` — both also enforced at contract-creation time.
-`byColorShape` combines `countable` (root-only doc count per `(color, shape)` pair) with `summable` + `rangeSummable` (per-node sums of `price`). Drive's dispatch table promotes this combination to **`ProvableCountProvableSumTree`** (PCPS) at the value-tree and `[0]` terminal levels — the only grovedb variant carrying per-node sums also carries per-node counts as a side effect, so the count side gets per-node tracking "for free" even though only the sum side was opted into provability. See [`DocumentTypePrimaryKeyTreeType::primary_key_tree_type`'s v1 dispatch table](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/drive/document/primary_key_tree_type.rs) for the full mapping.
+`byColorShape` combines `countable` (root-only doc count per `(color, shape)` pair) with `summable` + `rangeSummable` (per-node sums of `price`). Drive's dispatch table promotes this combination to **`ProvableCountProvableSumTree`** (PCPS) at the value-tree and `[0]` terminal levels — the only grovedb variant carrying per-node sums also carries per-node counts as a side effect, so the count side gets per-node tracking "for free" even though only the sum side was opted into provability. See [`DocumentTypePrimaryKeyTreeType::primary_key_tree_type`'s v1 dispatch table](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/drive/document/primary_key_tree_type.rs) for the full mapping.
The two indexes share the `color` prefix exactly as the count examples did, so the same shared-prefix layout still applies. What changes is the element types at every level from `'color'` downward — and the diagram below makes the compound case visible, because the `'shape'` continuation under each color is now *itself* a sum-bearing tree (since `byColorShape` is `rangeSummable`) and needs `Element::NotSummed<*>`-wrapping to keep its aggregate from leaking into the outer `byColor` sum.
@@ -608,7 +608,7 @@ Walking through how the aggregates layer:
##### Why PCPS at the value level
-PCPS is grovedb's only tree variant carrying per-node sums. When an index sets `countable: "" + summable + rangeSummable`, the dispatch table promotes the value tree to PCPS because there's no "ProvableSumCountTree" variant (per-node sum + root-only count) to land on. The count side gets per-node tracking "for free" — same storage cost as `ProvableCountSumTree`'s count-half since PCPS commits the same per-node count metadata. See [`primary_key_tree_type.rs`'s v1 dispatch table](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/drive/document/primary_key_tree_type.rs) for the full mapping.
+PCPS is grovedb's only tree variant carrying per-node sums. When an index sets `countable: "" + summable + rangeSummable`, the dispatch table promotes the value tree to PCPS because there's no "ProvableSumCountTree" variant (per-node sum + root-only count) to land on. The count side gets per-node tracking "for free" — same storage cost as `ProvableCountSumTree`'s count-half since PCPS commits the same per-node count metadata. See [`primary_key_tree_type.rs`'s v1 dispatch table](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/drive/document/primary_key_tree_type.rs) for the full mapping.
`byColor`, by contrast, has only `summable + rangeSummable` (no `countable`), so its value trees stay `SumTree` — root-only sum, no count tracking, no upgrade. The two indexes living side by side on the same widget contract show both sides of the dispatch.
@@ -648,13 +648,13 @@ Setting both `range_countable: true` AND `range_summable: true` on the same inde
The combined primitive is strictly cheaper than running two separate range queries: one proof envelope, one merk walk, and both metrics atomically bound to the same root hash (so they can't drift relative to each other across a concurrent write).
-The full dispatch table mapping `(countable, range_countable, summable, range_summable)` combinations to grovedb tree variants lives in [`DocumentTypePrimaryKeyTreeType::primary_key_tree_type`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/drive/document/primary_key_tree_type.rs)'s v1 arm; the index-walker dispatch in [`add_indices_for_index_level_for_contract_operations`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/drive/document/insert/add_indices_for_index_level_for_contract_operations) follows the same table at every recursion level.
+The full dispatch table mapping `(countable, range_countable, summable, range_summable)` combinations to grovedb tree variants lives in [`DocumentTypePrimaryKeyTreeType::primary_key_tree_type`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/drive/document/primary_key_tree_type.rs)'s v1 arm; the index-walker dispatch in [`add_indices_for_index_level_for_contract_operations`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/drive/document/insert/add_indices_for_index_level_for_contract_operations) follows the same table at every recursion level.
-End-to-end coverage for the sum surface lives in [`packages/rs-drive/benches/document_sum_worst_case.rs`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/benches/document_sum_worst_case.rs)'s tip-jar fixture (paralleling the count side's `document_count_worst_case.rs` widget bench), with the worked-example queries in [Sum Index Examples](sum-index-examples.md).
+End-to-end coverage for the sum surface lives in [`packages/rs-drive/benches/document_sum_worst_case.rs`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/benches/document_sum_worst_case.rs)'s tip-jar fixture (paralleling the count side's `document_count_worst_case.rs` widget bench), with the worked-example queries in [Sum Index Examples](sum-index-examples.md).
## Tree Type at the Terminal Level
-The decision happens in [`add_reference_for_index_level_for_contract_operations/v0/mod.rs`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/drive/document/insert/add_reference_for_index_level_for_contract_operations/v0/mod.rs):
+The decision happens in [`add_reference_for_index_level_for_contract_operations/v0/mod.rs`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/drive/document/insert/add_reference_for_index_level_for_contract_operations/v0/mod.rs):
```rust
if !index_type.index_type.is_unique() || any_fields_null {
@@ -722,7 +722,7 @@ Same convention as the layout diagram above: rectangles are tree-type elements,
## Null Handling
-The `any_fields_null` and `all_fields_null` flags are accumulated as Drive descends the index property list during insertion ([`add_indices_for_index_level_for_contract_operations/v0/mod.rs:170-171`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/drive/document/insert/add_indices_for_index_level_for_contract_operations/v0/mod.rs#L170-L171)):
+The `any_fields_null` and `all_fields_null` flags are accumulated as Drive descends the index property list during insertion ([`add_indices_for_index_level_for_contract_operations/v0/mod.rs:170-171`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/drive/document/insert/add_indices_for_index_level_for_contract_operations/v0/mod.rs#L170-L171)):
```rust
any_fields_null |= document_index_field.is_empty();
@@ -747,11 +747,11 @@ Putting it together, when Drive inserts a document into a contract `C` of type `
2. **`add_indices_for_index_level_for_contract_operations`** (recursive) — for each sub-level of the trie, pushes the property name and value onto the path, OR-accumulates `any_fields_null`, AND-accumulates `all_fields_null`, and recurses. If the current level has `has_index_with_type = Some(...)`, it also calls into step 3 *before* recursing further (because an index can terminate at a non-leaf trie level when another index continues past it).
3. **`add_reference_for_index_level_for_contract_operations`** — the terminal call. Decides between unique and non-unique-style storage using the matrix above; for the non-unique-style path it picks a `NormalTree` / `CountTree` / `ProvableCountTree` based on `countable`; finally inserts the document reference (or sub-tree containing it).
-Deletion mirrors the same walk in reverse — see [`packages/rs-drive/src/drive/document/delete/`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/drive/document/delete/).
+Deletion mirrors the same walk in reverse — see [`packages/rs-drive/src/drive/document/delete/`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/drive/document/delete/).
## Query Traversal
-When a query arrives at drive-abci, the document-query construction path picks one of the document type's indexes that "covers" the query — i.e., whose property prefix matches the query's equality clauses, in order. The picker is in [`packages/rs-drive/src/query/mod.rs`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/query/mod.rs) (look for `fn construct_path_query` and the index-selection helpers it calls). For count queries specifically there's a separate, count-tree-aware picker ([`drive_document_count_query/mod.rs`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/query/drive_document_count_query/mod.rs)) — see [Document Count Trees](document-count-trees.md) for that path.
+When a query arrives at drive-abci, the document-query construction path picks one of the document type's indexes that "covers" the query — i.e., whose property prefix matches the query's equality clauses, in order. The picker is in [`packages/rs-drive/src/query/mod.rs`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/query/mod.rs) (look for `fn construct_path_query` and the index-selection helpers it calls). For count queries specifically there's a separate, count-tree-aware picker ([`drive_document_count_query/mod.rs`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/query/drive_document_count_query/mod.rs)) — see [Document Count Trees](document-count-trees.md) for that path.
Once an index is picked, the query-engine builds a `PathQuery` whose path is exactly the prefix shape the insert code produced: `[DataContractDocuments, contract_id, 1, doc_type, prop, value, prop, value, …]`. GroveDB then walks the path in O(log n per level), reading the terminal sub-tree (or single reference) and returning matching documents.
diff --git a/book/src/drive/sum-index-examples.md b/book/src/drive/sum-index-examples.md
index c1c6e2a8ab4..48b5509f91f 100644
--- a/book/src/drive/sum-index-examples.md
+++ b/book/src/drive/sum-index-examples.md
@@ -1,10 +1,10 @@
# Sum Index Examples
-This chapter walks through a representative contract and shows what a sum-query proof actually proves — both the path query the prover signs and the verified element the verifier extracts. Every example uses the same `tip` document type on the **tip-jar contract** at [`packages/rs-drive/tests/supporting_files/contract/tip-jar/tip-jar-contract.json`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/tests/supporting_files/contract/tip-jar/tip-jar-contract.json), so the proof bytes, verified elements, and diagrams can all be cross-referenced against the same data once the bench fixture lands.
+This chapter walks through a representative contract and shows what a sum-query proof actually proves — both the path query the prover signs and the verified element the verifier extracts. Every example uses the same `tip` document type on the **tip-jar contract** at [`packages/rs-drive/tests/supporting_files/contract/tip-jar/tip-jar-contract.json`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/tests/supporting_files/contract/tip-jar/tip-jar-contract.json), so the proof bytes, verified elements, and diagrams can all be cross-referenced against the same data once the bench fixture lands.
The chapter assumes you've read [Document Sum Trees](./document-sum-trees.md) — that chapter explains the three tree variants (`NormalTree` / `SumTree` / `ProvableSumTree`), how `Element::NonCounted`-style "doesn't contribute to my parent's aggregation" wrappers work (now for sums as well), and how the schema's `documentsSummable` / `rangeSummable` flags select between them. Here we take that machinery as given and trace what each query *sees*.
-> **Status:** the bench at [`packages/rs-drive/benches/document_sum_worst_case.rs`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/benches/document_sum_worst_case.rs) lands the reproducible numbers below — same convention as the [Count Index Examples](./count-index-examples.md) chapter. All proof sizes are measured against a 100 000-row fixture; verified `sum` values are the actual sums the bench's matrix reports. The full surface — primary-key total, point lookups, In-fan-out, `AggregateSumOnRange` on both top-level and compound indexes, and the carrier-aggregate primitive from grovedb PR #670 — is fully wired and producing the byte counts in the table below.
+> **Status:** the bench at [`packages/rs-drive/benches/document_sum_worst_case.rs`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/benches/document_sum_worst_case.rs) lands the reproducible numbers below — same convention as the [Count Index Examples](./count-index-examples.md) chapter. All proof sizes are measured against a 100 000-row fixture; verified `sum` values are the actual sums the bench's matrix reports. The full surface — primary-key total, point lookups, In-fan-out, `AggregateSumOnRange` on both top-level and compound indexes, and the carrier-aggregate primitive from grovedb PR #670 — is fully wired and producing the byte counts in the table below.
## The Tip Jar Contract
@@ -1543,7 +1543,7 @@ sum_property = "amount"
prove = true
```
-This is the **carrier-aggregate sum** shape: an `In` clause on the index's prefix property combined with a range on its terminator, returning **one sum per resolved In-bucket** rather than a single aggregate across all matches. The `group_by = [recipient]` (not `[recipient, sentAt]`) routes through `SumMode::GroupByIn` — the routing table at [`mode_detection/v0/mod.rs`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/query/drive_document_sum_query/mode_detection/v0/mod.rs) maps `(GroupByIn, In, range, prove) → RangeAggregateCarrierProof`, which is what the per-In-bucket aggregation needs. A `group_by = [recipient, sentAt]` (`GroupByCompound`) routes to `RangeDistinctProof` instead — per-`(in_key, range_key)` distinct walk, a different proof shape entirely. Sum analog of count's [Range-Countable group-by carrier-aggregate](./count-index-examples.md#range-countable-group-by-carrier-aggregate). The primitive landed in [grovedb PR #670](https://github.com/dashpay/grovedb/pull/670) (head `e98bab5f`); the verifier is [`GroveDb::verify_aggregate_sum_query_per_key`](https://github.com/dashpay/grovedb/blob/e98bab5f/grovedb/src/operations/proof/aggregate_sum/mod.rs).
+This is the **carrier-aggregate sum** shape: an `In` clause on the index's prefix property combined with a range on its terminator, returning **one sum per resolved In-bucket** rather than a single aggregate across all matches. The `group_by = [recipient]` (not `[recipient, sentAt]`) routes through `SumMode::GroupByIn` — the routing table at [`mode_detection/v0/mod.rs`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/query/drive_document_sum_query/mode_detection/v0/mod.rs) maps `(GroupByIn, In, range, prove) → RangeAggregateCarrierProof`, which is what the per-In-bucket aggregation needs. A `group_by = [recipient, sentAt]` (`GroupByCompound`) routes to `RangeDistinctProof` instead — per-`(in_key, range_key)` distinct walk, a different proof shape entirely. Sum analog of count's [Range-Countable group-by carrier-aggregate](./count-index-examples.md#range-countable-group-by-carrier-aggregate). The primitive landed in [grovedb PR #670](https://github.com/dashpay/grovedb/pull/670) (head `e98bab5f`); the verifier is [`GroveDb::verify_aggregate_sum_query_per_key`](https://github.com/dashpay/grovedb/blob/e98bab5f/grovedb/src/operations/proof/aggregate_sum/mod.rs).
**Path query** (carrier-style: outer Query enumerates the In branches, subquery descends through the terminator's `AggregateSumOnRange`):
@@ -1967,7 +1967,7 @@ The two carrier-aggregate gates worth knowing:
- **`SizedQuery::limit`** caps the outer walk (here `limit = 100` accommodates all distinct recipients; for an open-ended outer `Range` clause it bounds how many In-branches the proof commits). The verifier rebuilds the same `limit` byte-for-byte; mismatched limits break the merk-root recomputation.
- **`SizedQuery::offset`** is rejected for carrier-aggregate (would change which `(outer_key, sum)` pairs end up in the proof; the use case isn't designed yet). Mirrors the count-side carrier-ACOR contract.
-The PCPS variant — `AggregateCountAndSumOnRange` on a carrier — exists too; same primitive, but returns `(outer_key, u64 count, i64 sum)` triples for indexes that opt into both `rangeCountable: true` and `rangeSummable: true`. Drive-side support is wired ([`DriveDocumentSumQuery::verify_carrier_aggregate_count_and_sum_proof`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-drive/src/verify/document_sum/verify_carrier_aggregate_count_and_sum_proof)); a worked PCPS-carrier example will live in a separate combined-feature chapter alongside its own contract.
+The PCPS variant — `AggregateCountAndSumOnRange` on a carrier — exists too; same primitive, but returns `(outer_key, u64 count, i64 sum)` triples for indexes that opt into both `rangeCountable: true` and `rangeSummable: true`. Drive-side support is wired ([`DriveDocumentSumQuery::verify_carrier_aggregate_count_and_sum_proof`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-drive/src/verify/document_sum/verify_carrier_aggregate_count_and_sum_proof)); a worked PCPS-carrier example will live in a separate combined-feature chapter alongside its own contract.
## Range Modes — Distinct Variant
diff --git a/packages/swift-sdk/Sources/SwiftDashSDK/PlatformWallet/PlatformWalletManagerShieldedSync.swift b/packages/swift-sdk/Sources/SwiftDashSDK/PlatformWallet/PlatformWalletManagerShieldedSync.swift
index 7d2429b56c4..c2990270cdf 100644
--- a/packages/swift-sdk/Sources/SwiftDashSDK/PlatformWallet/PlatformWalletManagerShieldedSync.swift
+++ b/packages/swift-sdk/Sources/SwiftDashSDK/PlatformWallet/PlatformWalletManagerShieldedSync.swift
@@ -28,7 +28,7 @@ public struct ShieldedIdentityCreateUnconfirmedError: LocalizedError {
/// Per-wallet outcome from a completed shielded sync pass.
///
/// Mirrors the Rust-side
-/// [`ShieldedSyncWalletResultFFI`](https://github.com/dashpay/platform/blob/v3.1-dev/packages/rs-platform-wallet-ffi/src/shielded_types.rs)
+/// [`ShieldedSyncWalletResultFFI`](https://github.com/dashpay/platform/blob/v4.0-dev/packages/rs-platform-wallet-ffi/src/shielded_types.rs)
/// with three states:
///
/// - `success == true`: sync succeeded; the numeric counters are