GET /cluster/ownership?bucket=&key= diagnostic endpoint to localize the
residual issue #37 read-after-write miss on a large cluster. From the responding
pod's own view it returns the key's owner, the data holders, where a request would_proxy_to, and whether this pod holds the key's metadata/data locally
(meta_present_local, data_present_local), plus the pod's ring_members. curl it against every pod for the same key: if they disagree on owner, the
placement ring is inconsistent across pods (the miss cause); if they agree but the
owner's data_present_local is false, it's data placement; if only meta_present_local lags, it's replication. Read-only and gated by the cluster
secret (X-Cluster-Secret) when one is configured, since it is served on the
public S3 port. On a healthy cluster every pod agrees on the owner, metadata is
present everywhere, and data is present only on the owner.
Cluster reads are no longer served by a node that holds no data (issue #37).
With replica_count = 1 an object's data lives on exactly one node. Routing forced
the candidate set to two nodes "for failover", so when a per-node failure detector
marked the true owner down — including a false positive from a stale/unreachable
probe address — a GET/HEAD could be answered by the second-ranked node (or
handled locally) even though it holds no data, returning a phantom Object not found for a live object that then "reappeared" on a retry routed elsewhere. This
is the read-after-write miss reported on a 12-node cluster that three prior
read-path fixes (v4.4.25–v4.4.31) did not reach, because the miss happens in
routing, before the consistent read runs. ShouldProxy now routes strictly within
the actual data-holder set (the first replica_count nodes): a read is served
locally only if this node holds the data, otherwise it is forwarded to the first
healthy holder, and if every holder is marked down it is still forwarded to the
primary owner rather than answered from an empty shard — so a falsely-down owner
still serves the read, and a genuinely-down single copy returns an honest upstream
error instead of a misleading 404. Legitimate failover with replica_count >= 2
(where other nodes really do hold the data) is unchanged. Unit-covered; happy-path
read-your-writes verified at 0 misses on a local multi-node cluster.
Opt-in read-404 cause tracing for cluster reads (VAULTS3_TRACE_READS=1), to
diagnose the residual issue #37 read-after-write miss on a large cluster. When
enabled, every GET/HEAD that returns 404 logs whether the cause is meta_nil (the object's Raft-replicated metadata hasn't arrived on this node — a
replication/consistency lag the consistent read waits out) or data_missing (the
metadata is present but this node has no data file — the read was served by a node
that isn't the shard owner, a routing problem), plus which node proxied it here.
On a local cluster (5 and 9 nodes, injected latency) the v4.4.29 read-your-writes
path reproduces 0 misses, so this trace is aimed at capturing the actual cause on
the reporter's 12-node topology. Off by default, zero overhead.
Opt-in per-hop latency tracing for cluster reads (VAULTS3_TRACE_FORWARD=1),
to diagnose issue #38 (a large fixed GET time-to-first-byte in some clustered
deployments). When enabled, every proxied request logs an httptrace breakdown of
the upstream hop — DNS resolution time, TCP connect time, whether the keep-alive
connection was reused, and upstream time-to-first-byte — so the ~fixed overhead can
be attributed to the extra pod hop, a slow DNS lookup (e.g. Kubernetes ndots
search expansion), a cold connection setup, or the owner's disk read. Off by
default with zero overhead. The forward path itself measures ~2ms end-to-end
(owner and forwarded reads alike) on a local multi-node cluster, so the fixed cost
reported in #38 originates in the deployment environment, not the request path;
this trace pinpoints where. Keep-alive to the owner is reused across requests once
a response body is fully read.
Configuration
📅Schedule: (UTC)
Branch creation
At any time (no schedule defined)
Automerge
At any time (no schedule defined)
🚦Automerge: Enabled.
♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.
🔕Ignore: Close this PR and you won't be reminded about this update again.
If you want to rebase/retry this PR, check this box
This PR contains the following updates:
| Package | Update | Change |
|---|---|---|
| [eniz1806/vaults3](https://github.com/Kodiqa-Solutions/VaultS3) | patch | `4.4.29` → `4.4.33` |
---
> ⚠️ **Warning**
>
> Some dependencies could not be looked up. Check the [Dependency Dashboard](issues/3) for more information.
---
### Release Notes
<details>
<summary>Kodiqa-Solutions/VaultS3 (eniz1806/vaults3)</summary>
### [`v4.4.33`](https://github.com/Kodiqa-Solutions/VaultS3/blob/HEAD/CHANGELOG.md#4433---2026-07-20)
[Compare Source](https://github.com/Kodiqa-Solutions/VaultS3/compare/v4.4.32...v4.4.33)
##### Added
- **`GET /cluster/ownership?bucket=&key=` diagnostic endpoint** to localize the
residual issue [#​37](https://github.com/Kodiqa-Solutions/VaultS3/issues/37) read-after-write miss on a large cluster. From the responding
pod's own view it returns the key's `owner`, the data `holders`, where a request
`would_proxy_to`, and whether this pod holds the key's metadata/data locally
(`meta_present_local`, `data_present_local`), plus the pod's `ring_members`.
`curl` it against every pod for the same key: if they disagree on `owner`, the
placement ring is inconsistent across pods (the miss cause); if they agree but the
owner's `data_present_local` is false, it's data placement; if only
`meta_present_local` lags, it's replication. Read-only and gated by the cluster
secret (`X-Cluster-Secret`) when one is configured, since it is served on the
public S3 port. On a healthy cluster every pod agrees on the owner, metadata is
present everywhere, and data is present only on the owner.
### [`v4.4.32`](https://github.com/Kodiqa-Solutions/VaultS3/blob/HEAD/CHANGELOG.md#4432---2026-07-20)
[Compare Source](https://github.com/Kodiqa-Solutions/VaultS3/compare/v4.4.31...v4.4.32)
##### Fixed
- **Cluster reads are no longer served by a node that holds no data** (issue [#​37](https://github.com/Kodiqa-Solutions/VaultS3/issues/37)).
With `replica_count = 1` an object's data lives on exactly one node. Routing forced
the candidate set to two nodes "for failover", so when a per-node failure detector
marked the true owner down — including a false positive from a stale/unreachable
probe address — a `GET`/`HEAD` could be answered by the second-ranked node (or
handled locally) even though it holds no data, returning a phantom `Object not
found` for a live object that then "reappeared" on a retry routed elsewhere. This
is the read-after-write miss reported on a 12-node cluster that three prior
read-path fixes (v4.4.25–v4.4.31) did not reach, because the miss happens in
routing, before the consistent read runs. `ShouldProxy` now routes strictly within
the actual data-holder set (the first `replica_count` nodes): a read is served
locally only if this node holds the data, otherwise it is forwarded to the first
healthy holder, and if every holder is marked down it is still forwarded to the
primary owner rather than answered from an empty shard — so a falsely-down owner
still serves the read, and a genuinely-down single copy returns an honest upstream
error instead of a misleading 404. Legitimate failover with `replica_count >= 2`
(where other nodes really do hold the data) is unchanged. Unit-covered; happy-path
read-your-writes verified at 0 misses on a local multi-node cluster.
### [`v4.4.31`](https://github.com/Kodiqa-Solutions/VaultS3/blob/HEAD/CHANGELOG.md#4431---2026-07-20)
[Compare Source](https://github.com/Kodiqa-Solutions/VaultS3/compare/v4.4.30...v4.4.31)
##### Added
- **Opt-in read-404 cause tracing for cluster reads** (`VAULTS3_TRACE_READS=1`), to
diagnose the residual issue [#​37](https://github.com/Kodiqa-Solutions/VaultS3/issues/37) read-after-write miss on a large cluster. When
enabled, every `GET`/`HEAD` that returns `404` logs whether the cause is
`meta_nil` (the object's Raft-replicated metadata hasn't arrived on this node — a
replication/consistency lag the consistent read waits out) or `data_missing` (the
metadata is present but this node has no data file — the read was served by a node
that isn't the shard owner, a routing problem), plus which node proxied it here.
On a local cluster (5 and 9 nodes, injected latency) the v4.4.29 read-your-writes
path reproduces 0 misses, so this trace is aimed at capturing the actual cause on
the reporter's 12-node topology. Off by default, zero overhead.
### [`v4.4.30`](https://github.com/Kodiqa-Solutions/VaultS3/blob/HEAD/CHANGELOG.md#4430---2026-07-20)
[Compare Source](https://github.com/Kodiqa-Solutions/VaultS3/compare/v4.4.29...v4.4.30)
##### Added
- **Opt-in per-hop latency tracing for cluster reads** (`VAULTS3_TRACE_FORWARD=1`),
to diagnose issue [#​38](https://github.com/Kodiqa-Solutions/VaultS3/issues/38) (a large fixed GET time-to-first-byte in some clustered
deployments). When enabled, every proxied request logs an `httptrace` breakdown of
the upstream hop — DNS resolution time, TCP connect time, whether the keep-alive
connection was reused, and upstream time-to-first-byte — so the \~fixed overhead can
be attributed to the extra pod hop, a slow DNS lookup (e.g. Kubernetes `ndots`
search expansion), a cold connection setup, or the owner's disk read. Off by
default with zero overhead. The forward path itself measures \~2ms end-to-end
(owner and forwarded reads alike) on a local multi-node cluster, so the fixed cost
reported in [#​38](https://github.com/Kodiqa-Solutions/VaultS3/issues/38) originates in the deployment environment, not the request path;
this trace pinpoints where. Keep-alive to the owner is reused across requests once
a response body is fully read.
</details>
---
### Configuration
📅 **Schedule**: (UTC)
- Branch creation
- At any time (no schedule defined)
- Automerge
- At any time (no schedule defined)
🚦 **Automerge**: Enabled.
♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.
🔕 **Ignore**: Close this PR and you won't be reminded about this update again.
---
- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box
---
This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4yNzIuMSIsInVwZGF0ZWRJblZlciI6IjQzLjI3Mi4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZSJdfQ==-->
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
This PR contains the following updates:
4.4.29→4.4.33Release Notes
Kodiqa-Solutions/VaultS3 (eniz1806/vaults3)
v4.4.33Compare Source
Added
GET /cluster/ownership?bucket=&key=diagnostic endpoint to localize theresidual issue #37 read-after-write miss on a large cluster. From the responding
pod's own view it returns the key's
owner, the dataholders, where a requestwould_proxy_to, and whether this pod holds the key's metadata/data locally(
meta_present_local,data_present_local), plus the pod'sring_members.curlit against every pod for the same key: if they disagree onowner, theplacement ring is inconsistent across pods (the miss cause); if they agree but the
owner's
data_present_localis false, it's data placement; if onlymeta_present_locallags, it's replication. Read-only and gated by the clustersecret (
X-Cluster-Secret) when one is configured, since it is served on thepublic S3 port. On a healthy cluster every pod agrees on the owner, metadata is
present everywhere, and data is present only on the owner.
v4.4.32Compare Source
Fixed
With
replica_count = 1an object's data lives on exactly one node. Routing forcedthe candidate set to two nodes "for failover", so when a per-node failure detector
marked the true owner down — including a false positive from a stale/unreachable
probe address — a
GET/HEADcould be answered by the second-ranked node (orhandled locally) even though it holds no data, returning a phantom
Object not foundfor a live object that then "reappeared" on a retry routed elsewhere. Thisis the read-after-write miss reported on a 12-node cluster that three prior
read-path fixes (v4.4.25–v4.4.31) did not reach, because the miss happens in
routing, before the consistent read runs.
ShouldProxynow routes strictly withinthe actual data-holder set (the first
replica_countnodes): a read is servedlocally only if this node holds the data, otherwise it is forwarded to the first
healthy holder, and if every holder is marked down it is still forwarded to the
primary owner rather than answered from an empty shard — so a falsely-down owner
still serves the read, and a genuinely-down single copy returns an honest upstream
error instead of a misleading 404. Legitimate failover with
replica_count >= 2(where other nodes really do hold the data) is unchanged. Unit-covered; happy-path
read-your-writes verified at 0 misses on a local multi-node cluster.
v4.4.31Compare Source
Added
VAULTS3_TRACE_READS=1), todiagnose the residual issue #37 read-after-write miss on a large cluster. When
enabled, every
GET/HEADthat returns404logs whether the cause ismeta_nil(the object's Raft-replicated metadata hasn't arrived on this node — areplication/consistency lag the consistent read waits out) or
data_missing(themetadata is present but this node has no data file — the read was served by a node
that isn't the shard owner, a routing problem), plus which node proxied it here.
On a local cluster (5 and 9 nodes, injected latency) the v4.4.29 read-your-writes
path reproduces 0 misses, so this trace is aimed at capturing the actual cause on
the reporter's 12-node topology. Off by default, zero overhead.
v4.4.30Compare Source
Added
VAULTS3_TRACE_FORWARD=1),to diagnose issue #38 (a large fixed GET time-to-first-byte in some clustered
deployments). When enabled, every proxied request logs an
httptracebreakdown ofthe upstream hop — DNS resolution time, TCP connect time, whether the keep-alive
connection was reused, and upstream time-to-first-byte — so the ~fixed overhead can
be attributed to the extra pod hop, a slow DNS lookup (e.g. Kubernetes
ndotssearch expansion), a cold connection setup, or the owner's disk read. Off by
default with zero overhead. The forward path itself measures ~2ms end-to-end
(owner and forwarded reads alike) on a local multi-node cluster, so the fixed cost
reported in #38 originates in the deployment environment, not the request path;
this trace pinpoints where. Keep-alive to the owner is reused across requests once
a response body is fully read.
Configuration
📅 Schedule: (UTC)
🚦 Automerge: Enabled.
♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR has been generated by Mend Renovate.