* fix: enable concurrent status computation to prevent multi-minute delays
Set StatusComputeWorkers=8 on DefaultStatusWatcher for Wait,
WaitWithJobs, and WatchUntilReady. This opts in to the async status
computation added in fluxcd/cli-utils#20, preventing the informer
notification pipeline from being blocked by slow API calls when many
resources are updated simultaneously.
Without this, status computation for resources like Deployments (which
require additional LIST ReplicaSets/Pods calls) runs serially inside
the informer, causing growing delays of 1-3+ minutes when upgrading
many resources at once (e.g., ~20 Deployments via Helm).
Signed-off-by: maplemiao <maplemiao@tencent.com>
* refactor(kube): expose StatusComputeWorkers as a WaitOption
Replace the package-level DefaultStatusComputeWorkers variable with a
WithStatusComputeWorkers WaitOption threaded through waitOptions into
the statusWaiter. This removes global mutable state from pkg/kube and
lets callers opt in explicitly.
SDK consumers (e.g. helm-controller) inherit the zero value, which
preserves the upstream cli-utils synchronous behavior and avoids an
unexpected fan-out of status-compute goroutines when many releases
reconcile concurrently. The Helm CLI continues to enable 8 workers by
default via a shared pkg/cmd/flags.go helper, so install/upgrade/
rollback/uninstall/test users still get the fix for multi-minute
informer stalls out of the box.
Signed-off-by: maplemiao <maplemiao@tencent.com>
* refactor(kube): clamp negative StatusComputeWorkers values to zero
Address review feedback: when a caller passes a negative value to
WithStatusComputeWorkers, coerce it to zero rather than propagating
it to the underlying cli-utils watcher, where the behavior is
undefined. Zero is the safe default and matches the SDK opt-in
contract.
Signed-off-by: maplemiao <maplemiao@tencent.com>
---------
Signed-off-by: maplemiao <maplemiao@tencent.com>
Signed-off-by: Maple Miao <maplemiao@yunzhanghu.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix missing conflict retry with server-side apply
Signed-off-by: Jakub Jaruszewski <jjaruszewski@man.poznan.pl>
* Only retry on resourcequota transient conflict
Signed-off-by: Jakub Jaruszewski <jjaruszewski@man.poznan.pl>
* Use isServerSideRetryable + naming nitpick
Signed-off-by: Jakub Jaruszewski <jjaruszewski@man.poznan.pl>
---------
Signed-off-by: Jakub Jaruszewski <jjaruszewski@man.poznan.pl>
- only cache kube client when creation succeeds
- add regression test for two failed reachability checks
Signed-off-by: Mohammad Abdolirad <m.abdolirad@gmail.com>
* fix: protect FailingKubeClient.RecordedWaitOptions from concurrent access
Add a sync.Mutex to guard the append to RecordedWaitOptions in
GetWaiterWithOptions, fixing a data race detected by -race when
concurrent goroutines (e.g. upgrade + rollback) both call
GetWaiterWithOptions on the same FailingKubeClient instance.
Fixes race failures in TestUpgradeRelease_Interrupted_RollbackOnFailure
and TestInstallRelease_RollbackOnFailure_Interrupted.
Signed-off-by: Terry Howe <thowe@nvidia.com>
* fix: extract appendRecordedWaitOptionsLocked helper with defer unlock
Signed-off-by: Terry Howe <terrylhowe@gmail.com>
---------
Signed-off-by: Terry Howe <thowe@nvidia.com>
Signed-off-by: Terry Howe <terrylhowe@gmail.com>
The previous change suppressed ctx.Err() whenever there were no
resource-specific errors, which incorrectly swallowed context.Canceled
and other non-deadline errors signalling an external interruption.
Refine the condition: only suppress context.DeadlineExceeded when there
are no resource-specific errors (resources are Unknown/NotFound, meaning
the delete wait succeeded or resources were already gone). Any other
context error — including context.Canceled — is always propagated.
Signed-off-by: Terry Howe <terrylhowe@gmail.com>
During informer initialization there is a brief window where watched
resources appear as Unknown before their real statuses are delivered.
The statusObserver skips Unknown resources when waiting for deletion
(they may have been deleted before the watch started), but if *all*
resources are in that transient Unknown state the skipped-resource list
is empty. AggregateStatus on an empty slice returns the desired status,
causing cancel() to be called immediately — before any real status event
has arrived.
Guard against this by tracking the count of Unknown-skipped resources.
When every resource was Unknown-skipped and none have a definitive status
yet, defer the early-cancel decision until at least one resource reports
a real status. This preserves the correct behaviour for resources that
were genuinely deleted before the watch started (they eventually receive
a NotFound or stay Unknown, and the aggregate succeeds), while fixing
the race for resources that are transiently Unknown at startup.
Also tighten the ctx.Err() check in waitForDelete: only append a
deadline error when there are resource-specific errors to accompany it.
A timeout while all resources are Unknown or NotFound is not itself an
error — the resources are in an acceptable state for a delete wait.
Fixes: TestStatusWaitForDelete/error_when_not_all_objects_are_deleted
Signed-off-by: Terry Howe <terrylhowe@gmail.com>
Move implementation to pkg/kubeenv per review; kube.RetryingRoundTripper
remains a type alias for API compatibility. pkg/cli uses kubeenv only.
Signed-off-by: Sumit Solanki <sumit.solanki@ibm.com>
The previous fix (increasing timeout / reducing deletion delay) did not
work because the flakiness is not a timing problem at all.
Root cause: fluxcd/cli-utils HasSynced() returns true after the initial
list item is *popped* from DeltaFIFO, which is before AddFunc delivers
the ResourceUpdateEvent to the collector. This creates a race where the
SyncEvent can arrive at the statusObserver *before* the pod's Current
status is recorded. When that happens:
- statusObserver sees pod as Unknown
- Unknown is skipped for WaitForDelete (by design, to handle resources
that were already deleted before watching started)
- AggregateStatus([], NotFoundStatus) == NotFoundStatus → cancel()
- The watch context is cancelled before DeleteFunc can fire
- Final check: pod still Current → error
The test intent is to verify that waitForDeleteCtx (not the cancelled
generalCtx) is selected. A non-existent resource satisfies this:
- With waitForDeleteCtx=Background(): informer syncs with empty list
→ Unknown → cancel → success ✓
- With generalCtx (cancelled, wrong): context immediately done
→ ctx.Err() appended → error returned ✓
Remove the goroutine-based deletion and the pod creation to eliminate
the race while preserving the context-selection assertion.
Signed-off-by: Terry Howe <terrylhowe@gmail.com>
TestMethodContextOverridesGeneralContext/WaitForDelete used a 1s
timeout with a 500ms deletion delay, leaving only ~500ms for the
fake watcher to propagate the delete event. On loaded CI runners
this window is too tight and causes intermittent failures.
Increase the timeout to 5s and reduce the deletion delay to 100ms
so there is ample headroom. Apply the same deletion-delay reduction
to TestStatusWaitForDelete which shares the same pattern.
Signed-off-by: Terry Howe <terrylhowe@gmail.com>
Remove pre-Go modules import path comments from pkg/kube test files
(ready_test.go, resource_test.go, statuswait_test.go) for consistency
with the rest of the package.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Terry Howe <terrylhowe@gmail.com>
Import path comments (e.g. `// import "helm.sh/helm/v4/pkg/kube"`) are
a pre-Go modules convention no longer needed in module-aware builds.
Some files in pkg/kube had these comments while others did not, causing
inconsistency that triggered downstream Kythe indexing errors.
Remove the import comments from all affected files to make the package
declaration consistent across the directory.
Fixes#31846
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: abhay1999 <abhaychaurasiya19@gmail.com>
Terry Howe
Matthieu MOREL
Zhaofeng Miao
Jakub Jaruszewski
George Jenkins
Mohammad Abdolirad
Austin Abro
Matheus Pimenta
Sumit Solanki
abhay1999
tsinglua