Skip to content

ci: disable sccache preprocessor cache mode#2221

Open
leofang wants to merge 1 commit into
NVIDIA:mainfrom
leofang:leofang/disable-sccache-preprocessor-mode
Open

ci: disable sccache preprocessor cache mode#2221
leofang wants to merge 1 commit into
NVIDIA:mainfrom
leofang:leofang/disable-sccache-preprocessor-mode

Conversation

@leofang

@leofang leofang commented Jun 15, 2026

Copy link
Copy Markdown
Member

Closes #2220.

Preprocessor cache hits are 0% in every CI run sampled, and the lookups cost ~20 min per Linux wheel job. Disable until the underlying pip-build-env path leak is fixed at the source. The object cache via GHAC is unaffected (still hits ~100%).

The preprocessor cache has produced 0 hits in every observed CI run
because pip's per-run build-isolation overlay path lands in -I flags
and sccache hashes raw compiler arguments without basedir stripping.
The lookup overhead is ~20 min per Linux wheel job; until the
underlying issue is fixed, just stop enabling it.

See NVIDIA#2220.
@leofang leofang self-assigned this Jun 15, 2026
@copy-pr-bot

copy-pr-bot Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@leofang leofang added the CI/CD CI/CD infrastructure label Jun 15, 2026
@leofang leofang added this to the cuda.core v1.1.0 milestone Jun 15, 2026
@leofang leofang added bug Something isn't working P0 High priority - Must do! labels Jun 15, 2026
@leofang

leofang commented Jun 15, 2026

Copy link
Copy Markdown
Member Author

/ok to test 369917e

@leofang leofang marked this pull request as draft June 15, 2026 17:50
@leofang

leofang commented Jun 15, 2026

Copy link
Copy Markdown
Member Author

Observed effect on Average preprocessor cache miss (same Build linux-64, CUDA 13.3.0 / py3.12py3.11 slot before/after; this PR's job ran py3.11):

build step before (main @ ae1617d, py3.12) after (this PR, py3.11)
Build cuda.bindings wheel 0.065 s 0.005 s
Build cuda.core wheel 13.780 s 0.003 s
Build cuda.core wheel (prev CTK) 7.476 s 0.002 s

Preprocessor cache hits stays at 0 across the board — preprocessor cache mode is still on for the local-disk tier (sccache's default), but local-only misses are essentially free. The GHA preprocessor tier (which never hit and accounted for all the network round-trips) is what's removed here.

@leofang leofang marked this pull request as ready for review June 15, 2026 18:10
@github-actions

Copy link
Copy Markdown

@mdboom

mdboom commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

and the lookups cost ~20 min per Linux wheel job

I think you mean ~20 sec? Still very worthwhile, just wanted to make sure I understand correctly.

@mdboom mdboom left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working CI/CD CI/CD infrastructure P0 High priority - Must do!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CI] Preprocessor cache hits = 0% across all Linux wheel builds (sccache + PEP 517 build isolation)

2 participants