GH-49831: [Python] Withhold py.typed from the wheel until stubs are complete#50168
GH-49831: [Python] Withhold py.typed from the wheel until stubs are complete#50168rok wants to merge 5 commits into
Conversation
|
|
github-actions
Bot
added
Component: Python
awaiting committer review
There was a problem hiding this comment.
Pull request overview
This PR aims to prevent downstream type checkers from trusting PyArrow’s currently-incomplete bundled type stubs by withholding the PEP 561 py.typed marker from the built wheel (per GH-49831), while keeping stubs and py.typed in-tree for ongoing CI/type-checking work.
Changes:
- Configure the wheel build to exclude
pyarrow/py.typedviawheel.exclude. - Add explanatory comments documenting the rationale and removal TODO.
rok
force-pushed
the
broken-annotations-downstream
branch
from
325b6ac to
860479a
Compare
github-actions
Bot
added
awaiting changes
| source_root = Path(__file__).resolve().parents[2] | ||
| stubs_dir = source_root / "python" / "pyarrow-stubs" / "pyarrow" |
There was a problem hiding this comment.
@rok I strongly recommend excluding the entire directory.
This is all that is in there? 🤨
arrow/python/pyarrow-stubs/pyarrow/__init__.pyi
Lines 24 to 29 in 4fce185
We've been pinning pyarrow<24 and the errors we see in (narwhals-dev/narwhals#3560) will still be there for type checkers that aren't mypy 😔
There was a problem hiding this comment.
Ok, I'll change this to do so. Sorry it's taking so long.
There was a problem hiding this comment.
Thanks, and the speed is fine!
This is a big job for one person to do, especially when you're still contributing stuff that isn't typing 🫡
github-actions
Bot
added
awaiting change review
| # TODO(GH-48970): Uncomment when stubfiles are complete | ||
| # assert any( | ||
| # info.filename == "pyarrow/py.typed" for info in wheel_zip.filelist | ||
| # ), "pyarrow/py.typed is missing from the wheel." | ||
|
|
||
| # TODO(GH-49831): Re-enable when incomplete stubs are shipped in wheels | ||
| # again. For now, wheels intentionally omit pyarrow-stubs because some | ||
| # type checkers consume .pyi files even without py.typed. | ||
| # |
| # TODO(GH-49831): Temporarily do not install pyarrow-stubs into wheels. | ||
| # The stubs are incomplete, and some type checkers consume .pyi files even | ||
| # without the py.typed marker. Re-enable this when the stubs are complete. | ||
| # |
There was a problem hiding this comment.
You might wanna look into this repo as a follow up 🎁
There was a problem hiding this comment.
Oooh, if this works we can get rid of quite some logic, thanks!
| # TODO(GH-48970): Remove this when stubfiles are complete | ||
| # Withhold the PEP 561 marker until the type stubs are complete. The .pyi files | ||
| # are also temporarily omitted from wheels, so type checkers don't rely on the | ||
| # incomplete stubs and break downstream users (GH-49831). py.typed is kept | ||
| # in-tree for CI type-checking. | ||
| wheel.exclude = ["pyarrow/py.typed"] |
github-actions
Bot
added
awaiting changes
Rationale for this change
Since 24.0, pyarrow ships a
py.typedmarker while the type stubs are still incomplete. This makes type checkers trust the partial stubs and report false errors in downstream code (e.g. Module has no attribute "all" forpyarrow.compute). Some type checkers also consume bundled.pyifiles even withoutpy.typed, so the stubs need to be withheld from wheels for now. See #49831.What changes are included in this PR?
This temporarily omits both
pyarrow/py.typedand the bundledpyarrow-stubs/.pyifiles from built wheels until the stubs are complete. Wheel-content validation now asserts that neitherpy.typednor.pyifiles are present, and wheel build scripts no longer request stub docstring injection while stubs are not installed.Are these changes tested?
Wheel-content validation has been updated to check the intended absence of
py.typedand.pyifiles.Are there any user-facing changes?
Type checkers no longer pick up pyarrow's incomplete stubs from wheels.