Skip to content

Replace lib2to3 in the proto code generation with a small script - #2881

Open
SammyTourani wants to merge 1 commit into
apple:mainfrom
SammyTourani:fix/issue-2880
Open

SammyTourani wants to merge 1 commit into
apple:mainfrom
SammyTourani:fix/issue-2880

Conversation

@SammyTourani

Copy link
Copy Markdown

Fixes #2880

Both proto rules in cmake/coreml-utils.cmake now run a small stdlib-only script,
scripts/fix_proto_imports.py, instead of python -m lib2to3 -wn --no-diff -f import.
It is still run with python, so the build needs nothing new.

On protoc output, lib2to3's import fixer makes only two rewrites, and the script makes
the same two:

  • import FeatureTypes_pb2 as FeatureTypes__pb2 → from . import FeatureTypes_pb2 as FeatureTypes__pb2
  • from FeatureTypes_pb2 import * → from .FeatureTypes_pb2 import *

The regenerated files are byte-identical to the checked-in ones (see below).

The script decides from the _pb2 module name, while lib2to3 only rewrote an import when
the sibling .py file already existed on disk. For the flat mlmodel/format directory the
two give the same result, and the new check doesn't depend on the order in which the files
are generated.

The first rule (the one writing into the build directory) passes
coremltools/<name>_pb2.py to the fixer rather than coremltools/proto/<name>_pb2.py.
No target depends on that rule's output, so it never runs; I left its path unchanged.

Verification

I built the bundled protoc (3.19.0) in a build directory configured with
cmake -DOVERWRITE_PB_SOURCE=ON. Then I ran the proto targets with python on PATH
pointing at each interpreter in turn, and checked the source tree afterwards.
protosrc regenerates the C++ sources, enum headers and _pb2.py files for all 33
protos into the source tree.

Command: make -j8 protosrc && git status --short (once per interpreter)

Result:

  • Before, on main with Python 3.13.15: No module named lib2to3, and the target fails
    (output above).
  • Baseline, on main with Python 3.9.6 (lib2to3 still present): all 33 tgt_*_python
    targets succeed and git status is empty. So the checked-in files are exactly
    protoc 3.19.0 output plus lib2to3.
  • With this change:
Python 3.9.6:    make protosrc exit=0  python steps: 33, c++ steps: 33, enum steps: 33, errors: 0  git status --short: (empty)
Python 3.13.15:  make protosrc exit=0  python steps: 33, c++ steps: 33, enum steps: 33, errors: 0  git status --short: (empty)
Python 3.14.6:   make protosrc exit=0  python steps: 33, c++ steps: 33, enum steps: 33, errors: 0  git status --short: (empty)
Python 3.15.0b3: make protosrc exit=0  python steps: 33, c++ steps: 33, enum steps: 33, errors: 0  git status --short: (empty)
  • The full make -j that build.sh --protobuf runs after configuring also succeeds with
    Python 3.13.15 (exit 0; mlmodel, modelpackage, milstoragepython and coremlpython all
    built). The source tree is unchanged afterwards.
  • With a current protoc (libprotoc 36.2), I generated all 33 protos twice and ran lib2to3
    (Python 3.9) on one copy and the new script (Python 3.13) on the other. diff -r finds
    no difference (52 from . import ... and 52 from .... import * lines). With
    protobuf 7.36.2, the rewritten package imports and a Model message round-trips.
    Without the rewrite, the import fails with
    ModuleNotFoundError: No module named 'VisionFeaturePrint_pb2'.

No pytest run: nothing in the coremltools package changes, the regenerated
coremltools/proto files are byte-identical to the committed ones, and no existing test
covers proto regeneration. Tested on macOS 26.5 (arm64) only.

lib2to3 was removed in Python 3.13, so regenerating the protobuf sources
(OVERWRITE_PB_SOURCE / scripts/build.sh --protobuf) failed with
"No module named lib2to3". Rewrite the sibling *_pb2 imports as relative
imports with scripts/fix_proto_imports.py instead. The regenerated files
are identical to the ones produced with lib2to3.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The protoc code generator tooling still uses lib2to3 which is no longer shipped

1 participant