Repository navigation
Replace lib2to3 in the proto code generation with a small script - #2881
Open
SammyTourani wants to merge 1 commit into
Open
SammyTourani wants to merge 1 commit into
SammyTourani wants to merge 1 commit into
Conversation
lib2to3 was removed in Python 3.13, so regenerating the protobuf sources (OVERWRITE_PB_SOURCE / scripts/build.sh --protobuf) failed with "No module named lib2to3". Rewrite the sibling *_pb2 imports as relative imports with scripts/fix_proto_imports.py instead. The regenerated files are identical to the ones produced with lib2to3.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2880
Both proto rules in
cmake/coreml-utils.cmakenow run a small stdlib-only script,scripts/fix_proto_imports.py, instead ofpython -m lib2to3 -wn --no-diff -f import.It is still run with
python, so the build needs nothing new.On protoc output, lib2to3's
importfixer makes only two rewrites, and the script makesthe same two:
import FeatureTypes_pb2 as FeatureTypes__pb2→from . import FeatureTypes_pb2 as FeatureTypes__pb2from FeatureTypes_pb2 import *→from .FeatureTypes_pb2 import *The regenerated files are byte-identical to the checked-in ones (see below).
The script decides from the
_pb2module name, while lib2to3 only rewrote an import whenthe sibling
.pyfile already existed on disk. For the flatmlmodel/formatdirectory thetwo give the same result, and the new check doesn't depend on the order in which the files
are generated.
The first rule (the one writing into the build directory) passes
coremltools/<name>_pb2.pyto the fixer rather thancoremltools/proto/<name>_pb2.py.No target depends on that rule's output, so it never runs; I left its path unchanged.
Verification
I built the bundled protoc (3.19.0) in a build directory configured with
cmake -DOVERWRITE_PB_SOURCE=ON. Then I ran the proto targets withpythononPATHpointing at each interpreter in turn, and checked the source tree afterwards.
protosrcregenerates the C++ sources, enum headers and_pb2.pyfiles for all 33protos into the source tree.
Command:
make -j8 protosrc && git status --short(once per interpreter)Result:
mainwith Python 3.13.15:No module named lib2to3, and the target fails(output above).
mainwith Python 3.9.6 (lib2to3 still present): all 33tgt_*_pythontargets succeed and
git statusis empty. So the checked-in files are exactlyprotoc 3.19.0 output plus lib2to3.
make -jthatbuild.sh --protobufruns after configuring also succeeds withPython 3.13.15 (exit 0; mlmodel, modelpackage, milstoragepython and coremlpython all
built). The source tree is unchanged afterwards.
(Python 3.9) on one copy and the new script (Python 3.13) on the other.
diff -rfindsno difference (52
from . import ...and 52from .... import *lines). Withprotobuf 7.36.2, the rewritten package imports and a
Modelmessage round-trips.Without the rewrite, the import fails with
ModuleNotFoundError: No module named 'VisionFeaturePrint_pb2'.No pytest run: nothing in the
coremltoolspackage changes, the regeneratedcoremltools/protofiles are byte-identical to the committed ones, and no existing testcovers proto regeneration. Tested on macOS 26.5 (arm64) only.