__static_attributes__ is not deterministic

Bug report

Bug description:

Tested on commit: e9b00cc

Order of strings in __static_attributes__ is not deterministic

Running the following a couple of times produces different hashes:

/opt/cpython/cross-build/build/python ../../Programs/_freeze_module.py zipimport ../../Lib/zipimport.py Python/frozen_modules/zipimport.h
sha256sum ./Python/frozen_modules/zipimport.h

I patched this generator to output dis as follows:

diff --git a/Programs/_freeze_module.py b/Programs/_freeze_module.py
index ba638ee..7e837e9 100644
--- a/Programs/_freeze_module.py
+++ b/Programs/_freeze_module.py
@@ -24,6 +24,9 @@ def compile_and_marshal(name: str, text: bytes) -> bytes:
     filename = f"<frozen {name}>"
     # exec == Py_file_input
     code = compile(text, filename, "exec", optimize=0, dont_inherit=True)
+    import dis
+    with open("/tmp/a", "at") as f:
+        dis.dis(code, file=f)
     return marshal.dumps(code)

Then disassembly differs in one place (after removing at 0x[a-f0-9]*):

diff --git a/tmp/a b/tmp/b
index 372a97c..4010046 100644
--- a/tmp/a
+++ b/tmp/b
@@ -291,7 +291,7 @@ Disassembly of <code object zipimporter , file "<frozen zipimport>", line 50>:
 289           LOAD_CONST              15 (<code object __repr__ , file "<frozen zipimport>", line 289>)
               MAKE_FUNCTION
               STORE_NAME              16 (__repr__)
-              LOAD_CONST              16 (('archive', 'prefix'))
+              LOAD_CONST              16 (('prefix', 'archive'))
               STORE_NAME              17 (__static_attributes__)
               RETURN_CONST             4 (None)

diff in .h files is quite small, so I assume only sorting is affected:

-    0,0,0,41,2,114,46,0,0,0,114,44,0,0,0,169,
+    0,0,0,41,2,114,44,0,0,0,114,46,0,0,0,169,

CPython versions tested on:

3.13

Operating systems tested on:

No response

Linked PRs

@iritkatriel

This is because we store u_static_attributes as a set inside the compiler, and set iteration order is nondeterministic. While we don't guarantee any order for this attribute, I can see how nondeterministic ordering is problematic if you want reproducible builds.

We could fix this either by using some structure other than a set (e.g., a dict with dummy values), or by sorting the attributes when we convert them from a set into a tuple. cc @iritkatriel

This patch seems to fix it

diff --git a/Python/compile.c b/Python/compile.c
index 7752a68..dfaa5a4 100644
--- a/Python/compile.c
+++ b/Python/compile.c
@@ -2544,7 +2544,14 @@ compiler_class_body(struct compiler *c, stmt_ty s, int firstlineno)
         return ERROR;
     }
     assert(c->u->u_static_attributes);
-    PyObject *static_attributes = PySequence_Tuple(c->u->u_static_attributes);
+    PyObject *static_attributes_unsorted = PySequence_List(c->u->u_static_attributes);
+    if (static_attributes_unsorted == NULL) {
+        compiler_exit_scope(c);
+        return ERROR;
+    }
+    PyList_Sort(static_attributes_unsorted);
+    PyObject *static_attributes = PySequence_Tuple(static_attributes_unsorted);
+    Py_CLEAR(static_attributes_unsorted);
     if (static_attributes == NULL) {
         compiler_exit_scope(c);
         return ERROR;

If it's ok I can open a pr to not waste core team time

Go ahead and create a PR. Sorting at compile time prior to tupling will work. Another option would be to use a dict instead of a set which preserves insertion order. But I think it is much wiser for us to have the tuple always be in sorted order so that changes in the order of otherwise logically equivalent code do not result in different ordering in the tuple.

@Yhg1s

@Yhg1s fyi (probably not a release blocker - distributors who need the canonical output here should wind up backporting the fix that we'd have for 3.13.1)

fix issue python#124442: make `__static_attributes__` deterministic b…

gh-124442: make `__static_attributes__` deterministic by sorting (#12…

[3.13] pythongh-124442: make `__static_attributes__` deterministic by…

[3.13] gh-124442: make `__static_attributes__` deterministic by sorti…

Can this be closed? I don't think we need more deterministic processing.

Navigation Menu

Repository navigation

`__static_attributes__` is not deterministic #124442

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

__static_attributes__ is not deterministic #124442

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

JelleZijlstra commented on Sep 24, 2024

kp2pml30 commented on Sep 24, 2024

gpshead commented on Sep 24, 2024

gpshead commented on Sep 24, 2024

picnixz commented on Oct 6, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`__static_attributes__` is not deterministic #124442