fix: improve the reproducibility system by p-e-w · Pull Request #303 · p-e-w/heretic

p-e-w · 2026-04-21T06:24:11Z

This is a pretty major overhaul of #191, aiming to get it ready for the Heretic 1.3 release.

Changes include:

The user can now choose between uploading reproducibility information with system information, reproducibility information without system information, or no reproducibility information at all.
Commit handling for models and datasets actually works now.
Non-relevant fields are stripped from settings before saving them, improving compatibility and privacy.
Changed the structure of reproduce.json with a view to the future.
Better and more detailed reproduction instructions.
Configurable shard size for upload (fixes problems with some connections and improves reproducibility).
Lots and lots of small formatting and correctness fixes.

Examples:

Full reproduction information: https://huggingface.co/p-e-w/Qwen3-4B-Instruct-2507-heretic-REPRODUCTION-TEST-4
Without system information: https://huggingface.co/p-e-w/Qwen3-4B-Instruct-2507-heretic-REPRODUCTION-TEST-4-no-sysinfo
Reproduction information disabled: https://huggingface.co/p-e-w/Qwen3-4B-Instruct-2507-heretic-REPRODUCTION-TEST-4-no-info

@Vinay-Umrethe Requesting review!

gemini-code-assist

Code Review

This pull request refactors the reproducibility system to support multiple levels of detail and includes new settings for model commits and shard sizes. Key changes include the addition of a privacy-focused settings filter and updated logic for generating reproduction assets. Feedback identifies style guide violations regarding missing default configurations, unrelated whitespace changes, and missing type annotations, alongside a suggestion to use more idiomatic Pydantic practices for attribute exclusion.

Vinay-Umrethe · 2026-04-21T12:34:33Z

-{accelerator_report}
+- **Heretic:** v{version_info.version}{f" (Origin: {version_info.origin})" if version_info.origin else ""}
+- **PyTorch:** {pytorch_version}
+- **Other dependencies:** See [`requirements.txt`](requirements.txt).


isn't this line redundant as the 1st line just after it in contents section has requirements.txt info

I noticed that too, but the fact is that the information legitimately belongs in both places. How would you structure it? The dependencies are part of the environment.

maybe it's very obvious for anyone, so keeping it in any one of them is enough "See [requirements.txt]" is there not not does not matter.

I agree that this looks a bit strange, but I tried removing both references and something is clearly missing then. I prefer a bit of redundancy over confusion.

Vinay-Umrethe

done review of what I catched

Vinay-Umrethe · 2026-04-21T13:03:42Z

I noticed in your conducted tests:

direction_index value was Null (for basic and full test)

https://huggingface.co/p-e-w/Qwen3-4B-Instruct-2507-heretic-REPRODUCTION-TEST-4-no-sysinfo/blob/main/reproduce/reproduce.json#L227

stepping value of CPU was Null (for full test)

https://huggingface.co/p-e-w/Qwen3-4B-Instruct-2507-heretic-REPRODUCTION-TEST-4/blob/main/reproduce/reproduce.json#L20

I suppose py-cpuinfo must be generic enough to do the same task of getting stepping value for both AMD and Intel CPUs even if they use different naming conventions. Or he system had some issue (unlikely) as a number is almost always associated.

Vinay-Umrethe · 2026-04-21T13:06:25Z

Commit handling for models and datasets actually works now.

Previously was it not working?

p-e-w · 2026-04-21T14:30:00Z

direction_index value was Null (for basic and full test)

Correct. This is how the system expresses the "global" direction (None = no specific index). See the abliterate method.

stepping value of CPU was Null (for full test)

Yes, I noticed that. I guess not every CPU provides that information.

Commit handling for models and datasets actually works now.

Previously was it not working?

No? Did you intend to add that functionality? Because there was no relevant code at all yet. The commit was stored in the settings, but not used when fetching the datasets and model.

p-e-w · 2026-04-21T14:39:29Z

Note to self: The dependencies are not sorted in reproduce.json. This must be fixed.

Vinay-Umrethe · 2026-04-21T14:44:29Z

No? Did you intend to add that functionality? Because there was no relevant code at all yet. The commit was stored in the settings, but not used when fetching the datasets and model.

maybe I haven't checked that time was it being used or not. Correct

Vinay-Umrethe · 2026-04-21T14:46:22Z

Note to self: The dependencies are not sorted in reproduce.json. This must be fixed.

Not sure what you are talking about, but I do see requirements list in reproduce.json

p-e-w · 2026-04-21T14:51:00Z

Not sure what you are talking about, but I do see requirements list in reproduce.json

Yes, but it's not sorted.

p-e-w · 2026-04-21T15:07:43Z

I'm also planning for the final puzzle piece: heretic --reproduce /path/to/reproduce.json.

This seems quite complicated to implement, because it will require Heretic to create a temporary venv, install the correct Heretic version and dependencies there, and then quit to hand processing over to that version of Heretic.

Vinay-Umrethe · 2026-04-21T15:31:11Z

I'm also planning for the final puzzle piece: heretic --reproduce /path/to/reproduce.json.

if 1.3 will release after the reproducibility suite is automatically useable that'll take long,

This seems quite complicated to implement, because it will require Heretic to create a temporary venv, install the correct Heretic version and dependencies there, and then quit to hand processing over to that version of Heretic.

We can have a reproduce.py module to handle it entirely

p-e-w · 2026-04-23T07:11:58Z

if 1.3 will release after the reproducibility suite is automatically useable that'll take long,

Correct, that part won't be in the 1.3 release most likely.

Vinay-Umrethe · 2026-04-23T13:17:57Z

+        exclude=True,
+    )
+
+    max_shard_size: int | str = Field(


I don't know if This sharding logic is related to reproducibility

The default size may change between Transformers versions (apparently, this has already happened at least once). Storing the size in settings increases the chance of reproducibility if the Transformers version doesn't match.

Vinay-Umrethe · 2026-04-23T13:35:16Z

+> [!TIP]
+> **This model is reproducible!**
+>
+> See the [README](reproduce/README.md) in the `reproduce` directory for more information.


Suggested change

> See the [README](reproduce/README.md) in the `reproduce` directory for more information.

> See the [README](reproduce/README.md) in the [reproduce](https://huggingface.co/{settings.model}/tree/main/reproduce) directory for more information.

I've thought about that but decided against it because if the model is forked, quantized etc., absolute paths will link to the wrong (parent) model. It's unfortunate that HF doesn't support linking directories properly, but hacks only create more problems.

Also, your suggestion is linking to the wrong model anyway, namely to the base model, which doesn't have a reproduce directory.

p-e-w · 2026-04-23T13:41:45Z

Merged! Several remaining issues have been fixed and the exclusion logic for settings in particular is now much cleaner. I have re-tested everything before merging and found no more problems.

Vinay-Umrethe · 2026-04-23T13:42:35Z

Merged! Several remaining issues have been fixed and the exclusion logic for settings in particular is now much cleaner. I have re-tested everything before merging and found no more problems.

yes i saw all 3 test repos

p-e-w · 2026-04-23T13:46:06Z

I'm deleting all my test models now because many people follow my account and I don't want them to be quantized/benchmarked etc.

Vinay-Umrethe · 2026-04-23T13:47:30Z

I'm deleting all my test models now because many people follow my account and I don't want them to be quantized/benchmarked etc.

Yeah I was thinking why you didn't deleted old tests, as they don't look good 👍

Vinay-Umrethe · 2026-04-23T13:54:17Z

how was stepping value issue was fixed in CPU? in previous tests it was null. now it works
@p-e-w

p-e-w · 2026-04-25T02:49:14Z

@Vinay-Umrethe

It wasn't "fixed", I changed nothing there. I believe some CPUs provide that value, while others don't. I was on a different server so I probably just got one with a CPU that has the stepping set.

* fix: various cleanups and improvements for the reproducibility system * fix: save only essential settings * fix: improve model commit handling * feat: make including system information optional * fix: improve formatting of reproducibility README * fix: fix remaining issues

p-e-w added 5 commits April 15, 2026 11:13

fix: various cleanups and improvements for the reproducibility system

b46396b

fix: save only essential settings

0592090

fix: improve model commit handling

b47a541

feat: make including system information optional

cc19c5a

fix: improve formatting of reproducibility README

f6cb7fb

gemini-code-assist Bot reviewed Apr 21, 2026

View reviewed changes

Comment thread config.default.toml

Comment thread src/heretic/config.py Outdated

Comment thread src/heretic/system.py

Comment thread src/heretic/utils.py

Comment thread src/heretic/utils.py

Vinay-Umrethe reviewed Apr 21, 2026

View reviewed changes

Comment thread src/heretic/utils.py

Vinay-Umrethe reviewed Apr 21, 2026

View reviewed changes

Comment thread src/heretic/main.py

Vinay-Umrethe reviewed Apr 21, 2026

View reviewed changes

Comment thread src/heretic/config.py

p-e-w commented Apr 21, 2026

View reviewed changes

Comment thread src/heretic/utils.py Outdated

p-e-w mentioned this pull request Apr 23, 2026

fix: Enable LoRAs #308

Merged

p-e-w added 2 commits April 23, 2026 13:04

Merge branch 'master' into reproducibility-fixes

54f5daa

fix: fix remaining issues

5c0f344

Vinay-Umrethe reviewed Apr 23, 2026

View reviewed changes

p-e-w merged commit 513e3ac into master Apr 23, 2026
4 checks passed

	> See the [README](reproduce/README.md) in the `reproduce` directory for more information.
	> See the [README](reproduce/README.md) in the [reproduce](https://huggingface.co/{settings.model}/tree/main/reproduce) directory for more information.

Conversation

p-e-w commented Apr 21, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Vinay-Umrethe left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Vinay-Umrethe commented Apr 21, 2026

Uh oh!

Vinay-Umrethe commented Apr 21, 2026

Uh oh!

Uh oh!

Uh oh!

p-e-w commented Apr 21, 2026

Uh oh!

p-e-w commented Apr 21, 2026

Uh oh!

Vinay-Umrethe commented Apr 21, 2026

Uh oh!

Vinay-Umrethe commented Apr 21, 2026

Uh oh!

p-e-w commented Apr 21, 2026

Uh oh!

p-e-w commented Apr 21, 2026

Uh oh!

Vinay-Umrethe commented Apr 21, 2026

Uh oh!

p-e-w commented Apr 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

p-e-w commented Apr 23, 2026

Uh oh!

Vinay-Umrethe commented Apr 23, 2026

Uh oh!

p-e-w commented Apr 23, 2026

Uh oh!

Vinay-Umrethe commented Apr 23, 2026

Uh oh!

Vinay-Umrethe commented Apr 23, 2026

Uh oh!

p-e-w commented Apr 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Vinay-Umrethe left a comment •

edited

Loading