Basic implementation of z image fun control union 2.0 #11304
Conversation
The inpaint part is currently missing and will be implemented later. I think they messed up this model pretty bad. They added some control_noise_refiner blocks but don't actually use them. There is a typo in their code so instead of doing control_noise_refiner -> control_layers it runs the whole control_layers twice. Unfortunately they trained with this typo so the model works but is kind of slow and would probably perform a lot better if they corrected their code and trained it again.
|
After carefully reading the code, I noticed this issue exists, but if the model's performance is unaffected, reassigning values to the weights seems to resolve the speed issue—am I correct? |
|
If I assign the weights from the first two layers of the control layer to the control_noise_refiner and then perform inference using these modified weights, it seems to achieve consistent results. |
|
Well, this issue doesn't seem as simple as I initially thought. It looks like the only way to resolve the slow speed is to retrain—the input for the latter segment is the output from the previous segment. I'm really sorry for causing this mistake. |
|
Hello, could you please take a look and see if the fix in this PR is correct? I'll try training with the new code: aigc-apps/VideoX-Fun#404. Thank you so much! |
|
Yes that seems fine. |
https://huggingface.co/alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.0/tree/main They have uploaded the 2.1 version which fixes the issue you mentioned. Currently this error occurs for a user on Bonodoco |
Thanks for your reply. |
The inpaint part is currently missing and will be implemented later.
I think they messed up this model pretty bad. They added some control_noise_refiner blocks but don't actually use them. There is a typo in their code so instead of doing control_noise_refiner -> control_layers it runs the whole control_layers twice.
Unfortunately they trained with this typo so the model works but is kind of slow and would probably perform a lot better if they corrected their code and trained it again.