The 3D Guided Generative AI Blueprint unlocks greater control over image generation by laying out the content in Blender to guide the image layout. Users can quickly alter the look of the 3D scene using generative AI, and the image outputs can be iterated on by making simple changes in the 3D viewport - such as changing the image perspective by adjusting the camera angle in Blender. Creators can ideate on scene environments much faster using generative AI, and adjustments are made much faster due to the control offered by using the viewport as a depth map.

The blueprint produces high-quality outputs by leveraging the FLUX.dev NIM, using Black Forest Labs' state-of-the-art FLUX.dev models, and ComfyUI provides a flexible and convenient UI. The models are quantized and accelerated on NVIDIA GPUs, doubling performance and enabling this workflow to run on consumer GPUs. Sample image generation times using 30 steps at 1024x1024 resolution on a GeForce RTX 5090:

This blueprint is ready for non-commercial use. Contact sales@blackforestlabs.ai for commercial terms.

This blueprint supports the following NVIDIA GPUs: RTX 5090, RTX 5080, RTX 4090, RTX 6000 Ada. We recommend at least 48 GB of system RAM. We are in the process of adding support for more GPUs.

The NIM Prerequisite Installer requires Microsoft User Account Control (UAC) to be enabled. UAC is enabled by default for Windows, but if it has been disabled, it must be enabled to ensure successful installation of the NIM Prerequisite Installer. More information on Microsoft UAC can found HERE

Download the NIM Prerequisite Installer, extract the zip file and run the NIMSetup.exe file, and follow the instructions in the setup dialogs. This will install the necessary system components to work with NVIDIA NIMs on your system.

You will need to reboot your computer to complete the installation.

Git is required and should be installed using winget from a command prompt::

winget install --id Git.Git 

Install Microsoft Visual C++ 2015-2022 Redistributable Package
https://aka.ms/vs/17/release/vc_redist.x64.exe or

winget install Microsoft.VCRedist.2015+.x64

This blueprint requires the installation of Blender. The blueprint has been tested with the Blender 4.27 LTS (Long Term Support) build.
https://www.blender.org/download/release/Blender4.2/blender-4.2.7-windows-x64.msi

Blender 4.27 can also be installed using winget from a command prompt:

winget install --id 9NW1B444LDLW

Open and command prompt and type the following:

set HF_TOKEN

If a value is shown for the HF_TOKEN like in the image below you can skip the steps for obtaining a HuggingFace API Access Token and proceed to Installing the Blueprint

If the command returns that the environment variable HF_TOKEN is not defined, complete the steps below.

Create a user account at https://huggingface.co/ or login to your existing account.
Select the “Settings” option on the left-hand menu.

From the Left-Hand Menu select Access Tokens

Create a new Access Token with READ permissions. (Note: If you have an existing Access Token with read permissions you may use it instead of creating a new access token)

Copy your access token.

Open a command prompt and issue the following command:

hf_access_token represents the actual access token value you created in the step above.

Once you have generated an access token you’ll need to agree to the FluxDev Non-Commercial License Agreement and acknowledge the Acceptable Use Policy by visiting: https://huggingface.co/black-forest-labs/FLUX.1-dev

Click the Agree and access repository button.

Repeat the above process to accept the license for the ONNX variant: https://huggingface.co/black-forest-labs/FLUX.1-dev-onnx

Open a command prompt at the location where you would like the Blueprint files to be installed.

Download the Blueprint using Git.

git clone https://github.com/NVIDIA-AI-Blueprints/3d-guided-genai-rtx.git

Then

cd 3d-guided-genai-rtx

From 3d-guided-genai-rtx folder, run Setup.bat (It is recommended to run this batch file from the command line)

The setup installer will install ComfyUI, the ComfyUI plugin for Blender, and other components required for the blueprint.

Installation will take up to 20 minutes depending on download speed.
Once complete the installation will list the ComfyUI Path, and Python Path, this information will be used to set up the Blender ComfyUI add-on.

Once installation is complete start Blender and press open Preferences from the menu: Edit >>Preferences

Select the Add-On section , and click the checkbox next to ComfyUI BlenderAI node.
Expand the ComfyUI BlenderAI node section by clicking on the >

In the ComfyUI Path and the Python Path configuration section, input the paths shown at the end of the blueprint installation process. Alternatively, you can click the folder icon and navigate to the installation location and select the ComfyUI folder, and the python_embedded folder in the ComfyUI installation.

From the Blender menu select File >> Open

Navigate to Documents >> Blender
Select the Guided_GenAI_BP.blend file

If necessary expand the panel in the upper left viewport by clicking on the < indicator. Alternatively move the mouse into the upper left viewport and press the “n” key on the keyboard.

Select the ComfyUI X Blender tab if needed. Click the Launch/Connect to ComfyUI button to start a local ComfyUI server instance.

It may take up to two minutes to start the ComfyUI service and establish a connection.

NOTE: The Blender system console can be opened from the Blender Menu selection Window >> Toggle System Console. The system console can help provide additional information about the ComfyUI startup process and provide updates while ComfyUI tasks are running.

Once ComfyUi has started and is ready the panel will change and a Run button will appear.

If the Run button does not appear or the Launch/Connect to ComfyUI reappears, check the system console for any error messages.

Click the Run button.

The first time the FLUX NIM is utilized it will need to download models from NVIDIA NGC and setup the FLUX NIM container, this process can take up to 20 minutes or more depending on the connection speed. Note that if it takes >30 minutes to run, it may time out and give an error. If so, you'll need to run it a second time, which should succeed.

There will be an initial connection delay when first connecting to the NIM during a session which may take between 2-5 minutes.

By default the sample workflow will use the viewport scene combined with the following prompt to generate an image that matches both the overall look of the 3D scene, and the text prompt:
“a stunning professional photo of a quaint village in the mountains in winter, a snow covered fountain is visible in the center of the town square”

You can change the output, by changing either the text prompt, the 3D viewport information or both. NOTE: When generating output, some parameter must be changed before it’s possible to generate a new output, either the 3D scene information, prompt, or some parameter. If nothing has been changed the workflow will not process a new image.

The ComfyUI Connector panel is linked to the Input Text Node, you can change the prompt information here.

In the prompt input area, add some additional information to the end of the existing text to change the output, for example try any of the following:
“At sunset”
“At night”
“In the rain”

With the mouse in the upper left viewport press SHIFT + ~ to enter navigation mode. You can fly though the scene using the WASD keys and using the mouse to point in a direction. The E and F keys raise and lower the camera. Navigate the scene to find different camera angles.

Click on the fountain object and click delete on the keyboard to remove the fountain.

In the lower left area of the screen grab the boat object and drag it into the upper left viewport to the general location where the fountain was previously.

Replace the entire prompt with one of these:
“A stunning profession photo of a modern luxury boat in the canals of Venice, classical buildings, at sunset”
“An abandoned boat sitting between rows of warehouses”

Change the output path in the SaveImage node to point to a location on your system where you would like to save generated images.

Closing Blender after the FLUX NIM has been loaded may leave the NIM running in the background which can unneccesarily consume GPU and System resources, best practice is to stop the NIM when it is no longer needed, or before exiting Blender.

Steps to stop the NIM:

1. In the node tree, locate the LoadNIMNode node
2. 
3. Switch the operation setting from Start to Stop
4. 
5. Click Run to run the workflow
6. This will cause an error in the NIMFLUXNode as the NIM is no longer available to run the workflow.
7. 

1. If a NIM isn't stopped via the Stop method of the LoadNIMNode it may be necessary to manually stop the NIM via the command console
2. Open a windows command console and type:

3. wsl podman ps -a

4. 
5. This will show the current active NIM containers
6. Note the name of the container FLUX_DEPTH
7. Issue the following command:

8. wsl podman stop FLUX_DEPTH

9. 
10. You can re-run the wsl podman ps -a command to verify that the FLUX_NIM is no longer running.

If errors occur when working with the NIM it may be necessary to restart the ComfyUI Server. To restart ComfyUI, place your mouse cursor in the ComfyUI node graph area and press “N” to display the panel.

Click the icon to stop ComfyUI.
Click the icon again to restart ComfyUI, or click the Launch/Connect to ComfyUI button.

Re-run the workflow.

Problem: NIMSetup.exe continuously requests that you restart to continue the setup process.

Solution: Changing the Windows display language to English will fix this issue. This is a bug and will be fixed in an upcoming patch.

Problem: The ComfyUI workflow times out on the first run, giving this error: {'type': 'TimeoutError', 'message': 'NIM Server did not start within the specified timeout.'}

Solution: If the time to execute a prompt, including NIM setup, takes longer than 30 mins (1800 secs), the NIM node times out so it doesn't wait indefinitely. The only time this should happen is during first run, when it needs to download all the NIM models. If this happens, simply run again - it should no longer time out since it no longer needs to do all the first time run steps again.

Problem: When running Setup.bat you receive an error "tar: LZMA codec is unsupported" "tar: Error exit delayed from previous errors."

Solution: Manually extract the ComfyUI_windows_portable_nvidia.7z file using 7-zip. If necessary install 7-zip using winget install 7-zip Then issue the following command from a command prompt at the location containing setup.bat "C:\Program Files\7-Zip\7z x ComfyUI_windows_portable_nvidia.7z Once the files have successfully extracted, re-run Setup.bat and choose the option to resume the installation.