Work in progress. This project is still under active development — things may break or change. If you run into a bug, please open an issue on GitHub Issues.

Convert an uploaded image into a LEGO-style LDraw model (.ldr) with AI-generated building instructions.

Click the image above to watch the demo video.

1. Upload an image from the browser.
2. A vision model describes the image as a text prompt.
3. Shap-E generates a 3D .obj from the prompt.
4. The .obj is voxelized into LEGO-sized bricks and exported as an .ldr file.
5. The .obj and .ldr are previewed live in the browser using Three.js.
6. Optionally, click Generate Instructions to render the LDR model from six angles (via LDView) and send those renders to the vision model, which produces three sections of structured instructions:
   • Structural review — integrity analysis, techniques, and weak points
   • Build guide — step-by-step instructions from first brick to last
   • Parts & sourcing — part IDs, availability, substitutions, and cost estimate
7. Token usage for the instruction generation run is shown at the bottom.

backend/
  main.py                          Flask app and API routes
  shape_server.py                  Local Shap-E model server
  integrations/shape_generator.py  Client for the Shap-E server
  llm_models/models.py             Vision model instances (OpenAI / Ollama)
  services/image_to_text.py        Image-to-prompt and instruction generation
  services/ldr_cam_angles.py       Renders LDR views via LDView for AI input
  services/prompts.py              Prompt templates for each instruction type
  services/obj_to_lego.py          OBJ/STL to LDR orchestration
  services/ldr_parser.py           LDR layer parser
  scripts/obj_to_ldr.py            Mesh voxelization and LDraw writer

frontend/
  lego_front.html                  Browser UI
  images/                          Frontend assets

Generated files are written to (all git-ignored):

• backend/models/ — uploaded images and generated .obj files
• backend/ldr_output/ — exported .ldr files
• backend/views/ — temporary LDR render images for instruction generation
• backend/instructions/ — saved instruction JSON files
• shap_e_model_cache/ — cached Shap-E weights

Two backends are supported, configured in backend/llm_models/models.py and selected in backend/services/image_to_text.py:

To switch, edit backend/services/image_to_text.py line 30:

self.model_name = model_openai    # OpenAI API (default)
# self.model_name = model_ollama  # Ollama (local)

• Python 3.10+
• Python packages in requirements.txt
• LDView — required for instruction generation (renders the LDR model from multiple angles). Install from ldview.sourceforge.net and ensure LDView64 is on your PATH.
• One of:
  • OpenAI API access with an API key (default)
  • Ollama running locally with qwen3-vl:8b pulled

Install Python dependencies:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt

PyTorch is included in requirements.txt. GPU users may want to install the CUDA-specific wheel from pytorch.org before the rest of the packages.

Pull the Ollama vision model:

ollama pull qwen3-vl:8b

Create a .env file in the project root:

FLASK_PORT=5000
SHAPE_API_URL=http://127.0.0.1:8000/

For the OpenAI API, add your API key:

OPENAI_API_KEY=your_api_key

PYTHON_PATH is optional. If omitted, the backend uses the current Python interpreter.

PYTHON_PATH=C:\path\to\python.exe

Start the Shap-E model server first:

python backend\shape_server.py

In a second terminal, start the Flask backend:

python backend\main.py

Open the app at http://127.0.0.1:5000/, upload an image, and wait for the 3D viewers to load. Then click Generate Instructions to produce the full build guide.

GET /health

Returns { "status": "ok" }.

POST /api/image-to-lego

Multipart form fields:

Returns the generated prompt, OBJ file info, and LDR file info including filepath_ldr (needed for instruction generation).

POST /api/ldr-to-instructions

Body:

{
  "ldr_file": "C:\\...\\backend\\ldr_output\\<model_id>.ldr",
  "original_description": "optional prompt text from image-to-lego"
}

Renders the LDR model from six angles using LDView, then sends the renders to the vision model three times (once per instruction type). Returns:

{
  "instructions": {
    "engineering": "...",
    "building": "...",
    "style": "...",
    "original": "...",
    "token_usage": {
      "prompt_tokens": 0,
      "completion_tokens": 0,
      "total_tokens": 0
    }
  }
}

GET /ldr/layers?file=<filename>.ldr
GET /ldr/layers/<layer_num>?file=<filename>.ldr

Parses an exported .ldr file into layer data.

GET /models/<filename>
GET /ldr_output/<filename>
GET /instructions/<filename>

Run the converter script directly without starting the server:

python backend\scripts\obj_to_ldr.py backend\models\model.obj backend\ldr_output\model.ldr 64

• Higher resolution values produce more detailed models but take longer and generate more bricks.
• The Shap-E server runs on CPU but is much faster with a CUDA GPU.
• Output quality depends on the image description produced by the vision model and the geometry generated by Shap-E.
• Instruction generation requires LDView to be installed and on PATH.
• The project exports LDraw data, not official LEGO building instructions.