touchfarm

Control a 3D printer with an attached touch stylus to operate a smartphone screen. Uses OctoPrint for printer communication and camera access. Optional VLM integration for screen-state analysis.

pip install -e .

cp .env.example .env
# Edit .env with your OctoPrint URL, API key, and VLM credentials

Add to ~/.vpype.toml:

[gwrite.profiles.touchfarm]
unit = "mm"
document_start = "G90\nG21\n"
line_start = "G0 Z5.0 F500\nG0 X{x:.3f} Y{y:.3f} F3000\nG0 Z0.0 F500\n"
segment = "G1 X{x:.3f} Y{y:.3f} F1500\n"
line_end = "G0 Z5.0 F500\n"
document_end = "G0 Z5.0 F500\n"

This profile is only needed for PhoneActions.swipe_svg(). Simple taps and linear/chained swipes work without it.

touchfarm
# Opens on http://localhost:8000

In the UI, click Home XY before any calibration.

1. Place your phone face-up under the stylus.
2. Open Phone Registration in the Calibration panel.
3. Choose a mode:
   • PIN pad — reference points are PIN digits 1, 3, and 0
   • Pattern lock — reference points are upper-left, upper-right, bottom-right dots
4. Click Start. The canvas shows the 3 reference points.
5. For each highlighted point: jog the stylus tip to physically contact that point on the phone screen, then click Record Point.
6. After all 3 points, click Finish.

This uses stylus-contact calibration: the nozzle position when the stylus touches each reference point is recorded. The resulting affine transform maps phone mm coordinates to printer coordinates, with the stylus offset implicitly baked in.

Use the Named Coordinates panel to save logical tap targets:

• Jog the stylus to a target location
• Click Record current pos to fill in the X/Y fields
• Enter a name (e.g. PIN_0) and click Save Coord

Named coords are stored in the phone profile JSON and can be used programmatically.

If taps are consistently off by a small amount, use the Refinement panel to apply a rotation and/or translation without re-running the full calibration.

import asyncio
from touchfarm.config import load_config, load_calibration, load_phone_profile
from touchfarm.printer import PrinterClient
from touchfarm.transform import ScreenTransform
from touchfarm.gcode import GCodeBuilder
from touchfarm.phone import PhoneActions
from touchfarm.sequencer import ActionSequencer, TapAction, ChainedSwipeAction, SnapshotCondition, WaitAction
from touchfarm.vlm import VLMClient

async def main():
    config = load_config()
    cal = load_calibration(config.calibration_file)
    profile = load_phone_profile(config.phone_profile)
    transform = ScreenTransform(cal)
    gcode = GCodeBuilder(cal.gcode)

    async with PrinterClient(config) as printer:
        phone = PhoneActions(printer, transform, cal, gcode)
        vlm = VLMClient(config)
        seq = ActionSequencer(phone, vlm, printer)

        coords = profile.coords

        # Tap a named coordinate
        await phone.tap(*coords["PIN_1"])

        # Pattern lock via chained swipe
        await phone.chained_swipe([
            coords["PATTERN_UL"],
            coords["PATTERN_UR"],
            coords["PATTERN_BR"],
        ])

        # Control flow: wake screen if off
        await seq.execute([
            SnapshotCondition(
                prompt="Is the phone screen on and showing content? Reply YES or NO.",
                expected="YES",
                else_actions=[TapAction(x_mm=coords["HOME_BUTTON"][0], y_mm=coords["HOME_BUTTON"][1])],
            ),
            WaitAction(duration_ms=500),
            TapAction(x_mm=coords["PIN_1"][0], y_mm=coords["PIN_1"][1]),
        ])

asyncio.run(main())

Phone profiles live in phone_profiles/*.json. They store:

• Screen physical dimensions (mm) and pixel resolution
• pixels_per_mm for SVG coordinate conversion
• coords: named logical tap targets (mm)

To add a new phone:

{
  "name": "My Phone",
  "screen_width_mm": 67.0,
  "screen_height_mm": 145.0,
  "screen_width_px": 1080,
  "screen_height_px": 2400,
  "pixels_per_mm": 16.12,
  "coords": {}
}

Save as phone_profiles/my_phone.json and set PHONE_PROFILE=my_phone.json in .env.

The server exposes a REST API at http://localhost:8000. Interactive docs at http://localhost:8000/docs.

Key endpoints:

Position tracking: OctoPrint's REST API does not expose printer coordinates. Positions are soft-tracked (updated after each jog/home command). Always home before calibration to establish a known reference frame.

Z axis: Z moves use a slower feedrate (feedrate_z, default 500 mm/min) to prevent crashes. The jog Z buttons in the UI are visually distinct from XY for this reason.

Stylus contact calibration: During phone registration, jog until the stylus tip physically touches each reference point, then record. The transform encodes the stylus-to-nozzle offset implicitly — no separate offset correction is applied during phone actions.