Sana connects a conversational check-up call, a personalized walking-assessment link, and a clinician dashboard. Built as a hackathon prototype for orthopedic and stroke mobility follow-up, it keeps survey responses and walking results linked to the same patient and call.
Prototype only: use synthetic patients and consenting test recipients. Survey scores, gait risk estimates, and AI summaries are not validated clinical decisions. This repository does not establish HIPAA compliance or readiness to store real patient data.
- Start a check-up: a clinician signs in at
/doctor, selects an existing patient and condition, enters a phone number, and clicks Start call. Starting the servers alone never places a call. - Talk naturally: Twilio carries the call; Deepgram handles speech recognition and voice output. Configured orthopedic calls start with the bundled recorded doctor greeting. The helper then asks six fixed questions and detects the end of a spoken turn without a button press.
- Keep answers trustworthy: OpenAI can interpret conversational replies into the survey's allowed options. A clear selection is accepted directly; an inferred answer or relative correction is proposed for patient agreement. Questions and options remain application-controlled. Short acknowledgments can vary within the companion's language guardrails.
- Continue through uncertainty: after three unsuccessful clarifications, or repeated silence on an active question, the helper leaves the answer blank and continues. The submitted survey includes the recognized transcript, unanswered-question metadata, and a Needs human review flag. An incomplete survey receives no total score. Stop requests and overall call limits still apply.
- Offer a walking link: after the survey, the helper asks permission to send a text. The backend saves the survey; with consent, the phone service texts an expiring signed patient link to the call's destination. Declining the link still saves the survey, without sending an SMS.
- Capture and review: the link opens a simple camera/video-upload screen, without scores or a skeleton overlay. Saved walking results carry the same patient, call, and attempt IDs. The clinician can review surveys, transcripts, missing answers, call status, and gait results in the portal and its authenticated, read-only Database tab.
The current phone flow asks the condition survey directly. Generic intake fields remain in the API contract but are sent as unknown/null; they are not inferred from condition answers. Transcripts are stored with survey submission, not continuously backed up during the call. Interrupted calls may therefore lack a saved survey or transcript.
Gait processing uses MediaPipe/OpenCV and joint telemetry to estimate stride
length, asymmetry, velocity degradation, cadence, knee range of motion, peak
ankle speed, and a prototype leg-length-normalized fall-risk indicator. Missing
landmarks can be interpolated; dropped_frame_pct reports repaired frames and
clips missing more than half their frames are rejected.
See docs/architecture.md for the full system overview: the patient client, voice-agent intake, doctor's portal, the end-to-end data workflow, the complete endpoint reference, and which pieces are implemented today.
- Node.js 20.19.x (the repository's
.nvmrcselects 20.19.0). - npm 10 or newer.
- Python 3.12 (the pinned MediaPipe build does not install on Python 3.13).
The repository's .python-version selects Python 3.12 for version managers
that support it.
Sana/
backend/
main.py FastAPI app and route definitions
patient_access.py signed patient-token and link generation/verification
processor.py GaitProcessor: frames -> GaitMetrics
video.py OpenCV + MediaPipe extraction for uploaded video
store.py patient records: surveys, sessions, synthesis
agent.py clinical summary (OpenAI gpt-4o-mini w/ template fallback)
frontend/
app/page.tsx public secure-link instructions
app/patient/[id]/page.tsx signed per-patient screening (webcam / upload)
app/doctor/page.tsx clinician dashboard (live-syncs)
app/components/PatientScreening.tsx patient screening view
app/components/ClinicianDatabase.tsx authenticated record browser
supabase/migrations/ integrated patient-store SQL
docs/ setup, architecture, and data contracts
data/
mock_patients.json seed patients for the doctor's portal
voice_agent/
phone_app.py Twilio + Deepgram outbound phone survey and gait SMS handoff
voice_app.py desktop push-to-talk variant of the same survey
app/ survey engine, conversation policy, telephony bridge
The automated voice agent lives in voice_agent/.
Main runs on 8000, phone on 8001, and the frontend on 3000.
Use separate Python virtualenvs: main pins openai==3.16.1, while voice
requires openai<3. Main starts and tests without any telephony credentials.
The copied voice-results dashboard is available from the optional desktop voice service on 8002 (keep 8000 reserved for main):
(cd voice_agent && .venv/bin/uvicorn voice_app:create_app --factory --host 127.0.0.1 --port 8002 --no-access-log)Open http://127.0.0.1:8002/results and enter the voice service's
OPERATOR_TOKEN. It shows standalone voice-demo answers, scores and transcripts
from Supabase or explicitly labeled local fallback, with search and automatic
refresh. It does not mix integrated phone/gait records into the standalone tables;
review those in http://localhost:3000/doctor. Both interfaces remain protected
by their existing authentication boundaries. Only synthetic data should be used
for local testing. Starting these servers does not place a phone call.
The authenticated clinician explicitly supplies orthopedic or stroke
condition metadata. Main reserves the call; phone collects HOOS JR/stroke items.
Main persists confirmed answers and review metadata in its configured patient
store and issues the signed patient URL. With SMS consent, phone texts that
exact URL, then polls the patient's walking events.
Only a persisted session with matching call/attempt IDs confirms completion.
The clinician timeline and synthesis join that saved result to its survey.
Unknown condition metadata is never inferred from complaints, age, or cohort.
See the integration/operator runbook for setup, the offline deterministic demo, failure recovery and live validation steps. The Sana screens use the shared pastel design system. Patient access still requires the complete signed SMS link; the public landing page does not offer patient-ID lookup. The API/data contract describes all three authentication boundaries and the call/survey/walking payloads. Integrated records can use the separate Supabase patient store or the explicitly configured local JSON demo store.
Clone the upstream repository (or use your own fork), then enter it:
git clone https://github.com/JiyaS12/Sana.git
cd Sanacd backend
python3.12 -m venv .venv
.venv/bin/pip install -r requirements.txtIn a new terminal, from the repository root:
cd backend
cp .env.example .env
# Fill the username/password and generate distinct clinician-session,
# patient-link, and survey-ingest secrets with:
# python -c "import secrets; print(secrets.token_urlsafe(48))"
.venv/bin/uvicorn main:app --port 8000 --env-file .envCopy example files only on first setup; do not overwrite an existing .env.
Keep actual credentials in ignored local environment files, never in the
tracked examples. Run one worker per Python service.
The three clinician values are required. Clinician APIs fail closed with 503
until CLINICIAN_USERNAME, CLINICIAN_PASSWORD, and a random
CLINICIAN_SESSION_SECRET of at least 32 characters are all present. /doctor
exchanges the credentials for a signed, HttpOnly, SameSite=Lax session cookie;
neither the credentials nor signing key belong in a NEXT_PUBLIC_* variable.
Runtime environment variables:
CLINICIAN_USERNAME/CLINICIAN_PASSWORD— local clinician sign-in.CLINICIAN_SESSION_SECRET— random server-only session signing key (minimum 32 characters).CLINICIAN_SESSION_TTL_SECONDS— session lifetime in seconds (default 28800, accepted range 60–86400).CLINICIAN_COOKIE_SECURE— defaults totrue; setfalseonly for explicit local HTTP development.CLINICIAN_LOGIN_MAX_ATTEMPTS/CLINICIAN_LOGIN_WINDOW_SECONDS— per-process, per-client login throttle (defaults 5 attempts per 60 seconds).CORS_ALLOWED_ORIGINS— exact comma-separated frontend origins. Localhost and 127.0.0.1 on port 3000 are the development defaults.*is rejected because clinician sessions use credentialed requests.
Required patient-link configuration:
PATIENT_LINK_SIGNING_SECRET— a deployment secret containing at least 32 bytes. Survey ingestion fails closed when it is absent or too short.PATIENT_APP_BASE_URL— public patient-app base URL used to construct the complete signed link (defaulthttp://localhost:3000; non-local URLs must use HTTPS).PATIENT_LINK_TTL_SECONDS— link lifetime from 60 to 604800 seconds (default900).
Service authentication and optional environment variables:
OPENAI_API_KEY— enable LLM-generated summaries/synthesis (deterministic template fallback without it). Summaries are cached inbackend/.cache/summaries.json;GET /api/summary-cache-statsreports cache stats.SURVEY_INGEST_TOKEN— required forPOST /api/submit-survey; callers send it inX-Survey-Token. It also disablesPOST /api/patients/{id}/ensure-demo(403) unlessALLOW_DEMO_PATIENTSis set.PHONE_SERVICE_BASE_URL— optional phone origin (localhttp://127.0.0.1:8001).OPERATOR_TOKENmust match the phone service's server-only bearer token.ALLOW_UNAUTHENTICATED_SURVEY_INGEST— local-only escape hatch. Set to1/true/yesto run survey ingestion without a token. Never enable this in a shared or production environment.ALLOW_DEMO_PATIENTS— local-only support for the clinician-authenticated demo-record endpoint. It does not make/patient/[id]public or bypass a signed patient link.
See the patient access contract for stable request/response examples used by the patient UI and voice/SMS integrations.
From the repository root:
cd frontend
npm ci
cp .env.example .env.local
npm run devThe app runs at http://localhost:3000. For same-origin API requests, configure
frontend/.env.local as follows and restart Next.js:
NEXT_PUBLIC_API_URL=
API_PROXY_TARGET=http://127.0.0.1:8000The intentionally empty public URL makes the browser call /api/* on the
frontend's own origin; Next.js proxies those requests to the backend. The
default direct-API alternative is NEXT_PUBLIC_API_URL=http://localhost:8000.
Never put credentials or access tokens in NEXT_PUBLIC_* variables. For local
HTTP testing set CLINICIAN_COOKIE_SECURE=false in the backend; use true
with public HTTPS and configure the exact frontend CORS origin.
From the repository root, create the separate phone environment:
cd voice_agent
python3.12 -m venv .venv
.venv/bin/pip install -r requirements.txt
cp .env.example .envConfigure voice_agent/.env with TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN,
TWILIO_FROM_NUMBER, and DEEPGRAM_API_KEY. For conversational interpretation,
set SURVEY_EXTRACTOR=openai and OPENAI_API_KEY; SURVEY_EXTRACTOR=exact
supports offline exact-option handling instead. The backend's OpenAI key is
separate configuration for optional clinician summaries.
Match these service settings (tokens are private, independently generated secrets, not the literal placeholders below):
backend/.env |
voice_agent/.env |
|---|---|
PHONE_SERVICE_BASE_URL=http://127.0.0.1:8001 |
MAIN_BACKEND_URL=http://127.0.0.1:8000 |
OPERATOR_TOKEN=<operator-secret> |
OPERATOR_TOKEN=<same-operator-secret> |
SURVEY_INGEST_TOKEN=<ingest-secret> |
SURVEY_INGEST_TOKEN=<same-ingest-secret> |
Keep PATIENT_LINK_SIGNING_SECRET only in the backend. The recorded greeting is
controlled by DOCTOR_GREETING_AUDIO and DOCTOR_GREETING_CONDITIONS on the
phone service; an explicitly empty audio value disables it.
Twilio needs to reach the phone service, and a patient's phone needs to reach
the website. With cloudflared installed, run these in two separate terminals:
cloudflared tunnel --url http://127.0.0.1:8001cloudflared tunnel --url http://127.0.0.1:3000Use the actual URLs printed by those processes:
| Environment file | Setting |
|---|---|
voice_agent/.env |
PUBLIC_BASE_URL=https://<phone-tunnel-host> |
backend/.env |
PATIENT_APP_BASE_URL=https://<website-tunnel-host> |
backend/.env |
CORS_ALLOWED_ORIGINS=https://<website-tunnel-host> |
backend/.env |
CLINICIAN_COOKIE_SECURE=true |
frontend/.env.local |
ALLOWED_DEV_ORIGINS=<website-tunnel-host> (hostname only) |
Use the frontend proxy settings above; the backend does not need its own public tunnel in this arrangement. Keep both tunnel terminals and all three services running. If tunnel URLs change, update the settings and restart the services; previously sent links pointing at an old tunnel will not work.
After configuration, run these in three separate terminals from the repository root (or restart existing instances; do not start duplicate servers):
(cd backend && .venv/bin/uvicorn main:app --host 127.0.0.1 --port 8000 --env-file .env --no-access-log)
(cd voice_agent && .venv/bin/uvicorn phone_app:create_app --factory --host 127.0.0.1 --port 8001 --env-file .env --no-access-log)
(cd frontend && npm run dev)Check backend /api/health and phone /api/config, then open /doctor on the
website URL. A readiness response checks configuration, not actual provider
delivery. Only click Start call for a number you control or have permission
to contact; real calls, texts, and model requests may incur provider charges.
Follow the Supabase setup guide to apply
supabase/migrations/20260920080000_integrated_patient_records.sql to your
project (the demo uses HackMIT2026). Set these in backend/.env:
PATIENT_STORE=supabase
SUPABASE_URL=https://<project-ref>.supabase.co
SUPABASE_SERVICE_ROLE_KEY=<server-only-key>
SUPABASE_SEED_DEMO=falseThe new store starts empty. For a deliberate synthetic-data demo, the guide
explains one-time seeding; existing local or standalone voice data is not
automatically imported. Supabase failures do not silently fall back to JSON.
For local-only storage, explicitly set PATIENT_STORE=json.
In Doctor portal → Database, search patients and inspect calls, destination numbers, survey answers, review flags, transcripts, and saved gait measurements. The same clinician sign-in protects this read-only view; no Supabase dashboard login is needed. It does not offer arbitrary SQL, editing, or deletion.
walking_patient_records stores the linked patient aggregate;
walking_assessment_results provides a survey/walking review view. Patient ID
is the permanent identity; the phone destination stays private on its call
record, not in the patient URL. Review/transcript fields use the existing JSON
record and need no additional migration. Signed links are bearer credentials:
anyone holding a valid link can use it until it expires, so do not share or log
complete links. Audio and raw video recordings are not stored by this flow.
The public home page never lists patients, accepts arbitrary IDs, or creates demo records. A local demo uses the same signed handshake as production:
-
In the backend's local
.env, set a randomPATIENT_LINK_SIGNING_SECRET, setSURVEY_INGEST_TOKEN=local-demo-only, and leavePATIENT_APP_BASE_URL=http://localhost:3000. -
Start the backend and frontend development servers.
-
Submit a local intake and copy the returned
patient_url(treat it as a credential and do not paste it into shared logs):curl -sS http://localhost:8000/api/submit-survey \ -H 'Content-Type: application/json' \ -H 'X-Survey-Token: local-demo-only' \ -d '{"patient_id":"RGN-0417","patient_name":"Local Demo Patient","pain_scale":3,"fall_history":{"falls_last_6_months":0,"injured":false},"dizziness":false,"primary_complaints":["local demo"]}'
-
Open the complete returned URL. A bare
/patient/RGN-0417path is expected to fail. Complete a Live Camera or Upload Video walk; only a detected walk can be saved. -
Open
/doctordirectly, sign in with the backend-configured clinician credentials, and confirm the saved session appears in that patient's timeline.
From the repository root, use separate Python 3.12 environments:
python3.12 -m venv backend/.venv
backend/.venv/bin/pip install -r backend/requirements.txt
OPENAI_API_KEY= backend/.venv/bin/python -m pytest -q backend/tests
python3.12 -m venv voice_agent/.venv
voice_agent/.venv/bin/pip install -r voice_agent/requirements.txt
(cd voice_agent && OPENAI_API_KEY= RUN_LIVE_SURVEY_TESTS= SURVEY_EXTRACTOR=exact \
.venv/bin/python -m pytest -q -m 'not live')
node --test voice_agent/tests/*.cjs
mkdir -p .cache
backend/.venv/bin/python -m pytest -q tests/integration/test_handshake.py --basetemp=.cache/handshakeThen use Node 20.19.x to install exactly the locked frontend dependencies, audit production packages, lint, type-check, and build:
cd frontend
npm ci
npm audit --omit=dev --audit-level=high
npm test
npm run lint
npm run type-check
npm run buildThese checks require no repository secrets. The handshake starts real main and phone HTTP processes on ephemeral loopback ports, using their separate virtualenvs, fake telephony and synthetic gait frames. It covers both conditions, unknown generic answers, SMS failure/ambiguity, retries, event ordering, scoped authorization and saved-session synthesis without cameras or paid providers. GitHub Actions preserves backend/frontend gates and adds offline voice Python, voice JavaScript and the integrated handshake.
MIT — see LICENSE.