Skip to content

Home

About me

So AI can be safer, I evaluate its performance. One cool way to do this is to figure out the hidden story inside a human expert's mind when they evaluate performance in their field.

Knowledge-dissection projects

  • τ-Discernment — What does a user latently require before an agent takes a consequential action? I lift each user's implicit requirement, buried in task prose, into a typed, provenance-grounded object a grader can score (UserPreflightRequirements) — surfacing failures a task-completion grader misses. github.com/borisdev/tau-discernment
  • NoBSmed — What makes a medication or supplement recommendation questionable given the evidence, missing evidence, and relevance to the patient?
  • HealthBench audit — What makes a "doctor-approved" answer or grading rubric unreliable? I found fabricated citations and 29 possible patient-harm issues in OpenAI's medical-AI benchmark family. github.com/borisdev/nobsmed-healthbench-audit
  • HealthBench EBM failure patterns — Where do frontier models fail on evidence-based medicine, and which failures are worth fixing? I ran GPT-5.2 and Claude Opus 4.8 on an EBM-verified slice of HealthBench and clustered the errors into seven canonical recall/precision failure patterns — each with anchored examples, a root-cause hypothesis, and a distinguishing experiment. github.com/borisdev/healthbench-ebm-verified
  • Muni-Resilience-Bench — How much does a specific resilience project (wildfire hardening, flood control, seismic retrofit) lower a city's borrowing cost? An open benchmark for the missing, auditable translation from a project's risk reduction to citywide fiscal impact. github.com/borisdev/muni-resilience-bench
  • Wolf Games — What makes an AI-generated murder-mystery storyboard coherent across interpolated scenes?
  • Inequality as a Graph (PhD thesis) — What makes an inequality metric capture nuanced gaps among social groups in the same city? Gini and variance silently weight every pair of groups the same. I lifted that hidden assumption into an explicit weights matrix — a graph whose edges say which gaps count, and how much — so different concerns imply different metrics. A Rawlsian asks whether institutions distribute justly, so the heavy edges run between racial neighborhoods and school districts. Sen asks who is deprived of capabilities, so the heavy edges run between occupational groups — the entitlement collapse that drove the Bengal famine. The same change can score in opposite directions under the two. Essay · Full dissertation (PDF)
  • SimpleLegal — What makes "Called the State Senator" lawyer-level work rather than an administrative task? The classification depended on the relevance of the output, not the wording of the task.

Selected technical impact

  • Audited OpenAI's HealthBench medical-AI benchmark family — surfaced fabricated citations and 29 possible patient-harm issues: github.com/borisdev/nobsmed-healthbench-audit
  • Built an AI agent-evaluation framework for Sindri.ai using Temporal.
  • Helped relaunch a stalled legal-billing AI feature by eliciting lawyer expertise and redesigning the rubric for 11 billing flags.
  • Built story-scene prediction for a gaming startup led by a Law & Order producer.
  • Contributed an experimental causal-graph agent to LangChain: langchain-ai/langchain#6255
  • Built backend systems for factory analytics, people analytics, and Tableau geospatial services.
  • Embedded subjective concerns into statistical analysis of income inequality in my social-science PhD: escholarship.org/content/qt8br7d5df

Non-tech fun points

  • Climbed Cotopaxi (about 19,300 ft).
  • Survived bodyboarding Mexpipe.
  • Taught geospatial data to students in Medellín, Colombia.
  • Taught kids snowboarding.
  • Managed an international restaurant team.
  • Counseled severely emotionally disturbed children.

Elsewhere

Writing

Essays on the patterns I've seen recur. Some are primarily a way to organize my own learning; I'm not an expert in everything I write about — especially the compiler-design piece.

# Title Topic
1 Beyond RAG: How Chomsky's I-Language and Compiler Design Converge on Knowledge Graphs LLVM-style IR, Chomsky's I-language, BFO ontology, grammar-first design
2 What Is Knowledge Engineering, Really? A working definition built around elicitation, evaluation, and 0→1 modeling in messy domains
3 Fine-Tuning LLMs Will Restructure Your Data Science Team How fine-tuning replaces annotation pipelines and the NN-optimization role; the new "fine-tuning analyst"
4 Why Domain-Specific Language AI Features Fail The customer-discovery process for niche language AI, and why a Lean Startup approach is required
5 Language AI Evaluation 101: Know Your User Why simplistic Ground Truth produces misleading accuracy metrics; cognitive empathy as the iteration loop
6 Hyper-Local Community Funding: A DAO Alternative to CDFIs Local digital tokens and DAOs as a delivery mechanism for under-served-neighborhood capital
7 Inequality with a Spatial View A note from my 2014 dissertation: the same income data can read as inequality going down or up depending on whether you keep the spatial structure. A spatial view is a graph
8 CV: Knowledge Engineering in Messy Domains The IR-compile pattern across clinical trials, legal billing, maritime construction, narrative gaming, and geographic inequality