I build LLM systems and measure them.

handoff-graphOct 2026

Agent orchestration · Python · LangGraph · FastAPI · Claude

Question: when an agent has to follow rules, where should they live: in the prompt, in the shape of the graph, or in a guard that checks every tool call?

A card-support agent in LangGraph: intent routing, context carried across turns by the checkpointer, an identity-verification gate, a guard that checks each proposed tool call against the policy, and a human handoff that pauses with interrupt() and files the ticket only after a person accepts. It's graded on what happened to the accounts, plus an independent audit of the action log that doesn't trust the guard.

Findings, on scripted planners with known behaviour: with a careful planner every design scores 40/40, so a demo can't tell them apart. With an eager one, the full graph still scored 40/40 because the guard stopped eight actions, while taking out the gate and the guard let it break policy in 12 of 40 conversations.

voice-latencyOct 2026

Voice agents · Python · Vosk streaming STT · TTS · voice activity detection

Question: in a voice agent, where does the time go between the caller going quiet and the agent starting to answer?

48 recorded requests (12 card-support questions in US, UK, Australian and Indian English) run through an energy endpointer, streaming Vosk recognition, a reply step with a swept time to first token (or Claude, streamed), and local text-to-speech, all offline on a 2020 laptop CPU.

Findings: the silence threshold, not the model, sets the floor. 300 and 500 ms cut off every caller who paused mid-sentence; only 1200 ms caught them all, and it adds over a second to every turn. Recognition left about 30 ms of work once the turn ended, but its error rate was 2.1% on the US voice and about 11% on the other three. Then the speech engine spent about 1.75 s starting up on every reply, so keeping it loaded would save three times what the faster model in the sweep did.

lora-intentsOct 2026

Fine-tuning · Python · PyTorch · Transformers · PEFT

Question: how many labelled examples does a LoRA adapter need before it beats the cheap options?

Learning curves on Banking77 (77 banking intents) from 5 examples per intent to all 10,003: TF-IDF, a frozen MiniLM sentence encoder with logistic regression, a rank-8 LoRA adapter, and a full fine-tune, in plain PyTorch on a laptop CPU. All the MiniLM methods read the same mean-pooled embedding, so only the way the backbone is updated differs.

Findings: the frozen encoder matched or beat LoRA at every data size and trained in minutes rather than half an hour. A full fine-tune only edged ahead with all the data (93.2% vs 93.0%, one seed). For an intent router, start frozen and fine-tune only when the embedding is the problem.

agentcheck

Agent evaluation · Python · Anthropic SDK

Question: when you grade a tool-using agent from its conversation, what do you miss?

A deterministic back-office world with seven tools and 36 tasks. Every run is graded on the end state of the world, a zero-tolerance policy check over the action log, and the facts in the answer. The environment enforces physics but not policy, so violations happen and get caught rather than being prevented and hidden.

Findings, on scripted reference agents with known behaviour: a grader that reads only the transcript agreed with ground truth worse than chance on the agent that skips checks, passing 16 runs that broke policy and failing every correct refusal. And an agent that looks 96% reliable per attempt gets all eight attempts right on only 75% of tasks. Real-model runs plug in through a Claude adapter on the Anthropic SDK.

Solquara: a production AI platform, built and run alone

Founder & lead engineer · PHP, JavaScript, WebGL, Python · Jul 2026 – present

An AI video-ad and automation studio at solquara.com. I'm the only engineer: a 40-route site, an accounts, plans and entitlements system with signed billing webhooks, a generative-media pipeline, and an MCP server that lets Claude drive Gemini, Veo and TTS.

The work I'm proudest of is the testing: 125 end-to-end checks covering IDOR, sessions, throttling and every subscription state, run locally on real PHP 8.3 and 112 of them against production through a self-expiring harness. Production found two traps no local run could: a host cache serving each user a stale plan, and a security plugin locking out IPs. Both are handled and documented.

pg-hybrid-search

Retrieval · TypeScript · PostgreSQL · pgvector · Docker

Question: can Postgres alone do hybrid search well, and what does it take?

Full-text, BM25 and pgvector retrieval fused with reciprocal rank fusion in a single SQL statement, benchmarked on BEIR SciFact (300 expert-judged claims, 5,183 abstracts). The same tests run on PGlite (Postgres 18 in WASM) and on a real pgvector server in CI, which caught two portability bugs the in-process run could not show.

Findings: hybrid search only helped with real BM25. Built on Postgres's own ts_rank_cd, which has no IDF, fusion made the top 10 worse than vectors alone. And pgvector's default ef_search silently returned 40 rows when asked for 100.

rag-document-qaCloud Run · Oct 2026

RAG service · Python · FastAPI · Terraform · Google Cloud Run

Question: can a RAG service say "I don't know" honestly, and what does it take to put it on the internet?

Citation-grounded Q&A over a folder of documents. A similarity threshold is the usual way to refuse unanswerable questions. Measured here, it can't work: an off-topic question scored higher than a relevant one. A vocabulary-coverage guard separates them cleanly, and a test fails if a future embedder makes the simpler guard viable again.

It ships ready for Cloud Run: Terraform for the service, Artifact Registry and Secret Manager, a non-root container, and a GitHub Actions deploy job that authenticates through Workload Identity Federation, so no cloud key is stored anywhere. Getting it ready for the internet turned up a hole in its own API: the indexing endpoint would read any path on the server. It's now off unless an index root is configured, paths can't leave that root, and CI checks it on every push.