JH
Jonathan Haber
Architecture· 20 min read

Building a System That Cares: The Technical Architecture of AI Coaching at Scale

How a solo-built production stack encodes specific beliefs about human development into every schema, endpoint, and prompt template.

Every technical architecture is a theory of its domain. In AI coaching, the domain is human development — and the architecture's job is to encode assumptions about how people grow, what growth requires from an AI system, and where that system should refuse to pretend it knows more than it does.

Next AI Labs · Human-Agent Research

The Schema Is the Philosophy

Consider a single field in a database document: a memory type classifier. It appears in the per-user knowledge store, where insights extracted from coaching sessions are persisted as individual records with confidence scores, relevance rankings, and vector embeddings for similarity search.

The field distinguishes kinds of knowledge such as what a person prefers, what resonates with them, what they resist, and what they believe. Each category represents a specific kind of knowledge the system can hold about a human being. The taxonomy is guidance rather than a closed list, open to extension, because the designers expect to discover new categories of human insight as the system encounters more people.

That single field contains an architectural thesis: a coaching system needs typed knowledge about its users, not just transcripts. A transcript is a record of what was said. A memory typed as a resistance — "not interested in meditation-based approaches, rejected guided visualization" — is a record of what was understood. The difference between those two is the difference between a system that stores conversations and a system that accumulates coaching intelligence. Concept Atomization

This article is about an architecture where decisions like that one — the choice to type knowledge, to embed it for similarity retrieval, to score its relevance and confidence separately — are not implementation details. They are design claims about what AI coaching requires. Every layer of the stack makes claims like this. The question is whether the claims are coherent, whether they serve the human on the other end, and whether they compose into something that gets better over time.


The Stack as Theory of Coaching

Frontend: The Coaching Conversation Surface

The React application handles the full user journey — intake flows, coaching sessions, vision capture, goal management, checkout, and an admin tooling layer for system calibration. But the architecturally significant choice is not what the frontend does. It is what it does not do.

The frontend does not contain coaching logic. It does not decide what the AI should say, how it should frame a question, or when it should stop talking. It is a conversation surface — a place where the human speaks and the system's response arrives. The coaching intelligence lives entirely in the backend and AI layer.

This separation is deliberate. Coaching quality is a backend concern because it requires context that the frontend cannot hold: the user's full session history, their goal trajectory over weeks, their memory vectors, their funnel stage, their engagement patterns. Putting coaching decisions in the frontend would mean making quality-affecting choices with partial information. The architecture says: the thing that talks to the human must have access to everything the system knows about that human.

Backend: The 2,500-Line Reality

The Node.js/Express API is the operational core. The authentication controller alone spans 2,500+ lines — not because of poor design, but because authentication in a coaching system carries more weight than in most applications. Account recovery helps users identify their accounts without exposing full credentials, account lookups are built so that failures do not reveal which accounts exist, and calls to the external auth provider tolerate transient failures instead of surfacing them to the person.

These are not authentication novelties. They are expressions of a specific priority: users who are in the middle of a coaching journey and cannot log in are experiencing a trust violation. The system promised to be a reliable presence in their development. Locked-out users do not perceive an auth bug — they perceive abandonment. The defensive engineering serves the coaching relationship, not just the login flow.

The billing layer follows the same principle. Subscription access is designed to fail open: a paying user in a trial period must still see full-quality coaching, so the access handler corrects for the cases where the plan record does not yet reflect a real subscription. The architecture encodes a specific belief: it is worse to degrade a paying user's experience than to occasionally give a non-paying user something for free.

Data Layer: Typed Knowledge, Not Just Storage

MongoDB holds user models, conversation history, goals, and two knowledge systems that are architecturally distinctive.

Client Memory Vectors store modular insights extracted from coaching conversations. Each record is a single assertion — "prefers visual learning over auditory," "struggling with consistency in morning routine," "said 'I always feel like I'm not enough' — core limiting belief" — carrying what kind of knowledge it is, how confident the system is in it, how relevant it is, and when and why it was extracted ("During discussion about work-life balance"). A memory can be retired without being deleted — a human changes, and the system's knowledge about them should change too.

The schema design embeds several coaching principles. Exact client quotes are preserved as their own kind of memory, because a coach who paraphrases loses the precise language that reveals how the person frames their own experience. What the user rejected, refused, or pushed back on is tracked alongside what resonates, because understanding what does not work for someone is as diagnostically valuable as understanding what does. Relevance scoring lets the system surface the most important memories first when assembling context for a coaching response, rather than treating all knowledge as equally weighted.

Knowledge Atoms are a second knowledge system, but they operate at the institutional level rather than the individual user level. Each atom is a typed assertion — a correction, a mental model, a finding, a decision, an open question — with provenance tracking, a record of how firmly it is known, and a correction chain that allows new atoms to supersede old ones. Concept Atomization

The correction chain is worth examining. When a new atom is saved with a reference to older atoms it supersedes, the system automatically marks the superseded atoms as deprecated and records the lineage. This creates a traceable history of how the system's understanding evolved. Correction-type atoms carry two additional fields that no other type has: what was believed before, and why the wrong belief seemed plausible. These fields exist because the system's designers believe that understanding why a mistake was made is more valuable than simply recording the correct answer.

Atoms get identifiers and human-readable labels automatically, and the design never blocks a save on bookkeeping like that. Knowledge preservation takes priority over indexing elegance.

AI Layer: Multi-Model Orchestration With Strategy Resolution

The coaching engine orchestrates across multiple LLM providers and models. A model strategy module defines named presets for different coaching situations, and a resolution function combines the chosen preset, any adjustments a user is allowed to make (paying users can tune reasoning depth and summary preferences), and the requested model into one fully resolved configuration.

The engineering details here serve a coaching concern. Some models spend tokens reasoning before they write, so a budget that works for one model can produce an empty reply on another. The resolver accounts for this per model, so a person never receives a response where the thinking used up the room for the answer.

The reasoning-effort setting is model-aware: if a user requests more reasoning than a model supports, the strategy resolver quietly brings it down to what the model can do and logs the adjustment. The user never sees a cryptic API error. They get the best available reasoning for their model.

The entire strategy resolution is wrapped in a fallback that returns safe defaults. The error logging includes a UX impact description — "A user starting a coaching session could get a suboptimal model or default settings because strategy resolution failed" — and expected-vs-actual behavior annotations. This is not standard error handling. It is error handling designed for a system where the person on the other end of the failure is a human who came for coaching, not a developer who can debug a stack trace. Zero Touch Observability


The Agentic Orchestration: A System That Learns How to Coach Better

The most architecturally distinctive layer is not the coaching stack itself but the meta-system that wraps it — a constellation of knowledge stores, search indexes, and agent tools that accumulate institutional knowledge about how the coaching system should operate. Meta Learning System

Swarm Memory

The agent swarm maintains 533+ session artifacts of operational knowledge — errors encountered, patterns discovered, solutions validated, lessons learned — accumulated across development sessions. Each entry is timestamped, tagged by repository and event type, and indexed for semantic search. When an agent begins working on a task, it searches this corpus to find relevant prior art: has this error been seen before? Was this approach tried and abandoned? Is there a known constraint that the current plan would violate?

The Intent Database

100+ UX intent statements live in a searchable intent database, each in the format: "When {all constraints in UX terms} we {testable UX statement}." Each intent carries a machine ID (e.g., IX-042) that appears in code annotations, commit messages, and audit logs. The human-readable statement appears in test names and changelogs.

The two-layer ID system is a design response to a specific problem: code comments need to be terse (machine IDs), but test names and changelogs need to be readable (full UX statements). A lookup function resolves the ID to the full statement — meaning any artifact in the system can be traced from a code annotation to a testable claim about user experience. Searchable Intent

Unified Semantic Search

A single query searches eight knowledge sources — operational logs, UX intents, business strategy, registered tools, session artifacts, commits, tasks, and documentation — with source authority weighting and profile-based relevance scoring. The default profile leans most heavily on UX intents, then on tools and API references, with operational logs and documentation counting for somewhat less. Knowledge atoms are weighted higher than general documents but are not a separately indexed source in the search pipeline. Source Authority Weighting

Different search profiles reweight sources for different contexts. A strategy-focused profile boosts business context to dominate results. A founder-focused profile elevates institutional memory and past decisions. The weighting system means the same query produces differently prioritized results depending on who is asking and why.

The output is structured into distinct sections — ranked results, session artifacts, commits, documentation, operational history, strategy entries, and matched tools — each with its own relevance threshold and deduplication logic. The design prevents the most common failure mode of multi-source search: the user drowning in loosely relevant results from every source, unable to identify the one document that actually matters. Unified Knowledge Retrieval

The Canonical Function Registry

A lightweight but architecturally significant component: a registry of canonical functions with warnings for agent and developer discoverability. When code uses a non-canonical pattern — looking up a price ID manually instead of calling the designated function, or computing churn without using the standardized calculation — the registry can emit a console warning pointing to the canonical implementation.

The registry currently tracks canonical functions across Stripe billing, Firebase auth, KPI computation, and UI components. Agents check the registry before writing new utility code. The purpose is not enforcement — the registry has no runtime power to prevent non-canonical usage. It is discoverability: making sure the system's preferred patterns are findable before a new pattern gets created by someone who did not know the old one existed.


The Claude Migration: Architecture for Coaching Quality

The system is actively migrating from OpenAI to Claude Opus for coaching content generation. The email coaching pipeline — the "email machine" — has been rebuilt twice on Claude, with V1 and V2 representing two distinct architectural approaches to the same problem.

V1: Separate Calls, One Per Stage

The first Claude implementation uses separate API calls, one per stage, each producing an audit record:

Step 1: Strategy Generation. Claude Opus proposes several candidate strategies based on assembled user context and sent email history. Each one has to be grounded in an emotional angle: the feeling or tension the strategy addresses. The requirement for emotional grounding prevents the common failure mode where strategy generation produces technically reasonable but emotionally hollow approaches like "remind user about their goals."

Step 2: Strategy Selection + Self-Assessment. Claude selects the best strategy and rates its own confidence. When confidence is low, the pipeline switches to conservative framing — questions instead of assertions, curiosity instead of claims. The system enforces this regardless of what the model reports as its preferred framing mode. The design says: the model is allowed to assess its confidence, but the architectural consequences of low confidence are not optional.

Step 3: Email Generation with Extended Thinking. The final step uses Claude's extended thinking, with room to reason before writing. The instructions define the coaching identity — a warm, direct presence rather than a performer — and subject lines must reference a feeling or tension the user expressed, not a feature or goal name. The output includes a personalized CTA label — not a generic "Continue your coaching" but a prompt tied to what this person is actually working on.

V2: Single-Call Tool-Use Architecture

The second iteration collapses the separate calls into a single Claude Opus call using structured tool use. The model receives all user context upfront and works through the same stages — taking stock, stating its intent, choosing a strategy, composing — and can optionally look deeper into coaching data when it needs richer context.

The architectural advantage is threefold: latency drops by roughly 3x (fewer round trips), coherence improves (the model holds the full reasoning chain in one context window instead of losing intermediate state between calls), and the audit trail preserves the same granularity by recording each tool call as a separate step.

V2 also introduces cost-aware model selection. Paying customers — active subscribers and credit card trials — get Claude Opus for retention-quality emails. Churned, free, and never-paid users get Claude Sonnet — still capable, but at one-fifth the per-token cost. The selection function explains its own choice, and when a user's status is unknown it defaults to Sonnet. The system fails toward lower cost, not higher, because an unidentified user getting a Sonnet-quality email is better than burning Opus spend on someone who may never convert.

Per-step dollar cost is calculated from token usage and model-specific pricing tables and stored in the audit log. Administrators can see exactly what each email costs to generate, broken down by input tokens, output tokens, cache writes, and cache reads. Zero Touch Observability

The Coaching Calibration Skills

Coaching behavior is not defined by a single prompt but by a decomposed skills system. A calibration skills module defines 10 toggleable quality dimensions — Emotional Intelligence, Contextual Depth, Effortless Re-entry, Timing Intelligence, Voice Consistency, Subject Line Craft, Anti-Spam & Respect, Model Quality & Reasoning, Template & Visual Polish, and Feedback Loop. Coaching Skills Architecture

Each skill is a self-contained piece of coaching intent that an administrator can switch on or off. All skills default to off, and a skill that is off leaves no trace in the pipeline — it behaves as if the skill does not exist.

The design is modular for a specific reason: coaching quality is not a single dimension. An email can have strong emotional intelligence but weak timing awareness. It can have a compelling subject line but a voice that sounds different from the in-app coaching sessions. By decomposing quality into 10 discrete dimensions, each independently toggleable, the system allows targeted calibration. If emails are scoring well on personalization but poorly on voice consistency, the administrator activates the voice consistency skill without touching anything else.

The skill definitions themselves are instructionally rich. The Emotional Intelligence skill asks the email to mirror the person's own language instead of translating it into clinical terms, to connect to the deeper motivation, to refuse anything that could have been sent to someone else, and to frame the coach as an ally continuing a conversation rather than someone checking in.

The Anti-Spam & Respect skill includes what may be the most revealing design test in the system: "A user who unsubscribes after reading this email should still recommend the product." That sentence encodes a belief about what email communication should feel like in a coaching relationship — respectful enough that even ending the relationship does not damage the user's opinion of the system.

Context Quality Scoring: Preventing Hallucination Through Self-Awareness

Before any email is generated, the assembled user context passes through a quality scorer that weighs how much the system actually knows about this person: how recent it is, how rich, and across how many kinds of signal — sessions, goals, memories, funnel stage and intake data each count, and not equally. The more grounded the system is, the more it may offer a connection; the less grounded, the more it asks instead of asserts. It is designed this way because a wrong insight from a coach costs more trust than a missed one.

The scorer is nuanced in ways that reveal coaching domain knowledge. Session count alone does not determine session signal strength — a user whose sessions are long past gets weaker credit than the count suggests, because the context is stale. Goal scoring checks when goals were last touched, not just whether they exist — goals created but never revisited suggest avoidance, not engagement. Funnel stage scoring gives full credit only to active users because only active engagement confirms recent context the AI can reference.

The scorer also describes in plain language what is missing — for example, that goals have gone stale and may no longer reflect current priorities. A recommendation step then adapts the framing to the gaps: for a paying user with low engagement, it acknowledges their investment and loyalty rather than assuming; when very little is known, it asks open-ended questions to re-engage, because minimal data means minimal assumptions.

The paying-user check demonstrates the defensive coding philosophy throughout the system. A Stripe customer ID is not treated as a paying signal — it only means checkout was initiated, not completed. Only confirmation from the payment system counts as definitive. The design principle is explicit: having started a checkout is not the same as having paid.


The Solo Builder Methodology

Everything described above — frontend, backend, AI orchestration, billing integration, email systems, admin tooling, agent swarm, knowledge atoms, canonical registries — is built and operated by a single person. This is not presented as a heroic achievement. It is presented as evidence of a methodology.

Defensive Coding as Operational Survival

The agent operating instructions contain a rule in bold: never gate user-facing features on an explicit production check. The reasoning: if the environment variable is missing or unset in production, user-facing features silently break. Instead, the codebase excludes the known safe case (local development) — so the default behavior favors the user. The only exception is for dev-only tools (logging, debug middleware), which require an explicit development check because those should not fire in unknown environments.

This is a coding convention born from operational reality. A solo operator cannot monitor every deployment in real time. The architecture must fail toward the user's benefit, not toward correctness. Paying users who hit an edge case should get full access, not degraded access. The system's error-handling paths reflect this: the subscription access handler's fail-open path checks whether a real subscription record exists in the database, not merely whether a checkout was initiated, because the former confirms a real subscription while the latter only confirms a checkout was started. Architecture As Philosophy

Agentic Automation for Institutional Memory

The agent swarm handles what can be automated: knowledge extraction, error pattern detection, code search across multiple knowledge sources, canonical function discovery. The canonical registry is a direct expression of this methodology — instead of relying on a solo builder's memory of what utility functions already exist, the system maintains a discoverable index that agents consult before writing new code.

The knowledge atoms system serves the same purpose at a higher level. When an agent discovers that a previous assumption was wrong — for example, about how a user's billing state gets set — it creates a correction atom recording what was believed before and why the wrong belief seemed plausible. The next agent working on billing code will find this correction through semantic search before making the same mistake.

The Intent DB as Living Specification

The 100+ UX intent statements function as a specification system that evolves with the product. Instead of static requirements documents that drift from reality, each intent is a testable claim about user experience — "When a user who is not paying or on a trial first lands on the home page, we do not force login" — traceable from code annotation to database record to human-readable statement.

New intents are mined from conversation when the founder says things like "users should never see..." or "when X happens they should see Y." The system does not wait for formal specification — it extracts specification from operational intent. This is how a solo builder maintains alignment between what the system should do and what it actually does, without the overhead of traditional requirements management.


The Architecture, Connected

                    +-------------------+
                    |    COACHING UI    |
                    +--------+----------+
                             |
                             v
                    +-------------------+
                    |        API        |
                    +--------+----------+
                             |
              +--------------+--------------+
              |              |              |
              v              v              v
     +--------+---+  +------+------+  +----+--------+
     |   MODEL    |  |  KNOWLEDGE  |  |    EMAIL    |
     |  SELECTION |  |    LAYER    |  |   MACHINE   |
     +-----+------+  +------+------+  +----+--------+
           |                |              |
           v                |              v
     +-----+------+         |      +-------+-------+
     |  COACHING  |<--------+      | PERSONALIZED  |
     |  RESPONSE  |                |    EMAIL      |
     +-----+------+                +---------------+
           |
           v
     +-----+------+        +--------------------+
     |   MEMORY   |        |    AGENT SWARM     |
     | EXTRACTION +------->| (meta-learning)    |
     +------------+        +--------------------+

The loop that matters: User speaks to the coaching UI. The API assembles context from the knowledge layer. The model strategy resolver selects the right LLM configuration. The coaching response streams back. Memory extraction captures new insights as typed vectors. The email machine — powered by the same knowledge layer — generates proactive coaching emails that bring the user back. The agent swarm watches the whole process and accumulates institutional knowledge about how to do it better.

The email machine has demonstrated a favorable revenue gain-to-loss ratio when operated at sufficient dose. But the more structurally interesting finding is the "wake-up call effect" — aggressive proactive coaching purges low-value subscribers (zombie accounts that inflate metrics without generating development outcomes) while dramatically improving retention for users who are genuinely engaged. The architecture treats this as signal, not loss: a user who leaves because they were not actually coaching is a more honest outcome than a user who stays because they forgot to cancel.


What We Still Do Not Know

Does decomposing coaching into discrete skills preserve the quality of integrated practice? A masterful human coach does not consciously select "technique #7: subject-object move" mid-conversation. They respond from integrated skill. The calibration skills system decomposes coaching quality into 10 independent dimensions — but human development may require something that emerges from the interaction between dimensions, not from their sum. The system can measure each dimension independently. It cannot yet measure the gestalt.

Does self-assessment accuracy improve or degrade over time? The pipeline asks Claude to rate its own confidence and quality. Early observations suggest the model is sometimes honest — re-reading its work and marking itself down when asked to check — and sometimes performs honesty. The system cannot yet distinguish between genuine calibration and sophisticated-sounding self-assessment that masks the same overconfidence it claims to check. If self-assessment inflates systematically, the entire conservative-framing mechanism degrades — the system will send personalized emails to users it does not actually understand well.

What is the right relationship between user memory persistence and human change? The memory vector system stores insights with relevance scores and active flags. But a human who was "struggling with consistency in morning routine" six months ago may have resolved that struggle — or may have abandoned morning routines entirely and no longer considers it relevant. The system can age memories by recency. It cannot yet distinguish between a memory that is old-but-still-true and a memory that is old-and-superseded-by-growth. The same knowledge system that prevents the AI from forgetting important patterns could also prevent it from recognizing that the person has changed.

Can a meta-learning system learn coaching quality, or only coaching patterns? The agent swarm accumulates knowledge about what works — which email strategies get engagement, which calibration skills improve scores, which model configurations produce better responses. But "what works" as measured by engagement metrics is not the same as "what works" as measured by human development outcomes. The system optimizes for the proxies it can measure. Whether those proxies correlate with the thing that actually matters — whether the human became more capable — remains an open empirical question that the architecture is not yet designed to answer.