← Blog

How michi-chat works

August 31, 20265 min read

In Introducing michi-chat I showed what the assistant does. This post is about how: the journey of one question through the system, why the model layer is a proxy called LiteLLM, and how swapping the AI model is a config edit rather than a rewrite. No AI background needed; if you can read a flowchart, you can read this.

The journey of one question

When a visitor types "what time do you open on weekends?" into the widget, this happens, in order:

visitor -> embed key (which business is this?)
        -> guardrails (is this bait? is the daily cap hit?)
        -> answer cache (have we answered this exact meaning recently?)
        -> the tool loop (the model asks for data, tools fetch it)
        -> the answer streams back word by word

Each step exists for a reason:

  1. The embed key selects the tenant. One deployment serves many businesses; the key on the widget says whose persona, whose tools and whose knowledge to use. It is a selector, not a secret, so everything after it must hold up on its own.
  2. Guardrails run before any AI. Requests that try to drag the bot off topic burn strikes and get a canned refusal with no model call at all, and a per-business daily cap protects the bill. The cheapest model call is the one you never make.
  3. The semantic answer cache can end the turn right here. More on it below, it earned its own war story.
  4. The tool loop is where the honesty lives. The model is not asked to remember the opening hours; it is given tools ("get the hours", "get upcoming events", "search the knowledge base") and the answer is grounded in what the tools return. Events, for example, come back already split into upcoming and recent past, computed in the cafe's own timezone, because language models are famously bad at date math and should never be asked to do any.
  5. The answer streams over server-sent events: little status chips ("Checking our info"), then the text token by token. The visitor watches the bot actually working.

Facts live in two places on purpose

Stable facts (the story of the beans, the venue rules, what the shop does NOT sell) live in a knowledge base: markdown files, chunked along their headings, each chunk stored with an embedding so a question in any wording, even in Filipino, finds the right chunk by meaning rather than by keywords. Fast-changing facts (events, weekly specials, weather) never touch the knowledge base; they come from live tools at answer time, so they cannot go stale.

The one rule above both: if the data does not contain the answer, the bot says so and points to a human. It is prompted, guarded, and eval-tested to never invent a fact, a price, or an event.

The cache that outlived a bug fix

First messages get cached by meaning: "how much is a latte" and "latte price?" hit the same cached answer, served instantly and for free. Entries live for 24 hours and are wiped the moment anyone edits the tenant or its knowledge.

That design produced my favorite lesson of this project. I fixed a real bug (the events tool was serving only past events), deployed the fix in five minutes, clicked the suggestion chip again, and got the same wrong answer. Nothing was broken: the chip sends the identical first message every time, the old answer was still cached, and deploys deliberately do not clear the cache. Three correct designs composed into "the fix didn't ship". The operational rule that fell out: after any fix that changes behavior, flush the cache or test with a phrasing the cache has never seen.

Why LiteLLM sits between the app and every model

michi-chat's application code never names an AI provider. It asks for aliases:

  • michi answers visitors
  • judge grades answers in the eval suite (never let a model grade its own homework)
  • embed turns text into the vectors behind the knowledge base search

What actually serves each alias is decided in one YAML file that the LiteLLM proxy reads:

model_list:
  - model_name: michi
    litellm_params:
      model: ollama_chat/qwen3.5:4b
      api_base: os.environ/OLLAMA_API_BASE

  - model_name: embed
    litellm_params:
      model: ollama/nomic-embed-text
      api_base: os.environ/OLLAMA_API_BASE
      drop_params: true

The reasons this is a proxy and not code:

  1. Swapping providers is an edit, not a migration. Point michi at a cloud model tomorrow and the app neither knows nor cares. My earlier attempt at this project (a .NET platform) spent half its code on provider routing, keys and budgets; LiteLLM deleted that half, which is exactly why the rebuild could be small.
  2. One place for keys and spend. The app holds a single LiteLLM key; provider keys live in the proxy. Budgets and rate limits are proxy config, not application features.
  3. It absorbs provider quirks. Real example from this config: the OpenAI SDK always sends a parameter that Ollama does not support, and drop_params: true makes that mismatch disappear. Translating quirks is the proxy's whole job.
  4. A/B testing is free. A tenant's model field can point at a trial alias (gemma, michi-mini) to test a different model on one business with zero code changes.

How a config change actually ships

In development:

# edit litellm/config.yaml, then:
docker compose up -d --force-recreate litellm

(The force-recreate is a hard-won detail: a plain restart keeps serving the stale file on Docker Desktop under WSL, because the config is a single-file bind mount.)

In production the config file lives in a small deploy repo, so a model swap is: edit the YAML, commit, push, and the server's runner restarts the proxy. The change is reviewable, revertable git history, like any other change.

Two aliases deserve extra respect before you touch them:

  • embed is sticky. The database column is sized to the embedding model's exact output (768 numbers), and two different models arrange meaning differently even at the same size. Changing embed means re-embedding every document, so it is a decision, not a tweak.
  • Any swap of michi goes through the evals first. The project ships two golden sets: one checks that knowledge-base search retrieves the right chunks, and one runs real questions through the real chat API while the judge model grades every answer for faithfulness and completeness. If a cheaper model tanks the score, the YAML edit never ships. The judge being a different model, behind its own alias, is the point.

The shape of the whole thing

That is the entire architecture: a Next.js app that trusts nothing, a Postgres that holds everything (facts, vectors, cache, transcripts, audit trail), and a proxy that makes the AI model a replaceable part. The code is at beany-vu/michi-chat, the docs at beany-vu.github.io/michi-chat, and the live example is the assistant on mugshotmnl.com, with its sibling answering questions about me right here on this site.