AI Platform Infrastructure Beyond the Model API

The gateway is the single entry point to every model provider. Product code does not know whose model answered.

• A circuit breaker per provider

  When a provider starts erroring or slowing down, traffic stops going there, so timeouts do not pile up behind it.

• Fallback chain

  If a provider is unavailable, the request goes to another model of a comparable class. The user sees a short delay instead of an error.

• Tier and intent routing

  Simple requests go to cheaper models and harder ones to stronger models. The decision rests on task type and cost.

• Graceful degradation

  If the preferred model is down, the system offers an available replacement instead of failing as a whole.

• Streaming

  The answer reaches the client as it is generated, over WebSocket or SSE, so nobody waits for the full text.

This is about storing and finding information. It does not involve training or fine-tuning models, only retrieving the right data.

• Semantic cache

  For requests close in meaning, an earlier answer found in PostgreSQL and pgvector is reused instead of calling a provider again.

• Three-layer memory

  Short-term conversation context sits in Redis, structured facts in PostgreSQL, and semantic retrieval over embeddings in pgvector.

• Session summarisation

  A long history is compressed into a summary, and what matters is saved to long-term memory.

• Prompt under a token budget

  Context is composed from prioritised layers to fit the budget, rather than being cut off arbitrarily.

• Retrieval instead of tuning

  New knowledge reaches the answer by retrieving relevant passages, while the model itself stays unchanged.

Swapping a model or editing a prompt without a check is an experiment on users. That is why evaluation lives inside the server.

• LLM-as-judge

  A judge model scores answers against defined criteria. It is a fast check, best complemented by human review.

• Model and prompt comparison

  The same scenarios run through different models and prompt versions, and the results are compared.

• Regression checks

  Before a model switch, a scenario set shows what got worse.

• A/B experiments

  Variants are compared on real usage scenarios rather than synthetic examples.

A prepaid balance demands care because a model call costs money before its outcome is known.

• Redis holds

  The amount is reserved before the call, settled by actual cost after a successful answer, and released on failure. Concurrent requests cannot push a balance below zero.

• A stream that breaks midway

  If a provider cuts an answer off, the balance stays correct: actual usage is charged and the surplus reservation is returned.

• Payment reconcile worker

  On a schedule it compares provider payments with internal records and raises any mismatch.

• Webhook signature checks

  Notifications from the payment provider are accepted only after their signature is verified.

• Prometheus and Grafana

  Several dashboards cover provider errors, latency, spend and queue state.

• Structured logs

  Machine-readable records make it possible to follow one specific request.

• Versioned migrations

  The PostgreSQL schema changes through versioned migrations, so database state is reproducible.

• Docker delivery

  Services run the same way locally and in deployed environments.