AI/LLM Product Engineering

• LLM gateways and multi-provider routing

  One entry point to several providers, with model choice driven by task, cost and availability.

• RAG and semantic search

  Retrieval over your own data using embeddings and pgvector, and context assembly for the answer.

• Embeddings and long-term memory

  What the system remembers about a user or session, how it is retrieved and how it reaches the prompt.

• Agent and workflow pipelines

  Multi-step processing across models, background jobs and checkpoints at each step.

• AI features inside existing products

  Wired into your backend, permissions and billing rather than a parallel system next to them.

• AI-assisted internal tools

  For example an analyst that answers questions through a read-only database session and prepares reports for an operations team.

• Evaluation

  Model and prompt comparison on real scenarios, with regression checks.

• Billing and quotas

  Token accounting, prepaid balances, per-plan and per-user limits.

• Moderation and guard pipelines

  Checks on the request and the response: jailbreak attempts, drifting out of the assigned persona.

This is about how a system stores and finds information. It does not involve training models.

• Three-layer memory

  Short-term conversation context in Redis, structured facts in PostgreSQL, semantic retrieval over embeddings in pgvector.

• Semantic cache

  Similar requests reuse a previously found answer from Postgres and pgvector instead of calling a provider again.

• Session summarisation

  Long history is compressed into a summary, and what matters is saved to long-term memory.

• Prompt assembly under a token budget

  Context is composed from prioritised layers to fit the budget, rather than being cut off arbitrarily.

• RAG and retrieval

  Finding relevant passages by meaning and inserting them into the request together with their source.

Switching a model or editing a prompt without a check is an experiment run on your users.

• Model comparison

  The same scenarios run through different models and are compared on quality and price.

• LLM-as-judge

  A judge model ranks answers against defined criteria. It is a fast check, best complemented by human review.

• Regression checks

  Before a model switch or a prompt change, a scenario set shows what got worse.

• A/B experiments

  Variants are compared on real usage scenarios, not synthetic examples.

• Prepaid balance with Redis holds

  The amount is reserved before the model call and settled afterwards, so concurrent requests cannot overdraw a balance.

• Payment reconcile worker

  Provider payments and internal records are compared on a schedule, and signatures are verified at intake.

• Provider fallback

  One provider being down does not stop the product.

• Token accounting

  The cost of every request is recorded against user, model and plan.

• Observability

  Prometheus and Grafana dashboards for errors, latency and spend surface problems before users report them.

• A prototype must become a dependable feature

  The demo works, but there are no limits, retries, monitoring or defined behaviour on error.

• Cost and quality are unpredictable

  Spend is growing and it is unclear which model suits which task.

• You need to add several providers

  You want to avoid dependence on a single API and pick models per task.

• AI has to live inside an existing backend

  The feature must use your data, permissions and billing.