Project Overview

• Purpose: Twitch chat assistant providing concise, persona-aligned replies; supports triggers, hybrid retrieval, and admin tools.
• Core services: FastAPI HTTP API (bootstrap.py:145-181), Event Bus (events/bus.py:20-52), DurableStore (storage/durable_store.py:12-33), LLM pipeline (services/llm_pipeline.py:76-141,143-201,202-253), Retrieval service (retrieval/router.py:63-169,170-233), Twitch Bridge Bot (bin/bridge_bot.py:158-181,235-281).
• Feature flags: NEXT_USE_OPENAI_REAL, NEXT_ENABLE_RETRIEVAL, NEXT_ENABLE_TOOLS, NEXT_TOOLS_ENDPOINT_ENABLED, NEXT_ENABLE_OTEL, NEXT_ENABLE_PROMETHEUS (config/settings.py:6-30).

System Architecture

• App orchestration & middlewares: Correlation ID (bootstrap.py:37-43), security headers/CORS/HTTPS (bootstrap.py:151-164), Prometheus (bootstrap.py:69-76), OTEL init (bootstrap.py:64).
• Eventing: Bridge Bot publishes IRC events to bus; ChatConsumer processes chat.message and produces replies (services/chat_consumer.py:20-36,77-117,118-160).
• Storage: Repos for messages, embeddings, triggers with DB fallback (storage/durable_store.py:34-41,84-101).
• LLM pipeline: Stage1 scout with optional tools; Stage2 finalizer; validator guardrails (services/llm_pipeline.py:83-119,143-201,202-253; agents/validator.py:20-44).
• Retrieval: Hybrid search across chunks/events/summaries; blend scores (retrieval/router.py:63-169,170-233).

graph TD
    A[Twitch IRC] --> B(Bridge Bot)
    B --> C{Event Bus}
    C --> D[ChatConsumer]
    D --> E[DurableStore]
    D --> F[Retrieval Service]
    D --> G[LLM Pipeline]
    G --> H[OpenAI Agent Chat]
    G --> I[Validator]
    D --> J[Triggers]
    D --> K[HTTP API Routers]
    D --> L[chat.send]
    L --> B
    E --> M[(DB / Memory Fallback)]

Functional Requirements

• Messages
  • Store chat messages: POST /messages persists and evaluates triggers (routers/messages.py:12-63).
  • List recent: GET /messages?channel&limit (routers/messages.py:92-115).
• Replies
  • Generate and optionally save reply: POST /reply (routers/reply.py:9-36).
• Retrieval
  • Hybrid search API: GET /retrieval/search (routers/retrieval.py:9-18).
• Triggers
  • Admin CRUD: GET/POST/DELETE /triggers, enable/disable endpoints with auth (routers/triggers.py:26-52,55-127).
• Tools
  • Admin tools: GET /tools, POST /tools/execute gated by NEXT_TOOLS_ENDPOINT_ENABLED and token (routers/tools.py:23-42; bootstrap.py:175-177).
• Health/Perf/Metrics
  • GET /health (routers/health.py:10-21), GET /perf/ping (routers/perf.py:10-28), GET /metrics/final-agent (routers/metrics.py:7-9).

Technical Specifications

• Languages/Frameworks
  • Python 3.11+ (tested on 3.13), FastAPI, SQLAlchemy, asyncpg, pgvector, OpenAI client, structlog, opentelemetry, prometheus.
• Database Schema
  • Messages: ops.messages (UUID id, text, meta) (storage/models/messages.py:9-19; 20251127_0001_create_ops_messages.py:11-25).
  • Chat chunks: ops.chat_chunks (text, embedding Vector(1536), meta, time window) (storage/models/embeddings.py:10-26; 496db14e3b68_create_hybrid_retrieval_schema.py:20-33,76-79).
  • Summaries/events: narrative/event tables with optional embeddings (496db14e3b68_create_hybrid_retrieval_schema.py:35-63).
  • Triggers: JSON rule, enabled/version (storage/models/triggers.py:8-16; 20251127_0003_create_ops_triggers.py:11-19).
• Data Flow
  • Ingestion buffer: aggregate lines → embed → write chunks → emit event (ingestion/buffer.py:33-107,110-161).
  • Extractor: parse structured events → embed → persist (ingestion/extractor.py:24-116).
  • LLM pipeline: build context (recent + retrieval), optional tool calls, persona finalizer, validator enforcement (services/llm_pipeline.py:83-119,143-201,202-253).

flowchart TD
    msg[Incoming chat.message] --> store[Store in ops.messages]
    store --> ingest[Buffer & Extractor]
    ingest --> chunks[Write ops.chat_chunks]
    ingest --> events[Write ops.chat_events]
    msg --> ctx[Build Context]
    ctx --> retr[Hybrid Retrieval]
    retr --> s1[Stage1 Scout]
    s1 --> tools{Wants Tools?}
    tools -- yes --> exec[Execute Tool & Append]
    tools -- no --> s2[Stage2 Finalizer]
    exec --> s1
    s1 --> s2
    s2 --> val[Validator]
    val --> out[Single-line reply]

erDiagram
    ops_messages {
        uuid id
        text text
        jsonb meta
        varchar channel_id
        timestamptz ts
    }
    ops_chat_chunks {
        uuid id
        varchar channel_id
        varchar doc_id
        uuid message_id
        text text
        vector embedding
        jsonb meta
        timestamptz ts_start_utc
        timestamptz ts_end_utc
    }
    ops_chat_summaries {
        bigint id
        varchar channel_id
        text summary_text
        vector emb
        jsonb meta
    }
    ops_chat_events {
        bigint id
        varchar channel_id
        timestamptz ts_utc
        varchar kind
        varchar actor
        varchar target
        jsonb payload
        text text
        vector emb
    }
    ops_triggers {
        uuid id
        varchar name
        boolean enabled
        int version
        json rule
    }
    ops_messages ||--o{ ops_chat_chunks : message_id

Quality Assessment

• Test Coverage
  • Integration: endpoints, flags, tools gating, reply flow, retrieval search (tests/integration/test_tools.py:6-23; test_reply.py:5-12; test_retrieval.py:6-13; test_reply_memory_flow.py:6-19; triggers admin suites).
  • Unit: durable store transactions, triggers eval, LLM pipeline prompts and validator stub policy (tests/unit/test_durable_store_transaction.py:59-122; test_llm_pipeline_stub_policy.py:12-21).
• Observability
  • Correlation IDs, optional OTEL init, Prometheus instrumentation (bootstrap.py:37-43; observability/otel.py:4-19; bootstrap.py:69-76).
• Technical Debt & Improvements
  • LLM tools coupled to twitchbot.database (commands/quotes) (services/tools_def.py:1-9,28-60).
  • Repo vector search vs hybrid retrieval duplication (repos/embeddings_repo.py:32-53; retrieval/router.py:170-233).
  • Stub metrics endpoint; expand metrics.

Operations Appendix

• Environment & Secrets
  • Required: OPENAI_API_KEY when NEXT_USE_OPENAI_REAL=True (llm/openai_orchestrator.py:12-16,88-93).
  • Admin token: NEXT_ADMIN_TOKEN for tools/triggers endpoints (routers/tools.py:31-38; routers/triggers.py:16-23).
  • DB: DATABASE_URL for SQLAlchemy async engine; memory fallback when unreachable (storage/durable_store.py:34-41,84-101).
  • CORS/Hosts: NEXT_ALLOWED_HOSTS, NEXT_CORS_ORIGINS (bootstrap.py:151-164).
• Deployment
  • Run app: uvicorn app:app or python app.py per existing scripts; ensure flags set for OTEL/Prometheus as desired.
  • Production: enforce strong NEXT_ADMIN_TOKEN; HTTPSRedirectMiddleware enabled (bootstrap.py:151-156).
• Scaling & Performance
  • Enable Prometheus for HTTP metrics; consider custom counters for LLM latency and ingestion flush counts.
  • Tune retrieval: CONTEXT_RETRIEVAL_K, CONTEXT_TOKEN_BUDGET, SNIPPET_MAX_CHARS (config/settings.py:42-47).
  • LLM caps: LLM_MAX_TOKENS_STAGE1, LLM_MAX_TOKENS_STAGE2 (config/settings.py:35-36).
• Runbooks
  • Investigate reply issues: check X-Correlation-ID header, inspect /perf/ping, verify NEXT_USE_OPENAI_REAL and API key presence, examine validator report in reply body (services/llm_pipeline.py:229-241).
  • Troubleshoot retrieval: call /retrieval/search, check DB connectivity, verify embeddings status in ops.chat_chunks.
  • Tools failures: confirm NEXT_TOOLS_ENDPOINT_ENABLED and NEXT_ADMIN_TOKEN alignment; review unified registry entries (tools/registry.py:1-50; services/tools_def.py:1-119).

Future Recommendations

• Expand metrics: message counts, trigger matches, ingestion flushes, LLM latencies.
• Harden admin auth scopes and audit logging; per-channel tool permissions.
• Add EventSub and backpressure tests for ChatConsumer.
• Normalize JSONB meta and indexes across tables; review pgvector index coverage.
• Maintain this spec as the onboarding guide; keep citations current as code evolves.