Self-Hosting Decision Models: Serving, Quantisation, Latency and Operations Training Course

Deployment Options

• Trade-offs between Hosted APIs and self-hosted solutions.
• Patterns for on-premises, edge, and air-gapped environments.
• Single model versus model portfolio strategies.
• Comparison of cost per decision.

Hardware Sizing and Capacity

• Memory footprint based on parameter count.
• CPU-only versus GPU acceleration.
• Planning for concurrency and throughput.
• Allowing headroom for task length growth.

Serving Stacks

• vLLM and SGLang options.
• llama.cpp and GGUF execution.
• Ollama for local development.
• Selection based on decision workload characteristics.

Quantisation Without Losing Calibration

• Precision formats and memory savings.
• Measuring quality and calibration loss.
• Per-tensor and importance-aware variants.
• Acceptance testing following quantisation.

Latency Engineering

• Median versus tail latency.
• Latency growth with context length.
• Warm start and model caching.
• Batching multiple decision questions.

Edge and Constrained Environments

• Small decision models on CPU.
• Apple Silicon and embedded deployment.
• Offline operation and update strategy.
• Air-gapped weight distribution.

Containerisation and Scaling

• Image design for model serving.
• Volume and cache management.
• Horizontal scaling and routing.
• Resource limits and GPU sharing.

Observability

• Request, latency, and error metrics.
• Confidence distribution monitoring.
• Tracking decision outcomes and escalations.
• Alerting on calibration drift.

Versioning and Rollout

• Model and adapter versioning.
• Shadow and canary deployment.
• Pinned model files for audit purposes.
• Rollback procedures.

Security and Multi-Tenancy

• Authentication and rate limiting.
• Tenant isolation for prompts and logs.
• Network isolation and egress control.
• Audit logging of decisions.