Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
Deployment Strategies
- Trade-offs between hosted APIs and self-hosted solutions
- Patterns for on-premises, edge, and air-gapped environments
- Single model versus model portfolio approaches
- Cost analysis per decision
Hardware Sizing and Capacity Planning
- Memory footprint relative to parameter count
- CPU-only versus GPU acceleration
- Planning for concurrency and throughput
- Reserving headroom for increasing task lengths
Serving Stacks
- vLLM and SGLang options
- llama.cpp and GGUF execution
- Ollama for local development
- Selecting stacks based on decision workload characteristics
Quantisation While Preserving Calibration
- Precision formats and associated memory savings
- Assessing quality and calibration degradation
- Per-tensor and importance-aware variants
- Conducting acceptance tests post-quantisation
Latency Engineering
- Median versus tail latency metrics
- Latency impact of context length growth
- Warm starts and model caching strategies
- Batching multiple decision queries
Edge and Constrained Environments
- Running small decision models on CPUs
- Deployment on Apple Silicon and embedded systems
- Offline operation and update protocols
- Distributing weights in air-gapped settings
Containerisation and Scaling
- Designing images for model serving
- Managing volumes and caches
- Horizontal scaling and routing
- Applying resource limits and GPU sharing
Observability
- Tracking request, latency, and error metrics
- Monitoring confidence distributions
- Logging decision outcomes and escalations
- Setting alerts for calibration drift
Versioning and Rollout
- Versioning models and adapters
- Implementing shadow and canary deployments
- Using pinned model files for audit purposes
- Establishing rollback procedures
Security and Multi-Tenancy
- Implementing authentication and rate limiting
- Ensuring tenant isolation for prompts and logs
- Configuring network isolation and egress controls
- Maintaining audit logs of decisions
Requirements
- Experience with Linux administration and containerisation.
- Familiarity with model inference servers and GPU management.
- Foundational knowledge of REST APIs, load balancing, and monitoring.
Target Audience
- MLOps and platform engineers managing inference infrastructure
- SRE teams supporting AI services
- AI engineers deploying decision models on-premises
- Architects designing air-gapped or regulated AI deployments
