Thank you for sending your enquiry! One of our team members will contact you shortly.
Thank you for sending your booking! One of our team members will contact you shortly.
Duration 14 hours
Course Outline
AI Sovereignty and Local LLM Deployment
- Addressing risks associated with cloud LLMs, including data retention, input utilization for training, and foreign jurisdiction issues.
- Exploring Ollama’s architecture: the model server, registry system, and OpenAI-compatible API integration.
- Benchmarking Ollama against alternatives such as vLLM, llama.cpp, and Text Generation Inference.
- Navigating model licensing agreements for Llama, Mistral, Qwen, and Gemma.
Installation and Hardware Configuration
- Setting up Ollama on Linux with support for both CUDA and ROCm ecosystems.
- Implementing CPU-only fallback strategies and leveraging AVX/AVX2 optimizations.
- Deploying via Docker with appropriate persistent volume mappings.
- Configuring multi-GPU setups and establishing effective VRAM allocation strategies.
Model Management
- Retrieving models from the Ollama registry using commands like ollama pull llama3.
- Importing GGUF formats from HuggingFace and TheBloke repositories.
- Evaluating quantization levels such as Q4_K_M, Q5_K_M, and Q8_0 to balance performance and quality.
- Managing model switching and understanding concurrent model loading constraints.
Custom Modelfiles
- Mastering Modelfile syntax, including FROM, PARAMETER, SYSTEM, and TEMPLATE directives.
- Adjusting generation parameters like temperature, top_p, and repeat_penalty.
- Engineering system prompts to define role-specific model behaviors.
- Building and publishing custom models to a local registry.
API Integration
- Utilizing the OpenAI-compatible /v1/chat/completions endpoint.
- Implementing streaming responses and enforcing JSON output modes.
- Connecting local LLMs with frameworks like LangChain, LlamaIndex, and bespoke applications.
- Securing endpoints with authentication and rate limiting through reverse proxies.
Performance Optimization
- Optimizing context window sizes and managing KV cache memory usage.
- Enhancing throughput through batch inference and parallel request processing.
- Configuring CPU thread allocation with NUMA architecture awareness.
- Monitoring GPU utilization rates and memory pressure metrics.
Security and Compliance
- Establishing network isolation for model serving endpoints.
- Implementing robust input filtering and output moderation pipelines.
- Maintaining comprehensive audit logs for all prompts and completions.
- Ensuring model provenance integrity through hash verification.
Requirements
- Proficiency in intermediate Linux administration and container management.
- A high-level understanding of machine learning concepts and transformer architectures.
- Familiarity with RESTful APIs and JSON data structures.
Target Audience
- AI engineers and developers looking to substitute cloud LLM APIs with local solutions.
- Organizations with data privacy constraints that preclude the use of cloud-based models.
- Government and defense teams requiring fully air-gapped language model capabilities.
