Deployment and MLOps · MLD-0837
On-device inference
Executing model predictions locally on the user's or application's device.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Run a small language model without a network request.
Original caveat
Local execution still has memory, energy, and security constraints.
Context and aliases
Level: Intermediate. Stages: Deployment · Monitoring.
Tags: ML · DL · LLM · SLM · MLOps.
Aliases: None recorded.
Related concepts
Concepts linking here
Technology connections
- Core ML — Apple framework for integrating models into native applications.
- ExecuTorch — PyTorch edge-deployment runtime and tooling.
- LiteRT — Google on-device inference runtime, formerly TensorFlow Lite.
- llama.cpp — Portable C/C++ inference engine for supported GGUF models.
- LM Studio — Desktop application for discovering and running local language models.
- MLC LLM — Compiler-based deployment stack for language models.
- Ollama — Local model-management and inference application with an API.
- OpenVINO — Toolkit for optimizing and deploying inference across supported hardware.
- Transformers.js — JavaScript execution of supported pretrained models.
- WebLLM — Browser LLM inference engine using WebGPU.
Original references
- Transformers glossary ↗ · Consulted
- Google Cloud MLOps architecture ↗ · Reference link
- scikit-learn User Guide ↗ · Consulted
- Original reference URL ↗