Language-model training and adaptation · MLD-0688
Low-rank adaptation
Learning low-rank weight updates while keeping original base weights frozen.
Original reference page · detailed expansion pending. Original definitions, examples, notes and citations are preserved below. New examples are illustrative, not benchmark results.
Original example
Adapt a model with two small update matrices per target layer.
Original caveat
LoRA reduces trainable parameters, not automatically the base model's stored size.
Context and aliases
Level: Intermediate. Stages: Training · Adaptation · Evaluation.
Tags: DL · SLM · LLM · NLP.
Aliases: LoRA.
Related concepts
Concepts linking here
Technology connections
- Axolotl — Configuration-driven model fine-tuning toolkit.
- bitsandbytes — Low-bit quantization and memory-saving optimizer primitives.
- Hugging Face PEFT — Parameter-efficient model adaptation methods.
- Hugging Face TRL — Post-training trainers for supervised and preference/RL-style methods.
- LLaMA-Factory — Unified training and fine-tuning toolkit for supported models.
- NVIDIA Model Optimizer — Model optimization tools including supported quantization, pruning and distillation paths.
- OpenRLHF — Distributed RLHF and post-training framework.
- TorchAO — PyTorch architecture optimization and quantization tooling.
- torchtune — PyTorch-native post-training recipes; development has wound down.
- Unsloth — Optimized fine-tuning tooling for supported model families.