Fine-tuning and quantization

Fine-tuning adapts an open model to your own data, and quantization shrinks a model so it fits in less memory. Both used to need a rack of servers. Today, for small and mid-size models, both can be done on a single gaming GPU or an Apple silicon Mac, which is why they belong in a directory of local tools.

Most local fine-tuning uses adapters. Instead of retraining every weight, the tool trains a small set of extra weights that sit alongside the original model. That uses far less memory and produces a small file you can share or swap out. Combined with loading the base model in reduced precision, it brings training within reach of hardware many people already own.

Before you start, be clear about what fine-tuning is good for. It is effective for teaching a model a format, a tone, a set of labels or a narrow task with consistent examples. It is a poor way to add facts that change often; for that, document retrieval is usually better. Many projects that begin as fine-tuning end up as a mix of both.

Data is most of the work. You need clean, consistent examples, written in the format the model will see when it is used, and you need to keep some aside to test against. Remove personal information you do not have the right to use, and check that the base model's license permits fine-tuning and the use you have in mind.

Featured local AI tools: fine-tuning and quantization

SponsoredFine-tuning and quantization This position is open Put your tool at the top of the fine-tuning and quantization page. Paid spots are always labeled Sponsored. From $99 a month. Get featured

What a featured listing costs ยท from $99 a month

5 local AI tools in fine-tuning and quantization

Unsloth

by Unsloth AI

A library for fine-tuning open language models faster and with less GPU memory, with ready notebooks.

  • Open source

Axolotl

by Axolotl AI

A configuration-driven tool for fine-tuning open models, from small adapters to full training runs.

  • Open source

bitsandbytes

by bitsandbytes Foundation

A library for 8-bit and 4-bit model weights, used to load and fine-tune large models on smaller GPUs.

  • Open source

MLX LM

by Apple

Tools built on MLX for running, quantizing and fine-tuning language models on Apple silicon Macs.

  • Open source
  • Runs on macOS

LLaMA-Factory

by hiyouga

A fine-tuning toolkit with a web interface that supports many open model families and training methods.

  • Open source

Quantization tools convert a model into smaller files at various precision levels. Test the result on prompts you care about, since quality loss shows up unevenly, often first in reasoning, math and less common languages.

Watch the hardware while training runs. Fine-tuning keeps a GPU or Apple silicon chip at full load for long stretches, and running out of memory partway through wastes the run. Start with a short test on a small slice of the data to confirm settings fit in memory, then launch the full job, and save checkpoints along the way so a crash does not cost the whole session.

Plan how the result will be used. An adapter can be loaded on top of the base model or merged into it and then quantized for a desktop app or runner. Check that your target tool supports the format you produce.

Expect to iterate. Training settings, data size and adapter choices interact, and the first run is rarely the one you keep. Keep notes on each run so you can compare.

Questions about fine-tuning and quantization

Can I fine-tune a model on a laptop?

Small models, yes, especially with adapters on an Apple silicon Mac or a laptop with a capable GPU. Larger models need more memory than most laptops have. Cloud GPUs are a common middle step for one-off runs.

Should I fine-tune or use document retrieval?

Use retrieval when the model needs facts from your documents, especially facts that change. Use fine-tuning to teach a consistent format, style or narrow task. Many setups combine both.

Does quantizing a model ruin its quality?

Moderate quantization usually costs little for everyday use. Very aggressive settings cause clearer drops, often in reasoning and math. Compare a few versions on your own prompts before settling.

Can I share a model I fine-tuned?

Only within the base model's license. Some licenses allow sharing derivatives with the same terms, some require attribution, and some forbid redistribution. Check your training data rights too.

Related categories