Sort Email Locally With Ollama and AI Email Filter

TL;DR. Install Ollama, pull a small instruct model, and point AI Email Filter’s Bring Your Own LLM setting at http://localhost:11434/v1. Classification then runs entirely on your own hardware — no email content is sent to a third-party API.

Related reading: Bring Your Own LLM · Self-hosted email sorting

If you already run your own mail server, sending every subject line and sender address to a cloud LLM for classification defeats half the point of self-hosting. Ollama fixes that: it runs an OpenAI-compatible API on your own machine, and AI Email Filter can talk to it directly through its BYO-LLM setting. This post walks through the setup end to end.

Why run the classifier locally

AI Email Filter classifies mail using the subject, sender, and a short body preview. With a hosted model, that data goes to whichever provider you pick. With Ollama, it doesn’t leave the box it runs on. The trade-off is speed and raw classification quality against a large hosted model — but for sorting mail into folders based on a clear prompt, a small local model is usually more than enough, and re-categorizing a sender you disagree with takes one click regardless of which model made the call.

The other reason to do this: cost. A local model has no per-token bill. If you’re running Ollama on hardware you already own, classification is effectively free after setup.

Step 1: Install Ollama

On Linux or macOS:

curl -fsSL https://ollama.com/install.sh | sh

On Windows, download the installer from ollama.com/download. Either way, Ollama installs as a background service and starts listening on localhost:11434 automatically — you generally don’t need to run ollama serve manually unless you stopped the service and want to run it in the foreground for debugging:

ollama serve

Verify it’s up:

curl http://localhost:11434/api/tags

An empty {"models":[]} response means Ollama is running but you haven’t pulled a model yet.

Step 2: Pull a model suited to classification

Email classification is a small, well-defined task — sender, subject, and a short preview, matched against a category description. You don’t need a large model for this. Small instruct models are fast, cheap on RAM, and accurate enough for the job:

ollama pull qwen2.5:3b-instruct
# or
ollama pull llama3.2:3b
# or, for the lightest footprint
ollama pull phi3:mini

qwen2.5:3b-instruct and llama3.2:3b both run comfortably on a machine with 8GB of RAM and no GPU; phi3:mini is smaller still and a reasonable choice on more constrained hardware like a Raspberry Pi 5 or an old laptop repurposed as a home server. If you have a GPU or don’t mind slower classification, a 7B-class instruct model (e.g. qwen2.5:7b-instruct or mistral:7b) will generally be more accurate at distinguishing similar categories.

Confirm the model downloaded:

ollama list

Step 3: Point AI Email Filter at Ollama

In AI Email Filter’s settings, open the LLM provider section and choose the custom/OpenAI-compatible endpoint option. Set:

• Base URL: http://localhost:11434/v1
• Model: the exact tag you pulled, e.g. qwen2.5:3b-instruct
• API key: Ollama doesn’t check this, so any placeholder value works — but the field usually can’t be left empty

This is the same BYO-LLM mechanism AI Email Filter uses for a hosted OpenAI-compatible endpoint; Ollama just happens to expose that same API shape locally.

One networking detail that trips people up: localhost only resolves to Ollama if AI Email Filter is running on the same machine. If AI Email Filter runs in a container or on a different host than Ollama, use the machine’s LAN IP or hostname instead of localhost, and make sure Ollama is bound to listen on 0.0.0.0 rather than just the loopback interface, and that your firewall allows the connection on port 11434.

Step 4: Test before trusting it

Run a dry test on a handful of real emails and check the results against your category descriptions before turning on any destructive actions. Small models are more sensitive to vague category prompts than large hosted ones — if two categories overlap (“Work” and “Project X”), a 3B model is more likely to blur the line than GPT-4-class models are. Tightening the category description usually fixes it faster than switching models.

Sender Smart Categories helps here too: once a sender is classified correctly, that decision is cached, so the local model only has to get it right once per sender rather than on every single email.

What this buys you

With Ollama as the backend, AI Email Filter’s classification step never leaves your infrastructure. Combined with a self-hosted mail server, that closes the last gap where email metadata would otherwise touch a third party — see self-hosted email sorting for how this fits into a fully self-run setup, and Bring Your Own LLM for the other endpoints this same setting supports if you decide a hosted model is a better trade-off later.

Model choice isn’t locked in either. If a small model isn’t accurate enough for a tricky category, pull a bigger one and repoint the model field — no other configuration changes.

Keeping Ollama running reliably

A couple of operational details matter if you want this setup to keep working unattended:

• Idle unloading. By default, Ollama unloads a model from memory after a few minutes of inactivity and reloads it on the next request, which adds a short delay to the first classification after a quiet period. If that delay matters, set OLLAMA_KEEP_ALIVE=-1 (or a longer duration) as an environment variable before starting the service to keep the model resident.
• Concurrent requests. If AI Email Filter classifies a batch of emails at once, Ollama processes them according to its configured parallelism (OLLAMA_NUM_PARALLEL), which defaults to a small number on CPU-only setups. For a personal mailbox this is rarely a bottleneck; for a shared inbox with high volume, it’s worth checking.
• Updating the model. Re-running ollama pull <model> fetches the latest version of a tag without any extra steps — useful if a model you rely on gets an updated release upstream.

None of this is specific to AI Email Filter — it’s standard Ollama operation — but it’s the part people usually hit only after the initial setup works and they start relying on it day to day.