Ask questions with a local LLM

hay ask retrieves the most relevant passages from your documents and has a local LLM answer from them, citing its sources. Haypile ships no LLM and never talks to the network; generation is delegated to a server running on your machine.

The fast path

If you do not already run a local LLM:

hay llm setup

This finds an already-running server (Ollama, LM Studio, llama.cpp, Jan), or installs and starts Ollama for you, asking before anything is downloaded. The one large download is the model itself, about 2GB. When it finishes:

hay ask "what notice period does the vendor agreement require?"

Answering with llama3.2:3b (http://localhost:11434/v1)...

The vendor agreement requires a 60-day written notice period for
termination [1].

Sources:
  [1] ~/Documents/contracts/vendor-deal.docx · chunk 2
  [2] ~/Documents/contracts/meridian-msa.pdf · page 4

Check the citations

The [1] markers are the point. The model is instructed to answer only from the retrieved passages and to cite which passage supports each claim. Small models sometimes blend sources or overreach anyway; the citations let you catch it in one glance instead of trusting blindly. If an answer matters, open the cited source.

Model quality matters here. A 3B parameter model or larger holds together well; 1B models are noticeably sloppier at synthesis. Pick a specific model with:

hay ask "..." --model qwen2.5:7b

Point at a specific server

Auto-detection probes the usual local ports (Ollama 11434, LM Studio 1234, llama.cpp 8080, Jan 1337). To use something else:

hay ask "..." --endpoint http://localhost:8080/v1

Or set it once:

export HAYPILE_LLM_ENDPOINT=http://localhost:8080/v1
export HAYPILE_LLM_MODEL=my-model

Without an LLM, nothing breaks

If no server is found, hay ask says so and shows the top passages for your question instead. Search never depends on an LLM, and neither does anything else in Haypile.

Scope the retrieval

hay ask accepts the same narrowing flags as search:

hay ask "when is the filing deadline?" --tag acme --limit 8

--limit controls how many passages are given to the model as context. More is not always better; 6 to 8 focused passages usually beat 20 loose ones.