# Getting started (/docs) This tutorial takes you from nothing to searching your own documents. It takes about two minutes, and nothing in it needs the internet after the install step. ## 1. Install [#1-install] On macOS, with Homebrew: ```sh brew install BenyD/tap/hay ``` On Linux or macOS, with the install script: ```sh curl -fsSL haypile.sh | sh ``` On Windows, run this in PowerShell: ```powershell irm https://haypile.sh/install.ps1 | iex ``` It installs `hay.exe` under `%LOCALAPPDATA%\Programs\hay` and adds it to your PATH. Open a new terminal afterwards so the PATH change takes effect. Either way you get one binary called `hay`. The embedding model that powers semantic search is already inside it. There is nothing else to download, ever. Check it works: ```sh hay --version ``` ## 2. Index a folder [#2-index-a-folder] Point Haypile at any folder with documents in it. PDF, docx, pptx, HTML, Markdown, plain text, and mbox email archives are indexed; everything else is ignored. ```sh hay add ~/Documents ``` A live progress line tracks the pass, first extracting text, then embedding it, then a summary lands: ``` Indexing /Users/you/Documents… extracting [######····] 62% · 194/312 files · ~1m left ``` ``` Indexed 214 files (1892 chunks), 0 unchanged. Embedded 1892 chunks for semantic search (bundled/all-MiniLM-L6-v2). Try: hay search "something you remember" ``` In a pipe or a CI log, where a line cannot redraw, the same progress arrives as one printed line per tenth of each phase. `hay add` also did the one-time setup: the index lives in a single SQLite file at `~/.haypile/haypile.db`, and a background daemon now watches the folder, re-indexing changed files within seconds. There is no server or database to manage. ## 3. Search [#3-search] Search for an idea, not just exact words: ```sh hay search "agreement cancellation" ``` ``` 1. ~/Documents/contracts/vendor-deal.docx (chunk 2) Termination Either party may terminate this Agreement with sixty days written notice. 2. ~/Documents/contracts/meridian-msa.pdf (page 4) Either party may terminate for convenience upon sixty (60) days prior written notice. ``` Neither result contains the words "agreement cancellation". That is semantic search doing its job. Exact identifiers work too, and they match exactly: ```sh hay search "2024-CV-01847" ``` Every result cites its source: the file, and for PDFs the page. You can always check the original. ## 4. Prove it stays private [#4-prove-it-stays-private] ```sh hay status ``` ``` Daemon: running (0.3.1, up 42s) Index: ~/.haypile/haypile.db Indexed: 1 sources, 214 files, 1892 chunks Model: bundled/all-MiniLM-L6-v2 Outbound connections: 0 ``` That last line is measured, not asserted. Haypile counts its own network connections and reports them. The target is zero, always. ## Where to go next [#where-to-go-next] * Get answers instead of search results: [Ask questions with a local LLM](/docs/guides/ask) * Prefer a browser to a terminal: [Use the web UI](/docs/guides/web) * Let Claude Code or Cursor search your documents: [Use Haypile from Claude Code](/docs/guides/claude-code) * Tag a folder and exclude drafts from the index: [Configure a folder](/docs/guides/folders) * Index scans and image-only PDFs: [Scanned PDFs (OCR)](/docs/guides/scanned-pdfs) * Update, stop the daemon, or uninstall: [Manage, update, uninstall](/docs/guides/manage) # How search works (/docs/explanation/how-it-works) This page explains what happens between `hay add` and a search result. Nothing here is required reading to use Haypile; it exists because understanding a tool builds the right instincts for it. ## The pipeline [#the-pipeline] ``` folders -> watcher -> extract -> chunk -> embed -> SQLite | you <- citations <- fuse <- keyword search + vector search ``` ### Extract [#extract] Each format gets a real parser. PDFs go through PDFium, the same engine Chrome uses to render them, compiled to WebAssembly so it runs inside the Go binary with no native dependencies. Extraction is per page and layout-aware: paragraphs, headings, and bullets are rebuilt from the page geometry, and icon-font glyphs are dropped instead of indexed as junk. The page number travels with the text from here on; it is what makes `contract.pdf (page 12)` possible later. A page with no text layer but an image (a scan) is rendered and transcribed by your local vision LLM when one is running; without one it indexes empty. Word documents are unzipped and their XML parsed directly. Markdown is sectioned at its headings. ### Chunk [#chunk] Search operates on chunks of roughly 500 tokens, not whole documents. Chunking is structure-aware: a chunk never crosses a page or heading boundary, so a citation always points at one coherent place. Consecutive chunks overlap slightly, so a sentence that straddles a boundary is findable from either side. ### Embed [#embed] Every chunk goes through a sentence-embedding model (all-MiniLM-L6-v2) that maps text to a 384-dimensional vector where similar meanings land near each other. The model ships inside the binary, quantized to 23MB, and runs in pure Go. This is unusual and deliberate: it is why semantic search works on a fresh install with zero downloads, no Python, and no GPU. Identical text is never embedded twice; a content-addressed cache sees through file renames and re-indexes. Embedding is also built to lose fights for the machine: it fans out across all CPU cores minus two of headroom, and the daemon runs itself at background priority. An idle machine indexes at full speed; a busy one yields to whatever you are actually doing, at the cost of a somewhat longer first index. ### Store [#store] Everything lands in one SQLite file: the chunk text in an FTS5 full-text index, the vectors as blobs beside them. SQLite in WAL mode means searches keep answering while indexing writes. There is no server, no schema to migrate by hand, and backup is `cp`. ## Why two searches [#why-two-searches] Semantic search and keyword search fail in opposite ways. Embeddings capture meaning, so "agreement cancellation" finds a termination clause it shares no words with. But they blur exact identifiers: `MSA-2024-117` and `MSA-2024-118` embed nearly identically, and only one of them is your contract. Keyword search (BM25 over FTS5) nails identifiers, names, and rare terms, but has no idea that "cancel" and "terminate" are the same intent. So every query runs both, and the two rankings are merged with Reciprocal Rank Fusion: each result scores by its rank position in each list, which needs no tuning and no score normalization, since BM25 scores and cosine similarities live on unrelated scales. A result that both retrievers like rises to the top; a result only one likes still surfaces. ## Why citations are non-negotiable [#why-citations-are-non-negotiable] Retrieval is probabilistic. The ranker can be wrong, and when `hay ask` hands passages to a small local LLM, the model can misread them. Citations are the honesty mechanism: every result and every answer points at a file and page you can open. The design rule inside the codebase is that if output cannot be traced to a source, it does not ship. ## What the eval set is for [#what-the-eval-set-is-for] Retrieval quality regressions are invisible: the code compiles, tests pass, and results are quietly worse. Haypile carries a query set with expected results (in [`eval/`](https://github.com/BenyD/haypile/tree/main/eval)) that runs in CI on every change that touches retrieval. Chunk sizes, fusion constants, and model choices change only when that eval says the change is an improvement. During development it has already caught a real regression that keyword-tuning introduced; that is the mechanism working as intended. # Privacy, verified (/docs/explanation/privacy) Local-first tools usually ask for trust. Haypile tries to ask for less of it, by making its central privacy claim something you can measure. ## The claim [#the-claim] Haypile makes zero external network connections. Not few, not anonymized: zero. * Indexing, embedding, and search run entirely in-process. The embedding model is inside the binary; there is nothing to download and no API to call. * The daemon listens on localhost only. * There is no telemetry, no update check, no crash reporting. If that ever changes it will be opt-in, documented loudly, and off by default. This commitment is versioned with the code. ## How to check it [#how-to-check-it] ```sh hay status ``` ``` Outbound connections: 0 ``` That number is measured live against the running daemon's actual sockets, not asserted from a README. If you want a second opinion, ask the OS directly: ```sh lsof -p $(pgrep -f "hay serve") -i TCP ``` On Windows: ```powershell Get-NetTCPConnection -OwningProcess (Get-Process hay).Id ``` You will find listeners on localhost and nothing else. Being auditable by standard tools, on an open codebase, is the point. ## The boundaries, honestly drawn [#the-boundaries-honestly-drawn] Four cases involve the network, each explicitly yours to choose: **Installation.** Downloading the binary is a network act, once, from GitHub Releases. The install script does nothing else, and it is short enough to read first. **`hay llm setup`, `hay ask`, and scanned-page OCR.** Haypile ships no LLM. Answer generation talks to a server on your machine (Ollama, LM Studio, llama.cpp, Jan) over localhost, and OCR of scanned PDF pages sends page images to that same local server, nowhere else. `hay llm setup` can download Ollama and a model for you; it asks before every download. Once the model is on disk, asking questions is fully offline. **A cloud model, if you bring a key.** `hay ask --endpoint https://... --key sk-...` (or `HAYPILE_LLM_API_KEY`) points generation at a cloud API instead. Indexing and search stay local regardless; what leaves the machine is the retrieved passages for that one question, sent to the endpoint you named, only when you configure it. Keys are refused over plain http to anything that is not localhost, so a typo cannot leak the secret in cleartext. **MCP clients.** If you connect Claude Code or another cloud-backed agent to Haypile, the passages it retrieves become part of that agent's context, which the agent sends to its own provider. Haypile's behavior does not change; the agent's reach is what you are choosing. For a fully offline answer path, use `hay ask` with a local model. ## Why this matters more than a policy [#why-this-matters-more-than-a-policy] Privacy policies describe intentions and can change. Architecture describes capabilities and is harder to walk back. Haypile's design removes the capability: there is no code path that phones home, no account system, no server side at all. For client-confidential, medical, or just personal documents, "cannot" beats "will not". # Ask questions with a local LLM (/docs/guides/ask) `hay ask` retrieves the most relevant passages from your documents and has a local LLM answer from them, citing its sources. Haypile ships no LLM and never talks to the network; generation is delegated to a server running on your machine. ## The fast path [#the-fast-path] If you do not already run a local LLM: ```sh hay llm setup ``` This finds an already-running server (Ollama, LM Studio, llama.cpp, Jan), or installs and starts Ollama for you, asking before anything is downloaded. The one large download is the model itself, about 2GB. When it finishes: ```sh hay ask "what notice period does the vendor agreement require?" ``` ``` Answering with llama3.2:3b (http://localhost:11434/v1)… The vendor agreement requires a 60-day written notice period for termination [1]. Sources: [1] ~/Documents/contracts/vendor-deal.docx (chunk 2) [2] ~/Documents/contracts/meridian-msa.pdf (page 4) ``` ## Check the citations [#check-the-citations] The `[1]` markers are the point. The model is instructed to answer only from the retrieved passages and to cite which passage supports each claim. Small models sometimes blend sources or overreach anyway; the citations let you catch it in one glance instead of trusting blindly. If an answer matters, open the cited source. Model quality matters here. A 3B parameter model or larger holds together well; 1B models are noticeably sloppier at synthesis. Pick a specific model with: ```sh hay ask "..." --model qwen2.5:7b ``` When several models are loaded, `hay ask` chooses a text chat model and skips vision models like `llava` and `qwen3-vl`, which answer text questions noticeably worse. So installing a vision model for [scanned-PDF OCR](/docs/guides/scanned-pdfs) never changes what answers your questions. Override the choice any time with `--model`. ## Point at a specific server [#point-at-a-specific-server] Auto-detection probes the usual local ports (Ollama 11434, LM Studio 1234, llama.cpp 8080, Jan 1337). To use something else: ```sh hay ask "..." --endpoint http://localhost:8080/v1 ``` Or set it once: ```sh export HAYPILE_LLM_ENDPOINT=http://localhost:8080/v1 export HAYPILE_LLM_MODEL=my-model ``` ## Use a cloud model (bring your own key) [#use-a-cloud-model-bring-your-own-key] Any OpenAI-compatible API works, with your key: ```sh hay ask "..." --endpoint https://api.example.com/v1 --key sk-... ``` Or set `HAYPILE_LLM_API_KEY` alongside the endpoint. The boundary: your documents are indexed locally, always. Opting in sends only the retrieved passages for that one question, to the endpoint you chose. Keys are refused over plain http to anything that is not localhost, so a typo cannot leak the secret in cleartext. ## Without an LLM, nothing breaks [#without-an-llm-nothing-breaks] If no server is found, `hay ask` says so and shows the top passages for your question instead. Search never depends on an LLM, and neither does anything else in Haypile. ## Scope the retrieval [#scope-the-retrieval] `hay ask` accepts the same narrowing flags as search: ```sh hay ask "when is the filing deadline?" --tag acme --limit 8 ``` `--limit` controls how many passages are given to the model as context. More is not always better; 6 to 8 focused passages usually beat 20 loose ones. # Use Haypile from Claude Code (/docs/guides/claude-code) Haypile's daemon speaks MCP (Model Context Protocol), the standard way to expose tools to AI agents. Once connected, Claude Code can search your indexed documents whenever a question calls for it: "what does our Meridian contract say about termination?" becomes answerable inside your editor, grounded in your actual files, with citations. ## Connect Claude Code [#connect-claude-code] One command: ```sh claude mcp add --transport http haypile http://localhost:11500/mcp ``` That is the whole setup. Claude Code now sees two tools: * `search_documents`: hybrid search over everything you have indexed, returning cited passages * `list_sources`: what folders are indexed, so the agent knows what it can search Make sure something is indexed (`hay add ~/Documents`) and the daemon is running. It starts automatically on `hay add` and stays up. ## Per-project setup with hay init [#per-project-setup-with-hay-init] For a project or case folder, `hay init` writes a `.mcp.json` in the folder: ```sh cd ~/cases/acme-litigation hay init ``` Claude Code picks up `.mcp.json` automatically when opened in that folder. Anyone who clones or opens the project gets the integration without running the `claude mcp add` command themselves. ## Cursor and other editors [#cursor-and-other-editors] Any MCP client that supports the Streamable HTTP transport can use the same endpoint: `http://localhost:11500/mcp`. For clients that prefer launching a process (stdio transport), configure: ```json { "command": "hay", "args": ["mcp-stdio"] } ``` `hay mcp-stdio` bridges stdio to the daemon and auto-starts it if needed. ## MCP bundles [#mcp-bundles] Every release also ships one `.mcpb` MCP Bundle per platform, on the [releases page](https://github.com/BenyD/haypile/releases). An `.mcpb` is a zip with a manifest that MCP clients, the official MCP registry, and Smithery understand: install it and the client launches `hay mcp-stdio` itself, no separate binary install or config file needed. Useful for clients that install servers from a bundle rather than a command line; on a machine where you also want the CLI, the normal install is still the way. ## Discovery endpoints for agents [#discovery-endpoints-for-agents] An agent pointed at `haypile.sh` can learn all of this without scraping HTML: * Every docs page has a Markdown twin: append `.md` to the URL, or send `Accept: text/markdown`. [`/llms.txt`](https://haypile.sh/llms.txt) and [`/llms-full.txt`](https://haypile.sh/llms-full.txt) index them. * [`/.well-known/mcp.json`](https://haypile.sh/.well-known/mcp.json) is the MCP server card: packages to install and the local endpoint to connect to. * [`/.well-known/api-catalog`](https://haypile.sh/.well-known/api-catalog) (RFC 9727) points at the API and MCP documentation. * [`/.well-known/agent-skills/index.json`](https://haypile.sh/.well-known/agent-skills/index.json) lists the Haypile skill file agents can load. The docs site itself also exposes two in-browser (WebMCP) tools to agents that drive a browser: `search_haypile_docs` and `get_haypile_install_command`. ## A privacy note worth understanding [#a-privacy-note-worth-understanding] Haypile itself makes zero external connections; that does not change here. But an MCP client is a separate program with its own behavior: when Claude Code calls `search_documents`, the passages that come back become part of Claude's context, which is sent to Anthropic like the rest of your conversation. Connecting an AI agent to your documents is a choice about that agent, not a change in what Haypile does. For fully offline question answering, use `hay ask` with a local LLM instead. ## Try it [#try-it] Open a Claude Code session in a folder with an indexed project and ask something only your documents can answer: > what did we agree with the vendor about payment terms? Claude will call `search_documents`, read the cited passages, and answer from them. The citations flow through, so you can verify against the source file. # Configure a folder (/docs/guides/folders) Folders you work in deserve more than a bare `hay add`: a tag for scoped search, patterns for files that should never be indexed, and editor wiring. `hay init` sets all of that up, and a plain YAML file keeps it adjustable. ## Run the wizard [#run-the-wizard] ```sh cd ~/cases/acme-litigation hay init ``` ``` Setting up /Users/you/cases/acme-litigation Tag for filtered search [acme-litigation]: Exclude patterns, comma-separated [none]: drafts/**, *.bak Wrote /Users/you/cases/acme-litigation/.haypile.yml Make these docs available to AI tools here (Claude Code, Cursor)? [Y/n] Wrote /Users/you/cases/acme-litigation/.mcp.json (Claude Code will pick it up in this folder) Indexing /Users/you/cases/acme-litigation… Indexed 214 files (1892 chunks), 0 unchanged. Embedded 1892 chunks for semantic search (bundled/all-MiniLM-L6-v2). Done. Try: hay search "something in acme-litigation" ``` While it indexes, a live progress line shows the phase, fraction, and time left. Three questions, each with a sensible default. If no local LLM is detected it also offers `hay llm setup` at the end. For scripts and dotfiles, skip the questions entirely: ```sh hay init --yes --tag acme --exclude "drafts/**,*.bak" ``` ## The config file [#the-config-file] `hay init` writes `.haypile.yml` in the folder. It is short enough to read in full: ```yaml tag: acme-litigation exclude: - drafts/** - "*.bak" ``` * `tag` scopes searches: `hay search "deposition" --tag acme-litigation` * `exclude` takes gitignore-style glob patterns, matched against paths relative to the folder. `drafts/**` skips a subtree; `*.bak` skips matching files at any depth. Hidden directories like `.git` are always skipped without any pattern. * Machine-managed directories are always skipped too: `node_modules`, `vendor`, `venv`, `__pycache__`, `target`, `dist`, `build`, `coverage`, `Pods`, `DerivedData`. Indexing a code project should surface your README and design notes, not a thousand dependency READMEs. ## Edit it anytime, by hand [#edit-it-anytime-by-hand] The config file is the source of truth and the daemon watches it. Add an exclude pattern, save, and the matching files leave the index within seconds. Remove the pattern and they come back. No re-add, no restart, no command to remember. This also means the config travels with the folder: sync it, commit it to a repo, or copy it to another machine, and the same rules apply wherever Haypile indexes that folder. ## What init does and does not do [#what-init-does-and-does-not-do] `hay init` is a writer for the config plus the normal indexing you would get from `hay add`. It does not create a separate index or a different kind of source. A folder set up with `init` and a folder added with `add` behave identically afterwards; `init` just gives you the tag, excludes, and `.mcp.json` in one pass. # Manage, update, uninstall (/docs/guides/manage) Haypile runs itself: the daemon starts when needed and the index migrates on its own. This page covers the few lifecycle moments where you take the wheel. ## The daemon lifecycle [#the-daemon-lifecycle] The daemon is a single background process serving the API on `127.0.0.1:11500` and watching your indexed folders. `hay add`, `hay web`, and `hay mcp-stdio` start it automatically when it is not running; its address and pid live in `~/.haypile/daemon.json`. It runs at background priority (nice 10, below-normal on Windows) so indexing never wins a fight with your foreground work; queries are millisecond-scale and never notice. To run it in the foreground instead, with logs in your terminal: ```sh hay serve ``` To stop a background daemon, on any platform: ```sh hay stop ``` It exits gracefully: in-flight requests finish, the index closes cleanly, and the runtime file is removed. (`pkill -f "hay serve"` still works on macOS and Linux; `hay stop` is the way that also works on Windows, where a detached process cannot receive signals.) Stopping it costs you nothing permanent. Search and every other command keep working through direct index access; folder watching pauses until the next command starts the daemon again. ## Update [#update] However you installed, update the same way: ```sh brew upgrade hay # Homebrew curl -fsSL haypile.sh | sh # install script (replaces the binary) irm https://haypile.sh/install.ps1 | iex # Windows PowerShell ``` Then restart the daemon so the new binary serves: `hay stop` and run any `hay` command. (Any command from the new binary also retires an old daemon on sight, so forgetting this step costs nothing but a moment.) On Windows the installer stops a running daemon itself before swapping the binary. Your index needs nothing from you across versions. There are no migrations to run, and when an update improves text extraction, affected files re-index themselves on the next pass; unchanged text hits the embedding cache, so even a full re-index is quick. ## Uninstall [#uninstall] Four steps remove every trace. Haypile never modifies your documents, so this is the complete list: ```sh hay stop # 1. stop the daemon brew uninstall hay # 2. remove the binary (script installs: rm /usr/local/bin/hay) rm -rf ~/.haypile # 3. delete the index and runtime files ``` On Windows, in PowerShell: ```powershell hay stop Remove-Item -Recurse -Force "$env:LOCALAPPDATA\Programs\hay" # the binary Remove-Item -Recurse -Force "$env:USERPROFILE\.haypile" # the index and runtime files ``` 4. Folders set up with `hay init` may contain a `.haypile.yml` and `.mcp.json`; delete them if you do not want the config to apply on a future install. There is no account to close and nothing server-side to request deletion from, because nothing ever left your machine. # Scanned PDFs (OCR) (/docs/guides/scanned-pdfs) Some PDFs carry no text at all: scans, faxes, photographed contracts. Haypile detects those pages, renders them, and has your local vision model transcribe them. The result indexes and cites by page like any other PDF, and the page images go to your local LLM server over localhost, nowhere else. ## What you need [#what-you-need] A local OpenAI-compatible server with a vision-capable model. With Ollama: ```sh ollama pull qwen3-vl ``` `hay llm setup` offers this download at the end of setup, so most installs already said yes or no to it once. Without a vision model, nothing breaks: scanned pages index empty, and `hay add` says so: ``` ! 2 scanned pages indexed empty: no vision model is running. hay llm setup installs one; re-add this folder after. ``` ## Pick a model that transcribes, not narrates [#pick-a-model-that-transcribes-not-narrates] Not every vision model takes transcription seriously. General vision chat models (llava especially) tend to describe the page in their own words, and sometimes invent details that are not on it; that text then sits in your index and comes back in search results as if your document said it. Models with strong OCR training (qwen3-vl, minicpm-v, or a dedicated OCR model) write down the words on the page. After indexing a scan, search for a phrase you can see on it; a miss means the transcription is not faithful and a better model is worth the download. ## Which model gets used [#which-model-gets-used] OCR prefers a vision-looking model from whatever your server lists: names containing `ocr` first (Unlimited-OCR, HunyuanOCR), then vision families like `-vl`, `vision`, `llava`, `minicpm-v`. If none match, it falls back to the first chat model. Override the choice explicitly: ```sh export HAYPILE_OCR_MODEL=qwen3-vl hay add ~/scans ``` If the selected model rejects images, OCR turns itself off for the rest of the pass instead of failing every page; the files still index with whatever text they have. ## Heavier models for hard scans [#heavier-models-for-hard-scans] Dedicated OCR models read dense tables, small print, and messy scans better than general vision chat models. Baidu's Unlimited-OCR (MIT licensed, about 3B parameters) serves an OpenAI-compatible API through vLLM or SGLang; point Haypile at it like any other endpoint: ```sh export HAYPILE_LLM_ENDPOINT=http://localhost:8000/v1 hay add ~/scans ``` The model name contains `ocr`, so it is picked for transcription automatically. It wants a GPU; on a laptop without one, qwen3-vl through Ollama is the practical choice. ## How pages are chosen [#how-pages-are-chosen] Only pages with no extractable text and at least one image are OCRed, so normal digital PDFs never pay the cost. Each qualifying page is rendered at 150 DPI and transcribed once at indexing time; searching is as fast as always afterwards. Expect a few seconds per scanned page, depending on the model and your hardware. ## Verify it worked [#verify-it-worked] Index a scanned document, then search for text you can see on the page: ```sh hay add ~/scans/lease-2019.pdf hay search "security deposit" ``` A hit citing `lease-2019.pdf (page 3)` means the transcription is in the index. If a scan was indexed before a vision model was available, its pages are empty in the index. Re-index that source to redo them: ```sh hay remove ~/scans && hay add ~/scans ``` ## Turning it off [#turning-it-off] ```sh export HAYPILE_OCR=off ``` Scanned pages then index empty, exactly as if no model were running. # Search your documents (/docs/guides/search) `hay search` runs both semantic and keyword retrieval and merges the results. This guide shows how to use each strength and how to narrow results when your index grows. ## Search by meaning or by exact term [#search-by-meaning-or-by-exact-term] Both of these work well, for different reasons: ```sh hay search "what happens if we stop paying" # meaning: finds payment default clauses hay search "MSA-2024-117" # exact: finds that contract number ``` Semantic retrieval catches paraphrases and related concepts. Keyword retrieval (SQLite FTS5 with BM25 ranking) catches identifiers, names, and rare terms that embeddings blur. You never choose between them; every query runs both and the rankings are fused. ## Read the citations [#read-the-citations] ``` 1. ~/cases/acme/contract.pdf (page 12) ...the indemnity cap shall not exceed two million dollars... ``` PDFs cite a page, and each slide in a pptx cites its slide as a page. In an mbox email archive each message is one unit, so `mail.mbox (page 47)` means the 47th message in the archive; the indexed text leads with the Subject, From, and Date headers, which makes senders and subjects searchable too. Formats without fixed pages (Markdown, text, docx, HTML) cite a chunk position instead. The citation is the contract: if a result cannot tell you where it came from, it does not belong in the output. ## Narrow with tags [#narrow-with-tags] Tag folders when you add them, then scope searches: ```sh hay add ~/cases/acme --tag acme hay add ~/personal/notes --tag personal hay search "deposition schedule" --tag acme ``` A folder configured with `hay init` gets its tag from `.haypile.yml`, so you set it once per folder rather than remembering flags. ## Control the result count [#control-the-result-count] ```sh hay search "termination" --limit 25 ``` The default is 10. Results are ranked, so the first few are usually what you want. ## Freshness is automatic [#freshness-is-automatic] While the daemon runs (it starts automatically on `hay add`), saved files are re-indexed within seconds, deleted files leave the index, and new files in watched folders appear on their own. There is no re-index command because you should never need one. To check what is indexed right now: ```sh hay list ``` # Use the web UI (/docs/guides/web) Everything the CLI does, in a browser tab: ```sh hay web ``` This starts the daemon if it is not already running and opens `http://localhost:11500`. The page is served by the same process that powers `hay search` and `hay ask`, from assets embedded in the binary. There is no separate server, no build step, and nothing leaves your machine. ## Search and read [#search-and-read] Type in the search box and results appear as you type, each cited with its file and page. Click a citation to read the passage in place, with the surrounding text for context, without opening the original document. Press `/` anywhere to jump back to the search box. ## Ask [#ask] Ask a question and the answer streams in live, with the retrieved sources listed above it. This uses the same local LLM as `hay ask`; if you have not set one up yet, run: ```sh hay llm setup ``` ## Manage sources [#manage-sources] The Sources panel does what `hay add`, `hay list`, and `hay remove` do: see what is indexed, add a folder or file, or remove one. When adding, the path field suggests completions as you type, and Browse opens an in-page folder picker. Where the platform has a dialog helper (macOS, Windows, Linux with `zenity` installed), a native file dialog is available from inside the picker. Indexing runs in the background; you can close the panel and keep searching while it works. ## Localhost only [#localhost-only] The daemon binds to `127.0.0.1` and refuses requests from other hosts and origins, so the web UI is reachable only from your own machine. Like the rest of the free tier it is single-user by design; team access is a paid-tier concern, not a flag you can flip. # REST API (/docs/reference/api) The daemon serves JSON over HTTP on `localhost:11500`. It listens on localhost only and has no auth in v1; do not expose it beyond the machine. Requests from other hosts or browser origins are refused with `403`. ## POST /api/query [#post-apiquery] Search. This is the endpoint your scripts want. ```sh curl -s -X POST localhost:11500/api/query \ -d '{"query": "termination clause", "tag": "", "limit": 5}' ``` Request: | Field | Type | Meaning | | ------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `query` | string | Required. Search text | | `tag` | string | Optional. Restrict to folders with this tag | | `limit` | int | Optional. Max results, default 10, cap 100 | | `mode` | string | Optional. `search` (default) returns only results that clear the relevance floors, possibly none. `answer` falls back to the nearest chunks instead of returning empty; it is what `hay ask` uses for retrieval | Response: ```json { "results": [ { "path": "/Users/you/Documents/contracts/msa.pdf", "page": 4, "chunk": 11, "snippet": "Either party may terminate...", "text": "Either party may terminate this Agreement...", "score": 0.0322 } ] } ``` `page` is 1-based and present for paginated formats; `0` means the format has no pages and `chunk` is the citation. `snippet` is the short display excerpt; `text` is the chunk's full text, for callers that answer or reason from the result. ## POST /api/ask [#post-apiask] Retrieval plus a streamed answer from your local LLM, as server-sent events. This is what `hay web` uses; point `curl -N` or an SSE client at it. ```sh curl -sN -X POST localhost:11500/api/ask \ -d '{"question": "what notice period does the vendor agreement require?"}' ``` Request: | Field | Type | Meaning | | ---------- | ------ | -------------------------------------------------------- | | `question` | string | Required. The question to answer | | `tag` | string | Optional. Restrict retrieval to folders with this tag | | `limit` | int | Optional. Passages given to the model, default 6, cap 20 | The response is `text/event-stream` with four event types, in order: | Event | Data | Meaning | | --------- | ----------------------------- | --------------------------------------------- | | `sources` | array of results with `label` | The retrieved passages, always first | | `token` | JSON string | One piece of the answer, repeated | | `error` | `{"message": "..."}` | Generation failed; the stream ends after this | | `done` | `{}` | Always last on success | Requires an OpenAI-compatible server, same as `hay ask`; when none answers, the response is `503` before any stream starts, so clients can branch on the status code. ## GET /api/chunk [#get-apichunk] The passage behind a citation, with its neighbors for context. The web UI's "read in place" view. ```sh curl -s "localhost:11500/api/chunk?path=/Users/you/Documents/msa.pdf&chunk=11&window=1" ``` | Param | Meaning | | -------- | ------------------------------------------------------ | | `path` | Required. Source file path as returned by `/api/query` | | `chunk` | Required. Chunk ordinal from the result | | `window` | Neighboring chunks on each side, 0 to 5, default 1 | ## GET /api/health [#get-apihealth] Liveness plus identity. Clients use `db` to confirm they are talking to the daemon for the right index. ```json { "ok": true, "version": "0.3.1", "db": "/Users/you/.haypile/haypile.db", "model": "bundled/all-MiniLM-L6-v2" } ``` ## GET /api/status [#get-apistatus] Everything `hay status` shows: the health fields plus uptime, sources, counts, pending indexing jobs, and the measured outbound connection count. | Field | Meaning | | ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `uptime_seconds` | Seconds since the daemon started | | `sources` | Indexed folders, each with `path`, `tag`, `files`, `chunks` | | `files`, `chunks` | Totals across all sources | | `pending_jobs` | Watcher changes queued for re-indexing | | `outbound_connections` | The daemon's live non-listening TCP connections, measured | | `outbound_note` | Present only when the count could not be measured; explains why | | `indexing` | Present only while an add pass runs: `phase` (`extracting` or `embedding`), `files_done`/`files_total`, `bytes_done`/`bytes_total`, `chunks_done`/`chunks_total` | `indexing` is what `hay add` polls to draw its live progress bar; scripts can poll it the same way. ## POST /api/shutdown [#post-apishutdown] Asks the daemon to exit gracefully; this is what `hay stop` calls. Responds `{"stopping": true}`, finishes in-flight requests, closes the index cleanly, and removes the runtime file on the way out. ## GET /api/sources [#get-apisources] Indexed folders with counts. ## POST /api/sources [#post-apisources] Index and watch a folder or file. Runs synchronously and returns the indexing stats. ```sh curl -s -X POST localhost:11500/api/sources \ -d '{"path": "/Users/you/Documents", "tag": "personal"}' ``` ## DELETE /api/sources [#delete-apisources] Un-index and un-watch. Body: `{"path": "..."}`. Returns `{"removed": true}` or `false` if the path was not indexed. ## GET /api/browse and POST /api/pick [#get-apibrowse-and-post-apipick] Helpers for the web UI's folder picking; scripts rarely need them. `GET /api/browse?path=/abs/dir` lists a directory's subfolders and indexable files (defaults to the home directory). `POST /api/pick?kind=folder|file` opens the native OS file dialog and returns the chosen path; `204` means the user canceled, `501` means the platform has no dialog helper. ## POST /mcp [#post-mcp] The MCP endpoint (Streamable HTTP transport, JSON-RPC 2.0). Supports `initialize`, `tools/list`, `tools/call`, and `ping`. Tools: | Tool | Arguments | Returns | | ------------------ | ---------------------------------- | --------------------------- | | `search_documents` | `query` (required), `tag`, `limit` | Cited passages as text | | `list_sources` | none | Indexed folders with counts | Point any MCP client at `http://localhost:11500/mcp`, or use `hay mcp-stdio` for stdio-transport clients. ## Errors [#errors] Non-200 responses carry `{"error": "message"}`. The codes in use: | Code | When | | ----- | --------------------------------------------------------------------------------- | | `400` | Malformed request | | `403` | Request from another host or browser origin | | `404` | `/api/browse` on an unreadable path, `/api/chunk` for a chunk that does not exist | | `405` | Wrong method, e.g. `GET /mcp` | | `409` | `/api/pick` while a dialog is already open | | `503` | `/api/ask` when no LLM endpoint answers | | `500` | Server fault | MCP tool failures come back inside the JSON-RPC result with `isError: true` so agent models can read and react to them. # CLI commands (/docs/reference/cli) The binary is `hay`. Any command that needs the daemon starts it automatically; you never manage it by hand. ## hay init \[folder] [#hay-init-folder] Per-folder setup: writes `.haypile.yml`, indexes the folder, optionally writes `.mcp.json` and offers LLM setup. Defaults to the current directory. | Flag | Meaning | Default | | ----------- | --------------------------------- | ----------- | | `--tag` | Tag for filtered search | folder name | | `--exclude` | Comma-separated glob patterns | none | | `--mcp` | Write `.mcp.json` for MCP clients | true | | `--yes` | Accept all defaults, no prompts | false | ## hay add \ [#hay-add-path] Indexes a folder (recursively) or a single file, and watches it for changes. Unchanged files are skipped on re-add; identical content is never embedded twice, even across files. | Flag | Meaning | Default | | ------- | ----------------------- | ------------------------------ | | `--tag` | Tag for filtered search | from `.haypile.yml`, else none | Supported formats: `.pdf`, `.docx`, `.pptx`, `.md`, `.markdown`, `.txt`, `.html`, `.htm`, `.mbox`. ## hay web [#hay-web] Opens the bundled local web UI in your browser: search as you type, ask with streamed answers, and click any citation to read the passage in place. Starts the daemon if needed; everything is served from `localhost:11500`. | Flag | Meaning | Default | | -------------- | ------------------------------------------ | ------- | | `--no-browser` | Print the URL instead of opening a browser | false | ## hay search "\" [#hay-search-query] Hybrid retrieval: semantic and keyword legs run in parallel and the rankings are fused. Results cite file and page. | Flag | Meaning | Default | | --------- | --------------------------------- | ------- | | `--tag` | Only search folders with this tag | all | | `--limit` | Maximum results | 10 | ## hay ask "\" [#hay-ask-question] Retrieves relevant passages and has a local LLM answer from them with citations. Requires an OpenAI-compatible server (auto-detected); without one it prints the top passages instead. | Flag | Meaning | Default | | ------------ | ------------------------------------------------ | ------------------------------ | | `--endpoint` | OpenAI-compatible base URL | auto-detect | | `--model` | Model to request | first chat model listed | | `--key` | API key for the endpoint, sent as a Bearer token | none, or `HAYPILE_LLM_API_KEY` | | `--tag` | Only retrieve from folders with this tag | all | | `--limit` | Passages given to the model | 6 | ## hay list [#hay-list] Indexed folders with file and chunk counts. ## hay remove \ [#hay-remove-path] Un-indexes a source and stops watching it. The path must match what was added (see `hay list`). ## hay status [#hay-status] Daemon state, index location, counts, model, queued indexing jobs, and the measured outbound connection count. ## hay stop [#hay-stop] Stops the background daemon gracefully: in-flight requests finish, the index closes cleanly, and the runtime file is removed. Folder watching pauses until the next command starts the daemon again; search keeps working through direct index access. Works on every platform, including Windows, where a detached process cannot receive signals. ## hay serve [#hay-serve] Runs the daemon in the foreground: REST API and MCP on `localhost:11500`, plus the folder watcher. Usually you never run this yourself. | Flag | Meaning | Default | | -------- | ------------ | --------- | | `--host` | Bind address | 127.0.0.1 | | `--port` | API port | 11500 | Binding beyond localhost prints a loud warning: the API has no auth in v1. ## hay llm setup [#hay-llm-setup] Guided path to a working local LLM for `hay ask`: detects running servers, installs and starts Ollama with your confirmation, downloads a recommended model with your confirmation, verifies with a real request. Finishes by offering a vision model (qwen3-vl) so scanned PDFs are searchable too; skipping it costs nothing except OCR. | Flag | Meaning | Default | | --------- | --------------------------------- | ----------- | | `--model` | Model to download if none present | llama3.2:3b | | `--yes` | Answer yes to all prompts | false | ## hay mcp-stdio [#hay-mcp-stdio] MCP stdio transport for clients that launch a process. Bridges stdin/stdout to the daemon's `/mcp` endpoint, auto-starting the daemon if needed. ## hay completion \ [#hay-completion-shell] Prints a shell autocompletion script for bash, zsh, fish, or powershell. `hay completion zsh --help` shows the install one-liner for your shell. ## Environment variables [#environment-variables] | Variable | Meaning | | ------------------------ | ------------------------------------------------------------------------------------------------------------ | | `HAYPILE_DIR` | Data directory (index database, daemon runtime file). Default `~/.haypile` | | `HAYPILE_ADDR` | Daemon listen address, overrides host/port flags | | `HAYPILE_NO_DAEMON` | `1` disables daemon auto-start and routing (direct index access) | | `HAYPILE_LLM_ENDPOINT` | OpenAI-compatible base URL for `hay ask` and scanned-page OCR | | `HAYPILE_LLM_MODEL` | Model name for `hay ask` | | `HAYPILE_LLM_API_KEY` | Bearer token for LLM endpoints that require one (refused over plain http off-machine) | | `HAYPILE_OCR` | `off` disables OCR of scanned PDF pages | | `HAYPILE_OCR_MODEL` | Vision model to use for scanned-page OCR (default: first vision-looking model listed, else first chat model) | | `HAYPILE_EMBED_ENDPOINT` | Optional external embedding server (replaces the bundled model) | | `HAYPILE_EMBED_MODEL` | Model name for the embedding endpoint | | `HAYPILE_MODEL_PATH` | Dev builds: path to embedding weights on disk | # Configuration (/docs/reference/configuration) Haypile needs no configuration to work. When you want control, there are exactly two places to look: a per-folder YAML file and a handful of environment variables. ## .haypile.yml [#haypileyml] Lives in a source folder, written by `hay init`, editable by hand. The daemon watches it: saving a change re-syncs the index within seconds. ```yaml tag: acme-litigation exclude: - drafts/** - "*.bak" - "**/archive/**" ``` ### tag [#tag] Applied to everything indexed under this folder. Search with `--tag` to scope results. A tag passed explicitly to `hay add --tag` wins over the config. ### exclude [#exclude] Glob patterns matched against paths relative to the folder, gitignore flavored: | Pattern | Matches | | --------------- | ------------------------------------------------------- | | `drafts/**` | everything under the `drafts` subtree | | `*.bak` | `.bak` files at any depth (bare names match everywhere) | | `**/archive/**` | anything under any directory named `archive` | Adding a pattern removes already-indexed matching files from the index on the next sync. Removing a pattern brings them back. A malformed pattern or broken YAML fails the indexing pass loudly rather than silently indexing everything. Hidden directories (`.git` and anything else starting with a dot) are always skipped; you do not need to exclude them. ## The data directory [#the-data-directory] Everything Haypile stores lives in one place: | File | What it is | | ------------------------ | -------------------------------------------------------------- | | `~/.haypile/haypile.db` | The entire index: files, chunks, vectors, FTS. One SQLite file | | `~/.haypile/daemon.json` | Runtime file: the running daemon's address and pid | On Windows the same directory is `%USERPROFILE%\.haypile`. Set `HAYPILE_DIR` to relocate it. Deleting the directory deletes the index and nothing else; your documents are never touched. Back it up by copying one file. ## Replacing the embedding model [#replacing-the-embedding-model] The bundled model is the default and the right choice for almost everyone. If you run an embedding server (Ollama and others expose one), `HAYPILE_EMBED_ENDPOINT` and `HAYPILE_EMBED_MODEL` switch Haypile to it, trading the zero-setup guarantee for a larger model. Vectors from different models are not comparable, so an index sticks with the model that embedded it: pointing an existing index at a different model is an error, not silent degradation. To switch, re-index (`hay remove` then `hay add`), or keep a separate index for the experiment with `HAYPILE_DIR`. ## Environment variables [#environment-variables] See the [CLI reference](/docs/reference/cli#environment-variables) for the full table. The two most useful: * `HAYPILE_DIR`: keep separate indexes (for tests, for work vs personal) by pointing this at different directories. * `HAYPILE_NO_DAEMON=1`: force direct index access with no background process, useful in scripts and CI. # Troubleshooting (/docs/reference/troubleshooting) Symptoms first, in the words the tool uses. If yours is not here, the codebase is open and the error messages say what file they came from. ## "listen 127.0.0.1:11500 (already running?)" [#listen-12700111500-already-running] `hay serve` found the port taken. Usually a daemon is already running, which is fine: every command talks to it automatically, and you do not need a second one. If something else owns the port, move Haypile: ```sh hay serve --port 11600 # one-off export HAYPILE_ADDR=127.0.0.1:11600 # permanent ``` ## Commands ignore the running daemon [#commands-ignore-the-running-daemon] The CLI refuses to route through a daemon serving a different database, which happens when the daemon was started with a different `HAYPILE_DIR`. It falls back to direct index access, so results are still correct; watching is what you lose. Fix by restarting the daemon under the environment you want: ```sh hay stop hay add ~/Documents # restarts it with the current HAYPILE_DIR ``` ## "unsupported format (want .docx .htm .html .markdown .mbox .md .pdf .pptx .txt)" [#unsupported-format-want-docx-htm-html-markdown-mbox-md-pdf-pptx-txt] You pointed `hay add` at a single file of a type Haypile cannot parse. This errors only for single files, because you named that file deliberately; folder indexing skips unsupported files silently. ## `hay add` has been running for a long time [#hay-add-has-been-running-for-a-long-time] Working as designed: `hay add` waits for the whole pass, however long it takes, with no timeout. A large folder of PDFs can legitimately take an hour; the live progress line (or, in a pipe, the milestone lines) shows the phase, the fraction done, and the time left. Indexing also deliberately yields to your foreground work: the daemon runs at background priority and embedding leaves two CPU cores free, so a busy machine indexes slower than an idle one. If the progress line has genuinely stopped moving, check from another terminal: `curl -s localhost:11500/api/status` reports the in-flight pass under `indexing`. ## "Warning: N files could not be read and were skipped." [#warning-n-files-could-not-be-read-and-were-skipped] Some files failed to open or parse: corrupt PDFs, permission problems, truncated downloads. One bad document never aborts an indexing pass, and if a previously indexed version of the file exists, it stays searchable until a readable version appears. ## A scanned PDF returns no results [#a-scanned-pdf-returns-no-results] Pages that are images need a local vision model to transcribe them; without one they index empty. See [Scanned PDFs (OCR)](/docs/guides/scanned-pdfs). After setting a model up, re-index the source: `hay remove && hay add `. ## `hay init` indexed fewer files than expected [#hay-init-indexed-fewer-files-than-expected] Machine-managed directories are skipped on purpose: `node_modules`, `vendor`, `venv`, `__pycache__`, `target`, `dist`, `build`, `coverage`, `Pods`, `DerivedData`. Indexing a code project should surface your README and design notes, not thousands of dependency READMEs. Hidden directories like `.git` are skipped too. If a document you want lives in one of these folders, move it elsewhere or index that file directly with `hay add `. ## "no local LLM endpoint found" [#no-local-llm-endpoint-found] `hay ask` needs an OpenAI-compatible server and probed the usual ports without finding one. Run `hay llm setup` for the guided path, or point at your server: `hay ask --endpoint http://localhost:PORT/v1`. Search never needs an LLM; only `ask` does. ## Saved changes are not showing up in search [#saved-changes-are-not-showing-up-in-search] Check that the daemon is running with `hay status`; watching only happens while it is up. Files re-index within seconds of saving, so if the daemon is up and a result is still stale, re-run `hay add `. It is cheap (unchanged files are skipped) and reconciles everything, including deletions. ## Index seems wrong in a way you cannot name [#index-seems-wrong-in-a-way-you-cannot-name] The reset is safe and complete, because the index is derived data: ```sh hay stop rm -rf ~/.haypile # Windows: Remove-Item -Recurse -Force "$env:USERPROFILE\.haypile" hay add ``` Your documents are untouched; only the index is rebuilt.