Building a vault index so your AI agent reads Obsidian fast

An Obsidian vault gives your agent a knowledge base, but there’s a scaling problem hiding in it. When the vault is small, the agent reads notes directly and all is well. Once it grows to dozens of notes, having the agent crawl them one by one to find something is slow and burns tokens on every search. The fix is a vault index for your AI agent: one compact file that maps the entire vault, so the agent reads a single map instead of walking every shelf. This is the technical companion to the Obsidian knowledge base article — here we build the index that makes a large vault fast.

The problem: crawling doesn’t scale

Picture a vault of forty-plus notes. You ask the agent something that touches three of them. Without an index, the agent has no map — so to answer reliably it may read many notes in sequence, each one a separate tool call, each one pulling text into its context window. That’s slow, and because you pay per token, it’s needlessly expensive. Worse, a big pile of note contents can crowd the agent’s working context and push out the thing you actually asked about.

The insight is that the agent rarely needs the full text of every note to decide where to look. It needs a map: what notes exist, what each is about, and how they connect. Give it that map in one compact file and it can pinpoint the two or three notes that matter, then read only those. One cheap read replaces dozens of expensive ones.

What the index contains

The index is a single machine-readable file — JSON works well — that summarises every note in the vault. For each note it records the structural metadata an agent needs to navigate, without the full body text:

• Title and aliases — how the note is named and referred to.
• Tags — for topic-based lookup.
• Headings — the note’s internal structure, so the agent knows what’s inside without reading it.
• Wikilinks (resolved) — which notes this note points to, mapped to real file paths.
• Backlinks — which notes point to this one.
• Word count — a quick sense of how big a read will be if the agent opens it.

All of that lives in one file, typically written to a dedicated location inside the vault (for example a _meta/ folder). The agent reads that one file and immediately has a model of the whole vault — titles, topics, structure, and the link graph — for the cost of a single tool call.

The clever part: resolving wikilinks and backlinks

The feature that turns a flat list into a genuine knowledge graph is link resolution. In Obsidian you link notes with [[double-bracket]] syntax — but [[Some Note]] is just a display name, not a file path. On its own it doesn’t tell an agent where the target actually lives.

The indexer resolves each wikilink to the real note it points at, and — just as importantly — computes the reverse: for every note, which other notes link to it. That gives you backlinks, the “who references this?” relationship. Together, resolved forward links and backlinks are exactly what a knowledge graph needs: the agent can start at one note and traverse to everything related, in either direction, using the index alone. A vault with zero broken links is a clean graph; the indexer will also surface any unresolved links so you can fix them.

The indexer script

The index is produced by a small script that walks the vault, reads each note’s metadata, resolves the links, and writes the single output file. It’s deliberately simple — standard-library Python, no heavy dependencies, and it runs in well under a second even on a vault of dozens of notes. A few design choices make it robust:

• Path priority. It finds the vault by checking, in order: an explicit argument you pass, then the vault-path setting the agent uses, then a sensible fallback. That way the same script works from the command line and when the agent runs it.
• Skip the noise. It ignores folders that aren’t knowledge — the .obsidian config, version-control folders, the trash, the _meta output folder itself, and any dependency folders.
• One output file. Everything lands in a single JSON file in the vault’s meta folder, so there’s exactly one thing for the agent to read.
• No dependencies required. It runs on the Python standard library alone. If PyYAML happens to be installed it uses it for frontmatter; if not, a lightweight built-in parser handles aliases and tags, so the script works either way.
• Links to attachments count too. A [[link]] to a PDF or a script that actually exists in the vault is resolved to that file rather than flagged as broken — only genuinely missing targets end up in the unresolved list.

The full script is below. Save it, point it at your vault, and run it once to generate your first index.

#!/usr/bin/env python3
"""
index_vault.py  --  Build an AI-readable map of an Obsidian vault.

- Walks every .md file in the vault ONCE.
- Extracts per note: title, aliases, tags, headings, outgoing [[wikilinks]], word count.
- Resolves wikilinks to real file paths (including aliases, same as Obsidian).
- Inverts the links -> computes backlinks.
- Writes everything to  <vault>/_meta/index.json

No third-party dependencies (Python stdlib only). PyYAML is used if present;
otherwise it falls back to a lightweight frontmatter parser.

Usage:
    python index_vault.py "C:\path\to\Vault"
Or set OBSIDIAN_VAULT_PATH in the environment and run without arguments.
"""

import sys
import os
import re
import json
from datetime import datetime, timezone
from pathlib import Path

# --- Config ---
VAULT = None  # Hardcoded fallback (override with CLI argument or env var)
OUTPUT_RELATIVE = os.path.join("_meta", "index.json")
# Directories we skip (Obsidian config, git, and our own meta output)
SKIP_DIRS = {".obsidian", ".git", ".trash", "_meta", "node_modules"}


def print_help():
    print(__doc__)
    print("Vault path resolution (priority):")
    print('  1. CLI argument:     python index_vault.py "C:\path\to\Vault"')
    print("  2. Environment var:  set OBSIDIAN_VAULT_PATH=C:\path\to\Vault")
    print("  3. Hardcoded:        edit VAULT in this script")
    print()
    print(f"Output: <vault>/{OUTPUT_RELATIVE}")


# --- Frontmatter parsing ---
FRONTMATTER_RE = re.compile(r"^---\s*\n(.*?)\n---\s*\n", re.DOTALL)

try:
    import yaml  # optional
    _HAVE_YAML = True
except Exception:
    _HAVE_YAML = False


def parse_frontmatter(text):
    """Return (frontmatter_dict, body). Body = text without frontmatter."""
    m = FRONTMATTER_RE.match(text)
    if not m:
        return {}, text
    raw = m.group(1)
    body = text[m.end():]
    if _HAVE_YAML:
        try:
            data = yaml.safe_load(raw) or {}
            if isinstance(data, dict):
                return data, body
        except Exception:
            pass
    return _light_frontmatter(raw), body


def _light_frontmatter(raw):
    fm = {}
    for line in raw.splitlines():
        for key in ("aliases", "alias", "tags", "tag"):
            if line.strip().lower().startswith(key + ":"):
                val = line.split(":", 1)[1].strip()
                norm = "aliases" if key in ("aliases", "alias") else "tags"
                if val.startswith("[") and val.endswith("]"):
                    items = [v.strip().strip("'\"") for v in val[1:-1].split(",")]
                    fm[norm] = [i for i in items if i]
                elif val:
                    fm[norm] = [val.strip("'\"")]
                else:
                    fm[norm] = []  # probably a YAML list on the lines below
    return fm


def as_list(value):
    if value is None:
        return []
    if isinstance(value, list):
        return [str(v).strip() for v in value if str(v).strip()]
    if isinstance(value, str):
        parts = re.split(r"[,\s]+", value.strip())
        return [p for p in parts if p]
    return [str(value)]


# --- Content parsing ---
WIKILINK_RE = re.compile(r"\[\[([^\]]+?)\]\]")
HEADING_RE = re.compile(r"^(#{1,6})\s+(.*)$", re.MULTILINE)
INLINE_TAG_RE = re.compile(r"(?:^|\s)#([A-Za-z0-9_][A-Za-z0-9_/\-]*)")
CODE_FENCE_RE = re.compile(r"```.*?```", re.DOTALL)
INLINE_CODE_RE = re.compile(r"`[^`\n]*`")


def extract_links(body):
    """Return list of targets from [[...]] -- alias/heading stripped.
    Wikilinks inside inline backticks or code fences are skipped."""
    body = CODE_FENCE_RE.sub(" ", body)
    body = INLINE_CODE_RE.sub(" ", body)
    links = []
    for raw in WIKILINK_RE.findall(body):
        target = raw.split("|", 1)[0]
        target = target.split("#", 1)[0]
        target = target.strip()
        if target:
            links.append(target)
    return links


def extract_headings(body):
    return [h.strip() for _, h in HEADING_RE.findall(body)]


def extract_inline_tags(body):
    cleaned = CODE_FENCE_RE.sub("", body)
    return list({t for t in INLINE_TAG_RE.findall(cleaned)})


# --- Main ---
def main():
    if len(sys.argv) > 1 and sys.argv[1] in ("-h", "--help"):
        print_help()
        return

    # Priority: CLI argument > env var > hardcoded VAULT
    vault = VAULT
    if len(sys.argv) > 1:
        vault = sys.argv[1]
    if not vault:
        vault = os.environ.get("OBSIDIAN_VAULT_PATH")
    if not vault:
        print("Pass the vault path or set OBSIDIAN_VAULT_PATH in your environment.")
        sys.exit(1)

    vault = Path(vault).resolve()
    if not vault.is_dir():
        print(f"Vault not found: {vault}")
        sys.exit(1)

    # 1) Collect all markdown files + other files (for link resolution)
    md_files = []
    all_files = {}          # lowercase relpath -> relpath (PDFs, scripts, ...)
    file_stem_to_path = {}  # stem.lower() -> relpath (first one wins)

    for root, dirs, files in os.walk(vault):
        dirs[:] = [d for d in dirs if d not in SKIP_DIRS]
        for f in files:
            rel = str((Path(root) / f).relative_to(vault)).replace("\\", "/")
            if f.lower().endswith(".md"):
                md_files.append(Path(root) / f)
            else:
                all_files.setdefault(rel.lower(), rel)
                file_stem_to_path.setdefault(Path(f).stem.lower(), rel)

    # 2) First pass: parse each file, build resolution maps
    notes = {}
    basename_to_path = {}  # "Note B" -> "folder/Note B.md"
    alias_to_path = {}

    for path in md_files:
        rel = str(path.relative_to(vault)).replace("\\", "/")
        try:
            text = path.read_text(encoding="utf-8")
        except Exception as e:
            print(f"  ! could not read: {rel} ({e})")
            continue

        fm, body = parse_frontmatter(text)
        aliases = as_list(fm.get("aliases", fm.get("alias")))
        fm_tags = as_list(fm.get("tags", fm.get("tag")))
        tags = sorted(set(fm_tags) | set(extract_inline_tags(body)))

        basename = path.stem
        title = basename  # Obsidian convention: title = file name

        notes[rel] = {
            "title": title,
            "aliases": aliases,
            "tags": tags,
            "headings": extract_headings(body),
            "links_raw": extract_links(body),
            "links": [],
            "backlinks": [],
            "unresolved": [],
            "words": len(re.findall(r"\w+", body)),
        }

        basename_to_path.setdefault(basename.lower(), rel)
        for a in aliases:
            alias_to_path.setdefault(a.lower(), rel)

    # 3) Second pass: resolve links + fill backlinks
    for rel, note in notes.items():
        resolved = []
        for raw in note["links_raw"]:
            key = raw.lower()
            target = (
                basename_to_path.get(key)
                or alias_to_path.get(key)
                or basename_to_path.get(Path(raw).stem.lower())
            )
            if not target:
                raw_path = raw.replace("\\", "/").lower()
                target = all_files.get(raw_path) or file_stem_to_path.get(Path(raw).stem.lower())
            if target:
                resolved.append(target)
                if target in notes:
                    notes[target]["backlinks"].append(rel)
            else:
                note["unresolved"].append(raw)
        note["links"] = sorted(set(resolved))
        del note["links_raw"]

    for note in notes.values():
        note["backlinks"] = sorted(set(note["backlinks"]))

    # 4) Write the map
    out_path = vault / OUTPUT_RELATIVE
    out_path.parent.mkdir(parents=True, exist_ok=True)
    payload = {
        "generated": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "vault": str(vault),
        "note_count": len(notes),
        "notes": notes,
    }
    out_path.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
    print(f"OK  {len(notes)} notes indexed -> {out_path}")


if __name__ == "__main__":
    main()

Run it from your project folder with the uv Python (see the foundation article on why you use uv rather than a bare python command):

uv run python index_vault.py

Index-first: how the agent uses it

Generating the index is half the job; the other half is teaching the agent to use it first. The rule to give Hermes is simple: before crawling individual notes, read the index. From that one file it can decide which specific notes are worth opening, then read only those.

That “index-first” instruction is what actually delivers the speed and cost win. Without it, the agent might still crawl; with it, the index becomes the default entry point to the whole vault. As with other durable rules in this series, the way to make it stick is to state it, have the agent repeat it back, and save it — as part of the vault skill, so every session starts index-first.

Keeping it fresh

An index is a snapshot. The moment you add or edit notes, it starts to drift from reality — so it needs regenerating. Two approaches work together:

• Staleness check. Have the agent check the index’s age before trusting it. If it’s older than a set threshold — say a day — it offers to re-index before answering, so you never act on a stale map.
• Scheduled rebuild. Better still, regenerate the index automatically on a timer, so it’s simply always current. A daily rebuild in the early morning keeps it fresh with no effort from you.

That scheduled rebuild is a perfect first real job for the agent’s task scheduler — a small, self-contained script that runs on a cron-style timer. Setting that up is covered in the scheduled tasks article, and the vault index is exactly the kind of maintenance work it’s built for.

Frequently asked questions

Why JSON instead of letting the agent read the notes?

Because JSON is compact and structured. A single JSON map of forty notes is a fraction of the size of forty note bodies, and the agent can parse it in one read. It gets the whole picture — titles, tags, links — without pulling every note’s full text into its context.

Does this replace the agent’s memory?

No — they’re complementary. Memory holds compact, always-loaded facts. The vault holds detailed knowledge, and the index makes that knowledge reachable cheaply. Together they keep the agent’s working context lean while still giving it access to everything.

What happens if a wikilink is broken?

The indexer flags it as unresolved rather than silently dropping it. That’s useful — a broken link is usually a typo or a renamed note, and seeing the list lets you fix it, keeping the graph clean.

Do my specialist bots get the index too?

If you run separate bot profiles, each can be pointed at the same index, so they share the vault’s knowledge. The index is just a file; anything you give the path and the index-first rule to can use it.

What you’ve built

You now have a vault index: one compact JSON map of your entire knowledge base, with resolved links and backlinks, that the agent reads first instead of crawling every note. It’s what keeps a growing vault fast and cheap, and what stops the vault’s detail from bloating the agent’s memory.

Next up: now that the agent has a brain and a knowledge base, it’s time to give it hands. The next article covers tools — what they are, the ~86 Hermes ships with, and how to enable them safely. This article is part of the Agentic AI series; the gentler knowledge base article is the place to start if you haven’t set up a vault yet.