🔍 Semantic Search & Indexing

On My Disk offers two ways to run search on your files: OnMyChat (advanced semantic vector search powered by ChromaDB) or PeARS (decentralized semantic search engine based on compact models).

1. Semantic Search with OnMyChat (ChromaDB)

When running the personal AI Node (OnMyChat), the system automatically deploys a local ChromaDB instance to power vector-based semantic search.

Instead of matching exact keywords, semantic search understands the meaning behind your query. For example, searching for “income” will retrieve files mentioning “revenue”, “salary”, or “financial reports”.

How Indexing Works

  1. Real-time Event Watcher: When you add or modify files in your shares, a filesystem watcher detects the change and queues the file for indexing.
  2. C++ Background Crawler: An idle network crawler periodically scans your files to build or repair the index.
  3. Python Indexer (search_node.py): The indexing task is run asynchronously in the background. Complex documents (PDF, DOCX, ODT, etc.) are converted to clean text via pandoc, split into chunks, vectorized using the sentence embedding model (all-MiniLM-L6-v2), and stored in the vector database.

The Index Collections

The database is structured using a Twin-Collection Architecture:

  • omd_search Collection (stored under memory_index/search_index/): Contains full-text indices of user files.
  • Thematic / shared collections (like omd): Created manually by importing documents, URLs, etc., for later referring in chats with the AI assistant. For example, the omd collection contains On My Disk documentation and should be present in your setup by default. See the RAG section for details.

2. PeARS Integration (Alternative Semantic Search Engine)

PeARS (People’s Search Engine) is also a semantic search engine. Unlike traditional keyword-matching systems, PeARS creates compact, optimized semantic signatures of documents using lightweight embedding models. This makes it extremely non-demanding on hardware, allowing it to run smoothly on low-power devices like single-board computers (SBCs) or desktop machines.

By index sharing, PeARS enables collaborative peer-to-peer web search without relying on centralized, data-collecting search giants. You can learn more about its design principles at pearsproject.org.

Setup Steps

  1. Install and run PeARS according to its user manual: PeARS-OMD on GitHub
  2. Open the Connector Web UI, go to settings, and enter your PeARS token in the Search engine token field (make sure the token matches the one defined in pears.ini).
  3. Restart the On My Disk service: 
    sudo systemctl restart onmydisk
  4. Reindexing begins automatically when new storage devices are detected.

Known Limitations:

  • Languages: Limited for now to supported languages: English, French, Russian, and Slovenian. You can restrict the search language by adding the language code (e.g. -fr at the end of the query to search in French exclusively).
  • Supported Formats: Scans txt, markdown, html, pdf, odt, docx, pptx, latex, epub, fb2, csv.
  • Non-textual Files: Search is limited to metadata (e.g. EXIF headers in photos, ID3 tags in music).