Skip to main content

​
Documents

Documents are the primary content unit used by BindAI Knowledge for working with external information. The Knowledge system provides document abstractions and processing components that can load, process, split, enrich, and prepare external content for retrieval. Documents can ultimately become searchable knowledge that agents can use as context during execution.

​
Documents and BindAI

A document represents external information that an application wants to make available to its Knowledge system. Examples include:
  • Markdown files
  • Text files
  • Documentation
  • Product manuals
  • Knowledge-base content
  • Application-generated content
  • Structured application data
  • Other supported external sources
The Knowledge layer separates document processing from agent execution. A typical flow is:
This allows documents to be prepared for retrieval before an agent needs to answer a question.

​
Document Processing

A Knowledge document can pass through several processing stages.
Not every application needs every stage. For example, a small text document may require only loading and chunking, while a large knowledge base may require parsing, metadata enrichment, embeddings, and retrieval indexing.

​
Document Loading

Document loaders are responsible for obtaining content from a source and converting it into Knowledge-compatible document data. Conceptually:
The loader separates source access from downstream document processing. This is useful because applications can process content from different sources without coupling the rest of the Knowledge pipeline to the source itself.

​
Ingestion

Document ingestion prepares source content for use by the Knowledge system. A typical ingestion flow is:
BindAI provides Knowledge pipeline components for organizing this processing. Ingestion should generally be treated as a preparation step rather than something that happens automatically for every agent request.

​
Parsing

Parsing converts source content into a form that can be processed as Knowledge. For example:
Parsing is particularly important for sources whose useful content is not already represented as plain text. The parsing strategy should match the source type and its structure.

​
Document Content

A document’s content represents the information that should be made available to downstream Knowledge components. Conceptually:
The content can then be split into smaller chunks for retrieval. Keeping source information and metadata attached to the content makes it possible to understand where retrieved information originated.

​
Metadata

Metadata provides additional information about a document. Examples include:
Metadata can be preserved during document processing and used later during retrieval and filtering. For example, an application may attach:
to a document so that retrieval can distinguish it from other content.

​
Document Chunking

Large documents are normally divided into smaller chunks before retrieval. For example:
Chunking allows retrieval systems to identify relevant portions of a document instead of returning an entire large document. Chunk size and overlap should be selected according to the source content and retrieval requirements.

​
Why Chunk Documents?

Sending an entire large document to a language model for every question can be inefficient. Chunking allows a retrieval system to select only relevant sections.
This reduces unnecessary context and helps the model focus on information related to the current query.

​
Chunk Metadata

Chunks can retain metadata from their source document. Conceptually:
This is important for retrieval because the application may need to know the source, category, document identity, or other attributes associated with a retrieved chunk.

​
Embeddings

Documents or document chunks can be transformed into embeddings for vector-based retrieval. The general flow is:
BindAI provides an embedding abstraction that separates embedding generation from the rest of the Knowledge pipeline. The current implementation includes OpenAI embeddings and a local/deterministic embedding implementation used for development and testing.

​
Document Retrieval

Once documents have been processed, their chunks can participate in retrieval. A simplified flow is:
BindAI currently provides multiple retrieval strategies, including:
  • Vector retrieval
  • BM25 retrieval
  • Hybrid retrieval
  • Metadata filtering
  • Reranking
The selected retrieval strategy depends on the application’s requirements.

​
Vector Document Retrieval

Vector retrieval uses embeddings to find semantically similar document chunks. For example:
The query and document do not need to contain identical wording to be considered related.

​
BM25 Document Retrieval

BM25 provides lexical retrieval based on terms and relevance. It can be useful when exact terminology matters.
BM25 can also be combined with vector retrieval through hybrid search.

​
Hybrid Document Retrieval

Hybrid retrieval combines lexical and semantic retrieval.
This approach can provide stronger retrieval when both exact terminology and semantic similarity are important.

​
Metadata Filtering

Metadata can be used to restrict which documents or chunks participate in retrieval. For example:
This can be useful for:
  • Tenant isolation
  • Document categories
  • Versions
  • Access scopes
  • Content types
Metadata filtering should complement application-level authorization rather than replace it.

​
Reranking Document Results

Initial retrieval can return multiple candidate chunks. Reranking can then improve their ordering.
BindAI provides a reranker abstraction and lexical reranking implementation. Reranking can be applied after vector, BM25, or hybrid retrieval.

​
Documents and Knowledge Pipelines

Documents can be processed through Knowledge pipelines. A typical ingestion pipeline is:
The pipeline approach keeps document processing organized and repeatable. It also makes it easier to change one processing stage without rewriting the complete application.

​
Documents and Agents

Processed documents become useful to an agent through Knowledge retrieval. A typical runtime flow is:
This allows agents to answer questions using information outside the model’s built-in knowledge.

​
Documents and Tools

Documents and tools can coexist. Tools are useful when an agent needs to perform an operation such as:
  • Fetching a live document
  • Calling an external document API
  • Reading application-specific files
  • Performing an external search
  • Updating a document system
Knowledge is useful when information needs to be processed and retrieved as part of a knowledge base. A common architecture is:

​
Documents and Memory

Documents and Memory serve different purposes. For example:
  • A product manual belongs in Knowledge.
  • A user’s preferred response format may belong in Memory.
An application can use both.

​
Document Collections

Applications can organize documents into logical collections. For example:
Collections can be represented using application-defined metadata, source identifiers, namespaces, or storage organization. The exact organization depends on the configured Knowledge and retrieval components.

​
Document Updates

Knowledge data may need to be reprocessed when source documents change. A typical update flow is:
Applications should define an appropriate update strategy based on how frequently their source documents change. For relatively static documentation, ingestion can happen during deployment or scheduled maintenance. For frequently changing data, ingestion may be part of an automated workflow.

​
Document Deletion

When a source document is removed or should no longer be available, its derived knowledge data should also be considered for removal. Conceptually:
The application should maintain enough source metadata to identify related derived records. Deletion and retention behavior depends on the configured storage and retrieval implementation.

​
Supported Sources

BindAI’s Knowledge architecture is designed to work with document loaders and processing components. Supported source types depend on the loaders implemented and configured by the application. Possible source categories include:
  • Text content
  • Markdown
  • Structured content
  • Application-generated documents
  • External content supported by an installed loader
Do not assume that every possible document format is supported automatically. For a specific source format, verify that an appropriate loader and parser are available.

​
Document Format Strategy

Different source formats may require different processing strategies. For example:
The goal is to preserve useful information and structure before retrieval.

​
Document Quality

Document quality directly affects retrieval quality. Important considerations include:
  • Clean source content
  • Appropriate parsing
  • Sensible chunk sizes
  • Useful metadata
  • Consistent document identifiers
  • Appropriate embedding models
  • Correct retrieval configuration
A poor ingestion process can produce poor retrieval results even when the language model itself is capable.

​
Document Security

Documents may contain sensitive information. Applications should consider:
  • Access control
  • Tenant isolation
  • Metadata filtering
  • Source permissions
  • Secure storage
  • Credential management
  • Retention policies
  • Document deletion
Knowledge retrieval should not bypass the application’s authorization rules. Metadata filtering is useful for retrieval boundaries, but authorization should be enforced independently where required.

​
Document Pipeline Example

A complete conceptual ingestion process looks like:
This pipeline separates document preparation from runtime question answering.

​
Best Practices

  • Use Knowledge for external or reference information.
  • Keep document loading separate from retrieval where practical.
  • Choose parsers appropriate to the source format.
  • Use chunking strategies appropriate to the document structure.
  • Preserve useful source metadata.
  • Keep document identifiers stable.
  • Use embedding models compatible with the configured retrieval storage.
  • Test retrieval using realistic questions.
  • Reprocess documents when important source content changes.
  • Track document provenance.
  • Enforce tenant and access boundaries.
  • Do not rely on metadata filtering as the only authorization mechanism.
  • Avoid storing unnecessary sensitive information.
  • Keep document ingestion reproducible.
  • Separate Knowledge retrieval from tools that perform live operations.

​
Current Project Status

BindAI currently provides a concrete Knowledge layer with document-processing capabilities. The implemented architecture includes:
  • Document abstractions
  • Document loading
  • Ingestion pipelines
  • Parsing
  • Chunking
  • Metadata
  • Embedding integration
  • Retrieval
  • BM25 retrieval
  • Vector retrieval
  • Hybrid retrieval
  • Metadata filtering
  • Reranking
  • Conversational retrieval
  • Knowledge pipelines
  • Agent Knowledge integration
The exact source formats available to an application depend on the implemented and configured loaders.

​
Summary

Documents are the content foundation of BindAI Knowledge. A typical document lifecycle is:
BindAI’s Knowledge architecture separates document processing from agent execution and provides the components required to prepare external information for retrieval-augmented applications. Documents can be combined with Memory, tools, workflows, embeddings, and retrieval strategies to build more capable AI applications.