🎉 Highlights

This release focuses on generic ingestion workflows with IngestionTool.

• ⛏️ IngestionTool: A flexible new component for document ingestion.
• 🤖 Ingest and Knowledge Agents: Demonstrating search, ingestion, and retrieval capabilities.
• 💤 Lazy Loading Node Templates: Enable agents to initialize components within it's chain graph

https://github.com/kreneskyp/ix/assets/68635/349466de-47f7-40e2-9467-28f891f08956

Details

⛏️ IngestionTool #268

Introducing the new IngestionTool component, designed for versatile document ingestion. This tool seamlessly integrates with any vector store and any document source. It's a flexible way to build a customized document ingestion agent and RAG system.

IngestionTool uses lazy loading NodeTemplates to parameterize properties in both the loader and vectorstore. This enables agents that can combine search, transform, and ingestion in a single request.

Prompt Example:

Search for TOPIC and add the urls to collection COLLECTION

🤖 Ingest and Knowledge Agents (#267, #269, #272)

With these agents, users can search, ingest, and retrieve content. Specifically:

• The ingest agent can search Wikipedia, capturing URLs and adding them to the knowledge collection.
• The knowledge agent can provide answers based on the stored content.

Agent: Ingest

Search & Ingest

@Ingest is parameterized:

• URL and COLLECTION are both variables.
• @ingest can fetches a URL and stores the content in a specified COLLECTION.

Web Loader Template Config

Chroma Template Config

💤 Lazy Loading Node Templates (#267)

Lazy loading templates enable agents to initialize components in the graph. The default loading method uses statically defined configuration. Node templates delay loading until runtime and then initialize with input variables. Combine that into a custom tool and agent can initialize the component using an LLM function call.

Templates & Variables

Custom tools may define any connector as a template. The branch of nodes off the connector is wrapped is lazy loaded when the chain graph loads. Any node within the branch may define any field value as a {variable}. The variables are formatted with input variables or tool args when it runs.

Tool Args

Lazy loading templates are compatible with LLM function calls. NodeTemplate builds Pydantic models for the templates automatically from variables within the branch. Custom tools can set args_schema from the templates schema. The schema is used to generate LLM function calls dynamically from the agent config.

Example

IngestionTool is the first example of NodeTemplate and demonstrates how to configure and format components from multiple components.

See #267 for more details on how to use templates