Release Notes

v1.25.0

⚡️ Enhancement Notes

• Add raise_on_failure flag to BaseConverter class so that big processes can optionally continue without breaking from exceptions.

• Upgrade Transformers to the latest version 4.37.2. This version adds support for the Phi-2 and Qwen2 models and improves support for quantization.

• Add support for latest OpenAI embedding models text-embedding-3-large and text-embedding-3-small.

• API_BASE can now be passed as an optional parameter in the getting_started sample. Only openai provider is supported in this set of changes. PromptNode and PromptModel were enhanced to allow passing of this parameter. This allows RAG against a local endpoint (e.g, http://localhost:1234/v1), so long as it is OpenAI compatible (such as LM Studio)

  Logging in the getting started sample was made more verbose, to make it easier for people to see what was happening under the covers.

• Added new option split_by="page" to the preprocessor so we can chunk documents by page break.

🐛 Bug Fixes

• Change the dummy vector used internally in the Pinecone Document Store. A recent change to the Pinecone API does not allow to use vectors filled with zeros as was the previous dummy vector.
• The types of meta data values accepted by RouteDocuments was unnecessarily restricted to string types. This causes validation errors (for example when loading from a yaml file) if a user tries to use a boolean type for example. We add boolean and int types as valid types for metadata_values.
• Fixed a bug that made it impossible to write Documents to Weaviate when some of the fields were empty lists (e.g. split_overlap for preprocessed documents).
• Correct page meta field for pdfs that contain pages without any text content