The highlight of this release is a new token-based text splitter, used by the prepdocs script when splitting content into chunks for the search index. The previous algorithm was based solely on character count, which meant that our prepdocs script did not work well for non-English documents or any documents which resulted in a higher than usual amount of tokens. If you do experience any regression in splitting quality as a result of this change, please file an issue.

What's Changed

• Improve text splitter for non-English documents by @tonybaloney in https://github.com/Azure-Samples/azure-search-openai-demo/pull/1326
• Restrict GitHub workflows run by @john0isaac in https://github.com/Azure-Samples/azure-search-openai-demo/pull/1366
• Improvements to load balancer setup script by @pamelafox in https://github.com/Azure-Samples/azure-search-openai-demo/pull/1348
• Update productionizing.md with link to search service size guide by @pamelafox in https://github.com/Azure-Samples/azure-search-openai-demo/pull/1354
• Update README.md to delete old links by @pamelafox in https://github.com/Azure-Samples/azure-search-openai-demo/pull/1372
• Update deploy_features.md link by @pamelafox in https://github.com/Azure-Samples/azure-search-openai-demo/pull/1373
• Add suggestion to use [azd auth login] in the free low-cost deploy tutorial by @elbruno in https://github.com/Azure-Samples/azure-search-openai-demo/pull/1214
• Bump the python-requirements group with 18 updates by @dependabot in https://github.com/Azure-Samples/azure-search-openai-demo/pull/1368

New Contributors

• @elbruno made their first contribution in https://github.com/Azure-Samples/azure-search-openai-demo/pull/1214

Full Changelog: https://github.com/Azure-Samples/azure-search-openai-demo/compare/2024-03-01...2024-03-06