OpenCompass v0.1.7 Release Notes 🎉

🌟 Highlights

• Sampling Control: Enforce do_sample=False for precise control over sampling behavior in HF model.
• Subjective Evaluation Guidance: Enhanced evaluation mechanisms for a more comprehensive understanding and analysis of models.
• Eval Details Dump: Now, evaluation details for certain datasets are available for a deeper insight and analysis.

🚀 New Features

• Eval Details Dump for a deeper insight on each test cases. (#517 by @Leymore).
• MathBench Dataset and Circular Evaluator to bolster mathematical benchmarking capabilities (#408 by @liushz).
• Support for Math/GMS8k Agent Config providing new avenues for configuration (#494 by @yingfhu).
• Default Example Summarizer making summarization tasks more accessible (#508 by @Leymore).
• Model Keyword Arguments Setting for HF Model enhancing customization (#507 by @Leymore).

🛠 Improvements & Refactorings

• Local API Speed Up with fixed concurrent users for better performance (#497 by @yingfhu).
• Local Runner Support for Windows expanding the platform support (#515 by @yingfhu).
• Sync with Internal Implements for updated and refined functionalities (#488 by @Leymore).

🐛 Bug Fixes

• Summary Default Fix for accurate summarization (#483 by @Leymore).
• Enforce do_sample=False in HF Model for correct sampling behavior (#506 by @Leymore).
• Invalid Link Fix in documentation for better navigation (#499 by @yingfhu).

📚 Documentation & Maintenance

• Subjective Comparison Introduction for organized documentation (#510 by @frankweijue).
• README Update for better project understanding (#496 by @saakshii12).
• Owner Update for correct ownership information (#504 by @Leymore).

🎊 New Contributors

We're delighted to welcome new contributors to the OpenCompass community!

• @saakshii12 made their first contribution in #496.
• @frankweijue stepped in with their first contribution in #510.

Changelog

• [Fix] Fix summary default by @Leymore in https://github.com/open-compass/opencompass/pull/483
• [Feature] Add mathbench dataset and circular evaluator by @liushz in https://github.com/open-compass/opencompass/pull/408
• [Sync] sync with internal implements by @Leymore in https://github.com/open-compass/opencompass/pull/488
• Update README.md by @saakshii12 in https://github.com/open-compass/opencompass/pull/496
• [Docs] update invalid link in docs by @yingfhu in https://github.com/open-compass/opencompass/pull/499
• [Feat] local api speed up with fixed concurrent users by @yingfhu in https://github.com/open-compass/opencompass/pull/497
• [Feat] support math/gms8k agent config by @yingfhu in https://github.com/open-compass/opencompass/pull/494
• [Feature] update .owner by @Leymore in https://github.com/open-compass/opencompass/pull/504
• [Feat] use example summarizer by default by @Leymore in https://github.com/open-compass/opencompass/pull/508
• [Feat] Add _set_model_kwargs_torch_dtype for HF model by @Leymore in https://github.com/open-compass/opencompass/pull/507
• Subdocs by @frankweijue in https://github.com/open-compass/opencompass/pull/510
• [Fix] enforce do_sample=False in HF model by @Leymore in https://github.com/open-compass/opencompass/pull/506
• [Feat] support local runner for windows by @yingfhu in https://github.com/open-compass/opencompass/pull/515
• [Sync] Sync with internal codes 2023.10.27 by @Leymore in https://github.com/open-compass/opencompass/pull/517
• Bump version to 0.1.7 by @Leymore in https://github.com/open-compass/opencompass/pull/518

The full list of changes is available in the changelog. A massive thank you to all the community members who contributed to this release. Your efforts are propelling OpenCompass further! 🙌

Embark on new explorations with OpenCompass v0.1.7!