The OpenCompass team is thrilled to announce the release of OpenCompass v0.3.4!

🎉 OpenCompass v0.3.4 brings major enhancements including new benchmarks, improved documentation, and numerous bug fixes. 🌈 Notable features include support for new datasets and the integration of lmdeploy pipeline API.

🔧 Support for New Datasets:

• Addition of GaoKaoMath Dataset for Evaluation.
• Support for MMMLU & MMMLU-lite Benchmark.
• Integration of Judgerbench and reorganization of subeval.
• Support for LiveCodeBench.

📝 Output Format Enhancements:

• Support for printing and saving results as markdown format tables.

🔧 Pipeline and Integration Improvements:

• Integration of lmdeploy pipeline API.
• Update of TurboMindModel through integration of lmdeploy pipeline API.
• Removal of prefix bos_token from messages when using lmdeploy as the accelerator.

🛠️ Miscellaneous Enhancements:

• Updates to the common summarizer regex extraction.
• Internal humaneval postprocess addition and updates.

📖 Documentation Updates

🐛 Bug Fixes

🎉 Welcome New Contributors 👋 New Contributors Joined the Team:

BobTsang1995 - Contributed support for MMMLU & MMMLU-lite Benchmark. noemotiovon - Provided NPU support fixes. changlan - Fixed RULER datasets. BIGWangYuDong - Added support for printing and saving results as markdown format tables. Thank you to all contributors who have made this release possible. For a complete list of changes, please see the full changelog linked below.

Full Changelog: https://github.com/open-compass/opencompass/compare/0.3.3...0.3.4