The OpenCompass team is thrilled to announce the release of OpenCompass v0.3.1!

🌟 Highlights

• 🚀 Support pip installation, update Readme and evaluation demo
• 🐛 Fixed various dataset loading issues.
• ⚙️ Enhanced auto-download features for datasets.

🚀 New Features

• 🆕 Introduced support for Ruler datasets.
• 🆕 Enhanced model compatibility.
• 🆕 Improved dataset handling, support auto-download for various datasets

📖 Documentation

• 📚 Updated README to reflect the latest changes.
• 📚 Improved documentation for dataset loading procedures.

🐛 Bug Fixes

• 🐞 Resolved modelscope dataset load issues.
• 🐞 Corrected evaluation scores for the Lawbench dataset.
• 🐞 Fixed dataset bugs for CommonsenseQA and Longbench.

⚙ Enhancements and Refactors

• 🔧 Retained first and last halves of prompts to avoid max_seq_len issues.
• 🔧 Updated Compassbench to v1.3.
• 🔧 Switched to Python runner for single GPU operations.

🎉 Welcome New Contributors

• 🙌 @Yunnglin for fixing modelscope dataset load problem.
• 🙌 @changyeyu for addressing max_seq_len issues with prompt handling.
• 🙌 @seetimee for updates to openai_api.py.
• 🙌 @HariSeldon0 for adding the scicode dataset.

What's Changed

• [Fix] Fix modelscope dataset load problem by @Yunnglin in https://github.com/open-compass/opencompass/pull/1406
• [Fix] the issue where scores are negative in the Lawbench dataset evaluation(#1402) by @yaoyingyy in https://github.com/open-compass/opencompass/pull/1403
• [Doc] Update README by @tonysy in https://github.com/open-compass/opencompass/pull/1404
• Retain first and last halves of prompts to avoid max_seq_len issues by @changyeyu in https://github.com/open-compass/opencompass/pull/1373
• [UPDATE] Compassbench v1.3 by @MaiziXiao in https://github.com/open-compass/opencompass/pull/1396
• [Fix] longbench dataset load fix by @MaiziXiao in https://github.com/open-compass/opencompass/pull/1422
• [Fix] Sub summarizer order fix by @bittersweet1999 in https://github.com/open-compass/opencompass/pull/1426
• [Update] Support auto-download of FOFO/MT-Bench-101 by @tonysy in https://github.com/open-compass/opencompass/pull/1423
• [Bug] Commonsenseqa dataset fix by @MaiziXiao in https://github.com/open-compass/opencompass/pull/1425
• [Feature] Add abbr for rolebench dataset by @xu-song in https://github.com/open-compass/opencompass/pull/1431
• [Feature] Add Ruler datasets by @MaiziXiao in https://github.com/open-compass/opencompass/pull/1310
• [Fix] Fix openai api tiktoken bug for api server by @liushz in https://github.com/open-compass/opencompass/pull/1433
• Update openai_api.py by @seetimee in https://github.com/open-compass/opencompass/pull/1438
• [Feature] Add model support for 'huggingface_above_v4_33' when using '-a' by @liushz in https://github.com/open-compass/opencompass/pull/1430
• Add scicode by @HariSeldon0 in https://github.com/open-compass/opencompass/pull/1417
• [Doc] Update Readme by @MaiziXiao in https://github.com/open-compass/opencompass/pull/1439
• [Fix] Update option postprocess & mathbench language summarizer by @liushz in https://github.com/open-compass/opencompass/pull/1413
• [ci] add commond testcase into daily testcase by @zhulinJulia24 in https://github.com/open-compass/opencompass/pull/1447
• [Feature] Switch to python runner for single GPU by @xu-song in https://github.com/open-compass/opencompass/pull/1308
• [Fix] Update SciCode and Gemma model by @tonysy in https://github.com/open-compass/opencompass/pull/1449
• [Bump] Bump version to 0.3.1 by @tonysy in https://github.com/open-compass/opencompass/pull/1450

Full Changelog: https://github.com/open-compass/opencompass/compare/0.3.0...0.3.1

Thank you for your continued support and contributions to OpenCompass!