The OpenCompass team is thrilled to announce the release of OpenCompass v0.3.0! This release brings a variety of new features, enhancements, and bug fixes to improve your experience.

🌟 Highlights

1. Support for OpenAI ChatCompletion
2. Updated Model Support List
3. Support Dataset Automatic Download
4. Support pip install opencompass

🚀 New Features

1. Support for CompassBench Checklist Evaluation
   • PR #1339 by @bittersweet1999
2. Adding support for Doubao API
   • PR #1218 by @LeavittLang
3. Support for ModelScope Datasets
   • PR #1289 by @wangxingjun778

📖 Documentation

1. Update NeedleBench Docs
   • PR #1330 by @DseidLi
2. Update Documentation
   • PR #1318 by @Leymore

🐛 Bug Fixes

1. Fix Typing and Typo
   • PR #1331 by @xu-song
2. Fix Lint Issues
   • PR #1334 by @DseidLi
3. Fix Summary Error in subjective.py
   • PR #1363 by @WenjinW

⚙ Enhancements and Refactors

1. Upgrade Default Math pred_postprocessor
   • PR #1340 by @xu-song
2. Fix Path and Folder Updates
   • PR #1344 by @tonysy
3. Update Get Data Path for LCBench and HumanEval
   • PR #1375 by @tonysy

🔗 Full Change Logs

• [Fix] Change abbr for arenahard dataset by @bittersweet1999 in https://github.com/open-compass/opencompass/pull/1302
• [Fix] Force register by @Leymore in https://github.com/open-compass/opencompass/pull/1311
• [Fix] add bc for alignbench summarizer by @bittersweet1999 in https://github.com/open-compass/opencompass/pull/1306
• [Fix] update Faq by @bittersweet1999 in https://github.com/open-compass/opencompass/pull/1313
• [Fix] Fix rouge evaluator of rolebench_zh by @xu-song in https://github.com/open-compass/opencompass/pull/1322
• [Doc] Update NeedleBench Docs by @DseidLi in https://github.com/open-compass/opencompass/pull/1330
• [Fix] Fix typing and typo by @xu-song in https://github.com/open-compass/opencompass/pull/1331
• [Fix] Fix lint by @DseidLi in https://github.com/open-compass/opencompass/pull/1334
• [Feature] support compassbench Checklist evaluation by @bittersweet1999 in https://github.com/open-compass/opencompass/pull/1339
• Add compassbench wiki&math part by @liushz in https://github.com/open-compass/opencompass/pull/1342
• Compassbench v1_3 subjective evaluation by @MaiziXiao in https://github.com/open-compass/opencompass/pull/1341
• [Fix] Update path and folder by @tonysy in https://github.com/open-compass/opencompass/pull/1344
• Upgrade default math pred_postprocessor by @xu-song in https://github.com/open-compass/opencompass/pull/1340
• commit inference ppl datasets by @Quehry in https://github.com/open-compass/opencompass/pull/1315
• CompassBench subjective summarizer added by @MaiziXiao in https://github.com/open-compass/opencompass/pull/1349
• Fix MathBench Generation Config by @liushz in https://github.com/open-compass/opencompass/pull/1351
• [Update] Update model support list by @bittersweet1999 in https://github.com/open-compass/opencompass/pull/1353
• [Update] update Subeval demo config by @bittersweet1999 in https://github.com/open-compass/opencompass/pull/1358
• [Fix] Fix the summary error in subjective.py by @WenjinW in https://github.com/open-compass/opencompass/pull/1363
• [Fix] Support HF models deployed with an OpenAI-compatible API. by @heya5 in https://github.com/open-compass/opencompass/pull/1352
• update docs by @Leymore in https://github.com/open-compass/opencompass/pull/1318
• [Feature] Make NeedleBench available on HF by @DseidLi in https://github.com/open-compass/opencompass/pull/1364
• 【bug fix】: Remove extra ampersands. by @baymax591 in https://github.com/open-compass/opencompass/pull/1365
• [Fix] minor update wildbench by @kleinzcy in https://github.com/open-compass/opencompass/pull/1335
• Adding support for Doubao API by @LeavittLang in https://github.com/open-compass/opencompass/pull/1218
• [Fix] origin_prompt should be None in llm-compression task by @mqy004 in https://github.com/open-compass/opencompass/pull/1225
• Calm dataset by @pengbo807 in https://github.com/open-compass/opencompass/pull/1287
• Add en and zh groups to longbench summarizer; Fix longbench overall score by @xu-song in https://github.com/open-compass/opencompass/pull/1216
• [Revert] "Calm dataset (#1287)" by @bittersweet1999 in https://github.com/open-compass/opencompass/pull/1366
• Charm by @jxd0712 in https://github.com/open-compass/opencompass/pull/1230
• Support ModelScope datasets by @wangxingjun778 in https://github.com/open-compass/opencompass/pull/1289
• [Feature] Update pip install by @tonysy in https://github.com/open-compass/opencompass/pull/1324
• add support for hf_pulse_7b by @QXY716 in https://github.com/open-compass/opencompass/pull/1255
• [Fix] Update get_data_path for LCBench and HumanEval by @tonysy in https://github.com/open-compass/opencompass/pull/1375
• [Bug] Fix bug in turbomind by @tonysy in https://github.com/open-compass/opencompass/pull/1377
• [Fix] Fix version mismatch of CIBench by @kleinzcy in https://github.com/open-compass/opencompass/pull/1380
• [Fix] Fix InternLM2.5-7B-Chat-1M config by @DseidLi in https://github.com/open-compass/opencompass/pull/1383
• [Feature] Support import configs/models/summarizers from whl by @tonysy in https://github.com/open-compass/opencompass/pull/1376
• Calm dataset by @pengbo807 in https://github.com/open-compass/opencompass/pull/1385
• [Feature] Support OpenAI ChatCompletion by @tonysy in https://github.com/open-compass/opencompass/pull/1389
• [Fix] Fix slurm env by @tonysy in https://github.com/open-compass/opencompass/pull/1392
• [Fix] Fix CaLM import by @tonysy in https://github.com/open-compass/opencompass/pull/1395
• [Bump] Bump version for v0.3.0 by @tonysy in https://github.com/open-compass/opencompass/pull/1398

🎉 Welcome New Contributors

• @MaiziXiao made their first contribution in https://github.com/open-compass/opencompass/pull/1341
• @Quehry made their first contribution in https://github.com/open-compass/opencompass/pull/1315
• @WenjinW made their first contribution in https://github.com/open-compass/opencompass/pull/1363
• @heya5 made their first contribution in https://github.com/open-compass/opencompass/pull/1352
• @LeavittLang made their first contribution in https://github.com/open-compass/opencompass/pull/1218
• @pengbo807 made their first contribution in https://github.com/open-compass/opencompass/pull/1287
• @wangxingjun778 made their first contribution in https://github.com/open-compass/opencompass/pull/1289
• @QXY716 made their first contribution in https://github.com/open-compass/opencompass/pull/1255

Full Changelog: https://github.com/open-compass/opencompass/compare/0.2.6...0.3.0