OpenCompass v0.1.4 is here with an array of features, documentation improvements, and key fixes! Dive in to see what's in store:

🆕 Highlights:

More Tools and Features: OpenCompass continues to expand its repertoire with the addition of tools like update suffix, codellama, preds collection tools, qwen & qwen-chat support, and more. Not forgetting our attention to Otter and the MMBench Evaluation! Documentation Facelift: We've made several updates to our documentation, ensuring it stays relevant, user-friendly, and aesthetically pleasing. Essential Bug Fixes: We’ve tackled numerous bugs, especially those concerning tokens, triviaqa, nq postprocess, and qwen config. Enhancements: From simplifying execution logic to suppressing warnings, we’re always on the lookout for ways to improve our product.

Dive deeper to learn more:

🌟 New Features:

📦 Tools and Integrations:

• Application of update suffix tool (#280).
• Support for codellama and preds collection tools (#335).
• Addition of qwen & qwen-chat support (#286).
• Introduction of Otter to OpenCompass MMBench Evaluation (#232).
• Support for LLaVA and mPLUG-Owl (#331).

🛠 Utilities and Functionality:

• Enhanced sample count in prompt_viewer (#273).
• Ignored ZeroRetriever error when id_list provided (#340).
• Improved default task size (#360).

📝 Documentation:

• Updated communication channels: WeChat and Discord (#328).
• Documentation theme revamped for a fresh look (#332).
• Detailed documentation for the new entry script (#246).
• MMBench documentation updated (#336).

🛠️ Bug Fixes:

• Resolved issue when missing both pad and eos token (#287).
• Addressed triviaqa & nq postprocess glitches (#350).
• Fixed qwen configuration inaccuracies (#358).
• Default value added for zero retriever (#361).

⚙ Enhancements and Refactors:

• Streamlined execution logic in run.py and ensured temp files cleanup (#337).
• Suppressed unnecessary warnings raised by get_logger (#353).
• Import checks of multimodal added (#352).

🎉 New Contributors:

Thank you to all our contributors for this release, with a special shoutout to our new contributors:

@Luodian (First PR) @ZhangYuanhan-AI (First PR) @HAOCHENYE (First PR)

Thank you to the entire community for pushing OpenCompass forward. Make sure to star 🌟 our GitHub repository if OpenCompass aids your endeavors! We treasure your feedback and contributions.

Changelog

• [Feature] Add and apply update suffix tool by @Leymore in https://github.com/open-compass/opencompass/pull/280
• support sample count in prompt_viewer by @cdpath in https://github.com/open-compass/opencompass/pull/273
• docs: update wechat and discord by @vansin in https://github.com/open-compass/opencompass/pull/328
• [Docs] Update doc theme by @gaotongxiao in https://github.com/open-compass/opencompass/pull/332
• [Feat] support codellama and preds collection tools by @yingfhu in https://github.com/open-compass/opencompass/pull/335
• [Feature] Add qwen & qwen-chat support by @Leymore in https://github.com/open-compass/opencompass/pull/286
• [Feat] Add Otter to OpenCompass MMBench Evaluation by @Luodian in https://github.com/open-compass/opencompass/pull/232
• [Docs] Update docs for new entry script by @gaotongxiao in https://github.com/open-compass/opencompass/pull/246
• [Fix] Fix when missing both pad and eos token by @Leymore in https://github.com/open-compass/opencompass/pull/287
• [Doc] Update MMBench.md by @kennymckormick in https://github.com/open-compass/opencompass/pull/336
• [Feat] Support LLaVA and mPLUG-Owl by @ZhangYuanhan-AI in https://github.com/open-compass/opencompass/pull/331
• [Feature] Ignore ZeroRetriever error when id_list provided by @Leymore in https://github.com/open-compass/opencompass/pull/340
• [Enhance] Add import check of multimodal by @fangyixiao18 in https://github.com/open-compass/opencompass/pull/352
• [Sync] [Enhancement] Simplify execution logic in run.py; use finally to clean up temp files by @gaotongxiao in https://github.com/open-compass/opencompass/pull/337
• [Fix] Fix triviaqa & nq postprocess by @Leymore in https://github.com/open-compass/opencompass/pull/350
• [Enhance] Supress warning raised by get_logger by @HAOCHENYE in https://github.com/open-compass/opencompass/pull/353
• [Fix] Update qwen config by @Leymore in https://github.com/open-compass/opencompass/pull/358
• [Fix] zero retriever add default value by @Leymore in https://github.com/open-compass/opencompass/pull/361
• [Enhancement] Increase default task size by @gaotongxiao in https://github.com/open-compass/opencompass/pull/360
• [Fix] Quick lint fix by @Leymore in https://github.com/open-compass/opencompass/pull/362
• [Docs] update code evaluator docs by @yingfhu in https://github.com/open-compass/opencompass/pull/354
• [Feat] support wizardcoder series by @yingfhu in https://github.com/open-compass/opencompass/pull/344
• [Feat] Support Qwen-VL-Chat on MMBench. by @yyk-wew in https://github.com/open-compass/opencompass/pull/312
• [Feature] Update claude2 postprocessor by @gaotongxiao in https://github.com/open-compass/opencompass/pull/365
• [Doc] Update Overview by @tonysy in https://github.com/open-compass/opencompass/pull/242
• [Feat] Update URL by @tonysy in https://github.com/open-compass/opencompass/pull/368
• [Feature] Update llama2 implement by @Leymore in https://github.com/open-compass/opencompass/pull/372
• [Feature] Add open source dataset eval config of instruct-blip by @fangyixiao18 in https://github.com/open-compass/opencompass/pull/370
• [Fix] Update bbh implement & Fix bbh suffix by @Leymore in https://github.com/open-compass/opencompass/pull/371
• [Feaure] Add new models: baichuan2, tigerbot, vicuna v1.5 by @Leymore in https://github.com/open-compass/opencompass/pull/373
• Bump version to 0.1.4 by @gaotongxiao in https://github.com/open-compass/opencompass/pull/367

For an exhaustive list of changes, kindly check our Full Changelog.