The OpenCompass team is thrilled to announce the release of OpenCompass v0.3.9!

🌟 Highlights ✨ This version introduces a number of new features and improvements that enhance the user experience and expand the capabilities of OpenCompass. Notable changes include support for G-Pass@k and LiveMathBench, as well as the introduction of the Bradley-Terry subjective evaluation method.

🚀 New Features -🆕 Support for G-Pass@k and LiveMathBench metrics to better evaluate model performance. (#1772) -🆕 Theorem QA 0shot CoT configuration has been added for more comprehensive evaluation scenarios. (#1783) -🆕 A customizable tokenizer for RULER offers greater flexibility in processing inputs. (#1731) -🆕 Added LiveStemBench Dataset to enrich our collection of datasets. (#1794) -🆕 Integration of JudgeLLM into o1 evaluation for improved assessment accuracy. (#1795) -🆕 Implementation of the Bradley-Terry subjective evaluation method on wildbench, alpacaeval, and compassarena datasets. (#1791)

📖 Documentation -📚 Updated OC academic content to the most recent information as of December 2024. (#1771)

🐛 Bug Fixes -🔧 Fixed Order error which was causing issues with sequence handling. (#1767) -🔧 Resolved an issue where the lark report was returning None. (#1769) -🔧 Corrected the path for saving Local Runner parameters. (#1768) -🔧 Amended the summarizer abbreviation for models to ensure proper identification. (#1789) -🔧 Fixed output_path errors to improve file handling reliability. (#1798)

⚙ Enhancements and Refactors -💪 Fullbench testcase has been integrated into the CI pipeline. (#1766) Volc status exception handling has been updated for more robust responses. (#1780) -💪 Removed daily step retry mechanism and updated PR score calculation for efficiency. (#1782) -💪Deploy Python version has been updated to the latest stable release. (#1784) -💪Pypi deploy workflow has been refined for smoother deployments. (#1786)

Thank you for being part of the OpenCompass community! Your support and contributions make each release possible.