v0.15.0
版本发布时间: 2023-11-15 03:06:30
huggingface/tokenizers最新发布版本:v0.15.0(2023-11-15 03:06:30)
What's Changed
- fix a clerical error in the comment by @tiandiweizun in https://github.com/huggingface/tokenizers/pull/1356
- fix: remove useless token by @rtrompier in https://github.com/huggingface/tokenizers/pull/1371
- Bump @babel/traverse from 7.22.11 to 7.23.2 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1370
- Allow hf_hub 0.18 by @mariosasko in https://github.com/huggingface/tokenizers/pull/1383
- Allow
huggingface_hub<1.0by @Wauplin in https://github.com/huggingface/tokenizers/pull/1385 - [
pre_tokenizers] Fix sentencepiece based Metaspace by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1357
New Contributors
- @tiandiweizun made their first contribution in https://github.com/huggingface/tokenizers/pull/1356
- @rtrompier made their first contribution in https://github.com/huggingface/tokenizers/pull/1371
- @mariosasko made their first contribution in https://github.com/huggingface/tokenizers/pull/1383
- @Wauplin made their first contribution in https://github.com/huggingface/tokenizers/pull/1385
Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.14.1...v0.15.0