v0.14.0.rc1
版本发布时间: 2023-09-07 19:39:02
huggingface/tokenizers最新发布版本:v0.15.0(2023-11-15 03:06:30)
Reworks the release pipeline. Other breaking changes are mostly related to https://github.com/huggingface/tokenizers/pull/1335, where AddedToken is reworked
What's Changed
- pyo3: update to 0.19 by @mikelui in https://github.com/huggingface/tokenizers/pull/1322
- Add
expect()
for disabling truncation by @boyleconnor in https://github.com/huggingface/tokenizers/pull/1316 - Re-using scritpts from safetensors. by @Narsil in https://github.com/huggingface/tokenizers/pull/1328
- Reduce number of different revisions by 1 by @Narsil in https://github.com/huggingface/tokenizers/pull/1329
- Python 38 arm by @Narsil in https://github.com/huggingface/tokenizers/pull/1330
- Move to maturing mimicking move for
safetensors
. + Rewritten node bindings. by @Narsil in https://github.com/huggingface/tokenizers/pull/1331 - Updating the docs with the new command. by @Narsil in https://github.com/huggingface/tokenizers/pull/1333
- Update added tokens by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1335
New Contributors
- @mikelui made their first contribution in https://github.com/huggingface/tokenizers/pull/1322
Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.13.4.rc2...v0.14.0.rc1