v0.14.1rc1

版本发布时间: 2023-10-05 21:56:09

huggingface/tokenizers最新发布版本:v0.15.0(2023-11-15 03:06:30)

What's Changed

pyo3: update to 0.19 by @mikelui in https://github.com/huggingface/tokenizers/pull/1322
Add expect() for disabling truncation by @boyleconnor in https://github.com/huggingface/tokenizers/pull/1316
Re-using scritpts from safetensors. by @Narsil in https://github.com/huggingface/tokenizers/pull/1328
Reduce number of different revisions by 1 by @Narsil in https://github.com/huggingface/tokenizers/pull/1329
Python 38 arm by @Narsil in https://github.com/huggingface/tokenizers/pull/1330
Move to maturing mimicking move for safetensors. + Rewritten node bindings. by @Narsil in https://github.com/huggingface/tokenizers/pull/1331
Updating the docs with the new command. by @Narsil in https://github.com/huggingface/tokenizers/pull/1333
Update added tokens by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1335
update package version for dev by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1339
Added ability to inspect a 'Sequence' pre-tokenizer. by @eaplatanios in https://github.com/huggingface/tokenizers/pull/1341
Let's allow hf_hub < 1.0 by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1344
Fixing the progressbar. by @Narsil in https://github.com/huggingface/tokenizers/pull/1353

Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.13.4.rc2...v0.14.1rc1