v0.13.4.rc1
版本发布时间: 2023-08-14 19:50:28
huggingface/tokenizers最新发布版本:v0.15.0(2023-11-15 03:06:30)
What's Changed
- Update all GH Actions with dependency on actions/checkout by @mfuntowicz in https://github.com/huggingface/tokenizers/pull/1256
- Parallelize unigram trainer by @mishig25 in https://github.com/huggingface/tokenizers/pull/976
- Update unigram/trainer.rs by @chris-ha458 in https://github.com/huggingface/tokenizers/pull/1257
- Fixing broken link. by @Narsil in https://github.com/huggingface/tokenizers/pull/1268
- fix documentation regarding regex by @chris-ha458 in https://github.com/huggingface/tokenizers/pull/1264
- Update Cargo.toml by @chris-ha458 in https://github.com/huggingface/tokenizers/pull/1266
- Update README.md - Broken link by @sbhavani in https://github.com/huggingface/tokenizers/pull/1272
- [doc build] Use secrets by @mishig25 in https://github.com/huggingface/tokenizers/pull/1273
- Improve error for truncation with too high stride by @boyleconnor in https://github.com/huggingface/tokenizers/pull/1275
- Add unigram bytefallback by @ArthurZucker in https://github.com/huggingface/tokenizers/pull/1217
- revise type specification by @hiroshi-matsuda-rit in https://github.com/huggingface/tokenizers/pull/1289
- Bump tough-cookie from 4.0.0 to 4.1.3 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1291
- Update path name: master -> main by @bact in https://github.com/huggingface/tokenizers/pull/1292
- import Tuple from typing by @kellymarchisio in https://github.com/huggingface/tokenizers/pull/1295
- Fixing clippy warnings on 1.71. by @Narsil in https://github.com/huggingface/tokenizers/pull/1296
- Bump word-wrap from 1.2.3 to 1.2.4 in /bindings/node by @dependabot in https://github.com/huggingface/tokenizers/pull/1299
- feat: Added CITATION.cff. by @SamuelLarkin in https://github.com/huggingface/tokenizers/pull/1302
- Single warning for holes. by @Narsil in https://github.com/huggingface/tokenizers/pull/1303
- Give error when initializing tokenizer with too high stride by @boyleconnor in https://github.com/huggingface/tokenizers/pull/1306
- Handle when precompiled charsmap is empty by @kellymarchisio in https://github.com/huggingface/tokenizers/pull/1308
- Derive clone for TrainerWrapper by @jonatanklosko in https://github.com/huggingface/tokenizers/pull/1317
- CD backports by @chris-ha458 in https://github.com/huggingface/tokenizers/pull/1318
- 0.13.4.rc1 by @Narsil in https://github.com/huggingface/tokenizers/pull/1319
New Contributors
- @sbhavani made their first contribution in https://github.com/huggingface/tokenizers/pull/1272
- @boyleconnor made their first contribution in https://github.com/huggingface/tokenizers/pull/1275
- @hiroshi-matsuda-rit made their first contribution in https://github.com/huggingface/tokenizers/pull/1289
- @bact made their first contribution in https://github.com/huggingface/tokenizers/pull/1292
- @kellymarchisio made their first contribution in https://github.com/huggingface/tokenizers/pull/1295
- @SamuelLarkin made their first contribution in https://github.com/huggingface/tokenizers/pull/1302
- @jonatanklosko made their first contribution in https://github.com/huggingface/tokenizers/pull/1317
Full Changelog: https://github.com/huggingface/tokenizers/compare/v0.13.4-rc2...v0.13.4.rc1