v0.11.2
版本发布时间: 2022-02-28 18:53:49
huggingface/tokenizers最新发布版本:v0.15.0(2023-11-15 03:06:30)
- [#919] Fixing single_word AddedToken. (regression from 0.11.2)
- [#916] Deserializing faster
added_tokensby loading them in batch.
MyGit
版本发布时间: 2022-02-28 18:53:49
huggingface/tokenizers最新发布版本:v0.15.0(2023-11-15 03:06:30)
added_tokens by loading them in batch.