This release addresses incompatibilities related to saving and loading models across a range of transformers versions following changes in transformers v4.31.

• Extend support to transformers v4.33 (#402).
• Add fallback to load state_dict with strict=False, due to incompatibilities related to state_dict keys between transformers v4.30 and v4.31 (#398).

Options for improved pipeline package compatibility

If you are packaging models with spacy package and you'd like to avoid having your users run into the fallback strict=False loading and warnings related to state_dict keys, you may want to add a stricter transformers requirement that corresponds to the version used during training.

If you're training with the newest supported transformers version (v4.33.x), you could currently add transformers>=4.31 to your package requirements. Note that this would restrict your users to Python 3.8+ and PyTorch 1.9+ due to transformers requirements.

Create meta.json with your additional requirements:

{
  "requirements": ["transformers>=4.31"]
}

Package with spacy package:

spacy package --meta meta.json input_dir output_dir

The requirement transformers>=4.31 will be added to any other package requirements that are automatically determined based on your pipeline config.