What's new?

🤯 14 new architectures

In this release, we've added a ton of new architectures: BLOOM, MPT, BeiT, CamemBERT, CodeLlama, GPT NeoX, GPT-J, HerBERT, mBART, mBART-50, OPT, ResNet, WavLM, and XLM. This brings the total number of supported architectures up to 46! Here's some example code to help you get started:

• Text-generation with MPT (models):

  import { pipeline } from '@xenova/transformers';
  const generator = await pipeline('text-generation', 'Xenova/ipt-350m', {
      quantized: false, // using unquantized to ensure it matches python version
  });
  
  const output = await generator('La nostra azienda');
  // { generated_text: "La nostra azienda è specializzata nella vendita di prodotti per l'igiene orale e per la salute." }
  

  Other text-generation models: BLOOM, GPT-NeoX, CodeLlama, GPT-J, OPT.

• CamemBERT for masked language modelling, text classification, token classification, question answering, and feature extraction (models). For example:

  import { pipeline } from '@xenova/transformers';
  let pipe = await pipeline('token-classification', 'Xenova/camembert-ner-with-dates');
  let output = await pipe("Je m'appelle jean-baptiste et j'habite à montréal depuis fevr 2012");
  // [
  //   { entity: 'I-PER', score: 0.9258053302764893, index: 5, word: 'jean' },
  //   { entity: 'I-PER', score: 0.9048717617988586, index: 6, word: '-' },
  //   { entity: 'I-PER', score: 0.9227054119110107, index: 7, word: 'ba' },
  //   { entity: 'I-PER', score: 0.9385354518890381, index: 8, word: 'pt' },
  //   { entity: 'I-PER', score: 0.9139659404754639, index: 9, word: 'iste' },
  //   { entity: 'I-LOC', score: 0.9877734780311584, index: 15, word: 'montré' },
  //   { entity: 'I-LOC', score: 0.9891639351844788, index: 16, word: 'al' },
  //   { entity: 'I-DATE', score: 0.9858269691467285, index: 18, word: 'fe' },
  //   { entity: 'I-DATE', score: 0.9780661463737488, index: 19, word: 'vr' },
  //   { entity: 'I-DATE', score: 0.980688214302063, index: 20, word: '2012' }
  // ]
  

  

• WavLM for feature-extraction (models). For example:

  import { AutoProcessor, AutoModel, read_audio } from '@xenova/transformers';
  
  // Read and preprocess audio
  const processor = await AutoProcessor.from_pretrained('Xenova/wavlm-base');
  const audio = await read_audio('https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/jfk.wav', 16000);
  const inputs = await processor(audio);
  
  // Run model with inputs
  const model = await AutoModel.from_pretrained('Xenova/wavlm-base');
  const output = await model(inputs);
  // {
  //   last_hidden_state: Tensor {
  //     dims: [ 1, 549, 768 ],
  //     type: 'float32',
  //     data: Float32Array(421632) [-0.349443256855011, -0.39341306686401367,  0.022836603224277496, ...],
  //     size: 421632
  //   }
  // }
  

• MBart +MBart50 for multilingual translation (models). For example:

  import { pipeline } from '@xenova/transformers';
  let translator = await pipeline('translation', 'Xenova/mbart-large-50-many-to-many-mmt');
  let output = await translator('संयुक्त राष्ट्र के प्रमुख का कहना है कि सीरिया में कोई सैन्य समाधान नहीं है', {
    src_lang: 'hi_IN', // Hindi
    tgt_lang: 'fr_XX', // French
  });
  // [{ translation_text: 'Le chef des Nations affirme qu 'il n 'y a military solution in Syria.' }]
  

  See here for the full list of languages and their corresponding codes.

• BeiT for image classification (models):

  import { pipeline } from '@xenova/transformers';
  let url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/tiger.jpg';
  let pipe = await pipeline('image-classification', 'Xenova/beit-base-patch16-224');
  let output = await pipe(url);
  // [{ label: 'tiger, Panthera tigris', score: 0.7168469429016113 }]
  

• ResNet for image classification (models):

  import { pipeline } from '@xenova/transformers';
  let url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/tiger.jpg';
  let pipe = await pipeline('image-classification', 'Xenova/resnet-50');
  let output = await pipe(url);
  // [{ label: 'tiger, Panthera tigris', score: 0.7576608061790466 }]
  

😍 Over 150 newly-converted models

To get started with these new architectures (and expand coverage for other models), we're releasing over 150 new models on the Hugging Face Hub! Check out the full list here.

🏋️ HUGE reduction in model sizes (up to -40%)

Thanks to a recent update of 🤗 Optimum, we were able to remove duplicate weights across various models. In some cases, like whisper-tiny's decoder, this resulted in a 40% reduction in size! Here are some improvements we saw:

• Whisper-tiny decoder: 50MB → 30MB (-40%)
• NLLB decoder: 732MB → 476MB (-35%)
• bloom: 819MB → 562MB (-31%)
• T5 decoder: 59MB → 42MB (-28%)
• distilbert-base: 91MB → 68MB (-25%)
• bart-base decoder: 207MB → 155MB (-25%)
• roberta-base: 165MB → 126MB (-24%)
• gpt2: 167MB → 127MB (-24%)
• bert-base: 134MB → 111MB (-17%)
• many more!

Play around with some of the smaller whisper models (for automatic speech recognition) here!

Other

• Transformers.js integration with LangChain JS (docs)

  import { HuggingFaceTransformersEmbeddings } from "langchain/embeddings/hf_transformers";
  
  const model = new HuggingFaceTransformersEmbeddings({
    modelName: "Xenova/all-MiniLM-L6-v2",
  });
  
  /* Embed queries */
  const res = await model.embedQuery(
    "What would be a good company name for a company that makes colorful socks?"
  );
  console.log({ res });
  /* Embed documents */
  const documentRes = await model.embedDocuments(["Hello world", "Bye bye"]);
  console.log({ documentRes });
  

• Refactored PreTrainedModel to require significantly less code when adding new models

• Typing improvements by @kungfooman