Overview

This is a new major release adding integer quantization and partial GPU (NVIDIA) support

Integer quantization

This allows the ggml Whisper models to be converted from the default 16-bit floating point weights to 4, 5 or 8 bit integer weights. The resulting quantized models are smaller in disk size and memory usage and can be processed faster on some architectures. The transcription quality is degraded to some extend - not quantified at the moment.

• Supported quantization modes: Q4_0, Q4_1, Q4_2, Q5_0, Q5_1, Q8_0
• Implementation details: https://github.com/ggerganov/whisper.cpp/pull/540
• Usage instructions: README
• All WASM examples now support Q5 quantized models: https://whisper.ggerganov.com

Here is a quantitative evaluation of the different quantization modes applied to the LLaMA and RWKV large language models. These results can give an impression about the expected quality, size and speed for quantized Whisper models:

LLaMA quantization (measured on M1 Pro)

ref: https://github.com/ggerganov/llama.cpp#quantization

RWKV quantization

ref: https://github.com/ggerganov/ggml/issues/89#issuecomment-1528781992

This feature is possible thanks to the many contributions in the llama.cpp project: https://github.com/users/ggerganov/projects/2

GPU support via cuBLAS

Using cuBLAS results mainly in improved Encoder inference speed. I haven't done proper timings, but one can expect at least 2-3 times faster Encoder evaluation with modern NVIDIA GPU cards compared to CPU-only processing. Feel free to post your Encoder benchmarks in issue #89.

• Implementation details: https://github.com/ggerganov/whisper.cpp/pull/834
• Usage instructions: README

This is another feature made possible by the llama.cpp project. Special recognition to @slaren for putting almost all of this work together

This release remains in "beta" stage as I haven't verified that everything works as expected.

What's Changed

• Updated escape_double_quotes() Function by @tauseefmohammed2 in https://github.com/ggerganov/whisper.cpp/pull/776
• examples : add missing #include by @pH5 in https://github.com/ggerganov/whisper.cpp/pull/798
• Flush upon finishing inference by @tarasglek in https://github.com/ggerganov/whisper.cpp/pull/811
• Escape quotes in csv output by @laytan in https://github.com/ggerganov/whisper.cpp/pull/815
• C++11style by @wuyudi in https://github.com/ggerganov/whisper.cpp/pull/768
• Optionally allow a Core ML build of Whisper to work with or without Core ML models by @Canis-UK in https://github.com/ggerganov/whisper.cpp/pull/812
• add some tips about in the readme of the android project folder by @Zolliner in https://github.com/ggerganov/whisper.cpp/pull/816
• whisper: Use correct seek_end when offset is used by @ThijsRay in https://github.com/ggerganov/whisper.cpp/pull/833
• ggml : fix 32-bit ARM NEON by @ggerganov in https://github.com/ggerganov/whisper.cpp/pull/836
• Add CUDA support via cuBLAS by @ggerganov in https://github.com/ggerganov/whisper.cpp/pull/834
• Integer quantisation support by @ggerganov in https://github.com/ggerganov/whisper.cpp/pull/540

New Contributors

• @tauseefmohammed2 made their first contribution in https://github.com/ggerganov/whisper.cpp/pull/776
• @pH5 made their first contribution in https://github.com/ggerganov/whisper.cpp/pull/798
• @tarasglek made their first contribution in https://github.com/ggerganov/whisper.cpp/pull/811
• @laytan made their first contribution in https://github.com/ggerganov/whisper.cpp/pull/815
• @wuyudi made their first contribution in https://github.com/ggerganov/whisper.cpp/pull/768
• @Canis-UK made their first contribution in https://github.com/ggerganov/whisper.cpp/pull/812
• @Zolliner made their first contribution in https://github.com/ggerganov/whisper.cpp/pull/816
• @ThijsRay made their first contribution in https://github.com/ggerganov/whisper.cpp/pull/833

Full Changelog: https://github.com/ggerganov/whisper.cpp/compare/v1.3.0...v1.4.0