This release gets Gemma2 working closer to how Google intended.

• af22695 Make gemma2-27b-it the same as aistudio.google.com
• 41678c8 Add sliding window mask for Gemma2
• 140eed5 Add soft-capping to Gemma2

This release fixes Android support. You can now run LLMs on your phone using Cosmopolitan software like llamafile. Thank you @aj47 (techfren.net) for bug reports and and testing efforts. See also other bug fixes described by the Cosmopolitan v3.5.4 and v3.5.3 release notes.

Our future replacement for the server now has an /embedding endpoint. On my workstation, it's currently able to serve 851 requests per second for a prompt with 52 tokens, using the all-MiniLM-L6-v2.Q6_K.gguf embeddings model. None of the requests fail and 99th percentile latency is 56.74ms.

• 1346ef4 Create /embedding endpoint in new server
• 263d39b Use float to string conversion
• 0d62d05 Reclaim llama_decode() memory on cancelation
• 617d841 Remove ggml_context cache
• 46dda4f Refactor new server and get leak checker working
• cd73243 Prevent vector overflow in llama.cpp

You can try the new embedding server as follows:

make -j o//llamafile/server/main
o//llamafile/server/main -m /weights/all-MiniLM-L6-v2.F32.gguf
curl http://127.0.0.1:8080/embedding?prompt=orange

Compatibility with the old server's API of posting JSON content will be added in upcoming changes. The same goes for the OpenAI API. The goal's to be compatible with everything.