Highlights

🏠 NAT traversal & relays. Now, servers can join the swarm automatically even if your machine is located behind a NAT or a firewall, or has a dynamic IP address. You don't have to manually set up port forwarding or provide any arguments to make it work.

• Please upgrade the Petals package and restart all your servers & clients to use this feature or access servers joined via relays:

  pip install --upgrade petals

• How does it work? If the server learns that it can't accept incoming connections due to NAT/firewall, it opens a long-term outcoming connection to one of relay nodes, then the relay node forwards all requests to this server through this connection. In turn, any server with a public IP may serve as a relay node if necessary. We use libp2p circuit relays under the hood: https://docs.libp2p.io/concepts/nat/circuit-relay/

💬 Chatbot app. We've released a chatbot app working over Petals: http://chat.petals.ml (source code).

• Disclaimer: This chatbot uses the regular BLOOM, which is not fine-tuned for question answering. Please do not expect it to behave like ChatGPT.

• How does it work? Under the hood, this web app uses our HTTP endpoint for running inference using the public Petals swarm. You can use this endpoint for your own projects, or set up another endpoint yourself (no GPU needed). See API docs here: https://github.com/borzunov/chat.petals.ml#http-api-methods

🏃‍♀️ Faster CPU-only clients. If your CPU supports the AVX512 instruction set, a CPU-only client now runs almost as fast as a GPU-enabled one. This way, you can rent cheap CPU instances to run the client or an HTTP endpoint, like the one we use for the chatbot app.

• How to use it? AVX512 is mostly present on late Intel Xeon CPUs. You can rent one by choosing a "dedicated CPU" instance with 16+ GB RAM on DigitalOcean.

🏥 Swarm health monitor. We've updated the swarm health monitor: http://health.petals.ml (source code). It provides an overview of servers who joined the public swarm and reports any connection issues.

What's Changed

• Add PyPI badge, update instructions and links in readme by @borzunov in https://github.com/bigscience-workshop/petals/pull/172
• Add link to PyPI by @borzunov in https://github.com/bigscience-workshop/petals/pull/173
• Add local tensor-parallel fwd/bwd by @justheuristic in https://github.com/bigscience-workshop/petals/pull/143
• Make Docker command more visible by @borzunov in https://github.com/bigscience-workshop/petals/pull/175
• Allow to disable chunked forward by @borzunov in https://github.com/bigscience-workshop/petals/pull/176
• Disable chunked_forward() on AVX512 CPUs by @borzunov in https://github.com/bigscience-workshop/petals/pull/179
• Use slightly less memory in .generate() by @borzunov in https://github.com/bigscience-workshop/petals/pull/177
• Import bitsandbytes only if it's going to be used by @borzunov in https://github.com/bigscience-workshop/petals/pull/180
• hotfix: add initial peer that did not crash :) by @justheuristic in https://github.com/bigscience-workshop/petals/pull/181
• Remove protobuf from requirements by @borzunov in https://github.com/bigscience-workshop/petals/pull/182
• Add more links to BLOOM to readme by @borzunov in https://github.com/bigscience-workshop/petals/pull/183
• Add link to health.petals.ml to readme by @borzunov in https://github.com/bigscience-workshop/petals/pull/184
• Add readme subsections by @borzunov in https://github.com/bigscience-workshop/petals/pull/185
• Fix GiBs in the "insufficient disk space" message by @borzunov in https://github.com/bigscience-workshop/petals/pull/187
• Support libp2p relays for NAT traversal by @Vahe1994 in https://github.com/bigscience-workshop/petals/pull/186
• Fix psutil-related AccessDenied crash, disable --load_in_8bit by default in case of TP by @borzunov in https://github.com/bigscience-workshop/petals/pull/188
• Bump version to 1.1.0 by @borzunov in https://github.com/bigscience-workshop/petals/pull/190

New Contributors

• @Vahe1994 made their first contribution in https://github.com/bigscience-workshop/petals/pull/186

Full Changelog: https://github.com/bigscience-workshop/petals/compare/v1.0.0...v1.1.0