What's Changed

garak's 1st birthday! 🎂

Headlines in this birthday release:

• Multimodal support! LLaVa + FigStep - HUGE thanks to @DavidLee528
• LiteLLM generator support - thanks to @Tien-Cheng!
• DoNotAnswer probe for prompts an LLM should decline - thanks to @AhsanAyub!
• LangChain Serve generator support - thanks to @GustavFredrikson!
• Support for NIM generators, NVIDIA Inference Microservices
• OpenAI abstraction with parallelisation support
• Windows compatibility enhanced, with optional testing in the workflow
• Hugging Face speedup

plugins

• LiteLLM generator by @Tien-Cheng in https://github.com/leondz/garak/pull/572
• Support for using 'langchain serve' endpoints by @GustavFredrikson in https://github.com/leondz/garak/pull/588
• Enable parallel OpenAI calls by @jmartin-tech in https://github.com/leondz/garak/pull/645
• Multi-modal Jailbreaking Attack on LLaVA by @DavidLee528 in https://github.com/leondz/garak/pull/587
• bump openai module version to match paths in latest litellm by @leondz in https://github.com/leondz/garak/pull/664
• generator: NIM by @leondz in https://github.com/leondz/garak/pull/637
• Probe: Do Not Answer by @AhsanAyub in https://github.com/leondz/garak/pull/608

architecture

• change supported pythons to 3.10-3.12 by @leondz in https://github.com/leondz/garak/pull/503
• add more detailed TAP docs by @leondz in https://github.com/leondz/garak/pull/504
• add multiple-result aggregator by @leondz in https://github.com/leondz/garak/pull/505
• add post buff hook by @erickgalinkin in https://github.com/leondz/garak/pull/506
• Add fleshed-out docs to all probes by @leondz in https://github.com/leondz/garak/pull/507
• add bibtex under citation info in readme by @leondz in https://github.com/leondz/garak/pull/511
• define broad test, all probes, 1 gen per by @leondz in https://github.com/leondz/garak/pull/514
• Feature/taxonomy payloads by @leondz in https://github.com/leondz/garak/pull/519
• include paraphrasing in broad conf by @leondz in https://github.com/leondz/garak/pull/521
• choose whether buffing will also include the original prompt by @leondz in https://github.com/leondz/garak/pull/523
• add config var for capping max # buffed prompts to add per buff by @leondz in https://github.com/leondz/garak/pull/526
• document Probe.probe(); skip a buff hook if no buffs by @leondz in https://github.com/leondz/garak/pull/527
• add type hints to base.Probe; fix base probe rst by @leondz in https://github.com/leondz/garak/pull/528
• Bump datasets package by @shubhobm in https://github.com/leondz/garak/pull/536
• Add ConversationalPipeline for huggingface models by @erickgalinkin in https://github.com/leondz/garak/pull/539
• add generator for supporting openai module v0.x by @leondz in https://github.com/leondz/garak/pull/553
• Update README.md by @erickgalinkin in https://github.com/leondz/garak/pull/558
• Minor typo in FAQ by @jmartin-tech in https://github.com/leondz/garak/pull/562
• Add additional error message when doc is None type by @DavidLee528 in https://github.com/leondz/garak/pull/566
• shared constant & string literal by @jmartin-tech in https://github.com/leondz/garak/pull/571
• Spelling corrections for multiple locations by @jmartin-tech in https://github.com/leondz/garak/pull/564
• Reduce Huggingface GPU utilization by @erickgalinkin in https://github.com/leondz/garak/pull/567
• skip verbose flag in secondary parser by @jmartin-tech in https://github.com/leondz/garak/pull/576
• Added project twitter link and corrected a grammatic error by @codebrain001 in https://github.com/leondz/garak/pull/578
• Convert GGML to expect GGUF format by @jmartin-tech in https://github.com/leondz/garak/pull/581
• Update workflows: CLA asst bump, PR & manual testing by @leondz in https://github.com/leondz/garak/pull/591
• add test de-duping using skip-duplicate-actions by @leondz in https://github.com/leondz/garak/pull/597
• Remove #! entries from files not intended as executables by @jmartin-tech in https://github.com/leondz/garak/pull/612
• Further align shebangs with code that has executable entry points by @leondz in https://github.com/leondz/garak/pull/613
• interactive mode intro by @leondz in https://github.com/leondz/garak/pull/614
• add tests for ggml generator by @jmartin-tech in https://github.com/leondz/garak/pull/618
• add var for generator context_len and populate this for some generators by @leondz in https://github.com/leondz/garak/pull/616
• allow generators.Base.generate() to take an optional param specifying generation count by @leondz in https://github.com/leondz/garak/pull/600
• Enable windows tests as github action by @jmartin-tech in https://github.com/leondz/garak/pull/626
• add on-demand macos testing by @leondz in https://github.com/leondz/garak/pull/631
• macOS test install from correct path by @jmartin-tech in https://github.com/leondz/garak/pull/633
• consolidate test file cleanup by @jmartin-tech in https://github.com/leondz/garak/pull/634
• bump discord link by @leondz in https://github.com/leondz/garak/pull/648
• meta the arguments a bit for GET vs other request types by @jmartin-tech in https://github.com/leondz/garak/pull/640

stability

• fail gracefully if nvcf rejects input; compact zalgo prompts by @leondz in https://github.com/leondz/garak/pull/509
• log & skip past NVCF 4xx errors by @leondz in https://github.com/leondz/garak/pull/533
• fix empty autodan prompts & poor detector behaviour by @leondz in https://github.com/leondz/garak/pull/534
• Fix AutoDAN issues by @erickgalinkin in https://github.com/leondz/garak/pull/537
• fix bad nonetype handling in atkgen probe by @leondz in https://github.com/leondz/garak/pull/538
• Division by zero error fixed in HTML report generation by @CoderMayhem in https://github.com/leondz/garak/pull/545
• cap cohere lib version by @leondz in https://github.com/leondz/garak/pull/569
• rm deprecated model from example by @leondz in https://github.com/leondz/garak/pull/575
• Attack fixes by @erickgalinkin in https://github.com/leondz/garak/pull/555
• More regex as string literal by @jmartin-tech in https://github.com/leondz/garak/pull/586
• Bugfix/action dedupe by @leondz in https://github.com/leondz/garak/pull/598
• wrap cli exec to gracefully catch keyboard exit signal by @jmartin-tech in https://github.com/leondz/garak/pull/603
• Enforce warning output for garak classes. by @jmartin-tech in https://github.com/leondz/garak/pull/605
• bump hf transformers v to avoid transformers#30076 by @leondz in https://github.com/leondz/garak/pull/636
• update avidtools to remove typing reference by @jmartin-tech in https://github.com/leondz/garak/pull/639
• torch v bump by @leondz in https://github.com/leondz/garak/pull/649
• Pause FigStepTiny by @leondz in https://github.com/leondz/garak/pull/652
• Bugfix/visual jailbreak pause by @leondz in https://github.com/leondz/garak/pull/653
• limit push test to main by @jmartin-tech in https://github.com/leondz/garak/pull/661
• Update MANIFEST.in so all resources are installed by @JKL98ISR in https://github.com/leondz/garak/pull/660
• handle extant but closed hitlogfile file by @leondz in https://github.com/leondz/garak/pull/665

New Contributors

• @CoderMayhem made their first contribution in https://github.com/leondz/garak/pull/545
• @jmartin-tech made their first contribution in https://github.com/leondz/garak/pull/562
• @codebrain001 made their first contribution in https://github.com/leondz/garak/pull/578
• @Tien-Cheng made their first contribution in https://github.com/leondz/garak/pull/572
• @GustavFredrikson made their first contribution in https://github.com/leondz/garak/pull/588
• @JKL98ISR made their first contribution in https://github.com/leondz/garak/pull/660
• @AhsanAyub made their first contribution in https://github.com/leondz/garak/pull/608

Full Changelog: https://github.com/leondz/garak/compare/v0.9.0.12...v0.9.0.13