Building production ready distributed machine learning pipelines can be a challenge for even the most seasoned researcher or engineer. We are excited to announce the release of SynapseML v0.11.0 (Previously MMLSpark), an open-source library that aims to simplify the creation of massively scalable machine learning pipelines. SynapseML unifies several existing ML Frameworks and new MSFT algorithms in a single, scalable API that’s usable across Python, R, Scala, Java, .NET, C#, and F#.

Highlights

ChatGPT and GPT-4 at Scale	Simple Deep Learning	LightGBM v2
Intelligent chat and embeddings. Simplified Prompting APIs.	Train custom image and text classifiers with ease	Higher performance, >10x lower memory footprint, same API
View Notebook	Learn More	Try an example

ONNX Model Hub	Causal Learning	Vowpal Wabbit v2
Embed >150 state of the art deep networks into your pipelines	Discover and measure causal treatment effects	New second generation integration
Learn More	View Docs	Explore Samples

New Features

General ✨

• R Support is no longer Beta! (#1586)
• Support for Spark 3.2.3

Open AI 🤖

• Add OpenAI Prompt Template support (#1843)
• Add Azure OpenAI embedding support (#1832)
• Add Azure Active Directory authentication for OpenAI (#1829)
• Add Null-value handling for OpenAI models (#1854)

Deep Learning 🕸

• Remove CNTK functionality and replace with ONNX (#1593)
• Add the DeepTextClassifier a simple API for fine tuning a wide array of Hugging Face 🤗 text transformers using PyTorch Lightning (#1591)
• Add the DeepVisionClassifier a simple API for deep transfer learning and fine-tuning of a variety of vision backbones (#1518)

Azure Cognitive Services for Big Data 🧠

• Add SpeakerEmotionInference transformer to generate emotion annotation tags for emotive reading in SpeechToText (#1691)
• Add new AnalyzeText API (#1760)
• Support Azure Active Directory (AAD) authentication for the cognitive services (#1778, #1797)
• Move different cognitive services into sub packages (#1746)
• Add audiobook generation example (#1852)
• Add a notebook for advanced cognitive service usage (#1825)
• Upgrade MVAD to v1.1 (#1788)
• Remove MVAD's dependence on hardwired credentials and azure SDKs (#1629)
• Add word-level timing to SpeechToTextSDK and ConversationTranscription (#1801)
• Add the descriptionExcludes parameter to AnalyzeImage (#1590)

Causal Learning 📈

• Add the causal DoubleMLEstimator for learning causal treatment effects from data (#1715)
• Add a DoubleMLEstimator document and sample notebook (#1730)
• Fix DML regression bug, should remove both treatment and outcome columns as feature columns (#1820)
• Add TreatmentCol type checking (#1816)
• Update test to validate ATE value should be positive (#1821)
• Fix issue with missing causal test coverage (#1799)

LightGBM 🌳

• Add LightGBM streaming execution mode for more reliable performance with orders of magnitude less memory. (#1580)
• Add maxNumClasses param to LightGBMClassifier for multi-class (#1841)
• Added the passThroughArgs feature which allows users to set low level LGBM parameters before they are wrapped in SparkML (#1749)

Vowpal Wabbit 🐇

• Vowpal Wabbit v2 (#1579):
  • Support Vowpal Wabbit input format using VowpalWabbitGeneric model
  • Support additional algorithms & label types (multi-class, cost sensitive one against all): sample notebook
  • Progressive validation (aka 1-step ahead) using VowaplWabbitGenericProgressive
  • New Contextual Bandit Offline Policy Evaluation Notebook
  • Data parallel training independent of cluster size

Additional Updates

Bug Fixes 🐞

• Support grayscale images in toNDArray (#1592)
• Adjust learning rate in VW example notebook (#1853)
• Correct copy/paste error in acr cleanup (#1838)
• Fix synapse test config, and isolation forest notebook (#1833)
• Add spark config to fix ArrayStoreException (#1757)
• Fix breeze NoSuchMethodError (#1807)
• Fix modelVersion param in TextAnalytics (#1756)
• Make logging infrastructure consistent and add logging checks (#1755)
• Fix website sidebars and vulnerabilities in packages (#1753)
• Remove Vowpal Wabbit exclusion, add Interpretability exclusion (#1708)
• Update isolation forest notebook (#1696)
• Remove error on invalid columns in DropColumns (#1695)
• Fix PyArrow failure in deeplearning test (#1689)
• Fix linked service setters on cog service base class (#1685)
• KernelSHAP throws error when the key type in the ZipMap output is LongType (#1656)
• Fix flaky translate tests (#1643)
• Fix speechToTextSuite serialization Fuzzing failure (#1626)
• Fix translator endpoint and update all endpoints for gov regions (#1623)
• Finder runtime issues (#1598)
• Clean up cluster if Databricks tests pass (#1599)
• Fix deep-learning test flakiness (#1600)
• Update DotnetTestBase assembly version (#1601)
• Fix flaky forms test (#1584)
• Fix namespace import for Experimental (#1780)

Build 🏭

• Automate cleanup of Azure Container Registry images (#1787, #1751, #1735, #1814)
• Add bot to remove stale issues (#1602)
• Return values from TaskKeys (#1775)
• Remove unnecessary SbtPlugin settings (#1771)
• Simplify E2E test pipeline with a test matrix
• Add welcome message to new PRs/Issues (#1573, #1583)
• Add workflow to label new and reopened issues (#1571)
• Add a secret scanner (#1724)
• Add workflow to open GitHub issues after a comment (#1676)
• Add workflow to remove awaiting-response issue label on comment (#1674)
• Publish test jars so downstream projects can depend on test configurations and utilities
• Making build secrets optional and cached to remove 1 min latency on sbt commands (#1726)
• Add nightly build to catch flakes early (#1774)
• Automatically delete accumulated models in build (#1758, #1729, #1759)
• Add Dependabot for updating GitHub actions (#1608)
• Update build pipeline to ubuntu 20.04 (#1624)

Documentation 📘

• Add a hyperparameter tuning sample with HyperOpt (#1828)
• Add docs for new LightGBM executionMode parameter (#1779)
• Add additional ONNX docs for model hub and slicing (#1781)
• Improve OpenAI notebook (#1596)
• Add dotnet installation & examples (#1567, #1570)
• Update deep vision docs (#1752)
• Add custom search engine creation video to website (#1581)
• Replace Boston housing dataset with California housing dataset (#1856)
• Improve overview section in README
• Fix typo in old versions of Interpretability - Explanation Dashboard.md (#1846)
• Add versioned docs (#1858, #1566)
• Fix Synapse installation instructions for Spark 3.2 (#1815)
• Update required spark and python version on website doc (#1812)
• Fix latex rendering issue in Data Balance Analysis (#1796)
• Fix Acrolinx issues (#1792, #1793, #1808, #1794)
• Pin binder to latest released version
• Improve python env creation instructions in developer readme (#1693)
• Remove unused docs and fix links
• Improve example notebooks
• fix command to launch Jupyter notebook (#1649)
• Add documentation for MLFlow logging and loading (#1641)
• Update spark version in Readme
• Fix .NET logo on website (#1604)
• Update v0.10.0 installation guidance (#1578)

Maintenance 🔧

• Bump spark to 3.2.3 (#1744)
• Update Scalatest and Scalactic dependencies (#1706)
• Keep GitHub actions up to date (#1839, #1766, #1737, #1680, #1688, #1777, #1773, #1770, #1786, #1610, #1611, #1612)
• Keep website up to date (#1826, #1809, #1785, #1719, #1709, #1609, #1740)
• Update container registry and service principal connections
• Maintain Synapse tests (#1835, #1823, #1810, #1803, #1658)
• Fix codecov.yaml
• Fix conda env creation flakiness in build (#1748)
• Bump SynapseML Version (#1857, #1738, #1628)
• Improve code style (#1736, #1716, #1790)
• Pin az and python versions in build (#1705)
• Fix ado issue linker configurations (#1704)
• Add synapse-internal to platform detector function (#1651)
• Update OpenAI service to official deployment (#1619)
• Pin binder version (#1607)

Deprecations and Removals 🗑️

• Deprecate old TextAnalytics APIs (#1627)
• Remove old TextAnalytics APIs (#1622)
• Remove deprecated LIME APIs (#1620)
• Deprecate CNTK classes and ModelDownloader (#1712)
• Delete CNTK and related utils (#1743)
• Move ImageFeaturizer to onnx namespace (#1711)

Testing 💚

• Add Additional E2E testing infrastructure (#1727, #1769)
• Improve ONNX test reliability (#1713)
• Stabilize flaky tests (#1576, #1842)
• Remove Synapse E2E test exclusions (#1757, #1699, #1798, #1698, #1837)
• Add automated tests for getters and setters and improve test coverage (#1631)

Contributor Spotlight

We are excited to highlight the contributions of the following SynapseML contributors:

Scott Votaw	Serena Ruan	Haizhou (Dylan) Wang
Scott Votaw is a Principal Engineer on the SynapseML team has solved some of SynapseML’s toughest challenges in record time. In this release, Scott contributed both the new LightGBM streaming execution mode, and fully replaced our deep learning stack with the ONNX Runtime. These efforts were massive lifts including huge changes to the LightGBM native libraries and complex dependency management jujitsu respectively. Scott brings his love for the craft to every project he works on so keep your eyes peeled for more amazing feats of engineering from him in future releases.	Serena is a Software Engineer II on the SynapseML team and operates on a separate plane of existence than the rest of us mere mortals. Following up on prior major contributions like .NET support, form recognition, translation, and creating the SynapseML Website, Serena contributed the Simple Deep Learning package for this release. This package makes it easy to train modern deep text and vision networks from Hugging Face and torchvision on Spark clusters. Serena seeks only the most difficult engineering challenges and her contributions have laid the groundwork for many more deep-learning based algorithms in SynapseML.	Haizhou (Dylan) is a Senior Software engineer in the CSX Data team and a first-time contributor to the SynapseML library. Dylan contributed the new SynapseML causal learning package for the v0.11 release. This package helps users discover the effectiveness of things like medical treatments or economic policies even without controlled experiments. With his elegant contributions, Dylan has laid the foundation for more causal collaborations with the EconML library.
Markus Cozowicz	Brendan Walsh	Jessica Wang
Markus is a Principal Applied Scientist who (just!) joined the SynapseML team. Despite only recently coming on board officially, Markus has long been a prolific contributor to the library and built the Vowpal Wabbit and Isolation Forest integrations. In this release, Markus contributed the second generation of the Vowpal Wabbit integration, improving its generality and applicability. He also expanded the OpenAI integration to support embeddings and simplified prompt templating. Our team is incredibly lucky to have such a consistent and thoughtful collaborator.	Brendan is a Senior Engineer on the SynapseML team who recently joined after a long tenure on the Cognitive Services team where he developed their containerized cognitive service effort and co-authored the SynapseML publication on large-scale microservices. Brendan used this expertise to onboard Emotion Detection for text to speech models. He then went on to use this new emotive reading capability to create and donate thousands of audiobooks to the open source. You can learn more about Brendan’s awesome technical philanthropy efforts at https://aka.ms/audiobook.	Jessica is Software Engineer who recently joined the SynapseML team. Already, Jessica has grown into the role of the SynapseML benevolent “doc”tator. This release Jessica has worked hard to ensure that the SynapseML notebooks work across a wide variety of Spark platforms and are easy and simple to get started with. This work requires knowledge of the entire library’s surface area, and we are thankful Jessica has worked so hard to learn this breadth of content. If you have been following notebook examples from https://aka.ms/spark you have Jessica to thank!
Kyle Rush	Avrilia Floratou	Jason Wang
Kyle is a Senior Software Engineer on the SynapseML team with a penchant for architecture and a streak of taking on big responsibility behind the scenes. Kyle has been instrumental in expanding our testing infrastructure to new platforms so that the lights stay on even as the number of contributions increases. This often requires nontrivial code and delicate cross-team collaboration, and Kyle has both the engineering might and the charismatic finesse to make sure these systems can be spun up successfully.	Avrilia is Principal Scientist Manager in the Grey Systems Lab, first-time SynapseML contributor, and a delightful collaborator. This release, Avrilia contributed the first prototype of the simplified OpenAI prompting transformer. This contribution makes it easy to ask ChatGPT and other LLMs questions about large datasets and to create new LLM-derived columns in databases. You can learn more about her work through the OpenAI Docs and prompting demo	Jason Wang is a Principal Software Engineering on the CSX Data team and has a long history of not only contributing huge features to SynapseML, but actively maintaining his contributions. This release, Jason’s work on the ONNX model hub protocol enables quick access to over >150 pretrained deep networks from the Java and Scala ecosystems. Jason has also been instrumental in fixing the most difficult and arduous bugs, some even stemming from the core Spark runtime. Finally, we deeply appreciate Jason’s leadership in the community: he consistently encourages and helps others contribute, and his impact extends far beyond his own personal contributions.

Acknowledgements

We would like to acknowledge the developers and contributors, both internal and external, who helped create this version of SynapseML

Eric Dettinger, Markus Weimer, Serena Ruan @serena-ruan, Scott Votaw @svotaw, Haizhou (Dylan) Wang @dylanw-oss, Puneet Pruthi @ppruthi, Markus Cozowicz @eisber, Brendan Walsh @BrendanWalsh, Jessica Wang @JessicaXYWang, Kyle Rush @k-rush, Avrilia Floratou, Jason Wang @memoryz, Mark Niehaus @niehaus59, Keerthi Yanda @KeerthiYandaOS, Ilya Matiach @imatiach-msft, Kashyap Patel @ms-kashyap, Martha Laguna @martthalch @marthalc, Sarah Shy @sarahshy, @ocworld, @adityakode, @nightscape, Alexandra Savelieva @alsavelv, Tom Finley, Jeff Zheng, James Verbus @jverbus, Chris Hoder, Misha Desai, Nellie Gustafsson, Eren Orbey, Beverly Kodhek, Louise Han @jr-MS, Raj Rikhy, Marcos Campos, Mike Estee, Brice Chung, Justyna Lucznik, Kim Manis, Mitrabhanu Mohanty, Bogdan Crivet, Anand Raman, William T. Freeman, Akshaya Annavajhala (AK), Guolin Ke, Spark.NET Team, ONNX Team, Azure Global, Vowpal Wabbit Team, LightGBM Team, MSFT Garage Team, MSR Outreach Team, Speech SDK Team, MLflow Team

Learn More

Visit our website for the latest docs, demos, and examples	Watch our AI Show episode on creating and donating thousands of audiobooks	Learn the basics of SynapseML
Read the v0.11 Update blog	Apply OpenAI language models to your large datasets	Learn how to check data quality with GPT-3

Changes:

• 7b23764a93e73eab9c8903cb5f37e2438caedce0 docs: make v0.11.0 docs (#1858)
• 795bbecf5d1c803c549e78a0a940f5d7ef607127 docs: Add audiobook generation example (#1852)
• 8f9e970dd5df96db28a64b45687bf7e17adb8dd8 chore: bump version to 0.11.0 (#1857)
• 09648a32308f055d0cecf63eda2f82a2d2914d4c docs: replace Boston housing dataset with California housing dataset (#1856)
• 2b2419f1458634208b7127c557dc98717ce0b455 fix: OpenAI completion & prompting null handling (#1854)
• 66f30e30dfbcb8295ab29fe27b6e2501e87d5ac8 fix: adjust learning rate in VW example notebook (#1853)
• d64563f8efcb55a855d48ddea066ee361722ba5a chore: Upgrading ONNX version to fix assembly bug (#1849)
• a64b6e09e9c7e96f62ae8fd9a72663f66474d1a9 chore: fix mcr connection
• d23dd46e87cf0688c46c1ebedae1a4a7048186a2 chore: update build service principal
• 6876d53a95d71dc5c1fe66650d3f7b72b839fca1 chore: Update the build pipeline