Prefect Technologies, Inc., the company that produces the open-source data workflow automation platform Prefect, announced the release of a new report by leading data and AI consulting firm Gradient Flow titled the “2022 State of Workflow Orchestration.” Sponsored by Prefect, the survey report reveals a growing demand for workflow orchestration solutions, which is being driven
by ongoing company initiatives to modernize their data infrastructures and enhance datasets for machine learning (ML) and AI models. Data professionals at these organizations are looking to orchestrators to help write, schedule, monitor, and manage their data and ML pipelines.
Usage of orchestration
For the report, Gradient Flow surveyed 581 respondents from a variety of industries, with roles including data scientists and analysts, data and ML engineers, software engineers and developers, and DevOps engineers. The responses indicated strong adoption of workflow orchestration and dataflow automation tools, with 43% of respondents stating that they currently use an orchestrator to handle over half of their recurring tasks. Notably, among the data scientists and analysts surveyed, 52% reported using an orchestrator for up to a quarter of recurring tasks. Moreover, 91% of the data and ML engineers said they use orchestrators for more than a quarter of their tasks.
When asked about their primary use cases for workflow orchestration, 29% of respondents all cited data science, making it the most popular application of workflow orchestration. The use cases shifted when surveying data and ML engineers, given their focus on moving and transforming data. For these respondents, data movement (21%), MLOps (16%), and data transformation (14%) were their top use cases for orchestration.
Ben Lorica, Founder and Principal of Gradient Flow, said, “More organizations now want to enable analytics and AI to benefit their businesses, creating demand for not only data talent but also foundational data technologies. This includes everything from data integration and DataOps to data governance and platforms—as well as orchestration. Organizations need orchestrators to make data available for downstream applications including data science and AI systems. The growing demand for workflow orchestration has led to the emergence of a host of open source and SaaS solutions in an area ripe for innovation.”
Also Read: Mage Data™ Selected as SC Media Trust Award Finalist
Orchestration solutions
The two most popular orchestration solutions among the respondents are open-source workflow management platform Apache Airflow—used by 36% of the respondents—and the Prefect dataflow automation platform—used by 14% of all respondents.
While 46% of the surveyed data scientists and analysts reported they use Airflow, a notable 82% of respondents in these roles indicated that they plan to use a new orchestration tool in the next year. The data and ML engineers in the survey also expressed strong interest in trying Prefect and Dagster (both 18%) over Airflow (15%), while Prefect (at 14%) was the top solution that software engineers, developers and those in DevOps wanted to try for their next project.
“This report shows that the market for orchestration is growing significantly and starting to evolve beyond legacy products,” added Jeremiah Lowin, Founder and CEO of Prefect. “Data scientists, engineers, and developers all ultimately need dataflow automation that will ensure their code runs and their data arrives as expected, while most importantly making it easy to identify when a failure occurs and how to fix it. The result is less time spent writing defensive code to protect against failure, and more time on productive objectives.”
Prefect is a dataflow automation platform for data engineers to build, automatically run, and monitor complex pipelines at any scale. Prefect is used by organizations of all sizes across the globe,