Tech Trend - Developer Led B2Bi Infrastructure
5 Areas to Watch: Part 4
As early-stage technology investors, we see a wide range of companies working on game-changing businesses. This 5 Part Blog Series shares what our firm has been most intrigued by within the early-stage Enterprise B2B IT Infrastructure space in the first half of 2019.Trends we noticed led to the 5 Areas to Watch which we’ll cover in this 5 Part Series:
- Part 1: Privacy Operations Infrastructure (PrivOps)
- Part 2: Kubernetes Management
- Part 3: API Security
- Part 4: Developer Led B2Bi Infrastructure (3rd-Gen Data Flow/ Integration Tools)
- Part 5: Intelligent Enterprise Applications
Developer Led B2Bi Infrastructure (3rd-Gen Data Flow/ Integration Tools)In order to compete in the modern economy, you must be a data-driven enterprise, which requires you to find ways to ingest and manage large volumes of data. Developers want to build, manage and scale these data pipelines. First Generation Companies (Informatica), Second Generation Companies: SnapLogic (pre-built) and MuleSoft (API centric) don’t fit the bill. One exciting project we came across is Apache Airflow, which is a data workflow orchestration tool developed and open-sourced by Airbnb. This technology is being heavily leveraged by top data-driven companies like Lyft, Twitter, PayPal, and Apple. Airflow saw a record 303,000 downloads in August 2019 — a 3x increase compared to August 2018. (Update April 2021: in March 2021 Apache Airflow had about 1,750,000 downloads). Data workflow orchestration is a critical part of any modern data stack and Airflow has emerged as the winner. Sierra Ventures recently made an investment in a company in the space called Astronomer that is committing to further enhance Airflow and build enterprise-grade features around it. We ultimately liked the company’s approach for the following reasons:
- Developer Led. Data Engineers love Airflow because data workflows are authored as code, which means they are more configurable, maintainable, and testable.
- Extensible. Every organization has a unique set of business rules and requirements; Airflow can be extended to fit the level of abstraction that suits any environment.
- Scalable. Leveraging Kubernetes enables Airflow to scale up to meet demand and down to almost nothing when idle.
Resources: