Docker Acqui-hires Engineering Team Behind Aurora, A Crucial Twitter Technology

Docker on Thursday acquired Conductant, a San Francisco-based startup so young, so stealthy, hardly anyone had heard of it.

And instead of a grand product vision, or even any unique intellectual property, what Docker got for its undisclosed sum was three engineers behind a technology that is now a foundation of Twitter's infrastructure.

The acqui-hire of Bill Farner, David Chung and John Sirois -- Conductant's core team -- will bring under Docker's roof unprecedented experience in massively scaling container clusters, Patrick Chanezon, chief developer advocate at Docker, told CRN.

[Related: Docker Introduces Datacenter, A Comprehensive Container-as-a-Service Suite]

"What we're getting is three world-class engineers, who have lots of experience, at places like Google, Twitter and Zynga, in scaling infrastructure for running containers," Chanezon said.

When Farner arrived at Twitter in 2009 after leaving Google, he found himself longing for the Borg scheduler, secretive software used at the Mountain View, Calif.-based Internet giant. That prompted him to launch the Aurora Project.

Aurora, a framework for the open-source Mesos cluster manager, went live at Twitter in 2010, and was open-sourced a few years later. The scale of Twitter's operations -- tens of thousands of machines and about a thousand developers -- uniquely battle tested Aurora, Chanezon said.

"Aurora was built with a very specific goal in mind: technical and operational scalability," Chanezon said. "Lots of teams innovating at high pace using a single infrastructure."

Chung had worked with Farner at Google, then went to Zynga. Sirois, another Google veteran, helped Farner develop Aurora at Twitter.

Conductant got off the ground just last year. Farner, Chung and Sirois were batting around ideas for commercializing Aurora through the startup, or developing related solutions based on the Aurora stack for scaling micro-services in a way that only computing giants like Google and Twitter could.

"To achieve this, we couldn't ignore the impact and adoption of a game-changing technology that is Docker," Farner wrote in a blog. "As a platform, Docker has solved many of the problems we had to overcome when building Aurora on Mesos. As a tool for developers, Docker has made infrastructure approachable."

The Conductant team set out to develop an orchestration engine that could scale Docker containers. The vision aligned with what Docker, the commercial entity behind the container project, was doing internally, according to Farner.

"As they explored and started talking to partners, they realized Docker had already been down the path" with Docker Datacenter and Docker Cloud, a product stemming from its Tutum acquisition, said David Messina, Docker's senior vice president of marketing.

Docker is still exploring options for how best to leverage the capabilities of the Conductant engineers, Messina said.

One is just to integrate Aurora into its own orchestration engine, Docker Swarm. Another approach would be to take ideas from the Conductant team and develop those anew as features in Swarm.

The overarching goal is to build something that, both organizationally and operationally, scales, Messina said.

"What we're finding in the Docker ecosystem is a fair amount of noise about how orchestration happens," Messina said. Enterprises adopting Docker in production want to avoid the need to staff operations teams with dozens of cloud-skilled engineers supporting the developers.

And, thanks partly to the work done by some of its new engineers, "Twitter is really the tip of the spear in terms of scaling containers. Hard to find any other place doing that at that scale," Messina said.

Aater Suleman, CEO of Flux7, a systems integrator based in Austin, Texas, that was a founding Docker partner, told CRN that Docker and Conductant together are positioned to solve challenges in developing micro-services through the use of Linux containers.

The union "can lead to even higher productivity development workflows that will enable teams to be more productive in both dimensions: faster cycle time and better scale," Suleman told CRN.

Flux7 recently demonstrated the power of joining micro-services and containers in an engagement with a Fortune 1,000 financial customer for which it built a Docker-based infrastructure.

The Conductant team has the credentials to help Docker partners succeed at similar projects, Suleman said, so as a Docker partner, "we are excited see this effort and are waiting to see what will come out of this synergy."