NetApp’s Data Mechanics Buy Latest Move To Optimize Hybrid Cloud

‘Ocean takes care of the infrastructure, while Data Mechanics takes care of the application. It looks at every job running in a Spark cluster, collects information as they are running, and then for the next run automates those resources. ... Data Mechanics really understands what jobs need in terms of compute and storage and makes those resources available,’ says Amiram Shachar, vice president and general manager for Spot by NetApp.

NetApp on Tuesday expanded its cloud capabilities with the acquisition of cloud analytics and big data technology developer Data Mechanics to better optimize the growing use of Apache Spark analytics in conjunction with NetApp’s Ocean Kubernetes DevOps technology and its Spot technology for optimizing cloud compute.

NetApp also expanded its Ocean technology with new continuous delivery as a way to automate the process of delivering applications from development to production.

With Data Mechanic, NetApp continues to expand beyond its roots as a storage vendor to embrace new technologies to enhance workload optimization on the cloud, said Amiram Shachar, vice president and general manager for Spot by NetApp.

[Related: NetApp Leans Hard Into Hybrid Cloud With New Ontap, FlexPod as a Service]

NetApp 12 months ago acquired Spot, a developer of technology to manage and optimize compute on public clouds, as a way to target cloud-native business beyond storage, Shachar said.

“We’re now focused on continuous optimization in the cloud,” Shachar told CRN. “We now operating across multiple areas with Spot. There’s a lot of interest in containers and Kubernetes to modernize applications and launch new services.”

NetApp two-and-a-half years ago also launched Ocean, an automation and optimization technology that lets customers run containers without the need to manage their infrastructure, Shachar said.

“Ocean is a DevOps suite for Kubernetes in the cloud that starts with automation to help customers run containers without virtual machines,” he said. “Whatever is needed from infrastructure-as-a-service, Ocean provides. It’s completely server-less.”

On top of that, Ocean adds optimization of storage and compute resources, with algorithms in to place workloads in the right infrastructure in an optimized fashion, Shachar said.

“It makes sure Kubernetes runs in the most efficient location,” he said. “For example, think of two applications that talk with each other a lot. Ocean sees this, and moves the apps to be close together to cut communication costs.”

Data Mechanics sits on top of that to make sure Ocean and Spot optimize big data and data analysis for Apache Spark workloads, Shachar said.

“Ocean takes care of the infrastructure, while Data Mechanics takes care of the application,” he said. “It looks at every job running in a Spark cluster, collects information as they are running, and then for the next run automates those resources. So a job that may take four hours normally might run in under 40 minutes next time, or a 60-minute job may run in 10 minutes. Data Mechanics really understands what jobs need in terms of compute and storage and makes those resources available.”

Data Mechanics is tailor-made for Apache Spark, which has become the second-most-common workloads in Kubernetes, Shachar said. The most common workload is web services from a variety of vendors. “But Spark is the biggest platform running on Kubernetes,” he said.

Future versions may be expanded to other big data platforms including Kafka and Elasticsearch, but it is still early to say for sure, Shachar said.

Data Mechanics is the ideal acquisition for NetApp to increase the reach of some of its other technology including Spot, said John Woodall, vice president of engineering west at General Datatech, a Dallas-based solution provider and long-term NetApp channel partner.

NetApp recently introduced Spot Wave, which is focused on automating the compute layer for cloud applications, Woodall told CRN. Wave is Apache Spark-aware to automatically size resources and deploy the application, he said.

“Data Mechanics logically sits on top of Spot to do automation for Spark and dynamic allocation of resources,” he said. “It optimizes the provisioning of resource utilization and cost over time of applications running on Spark.”

NetApp is clearly focused on hybrid clouds and on optimizing data services and cloud resources for Kubernetes, Woodall said.

“But NetApp still does storage in the cloud,” he said. “So on the compute and app side, everything can now sit on NetApp storage across all three major hyperscalers. Big data workloads definitely need optimization of resources to make them easy to provision. And that seems to be the goal.”

With NetApp, customers get Spot Wave for infrastructure optimization including compute resources and optimization, and Data Mechanics adds automated operations for applications, particularly Spark, Woodall said.

“So now a workload--Spark--can be optimized over multiple dimensions,” he said.

Data Mechanics is something that NetApp can use immediately to add value to customer engagements, Shachar said.

“We already have customers on-boarding both Ocean and Data Mechanics,” he said.

As for NetApp Ocean, NetApp Tuesday introduced Ocean CD, or continuous delivery.

Ocean was developed originally to provide continuous operations, but is now shifting to continuous delivery, Shachar said.

“Customers now write code, review it, and send to continuous integration for testing, and then ships it to the servers,” he said. “This is all ‘automated,’ but is still very much a manual process. The customer deploys the application, then looks at metrics and logs and checks with users, and then deploys further versions.”

Ocean CD changes that, Shachar said.

“Ocean CD automated these manual interventions,” he said. “It’s very focused on Kubernetes. Ocean CD automatically detects when a new app is deployed to Kubernetes, and watches all the events. Customers start with a pre-deployed strategy. For instance, apps may be deployed one after another in two-minute intervals, or deploy half now and half later. This is a big, big part of the modernization of applications.”