Rubrik CEO: Gartner Data Center Magic Quadrant ‘Seriously Flawed’

Rubrik and another vendor cite issues in how they say Gartner's 2019 Gartner Magic Quadrant for Data Center Backup and Recovery Solutions was prepared. Gartner says it found ‘nothing improper.’

The chief executive of cloud data management company Rubrik is taking issue with the latest version of the Gartner Magic Quadrant for Data Center Backup and Recovery Solutions, citing inaccuracies and potential bias that seems to have stemmed in part from changes in the Gartner analyst team.

Bipul Sinha, co-founder and CEO of Palo Alto, Calif.,-based Rubrik, wrote in a Thursday blog post that Gartner's 2019 Gartner Magic Quadrant for Data Center Backup and Recovery Solutions is "seriously flawed."

Sinha wasn’t alone in his criticism. Oussama El-Hilali, chief technology officer for Eden Prairie, Minn.-based Arcserve, told CRN that, despite a heavy investment in following Gartner guidelines for growing its business, there was no positive movement in Arcserve's positioning on the Magic Quadrant.

[Related: 19 Hot Storage Products That Bring Power And Performance To The Cloud]

Stamford, Conn.-based analyst firm Gartner publishes its Magic Quadrant for a variety of tech industries. In the Magic Quadrant, or MQ, Gartner reviews how competitors are faring against each other, how they are executing on their stated visions, and how well they are performing against Gartner's market views.

Gartner says the vendors are placed in one of four quadrants: "Leaders" who execute well against their visions and are well positioned for the future, "Visionaries" who have a good vision of where the market is going but have not yet executed on that vision well, "Niche Players" who focus on a small market segment or who do not have outstanding innovation or performance, and "Challengers" who dominate a market challenge but may not demonstrate it knows the market's directions.

For its part, Gartner referred CRN to a new blog post posted Friday from its ombudsman and written by Nancy Erskine, Gartner's group vice president of ethics and integrity.

Erskine wrote that the Gartner Magic for Data Center Backup and Recovery Solutions was significantly delayed following a complaint from Rubrik that claimed its evaluation was inaccurate and that one of the report's authors unsuccessfully applied for a job at Rubrik a year before that analyst was assigned to the Magic Quadrant.

Gartner investigated the preparation of the report and its underlying analysis, and brought in a senior analyst from another team to do a deep review.

"He found nothing improper in the data, research process, analysis, the analyst’s interactions with Rubrik, the advice provided by this analyst to more than 100 end-user clients regarding this vendor, or the conclusions presented in the draft Magic Quadrant. … We believe Gartner’s opinion on vendor capabilities in this market is accurately expressed in the Magic Quadrant, a rigorous, independent analysis that helps buyers navigate technology purchase decisions," Erskine wrote.

The 2019 Gartner Magic Quadrant for Data Center Backup and Recovery Solutions has yet to be made public. The last version, published July 31, 2017 and updated January 1, 2019, was written by Gartner analysts Dave Russell, Pushan Rinnen, and Robert Rhame.

Russell is currently vice president of enterprise strategy at Veeam Software, but at one point had accepted an offer to work at Rubrik. Rinnen is currently director of product strategy for Rubrik. Rhame is director of EMEA product marketing at Rubrik. At least one other former Gartner analyst, Ray Schafer, is director of competitive intelligence at Rubrik.

In his blog post, Sinha noted that the 2018 Magic Quadrant for Data Center Backup and Recovery Solutions was not published, and that Rubrik worked for several months to remedy a number of what he termed significant issues and concerns.

"So we felt it was important that the market, including our customers, potential customers, partners and employees, have the full set of facts that are pertinent in objectively evaluating the information contained in this MQ," he wrote.

Between February 2017 and February 2019, Rubrik saw dramatic progress with a 500-percent growth in bookings and employee count, making its business comparable to that of other leading vendors like Commvault and Veeam, and during that time added "tech titans" including former Microsoft Chairman John Thompson, former Symantec CEO Enrique Salem, and Cisco Chairman Emeritus John Chambers as board members and advisors, Sinha said.

Despite Rubrik's efforts to work with Gartner analysts, Gartner made several errors in its analysis, and even confused Rubrik with a smaller competitor in one case, Sinha wrote. Gartner at one point felt Rubrik might be too young to be in the Leader quadrant, but Sinha wrote the vendor gave examples of multiple companies across several Magic Quadrants that were younger and smaller than Rubrik and yet made the Leader quadrant.

Sinha also wrote that the Gartner analyst who did the backup and recovery Magic Quadrant report may have had a bias against the company, noting that five Gartner analysts in the backup and recovery part of its business separately left the analyst firm and approached Rubrik for employment. Rubrik made offers to four of those analysts.

"What happened to Analyst #5? After his sustained pursuit of a position with Rubrik, we declined to offer Analyst #5 a role, and he expressed his clear disappointment in light of his colleagues’ hirings at Rubrik. Approximately four months after communicating this displeasure, Analyst #5 would change teams and be named as a lead analyst on the MQ covering Rubrik. We expected that this conflict of interest would have been disclosed, as Gartner’s policy requires, and it was not," he wrote.

El-Hilali told CRN that his company, Arcserve, had a similar experience to that of Rubrik.

"We thought it would be good for partners, customers, and the tech community at large to be aware this didn't happen only to Rubrik," he said.

Arcserve worked closely with Gartner on developing a plan for growth, at one point directing about 30 percent of its research and development resources to follow the analyst firm's guidelines to reflect Arcserve's efforts towards growth, and saw positive results from the changes it implemented, El-Hilali said.

"When we wanted to correlate the changes to the MQ process, we had multiple meetings with Gartner to discuss the salient points," he said. "But we found the process wasn't followed. They told us we were one of a couple companies who showed double-digit growth, but we didn’t see changes in the MQ. The results put into question their entire process."

Arcserve identified about 40 inaccuracies in Gartner's analysis of the company, including errors in the descriptions of some of Arcserve's functionalities and errors in the growth numbers of its partner ecosystem, El-Hilali said.

Just as important, Arcserve displayed all its roadmaps and projections to the Gartner analysts involved in the Backup and Recovery Solutions Magic Quadrant just before many of them departed the analyst firm, El-Hilali said.

"We didn't see any protections for that data," he said. "There were NDAs (non-disclosure agreements) in place. But if an analyst under NDA goes to one of our competitors, I would think it would be hard to imagine they didn't bring some level of comparison to their new employers."

Several solution providers told CRN that clients look to Gartner's Magic Quadrants for guidance, but typically do not depend on them as a major part of their decision-making process.

The largest enterprises can afford to have their own IT proving grounds and do their own proofs of concept and testing, but not all of them have that luxury, said Glenn Dekhayser, field chief technology officer at Red8, a Costa Mesa, Calif.-based solution provider and Rubrik partner.

"Those companies have limited staff that are more devoted to production, and depend more on the Gartner Magic Quadrant to cut the number of vendors that they focus on," Dekhayser told CRN. "The 'Leader' Quadrant is a great shortcut for many people."

The Magic Quadrants do not focus so much on the promise of a vendor but instead more on where a vendor is at a point in time, Dekhayser said.

"Gartner's not including Rubrik in the top right [Leader] quadrant does not mean Rubrik has an issue," he said. "Gartner clearly points out that companies at the farthest top-right are not necessarily the best companies. Customers have different needs. If you have a 100-percent VMware environment, you'd be a fool not to include Rubrik as part of your search criteria. Gartner is allowed to say what they want, and Rubrik is allowed to not agree."

Scott Miller, senior director of strategic partnerships at World Wide Technology, a St. Louis, Mo.-based solution provider, said that his company subscribes to Gartner's reports, and that customers pay huge attention to what Gartner says.

However, Miller told CRN, that doesn't necessarily mean that the Magic Quadrant reports are totally objective given that Gartner analysts do not necessarily actually test the products from the vendors featured on the reports.

"It's my understanding that the people Gartner hires to build the Magic Quadrants do interviews of customers and gather other information and use it to build the results, and hence it may be more subjective than objective," he said. "When we talk to customers, we go to the actual data based on our hands-on testing in the ATC (WWT's Advanced Technology Center)."

Miller said that, in his experience, the Magic Quadrants are published roughly once a year at most.

"But the pace of this industry is way faster than that," he said. "I don't think Gartner analysts have the bandwidth to keep up with the pace. After 30 or 45 days, product information is already dated. And so a Magic Quadrant may be based on a version of technology introduced a year earlier."

A lot of clients attend the Gartner conferences, and the bigger customers will look at Magic Quadrants, said Chris Case, president of Sequel Data Systems, an Austin, Texas-based solution provider.

"But I can count on one hand in 20 years where a customer made a decision based on the Magic Quadrant," Case told CRN. "Smaller and midsized customers will look at it, but I'm not sure how much weight they give to it."

The Gartner Magic Quadrants are good places for information when customers are looking for a starting point of reference for some technology like backup and recovery, Case said.

"But today, people want to see demos of the software, and try the software before making purchases," he said. "And vendors usually do try-and-buys. And vendors in the 'Leader' quadrant like to point out their position to customers. The real value of Gartner is as a place where customers can bounce ideas off their analysts for major solutions like how to move to the cloud or how to implement ERP."