Cisco Partners Paying Price For Massive Product Replacement Effort That Will Cost 'Millions' To Fix

Some partners are already paying the price for a costly component malfunction that causes some of Cisco Systems' mission-critical enterprise products to fail after 18 months.

The "Clock Signal Component Issue" impacts some of Cisco's popular product lines including ASA firewalls, Nexus and Meraki cloud-based managed switches, and ISR routers. Cisco has cautioned that once the "component has failed, the system will stop functioning, will not boot and is not recoverable."

Partners say they are being forced to shoulder the cost for what amounts to a massive product replacement effort.

[Related: 10 Networking Predictions For 2017]

"This is an issue that could cost the channel community tens of millions of dollars," said one of Cisco's top national enterprise partners, who did not want to be identified. "It is a shock to the system. The clock is literally ticking on these devices... We have over 500 customers and more than 10,000 devices impacted by this. Cisco needs to step up and take responsibility to fix this from an equipment, installation and cost perspective."

The executive likened Cisco's stance on the component malfunction to a major automaker replacing a malfunctioning part but refusing to pay auto dealers to do the work. "This just doesn't make sense," he said. "Cisco gets it but they are just not willing to put money into a pool to make sure that channel partners can help customers."

Cisco issued a notice on Feb. 2 explaining it discovered in November that a clock signal component inside certain devices degraded over time.

"Although the Cisco products with these components are currently performing normally, we expect product failures to increase over the years beginning after the unit has been in operation for approximately 18 months," said Cisco in the notice posted on its website.

The San Jose, Calif.-based networking giant said the issue stems from a component manufactured by one supplier, which it did not identify.

Although Cisco said it will provide a replacement for the products that are under warranty or covered by any valid services contracted dated as of Nov. 16, 2016, it will not provide any reimbursement funding to channel partners who are providing replacement and installation services of the faulty products.

"Unfortunately, because our funding is focused on providing the products, we're unable to reimburse for on-site services to replace the affected devices," said Jennifer Ho, manager of Cisco's Business Critical Communications, in an interview with CRN. "Because the product doesn't actually have any issues for 18 months, we really wanted to focus our funding for the issue on a replacement program. So we're proactively offering the replacement of products even before the product fails, and we really believe that this focus on providing the best-quality products is the right focus for our Cisco customers and the partners' customers.

"Customers may have field engineering service as an option for their services contract for products which already have failed, in which case the field engineering support would be included with the replacement," she said.

Many partners who offer their own services on top of Cisco products are already spending their own money to fix the issue for customers.

"We got a ton of people spending a lot of time right now on it – scrubbing through customer lists of who had what product and service contract to determine who we have under our own branded support contract, which is our top priority," said one executive from a solution provider who is a Cisco Gold partner, who did not wish to be identified. "We also have to dig deeper for general support customers and go through our purchasing systems to go over multiple years of what customers bought from us."

The executive said his company is already shouldering the financial burden of sending out engineers and systems integrators to customers who own support or managed services contracts in order to replace the faulty products.

"We're a support services partner with Cisco and we have our own branded offerings … anybody under our service programs, yes, we are shouldering the cost," he said, adding that more than 15 percent of its customers are affected by the clock signal malfunction. "It's going to take some time to coordinate those sites and get out there and preform the replacements – it's going to be many months, I suspect."

Another top executive from a solution provider ranked on CRN's 2016 Solution Provider 500 list said sending out employees to replace all the faulty products is a "drain on internal resources."

"Imagine you have customers with hundreds of ASAs on their network that is their main termination point possibly for their WAN services or security device -- obviously it's really critical that they replace all of those before they become disabled, so we're working hard on it," said the SP 500 executive, who did not wish to be named. "Cisco isn't offering anything right now to offset [our] expenses."

The products affected by issues include certain models of its Nexus 9000 switches, Series 4000 Integrated Services Routers (ISRs), Cisco Adaptive Security Appliances (ASAs) to Meraki switches and its Network Convergence System (NCS) 5500 series.

Cisco said there is no risk of fire or other hazards on these devices. The only symptoms are that once the clock signal component fails, the system will stop functioning and is not recoverable.

"It's a pretty severe impact," said the Cisco Gold partner. "I haven't seen any of our customers have those at this point in time. It's some newer product SKUs. … Obviously everybody is concerned that their product might stop operating."

Partners also said there are concerns about the availability of the replacement products due to the high demand.

Due to the time-based nature of the failure and the volume of replacements, Cisco said it is prioritizing orders based on the product's time in operation.

"We are prioritizing based on the time in operation. So really looking at trying to replace those products that are hitting that 18-month period or longer in operation," said Cisco's Ho.

Fulfillment requests may be delayed and may be fulfilled in multiple shipments, according to Ho.

"If [a partner] puts in an order for a bunch of shipments, we may be able to replace some now and then some a little bit later. We'll try to replace what we can as quickly as we can, then fulfill the rest of the shipments potentially at a later date," said Ho. "So the shipments may be split up."

Partners expressed their long-term concerns if Cisco did not provide them with reimbursements.

"If we have to shoulder this cost ourselves, it is going to cost us millions of dollars," said the top executive for one Cisco partner. "Everyone is going to be hit by this."