10 Things You Need To Know About Cisco's Clock Signal Failure And Product Replacement Priorities

The Clock Is Ticking

Cisco's clock signal component issue inside some of its most popular products lines has the company and its channel partners scrambling to fix problem for customers. The component malfunction inside Cisco's ASA firewalls, Nexus and Meraki cloud-based managed switches, and ISR routers causes the product to fail after 18 months. The San Jose, Calif.-based networking giant said that once the "component has failed, the system will stop functioning, will not boot and is not recoverable."

CRN takes a deep dive into where the faulty clock signal issue stems from, how Cisco will not reimburse channel partners for replacement services, the networking giant's priority plan, and the specific products involved.

Intel's Atom Chips

The culprit that caused the massive product replacement appears to be Intel's low-powered Atom C2000 chips. The Register reports that Intel indicated in January that the chips contained a clock flaw, according to documentation about the C2000 family. The document said the Low Pin Count (LPC) clock outputs could stop working causing systems with the faulty processors to no longer be able to boot up.

An Intel spokesperson said they don't comment about other companies, but did say the issue with the C2000 chips "has been identified, root caused and there are immediate fixes available."

"Intel has identified an erratum in a circuit element in the low pin count bus (LPC) in the Intel Atom Processor C2000 product family. The issue has been root caused to a circuit design issue that causes a degradation of the circuit element under high use conditions at a rate higher than Intel's quality goals after multiple years of service," said the Intel spokesperson. "Intel is working now with customers on a board level workaround for the existing products which resolves the issue. Additionally, Intel will implement a minor silicon fix in a new product stepping that resolves this issue."

A Cisco spokesperson declined to provide who the supplier is.

No Reimbursement Funding For Partner Replacement Services

Cisco partners told CRN that they are already paying the price for the costly component malfunction. Cisco said it would not reimburse partners who are sending out engineers and system integrators to replace and install the new products.

"Unfortunately, because our funding is focused on providing the products, we're unable to reimburse for on-site services to replace the affected devices," said Jennifer Ho, manager of Cisco's Business Critical Communications, in an interview with CRN. "Because the product doesn't actually have any issues for 18 months, we really wanted to focus our funding for the issue on a replacement program. So we're proactively offering the replacement of products even before the product fails, and we really believe that this focus on providing the best-quality products is the right focus for our Cisco customers and the partners' customers … Customers may have field engineering service as an option for their services contract for products which have already failed, in which case the field engineering support would be included with the replacement," she said.

Flawed Security Products

The specific security products containing the clock signal issue include ASA firewalls models 5506, 5506W, 5560H, 5508 and 5516. Additionally, Cisco's ISA3000 Industrial Security Appliance (ISA) and its Meraki MX84 cloud-managed security appliance have the faulty component.

Flawed Networking Devices

Specific networking products containing the faulty component are ISR routers models 4331, 4321 and 4351. Cisco's industrial integrated service routers models IR809 and IR829 also include the flaw.

Cisco's Nexus 9000 Series models C9504-FM, C9508-FM-E and the X9732C-EX are also flawed along with the Meraki MS350 Series. The company's NCS5500 line cards might also fail after 18 months as well as its optical networking solution: the NCS1K-CNTLR.

UCS-E Series

Cisco also had one model of its Unified Computing System (UCS) servers impacted: its UCS-E120. The UCS E-Series are x86 router-integrated blade servers for branch network services and application hosting.

Expect Product Delays

Due to the high demand for these specific products, Cisco said there cloud be delays in obtaining the devices. The company also said requests might be fulfilled in multiple shipments.

"If [a partner] puts in an order for a bunch of shipments, we may be able to replace some now and then some a little bit later. We'll try to replace what we can as quickly as we can, then fulfill the rest of the shipments potentially at a later date," said Ho. "So the shipments may be split up."

Older Production Deployments Get Priority

Cisco's top priority is replacing products that have been in production the longest, not based on the specific solution.

"There's not a prioritization based on the product. Specifically, we are prioritizing based on the time in operation. So really looking at trying to replace those products that are hitting that 18-month period or longer in operation," said Ho.

Cisco Knew About The Issue In November

Cisco issued a notice on Feb. 2 explaining it discovered in November that a clock signal component inside certain devices degraded over time. For the past few months, Cisco said it has been working with the supplier to identify and fix the issue.

"We had to work very closely with the supplier to identify the exact issue, work on creating a fix for the issue, and then testing that fix with our products. So there's a lot of work on that end," said Ho. "We also worked with our executives to create our remediation plan for our customers."

All Flawed Products Now Fixed

All of the affected products have been taken out of the rotation, and the devices currently being shipped are problem-free. "Anything shipping now does not have the faulty component," said Ho.

Cisco Not The Only Vendor With The Clock Signal Problem

Cisco said other companies also use the same component from "the supplier" inside certain products.

Other vendors who have implemented Intel's Atom C2000 chip line include Hewlett Packard Enterprise – including HPE Altoline switches – Aaeon, NEC, Netgear and Super Micro, to name a few.