This post originally appeared on the Packet Pushers’ Ignition site on January 9, 2020.
This slide from the Cisco Live BRKOPT-2006 presentation on “Preparing for 400 GbE” jumped out at me. I recommend you download the whole presentation and keep it for future reference. It’s an excellent resource with lots of useful information.
Optics costs for 400G networking will be 80% of the Bill of Materials (BoM) for projects. Most enterprise IT projects are unlikley to use 400G in the next ten years because 100G is sufficient.
Is this justifiable? What are the issues behind this?
400G Is Complex
Reaching 400G speeds is basically magic. It boggles my mind that Ethernet has reached 400G just a couple of years after 100G. The underlying signalling data rate of 50Gbps is a substantial technical achievement. Putting 8 x 50G transmitters and receivers into the QSFP is just amazing.
Sure, some heat sinks are jammed on the front…
…and it requires fancy MPO connecters to get all those optical cores into the QSFP.
Market Transition
I find it interesting that both Juniper and Cisco have acquired companies that design 400G components. At Juniper NXTWORK I saw a lot of discussion about Juniper’s optics business and how it was critical to meeting 400G networking for customers. Shortly after than, Cisco’s Silicon One event allocated a substantial amount to time to its optics business.
It’s clear that owning the optical interfaces is key to sustaining Cisco’s and Juniper’s revenues going forward. Customers should understand the impact: Vendors are going to be pushing their own optics more then ever and customers should push back as hard as possible.
ODM Vs. Third Party
Optics markups of 1000% – 2000% are common when buying directly from the vendor. I believe that there several angles to consider:
- Some customers will blindly buy recommended optics for convenience or assurance. It’s not smart and maybe it’s worth it.
- Vendor pricing activities to maximize profits are an issue. When the list price is some ridiculous number and 50-90% discounts are possible, the real prices of the products are hidden from the customer. Remember Cisco power cables cost $350 List because a 90% discount gives a real price of $35. Anyone with lesser discount is getting gouged.
- If you choose third-party optics, then you have some responsibility here. You can’t ring the vendor and expect them to troubleshoot the optical component. Yes, the vendor TAC will blame anything they can so they can close the case. This is a trivial problem in real life.
- Sales people will tell FUD stories about vendor support being withdrawn, but these are exaggerated or lies. Vendors do support third-party optics and will continue to do so.
Breakouts
I’m not sure we need 400G-100G breakouts though. Is it really necessary to add complexity to the 400G standard and to switches?
If the switches are cheap to buy (only 20% of the BoM) and easy to replace (hello SDN controller and ECMP network architecture) why do we need to have 400G->100G breakouts? Surely it’s more sensible to buy 100G switches and use the cheaper 100G optics until 400G is mature.
This makes me wonder if breakouts are a way for vendors to upsell and lock in customers more than they are an actual customer requirement.
The EtherealMind View
- Buy only what you need. Prices are expected to drop rapidly in the next few years as the cloud market will drive demand and volumes will increase.
- 400G requires new optical cabling. Plan for it.
- OM5 Multimode is required for future 400G short haul (up to 100M) but I wouldn’t recommend using MM. Stick with SM to have sufficient performance as the cable degrades.
- MPO connectors for optics, plus using 8 or 16 cores for a single connection.
- Don’t install more fixed fiber; use a modular cabling system instead.
- 400G would be useful for DCI over dark fiber, 400G ZR can reach 80-120km, 400G ZR+ up 1000km for metro.
References
Cisco Live BRKOPT-2006 San Diego 2019
<https://www.ciscolive.com/c/dam/r/ciscolive/us/docs/2019/pdf/BRKOPT-2006.pdf>
