Senior Vice President, Cloud Network Strategy and Technology
Previously, Mike Capuano discussed the advantages of packet aware transport systems in terms of improving network efficiency, and in Infinera’s press release from March 23, we launched the concept of Layer T, a combined packet optical transport layer. Today I would like to expand upon Layer T and how emerging technologies will enable an entirely new breed of Layer T platforms.
Data Center Interconnect (DCI) packet networks are bounded in service complexity and routing scale because their primary purpose is to provide secure reliable attachment points for Cloud hosted services that require the spanning of multiple data centers. This is in sharp contrast to the complexities of multi-service, multi-tenant and Internet scale routing required by carrier wide area networks. Ideally, the operators that provision a service should need to be concerned only with the service level agreements required by that service and not with the underlying complexity of transport layers – optical, digital, and packet. For more than a decade, network operators have understood the economic reality that switching digital information at the optical layer is less costly on a per bit basis than switching it at the digital (circuit) layer, and both are much less costly than switching at the IP layer. Costs include capital, power, and operational complexity.
The underlying transport layer, which we call Layer T, should provide a completely flexible and programmable mix of optical and packet transport such that any given data path is optimized with respect to what layers of switching it consumes at which points along its path. Furthermore, Layer T should trend in line over time with respect to price-performance as predicted by Moore’s Law. Until recently, this has been a vision – or a fantasy, depending upon your level of skepticism – but we believe that the tools are now emerging to enable the construction of a Layer T platform: photonic integrated circuits (PICs), merchant silicon packet switching, and software defined networking (SDN). These tools include:
- Optics that scale like integrated circuits (ICs) – Infinera has been and continues to be a leader in designing and fabricating large scale PICs (the current generation integrates over 600 optical functions operating at 500 Gb/s DWDM (dense wave division multiplexed) super-channel)
- Packet switching that trends similarly to consumer volume ICs – the industry now supports several IC vendors delivering commercial off the shelf (COTS) packet switching chipsets that are approaching Tb/s switching capacities.
- SDN – the ability to separate service activation, service assurance, and policy management from the data path platform onto COTS servers not only isolates the software development cycle from the hardware development cycle, but enables greater disaggregation for greater flexibility in architectures and more options for vendors.
The disruption that such a platform will bring is not simply due to the Moore’s Law cost-performance curve it can ride and the complete elimination of high bandwidth optics needed between separate packet and optical platforms; it is the ability to truly achieve multi-layer optimization. In principle, every service or every flow could be handled dynamically at the lowest possible layer based upon the Cloud operator’s policies. When this multi-layer transport optimization is achieved, we are no longer bound by the traditional seven-layer networking model; we enter a much simpler architectural framework that consists of only two layers – the transport layer and the Cloud services layer, which is manifested in the Cloud as applications running on servers or network functions virtualized (NFV) on servers. This dramatic simplification is driven by and enabled by data center requirements on scale and virtualization.
We believe that the combination of PICs, merchant packet switching silicon and SDN will revolutionize the data center interconnect market through a disruptive cost structure, improved network efficiency, full automation across the packet optical transport layer, and enhanced service velocity. Over time we believe this approach will likely find its way into the metro and backbone networks as well.
Senior Product Manager Capacity & IP at BICS
Geoff Bennett: Hi Eric, thanks for letting me interview you today. We saw quite a bit of interest in last month’s trial of Large Area/Low Loss fiber. So let me begin by asking you why is it important for an international service provider like BICS to get involved in this sort of trial?
Eric Loos: At the end of the day all of our traffic is carried over optical fiber, and there are many different kinds of fiber in the ground today. In Europe especially there’s a lot of unlit fiber, and one of the business planning decisions I have to make at BICS is whether it makes more sense to use up this existing fiber, or to think about deploying newer, more efficient fiber types like the OFS Terawave™ fiber we tested here.
Geoff Bennett: How would you make a decision like that? What are the most important criteria?
Eric Loos: It’s really all about the money. If it’s more cost effective to deploy new fiber then that’s what we’ll do. But we have to have solid data to make the decision. It’s one thing to read a fiber data sheet, or run a simulation, but lab trials like this give us a far higher level of confidence in the data that we can plug into our planning model.
It’s the only reliable way we can figure out if it’s better to sweat existing fiber assets, or to deploy new types of fiber like the one we tested here.
And for sure these new fibers are a key step in the deployment of higher order modulation like 16QAM.
Geoff Bennett: Why is 16QAM so important?
Eric Loos: It’s not that 16QAM or 8QAM or any other particular modulation is important. It’s all about extracting the maximum economic capacity on a route. Higher order modulation is a mechanism to help us do that, but you don’t get something for nothing.
For example, we read Powerpoint slides and conference presentations about 16QAM, and the promise is a doubling of fiber capacity. But over existing fiber types that comes at the price of about one-sixth of the reach compared to QPSK – which is the most popular modulation used for 100G transmission today.
Is that a good economic trade-off? It would be if 16QAM still had enough reach to close our routes in an economic way, but the fact is that on existing fibers that isn’t the case. It’s a particular problem with G.655 or LEAF fiber because these have significant non-linear penalties with 16QAM.
We also need to look at the age of the fiber. Did you know that the last widespread deployment of fiber in Europe was during the dot com boom? So in addition to losses that fiber may incur because of repairs and splices, there’s a well-known mechanism of hydrogen aging in optical fiber. This is one reason why service providers will insist on a safety margin of about 2dB on any deployment. Usually this isn’t included in headline reach numbers for modulation trials of high order modulation like 16QAM.
Geoff Bennett: What kind of real-world reach would you expect from 16QAM on existing fiber?
Sr. Director, Field and Segment Marketing
Recently, it was announced that Facebook deployed the world’s longest terrestrial optical route capable of lighting up to eight terabits per second (Tb/s) of traffic over a 4000 kilometer link without any regeneration. This route starts in Lulea, Sweden, where massive data centers can be built north of the Arctic circle for very cost efficient cooling. However, all this traffic needs to find its way to the end user, so the route stretches to a major hub without any optical-electrical-optical (OEO) regeneration. 8 Tb/s is 8 trillion bits being transmitted over the fiber every second, which helps transport the increasing amount of videos and other rich media across Europe and ultimately across the world.
Facebook is just one of many Internet content providers and Cloud operators that have fallen in love with Infinera’s DTN-X, and people are always asking me why. Specifically, many are asking me, “Didn’t you just launch your Cloud Xpress product? Why are they deploying DTN-X?” What is important to remember is that Infinera is now a multi-market company and we are building products targeted at specific applications. The DTN-X is an industry-leading packet optical transport networking platform focused on subsea, long haul and regional applications. This platform leverages powerful coherent optical transmission technologies to carry signals very long distances. In this case the deployment was a long haul deployment that Infinera believes to be the longest 8 Tb/s capable route, with multiple terabits already deployed and spanning approximately 4,000 kilometers with no regeneration.
The Cloud Xpress platform is targeted at a second market, what we call metro Cloud and some others refer to as metro data center interconnect. This is a separate and large market in its own right where data centers are connected across a metro or even across a campus with a 2RU rack and stack platform designed to be quickly deployed and easily turned up.
Let me spend a few minutes on some background info and more detail:
Director, Corporate Marketing
OFC 2015 is in full swing. At our booth, Infinera is demonstrating an SDN-enabled packet-optical solution and showcasing packet Ethernet VLAN-based transport services running over a sliceable super-channel being controlled by an SDN OpenDayLight (ODL) Controller.
Because our customers are looking to deploy a single network with multi-layer functions working in harmony, we are demonstrating these capabilities as a single solution and not individually. The technologies involved in this effort include:
Infinera Technology Evangelist
Coherent transmission opens up a fascinating toolbox of modulation options, as we’ve seen with Infinera’s pioneering FlexCoherent™ modulation technology.
For those of you who are not aware of the issue, here’s a quick summary.
Phase modulation (part of modern coherent transmission) allows us to encode varying numbers of bits per symbol in order to increase spectral efficiency (and thereby optical fiber capacity).
The current “workhorse” modulation technology is Pol-muxed (PM) Quadrature Phase Shift Keying (QPSK), which carries four bits per modulation symbol.
In various trials, Infinera has demonstrated alternative modulations that have longer reach but lower capacity than QPSK (i.e. BPSK, Enhanced BPSK and 3QAM), as well as modulations that have greater spectral efficiency, but shorter reach than QPSK (i.e. 8QAM and 16QAM).
The key to FlexCoherent modulation is to present all modulations in a software selectable format on a single line card so that the service provider can choose the optimum balance between optical reach and fiber capacity, and historically QPSK has been the most commonly used modulation precisely because it offers a great reach/capacity balance.
Looking at one of these newer modulations in particular, in two recent trials we’ve seen how PM-8QAM not only offers a 50% increase in fiber capacity versus QPSK, but also represents a “sweet spot” in terms of optical reach on both existing optical fiber types, and new types of large area/low loss fiber, such as OFS Terawave™.
In the first trial with Telstra, we achieved a reach of 2,200 kilometers over an existing submarine fiber.
In trial announced today, we achieved an astonishing increase in reach over OFS Terawave fiber – 7,400 kilometers, in fact – which would be enough to close Atlantic submarine routes! Just to be clear the specific Terawave fiber we used in this test is a type that’s optimized for terrestrial transmission, and we may have been able to do even better if we’d used the submarine-optimized Terawave SLA+ or ULA Ocean Fibers.
These numbers are important because optical reach has a direct impact on the total cost of ownership for a DWDM system. Basically, the longer the reach, the lower the cost. But since fiber capacity also affects cost of ownership, we can imagine that there are reach values that are “just enough” to close a given set of routes without resorting to regeneration, and yet by using one of the higher order modulation formats we could achieve much greater fiber capacity.
In other words, we can imagine a set of sweet spots emerging because a given proportion of city pairs are a given distance apart.
These sweet spots are reasonably well understood. We know that analysts generally break down markets on the basis of reach – so that the ultra-long haul market would be distances of greater than 3,000 kilometers; long haul is distances greater than 1,800 kilometers; and at distances of less than 1,800 kilometers we have regional and metro networks.
So it’s clear that, with reach numbers of between 400 and 1,200 kilometers on conventional fiber depending on the specific type of fiber and amplification used, 16QAM is not a viable long haul modulation format. But 8QAM, with reach of over 2,000 kilometers on conventional fiber, certainly could be.
When we include optical protection techniques, the picture becomes even more clear. Figure 1 shows two common topologies: a ring and a mesh. Generally speaking, metro networks still tend to be laid out in rings. Regional and long haul networks are more likely to be logical meshes, but historically the fiber may have been deployed in a physical ring.
In the ring example, we seen Nodes A and B are connected by a relatively short working path, shown by the solid green line. Let’s say that in this example, X = 500 kilometers – well within the reach of 16QAM modulation. But if there is a fiber cut between A and B, the path length increases dramatically to three times that, or 1,500 kilometers. This is currently beyond the vendor claims we’ve seen for 16QAM modulation, even using Raman amplification.
In the mesh example, the differences are generally less extreme, but the absolute distances are more likely to be longer because we’re now looking at a regional or long haul deployment.
So you could imagine that Y = 1,000 kilometers; which would be possible to close with a very high performance 16QAM solution, probably including Raman amplifiers. But if the A-B link breaks and there is purely optical protection, then the link length increases to 2,000 kilometers, and once again 16QAM would not be able to cope.
But in both cases 8QAM could close the link, and so we can immediately appreciate there could be a reach sweet spot for this modulation technology.
By the way, if you’re wondering why I’m emphasizing optical protection, it’s because the companies who are heavily promoting 16QAM technology are also companies who promote optical protection.
Let’s move on to implementation, because 8QAM is not an “easy” modulation technology for vendors who are using discrete optical components.
Table 1 shows the data rates for the “new breed” of line cards that some vendors are now announcing. Each line card can run in several different modes – which are usually a simplified form of Infinera’s FlexCoherent capability.
In BPSK mode the card runs at 50 Gb/s, in QPSK mode it runs at 100 Gb/s, in 8QAM mode it runs at 150 Gb/s, and in 16QAM mode (which is the mode that is normally heavily marketed) it runs at 200 Gb/s.
I’ve already mentioned that 16QAM probably has too short a reach to be generally useful, and it’s not clear how many of these “200 Gb/s” line cards are being used in that mode today.
Oddly none of the vendors who are promoting 200 Gb/s operation seem to be talking about 8QAM (apart from hero experiments), and this may be because it delivers an odd sort of data rate of 150 Gb/s for this type of discrete component card design.
We know already that some DWDM systems struggle to provide a non-blocking switching capability across the backplane between line cards at even 100 Gb/s, and that means it could be difficult to amalgamate these separate 8QAM signals into a useable data rate – typically a multiple of 100 Gb/s.
Last August I took part in a field trial on the GÉANT network with a team of colleagues, and we showcased a prototype terabit PIC technology. We showed BPSK, 3QAM, QPSK, 8QAM and 16QAM technology over this link, using the same line card with FlexCoherent modulation.
But an important aspect of Infinera’s implementation, shown in Figure 3, is that the production version of this line card will deliver the full 1.2 Tb/s capacity for all of the “terrestrial” modulation types – which means QPSK, 8QAM and 16QAM. That means that instead of having to implement anywhere between six and 12 line cards to achieve 1.2 Tb/s, an Infinera customer can simply plug in one card.
Moreover the DTN-X platform that uses this line card will have a full 12 Tb/s non-blocking OTN switching capability to turn all of the link bandwidth into a virtual pool of digital capacity. Any service – including 1GbE, 10GbE, 40GbE, 100GbE and future 400GbE can be easily and efficiently supported using this OTN switching capability – including 8QAM modulation. Thus it is not only important to have the line card with the right modulation options; the system must be designed with forward scale to accommodate those line cards when they become available. Below is a video showing how any DTN-X ever deployed can be upgraded from 5 Tb/s to 12 Tb/s in service to accommodate 1.2 Tb/s line cards when they become generally available.
So we firmly believe that 8QAM will offer a “reach sweet spot” for high order modulation, but we also believe it’s essential to choose the right implementation to get the best out of this technology.
Terawave is a trademark of OFS Fitel, LLC.
Director, Corporate Marketing
There’s a crater on the near side of the moon called Alhazen that is connected to the industry I work in. It is named after the Arab scholar Ibn al-Haytham, who was a pioneer in the field of optics way back in the 11th century. He authored the classic tome “Kitab al-Manazir,” or the Book of Optics. His work transformed the way in which light was understood, earning him the title of “the father of modern optics.”
As I dug deeper into Ibn al-Haytham’s work I realized how the same physical phenomenon can have two very different interpretations (for an example, read about the emission theory and the counteractive intromission theory on vision and light rays). I was fascinated by how much optics has evolved since then and its application today across all facets of life, including the Internet. Simply put, life as we know it would not exist without the practical application of optics technology (aka photonics).
Vice President, Network Strategy
Today, Pacnet announced the deployment of Infinera’s Open Transport Switch (OTS) product to enable a new SDN service that extends network virtualization to the optical transport layer. As a pioneer in the realm of transport SDN, this is truly an exciting milestone for Infinera, and a remarkable demonstration of the benefits that DevOps processes combined with industry cross-pollination can yield. By leveraging Infinera’s Web 2.0-empowered OTS and engaging jointly in the development and refinement of the service concept, Pacnet and Infinera were able to achieve specification, development, and deployment of a new service capability to the market in a matter of months, much more quickly than what is typically achievable using conventional processes and technologies. With this new service capability, Pacnet’s customers will now be able to request on-demand or scheduled high-speed bandwidth services between key Pacnet data center locations, from 10Gb up to 100Gb (and beyond), and pay only for the duration the service is requested, all without manual intervention.
What makes this feasible is a unique mix of technologies that includes both software and hardware, each contributing a key piece of the overall solution.