Teleprotection over packet networks
Created by Dominique Verhulst
Edition 2
Legal notice
Copyright© 2013, 2014 Alcatel-Lucent. All rights reserved Alcatel, Lucent, Alcatel-Lucent and the Alcatel-Lucent logo are trademarks of Alcatel-Lucent. The information presented is subject to change without notice. Alcatel-Lucent assumes no responsibility for inaccuracies contained herein. All other trademarks are the property of their respective owners.
i
Editor’s notes Edition 1.2 This edition contains new information gathered since the first edition. More specifically section 3 of chapter 2 has been updated with new user data and illustrations. New illustrations were added to chapter 4. Edition 2.0 All animations have been reworked. More data was added from real current differential and distance protection deployments over IP/MPLS networks. Added references to standards. A new section was added in chapter 2, covering bandwidth and latency considerations to make when packetizing TDM data The technical validation testing examples and practical use cases have been grouped in a dedicated section in chapter 2. This allows more flexibility to add new examples in later editions. A new chapter was added to clarify why IP/MPLS is chosen over MPLS-TP A new chapter was added to explain the importance of synchronization
ii
Acknowledgements This book was made possible through the help of numerous colleagues, friends, customers and partners. At this point I would like to thank all the people who have helped create this content, provided inputs, feedback and inspired me to put this document together. In particular, I would like to extend my thanks to the following people. Dave Richards, Fai Lam, Benoît Léridon, Bram De Valck, Dave Sargent, Peter Merriman, Carl Rajsic, Colenso Van Wyk, Lieven Levrau, Matthew Bocci, Hannu Ahola, Danny Knezevic of AlcatelLucent Canada, France, Belgium, UK, South Africa, Finland, Australia Cory Struth, Clinton Struth of Falling Apple Solutions, Canada, Patrick Colling, Michael Wener of Creos, Luxembourg, Campbell Booth, Steven Blair, Federico Coffele of Strathclyde University, Scotland, Andrej Görbing, Siemens, Germany Fernando Castro, Joan Sans, Dimat-CG, Spain Sincerely, Dominique Verhulst iii
Foreword
Over the last few years Alcatel-Lucent has met many people from the power utility and telecommunications industry and have come to the conclusion that people who deal with some of the most mission critical applications in power utilities, such as teleprotection, have little understanding of the telecommunications infrastructure that assures the proper operation of their applications. It is considered that the telecommunications system is always up and has a guaranteed service performance.
Likewise, the people with a
telecommunications background in general lack a good understanding of the mission critical applications that power utilities need to operate in order to assure 24/7 electrical power to our homes and businesses. This book is meant to bridge this gap between the telecom and operational parts of the power utilities organizations. The hope is it can help the telecom people to appreciate the critical character of the teleprotection applications and design better, future proof telecom networks. Conversely, we also hope that this book can help instill confidence with the people who operate the teleprotection applications of the power utilities that IP/ MPLS can meet and exceed their requirements. It can also be useful for service providers to understand the strict constraints of the power utility applications in order to enable service providers to provide the appropriate service SLA’s that are required in the power utility market.
iv
The Need to Change Since the “War of Currents” was won by the AC camp in the late 19th century, electrical power generation and distribution systems haven’t changed much, other than that they have scaled massively. New central power generation technologies have been developed since then but the continuing search for sustainable energy has been driving the development of smaller scale distributed power generation. Power utility companies can no longer maintain the “status quo” of how they deploy and manage their power grid, there is an urgent need to change the way they run their business, how they control and protect the power grid. The grid has become a lot more dynamic and this has a direct consequence on the telecom network that is needed to control it. Power utilities will need to embrace this change and adopt new technologies to cope with the change.
Nicola Tesla is considered to be the “father” of the AC power system. His patented inventions of the AC induction motor and the transformer were licensed by Westinghouse. This put Tesla in the AC camp in the “War of Currents”
v
Chapter 1
Introduction Power Utility Assets - swipe to scroll through images
Power Utilities are managing one of the most critical assets of our times: Electricity. In order to ensure that the power grid is always on, power utilities put systems in place that continuously monitor the electrical infrastructure and provide information about the state of the power grid at key points in the network. These systems are designed to help protect the power grid against failures and avoid cascading of problems through the grid. One of the key systems that help power utilities protect their grid is called the Teleprotection system. This interactive book will provide you some insights into this application, how it works and how it works in a telecommunications network.
Power substation switchyard
Section 1
Introduction to teleprotection IN THIS SECTION 1. Why is teleprotection required?
Why teleprotection? Operating an electric grid requires the provision of safeguards. In particular, fault
2. What different systems are there?.
detection and subsequent corrective action are extremely important.
3. Pilot based protection systems
Teleprotection is an essential requirement for operating and maintaining a reliable,
4. Constraints of protection systems 5. Who are the players in teleprotection?
robust and safe electric grid. This functionality has always been required and its
Illustration 1.1 Power grid protection
Example of how power grids deal with failures - pinch open to enlarge, tap to start animation 7
importance is magnified when Smart Grid deployments include
Illustration 1.2 Standalone teleprotection system
increasingly diverse sources of electric power that are combined and channeled to increasingly diverse consumers of electric power. Teleprotection is a term that is often used to describe the systems which will detect faults in the power grid and will activate circuit
Circuit breaker High Voltage power line
breakers or reclosers to prevent faults from rippling through the network or to restore power to a part of the grid after an outage. Teleprotection systems can be classified into two main
Teleprotection Relay
groups: 1. Standalone protection systems 2. Pilot based protection systems
The first group of teleprotection systems are referred to as standalone teleprotection systems (also known as distance protections).
Tap on the labels for more details, pinch to open to full screen.
The second group of protection systems are referred to as pilot based protection systems. The difference between this type of
Standalone teleprotection systems are monitoring the high
system and the standalone teleprotection systems lies in the fact
voltage line impedance and in case of a failure (and a resulting
that pilot based systems use a communications channel to allow
change in impedance) determine where the fault is on the line and
the teleprotection relays on either side of the high voltage line to
make an autonomous local decision whether to trip the circuit
“talk” to each other and exchange information about what they
breaker or not.
see on the power grid.
The standalone protection system is shown in illustration 1.2
The illustration below shows how pilot based systems work. 8
Illustration 1.3 Pilot based teleprotection channel between adjacent nodes.Distance protection systems are measuring the high voltage line impedance in order (see illustration 1.4) to determine where the fault is, just like the standalone system, however since they work in pairs to monitor both sides of the line, pilot based Teleprotection systems can make better and faster decisions with regards to tripping their local circuit breaker. There is a possibility that the fault is further upstream or downstream in the grid. Therefore, information from an upstream or downstream neighbor’s Teleprotection relay can be essential in making the right decision to trip the circuit breaker as close as possible to the fault. Teleprotection systems send Pilot communications line
commands on an event basis. They can tell their neighbor to
Illustration 1.4 Principle of teleprotection
Within the pilot-based Teleprotection systems group, we can also make a distinction between two types of protection systems. Pilot Based Protection Systems: 1. Distance Protection or Teleprotection Systems 2. Current Differential Protection Systems
The pilot based Teleprotection systems are the ones that are of
Tele- or Distance Protections work on the principle that they measure the impedance of the high voltage line to determine the location of a fault.
interest for this book since they rely on a communications 9
block or to trip in case they see a fault.
These protection
Illustration 1.6 Distance protection schemes
schemes are generally referred to as blocking and permissive protection schemes. The animation in illustration 1.4 shows the principles of these protection schemes. Current Differential Protection systems work differently from the Tele- or Distance Protection systems. They use a different mechanism to determine if there is a fault condition on the grid. The principle of operation of Current Differential protection systems is based on a basic rule of electrical systems: Kirchoff’s point rule. This rule says that the sum of the currents flowing into a point is always identical to sum of the currents flowing out of the same point. If there is any difference, there is a fault.
Illustration 1.5 Kirchoff’s point rule
Current Differential relays continuously send sample values to each other to
This animation shows the principles of operation of blocking and permissive protection schemes.
compare the current
sample values and use a fixed offset to compensate for the
values that they see
communications latency. Hence the need for low latency,
(as in illustration 1.6).
accurate timing and very low variation in the latency.
The differential relays rely on a fixed latency time to transmit and receive their data because they need to compare real-time
Constraints of Teleprotection systems A failure such as a short circuit in a high voltage power system can cause major damage that can lead to an avalanche effect on the entire power system of a city, region or country. It is therefore
10
essential that failures are detected rapidly and that protection
figure) and fault resolution delay (4 in figure) take between one
systems can be engaged immediately after a failure is detected.
and three-to-five cycles, respectively. This leaves one cycle or
Today’s teleprotection applications are typically developed using a total fault clearing time of five to seven electricity cycles. The electricity cycle is at the rate of 50Hz (20 ms per cycle) in most parts of the world and typically at 60Hz (16.66 ms per cycle) in the Americas. As shown in illustration 1.8, the fault inception (1 in
16.6 ms (20 ms in a 50Hz grid) for total end-to-end delay comprised of teleprotection (TPR) equipment delay (2 in figure) and telecom network delay (3 in figure). So if teleprotection relay delay was found to be around 3 ms at each terminal, this leaves approximately 10 ms for an acceptable total telecom network delay. The 10 ms maximum latency for digital systems is defined
Illustration 1.7 Current differential protection scheme
Illustration 1.8 Teleprotection fault clearing
Current differential relays continuously send and receive sample values to compare currents, if they measure different values, the relay is tripped.
This animation shows the different steps in Teleprotection fault clearance and the time associated with each step.
in the IEC 834-1 standard. Older teleprotection systems that are 11
using analog (voice band) communication interfaces can be
Pilot based teleprotection systems rely on a communications
allowed up to 15, 20 and even 40 ms in some cases.
channel between them in order to send and receive information.
Latency is not the only constraint to which the teleprotection application is subjected, the variation of the latency (even if it is still within the overall 10 ms limit) is also a very important factor. Latency variation is also often referred to as Jitter.
This means that information is sent from both sides to the other side of the protected line. Hence we have two communications paths, one forward and one reverse path. For Teleprotection to work properly it is important that the latency is the same for both forward and reverse paths, the difference has to be less than 2ms
For teleprotection applications the Jitter tolerance is less than
and needs to be constant (within the Jitter tolerance).
This is
1ms, typically 0.5 ms.
often referred to as path asymmetry, or the difference between the forward and reverse path.
Illustration 1.9 Time constraints of teleprotection
Another standard that defines delivery times for power substation applications is IEEE 1646. Generally it defines latency for protection applications between 8 and 16 ms. Who are the players in the teleprotection market? The vendors who are active in the teleprotection relay market have been there for quite some time.
Names such as ABB,
Alstom Grid, General Electric, Schneider Electric, SEL, Siemens and ZIV-Dimat are often found in the Power Substations environments.
12
Section 2
Test your knowledge of teleprotection Review 1.1 Teleprotection Principles
Question 1 of 5 What is the purpose of teleprotection?
A. To monitor the condition of the power grid B. To isolate faults in the power grid C. To prevent damage to critical parts of the power grid D. All of the above
Check Answer 13
Chapter 2
Communication Networks for Teleprotection This chapter will cover the communication technologies that are used in power utility networks and more specifically the ones that are used for real time applications such as teleprotection.
Section 1
Interfaces of teleprotection equipment IN THIS SECTION
Before we start going into more detail about the communication networks which are used for teleprotection applications, it is worth elaborating a little on the
1. Analog interfaces 2. Digital interfaces
communications interfaces used by teleprotection equipment. Most older generation teleprotection equipment is using analog voice interfaces. The teleprotection equipment basically sends audio tones across the communications link which could be as basic as simple copper wires. There are
Illustration 2.1 Frequency spectrum of analog teleprotection systems
different frequencies that are used for the guard tone (to signal that the teleprotection relay is operational) and for the commands channels.
The
frequencies are all within the audible spectrum from 1100Hz up to 4000Hz. The frequencies and modulation schemes are defined by ITU-T standards. Commands are typically sent at very low speed (200 baud or lower) using frequency modulation. Channel
The physical interface used by analog teleprotection equipment is most commonly the 4 wire E&M interface. This interface uses 6 wires; 2 wires for transmit, 2 for
Guard Tone
receive and 2 for signaling. More recent teleprotection equipment is using digital interfaces. Over the past 20 years or more, there has been an evolution of these digital interfaces; the most common ones found on teleprotection relays are X.21, G.703, IEEE C37.94 and most recently Ethernet interfaces.
15
Section 2
Legacy Networks IN THIS SECTION
Teleprotection relays rely on a stable, symmetric, constant delay telecom network.
1. Traditional network architectures
Traditional telecom networking utilizes TDM transmission architectures based on
2. Limitations of traditional networks 3. Other factors which drive the need to change
PDH/SONET/SDH to provide the communication channel between relays.
The
circuit switched nature with fixed frame lengths provided some guarantee of delay limit, delay stability and transmission symmetry.
Existing PDH/SONET/SDH
Illustration 2.2 Typical traditional network architecture and frame structure
“The trouble with our times is that the future is not what it used to be.” - Paul Valéry - French Poet and critic (1871-1945)
16
networks have proven to be well suited to the task of supporting
Besides the limitations of the existing power utility
current differential protection, which has demanding performance
communications networks, there are a number of other business
imposed upon the communications networks.
related factors that are driving the need to change.
Over the past few years, there has been a growing demand for
The drive for Power Utilities to transform their telecom and
more Ethernet and IP based services to be provided for power
corporate Information and Communication Technology
substations.
department (ICT) into an integrated operations and ICT
Engineers are expecting to be able to connect to
their corporate intranet from within the substation.
Analog
organization is the direct result of the need to reduce costs,
telephones are being replaced by Voice over IP models, CCTV
maintain or increase performance while making the transition to a
cameras are moving to IP and multicast based solutions.
smart gird enabled company.
But perhaps the most important driver towards Ethernet and IP in the substation is the deployment of IEC 61850 based substation automation systems.
Illustration 2.3 Power Utility telecom transformation
These require Ethernet connectivity and
hence we see more Ethernet switches and routers being deployed in the power utility’s communications networks. While recent SONET/SDH equipment does provide Ethernet connectivity, they are ill equipped to deal with the scale and complexity of offering Ethernet and IP services for large power utility networks. The problem with non-MPLS based IP routing and Ethernet switching technologies however is that they aren’t well suited to deal with real time data from applications such as SCADA and teleprotection, so other solutions need to be considered.
This animation shows how Power Utility companies are transforming their operational and ICT departments into a converged organization in multiple steps.
17
This organizational transformation poses some significant challenges for the network infrastructure because the operational telecoms applications are mission critical and have fundamentally different requirements in terms of bandwidth, end-to-end-delay and jitter compared to the corporate ICT applications. The latter have been deployed on Ethernet and IP networks for a decade or more while the first have relied on more conventional TDM technologies for several decades.
Bringing those two different
worlds together onto a common telecommunications infrastructure is by no means a trivial task. However, the benefits are significant and the transformation must happen. Why is this happening and why now?
Security Issues. Coupled with government demand for smart grid capabilities is the increase in security directives. These are a
There are several key drivers forcing this transformation and
result of the increased terrorist and intrusion threats to power
making it very relevant and timely.
infrastructures. These directives are having a direct impact on the
The Smart Grid.
There is a big push from governments
network.
worldwide towards smart grid capabilities which have a significant impact on how power utilities have to manage demand and supply and be able to more rapidly adjust to both. This results in more sensors and actuators to be deployed producing more data to be collected and transmitted in real time. At the same time, applications using that data are communicating with various systems in the network.
18
Technology Lifecycle Issues. Most of the TSO and DSO
infrastructure to packet based technologies in order to reduce
communications networks have been using TDM technologies for
their costs, deal with increased bandwidth demand, support new
the past 20 years or more. Government regulations such as RoHS
applications and offer new revenue generating services. Service
and major shifts in technology in the service provider and
providers therefore are greatly reducing traditional leased line
enterprise markets are making it very difficult for the vendors of
services to the point where they are no longer offering TDM
TDM technologies to maintain their investment in the
based leased line services in many areas. Power utilities often rely
development and production of this technology. Most service
on leased line services either as backup to their own network or
providers and enterprise customers have evolved their
to reach remote substations. As a consequence of the service
communications networks to other technologies such as Ethernet
providers decision to no longer offer these services they are forcing packet services on power utilities.
and IP which resulted in a steady decrease of demand for TDM products over the past decade.
Illustration 2.4 MainStreet 3600
Consequently,
Own Fiber Optic.
Power Utilities
have deployed fiber optic cable
there is considerably less investment in
(for instance through new OPGW
the development and support of TDM
– Overhead Powerline Ground
technologies. There are fewer vendors on
Wire) through much of their
the market with native TDM products
infrastructure which provides
while vendor interoperability and end-to-
considerably more available bandwidth
end management is often a big issue.
to enable the transition to converged, packed-
Along with this aging of technology,
based networks.
maintenance costs are increasing and finding skilled staff to operate and
Evolution Of Microwave Technology.
maintain these legacy technologies and
power utilities have deployed microwave
equipment is proving to be very difficult.
communications systems to connect
Leased Line Service Migration.
As
The Newbridge MainStreet 3600 was a TDM multiplexer that has been sold from 1987 till
already mentioned earlier, service
2010. It was the world’s first software managed
providers are changing their network
TDM multiplexer
Many
substations to the communications network in areas where fiber or leased line services were not available or because microwave was the 19
Illustration 2.5 AlcatelLucent 9500 Microwave Packet Radio
most cost-effective solution. As
New Standards Evolve.
well, service providers have been
working on standards for packet based intra- and inter-substation
upgrading their mobile backhaul
automation and telecommunication (IEC61850-90-1).
networks from TDM to packet based technologies to keep up with the demand for bandwidth and support of IP. This is driving the development of high capacity and quality of service (QoS) based
microwave
communications systems, well suited to the utility and Smart Grid needs. New Technologies Mature. Viable packet based alternatives to TDM and SDH/SONET technologies are mature now and have been deployed widely.
Standards bodies such as IEC are
Cost Reduction. Service providers have moved away from SDH and PDH technologies because Ethernet has become a lot more cost effective. As a consequence, they are offering higher speed packet based services as a replacement for leased line services at much more attractive price points. Power utilities can take advantage of this and will save costs on leased lines and also on their own network equipment by moving to more Ethernet based solutions. Government Funding. In order to ease the move towards Smart Grids and the use of more green energy, governments are making
Some vendors have implemented new standards on some of their
funds available for research, trials and full scale rollouts.
products that support mission critical power utility applications
All of the above is resulting in an accelerated demand from Power
such as Teleprotection, SCADA, Telemetering and Telecontrol to work flawlessly on an IP/MPLS based packet network. However, all IP/MPLS implementations are not equal so power utilities will have to ensure their needs are met when selecting vendors.
Utilities to help them understand the new technologies and methods to transform their networks to modern, converged, reliable and agile infrastructures.
20
Section 3
Addressing the challenges with IP/MPLS IN THIS SECTION
With the advent of IP and Ethernet devices in the power systems, many utilities are
1. Misconceptions about IP/MPLS
migrating their telecom networks to IP/MPLS in order to support next-generation
2. Underlying principles
interfaces and to achieve a converged network infrastructure for existing, new as well as smart grid applications.
From a telecommunications requirement for
3. Performance of Teleprotection over IP/ MPLS
teleprotection, it is necessary to continue providing the connectivity between
4. Provisioning and Managing Teleprotection services on an IP/MPLS network
and asymmetry.
relays with a high quality of service quantified by limits on delay, delay variation An IP/MPLS network can meet these as well as other key
requirements such as reliability, flexibility, manageability and maintainability. A major concern for utilities is whether the IP/MPLS network can meet the strict latency requirements for protection signals between transmission substations. The doubts over IP/MPLS usually concern the ability to guarantee low latency service. First, IP/MPLS is sometimes and incorrectly perceived as connection-less IP-technology that can provide data transport but only with a “best-effort” like quality of service (QoS). This is the case for IP only however and in contrast, the MPLS part of IP/MPLS makes the solution connection-oriented and capable of multiple guaranteed quality of service (QoS) levels.
It is not the strongest of species that survives, not the
Another concern about IP/MPLS networks, as applied to teleprotection schemes,
most intelligent, but the one most responsive to change”
is the notion that the statistical nature of packet networks will impact the
–Charles Darwin English naturalist & geologist(1809-1882
performance of teleprotection relays.
This notion has been disproved through
extensive testing and implementation and is not applicable when the Utility 21
operates a “private” network based on IP/MPLS and, as with a
•
traditional PDH/SONET/SDH network, suitable care is taken in the
can be assigned a high priority to guarantee that the criticality of
engineering and provision of the network.
teleprotection requirements are met and reduced packet delay
SONET/SDH networks can be provisioned to provide alternate
The packets associated with teleprotection communication
variation through the network assured.
routes for mission critical traffic such as that between
•
The Alcatel-Lucent IP/MPLS network supports many
teleprotection relays. When operating correctly, the network
synchronization options to ensure that the network is properly
provides less than 50 ms switchover time. This 50 ms time is the
synchronized. Since the IP/MPLS routers are synchronized, they
“reference” in terms of resiliency for any new telecom technology.
can provide a good reference clock to the relays that are
IP/MPLS network technology provides several different fully
connected using serial interfaces by using the Network Clock to
standardized resiliency mechanisms to provide a switchover time
generate the Service Clock. Next generation relays connected
of less than 50 ms. These include end-to-end alternate paths and
using Ethernet can also be synchronized since the Alcatel-Lucent
MPLS Fast ReRoute (FRR). MPLS Fast ReRoute provides many pre-calculated alternate paths that can overcome any failure scenario in less than 50 ms. On top of that Alcatel-Lucent
Illustration 2.6 Alcatel-Lucent 7705 SAR, a teleprotection capable IP/MPLS router
provides Non-Stop Service (NSS) capability to the applications riding over the IP/MPLS network. More details about IP/MPLS and the reasons why this is the best long term solution over for instance MPLS-TP is provided in chapter 4. The Alcatel-Lucent IP/MPLS network supports teleprotection with many features: •
IP/MPLS networks utilize Label Switched Paths (LSP) to
ensure that all packets associated with a particular service, such as teleprotection, follow the same path. This is often referred to
The Alcatel-Lucent 7705 SAR-8 is a fully redundant and environmentally hardened IP/MPLS router with eight modular slots.
as strict paths. This ensures that the predetermined latency target is always met. 22
IP/MPLS routers support ITU-T Synchronous Ethernet (SyncE) and IEEE 1588v2 Precision Time Protocol (PTP). See chapter 5 for more details on synchronization. •
transport delay and jitter buffer/depacketization delay. Packetization delay relates to the process of transforming TDM traffic into packet data. For network delay, there is fixed delay
Alcatel-Lucent IP/MPLS routers natively support commonly
based on physical link speed and distances involved and a
used teleprotection interfaces including E&M, RS-232, X.21, G703
variable delay depends on the number of hops (nodes) in
and IEEE C37.94. It also supports Ethernet for next generation of
between.
Ethernet based relays. To reduce latency, it is advantageous to
Jitter buffer and de-packetization delay relates to the time
Thereby each node adds transit equipment latency.
support a direct interface from
required for a data packet to
the teleprotection relay to the IP/
move out of the jitter buffer and
MPLS routers.
This eliminates
Illustration 2.7 Packetization delay
to get de-packetized into a TDM
the need for a channel bank and
s t re a m c o n n e c t i n g t o t h e
the additional latency that is
teleprotection relays.
added.
The majority of the telecom An IP/
network delay occurs at the
MPLS network supports
edge where low speeds are
traditional relays using Circuit
present.
Emulation Service.
The key
depends on number of nodes
design consideration for
traversed. The use of coloring
supporting Teleprotection over a
and strict paths can be used to
packet network is how to
engineer the network to avoid
minimize latency.
path divergence and
Underlying principles.
For an IP/
Latency in the core
MPLS network, the telecom
asymmetrical delays.
network latency for TDM traffic
addition, the IP/MPLS router or
over IP/MPLS consists of packetization delay, network
This animation shows how serial data is packetized and sent across an IP/MPLS tunnel to the remote location where the jitter buffer ensures a smooth play out of the serial data
In
a GPS clock can provide a realtime clock pulse that can be 23
used for relays to time-stamp their data and thus ensure accurate
continuous play-out, thereby avoiding discards due to overruns
data comparison and processing.
and underruns.
Packetization delay is the delay imposed at ingress as the TDM
Another key element to assure the correct functioning of TDM
data is packetized. Smaller payload sizes with a higher number of packets per second result in lower packetization delay and lower one-way delay. Larger payload sizes with a lower number
Illustration 2.8 IP/MPLS network for power utilities
of packets per second result in higher packetization delay and higher one-way delay. The packet payload size is configurable. Network transport delay in the core depends on the number of nodes, distance and communications medium, but results mainly from transmission delays.
With IP/MPLS traffic engineering, a
service such as teleprotection follows a pre-determined path through the network that meets the strict latency requirement. On play out, a Circuit Emulation Service uses a jitter buffer to ensure that received packets are tolerant to packet delay variation (PDV).
This ensures the successful de-packetization of the
payload back into the TDM stream needed for communication with the teleprotection relay. The smaller the jitter buffer, the less
IP/MPLS enables simplification of the network and supports all critical services in a deterministic way.
delay is imposed. However, the jitter buffer needs to be set at a
applications over a packet network is the timing or
large enough value to ensure that jitter cannot cause a
synchronization.
communications failure on the teleprotection relays. The selection
have been developed to provide high precision timing over packet
of jitter buffer size must take into account the size of the TDM-
networks. The most important ones to consider for the use in
encapsulated packets. Larger payloads will require larger jitter
teleprotection applications are Synchronous Ethernet and IEEE
buffer sizes.
1588v2. Both technologies scale well in large networks and
A properly configured jitter buffer provides
Over the past few years, several techniques
provide timing precision that is appropriate for the teleprotection 24
systems.
However Synchronous Ethernet does have the
Manager (SAM). This product leverages more than 25 years of
advantage that it is a layer 1 based synchronization solution and
know how in network and service management software
therefore is totally immune to packet delay variation and it
development. It is building onto the experience of the 5620NM
interworks well with SONET/SDH network synchronization. This
products that were developed to manage the former
is an element that can be important in migration scenarios.
MainStreet3600 TDM network products.
Alcatel-Lucent IP/MPLS routers support both Synchronous Ethernet and IEEE 1588v2, allowing complete flexibility in the synchronization design of the network. Next generation relays now utilize Ethernet interfaces. Point-to-
5620 Service Aware Manager is a client-server based management system that supports up to 50 concurrent clients which can be set up according to specific rules related to span of control and scope of command.
point relay communication can be supported with an IP/MPLS network using Ethernet Virtual Leased Line (VLL) service and multipoint communication such as IEC 61850 GOOSE messaging can be supported using Virtual Private LAN Service (VPLS).
Illustration 2.9 5620 Service Aware Manager
Managing an IP/MPLS network for power utility applications. IP/MPLS is proving to be capable of handling the teleprotection traffic.
However one of the key concerns of power utilities to
embrace this technology is the perception that managing IP/ MPLS networks, provisioning services and managing alarms is very complex. Alcatel-Lucent has spent the past ten years developing a suite of network and service management products which leverage the many years of experience in managing PDH, SDH and ATM networks in order to simplify the task of managing IP/MPLS networks.
One of the main components of this suite of
5620 Service Aware manager, manages end-to-end services, network elements, all parts of the network layers, synchronization, SLA’s and much more.
management products is Alcatel-Lucent 5620 Service Aware 25
5620 SAM can be configured in an active/standby server
integration with OSS systems from IBM, HP and others. It also
configuration. The switchover happens automatically in case the
offers the ability to create web based service portals in order to
active server fails, or it can be triggered from the command
further simplify the day-to-day operation of the network, creating
interface.
reports etc. To ensure secure communication between 5620 SAM
5620 SAM allows thousands of nodes to be managed from the server. It allows the configuration of the nodes, creation of the point-to-point, point-to-multipoint and meshed services from an easy to use graphical interface.
It monitors services against
specific service level agreements (SLA). SAM provides the visualizing of all the paths and the services in the network in the network and can raise an alarm in case a path is re-routed and fall outside the limits set by the service SLA. Furthermore, 5620 SAM has a northbound interface, based on XML that allows the
Illustration 2.10 5620 SAM screenshot samples
and the OSS system, the XML interface supports encryption. The Alcatel-Lucent 5620 SAM provides: • Easy-to-use graphical forms for point-and-click element, network and service configuration • Wizards to guide administrators step-by-step through complex tasks • Advanced scripting, templating and rules-based configuration, allowing customization of the Alcatel-Lucent 5620 SAM for specific network or service requirements. This customization also allows non-expert resources to handle more complex tasks and eliminates repetitive data-entry activities It is not only important to have the tools to manage the elements, services, protocols, it is equally important to be able to plan ahead. For this Alcatel-Lucent offers different options. First there is the 5650 Control Plane Assurance Manager which is integrated with the 5620 Service Aware Manager. It allows to perform some off-line simulations. This means that users can look at different “what if” scenarios such as, “what if this link fails” or “what if this node fails”.
Network modeling an capacity planning can be
integrated with third party applications such as these from Aria Networks and Opnet, seamlessly integrated with 5620 SAM through the open XML interface. 26
Further automation and zero-touch configuration, if required, can be provided by deploying a service portal such as the Service Portal Express for Utilities. The Alcatel-Lucent Service Portal Express for Utilities is a lightweight, web-based application tightly coupled with the Alcatel-Lucent 5620 Service Aware Manager (SAM). It is purposebuilt for utility operators to simplify IP/MPLS network operations and management, delivering utility-specific workflows and terminology to bridge the gap between network experts and operators with less extensive training. For added security and control, workflows may also be routed to proper authorizations for review and approval. The Alcatel-Lucent Service Portal Express for Utilities enables rapid, cost-effective deployment by providing a frameworkbased architecture that is both modular and extensible.
Illustration 2.11 Service Portal Express for Utilities is making provisioning, monitoring and reporting extremely simple.
Base modules are included to provide the following capabilities: • Provisioning • Monitoring • Troubleshooting • Reporting
27
Section 4
Bandwidth & Latency considerations IN THIS SECTION 1. Packetization principles
Packetization Principles In the previous section we covered the underlying principles of the packetization
2. Impact of jitter buffer and payload size configuration on bandwidth requirement
technique which is used to take data from a serial (i.e. X.21) or n x 64kb (i.e. E1, G.
3. Validation testing
there is an optical fiber infrastructure available that offers plenty of bandwidth, so
4. Examples of implementations
there is no immediate need to focus on the bandwidth consumption impact of
703, C37.94) interface and send it across an IP/MPLS network. In most cases
packetization. However, there are a number of cases where there is no fiber optic network available as is the case when using microwave radio links or copper lines with xDSL modems. In those cases, it is important to understand the impact of the parameters, used to configure the packetization system, on the bandwidth requirement that results from these parameter settings. Therefore it is worth looking at the principles of packetization in a little more detail. Packetization is the process that occurs when legacy interface data (serial data or data coming from DS0 channels in a T1/E1) is presented to a router’s ingress port. The router needs to take a series of bytes and put them into packets (=encapsulation) at a certain constant rate which then get routed to their destination. This is done according to a standard mechanism as defined in the CESoPSN standard (RFC 5086). An alternative mechanism to transport TDM over
28
a packet network is Structure Agnostic Transport over Packet (SAToP).
Illustration 2.12 The E1 frame structure - tap to play
This method is not taking the DS0 channels into
consideration and basically transports the complete E1 or T1 transparently. The Jitter Buffer assures a smooth play-out of the serial data on the egress port and compensates for packet delay variations that may occur during transit of the packet through the network and intermediate network nodes. When a CESoPSN circuit is provisioned on IP/MPLS routers we will need to define the payload size (in bytes) at the ingress port and the Jitter buffer depth (in ms) at the egress port. The Impact Of Payload And Jitter Buffer The configuration of both parameters will have an immediate impact on the number of packets that will be generated and the bandwidth consumed on the network ports. Let’s consider an example whereby we packetize data from an E1 interface. The E1 frame structure consists of 32 bytes with each byte representing a different time slot of the E1 frame: TimeSlot 0 to TimeSlot 31. With the E1 interface speed being 2.048Mb/s, each E1 frame takes 125 microseconds. The structure of an E1 frame is illustrated in illustration 2.14. The E1 structure consists of cycles of 16 frames which ensure CRC4 error checking. The time
An E1 frame consists of 32 PCM channels or time slots , which means they are 8 bits in length each. Timeslot 0 is used for framing and error correction, timeslot 16 is used for signaling purposes of the voice channels in timeslots 1-15 and 17-31.
it takes to complete a full E1 cycle (16 frames) becomes 16 x 0,125 ms = 2 ms. When packetizing data such as that from an E1 or T1 circuit, there are a few key parameters that will have an important influence on the resulting end-to-end latency and the bandwidth requirements. These parameters are the payload size (PS), this is the amount of data (in bytes) we take from the E1 or T1 circuit (or other serial data) to put in each packet. For an E1/T1 circuit this 29
would be the number of time slots (N) multiplied by the number of
the resulting jitter buffer delay (JBD) is half of the configured JB
frames (F) we want to send per packet. For the purpose of
value.
calculating the formula is:
The total delay (TD) therefore becomes the sum of
PS = N x F is expressed in bytes
the
packetization delay, the fixed delay and the jitter buffer delay.
We know that our frame rate (FR) is one frame every 125 microseconds.
TD = PD + FD + JBD
A parameter which will have a crucial impact on the bandwidth
With this information we can calculate the packetization delay
utilization is the packet rate (PR) in packets per second, this is the
(PD) by multiplying the frame rate with the number of frames we
result of the product of the number of E1/T1 frames we chose to
want to send per packet.
put in each packet (F) with the frame rate (FR) which is 0,125 ms
PD = FR x F typically expressed in ms
Every telecom equipment such as a router will introduce a fixed
or 0,000125s for E1. The formula to calculate the packet rate is
PR = 1/(FR x F) in packets per second
delay, this will be in the range of tens or hundreds of
To calculate the resulting bandwidth (BW) in bits per second we
microseconds depending on the architecture of the product.
need to add the protocol overhead (MPLS, Control Word,
This fixed delay (FD) needs to be added to the calculation of our total delay.
size (PS) and multiply by the packet rate times eight (eight bits per byte).
The next important parameter is the jitter buffer size. The concept of the jitter buffer has been explained in section 3.
Ethernet, ML-PPP,...), which is typically 42 bytes to the payload
The Jitter
Buffer is needed to assure a smooth flow of data and minimize packet delay variation. The jitter buffer size (JB) is a parameter that is configurable, typically in increments of 1ms. When a router starts to play-out the data it receives at 50% of the jitter buffer,
BW = (PS+42) x PR x 8 = bandwidth in bits per second
Let’s consider the following example. We are transporting one timeslot of an E1(N=1) and we will put 16 frames in each packet (F=16), therefore the payload size is
PS = N x F = 1 x 16 = 16 bytes 30
The packetization delay becomes
PD = FR x F = 0,125ms x 16 = 2 ms
Illustration 2.13 Sample table of packetization bandwidth
Let’s assume a fixed delay of 0,3 ms and we configure a jitter
800
buffer to one ms, then the total delay becomes
736
TD = PD + FD +JBD = 2 + 0,3 + 0,5 = 2,35 ms
The packet rate in this example would become PR = 1/(FR x F) = 1/(0,000125 X 16) = 500 packets per second And so the resulting bandwidth requirement would be
600 bandwidth in kb/s
400
400
200
BW = (PS + 42) x PR x 8 = (16 + 42) x 500 x 8 = 232 kbits/s. As can be seen from these calculations, careful considerations must be made when putting TDM traffic over packet networks.
232 148
0
32
16
8
4
number of E1 frames per packet
When available bandwidth is constrained for instance over copper lines or microwave links, one must pay attention to the parameters used for the packetization of the data. It is important to understand the boundaries of the application in terms of
The illustration below shows a sample graph of how the number of E1 frames one choses to put into a single packet has an impact on the bandwidth utilization.
maximum latency, apply a margin to it and work the numbers back to find the right balance in terms of end-to-end latency and bandwidth requirements. In networks where the links are built with fiber optic connections, bandwidth is less on an issue and therefore tuning the parameters towards the lowest possible latency is not a problem. 31
Section 5
Performance Validation and Use Cases IN THIS SECTION
Performance of teleprotection over IP/MPLS
1. Performance and validation of Teleprotection over IP/MPLS
Alcatel-Lucent engaged Iometrix, the networking industry’s preeminent testing and
2. Examples of implementations
certification authority, to test and validate the ability of the Alcatel-Lucent IP/MPLS based 7705 Service Aggregation Router (SAR) and 7750 Service Router (SR) products for implementing an IP/MPLS network to support teleprotection. This testing was done in collaboration with Toshiba who provided their GRL100 relay
Illustration 2.14 Toshiba GRL-100
equipment and engineering personnel who participated in the testing and verified proper performance of the relays. The Toshiba equipment was available in both X. 21 interface and Ethernet interface versions, allowing the verification of support for both traditional and next generation teleprotection relays. Based on a comprehensive battery of tests, it was concluded that a network comprised of Alcatel-Lucent IP/MPLS routers will comply with all the requirements of teleprotection with substantial margin. The IP/MPLS network performed well within the requirements of the teleprotection application that has, to this point, only been supported by circuit switched (TDM) network (e.g. based on SONET/ SDH) devices. The Alcatel-Lucent routers support legacy and next-generation
“If I have seen further it is because I could stand on the
device interfaces such as Ethernet and consequently can support both existing
shoulders of giants”
teleprotection devices as well as those that will be deployed in the future. This capability is crucial for smooth migrations of utility networks.
–Isaac Newton English scientist (1643-1727)
32
In addition to the Iometrix validation, Alcatel-Lucent and third
control of the bandwidth required per application could be
party laboratories have also conducted testing, sponsored by
achieved with an Alcatel-Lucent IP/MPLS network.
utilities that evaluated teleprotection equipment from ABB, Areva (Alstom) and Siemens with Alcatel-Lucent IP/MPLS communications equipment. Further tests have been conducted with teleprotection equipment from ZIV/DIMAT, the TPU-1 with E&M, X.21 and G.703 interfaces.
The capability of IP/MPLS networks to support teleprotection is not only proven in the lab, it is proven in actual deployment. Altalink, a Transmission Operator in Alberta, Canada with 11,800 km of lines and more than 300 substations, has successfully deployed an Alcatel-Lucent IP/MPLS network in a live
These tests again demonstrated that low end-to-end delay could
environment since September 2010 supporting teleprotection
be achieved, that failover capabilities met or exceeded SONET/
alongside general utility SCADA and other operational voice and
SDH standards for traffic re-routing (< 50 ms) and that total
data traffic. More validation tests have happened in November 2012 with Creos, the power utility of Luxembourg. The tests were meant to
Illustration 2.15 Teleprotection equipment successfully tested over Alcatel-Lucent IP/MPLS networks
validate the use of differential protection relays over the Creos IP/ MPLS network. The tests have proven that differential protection works well over IP/MPLS, using C37.94 and E1 interfaces. The tests were conducted over different network paths; once over a short path between two adjacent nodes and a second time over a long path which consisted of 11 routers and a total fiber length of 104km. The end-to-end latency observed during the tests was 3.87ms on the short path between the relays using C37.94 interfaces and 4.56 ms over the long path. With the E1 interfaces on the differential relays, the end-to-end latency was 3.37ms over
The ZIV-DIMAT TPU-1 was tested with E&M, X.21 and G.703 (E1) interfaces. 1 of 12
the short path and 4.12ms over the long path. The tests showed consistent results over a longer period of time and prove that IP/MPLS is suitable for differential protection even 33
if the data is sent over a large number of hops. Each intermediate
Current differential protection schemes have been in operation on
hop between the ingress and egress hops adds 77microseconds
the IP/MPLS network at Creos since January 2013.
to the end-to-end latency.
Illustration 2.16 Differential protection testing at Creos
34
Table 2.5.1 Summary of teleprotection over IP/MPLS performance tests Vendor
Model
Type
Latency (ms)
Jitter (microseconds)
Interface
Siemens
7XV
Distance
3.12
5
X.21
ABB
NSD570
Distance
3.6
5
Ethernet
Areva
DIP5000
Distance
3
5
X.21
ZIV/Dimat
TPU1
Distance
5.92
E1
ZIV/Dimat
TPU1
Distance
8
E&M
Nokia
TPS 64
Distance
8.3
G.703 codir
Siemens
7SD52
Differential
3.12
Siemens
7SD52
Differential
3.87(*)
C37.94
Siemens
7SD52
Differential
3.37 (*)
E1
Areva
P541/P591
Differential
3.25
Alstom
P545
Differential
3.48
Toshiba
GRL100
Differential
2.9
3
X.21
Toshiba
GRL100
Differential
0.1
3
Ethernet
5
5
X.21
X.21 C37.94
These tests all included at least 4 IP/MPLS routers in the path between the Teleprotection equipment, with congestion on the links. (*): Tested on live power utility network between adjacent nodes. 35
Examples of implementations
Illustration 2.17 Current differential scheme on 138kV over a microwave link - tap to play
In this particular case multiple T1 links are used from the microwave radio to create an aggregate bundle of 12Mb/s. The protection equipment is GE L90 using RS232 interfaces at 38.4 kb/s.
Illustration 2.18 Current differential scheme on 500kV line over microwave links
The example above also uses microwave links, albeit longer reach. The payload is kept low which increases the packet rate.
36
Illustration 2.19 Same link as the previous example with a separate set of differential relays
Illustration 2.20 Distance protection scheme of 240kV line over fiber optic links
The secondary set of protection relays are SEL 311L and are connected over G.703 interfaces at 64kb/s. The jitter buffer setting is a bit more narrow, hence the lower latency. The same microwave MLPPP bundle is used as network link.
In this example, there are two fiber optic links between the nodes, one is over OC-3 (155Mb/s) and the other is over a 10Gb connection. A second distance protection scheme is used over a diverse path to protect the same HV line. The second set of protection relays is Siemens 7SA522, also connected using G.703 interfaces.
37
Illustration 2.21 Four leg current differential scheme Table 2.5.2 Master ring with E1 interfaces between A and B E1 Interface
Jitter Buffer ms
Payload Bytes
SITE B B to A
SITE A
Site B
A to B A to B B to A
CESoPSN
5
480
7.33
7.33
7.46
7.46
SAToP
1
64
1.15
1.15
1.26
1.26
Table 2.5.3 Slave ring with E1 interfaces between A and B E1 Interface More complex and higher performance current differential protection schemes are built in ring topologies as shown here.
Jitter Buffer ms
Payload Bytes
SITE B B to A
SITE A
Site B
A to B A to B B to A
CESoPSN
5
480
7.46
7.46
7.53
7.53
SAToP
1
64
1.17
1.17
1.29
1.29
38
Table 2.5.4 Master ring with E1 interfaces. Readout of the 7SD52 relay latency in ms E1 interface
Jitter Payload Buffer SITE B Bytes ms B to D
SITE D
SITE C
SITE A
D to B D to C C to D C to A A to C A to B
SITE B
B to A
CESoPSN
5
480
5.07
5.07
4.81
4.81
5.31
5.31
5.08
5.08
SAToP
1
64
1.21
1.21
1.11
1.11
1.10
1.10
1.16
1.16
Table 2.5.5 Slave ring with C37.94 interfaces. Readout of the 7SD52 relay latency in ms
C37.94
Jitter Payload Buffer SITE B Bytes ms B to D
SITE D
SITE C
SITE A
D to B D to C C to D C to A A to C A to B
SITE B
B to A
CESoPSN
5
32
4.00
4.00
3.87
3.87
3.93
3.93
3.93
3.93
SAToP
2
32
2.50
2.50
2.37
2.37
2.43
2.43
2.43
2.43
39
Illustration 2.22 Differential protection over xDSL with IP/MPLS
Extensive testing has been conducted at the Strathclyde university by Dr. Steven Blair and Dr. Campbell Booth. The full test report is available on the Strathclyde university website: “Real Time Teleprotection testing using IP/MPLS over xDSL”.
Illustration 2.23 Bandwidth versus distance for xDSL
This chart illustrates the available bandwidth versus the distance on typical copper lines depending on the DSL technology used. 40
Section 6
Test your knowledge on communication networks for Teleprotection Review 2.1 Questions on Chapter 2
Question 1 of 8 Place the labels on the image where they fit in terms of interface type and typical speed
C37.94
X.21 E&M
E&M
X.21
C37.94
41 Check Answer
Chapter 3
Migration Options In the previous chapter, we have established that IP/MPLS is ideally suited to replace the SONET/ SDH based TDM networks in use by power utilities. The challenge is how to seamlessly migrate from SONET/SDH to an IP/MPLS network. In this chapter we will cover a number of possible scenarios.
Section 1
Migration options IN THIS SECTION
Migration options will vary depending on a number of factors, most importantly,
1. When spare optical fiber is available
the availability of bandwidth for the IP/MPLS routers on the communications
2. Reuse of SDH/SONET backbone 3. Introducing CWDM optical multiplexing
infrastructure. In cases where unused fiber optic cables are available in the cable bundle that connects the substations, a parallel network can be created which allows the most smooth migration scenario to be used. The picture below illustrates how this could work.
Illustration 3.1 Using separate fiber optic cables in the same bundle Moving from SDH to an IP/MPLS network involves careful planning and depending on the availability of optical fiber, different scenarios may be applied.
43
In cases where there is not extra optical fiber available to create a
Once this is completed, the TDM/PDH layer can be removed,
separate IP/MPLS network, the IP/MPLS network can be built as
leaving the SDH/SONET multiplexer as the only legacy network
an overlay on top of the existing SDH/SONET infrastructure. For
element in the network. In order to remove this SDH/SONET
instance, if there is already an Ethernet over SDH/SONET
element from the network gracefully, it is best to introduce a
available at the substation, the IP/MPLS router can be connected
WDM layer in the optical network.
to that interface and immediately replace the router connected to
existing WDM infrastructure. Alcatel-Lucent offers a full range of
it at the substation. If there is no Ethernet interface available on
DWDM and CWDM solutions. The 1830PSS family of products
the SDH/SONET multiplexer, then the IP/MPLS router can be
offers very advanced DWDM capabilities and is also fully
connected through an STM-1 interface for instance. This would
integrated within the Alcatel-Lucent 5620 Service Aware
enable IP and Ethernet services to be deployed at the substation.
Management solution. Furthermore, the 7705 Service
Following that, other services like SCADA and Teleprotection can be migrated across to the IP/MPLS platform.
In many cases there is an
Aggregation Router product family offers an integrated (passive)
Illustration 3.3 Introducing WDM as a migration enabler
Illustration 3.2 Scenario 2, re-use of SONET/SDH
44
CWDM solution which can further simplify the migration process. The illustration below shows how a 7705 SAR can be used initially as a passive CWDM solution to initially provide more optical cable capacity before the IP/MPLS capability is added.
Illustration 3.6 7705 SAR CWDM modules
Illustration 3.4 Implementing CWDM with the 7705 SAR
Single color Passive CWDM Optical Add/Drop Multiplexer card (OADM) for 7705 SAR-8 and 7705 SAR-18.
Illustration 3.5 Sample CWDM setup with Alcatel-Lucent 7705 SAR
There are several CWDM cards available for the 7705 SAR
7705 SAR-8
products. The illustrations on the right show a couple of these cards and how they can be used.
7705 SAR-M
The 4 channel card also
supports the 1310nm wavelength which is typically used in SDH/
7705 SAR-8 7705 SAR-8/18
SONET networks. This makes the 7705 SAR ideally suited to integrate with existing SONET/SDH networks.
7705 SAR-8
7705 SAR-8
7705 SAR-8
45
Chapter 4
MPLS Technologies
for Teleprotection In this chapter we analyze the different options for next generation packet networks to support mission critical applications such as Teleprotection.
Section 1
Market evolution of IP/MPLS and MPLS-TP This chapter is meant to help power utilities to understand which
Some communications vendors are in the early stage of MPLS-
technology is most appropriate for their needs and to help
TP consideration and implementation as those standards
understand the basic differences between the proven technology
become more formalized. Before any significant uptake occurs
of IP/MPLS and more recent MPLS-TP.
though, two crucial questions need answering:
IP/MPLS, contrary to what many people believe, is not a new
• Is it a viable technology?
technology. It’s development started in the mid 1990’s in order to improve the performance of routers. The idea of using labels
• Is it a compelling technology?
instead of IP addresses was driven by the limited performance of
These are even more significant for non-carrier, mission-critical
routers to perform address lookups and their abilities to scale in
networks for organizations such as power utilities. Initially, the
large networks. Since those days, the paradigm has shifted,
principle driver for MPLS-TP was to apply a simplified subset of
router performance no longer is the prime issue, however
IP/MPLS to ‘bridge the gap’ between the packet and transport
network and service scaling is.
worlds by combining the packet efficiency, multi-service
The telecommunications industry has been focusing on IP/MPLS since the year 2000 and has created many standards ensuring a fully featured and interoperable framework. IP/MPLS has become the standard communications solution for service providers worldwide and has found great success in mission-critical networks in such industries as rail, electrical utilities, government and public safety organization, defense and some large
capabilities and carrier-grade features of IP/MPLS with the transport reliability and OAM tools traditionally found in SONET/ SDH. This seems like a reasonable approach on the surface but as you will see, there are a number of drawbacks and limitations associated with this simplified subset. In fact many of the objectives of MPLS-TP have meanwhile been achieved by IP/ MPLS technology implementations.
enterprises. 47
Alcatel-Lucent has already achieved the objectives of MPLS-TP through it’s extensive IP/MPLS product portfolio and management by the extensively proven 5620 Service Aware Manager (SAM) network management tool. This end-to-end management tool supports not only the Alcatel-Lucent IP/MPLS portfolio of routers and switches but also the DWDM long-haul optical transport products (1830 PSS) and packet microwave radio products (9500 MPR) with extensive capabilities and time saving, error preventing solutions. The next sections in this chapter will describe the fundamental distinctions between IP/MPLS and MPLS-TP, as it pertains to mission-critical industries such as power utilities.
48
Section 2
MPLS Technologies The concept of a unified communications framework based around IP/MPLS consists primarily of two main functions. These functions are
•
Transport Layer Functions
•
Services Layer Functions
IP/MPLS There are two major control plane protocols that allow the creation of an IP/MPLS service network, which are RSVP-TE and LDP. RSVP-TE provides many tools to achieve the same level of control on the network as an SDH network, where LDP provides a much more dynamic environment more equivalent to a full IP
There is a wide variety of transport layer functions used today in
network (with IP network we mean typically a connectionless IP
the MPLS context, including LDP, RSVP-TE, BGP labels and
network). It is important to note that both technologies can easily
MPLS-TP.
co-exist on a single network and that it is at the design stage that
For the purpose of transport, it is really a matter of comparing the applicability of RSVP-TE and LDP to static MPLS-TP as a transport mechanism linking to the service layer functions. This service layer function, driven by applications, is converging on IP.
one or the other protocol will be used to answer the needs of the respective applications in terms of resiliency and control. This flexibility may depend upon the vendor’s implementation in their respective products.
This is true in the railway metro and mainline market segments,
The development of RSVP-TE based traffic engineering has been
for example, even though some TDM based applications remain
ongoing for more than 15 years and is still evolving today to meet
(GSM-R, Interlocking, SCADA, operational telephony...).
The
the needs of ever converging applications. This technology is
same applies to utilities, where the applications are rapidly
widely deployed and very successful in meeting high availability
converging on IP (IEC104, Goose, eSCADA...) but where some
demands. This includes the functionality to facilitate Dynamic
TDM applications will remain for a long time (Teleprotection,
Control Planes for automatic bandwidth adjustments and re-
SCADA...).
routing or optimization, as well as protection and rerouting 49
mechanisms.
The control plane used for facilitating these
functions is indirectly dependent on the presence of an IP based control plane. The IGP will update the traffic engineering database (TED), with all the CSPF (Constrained Shortest Path First), link coloring information etc and RSVP-TE will then utilize the information in the TED. This architecture creates an abstraction between the IGP and IP/MPLS meaning that a failure in the IGP will not cause RSVP-TE based paths to fail.
IP/MPLS For Mission-Critical Networks A key enabler for the safe and efficient transformation of power utility communications network is a modern, reliable and flexible infrastructure that forms the core network in order to route the critical application traffic such as Teleprotection, grid monitoring, control and status and key corporate data effectively, efficiently and on time. Furthermore, non-critical applications around substation automation services such as voice, video surveillance,
It is within this market that Alcatel-Lucent has pioneered the adoption of the use of this IP based control plane to make use of RSVP-TE and LDP based transport mechanisms to facilitate the
Illustration 4.1 IP/MPLS supports all protocols and any transport mechanism
establishment of a framework for delivering both Point to Point and Point to Multipoint L2 Services. In addition to the use of these mechanisms, Alcatel-Lucent has also been able to enhance the L2 point to point and multipoint services with L3, L4 and application level intelligence. This allows critical networks to continue to evolve the services on the installed platforms as the industry demands. This flexibility that ALU has brought to the market is unprecedented and fundamentally based on the in-house based data path hardware and software developments allowing the continuous adoption of new technologies and standards as the industry evolves. Only this approach can ensure investment protection for periods exceeding 10 years.
The Multi Protocol nature of IP/MPLS means that it can run on any transport network, over any media and it natively supports any protocol on top of it.
corporate LAN/ or VPN and even public internet, are also able to leverage the same infrastructure.
These types of requirements
are forcing many industries, governments and enterprises to 50
consider an evolution of their communications infrastructures that
Telecommunication Union (ITU-T) have undertaken a joint effort to
would be very different from their traditional Time Division
standardize a new transport profile (TP) for the multi-protocol
Multiplexing (TDM) centric networks. A flexible transformation is
label switching (MPLS) technology that is intended to provide the
required to preserve existing investments and to minimize risks.
basis or the next generation packet transport network.
Alcatel-Lucent IP/MPLS communications infrastructure
fundamental idea of this activity is to extend MPLS where
incorporates state-of-the-art technologies to enable an industry,
necessary with Operations, Administration and Maintenance
government or enterprise to deploy a future-proof, highly
(OAM) tools that are widely applied in existing transport network
available IP network to continue supporting existing TDM and
technologies such as SONET/SDH or OTN and to enable it to
legacy applications while providing a smooth migration path to IP,
operate in the absence of a dynamic IP-based control plane.
Ethernet and IP/MPLS-based services. This new IP/MPLS infrastructure will maximize the cost-effectiveness and efficiency of the network without jeopardizing reliability, while enabling the deployment of new devices and applications that can improve operational and workflow efficiency.
The
MPLS-TP, as defined in RFC5921 in the IETF, is positioned to enable MPLS to be deployed in a transport network and operated in a similar manner to that of existing transport technologies. It enables MPLS to support packet transport services with a similar degree of predictability, reliability and OAM to that found in
A highly available IP/MPLS communications infrastructure is
existing transport networks. It defines a subset of MPLS
ideally suited to support both mission-critical operations and
protocols for Layer 1 (L1) and Layer 2 (L2) only.
corporate communications requirements. In addition, the AlcatelLucent network and service aware management platform allows organizations to improve their efficiency by automating and simplifying operations management for communications services, thus reducing the barrier in introducing IP/MPLS-based technologies and services. MPLS-TP The Inter net Engineering Task Force (IETF) and the Telecommunication Standardization Sector of the International
Some of the rationale behind the push for MPLS-TP is that there is an increasing demand for legacy transport networks to support packet based services and hence the need for evolution to MPLS like behavior, as an evolution of SDH. IP/MPLS networks are perceived as complex by some in the transport world and there is sometimes a further requirement to keep the layer 2 and layer 3 networks separate. In many cases the existing SDH infrastructure and transport switches lack native IP support. The alternative to providing native IP support is to combine the architectural, 51
management and operational models of Circuit Switched
applications. In short, these are the counter-arguments for IP/
transport networks with Packet switching optimizations using an
MPLS.
MPLS data plane and additional OAM and Protection capabilities.
• IP/MPLS can deliver very simple transport solutions just as well,
The main arguments that are generally used to promote MPLS-TP
however they offer the advantage of allowing very rapid and
are the following:
flexible changes (adding nodes, links, applications) which is a
• Transport solution leveraging SDH experience and therefore with no control plane, leading to a fully manual provisioning • High performance set of OAM and protection tools to allow fast failover times • Capacity to integrate legacy traffic inherited from SDH evolution. • Bi-directional transport of data in LSPs reducing the risk of path mismatch between a source and a destination and its return path. • Reduced cost of equipment (CAPEX), as the technology requires less intelligence due to the absence of a control plane. • High level of security due to the absence of IP based control plane.
lot more cumbersome with SDH and MPLS-TP. • Alcatel-Lucent offers even more OAM tools as part of its IP/ MPLS solution not only to assure fast failover times in the transport layer, but also to assure the services at higher layers. • The capacity to integrate legacy traffic is no different in MPLSTP from IP/MPLS, both have CESoPSN and SAToP capabilities. • The bi-directional LSP behavior is guaranteed by AlcatelLucent’s 5620SAM management tool, so there is no divergence of forward and reverse LSP paths possible. • The idea that MPLS-TP reduces CAPAEX is a major mistake because it is forcing another layer 3 overlay to be added to the network which is not the case with IP/MPLS. • The control plane protocols used in IP/MPLS are not part of the MPLS labeled traffic for the applications, they are secured and authenticated. Furthermore control plane protection
These arguments need to be put in perspective when comparing
mechanisms are in place to prevent any control plane attack to
to IP/MPLS technology and implementations, but also when
have any impact on the user/application traffic.
determining the applicability of MPLS-TP for industry 52
Building a mixed network with IP/MPLS and MPLS-TP
still consider that all the services part is to be provided by IP/
technologies
MPLS. Whether this vision applies to private networks which are
The relationship between IP/MPLS and MPLS-TP can be depicted in this diagram (used at the MPLS World Congress) where MPLS-TP was promoted for large access/aggregation networks for transport scalability while IP/MPLS remains in edge and core for services, however interest for MPLS-TP at MPLS World Congress has subsided.
typically smaller than large carrier networks is very questionable. Are private industry organizations willing to train two teams on two different technologies, quite possibly from two different vendors if not necessary? Such a choice has a very high OPEX cost and probably does not apply in markets other than mobile operators. From a technical perspective, there are also several challenges
Illustration 4.2 End-to-end interworking challenge
when interworking IP/MPLS and MPLS-TP domains. The first is that there is no control plane interworking between the two.
Illustration 4.3 End to end vision of large carrier networks including MPLS-TP
Providing end-to-interworking between MPLS-TP and IP/MPLS is quite challenging because of the different control plane architectures, disjoint protection mechanisms and different Operations Administration & Maintenance (OAM) tools.
The first information that can be extracted from this drawing is that carriers consider MPLS-TP as a backhaul technology, but
MPLS-TP is being positioned as the aggregation network for large carrier networks. 53
The second is that end-to-end interworking is very difficult in a multi-vendor environment and no mechanisms to ensure end-toend resilience interworking as shown in Illustration 4.3. Note also that from an operations perspective, there is little chance to find an end-to-end network management tool because MPLS-TP vendors often do not have IP/MPLS solutions and IP/ MPLS vendors which may have MPLS-TP solutions generally have two different management solutions. Alcatel-Lucent however have both IP/MPLS and MPLS-TP solutions, managed by the same management system, 5620 Service Aware Manager.
Illustration 4.4 Screen shot of 5620 SAM
The 5620 SAM manages both IP/MPLS and MPLS-TP.
54
Section 3
Comparing technologies This section will review the positioning used to promote MPLS-
easier with IP/MPLS due to the fact the provisioning can be
TP into certain markets, demonstrate how IP/MPLS can meet at
static end-to-end as in MPLS-TP but can also be a mix of strict
least the same level of functionality of MPLS-TP and review other
and loose therefore alleviating the load of provisioning required.
values of IP/MPLS that cannot be met by MPLS-TP.
This solution has been validated by several teleprotection and
This section will also describe how Alcatel-Lucent has enhanced the IP/MPLS standard to provide a unique, secure, complete end-to-end solution.
SCADA vendors. • High performance set of OAM: IP/MPLS has many OAM features that are similar to that of MPLS-TP. The only difference
Comparison between MPLS-TP and IP/MPLS technology
is in the way it is used, but overall, it provides the same services. The OAM trigger resiliency features available as part
Some applications like teleprotection in
of IP/MPLS (Fast ReRoute, primary/secondary LSP, active/
utilities and SCADA in many different industries, built on the
standby pseudowire) can also be used to control SLAs. Table
blue/red model, require the avoidance of common mode failure
4.4.1 below lists all OAM tools which are available on the
scenarios. For that matter, in the case where a single physical
Alcatel-Lucent IP/MPLS platforms. Bi-Directional Forwarding
network is built (typically a ring), there is a need to control the
Detection (BFD) can be used to trigger fast failover of L3
traffic between the source (SCADA RTU) and destination
protocols such as RSVP, VRRP at 10ms intervals.
• Manual provisioning:
(SCADA MASTER), in order to ensure that they don’t share a common path or equipment. MPLS-TP being built around manual provisioning can provide this feature, but, IP/MPLS with the use of RSVP-TE (traffic engineering), can also provide exactly the same feature. However, the provisioning process is
• Capacity to integrate legacy traffic: The legacy transport (typically E1) is part of the MPLS standard (CESoPSN and SAToP) and is therefore supported by IP/MPLS and MPLS-TP. Main mobile operators are successfully transporting E1 traffic of GSM over IP/MPLS for more than 5 years, demonstrating the 55
technology is totally suited for such legacy transport.
Table 4.3.1 OAM tools on Alcatel-Lucent IP/MPLS routers
• Bi-directional transport of data in LSPs:
In order to totally
eliminate the risk, for applications that are sensitive, IP/MPLS
OAM function ICMP & ICMPv6
Ping Traceroute
Two Way Active Measurement
TWAMP
LSP diagnostics
LSP Ping, LSP traceroute
SDP diagnostics
SDP ping, SDP MTU path discovery
RSVP-TE supports static provisioning of paths which allows control of traffic end-to-end and ensures that bi-directional traffic takes the same path, exactly as MPLS-TP does. Furthermore Alcatel-Lucent’ 5620SAM will always provision symmetrical LSP paths. • High level of security: IP/MPLS has been deployed by carriers
VLL diagnostics
VCCV ping, VCCV trace
for years and has been demonstrated as a very solid technology
VPLS MAC diagnostics
MAC ping, MAC trace, MAC populate, MAC purge, CPE ping
to protect data. However, as with all security frameworks, it is
Ethernet OAM
802.1ag, Y.1731, 802.3ah
and implementation, to ensure full security. The utility market
Ethernet loopbacks
Line, internal
have rolled out a specific security framework called NERC-CIP
OAM propagation to attachment circuits
ATM ports, E1 ports, Ethernet ports, Pseudowire status signaling propagation
not only a matter of technology, but also a matter of process
(in North America) to protect their mission-critical applications. It is important to note that many utilities networks world-wide have successfully met these criteria with an IP/MPLS network.
LDP signaling
Many critical Defense networks are using an IP/MPLS Multicast debugging
Multicast trace, Multicast stats, Multicast route info
LDP Tree trace
Allows to see multiple paths for a given service
Service Assurance Agent
SLA monitoring
MPLS-TP platform? Both can use the same kind of silicon
Control Plane Assurance Management
OSPF, ISIS, BGP, PIM,... network control plane visualization
technology, so, there is no reason for a price difference.
BFD
Bi-directional Forwarding Detection
context of network requirements on the platform itself, not only
infrastructure and have so for years. • Platform cost: Is an IP/MPLS platform more expensive than an
Therefore, this question needs to be looked at in the broader in the context of technology. For example: 56
Illustration 4.5 Cybersecurity enabled by IP/MPLS
Is accurate synchronization through packet-based
synchronization transport mechanism like IEEE1588v2 important? If yes, the platform must make use of specific hardware to be able to insert and extract timestamp information with enough precision. Also a high quality oscillator needs to be used to minimize frequency optimize synchronization recovery.
With the advent of next generation silicon and processor
technology, packet processing and forwarding (switching, bridging or routing) in the data plane is no longer a dominant cost factor when designing a new platform. When comparing platforms of different packet-based technologies, close attention Because of it’s native IP capability, IP/MPLS can offer a more comprehensive cybersecurity solution to mission critical networks.
must be paid to understand whether the platforms meet other
Is the capability to absorb traffic burst to minimize packet
are equally pivotal when building a mission-critical network.
loss crucial to the operational applications in mission-critical
When all factors are considered, the platform cost difference
networks? If yes, then the memory cost would be the same on
should be negligible.
both platforms.
non-data plane requirements, such as those listed above, which
As well, since IP/MPLS supports layers 1 through 3 and MPLS-TP
Is supporting a large number of OAM sessions (e.g. Y.1731
only supports layers 1 and 2, there is significant cost to having
continuity check) with a small transmission period crucial to
separate layer 1 and 2 from layer 3 solutions including equipment,
achieve SDH/SONET-like fault detection speed? If yes, then the
management, training, sparing, etc. This is discussed further in
platform requires either a powerful processor or specific hardware
the document.
assist to process in-service and out-of-service OAM protocol data units (PDU) continually for tasks such as frame loss
Other values brought by IP/MPLS
measurement and delay/jitter measurement in a scalable fashion.
As mentioned, MPLS-TP generally ignores the end to end
MPLS-TP Continuity Check does require this.
requirements, mainly for IP applications and this is where IP/ 57
Illustration 4.6 IP/MPLS supports all services at L2 and 3
collaboration between the two layers. Main concerns are the following:
Illustration 4.7 Overlay IP network on top of MPLS-TP Transport
Whatever the application requirements at L2 or at L3, IP/MPLS supports all of them, either for point-to-point or point-to-multipoint applications.
MPLS fulfills the complete range of services for mission-critical applications. • IP Awareness: Typically MPLS-TP has no IP awareness and therefore, for all applications that are IP based (eSCADA, CBTC, VoIP, meter reading, CCTV, PMU’s...) customers using an MPLS-TP path will have to add an IP layer on top. Therefore, the network will look like the one in Illustration 4.7. Such architecture creates issues because the two layers operate
Using separate technologies for aggregation and edge plus core leads to more complexity and is more difficult to manage.
• MPLS-TP provides poor efficiency of bandwidth because there is a need for native IP traffic to be transported between the two layers and typically the end to end path will likely not be optimized because it is fixed.
independently meaning that there is absolutely no dynamic 58
• End to end resiliency for IP applications will be triggered by
through IP-VPNs and therefore can ensure through a single
MPLS-TP features and Layer 3 protocols, typically OSPF,
technology (single vendor and, potentially, single management
therefore, convergence times can be much higher than 1
tool) a fully optimized transport of data end to end. That is the
second in many scenarios which may create big outages for IP
typical solution that has been used by carrier networks for more
applications such as operational telephony over IP. This can
than ten years and has recently become the solution of choice
create impacts on latency in the case where the lower layer
for utilities, government organizations, railways and other
backup scenarios are different than the upper layer scenarios.
mission-critical networks.
• Provisioning of services will be considerably more complex as
• Fast provisioning of services: mainly applications such as
there is a need to fully mesh the IP routers which will typically
SCADA, Phasor Measurement Units, GOOSE... are not only
require a lot of LSP manual provisioning.
point to point but becoming more point to multipoint and often
• Troubleshooting will be more complex as there is a need to use different tools and different technologies.
require the flexibility for disaster recovery scenarios. Therefore, doing manual provisioning like MPLS-TP for such applications doesn’t scale at all. That is why for these applications, having
• Most private network operators need to segment their IP
the choice to adapt provisioning from very manual to fully
applications in different networks for organizational or regulatory
dynamic, as IP/MPLS provides, answers all needs of all
issues including CCTV, intranet, telephony, financial payment
applications. MPLS-TP can only provide solutions for some
applications, etc. With a basic IP network, this is not possible
challenges of some applications but typically does not take a
unless using complex VRF features which, again are very
real end to end view of all requirements for railway, utilities and
statically configured hop by hop and this activity doesn’t scale
defense applications in a network.
well.
• Multiple failure scenarios: IP/MPLS supports many scenarios for
• Most vendors of MPLS-TP have little IP knowledge and
resiliency and always ensures that as long as there is a path
therefore railway, utility, government and other organizations will
available between a source and a destination, the traffic may
have to deal multiple vendors and their respective network
use it, if it makes sense for the application. MPLS-TP has only
management tools which can lead to a lack of clear ownership
one alternate path, if the first one fails, which may not be
when a problem arises. IP/MPLS natively understands IP traffic
enough to reach 99.999% availability expected by mission59
critical and safety applications. It is important to note that in
Clearly, a major impact on the cost of equipment and
very meshed networks, ensuring that the primary LSP does not
bandwidth, a fact often not mentioned by promoters of MPLS-
share any path or device with its redundant LSP is not always
TP technology.
that easy to control. Relying on manual provisioning or on unproven network management tools can be very risky compared with the much proven and dynamic technology of IP/
Alcatel-Lucent added value to a standard IP/MPLS implementation
MPLS, with LDP for example, that has years of effective demonstration of resiliency capabilities.
Illustration 4.8 IP/MPLS can cope with multiple failures
• Multicast Awareness: the number of video cameras is increasing heavily in many organizations for CCTV and is the most bandwidth hungry application that may be transported over a mission-critical communications network. In order to optimize any video traffic delivery, IP has been enhanced with multicast protocols which have a simple role of ensuring that a video stream is transported once over a physical link independently of the number of receivers being registered for the stream. The multicast protocols basically use the different network equipment (routers, switches) to replicate the traffic. While IP/MPLS has native multicast capabilities, MPLS-TP has none, which means that for example if a CCTV camera stream
MPLS-TP is not able to cope with multiple failures which is problematic for critical infrastructure networks at times of natural disasters.
has to be stored in two locations (for resiliency) and has to be
To make a clear and fair comparison between MPLS-TP and IP-
monitored by two OCCs, the traffic of each camera will be sent
MPLS, organizations should also include in their study all
four times over the links in access and core. Multiply this by
enhancements that have been made to IP/MPLS standards by
3Mbps, which is the average bandwidth for good quality video,
some specific design or implementations which are typically a
you realize that with 100 cameras for example, you will generate
demonstration of a more mature technology and which applies to
900 Mbps more traffic with MPLS-TP than with IP/MPLS. 60
railway, utility, public safety and other organization running
support such a variety of synchronization options that include
mission-critical networks.
native GPS, native BITS ports, SyncE, 1588v2 and ACR.
• Legacy integration: where most of MPLS-TP vendors only
• End to end application resiliency: most of the IP devices are not
transport E1, Alcatel-Lucent has developed natively in its
directly connected to an MPLS router, but rather to an Ethernet
access router portfolio, some interfaces to cope with many
switch. In order to ensure end to end resiliency in line with
applications which use E&M (operational telephony and LMR),
critical applications requirements, Alcatel-Lucent has developed
FXO/FXS (for internal telephony), serial interfaces – X21, RS232,
features that ensure that the first IP/MPLS node traversed (the
V35 (for SCADA), G.703 (for interlocking), C37.94 (for
PE) can be made redundant and that resiliency is not triggered
teleprotection). This allows organizations to install a single piece
by Spanning Tree which is not only hard to manage, but also
of equipment capable of connecting all legacy devices as well
very slow to converge. Features such as MC-LAG (Multi-
as all IP applications. Compare this with MPLS-TP solutions
Chassis Link AGgregation) allow a switch, or server to connect
that typically would need the MPLS-TP switch, plus an IP router
to two different IP/MPLS routers to provide fast resiliency. This
plus a PDH mux. In those cases, the low price argument of
feature provides 250 ms failover time. For trackside switches in
MPLS-TP is no longer relevant and CAPEX and OPEX (3
railways for example, the architecture leverages the standard
training, 3 management solutions, 3 contracts, 3 spare lots....)
ring protocol G.8032 and in the same way, ensures end to end
will be in favor of an IP/MPLS solution.
resiliency between this access technology and IP/MPLS. As per
• Synchronization: Alcatel-Lucent has developed many synchronization capabilities into the IP/MPLS portfolio and can claim support equal to or better than SDH, which is increasingly being replaced by IP/MPLS networks. Relying on years of deployments in mobile networks and in GSM-R networks ensures that all features and tools to control clocking are fully embedded for all applications such as GSM-R, Tetra, teleprotection, IEC61850, etc. MPLS-TP solutions cannot
MC-LAG, resiliency in that case is ensured in much less than 250 ms. We have observed convergence times of less than 50ms with G.8032. Other technologies are available to ensure resiliency with non standard protocols and all these features, allow network operators to define a real end to end architecture that answers the requirements of all applications, only available through IP/MPLS • Richness of transport technologies: many critical network operators (railways, air traffic control, power utility, oil & gas, 61
defense...) can leverage their own fibre infrastructure but for
based applications, the full integration provides a much quicker
resiliency purposes they sometimes have to create loops and
time to solve potential problems compared to an end to end
don’t have fibre to close the ring. In such cases, the native
network with different products and technologies.
implementation of microwave IDU in the IP/MPLS router portfolio provides a very good solution to provide a seamless end to end solution, with microwave awareness. In other situations, fibre has become a scarce resource and in those cases, optical multiplexing is becoming necessary to provide the expected bandwidth. In these cases, Alcatel-Lucent provides a full end to end portfolio of optical CWDM and DWDM solutions which are controlled by the same management product as the IP/MPLS and microwave products. As well, some of the IP/MPLS routers have integrated passive CWDM multiplexers to provide an optimized and small footprint solution. • Easy Network Management: MPLS-TP’s claim that IP/MPLS is a complex technology requires customers to look at the operations of a full end to end communication network before deciding. The Alcatel-Lucent 5620 SAM solution provides complete end to end management further simplifying operations and maintenance of an IP/MPLS solution beyond that of MPLSTP and certainly beyond requiring separate solutions and management for layers 1 / 2 and layer 3. • Provisioning is easy and the time to service is very short compared to MPLS-TP’s tedious manual provisioning. For IP 62
Section 4
IP/MPLS or MPLS-TP for mission critical applications In the mission-critical market segments of rail, utilities, public safety, etc., most of the new applications rely on IP, even for operational telecoms: eSCADA, signaling, teleprotection, LMR
Illustration 4.9 End to end service with IP/MPLS and MPLS-TP technologies
such as Tetra, 3G and LTE mobile communications and a lot of applications that include video such as video surveillance. All of these IP applications won’t be carried efficiently on a transport centric infrastructure, creating an additional burden on provisioning to compensate for the inefficiency of the transport. IP/MPLS however, handles both services and transport, reducing the complexity of a multi-technology model. This is depicted in the diagram below where it shows that IP/MPLS provides both the service intelligence as well as the necessary OTN interface to the Transport network. IP/MPLS covers the transport as well as the services while MPLS-TP only covers the transport part. Alcatel-Lucent’s interworking with the optical layer makes it possible to manage the service, transport and optical layers from a single management platform.
Typically, MPLS-TP suits the needs of a very hierarchical, large network and has therefore been adopted mainly by large mobile operators in China. In this scenario all communications are built 63
Table 4.4.1 A comparison of the applicability of IP/MPLS and MPLS-TP for network transformation Attribute
IP/MPLS
MPLS-TP
Maturity
proven for more than 10 years in carriers and with many railway deployments, (typically pushed by UIC group), utilities and public sector networks for government, Defense, Public Safety, ...
few deployments in large mobile service provider networks (mainly Asia). Infancy, little or no proven deployment experience for railways, utilities and government networks
Standardization
fully standardized since 1997
some still in progress
Multi-vendor interoperability
proven for more than 10 years in many customers with different vendors in the core, edge and access
limited to trials
TDM transport
yes, CESoPSN or SAToP
yes, CESoPSN or SAToP
VPN support
Layer 2 and Layer 3
Layer 2 only
Service support
full services
transport only
IP Multicast delivery
fully supported and optimized
none
Protection
Fast ReRoute, active/standby
Resilient to multiple failures
YES
NO, would require a control plane protocol
Traffic Engineering
dynamic, easy to accommodate change
static, difficult to cope with changes
Convergence of transport and services network
single solution covers all layers, less hardware diversity, less pares, common management, one training
more hardware diversity, separate management systems, more training to attend
OAM
single platform from L1 up to L3 (and above)
different platform needed for L3 , MPLS-TP covers L2 only
IP/MPLS & MPLS-TP interworking
unproven
unproven 64
manually and therefore would not support large scale any to any
Some MPLS-TP users have already moved away from this
service connectivity. IP/MPLS would be required for scalable
technology, towards a full IP/MPLS implementation because they
layer 3 any to any connectivity. Therefore, the organization would
have come to realize that multiple network failures can occur and
need to maintain two technologies, which can only be justified by
at times of crisis such as natural disasters, the resilience of the
very large organizations.
network is of utmost importance.
65
Section 5
Conclusion on MPLS Conclusion on IP/MPLS vs MPLS-TP Power Utility network operators who are considering to build new communication infrastructures for their applications should compare technologies in the context of end to end application needs not in terms of a subset of the solution considering just the transport infrastructure. The most important thing to consider is that IP/MPLS does provide the same services as MPLS-TP, but also much broader services such as IP optimized transport, much easier
utilities that IP/MPLS was the way forward and that MPLS-TP could not provide the adequate infrastructure for all applications. Other users of MPLS-TP have realized that their infrastructure should be resilient to multiple failure scenarios which often occur when natural disasters happen such as floods, earth quakes etc. These customers have chosen to move to IP/MPLS for this important reason. In fact a lot of these utilities are taking the IP/MPLS philosophy from the WAN deeper into the network, into the substation LAN.
provisioning, larger flexibility to accommodate different
In this way, IP/MPLS ensure seamless provisioning of all services
application requirements and more.
down to the application inside the substation.
From an industry point of view, in the railway industry for
All in all, comparing the installed base, technology maturity,
instance, the UIC has clearly pointed to IP/MPLS as a technology
development investment, standards evolution, market knowledge
to accommodate the needs of railway applications, MPLS-TP is
and adoption, critical network operators should find in IP/MPLS
not considered a viable solution to fulfill the end to end needs of
not only the technical solution but also the market to allow them
railway operations. In the last 5 years, many utilities have also
to securely deploy maintain and grow their network
transformed their network to prepare for Smart Grid applications
infrastructures for decades to come.
which are typically IP based and more and more Any to Any types of communications. Therefore, it has become clear to all
66
Chapter 5
The Importance of Synchronization
In previous chapters we have addressed the need to support legacy interfaces and the requirement of accurate synchronization throughout the network. In this chapter we will cover the synchronization options provided by packet networks, the accuracy they deliver and their application in power utility networks.
Section 1
Introduction This chapter is mainly based on the Synchronization over
service providers.
Ethernet Networks white paper from Alcatel-Lucent.
scaling required for these networks.
Many applications in power utility networks require an accurate
Over recent years, several (packet based) technologies have
synchronization mechanism. This can be required in order to
emerged to meet these timing needs. These range from changing
provide timestamp information to make sure that the application
the Ethernet physical layer to provide synchronization reference
itself can assess the validity of the data. Especially when
distribution (along the lines of SDH and SONET) to Timing Over
samples are made of current and voltage in the power grid, it is
Packet (ToP) techniques such as Adaptive Clock Recovery (ACR)
important to know exactly at what time this data was recorded in
on Circuit Emulation Services, enhanced NTP and IEEE1588v2.
order to make the right decision about the status of the power
The latter two technologies also address the requirement for the
grid at that point in the network. SONET and SDH technologies
distribution of highly accurate time for applications that need
have by the very nature of the technology had built-in
time-of-day or phase accuracy.
synchronization capabilities, the “S” in the acronym stands for “Synchronous”. In every SONET/SDH network where an accurate time-stamping capability is required, there is one master clock source (or more than one for redundancy) which is used to propagate timing information to all nodes in the network. In the past, packet networks have lacked the capability to provide timing accuracy which is required in real time network environments of power utilities, railway operators and mobile
They lacked the accuracy, resiliency and
When power utilities, railway operators and service providers look toward these new technologies, they need to view the transport of “timing” and “time” as a network service. These services will have network planning considerations along with OAM&P tools in order to ensure proper operation within the network. Traditionally, reference timing has been distributed using synchronous physical layer technologies.
This means that the
68
devices driving the interfaces have the ability to both transmit the
interfaces (Synchronous Ethernet) or a higher layer timing
data using a timing reference and to recover the timing from the
distribution protocol needs to be used (Timing Over Packet).
received data for reference purposes. Many technologies include
These two approaches are depicted below.
this capability. For example: PDH, SDH, SONET, POS, PDH Microwave, DSL, GPON and WDM. Traditional Ethernet does not fall into this class since it uses local interface timing rather than a timing reference for its transmissions. The Ethernet interfaces either need to be modified to act as synchronous physical layer
Synchronous Ethernet, which was introduced in 2008 is an enhancement of a legacy Ethernet interface and includes the ability to relay accurate timing information along with the Ethernet frames over the physical media. It is the highest performing solution for timing over Ethernet, but may cause some issues for complete network rollout. For instance when a part of the network
Illustration 5.1 Two main principles of synchronization
does not have the hardware capability to support Synchronous Ethernet, because it is going over another service provider network for instance. Several ToP techniques exist that allow for the transport of timing information without the need for a synchronous physical layer. These include NTP, IEEE1588 and ACR. Network Time Protocol (NTP), defined in IETF RFC 1305, has been used for many years to allow distributed devices to synchronize with respect to Time of Day. It uses timestamps embedded in packets to accomplish this synchronization. This protocol has been deployed in networks for over 20 years and has been proven to provide reliable time distribution to accuracies in the order of milliseconds. This level of accuracy is often acceptable for alarm time-stamping, billing and statistics; however, for applications like
Synchronous Ethernet and Timing over Packet are two different mechanisms that can be used to provide synchronization in Ethernet networks.
E1/T1 timing and mobile base-station phase and frequency references, accuracies of approximately 1 microsecond are 69
required.
IEEE1588 has, at its core, a timestamp distribution
mechanism very similar to NTP. Version 1 was targeted at time distribution over an Ethernet LAN while Version 2 (introduced in 2009) adds some key capabilities to address the distribution over the WAN environment. Both versions were designed to allow higher accuracy time distribution than possible with NTP. Circuit Emulation Service (CES) implementations use the constant bit rate of the TDM interfaces to generate packets. Adaptive Clock Recovery relies on this constant rate of packet generation to recover the original timing information. This is primarily intended to allow for the transport of the originating E1 or T1 service clock, but if the originating interface is known to be locked to the PRC traceable reference, CES can then be used to distribute a highly accurate timing reference. In the following sections, we will discuss these different options that are available to ensure proper synchronization of the network and provide support for Teleprotection, SCADA and PMU applications on a converged IP/MPLS network. It is important to understand the different techniques available in order to achieve network synchronization. Implementing the right synchronization solution will make the transition from the legacy installed base to a new network a lot easier. 70
Section 2
Synchronous Ethernet During early 2006, several European Telecom companies started
clock at one of its input interfaces or from a dedicated timing
an initiative within the ITU-T to define the requirements for having
interface such as a BITS port. Synchronous Ethernet is based on
the traditional Ethernet interfaces meet comparable timing
the same architectural structure as SONET/SDH. It is important
performance targets to those of SDH/SONET interfaces. They
to note that Synchronous Ethernet works at layer 1 and is
recognized that the Layer 1 relaying of synchronization
concerned only with the precision of the timing of signal
information would potentially be the most reliable form of
transitions to relay and recover accurate frequencies. It is not
synchronization transfer and would not have any impact from the
impacted by the traffic load. For this reason, it has been shown
packet delivery load over the interface.
to have a performance equivalent to that seen in SDH/SONET
By deploying a network of Synchronous Ethernet interfaces, or a
networks.
hybrid of SDH/SONET and Synchronous Ethernet, the network
Illustration 5.2 shows a test environment created within the
provider can ensure the delivery of the same quality of timing
Alcatel-Lucent laboratories to analyze the performance of a long
references through the network as is currently achieved using
chain of a total of 20 Service Routers and Service Aggregation
SDH/SONET only. This can then be used for TDM services at the
Routers using Synchronous Ethernet as the reference frequency
network edge or as a timing reference into Mobile base-stations
distribution method. For this test, six of the devices were
for carrier frequency derivation.
deployed within a thermal chamber to introduce extreme
Synchronous Ethernet uses the physical layer of the Ethernet link to distribute the clock among nodes in the network. In an analogous manner to SONET/SDH, each node has a local or system clock which determines the outgoing clock rate of each interface. The system clock is derived from the incoming
temperature variations into the environment. Even with this long chain and a 65° C temperature range, the timing reproduced at the end of the network was two orders of magnitude better than the network limit as defined by the ITU-T as is illustrated in illustration 5.3.
71
Illustration 5.2 Synchronous Ethernet test over 20 nodes
Illustration 5.3 Test results over 20 nodes
The chart shows how well the Synchronous Ethernet technology exceeds the ITU G.823 and ANSI DS1 standards, even after 20 nodes and in extreme temperature variations. Synchronization quality tests were conducted over 20 nodes, 6 of which were placed in a thermal chamber with temperature changes of 65°C over more than 24 hours
these discussions and is developing applications to make use of these capabilities.
The Synchronous Ethernet specification also includes the same ability to relay timing source quality level information as is found in the Synchronization Status Messages (SSMs) of PDH and SDH/SONET. Since Synchronous Ethernet uses Ethernet OAM messages for this purpose, there is discussion within the standards bodies of these SSMs being expanded to support many new features to assist in the management of Synchronization Distribution. Alcatel-Lucent is actively involved in 72
Section 3
Precision Timing Protocol IEEE 1588v2 While Synchronous Ethernet is a Layer 1 enhancement providing
performance is the variation in the Packet Transfer Delay (PTD)
for the relay of timing information completely independent of user
across the network or the Packet Delay Variation (PDV). There
traffic volume, the remaining technologies to be discussed are all
are different Timing over packet technologies such as Adaptive
packet based and are impacted by the user traffic. This means
Clock Recovery (ACR), Differential Clock Recovery (DCR),
that the packets carrying the timing information must compete
Network Timing Protocol (NTP) and IEEE1588v2. Because of the
for network resources with all of the other data services and the
requirements of synchronization accuracy in power utility
routing protocol packets. Significant work has been conducted
networks, we will focus on the IEEE1588v2 standard because
over the past several years which has led to implementations of
none of the other Timing Over Packet solutions meet the
these technologies being able to meet the required performance
requirements in terms of number of hops they support and
targets over relatively noisy environments where variable queuing
accuracy hey provide.
and processing delays can be introduced.
IEEE1588v2 and its Precision Time Protocol (PTP) message
Since changes in network conditions with respect to traffic load
exchange is a mechanism that can be used to synchronize time
can affect the performance of these technologies, there is the
and timing within a network. Version 1 of this standard is
need to provide more management and monitoring ability for
currently being used in the LAN environment of industrial
these services than might be required for Layer 1 technology like
manufacturing. It uses a very similar concept of time-stamped
Synchronous Ethernet. Alcatel-Lucent has architected its Timing
packets between master and slave network elements to NTP but
Over Packet solutions to include the OAM&P capabilities to allow
includes some enhancements such as higher packet rate and
these technologies to be deployed with success.
hardware-based time-stamping to improve on the accuracies of
As indicated above, all of these ToP technologies are affected by traffic load in the network. The key parameter that relates to the
the recovered time. IEEE1588v1 has demonstrated accuracies in the one microsecond range in the LAN environment. However,
73
when applied to the noisier WAN environment, it cannot
measures the residence time of each PTP message as the
guarantee this performance. Version 2 was created to try to
message transits the node and updates the message with this
address this noisier environment.
residence time. Since most of the PDV is caused by queuing within the nodes, the transparent clock can remove this unknown.
Illustration 5.4 1588v2 principles
If every device between the master port and the slave port performs as a transparent clock, then the actual transit time for each message can be measured and corrected. In an ideal IEEE1588v2 network, every device along the path — from the Grandmaster clock to the slave clock — is IEEE1588v2 aware and acts as a boundary or transparent clock. However, many implementations have been developed that can meet the performance targets in environments where there are a number of non-IEEE1588v2-aware devices between the master and slave clocks. 1588v2 defines 5 types of PTP devices.
Two significant concepts within IEEE1588v2 are the boundary clock and the transparent clock. The boundary clock is a device which has at least one slave port recovering timing/time from an upstream master and it then uses this recovered timing/time as a basis for one or more master ports toward downstream slave
• Ordinary clock • Boundary clock • End-to-end transparent clock • Peer-to-peer transparent clock • Management node
ports. The boundary clock can then be used both for scaling purposes and as an intermediate device to break up the PDV between the grandmaster and the slave devices. The transparent clock is a device that participates in IEEE1588v2 but does not perform any timing/time recovery. The transparent clock 74
Ordinary Clock An ordinary clock is a PTP device which has a single PTP port in a domain, where a domain consists of a logical grouping of clocks communicating with each other using the PTP protocol. An ordinary clock can be the source of time into the network (grandmaster) or recover frequency from the network(slave). Boundary Clock
correction field within the PTP packet header.
There are two
types of transparent clock, end-to-end and peer-to-peer. End-To-End Transparent Clock An end-to-end transparent clock compensates for the PTP packet residence time within a device. Peer-To-Peer Transparent Clock A peer-to-peer transparent clock compensates for the residence
A boundary clock is a PTP device with multiple ports in a domain.
time and the propagation delay from the upstream PTP clock.
It can have one port operating as a slave recovering frequency
Use of Peer-To-Peer Transparent Clocks requires the use of the
from an upstream master port and other ports acting as masters
Pdelay messages rather than the Delay messages in all clocks.
to redistribute the recovered frequency to downstream slaves. As the number of hops between a master and slave increases, typically so does the PDV incurred. Boundary clocks mitigate the end to end PDV by breaking the PTP packet flow into more manageable per hop segments. Transparent Clock A significant portion of the end to end PDV between a master and slave is caused by queuing within the intermediate network equipment. A transparent clock forwards and modifies PTP packets to compensate for the residence time within these intermediate devices. The residence time is placed in the
Management Node A management node is a device that communicates with other PTP devices using PTP management messages. It can be a standalone device or a logical entity within another PTP device. Packet Delay Variation The variation in the time it takes packets to transit a network is referred to as PDV. PDV is an important factor that impacts all ToP techniques like ACR, NTP and IEEE1588. If the packet delay through the packet network is constant, the arrival rate of packets at the destination node is constant. It will be relatively easy for the clock recovery module to recreate a stable frequency, phase and time of day from the delivered packets. If the packet delay varies, 75
then the recovery algorithm is more rigorously exercised. The
against these limits. Unfortunately, ToP implementations as they
algorithm has to deal with both sudden and slow changes in
exist today cannot always quantify the PDV metric or the specific
packet delivery based on reroutes or slowly increasing loads in
limit. There have been several proposals for how to measure the
the network, as well as possible changes in the master timing due
PDV (peak-to-peak level, TDEV, minTDEV, MAFE, etc.) but since
to reference switches (needs to be followed) and/or drift in the
implementations use different filtering techniques, it is not clear
local oscillator at the core of the local clock (needs to be
that one single PDV metric will apply to all solutions. Even within
removed). Therefore, a clock recovery design must incorporate a
one implementation there may be factors built into the timing
filtering capability to recognize and remove these effects. There
recovery algorithm which filter the PDV in multiple ways. In these
are several causes of packet delay variation on a packet network.
cases, one single PDV metric would not be sufficient to define the
Some examples include:
algorithm’s behavior.
• Random delay variation (e.g., packet arrivals at queuing points) • Low frequency delay variation (e.g.,day/night traffic load patterns) • Systematic delay variation (e.g., store and forward mechanisms in the underlying transport layer) • Routing changes and congestion effects The most significant factors in the PDV are: • The number of nodes (which contain queuing points) • The speed of the interfaces (GE vs. FE, for example)
When a PDV metric (or set of PDV metrics) is available, the target deployment network will need to be measured to ensure that the PDV metrics are not exceeded. It may be difficult to create all the network conditions over which the solution must operate, so some scenarios will have to be estimated. It will be essential for any deployed solution to provide some indication of its performance so that it can be monitored to ensure that performance limits are being met.
• Technology of the interfaces (Ethernet, DSL, μWave) • The packet size, randomness and load of the aggregate traffic in the network In order to know whether a particular ToP implementation will meet the performance targets in a given network deployment, it is desirable to characterize the limits on the PDV that the implementation can support and to also measure the network 76
Section 4
Experience and recommendations Leased Line Replacement
Rollout of Synchronous Ethernet
An Ethernet network can provide traditional E1/T1 interfaces and
SyncE – SDH interop
perform circuit emulation across the network. This configuration could replace the traditional leased E1/T1 services offered by wireline carriers or the E1/T1 services provided in the owned power utility network. In this deployment, the Circuit Emulation Service (CES) must meet the same performance characteristics of the SDH/SONET transport networks. The principle concern is the transport of the E1/T1 bits from one location to another. This means that support for service clock transport is required. This service clock transport support can be provided using either an ACR or DCR timing method with the CES.
If the network is providing Ethernet ports on a SDH/SONET backbone, then there is likely to be an option to upgrade only the Ethernet interface modules from legacy Ethernet to Synchronous Ethernet capabilities. The SDH/SONET equipment will already support a clock architecture conformant to the SDH/SONET requirements and should be able to operate in Hybrid mode between SDH/SONET synchronization distribution and Synchronous Ethernet distribution. In this environment, the Synchronous Ethernet interfaces can be rolled out only where needed at the network edge. This allows for a phased transition
When access to a traditional synchronization distribution network
from the SONET/SDH network over to an all Synchronous
with BITS devices is available, then the preferred clock recovery
Ethernet backbone while providing immediate support of
technique is DCR (as it is immune to PDV). If such a distribution
Synchronous Ethernet services at the network edge.
network was not available, then the option is to either use DCR on the E1/T1 circuits along with a dedicated ToP technique to
SyncE with ToP
distribute the common reference or to use ACR on the individual
The ToP technologies have to be concerned with the PDV
E1/T1 circuits.
between the timing master and the timing slave points. Using SyncE as the synchronization distribution technique over the 77
core links (and then ToP only over the last mile links) can be a
and lower loading. As there are many more links at the network
cost-effective deployment option while ensuring performance is
edge and usually multiple transmission technologies, this permits
maintained. This scenario is depicted in illustration 5.4.
a cost-effective and consistent solution over the last mile.
Illustration 5.5 Using PTP in the “last mile”
End-to-end Timing Over Packet As discussed above, the principle concern with the Timing Over Packet techniques is the control of the PDV. The larger the network span between the master and slave agents, the larger the peak-to-peak PDV and the greater the trend to Gaussian distribution. In all Timing Over Packet implementations there will be an engineering limit to the network span over which it will work. There are three methods to address this. The first is to use distributed masters where GPS-based masters are placed within the network to ensure that the span between the master and slave agents is reduced to an acceptable limit. The second method is to use boundary clocks within the network between the primary master and the edge slave devices. The third method, applicable to IEEE1588v2 deployments only, is to use transparent
Combining PTP for the last mile with SyncE on the main network provides a viable and cost effective solution.
clocks within the network. The use of distributed timing masters, or boundary clocks using a central primary master, are similar concepts. Before the availability of network elements
Since traffic loading is lower at the network edge than near the
incorporating boundary clock functionality, the use of distributed
core hub, the smaller number of core links can be upgraded to
timing masters will be more common. While the distributed GPS-
Synchronous Ethernet. This avoids the high PDV that can occur
based masters are operating in plesiochronous mode, the
when the links are run with high loading, while at the network
difference in frequencies at the various receivers is so small (0.01
edge ToP can maintain performance when there are fewer links
ppb) that there should be no real impact from this mode of 78
operation. Once the networking elements are upgraded to support boundary clock functionality, the use of one central GPSbased timing master and then operating through boundary clocks
Illustration 5.7 PTP test over 13 nodes, including microwave links and interfering traffic
to reach the network edges will become more common. In this case, the end-to-end PDV is replaced by several segments of low PDV, but with the tradeoff of operating several clock recovery control loops in series. More analysis is needed to determine if the use of boundary clocks will become more common. More data needs to be collected on the tradeoffs between using boundary clocks to reduce the PDV of a single large network span into multiple master-slave spans. Is there some optimal number of internodal links that provide enough PDV reduction while not introducing too many recovery steps? At least for frequency distribution, preliminary testing has shown some benefits to having boundary clocks in every network element in order to ensure the PDV is minimized for each master-slave association. Some of the results of our tests are shown in illustration 5.5 below. What this tells us is that it is good practice to combine PTP and Sync E, the former for Time/Phase, the latter
These test illustrate how the use of PTP for ToD and Frequency can be combines with SyncE to achieve the best possible result for Phase and Frequency accuracy.
for frequency accuracy. All Alcatel-Lucent IP/MPLS routers, the 7750SR, 7705 SAR and 7210 SAS products
For a number of the 7705 SAR routers, Alcatel-
Illustration 5.6 GPS module for 7705 SAR-H
Lucent is offering a GPS module to be built-in, this helps to address situations on remote sites
support both SyncE and 1588v2 to allow for
where PTP is not possible due to excessive PDV
the most flexible and robust synchronization
and SyncE is not available.
schemes.
79
Illustration 5.8 External synchronization inputs and outputs on 7705 SAR-8
Sync In and Out
Time of Day in and Time of Day Out
operation of the distribution network will be relevant to the new Synchronization Management
environment. New capabilities unlocked by enhancements to the SSM of Synchronous Ethernet will become available to assist and
In networks where synchronization is of utmost importance, as is
expand on the management of the distribution network. These
the case in power utility networks, it is important to be able to
capabilities can be integrated into dedicated management
troubleshoot synchronization issues very rapidly.
systems or incorporated as subsystems within the overall service management agents of the network. The intention will be to
As Synchronous Ethernet interfaces roll out in a network and are
develop a comprehensive system to allow for the analysis of the
used for synchronization distribution, the methods and
distribution network to look for optimizations and analyze network
procedures developed by the SONET/SDH timing experts in 80
Illustration 5.9 Managing Synchronization with 5620 SAM
PDV filtering algorithm. If the algorithm is having trouble generating a consistent output, then this will be reflected in variations of the recovered frequency. Since this situation will be impacted by network loading, this needs to be monitored on a constant basis and indication given when stability drops. These indications can then be correlated to changes in network conditions and corrective action can be performed in order to reduce the variability seen by the slave. These OAM&P capabilities need to be included in the solutions as they are provided. A network manager can monitor performance and run statistical collection continuously, or on demand, to investigate network characteristics and performance. Alcatel-Lucent is working together with Chronos and it’s
5620 SAM provides information about the synchronization state of each node
SynchWatch product to deliver an end-to-end synchronization
in the network. It shows PTP peering alarms, Threshold Crossing Alerts, path
management solution as part of the 5620 Service Aware Manager.
information between slaves, boundaries and masters
scenarios. In the case of the newer concept of ToP, there is an even greater requirement for management capabilities to control and monitor the performance. These capabilities start in the clock recovery slaves that run in the edge devices but also apply to the
The Sync management solution provides topological map information about the synchronization tree, either upstream or downstream. It also shows the sync state of each element in the network, whether it is in acquiring, holdover or lock status.
management of the timing masters and the delivery paths
It is also possible to get historical information on synchronization
between the masters and slaves. The implementations within the
paths which allows users to trace sync path changes back to
slaves should be capable of indicating some level of confidence
routing changes in the network.
in the accuracy and stability of the recovered timing and time of day. This will have to be based on the stability of the output of the
The solution also allows sync path audits to be performed to check for packet loss, excessive PDV etc. 81
Chapter 6
Conclusion IP/MPLS has proven to be the right technology to address the challenges of power utilities as they need to upgrade their SONET/SDH networks. This section highlights the most important aspects to take into consideration.
IP/MPLS is the technology of choice for utilities wanting to
examples illustrated in this book show that excellent results can
implement converged Smart Grid networks.
be achieved.
IP/MPLS offers a superior solution to MPLS-TP with no
Synchronization is crucial to enable migration of the TDM
compromises regarding layer two or layer three services,
application onto the IP/MPLS network and to enable next
multicast support, scaling, integrated cybersecurity and end-to-
generation teleprotection and phasor measurement applications.
end capabilities from access, over aggregation, edge and core elements of the network. IP/MPLS is capable to resist to multiple failures, this is of paramount importance for critical infrastructures such as power grids. IP/MPLS is also topology agnostic, it can support ring, star and full mesh topologies. With the right product choices utilities can implement a utilitygrade network that supports the flexibility and scalability of IP, while maintaining the reliability and predictability of traditional mission-critical networks.
Having the ability to combine SyncE and 1588v2 brings the best of both worlds to the network in terms of frequency and phase synchronization. A very interesting observation made during the numerous tests was that teleprotection equipment, designed around Ethernet interfaces performed significantly better than the equipment conceived with legacy interfaces. Because the performance difference is an order of magnitude better, Ethernet based teleprotection equipment allows for new protection algorithms
Over recent years, IP/MPLS networks have demonstrated in numerous power utility networks around the world the ability to support the most critical and latency-sensitive utility applications, including Teleprotection. IP/MPLS has also proven to provide highly secure, reliable and manageable VPN communications services for multiple applications that support utility operations and business objectives, while reducing capital and operational expenses. It is important to consider bandwidth requirements and latency as a consequence of packetization, however the real world 83
and schemes to be adopted. Last but not least, it is important to have access to the right management tools which will make the provisioning of services, element-, alarms- and synchronization management as well as service performance management simple
“Moving a power utility telecom network to IP/MPLS is no longer a technology issue, it has become an HR issue” - Cory Struth - Falling Apple Solutions, Canada
and intuitive. This will in turn contribute to building better, safer and smarter power grids that are future proof and Alcatel-Lucent is here to
Illustration 6.2 Differential protection testing
help make this a reality.
Illustration 6.1 Summary
Creos 200kV Substation.
The Substation WAN & LAN solution offered by Alcatel-Lucent consists of ONE technology, based on IP/MPLS,ONE operating system-SROS, ONE management system -5620 SAM without making compromises. 84
ATM Asynchronous Transfer Mode (ATM) is a standard communications protocol that was designed to carry all types of data, voice and video at high speed. It was defined in the late 1980’s. It uses a fixed packet length of 53 bytes. ATM technology has been widely deployed in enterprise and carrier networks since the early 1990’s however most of these networks have been replaced by IP/MPLS networks as of the early 2000’s.
Related Glossary Terms Drag related terms here
Index
Find Term
C37.94 IEEE standard defined as: An optical interface for use between teleprotection and digital multiplexer equipment that can operate at a data rate of N times 64 kilobit per second where N = 1, 2... 12 is described. Requirements for both physical connection and the communications timing are also included.
Related Glossary Terms Teleprotection
Index
Find Term
CESoPSN Circuit Emulation Service over Packet Switched Networks. This is a standard mechanism defined in RFC5086 to encapsulate data from circuit switched interfaces (n x DS0 in E1/T1) or legacy serial interfaces (RS232, X.21, V.35) into Ethernet packets.
Related Glossary Terms Drag related terms here
Index
Find Term
Circuit breaker Device used in electrical systems to interrupt the flow of electricity. These devices are used to protect electrical power systems from damage caused by faults such as for instance short circuits.
Related Glossary Terms Recloser
Index
Find Term
CWDM Coarse Wave Division Multiplexing is a technology that allows the bandwidth of an optical cable to be increased by using light at different wavelengths (colors) simultaneously. Each color represents an independent channel of 1 or 10Gbps.
Related Glossary Terms Drag related terms here
Index
Find Term
DS0 The smallest entity within the structure of a SONET/SDH network, typically a 64kb timeslot of an E1 for instance.
Related Glossary Terms Drag related terms here
Index
Find Term
DSO Distribution System Operator. This refers to the electric power distribution companies who operate the medium voltage (below 50kV) and low voltage (1kV and less) electrical network. Typically the DSO’s connect the end users to the electrical network.
Related Glossary Terms TSO
Index
Find Term
E&M Analog voice interface used in telephone systems, often to connect older telephone exhanges. E & M stands for Earth & Magneto but is often said to be Ear & Mouth. It can use 2 or 4 electrical wires for the audio plus 2 more wires for the signaling.
Related Glossary Terms Drag related terms here
Index
Find Term
Ethernet Standard layer 2 protocol used for data communications networks. Initially it was designed for Local Area Networks (LAN) at 2, 5 or 10 megabits per second on coaxial cable. In the late 1980’s it was competing with other technologies but these have all disappeared. Today, Ethernet runs on twisted pair and optical fibre at speeds of 1, 10, 40 or 100 gigabit per second. Ethernet has become the standard for enterprise networks and is now becoming a popular Wide Area Network technology.
Related Glossary Terms Drag related terms here
Index
Find Term
G.703 ITU-T standard for transmitting data or voice over T1 or E1 communications circuits. This is typically n x 64k. The physical interface can be a 75Ohm BNC (coax) interface or a 120 Ohm copper interface (RJ48)
Related Glossary Terms Drag related terms here
Index
Find Term
GOOSE Generic Object Oriented Substation Event. This is part of an IEC 61850 defined Generic Substation Event (GSE) model which defines a fast and reliable way of sending substation data to multiple devices. GOOSE relies on multicast and broadcast mechanisms.
Related Glossary Terms Drag related terms here
Index
Find Term
ICT Information & Communications Technology. ICT is often used to refer to the industry of information and telecommunications or the information and communications technology department of a company.
Related Glossary Terms Drag related terms here
Index
Find Term
IDU In-door unit. Typically refers to the equipment that needs to be installed inside a building or a shelter. Often in relation with ODU (out-door unit) as the antenna of a microwave radio system.
Related Glossary Terms ODU
Index
Find Term
IEC International Electrotechnical Commission. Standard conformance organization for electrical, electronic and all related technologies.
Related Glossary Terms Drag related terms here
Index
Find Term
IEC 61850 A standard that defines power substation automation systems design.
Related Glossary Terms Drag related terms here
Index
Find Term
IEEE 1588v2 Precision Timing protocol defined by IEEE in 2009, designed to deliver Time synchronization using packets.
Related Glossary Terms PDV, PTP, SyncE, ToP
Index
Find Term
IETF Internet Engineering Task Force. Standards organization that has defined the IP, and MPLS standards
Related Glossary Terms IP, IP/MPLS
Index
Find Term
IP Internet Protocol (IP) is a connection-less Layer 3 protocol used by routers to relay datagrams or packets over telecommunications networks. It defines the addressing and encapsulation mechanism used to exchange data over router networks. IPv4 is the most widely deployed variant, the more recent IPv6 is gaining ground because of the depletion of public IPv4 addresses. IP is used in combination with TCP (Transmission Control Protocol) which provides the connection oriented context to an IP based communication and will augment the reliability of the IP communication (TCP/IP).
Related Glossary Terms IETF
Index
Find Term
IP/MPLS Internet Protocol/Multi Protocol Label Switching is an IETF standard to allow packet networks to be used on top of existing Ethernet, IP, SONET/SDH and ATM networks and can carry any type of traffic on top of it.
Related Glossary Terms IETF, MPLS, MPLS-TP
Index
Find Term
Jitter Small rapid variations in a waveform resulting from fluctuations in the voltage supply or mechanical sources or other sources. Small irregular movement. In the context of this book jitter refers to the transmission delay variations that can occur due to several telecoms network conditions.
Related Glossary Terms PDV
Index
Find Term
LAN Local Area Network. Most commonly refers to a L2 Ethernet network in a building or campus.
Related Glossary Terms Drag related terms here
Index
Find Term
LSP Label Switched Path This is a function of MPLS and defines the path along which data is sent across multiple hops. The path can be set up in a “strict” manner defining the exact path packets need to follow between two end points. LSP’s can also be set up to be “loose”, which allows another control plane protocol to determine the best route.
Related Glossary Terms MPLS, MPLS-TP
Index
Find Term
MPLS Multi Protocol Label Switching
Related Glossary Terms IP/MPLS, LSP, MPLS-TP
Index
Find Term
MPLS-TP Multi Protocol Label Switching Transport Profile This variant of MPLS is limited to transport only, has no L3 awareness and does not rely on a control protocol to setup Label Switched Paths.
Related Glossary Terms IP/MPLS, LSP, MPLS
Index
Find Term
ODU Out door unit. Part of a system that is installed outdoor, for instance an antenna or radio of a microwave transmission system.
Related Glossary Terms IDU
Index
Find Term
PDH Plesiochronous Digital Hierarchy or PDH is a predecessor of SDH. It is a Time Division Multiplexing technology that runs up to 2 megabits per second and provides up to 32 timeslots of 64 kilobit.
Related Glossary Terms Drag related terms here
Index
Find Term
PDV Packet Delay Variation This refers to the difference in delay that may occur for packets to traverse the network between the same endpoints.
Related Glossary Terms IEEE 1588v2, Jitter, PTP, SyncE, ToP
Index
Find Term
PTP Precision Timing Protocol. Refers to any packet based protocol designed to deliver timing with high precision. Several variants have evolved over the past years and new standards are still in the process of being developed.
Related Glossary Terms IEEE 1588v2, PDV, ToP
Index
Find Term
Recloser A recloser is a circuit breaker which is equipped with a mechanism to close itself automatically after a fault has been cleared.
Related Glossary Terms Circuit breaker
Index
Find Term
ROHS Restriction Of Hazardous Substances. ROHS is a directive of the European Union which was introduced in 2003 and came into effect in 2006. It’s purpose is to restrict the use of a number of chemicals in electric/electronic equipment and appliances of all sorts. The chemical substances that are banned are: Lead (Pb), Cadmium (Cd), Mercury (Hg), Hexavalent Chromium (Cr6+), Polybrominated biphenyl (PBB) and Polybrominated diphenyl ether (PBDE). The latter two were often used as flame retardants in equipment housing and cable insulation. Lead was mostly used in electronic printed circuit boards for soldering components.
Related Glossary Terms Drag related terms here
Index
Find Term
SAToP Structure Agnostic Transport over Packet. This method of sending TDM data across packet networks does not take the structure of SDH/SONET or TDM traffic such as DS0 channels into consideration. It is a fully transparent transport mechanism.
Related Glossary Terms Drag related terms here
Index
Find Term
SCADA Supervisory, Control & Data Acquisition (SCADA) is a term used to define systems that are used to monitor and control industrial processes. It often is based on a master-slave type of communication system to get data from the process and control it in real time.
Related Glossary Terms Drag related terms here
Index
Find Term
SDH Synchronous Digital Hierarchy (SDH) is a standard multiplexing protocol used in telecommunications networks. It is formalized in the ITU standards G.707, G.784 and G. 803. The basic unit of SDH is STM-1 which runs at 155Mb/s, STM-4 runs at 622Mb/S, STM-16 runs at 1.2Gb/S, STM-64 runs at 10Gb/s and STM-256 runs at 40gb/s
Related Glossary Terms SONET, TDM
Index
Find Term
SONET North American variant to SDH.
Related Glossary Terms SDH, TDM
Index
Find Term
Substation Part of the electrical network where voltage is transformed from high voltage to medium voltage, from medium voltage to low voltage or the other way around.
Related Glossary Terms Drag related terms here
Index
Find Term
SyncE Synchronous Ethernet is a standard defined in 2008 and implements L1 synchronization on Ethernet interfaces in a similar way as it is done in SDH/SONET networks. SyncE is to date the best mechanism to provide accurate frequency synchronization.
Related Glossary Terms IEEE 1588v2, PDV
Index
Find Term
TDM Time Division Multiplexing (TDM) is the technology which is used by PDH and SONET/SDH systems. It divides a fixed length frame into a number of equal sized time slots. These timeslots can contain data or voice traffic which are transmitted at fixed speed and constant intervals.
Related Glossary Terms Drag related terms here
Index
Find Term
Teleprotection A system or device designed to protect high voltage power networks from damage caused by faults on the power network.
Related Glossary Terms Drag related terms here
Index
Find Term
ToP Time over Packet Refers to any technology used to deliver timing information over packet networks. IEEE1588v2 is an example of a ToP implementation.
Related Glossary Terms IEEE 1588v2, PDV, PTP
Index
Find Term
TSO Transmission Service Operator. This is a company that operates a high voltage network to carry electricity from the generation to the distribution networks. These are usually operating at 100kV- 400kV or higher. These companies are mostly regulated by a government instance and are reduced to mostly one per country in Europe. The reduction of the number of TSO’s it to prevent network instabilities.
Related Glossary Terms DSO
Index
Find Term
UIC Union International de Chemins de Fer. The International union of Railways organization, headquartered in Paris.
Related Glossary Terms Drag related terms here
Index
Find Term
WAN Wide Area Network. This refers to the network which connects different remote locations together.
Related Glossary Terms Drag related terms here
Index
Find Term
X.21 Serial communications interface standard as defined by the ITU-T in the 1970’s. It typically uses a 15-pin Sub-D connector. It was designed to run at higher speeds than RS232. It is using balanced (pairs) transmit and receive channels which can scale to 10Mbps although it is mostly used between 600bps and 64kbps. X.21 was designed to support synchronous communications.
Related Glossary Terms Drag related terms here
Index
Find Term
