Expert Reer Reerence ence Series Ser ies o White Papers
Troubleshooting Slow Networks with Wireshark
1-800-COURSES
www.globalknowledge.com
Troubleshooti roubleshooting ng Slow Networks with Wireshark Laura Chappell, Founder, Wireshark University and Chappell University
Introduction Your phone begins ringing beore you fnd a suitable spot spo t to put down your frst comorting cup o coee in the morning. Users are complaining that the network is slow slo w – web browsing sessions are painully sluggish and email takes orever to download. They state that they simply can’t work this way. way. The problem appears to be widespread as your coee cools aster than the users’ tempers. A lack o error messages or network alarms makes the problem more elusive and guarantees you’ll be hunting down the problem well through lunchtime – at least. Could the problem be related to the inrastructure devices? Is a rogue switch dropping packets periodically? What about the servers? Could the email server fnally be giving in to the pressure o handling all those email chain letters the users pass amongst themselves? What is the chance that the users’ systems have been compromised with a virus or bot that is spreading stealthily through the shadows o the network like the plague? In this white paper, paper, we examine how to use Wireshark, the world’s world’s most popular open-source network analyzer, to troubleshoot some o the th e top causes o poor network perormance, including • High latency • Packet loss • Inefcient window sizes • Intercepting devices • Application dependencies First, we’ll we’ll look at Wireshark and examine methods used to “see” network communications.
Wireshark: The Open-Source Network Savior Wireshark, ormerly Ethereal, is the world’s world’s most popular open-source network analyzer and the ideal frst-responder tool on a troubled network. Wireshark enables you to “see” the network communications and defnitively point to where the problem lies. Although it cannot tell you why the problem exists, Wireshark Wireshark reduces the troubleshooting time and eort drastically by providing a defnitive answer to the location o the problem – removing the guesswork that typically consumes the IT proessional’s time while users impatiently wait or their network services to be restored. Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
2
A system loaded with Wireshark is connected to the network using one o the methods defned below. Network trafc is captured and decoded by Wireshark’s dissectors, dissectors, predefned code that breaks apart the packets into their felds and feld contents. Wireshark Wireshark also contains an Expert system that identifes possible problems in network communications, thereby shortening the problem isolation process urther. For For more inormation on Wireshark, visit www.wireshark.org. www.wireshark.org.
The Naked Network The frst step in analyzing network perormance is to capture the network trafc. Ideally, Ideally, you’ll capture the trafc to and rom a complaining host system rom a location as close to that user as possible. You want to experience the slow perormance rom their perspective and their location on the network. There are our basic options available to capture network trafc. • Load Wireshark directly on one of the host systems. • Insert a network hub between a host and a switch (half-duplex). • Insert a network tap between a host and a switch (full-duplex). • Span the switch port of a user to an analyzer port.
Loading Wireshark on the User’s System This option makes my skin crawl a bit. I detest the idea o being so invasive and have nightmares imagining the users running Wireshark on their systems with little or no knowledge o network communications. This would be my least-avorite recommendation.
Hubbing Out This is a great option for half-duplex networks. Simply remove the cable from the user’s system and connect it to a hub. With another cable, connect the user’s system and your analyzer to the hub as shown in the diagram below. below. Hubs are stupid – they only know 1s and 0s, and forward all bits down all active ports. All trafc to or rom your user’s system will be copied to your analyzer as well.
Tapping Out Hubs work great on half-duplex networks, but most of us have migrated to full-duplex networks. Hubs can’t handle these ull duplex communications; this is the job or a ull-duplex tap. The connection process would be the same as shown in Figure 1, provided you have an aggregating full-duplex tap. An aggregating tap combines both transmit and receive channel inormation between the user and the switch into a single data stream to the analyzer system.
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
3
Figure 1: Use ull-duplex taps to listen in on all trafc to and rom the user’s system on a ull-duplex network.
Spanning Spanning requires reconguration of the switch that the user’s system connects to. A switch that is congured with a spanned port sends a copy o all trafc to/rom that spanned port down another port – the port that the analyzer is connected to. This method o tapping-in is ideal or listening to trafc to/rom a server as you are unlikely to break the server’s network connection to install a hub or tap.
High Latency: Latenc y: Somebody’ Somebody’ss Dragging Dragg ing their the ir Feet Latency is a measurement of travel time from one host to another or the roundtrip between hosts. Although packets on a 100Mbps network always travel at a rate of 100Mbps, latency is introduced by distance and inter connecting devices that process packets. packets. Slow travel from one endpoint to another is dened as high latency. latency. High latency has a tremendously negative eect on network communications. As an example, in Figure 2, we examine the roundtrip time o a fle download process on a high-latency path. At times, the roundtrip latency time reaches 1 second, which is completely unacceptable.
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
4
Figure 2: Use Wireshark’s Statistics > TCP Stream Graph > Round Trip Time Time Graph to determine the current roundtrip latency or a fle download.
We use Wireshark to determine the roundtrip time on a path to determine i this is the reason or poor network performance for Transmission Transmission Control Protocol (TCP) communications. TCP is used for web browsing, email receipt and transmission, fle transer protocol, and many other popular applications. In many situations, especially when hosts are using Windows XP, XP, the operating system can be adjusted to work more efciently on high-latency paths.
Packet Pack et Loss: Losing Data Dat a in Bits and Pieces Packet Packet loss is one of the most common problems I see on networks. When a user accesses a web site and begins to download the elements of the site, lost packets packets trigger retransmissions, increasing the overhead required to download the site elements and delaying the total download process. In addition, when an application uses TCP, TCP, the effect of lost packets is especially detrimental. Each time a TCP connection senses a lost packet, the throughput rate automatically throttles back dramatically to account or Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
5
network problems. Slowly Slowly,, it recovers to a more acceptable rate until the next packet is lost again, causing a drastic cut-back in data throughput. Packet Packet loss has a tremendously negative effect on large le downloads that should otherwise stream across a network smoothly. smoothly. What does packet loss look like? It depends. If the application is running over TCP, TCP, packet loss has two different looks. In one case, case, the receiver tracks packets based on their sequence numbers and notices a packet is missing. The client requests the missing packet three times (duplicate acknowledgments) which triggers a retransmission. I the sender times-out when it notices the receiver has not n ot acknowledged receipt o a data packet, the sender retransmits the data packet. In Figure 3, Wireshark indicates that packet loss has occurred and duplicate acknowledgments trigger the retransmission. A high number o duplicate acknowledgments indicates that a network has experienced packet loss and is also acing high latency. latency.
Figure 3: Wireshark Wireshark indicates that packet loss has occurred by color coding the problematic trafc.
Locating the exact point of packet loss is imperative in improving network performance. When packet loss is experienced, we move the Wireshark along the path until we can no longer see packet loss. At this point, we are “upstream” rom the packet loss point, and we know where to concentrate our troubleshooting eorts.
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
6
Inefcient Window Window Sizes: Peer Peering ing Through Through Small Windows Windows There are several “windows” in TCP/IP networking besides the Microsoft operating system. • Sliding window • Receiver window • Congestion window As a set, these windows dene the TCP-based communication performance on the network. First, let’s dene each o these windows and their individual eect on network throughput.
The Sliding Window Window The sliding window is a process used to send the next TCP segments onto the network as data is acknowledged. The sliding window expands as the sender receives acknowledgments or previously transmitted segments. Larger amounts of data can be sent on the network as long as there are no dropped packets on the network. When packet loss occurs, the sliding window shrinks under the assumption that the network cannot handle the larger amount o data on the wire.
The Receiver Window (rwin) The receiver window is a buffer space in the TCP stack. When data is received, it is held in this buffer space until an application picks up the data. When an application does not keep up with the receive rate, the receiver window flls and may eventually lead to a “zero window” condition. When a receiver advertises a zero window condition, all data transmission to the host must stop. Throughput drops to zero. A process called Window Scal ing (RFC 1323) enables a host to scale the receiver window to a larger value and reduce the chance of a zero window condition occurring. Figure 4, below, below, depicts a zero window condition that caused a 32-second delay in network communications.
The Congestion Window (cwnd) The congestion window defnes the actual amount o data that the network can support. It is defned by the sender’s packet transmission rate, the network packet loss rate, and the receiver’s window size. During a healthy network communication, the congestion window increases consistently until the transer completes or it hits a “ceiling” defned by the network health, the sender’s transmit capabilities, or the receiver’s window size. Each new connection begins the process o window size negotiation anew.
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
7
Figure 4: It took over 32 seconds to resolve the zero window condition, denoted by Wireshark’s Expert system.
Interceptin Inte rcepting g Devices: Devices: When Network Trafc Cops Go Bad Bad “Network Trafc Cops” are interconnecting devices, such as switches, routers, and frewalls that make orwarding decisions. When packet loss occurs, it is best to look at these devices as the possible cause. These interconnecting devices can also add latency to the path. For example, i trafc prioritization is in use, we can see additional latency injected into a stream that meets a low priority level.
Application Dependencies: Choose Your Application Fr Friends iends Careully Some applications have dependencies on other applications, processes or communications with other hosts. For example, i your database application relies on connecting to numerous servers to pull database elements, slow perormance to those other servers can aect the local application’s load time. As a simple example, we can look at a web browsing session in which the target server reerences numerous other websites. In order to load the main page of the site, www.espn.com, www.espn.com, for example, you must access 16 hosts that provide advertisements and content or the main www.espn.com page. page. Figure 5 shows the list o hosts that you must contact when you load the www.espn.com home page. Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
8
Figure 5: Wireshark’s Wireshark’s HTTP Load Distribution window lists all servers reerenced by the www.espn.com home page.
In addition, poorly-written applications can aect the perormance on both the sending side and the receiving side. No matter how healthy and ree o dropped packets the network is, an application may not take advantage o the network’s capabilities, because it has its own throttling mechanisms limiting the amount o data that it sends. On the receiving side o the connection, an application that does not pull data out o the receive buer in a timely manner can lead to a limited or zero window condition. In the case o poorly perorming applications, consider researching the possibility that the application can be tuned or better perormance.
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
9
First Steps to a Healthier Network Most networks can be tuned for better performance through a series of tasks. • Learn to use Wireshark as a rst-responder task to identify the cause of poor performance quickly and efciently. • Determine the source of latency on o n the network path and, if possible, reduce the latency to an acceptable value. • Identify the location of packet loss and resolve the problem. • Examine the window size in use on data transfers and consider optimizing the receiver window size, if possible. • Examine the performance of intercepting devices to determine if they are adding latency or dropping packets. • Optimize applications to send larger amounts of data and pull data out of the receiver window, if possible. We’ve discussed the primary causes o network perormance problems, but one cause – lack o insight into network communications behavior – cannot be overlooked. After 20 years of analyzing network trafc and teaching trafc interpretation and problem resolution, it is clear that network analysis is a skill that every IT proessional should possess. Wireshark oers an insight into networks in a similar way that X-rays and CAT scans oer an insight into the human body or accurate and timely diagnoses. And, just like like those indispensable technologies in the medical feld, Wireshark has become an essential tool to locate and diagnose the cause o network problems in the most efcient and cost-eective method possible. po ssible. Note: This white paper was developed as a ollow-up to the Global Knowledge webinar by the same name. Visit our Knowledge Center at www.globalknowledge.c www.globalknowledge.com/knowledgecenter om/knowledgecenter to view the related webinar.
Learn More Learn more about how you can improve productivity, productivity, enhance efciency, efciency, and sharpen your competitive edge. Check out the ollowing Global Knowledge courses: Analyzing TCP/IP Networks with Wireshark Troubleshooting and Securing TCP/IP Networks with Wireshark TCP/IP Networking For more inormation or o r to register, register, visit www.globalknowledge.com or call 1-800-COURSES to speak with a sales representative. Our courses and enhanced, hands-on labs oer practical skills and tips that you can immediately put to use. Our expert instructors draw upon their experiences to help you understand key concepts and how to apply them to
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
10
your specic work situation. Choose from our more than 700 courses, delivered through Classrooms, e-Learning, and On-site sessions, to meet your IT and management training needs.
About the t he Author Laura Chappell is the Founder of Wireshark University and Chappell University, University, and has been researching, writ ing, and lecturing on network analysis for over 20 years. Laura designed and created Wireshark University in cooperation with Gerald Combs, the creator of Ethereal/Wireshark. In March 2009, the Wireshark Certication Program was released to validate a candidate’s knowledge of Wireshark functionality, TCP/IP troubleshooting, and network orensics/security. orensics/security. Wireshark University (www.wiresharkU (www.wiresharkU.com) .com) Chappell University (www.chappellU (www.chappellU.com) .com)
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
11
