Why BitTorrent causes so much jitter (high ping) and how to fix it June 1st, 2008 George Ou Leave a comment Go to comments Anyone VoIP or online gamer who has a roommate or a family member memb er who uses BitTorrent (or any P2P application) knows what a nightmare it is when BitTorrent is in use. The ping (round trip latency/jitter) goes through the roof and it stays there mak ing VoIP packets drop out and an d game play impossible. I’ve personally experienced this and man y of my friends have experienced it. When I had a roommate in 1999 and I had my first broadband connection, my roommate started using the hot P2P (peer-to-peer) application of its day called Napster. As soon as he started sharing, my online game would become slow as molasses and I’d experience a huge 500ms second spike in round-trip latency and any gamer knows that pings over 100 are problematic. When it’s just myself using BitTorrent, I can pause it while I make Vo IP phone calls or when I want to game. But when it’s time to ask your family member or roommate to stop using their application, it becomes awkward because why should they stop using their application just so you can use your application? My roommate paid for half the broadband fees so I couldn’t always tell him to stop and he couldn’t always alwa ys ruin my latency making it a contentious situation for both of us. What makes it even more frustrating is t hat there’s plenty of bandwidth capacity to theoretically satisfy both our needs, but something about these P2P applications make them them very bad housemates.
I decided to do some research on o n this and ran a series of o f tests which resulted in some very interesting data. I fired up various applications at var ying rates of bandwidth utilization in the upstream and downstream direction and I measured ping to my first router hop (ISP router) beyond my home router. I ran continuous pings to the ISP router to check the round-trip latency above the normal baseline latency latenc y of 12 ms and plotted out the results. This test methodology measures network jitter (the variation of delay) on the ―last mile‖ which is what I’m primarily concerned about because that’s where most of the damage is done by local usage of P2P applications. BitTorrent downloads cause huge amounts of latency
The first set of tests I did were download tests. First I downloaded a file using HTTP which ran at 260 KB/sec (2.08 Mbps) which is roughly 87% of my peak download performance. I then set BitTorrent to also also download at 260 KB/sec to compare the effect on my ping times. To my surprise, there was a significant difference in the amount of ping increase between HTTP downloading and BitTorrent downloading despite d espite the fact that both applications were downloading at the same average rate.
When you look at the two graphs immediately above, you see that BitTorrent causes an average increase in ping of more than 117 ms (milliseconds). Note that those 6 ping spikes in the graph were actually bad enough that they caused the ping to time out which means the ping was higher than one second. HTTP on the other hand didn’t increase the ping nearly as much but it still caused an average of 46.7 ms increase in ping and it only peaked at 76 ms. So how is this possible when both applications are using the same amount of bandwidth? I did some thinking and this is the hypothesis I came up with. Burst traffic fills up transmit queues
When you look at the two graphs immediately above, you see that BitTorrent causes an average increase in ping of more than 117 ms (milliseconds). Note that those 6 ping spikes in the graph were actually bad enough that they caused the ping to time out which means the ping was higher than one second. HTTP on the other hand didn’t increase the ping nearly as much but it still caused an average of 46.7 ms increase in ping and it only peaked at 76 ms. So how is this possible when both applications are using the same amount of bandwidth? I did some thinking and this is the hypothesis I came up with. Burst traffic fills up transmit queues
The difference in ping time is caused by the fact that HTTP is a single source of data whereas BitTorrent is about 20 sources of data. Incoming data d ata from from multiple sources via the fast core of the Internet can sometimes clump closely together when multiple sources happen to transmit data around the same time. So in the period of 20 ms (the interval between typical VoIP packets), up to 200 max-size 1472 BYTE packets can occasionally build up in the DSLAM transmit queue (blue square in illustration above labeled as ―download queue‖) causing my ping requests to time out above the 1 second mark. But on average, we get around 23 of these packets causing sitting in the DSLAM transmit queue causing an average increase in ping of 117 ms. When there is a single transmitter, it might burst and clump pac kets close together but it won’t be at the level of 20 transmitters. With HTTP causing 76 ms of downstream delay, dela y, that means 15 of these 1472-BYTE 1472 -BYTE packets are sitting in the DSLAM transmit queue causing a less extreme increase in ping. This is still problematic for VoIP communications and it can certainly ruin online gaming. So despite the fact that there is plenty of remaining bandwidth for my VoIP or online gaming traffic, it’s the non-uniformity of the incoming Internet traffic that causes my VoIP phone calls and games to perform badly. Unfortunately since this is the downstream we’re talking about, the consumer can’t do much about it on their own end of the pipe because the delay is at the DSLAM which belongs to the ISP. The only way to fix this problem is for the ISP to implement intelligent packet scheduling and application prioritization at the DSLAM to re-order those VoIP or gaming packets to the front of the transmit queue. With packet prioritization (generally referred to as QoS), your family member, your roommate, or your own video downloads won’t need to be stopped and they won’t interfere with your VoIP or gaming applications which makes everyone happy. Unfortunately, these types of QoS services may never see the light of day if poorly conceived Net Neutrality legislation gets passed that ban the sale of packet prioritization. BitTorrent uploads cause excessive jitter
BitTorrent or P2P uploads also cause a lot of upstream jitter. I compared various types of uplo ad traffic patterns to see what kind of increase in the upstream ping times would result. First I tried running BitTorrent with a 47 KB/sec (376 Kbps) bandwidth cap which was about 90% of my upload capacity, then I ran BitTorrent with a 28 KB/sec (224 Kbps) bandwidth cap at 54% of my upload capacity, and then I ran BitTorrent with a 10 KB/sec cap at 19% of my upload capacity.
In either the 28 KB/sec test or 10 KB/sec test, I’m not being greedy by hogging all the available upstream bandwidth and I’m leaving more than the 11 KB/sec of bandwidth needed for VoIP and online gaming. Yet I found that the additional ping caused by BitTorrent uploads (seeding) was unacceptable for gaming and problematic for VoIP applications. Even when I severely limited upload throughput to 10 KB/sec, it didn’t reduce the ping time spikes although it reduced the frequency of those spikes. However, even fewer spikes in ping time can pose the same problems for VoIP applications because they have to adjust their buffers to account for the
worst-case network conditions. This would seem to indicate that BitTorrent is bu rsting packets rather than releasing the packets in a uniform and evenly spaced stream. Next I tried using FTP at full throttle and I managed to get an FTP session going at 47 KB/sec (90% of my peak load) yet the jitter caused by FTP at this extreme rate of throughput was less than the jitter caused by operating BitTorrent at an average of 10 KB/sec. This would seem to indicate that FTP is outputting data in a more consistent manner that BitTorrent.
Lastly, I ran some ping tests during some VoIP phone calls using the Lingo service (competitor of Vonage). I had set Lingo to use the uncompressed G.711 which uses 11 KB/sec for both upload and download which makes it very comparable to BitTorrent uploading at an average of 10 KB/sec. But as soon as I ran the ping tests, I was shocked to see virtually no increase in ping times. I realized that this is because the VoIP ATA (Analog Telephony Adapter) device pulses small packets at exact intervals of 20 milliseconds at 50 times a second.
Smoothing out the packet spacing to reduce jitter
After running these tests, I am beginning to conclude that it isn’t so much the amount of data that causes excessive jitter; it’s the uniformity of data transmission. If the transmissions are spaced evenly, other packets from other applications can slip in between the packets rather than getting stuck behind multiple packets in the transmit queue. So would it be possible to engineer BitTorrent to transmit data uniformly and what would be the effect? I came up with the following chart to illustrate what this could mean for peaceful co existence between VoIP/Gaming and BitTorrent/P2P.
I calculated that the max-size 1472 BYTE packet takes 28.3 milliseconds to transmit over my 52,000 BYTE/sec broadband uplink. If BitTorrent bursts 3 packets upstream over a 100 Mbps FastEthernet LAN connection, they will all sit in the upload queue of my DSL modem for 85 ms. Meanwhile, my VoIP or gaming packets get stuck behind those three packets for an additional 57 ms in the queue. This is shown in the first example in the illustration below with large 1472BYTE red packets representing BitTorrent and small 223-BYTE green packets representing VoIP.
In the second example, I show what happens if BitTorrent simply spaced their transmissions evenly. It’s still less than ideal but at least it significantly reduces the jitter for VoIP or gaming packets. In the third example, I show the ideal scenario where BitTorrent would reduce its packet size to 815 BYTES or less and pulse them out in 20 ms intervals at 50 packets per second. BitTorrent could essentially create a ―VoIP friendly mode‖ that allows VoIP packets to fit cleanl y between the BitTorrent packets and the increase in jitter would be no greater than 15.7 ms and would typically average around 8 ms increase. BitTorrent could also h ave a ―Game friendly mode‖ that uses 679-BYTE packets at 60 packets per second. Now it is possible to solve this problem on the network level by prioritizing VoIP and gaming packets in the home DSL modem upload queue. Unfortunately, I don’t have administrative access to the modem and implementing VoIP or gaming prioritization on my home router
seemed to have no effect. Packets in the home router get forwarded as soon as they arrive with 100 Mbps Ethernet on both ends so there is nothing to reorder in the queue. More advanced business-class routers like those from Cisco will allow you to configure the speed of the FastEthernet connection to match your DSL throughput so that the queue will migrate from the DSL modem to the router but this isn’t very practical for most people. So it would make sense for application writers to try and make their application work as well as possible on the majority of home networks and broadband networks without QoS. While modifying BitTorrent or the P2P application may not significantly fix the downstream problem, it would definitely fix the upstream jitter problem which means th at people will be more willing to seed and contribute to the health of BitTorrent. The download jitter may improve if the dozens of peers sending information to me did it in a more uniform manner, but it is still possible that those peers will transmit at the same time which still causes packet clumpin g and bursting. So why would BitTorrent or any P2P application vendor want to do this? Why couldn’t they just say ―it’s not my problem‖? Because it would fix their reputation as a bad housemate and more people would be willing to seed if it didn’t interfere with their other applications lik e VoIP or gaming. Corporate IT departments may also ease their ban on the technology if it doesn’t trash their Internet connection for the entire office. I am certainly a huge proponent of network-based solutions for dealing with jitter problems, but improvements in the application can act to further reduce network jitter even in the presence of existing network-based solutions. Furthermore, the vast majority of consumers do not have access to network-based solutions and it only makes sense for a P2P application vendor to cater to this market. I will try to reach out to BitTorrent corporation to see if there is any interest in this and perhaps someone would be interested in modifying open source Azureus. It would be a very good feature to have or it would be a very interesting academic experiment at the very least. 0diggsdigg Categories: Networking, P2P, Policy Tags: Comments (147) Trackbacks (9) Leave a comment Trackback
1. George Ou June 3rd, 2008 at 07:11 | #1 Reply | Quote I did use DD-WRT with a Buffalo router and the QoS latency gains were questionable. I prioritized the ports for Team Fortress 2 (same as Counter Strike) and for ICMP (for ping testing) and it didn’t do anything for the ping tests and the game play still sucked.
2. website design June 3rd, 2008 at 08:52 | #2 Reply | Quote I think the best program is miu torrent regards: http://ooyes.net
3. The 8472 June 3rd, 2008 at 11:10 | #3 Reply | Quote I must say that your analysis is flawed at best, politically motivated at worst. First of all: TCP is stream-based from the application perspective, while you can control packet scheduling and sizing to a certain extent by disabling nagle’s algorithm that’s an ugly hack. TCP is a stream and it’s the responsibility of the whole network stack and the routing equipment to take care of fair scheduling. Technically speaking it’s a no-brainer to give priority to outgoing, small-sized packets, either at the OS or at the router level. You shouldn’t blame bittorrent for it just because OS or router manufacturers don’t implement it. Decreasing packet sizes for a bulk transfer is also horrible from a network engineering perspective, you get the maximum efficiency out of a latency-insensitive bulk transfer if the packet size is maximized. That’s the one point of TCP … it auto-tunes itself to the MTU. And your take on network neutrality is also wrong, most aims would not prevent QoS (which we’re talking about here) but only discriminating equal services based on origin, destination or protocol being used. E.g. a comcast video store vs. a bittorrent video store. Or vonage VOIP vs. Skype. QoS to prioritize vonage AND skype over the bittorrent/comcast video store would still be allowed.
4.
George Ou June 3rd, 2008 at 14:02 | #4 Reply | Quote 8472 says: "First of all: TCP is stream-based from the application perspective, while you can control packet scheduling and sizing to a certain extent by disabling nagle’s algorithm that’s an ugly hack." So you admit that it is possible for the application to trickle data to the TCP stack so you are basically admitting that I am technically correct. I’m not so much "blaming BitTorrent"; I’m saying there’s an opportunity to tweak the application in such a way that it could address the vast majority of consumers who don’t have a network- based solution in place and who aren’t network engineers. For the longest time in the VoIP world, VoIP vendors expected the network to be re-engineered to support Voice. Skype came along and made the application support the network and made it "just work" and it took off like wildfire. So perhaps there is a lesson to be learned here.
The "too many packets" argument is overblown. Most people have the wrong MTU set for their DSL PPPoE broadband service; it’s the default 1500 BYTE instead of the optimized 1492. That means the packets are being fragmented anyways so the 815 BYTE packets isn’t so bad. Furthermore, the number 815-BYTE is only needed for slower uplink connections and you’ll be able to use optimized 1464-BYTE packets at 50 PPS if your uplink is faster. You’re so busy criticizing me for my "political motivations" that you don’t even bother to read the legislation. I’ve cut-pasted the exact language of the regulation in a comment a bove and I’ll paste it here again for you. It absolutely does ban the SALE of QoS services. So if you offer it to one customer you have to offer it to every customer which is the forced bundling of services. That means those who don’t need QoS would be for ced to subsidize those who do. Either that or the ISP simply decides that not enough people want it and they give it to no one.
Markey proposal from 2006: SECTION 201. NETWORK NEUTRALITY. (b) IN GENERAL.-Each broadband network provider has the duty(3) if the provider prioritizes or offers enhanced quality of service to data of a particular type, to prioritize or offer enhanced quality of service to all data of that type (regardless of the origin of such data) without imposing a surcharge or other consideration for such prioritization or enhanced quality of service; Snowe-Dorgan proposal from 2006: (5) only prioritize content, applications, or services accessed b y a user that is made
available via the Internet within the network of such broadband service provider based on the type of content, applications, or services and the level of service purchased by the user, without charge for such prioritization; Conyers-Lofgren proposal from 2008: "If a broadband network provider prioritizes or offers enhanced quality of service to data of a particular type, it must prioritize or offer enhanced qualit y of service to all data of that type (regardless of the origin or ownership of such data) without imposing a surcharge or other consideration for such prioritization or enhanced quality of service." So what you have shown is that I am technically correct and that at best, your own politics and ignorance blinds you to the truth about the legislation and at worst, you’re trying to mislead the public on what Net Neutrality legislation is. George Ou
5. The 8472 June 3rd, 2008 at 15:41 | #5 Reply | Quote > So you admit that it is possible for the application to trickle data to the TCP stack so you are basically admitting that I am technically c orrect. yes, absolutely. To quote RFC 1925: (3) With sufficient thrust, pigs fly just fine. However, this is not necessarily a good idea. It is hard to be sure where they are going to land, and it could be dangerous sitting under them as they fly overhead. > "If a broadband network provider prioritizes or offers enhanced quality of service to data of a particular type, it must prioritize or offer enhanced quality of service to all data of that type (regardless of the origin or ownership of such data) without imposing a surcharge or other consideration for such prioritization or enhanced qu ality of service." exactly my point. The ISP could give "enhance quality of service" to ALL voip traffic, "regardless of origin or ownership". Which is exactly what ever yone wants. Schedule small voip packets first, then let the bulk transfers through (such as VOD or bittorrent). This is exactly the kind of network neutrality that d oes not interfere with QoS.
6.
George Ou June 3rd, 2008 at 16:17 | #6 Reply | Quote I’m not sure what your objection is, we have cheap $50 multi-core processors today that would hardly notice the extra load of trickling data periodically to the TCP stack. If an application can be made to be latency-friendly in the absence of other network-level or TCP stack-level fixes, which is the vast majority of customers, it only makes sense to cater to that market. Your thinking is too ―in-the- box‖ and this kind of narrow minded thinking that prevented PC telephony from taking off until Skype came along and made their software just work. What consumers want is a P2P client that ―just works‖ and doesn’t require them to buy QoS capable hardware or require them to be network engineers. I use to build large enterprise networks for a living and I can configure QoS all day long and even I don’t want to mess with this stuff if I don’t have to.
If these bans on QoS charges are turned in to law, then customers who never use VoIP services because they prefer traditional land line services will be forced to subsidize QoS services for those who do use VoIP. Then what if the ISP decides that it isn’t worth doing if they can’t charge for it? Then you don’t get QoS even if you’re willing to pay for it. I would love to get free massages too, but the last thing I want is the Government to mandate free massages because that’s a sure way to effectively ban massages. What you’re saying is that because you want free QoS, you want the Government to mandate "free" QoS which is effectively bundled QoS or no QoS. What I’m saying is that this kind of thinking is short sighted, selfish, and ultimately foolish. Then what about IPTV service? You realize that IPTV service has to dynamically carve out a portion of an FTTN broadband pipe with low latency properties right? Are you suggesting that if a DSL provider like AT&T prioritizes their own U -Verse video, then they must prioritize all other video streams because they happen to use the same application? Do you not understand the unintended consequences of this? I’ll give you a hint. You go passing these short sighted laws and you’ll see DSL providers switch to HARD allocations of bandwidth for their IPTV service. So even if you don’t want IPTV service from your DSL provider, you’re not getting that bandwidth back. Now what do you do smart guy?
7. The 8472 June 3rd, 2008 at 19:34 | #7 Reply | Quote
This has nothing to do with subsidizing or forcing the use of QoS. ISPs just take the wrong approach right now. Instead of fairly throttling all bulk protocols to ease congestion (and thus make VoiP usable for other customers) they just throttle protocols that do not represent their interests. The point is that all this is anti-competitive as you can see with bell canada which rolls out its own video store but shapes bitto rrent inc.’s and vuze’s video store. So, assuming that the overselling business model won’t go awa y one form of shaping or another is necessary anyway and mandatory (as the user can’t choose). Net neutrality is only about ensuring fairness of this shaping. To make it a proper QoS instead of Quality of Business. They alraedy have QoS devices… what do you think are ellacoya applicances doing? So it’s not a matter of investment but a matter of policy/configuration.
Regarding IPTV. IPTV does not require low latencies, it only requires bandwidth guarantees. And under net neutrality laws ISPs might be required to wholesale IPTVallocations to other IPTV providers if those other providers wish to offer similar services over the internet. But unbundling is never a bad thing as you can see here in europe. Anyway, we’re digressing… this is about subpar packet scheduling. I have initially stated that it’s "possible to a certain extent" and "a hack". You make that into a "possible" and turn that one into a "should be done". No, it should not be done, because TCP applications are bandwidth-latency-product- and jitter-agnostic. Doing any "tweaks" on the application lev el would be detrimental to performance _at least_ under some edge cases, most likely under many more conditions.
If you have an issue then ask your OS manufacturer or router manufacturer why their network stacks don’t do what a network stack should do. The optimal and feasible solution lies in properly applied QoS-implementations. You demand an automagic solution from p2p-developers… why should this be any easier for them to implement – despite having less knowledge what’s cur rently going on on the network than the OS stack or the router – than the OS/router manufacturer? I suggest you aim your demands at them instead… i.e. demand a default-on QoS implementation that gives priority to latency-sensitive services… and hey, windows even claims to have a QoS-driver installed by default, maybe the VoiP application would just have to request higher- priority scheduling and doesn’t… so it might even be the fault of the VoiP application. Point is that you’re looking for solution in the wrong place.
8. Erick Art.
June 3rd, 2008 at 19:34 | #8 Reply | Quote I did use DD-WRT with a Buffalo router and the QoS latency gains were questionable. I prioritized the ports for Team Fortress 2 (same as Counter Strike) and for ICMP (for ping testing) and it didn’t do anything for the ping tests and the game play still sucked. The one thing that is CRITIAL to do is enable QoS for both LAN and WAN and prioritize them to the "premium" option. Also make sure your DD-WRT version is up to date some of the older one had bugs Try using this firmware ver. this is what I am running: DD-WRT v23 SP2 (09/15/06) std Also best thing to test is http traffic latency. If bittorrent slows down your http traffic enable QoS for http and test. it dramatically changed my experience with websurfing w/ bittorrent running. (another thing i should mention is that I am using rtorrent as my bittorrent client)
9. Craysh June 3rd, 2008 at 19:35 | #9 Reply | Quote @host There’s a difference between QoS and Tiered internet. If you’re non-discriminitive, QoS will NOT be banned. However, what the ISP’s are doing now IS discriminitive. When they have competition like Skype or vonage, and a phone com
10. AaronCompNetSys June 3rd, 2008 at 20:35 | #10 Reply | Quote Internal network QoS: Tomato http://www.polarcloud.com/tomato Works amazingly well for my sad DSL. I primarily use Xbox Live while Bittorrent-ing.
Its not magic, but it works as advertised, and once a particular connection is established is prioritized, no worries.
11. George Ou June 3rd, 2008 at 20:39 | #11 Reply | Quote You’re dodging the question on why you would prohibit the sale of QoS. Now answer the question. Why do you want the Government to mandate free QoS or no QoS for Broadband Internet? You’re changing the subject to an irrelevant topic. Vuze isn’t a "service" that delivers packets; Vuze offloads their bandwidth costs to the ISP using P2P and has absolutely no comparison to a CDN Video on Demand model. Throttling P2P application is strictly a fairness issue because P2P uses the multi-stream cheat to get a disproportionate amount of bandwidth. Jaco bson’s algorithm is being thoroughly exploited by P2P. http://blogs.zdnet.com/Ou/?p=1078. It’s also quite ignorant of you to believe that IPTV d oesn’t require low latency. You’re confusing it with Video on Demand which is buffered enough to not need low latency and only flow rate assurance. IPTV is a live streaming service with very low buffering primarily because consumers want to channel-surf. You cannot have a half-second delay any time you flip the channel because it wouldn’t work the way people expect it to when they compare it to Cable or Satellite TV service. You also need the low latencies if you want the audio to sync up perfectly with the video.
So you must absolutely prioritize IPTV service with minimal ban dwidth guarantees and low latency guarantees if you want a functional TV service competitive with Cable and Satellite. This is being prioritized at the expense of all other Internet traffic but you have to do this or else the system doesn’t work. If you give other Internet-based video streaming services (which are typically VoD services that don’t need low latency) equal priority as the Net Neutrality bills require, you kill the IPTV service. Because the DSL provider can’t afford to lose the IPTV business, they will simply be forced to use a fixed slice of bandwidth from their FTTN service and never make that available to you or any other content provider and this would be a horrible unintended consequence of these rules. 8472 says: "I suggest you aim your demands at them instead… i.e. demand a default-on QoS implementation that gives priority to latency-sensitive services… and hey, windows even claims to have a QoS-driver installed by default, maybe the VoiP application would just have to request higher- priority scheduling and doesn’t… so it might even be the fault of the VoiP application."
Again, you want the Government to mandate "free" QoS which makes about as much sense as mandating "free" car wash service with gas. S ure you can demand it, but you’re just mandating bundled service which ironically the advocates claim they’re against. As for the QoS-driver in Windows, I would suggest you d o a little more research about that before you embarrass yourself further. George Ou
12. Craysh June 3rd, 2008 at 21:45 | #12 Reply | Quote @George Ou What are you talking about? They’re not "offloading" their bandwidth costs, they pay (exorbitantly) for their bandwidth to their ISP. These Internet companies pay much more than the average web operator because the
13. George Ou June 3rd, 2008 at 21:52 | #13 Reply | Quote Craysh says: "What are you talking about? They’re not "offloading" their bandwidth costs, they pay (exorbitantly) for their bandwidth to their ISP."
Google (YouTube), Apple, Microsoft, Netflix etc pay a ton of money on bandwidth and CDN costs. They pay their own end of the pipe and the internconnect fees are already covered by the peering arrangements between the carriers. These companies for example are not offloading their costs and they really can’t use P2 P for their distribution model because they need in-order delivery so that the user can watch the movie as they are downloading it. Vuze uses the P2P distribution model. They pay just enough to seed the content but the vast majority of the distribution costs are pushed out to the ISPs and their customers. Craysh says: "If the ISP’s offers "Unlimited" internet at X speed they need to deliver that."
ISPs don’t offer unlimited. Comcast for example does not advertise unlimited. No ISP offers unlimited or guaranteed throughput. All broadband conn ections are shared.
14. The 8472 June 3rd, 2008 at 23:38 | #14 Reply | Quote > Because the DSL provider can’t afford to lose the IPTV business err… are we talking about a DSL provider or an IPTV provider? Because if you’re saying that the DSL provider and IPTV provider are not only the same entity but also are inseparable you obviously get a conflict of interest and an anti-competitive market as all IPTV competitors would have a disadvantage compared to the DSL/IPTV conglomerate.
So i suggest you make a clear distinction between access providers and content providers, otherwise it would like you’re advocating lock -in schemes and mono-/duopolies. > ISPs don’t offer unlimited. maybe not in their fineprint, but in their adverts. And most people expect what they’ve been promised and not what some weasels sneak into their contract…. most people don’t even read it. So it’s a marketing-failure among other things. People have been fed with big promises, now that they actually use what they have been promised the ISPs reap the fallout. Deceptive advertising is hardly something one has to d efend. > You’re dodging the question on why you would prohibit the sale of QoS
i never said i would prohibit the sale of QoS. QoS can be sold to customers, as long as it is applied equally to all services of the same type. E.g. you could sell them a "speed up VoIP and games"-package. Which would apply to all VoIP services (as far as technically possible) and not just prefer a specific vendor. We have something similar in germany (although not techn ically related). DSL can operated in interleaved or fastpath mode. The latter has (theoretically) higher error rates but better RTTs, hence it’s sold as an additional option and frequently used by gamers. This option applies to all services equally, and yet it can be sold. > You also need the low latencies if you want the audio to sync up perfectly with the video.
not really, as long as audio and video frames that belong together are in the same buffer window it does not matter which one arrives earlier, it’s only important that you have both of them. > You cannot have a half-second delay any time you flip the channel because it wouldn’t work the way people expect it to when they compare it to Cable or Satellite TV service. people actually prefer VoD over TV, so i’d say they can live with "slow" channel f lipping if you replace it with a low-res preview feature for ex ample. Not to mention that nearguarantees for latency of 250ms is feasible with good QoS solutions.
> So you must absolutely prioritize IPTV service [...] if you want a functional TV service competitive with Cable and Satellite. oh really? is that the only mode of competition? With IPTV you have the advantage that only that what is currently watched must be delivered. With cable and satellite you have to deliver all at once through a limited resource (frequencies). Thus you can offer a much broader choise with IPTV, i’d say this is a quite neat m ode of competition. > Vuze offloads their bandwidth costs to the ISP using P2P and has absolutely no comparison to a CDN Video on Demand model. The users serve as your CDN, that’s the only difference. Regarding your claim that p2p applications cheat… this one is actually simple to solve with QoS and Diffserv. Simply stuff P2P traffic into a traffic category below bulk and let them compete for the remaining bandwidth, i.e. that’s what left after allocating low-latency, best-effort and reliability traffic. The TOS/Diffserv value for that is 0×20 iirc. And your claim that what ISPs doing right now is about fairness is bogus. If it would be about fairness ISPs could still allocate all remaining bandwidth (see above, just via DPI instead Diffserv) to P2P applications. But they don’t they simply hard-cap the traffic to rediciously low amounts during certain hours (bell), completely sever TCP con nections 24/7 instead of shaping them (comcast). See http://arstechnica.com/news.ars/post/20080602-new-filings-reveal-extent-damage-ofbell-canada-throttling.html Does this look like adaptive shaping to ensure fairness to you? It certainly doesn’t to me. It’s just a "let’s reduce the overall traffic, it’ll save us coasts"-approach, regardless of necessity. Another point is: Over here in mainland europe most ISPs do NOT shape p2p traffic. Even in sparely populated (=> more expensive infrastructure) countries like sweden where they have internet connections comparable to those in japan (100Mbit symmetric to the home) there is near to none traffic-shaping. Especially nothing comparable to Comcast. And that’s despite the fact that sweden is one of the world leaders in filesharing. The fact that such ISPs do exist alone proves that it IS possible to build the necessary
infrastructure and stay competitive… thus it’s a failure of ISPs to build out infrastructure or a lack of a larger scale infrastructure like open IXPs (the 3 largest IXPs are all in europe: AMS-IX, DE-CIX, LINX). You can hardly blame the endusers for market failure.
> Again, you want the Government to mandate "free" QoS i said — [...] than the OS/router manufacturer? I suggest you aim your demands at them instead… — -
where did i mention the govt in that paragraph? Making QoS-hardware/software available to the enduser who has "issues" with his gaming+p2ping does not involve politics. I only said it should be part of most network stacks, not that it must be. The govt should only enforce fairness (no lock-in, no market distortion, non-anticompetetive etc. etc.) on QoS on the ISP-side, that’s what net neutrality is about at its core, subpar law proposals do not define net neutrality, they are only attempts at its implementation.
15. Craysh June 3rd, 2008 at 23:38 | #15 Reply | Quote So what happens when Comcast decides that Netflix new download services are eating into their on-demand income? Or if Hulu starts getting more popular and people drop Comcast cable and only keep their internet? With no limits on what the ISP can do, the connection can be degraded to encourage (read:force) people to keep the ISP’s offering. In an ideal free market situation, there would be competition so this wouldn’t be an issue, since you could simply go with another ISP. Unfortunately, most cities only have one or two choices at the most. And you’re right, Vuze doesn’t have as much bandwidth costs as some of the major sites, but they pay for that which they use. Any additional bandwidth that is used across the network distributing those files are covered by the pa ying customers on either end of that connection. If you believe that they should pay additional fees because other people are using the service, then what’s to stop the ISP’s from charging other companies because use of their programs consume large amounts of bandwidth too? Let’s take FTP for
instance: To use FTP you only have to download a client such as Filezilla. Should Filezilla be required to pay a surcharge every time someone uses their client to transfer large files via FTP? No, they shouldn’t because the bandwidth that is being used is being paid for by the person who’s using the client and the person paying the FTP server’s bandwidth. Or an even more common comparison: perhaps Mozilla should be charged because the price in bandwidth they pay for (5.7MB for each copy of FireFox) pales in comparison to the amount of data transferred by using it (viewing web pages daily etc). It’s the same situation with Vuze. Vuze pays for the initial distribution bandwidth costs (several times for each item I’m sure) because that’s all they use. Should V uze pay extra because the users of their product are swapping data between each other simply because they use Vuze’s client? No, because the people on either side of that P2P network are paying for the bandwidth they are using. So you see, every end of bandwidth usage is paid for by the people who use it As for ISP’s not being unlimited, they do not list li mits on their site, contracts, commercials, fliers, or anything else (unless you count Time Warner testing it’s usage plan in Texas). If no bandwidth limits are disclosed, it’s "un-limited." You’d be pretty pissed if you bought a car and for some reason it has an undisclosed "30 miles a day" limit imposed on it.
16. Craysh June 3rd, 2008 at 23:40 | #16 Reply | Quote So what happens when Comcast decides that Netflix new download services are eating into their on-demand income? Or if Hulu starts getting more popular and people drop Comcast cable and only keep their internet? With no limits on what the ISP can do, the connection can be degraded to encourage (read:force) people to keep the ISP’s offering. In an ideal free market situation, there would be competition so this wouldn’t be an issue, since you could simply go with another ISP. Unfortunately, most cities only have one or two choices at the most. And you’re right, Vuze doesn’t have as much bandwidth costs as some of the major sites, but they pay for that which the y use. Any additional bandwidth that is used across the network distributing those files are covered by the pa ying customers on either end of that connection. If you believe that they should pay additional fees because other people are using the service, then what’s to stop the ISP’s from charging other companies because use of their programs consume large amounts of bandwidth too? Let’s take FTP for instance: To use FTP you only have to download a client such as Filezilla. Should Filezilla be required to pay a surcharge every time someone uses their client to transfer large files via FTP? No, they shouldn’t because the bandwidth that is being used is being
paid for by the person who’s using the client and the person paying the FTP server’s bandwidth. Or an even more common comparison: perhaps Mozilla should be charged because the price in bandwidth they pay for (5.7MB for each copy of FireFox) pales in comparison to the amount of data transferred by using it (viewing web pages daily etc). It’s the same situation with Vuze. Vuze pays for the initial distribution bandwidth costs (several times for each item I’m sure) because that’s all they use. Should V uze pay extra because the users of their product are swapping data between each other simply because they use Vuze’s client? No, because the people on either side of that P2P network are paying for the bandwidth they are using. So you see, every end of bandwidth usage is paid for by the people who use it As for ISP’s not being unlimited, they do not list limits on their site, contracts, commercials, fliers, or anything else (unless you count Time Warner testing it’s usage plan in Texas). If no bandwidth limits are disclosed, it’s "un-limited." You’d be pretty pissed if you bought a car and for some reason it has an undisclosed "30 miles a day" limit imposed on it.
17. George Ou June 4th, 2008 at 00:37 | #17 Reply | Quote 8472, you started off questioning my knowledge and now you show that you don’t even grasp the basics. You don’t answer the questions and you chang the subject.
Do you not understand that DSL providers are the IPTV providers? The DSL provider’s key incentive to building out FTTN infrastructure (AT&T U-Verse) is to be able to sell IPTV services, it’s also important to the FTTH providers like Verizon FiOS. You remove those incentives, they don’t build those bigger networks. These are private businesses that have to pay back their investors. They’re not going to do it for charity just like Google won’t spend a dime on building Broadband infrastructure or municipal Wi-Fi. When they build out that infrastructure, they get to carve out a piece of that infrastructure for their TV business. Verizon uses a separate optical wavelength separate from the broadband service but AT&T and other FTTN providers dynamically allocate resources from the broadband service and carve out a virtual circuit that assures sufficient throughput and low latency. It is understood that FTTN and FTTH providers get to have this IPTV service to justify their investment and it is understood that this creates better competition in the TV space. As for "unlimited", Comcast for example does NOT advertise unlimited speeds. However, I have always said that I am for more transparency both from a public policy point of view and a good business stand point. In fact I believe that Governmen t
oversight on transparency is good for the free market. Ho wever, you are being unreasonable in your ignorant expectation that advertised peak speeds are supposed to be for minimum speeds. "i never said i would prohibit the sale of QoS. QoS can be sold to customers, as long as it is applied equally to all services of the same type." Yes but you’re supporting Net Neutrality legislation which does prohibit the sale of QoS so you’re a hypocrite. I’m saying by definition, the FTTN provider should be able to protect their own IPTV service and by definition they’re not allowing other Internet content companies access to that bandwidth when the customer is using their IPTV. However, the customer is in effect saying they want their IPTV service and they have the choice to not purchase that IPTV service. This is what the consumer wants and the consumer should be able to demand this type of QoS that does NOT apply to all content providers. "people actually prefer VoD over TV, so i’d sa y they can live with "slow" channel flipping if you replace it with a low-res preview feature for ex ample. Not to mention that near-guarantees for latency of 250ms is feasible with good QoS solutions." Again, don’t confuse the context of "VoD" here. We’re talking TV services for NORMAL people and what NORMAL people expect out of their TV service. If YOU don’t like it, YOU DON’T BUY THE IPTV service and you get that entire pipe for the Internet. What is your problem?
"[...] than the OS/router manufacturer? I suggest you aim your demands at them instead… " Again, what does your OS and your routers have to do with download latency? Hello? Is someone home up there? We’re talking DOWNLOAD. We’re talking abou t the consumer’s right to tell their ISP what services and content providers they want prioritized going over their own broadband link. We’re not talking about the ISP deciding what to de-prioritize and prioritize. Net Neutrality legislation, which you say you support, would ban consumer’s choice. George Ou
18. George Ou June 4th, 2008 at 01:23 | #18 Reply | Quote
I set BitTorrent (official client) to use 10, 28, and 47 KB/sec upstream out of a peak sustainable upstream of 52 KB/sec. Then I did a tracert to a common web site like yahoo.com just to figure out the first hop beyond my home router and determined it was 12 ms minimum latency under clear conditions. Then I ran a ping to that first hop beyond my home router continuously as I downloaded or uploaded data. If you’re in Windows, be sure to set the CMD console buffer to a minimum of 3000 lines which is something I do anyways and I keep this as a permanent setting and I set the window to display double the number or vertical lines. I then do a select all and pasted it in to Excel 2007 (any version will do). Then I did a global replace to erase everything except the ping number. Then I created a new column that detected 12 ms from that first column of data. This new column is th e increase in ping over time that I get to plot out in Excel.
19. Jesse June 4th, 2008 at 01:47 | #19 Reply | Quote Great article George. I was wondering what tool you used for generating the graphs. I want to see if I get similar result in my home network.
20. Craysh June 4th, 2008 at 02:04 | #20 Reply | Quote They never say that it would be illegal to buy/use QoS software. But if they use that as an excuse to throttle competitors, then yes it should b e regulated.
21. George Ou June 4th, 2008 at 02:15 | #21
Reply | Quote "They never say that it would be illegal to buy/use QoS software. But if they use that as an excuse to throttle competitors, then yes it should be regulated." Learn to read Craysh. I’ve written it many times and the proposed Net Neutrality laws are clearly preventing the ISP from offering QoS services. I’ve patiently explained why you need to permit this kind of service and you’re not even taking the time to read or be genuine about your arguments. You’re wasting my time here.
22. George Ou June 4th, 2008 at 04:16 | #22 Reply | Quote Craysh says: "So what happens when Comcast decides that Netflix new download services are eating into their on-demand income? Or if Hulu starts getting more popular and people drop Comcast cable and only keep their internet? With no limits on what the ISP can do, the connection can be degraded to encourage (read:force) people to keep the ISP’s offering." You’re changing the subject here to Comcast, but I’ll answer it. If Comcast or any ISP degrades a specific service beyond reasonable network management, that should be a anti-trust issue and it is illegal. But this is a totally separate issue from banning consumers from purchasing QoS and harming the IPTV business.
Reasonable network management has multiple forms. Primitive forms try to make things a little more fair by throttling the bandwidth hog a pplications but it’s not the ideal solution since it isn’t completely accurate. This is the cheap and less-than-ideal way of dealing with the problem but it shouldn’t be illegal. Comcast is going to a more advanced form of network management that is independent of the protocol by the end of this year and they’re already starting field trials. Comcast’s new system simply decides that if a customer uses a disproportionate amount of bandwidth, say for example 2% uses 50% of resources, then they throttle that 2% down towards 2% of the resources. It turns out that the congestion problems are usually gone by the time they knock them down to 25% of the resources and they don’t need to throttle an y more. So the hogs still hog a huge amount of bandwidth but nowhere near as much. When there is no congestion, they go back to hogging 50% of the resources. As to the rest of your post, I’ve already explained it before but I’ll keep it brief. At no point should Vuze or Mozilla or Google or Microsoft or any other company be charged for interconnect or off-ramp fees under the threat of blockage or unreasonable degradation. The Internet is a best-effort service and there a re no guarantees on
throughput or latency. As long as there is no specific targeting of a service to harm that service beyond reasonable network management as I defined in the previous paragraph, then there should be no problems. Craysh says: ―If no bandwidth limits are disclosed, it’s "un-limited." Now you’re just being irrational and unreasonable and you’d be laughed out of court if you tried to bring that complaint. I’m all for MORE transparency and I would be for more Government oversight on this matter but you’re just being flat out unreasonable with that kind of statement.
As for the ArsTechnica article on Bell Canada and the claims from CAIP, there are some serious problems with CAIP’s allegations about VoIP blockage. They tried to shove 4.5 Mbps of traffic through a VoIP protocol which is clearl y ludicrous since VoIP never exceeds 83 Kbps in the worst case. It seems that they are deliberately trying to sca re the VoIP users when it’s clear that their test was designed to trigger the s ystem to identify a 4.5 Mbps VoIP UDP stream when this would never happen in reality and the fact that no Lingo or Vonage customers are complaining prove it. Because if you took a 32 Kbps or 82 Kbps VoIP stream and forced it down to 30 Kbps, you pretty much break the latter completely and make the former very problematic with that much packet loss. George Ou
23. The 8472 June 4th, 2008 at 15:01 | #23 Reply | Quote As i have stated before (which has been ignored): Looks like you (Ou) are confusing the concept of net neutrality with (potentially flawed) law proposals implementing said concept. Most QoS advocates are NOT against QoS, it’s useful to use various services (p2p, voip, vod) simultanously after all. Net neutrality at its core is about preventing discriminating, unreasonable and anti-competetive traffic-shaping. You repeatedly cite law proposals. So yes, these proposals might need some work (or maybe not, depending on how they are interpreted). But you mistake them for the concepts advocated by network neutrality proponents themselves. > Again, what does your OS and your routers have to do with download latency? Hello? Is someone home up there? We’re talking DOWNLOAD. Oh, and why the hell are you talking about "spacing" packets if this is all about downloads? Spacing packets would only make (a small amount of) sense if we were
talking about upload, at least there wouldn’t be any intermediate steps. If you talk about download then this is a lost cause as packets get rescheduled en route.
But we are not talking about download, we ARE talking about upload, looking at your own article would tell you as much: > If BitTorrent bursts 3 packets upstream over a 100 Mbps FastEthernet LAN connection, they will all sit in the upload queue of my DSL modem for 85 ms. And what is comcast traffic-shaping? Right… bittorrent-uploads (seeding). So Net neutrality in this context is also about upload. Although net neutrality _in general_ is about fair QoS regarding download AND upload.
24. The 8472 June 4th, 2008 at 15:01 | #24 Reply | Quote i couldn’t say it any better: http://arstechnica.com/news.ars/post/20080604-arpanetarchitect-bring-fairness-to-traffic-management.html
25. Hotkees June 4th, 2008 at 15:01 | #25 Reply | Quote I don’t have any problem with Bittorent. Last night while sleeping I downloaded 10 movies. Before I left for work I disabled Bittorent in the Network P roperties. BT uses 2 ports and I simply uncheck them. Next time I want to download, I’ll go in and turn the ports on.
26. notgonnatellya June 4th, 2008 at 18:58 | #26 Reply | Quote
8472 said —– You repeatedly cite law proposals. So yes, these proposals might need some work (or maybe not, depending on how they are interpreted). But you mistake them for the concepts advocated by network neutrality proponents themselves. —– If the laws are can be interepreted in such a way that it outlaws QOS, you should assume that that interpretation will prevail at some point. Even if you can’t believe that, you can be certain that there will be plenty of costly lawsuits making that claim. 8472 said —– err… are we talking about a DSL provider or an IPTV provider? Because if you’re saying that the DSL provider and IPTV provider are not only the same entity but also are inseparable you obviously get a conflict of interest and an anti-competitive market as all IPTV competitors would have a disadvantage compared to the DSL/IPTV conglomerate. —– Using your logic, you should be fighting for legislation that forces Echostar to let directTV use their satellites to transmit video. If the cable company wants to set aside X MB/S for their video service, I see nothing wrong with that. Would you also argue that I should be able to sell my gasoline using my local Chevron’s Pumps? Infrastructure isn’t free. I object to them giving preferential treatment to some external video suplier over another. I’d also object if they made using another IP more expensive than it currently is. But there’s a difference between Skype or Vonage, which don’t require much bandwidth, but do require QoS, and a Cable Network. If you’re actually arguing that Verizon can’t reserve part of their network exclusively for their video service, then you’ve lost me, and I’ve been on the NN train for several years. If you don’t like that plan, you can go the route of Lafayette Louisiana, and have your city build it’s own Fiber network…oh wait, they’re providing their own video service as well.
If your goal is to get people to call their congressmen up and tell them to oppose net neutrality, you’re doing a good job. With that said, your link to the engineer that helped create arpanet was interesting. I’m f airly certain that it’s an extremely inefficient use of resources and will likely mean everyone ends up with less up and down bandwidth, but it would force real numbers when selling service.
Personally, I think a better solution is that that figure is the minimum level of service, but that someone can use more of it, (albeit at a lower priority than anything up to the min) if its’ available.
27. notgonnatellya June 4th, 2008 at 19:06 | #27 Reply | Quote There’s no order to them. 5/31 at the top, which goes to 6/1……6/1 at the bottom which goes to 6/2 then 3 then 1 then 4
It makes it very hard to follow threads.
28. George Ou June 4th, 2008 at 19:07 | #28 Reply | Quote I called tech support and they said it’s a bug in this blogging module. Maybe I should ask them to reboot my instance or something.
29. George Ou June 4th, 2008 at 19:11 | #29 Reply | Quote I was the one who introduced Nate Anderson to Anagran’s Flow Management technology and I’ve stated in these threads that Anagran’s protocol agnostic solution is a much better solution. However, I don’t believe that Government should outlaw inferior and less expensive methods. The market place should shame ISPs in to using the better solutions and this is what happened with Comcast.
30.
The 8472 June 5th, 2008 at 14:10 | #30 Reply | Quote @notgonnatellya > Infrastructure isn’t free. right. That’s why you pay subscriber fees for your broadband access. Just like telco infrastructure isn’t free and you pay your telco fees. That doesn’t mean the telco g
31. George Ou June 5th, 2008 at 14:19 | #31 Reply | Quote 8472 says: "right. That’s why you pay subscriber fees for your broadband access. Just like telco infrastructure isn’t free and you pay your telco fees. That doesn’t mean the telco gets an exclusive/reserved part of that line to shove other services towards you. At least over here telcos are forced to unbundle and resale services and it works quite well as it allows smaller competitors to achieve greater coverage by doing wholesale in some areas and run their own services in other areas."
In Germany, my friend gets IPTV service from his VDSL FTTN provider BUNDLED. When you choose that service, it MUST have QoS in the form of bandwidth guarantees AND latency guarantees. 8472 says: "My point is that reserving bandwidth is ok, as long as the user has the choice what to reserve the bandwidth to." You do have a choice, you don’t have to order the IPTV service if you don’t want that bandwidth reserved. NOBODY buys IPTV service with the exp ectation that BitTorrent will mess up your TV service and you’re being silly to suggest otherwise. It’s not shoved down your throat at all, you have a choice.
The consumer also should have a choice to tell their ISP what services they want to prioritize. You could have a situation where a household uses two Video on Demand services but they want one to get more priority than the other. So why shouldn’t the consumer tell their ISP to prioritize downstream iTunes more than XBOX Market Place or vice versa? The point is that for downstream prioritization, the user has to rely on the ISP’s gear because it is the transmitting device can ex ert the most control. We’re not
talking about some sinister plot where Apple pays an ISP to lock out a competitor like Microsoft or vice versa and certainly I would be open to making that form of exclusive access illegal. But we’re talking about consumers choosing to prioritize a certain web site above other web sites and the consumer should be able to block or prioritize any service they want.
32. Clifton June 6th, 2008 at 01:17 | #32 Reply | Quote Recent interesting post on how some colleges are dealing with bit torrent: http://netequalizer.wordpress.com/2008/05/17/curbing-riaa-requests-on-your-studentnetwork/
33. Jacob June 6th, 2008 at 03:57 | #33 Reply | Quote There are some great solutions here, but I have an ultimate solution that I use. Route your bittorent traffic over a VPN service like Relakks (there’s many others out there). These low cost services allow you to route all your traffic through one connection. Solves the latency hands down, you’ll still need to throttle in your app as n ot to choke off your pipe, but the effect of a VPN is that all those connections behave as one. It also adds privacy so you have less to worry about from the RIAA. Relakks will result in a slower top download speed, but I’ve found others which will run at full speed.
34. Saka June 18th, 2008 at 17:50 | #34 Reply | Quote
I had the same problem with BT lagging everyone at home as well, I found that if we limit the upload rate on our BT applications, it doesn’t cause as much of a problem for others…
35. Slade Wilson June 18th, 2008 at 17:51 | #35 Reply | Quote The 8472 said: "@notgonnatellya > Infrastructure isn’t free. right. That’s why you pay subscriber fees for your broadband access. Just like telco infrastructure isn’t free and you pay your telco fees. That doesn’t mean the telco gets an exclusive/reserved part of that line to shove other services towards you. At least over here telcos are forced to unbundle and resale services and it works quite well as it allows smaller competitors to achieve greater coverage by doing wholesale in some areas and run their own services in other areas." If you say so, but it sure doesn’t sound like a free market to me. Should the cable companies be forced to unbundle? What about satellite providers? Gas stations?
Why do you single out telcos? If AT&T or Verizon spend billions to create upgraded fiber-based networks, why should they be forced to allow other providers to use those networks when said providers didn’t invest a dime? When there was only one network (the telephone network), an argument for heavy regulation, etc. made some sense. But now there is competition: if I don’t want TV from Comcast, I can get it from AT&T or DirecTV. If I don’t want Internet from AT&T, I can get it from Comcast or Hughesnet or a fixed wireless provider or even a cellular company. And so on. If you force such unbundling, what you are really doing is discouraging such infrastructure build-outs. Why should AT&T spend all that money to b uild out a new fiber network? They’ll never make it back on $25/mo or $35/mo DSL services.
36. Peter Gerassimoff
June 27th, 2008 at 04:17 | #36 Reply | Quote The problem isn’t with the last mile link, but in the ISP itself. The latency seen is due to bandwidth constraints somewhere along the path of the VOIP traffic. Low latency is an artifact of a IP network with spare capacity. The spare capacity means that no packet is lost and all packets have a low consistent latency. VOIP doesn’t really need low latencies, but stable latencies. A call to the exact opposite point in the world will need 68ms just for the trip of 12.5K miles alone at the speed of light. And that is best case. Most people won’t notice even a 100ms delay from what you say to be heard on the other end. Even a satellite based phone call has a minimum 300ms delay with one hop (you to POTS to sat to POTS to other). What gets VOIP screwed up is latency changes. Some at 200ms and some at 5ms will cause many low grade VOIP systems to mess up. But a VOIP system that buffers 250ms at least would never have any problems.
But that variation is not just due to your end, but any blockages down the road. If an ISP is shaping bittorrent traffic at his forwarding connection, the packets will pile up in your router regardless of what BT does. Even 10KB/s will cause them to pile up. Yes slower and less often, but they will pile up even with one per connection. The BT doesn’t see that TCP has to retransmit that 1.5KB packet an y number of times to get it through UDP. It just sees 1.5KB of upload. And it will send about 6.6 such packets per second using some moving average plus some overhead ones as well. But your router just sees the average retransmissions required times 1.5KB IP packets times 6.6 or 10′s of KB/s usage in the upload side. Those latenc y spikes you see are the IP retransmit packets piling up. And that retransmission can be required through any given part of any BT peer connection going through a site that is capacity constrained whether its deliberate or not. The VOIP can similarly be messed by somewhere a network provider dropping packets at random to shoehorn traffic through a bottleneck. Strict network neutral carriers of that traffic actually would help streaming traffic by randomly dropping the pack ets. Since the packets are dropped randomly, the latenc y would stay within the typical gaussian distribution and slowly change allowing the applications to adjust. Since shaping requires intelligence, dumb networks have much higher capacity, 2-10 times as much as smart shaping ones. That extra capacity usually works to drastically reduce dropped packet rates and thus make latency both low and stable. Which makes them look like the ideal non blocking network. That is why most non QOS networks look like ideal ones. Also a faster last mile takes care of more problems. That is why a 100/100mbps FTTH connection doesn’t really have these issues. The bottlenecks are all somewhere else. Even a 1.5KB packet is only 0.12ms on such a connection. 8000 packets per second can make any problem with 100-200 peer connections seem effortless. Think about a HDTV signal. Its 19.2Mbps. Even composed of 1.5KB packets, thats 1600 packets per second. 250ms delay means 400 packets are buffered. One or two packets either way won’t cause any problems. And I had U-Verse. The HD channel switch times
were more like 2-5 seconds, not 1/2 second. So latency hiccups shouldn’t affect VOIP traffic in the last mile.
37. kelly March 29th, 2009 at 22:48 | #37 Reply | Quote for me…using bittorrent at all causes online game play to be erratic
downloading at 3kb..downloading at 300 kb uploading nothing or uploading at the max speed it all causes games to feel screwy like there is some kind of packet loss or something it doesnt have anything to do with limiting the upload…
38. Switeck June 10th, 2009 at 06:30 | #38 Reply | Quote We do not know from the article alone: 1.What BitTorrent client is used in testing. …although in reading the 100+ replies, about the 100th reply in: "I set BitTorrent (official client) to use 10, 28, and 47 KB/sec upstream out of a peak sustainable upstream of 52 KB/sec." 2.What settings are used in the BitTorrent client, BESIDES upload an d download speed limits! 3.What torrent/s are the tests run on, specifically how many peers an d seeds are there…and if the torrents are new, old, or even POISONED! 4.Firewalled conditions in BitTorrent? 5."Clean Line" tests…we don’t know if the tests were done on an internet connection that is not also being hammered by spyware/viruses both incoming and out going…or old BitTorrent traffic (DHT and peers/seeds retrying torrents you long-ago stopped). 6.If the tests’ computer/s is also running networking software and hardware known to interfere with BitTorrent traffic or the internet connection in general. (Zone Alarm, I choose you!) 7.What ISP service is being used? (some are real lemons who cripple the WHOLE line
when BitTorrent traffic is detected!) 8.Even how testing was done between upload speed changes can have a huge effect on the outcome. A short "cooldown" time between each upload speed reduction might make the latency temporarily worse, as both the line and the BitTorrent client hasn’t adapted to the new conditions. In conclusion, the test is not scientific and the con clusion should not be treated as fact. This old article is noteworthy in explaining much of the problem: http://www.stuartcheshire.org/rants/Latency.html Lastly, a BitTorrent/uTorrent troubleshooting guide: http://forum.utorrent.com/viewtopic.php?id=15992
39. George Ou July 19th, 2009 at 07:56 | #39 Reply | Quote @yacc143 Here’s another hint smart guy. Read the actual article. It tells you that the speed was limited to 10% of the link speed yet it caused huge amounts of jitter, albeit lower frequency when the percent saturation is lower.
40. leuboo December 11th, 2009 at 19:08 | #40 Reply | Quote I now use widevpn to unblock so mcuh high ping
41. Phil February 2nd, 2010 at 20:35 | #41 Reply | Quote
I was disappointed to see this thread devolve into a lot of heated yelling. The technology exists right now to allow your Bit Torrent transfers to peaceably coex ist with VoIP and any other latency-sensitive applications on the same Internet connection, and the politics can be limited to those parties sharing the same link (e.g., you and your roommates). No need to involve the ISPs. The answer is to implement a traffic-shaping router that prioritizes and controls your upstream traffic, which you can do directly. Nearly ever y residential Internet link has a much slower upstream than downstream, so addressing that bottleneck will mak e a huge difference even if nothing at all is done on the downlink by the ISP. I use a small single-board computer running Linux with the traffic shaping features turned on. Its has two primary functions. First, it rate-limits the traffic it sends into my broadband modem to ensure that a queue never builds within the modem – that’s what causes the long latencies when Bit Torrent gets going. Second, it examines the IP Type of Service (TOS) bits for the 6-bit DSCP (Differentiated Services Code Point) field. This is the packet’s priority, as locally defined by me within my own network – it doesn’t necessarily have to have any meaning once a packet gets to my ISP. Some Bit Torrent clients such as Azureus/Vuze already provide a configuration page to specify the DSCP value that should go into every packet it generates. (Unfortunately I am running Vuze on OSX which still has a bug preventing this feature from working, so I am temporarily having to use the iptables facility in Linux on the router to recognize my Bit Torrent traffic by other means, specifically IP address and port numbe r). By throttling the flow into my modem, Linux pulls the queues back into itself where they can be managed — where a new high priority packet can be sent ahead of an earlier, low priority packet. Without this throttling, there would never be a queu e in Linux because its Ethernet link to my modem is far faster than the link beyond. The third feature – and one that’s surprisingly useful all by itself – is to enable Stochastic Fair Queueing within each priority level. So even when several streams all compete at the same priority level, they are served in a round robin fashion; one bandwidth hog TCP connection cannot force everyone to wait for an entire window to drain each time before sending a packet. Google for something called the ―wonder shaper‖ – this is the name given to these scripts by their original author. They really do work wonders.
42. Broken_bazooka February 28th, 2010 at 18:23 | #42 Reply | Quote
George Ou, you should have listened to what The 8472 had to say instead of just reiterating yourself and calling him names. From reading both of your posts, it is quite obvious that you two are in way different leagues when it comes to skills in networking. In networking, there’s the business standpoint, and there’s the technical standpo int. When looking at history, the business standpoint doesn’t mean much, while the technical standpoint is what stands the test of time. Keep that in mind.
43. Ron Barone March 12th, 2010 at 12:46 | #43 Reply | Quote You can use rate-limiting to proactivly advertise a cong ested network. Cisco IOS supports this (aka traffic policing) but I have never seen this feature in home routers. Here’s a rundown of this… as I didn’t read the above articles since they were more motivated at ISP’s control over data and not the problem you posted. Slowing down bittorrent is a pill to fix the symptom under ideal circumstances. This would not be the correct way to handle this since other transfers (FTP downloads etc) would cause your queue to fill up. It isnt bit-to rrent it’s simply the faster connection fire-hosing your connection with packets (i will continue to explain) Let me start by saying we’re talking TCP not UDP here. As you may already know the only way to communicate to the sender to SLOW DOWN because you’re queues are filling up is to drop the packets. When packets are dropped the upstream router sees a congested network that’s unable to accept the packets ―that fast‖ and starts slowing down the streaming of the packets. It’s much like a funnel (queue) where each connection is pouring water (sending packets) into the same funnel (queue) and when you start spilling (packet drop) the person that caused the spill (packet drop) slows down and the cycle of slow-pour/fast-pour (fair packet queue) continues. Now… What you can do is make your receive queue on YOUR equipment smaller and make your committed bandwidth rate less (~1-5%) than your total available bandwidth. This will cause the packets to be dropped keeping your queues mostly empty (except possibly when starting a transfer). Doing this has each connection slow down before they actually start building up packets (funnel spill over lol). This means a ny packet (ICMP, UDP, TCP) is placed into a mostly empty queue making data reception faster because everyone was told to slow down a small bit.
This, of course, is exactly what happens now, so dropping those packets before it becomes a problem is not a bad thing. It just causes it to happen sooner smoothing out that nasty queue problem. It’s much like allowing your funnel to fill up but hitting the person in the knees if they cause that water level to rise hogging the exit of other people’s fluid. Ok so maybe that’s a bad analogy but I hope you get the point. Also any kind of QoS or traffic shaping is complicated on shared networks (cable) due to the up to millisecond calculations on everyone’s committed access rate and the total available bandwidth available. The QoS doenst solve the problem it just makes the problem go away for select items. This doesnt exactly help when you are playing games on dynamic ports, but it does when it’s VoIP and the ports are predictable. Remember though, VoIP QoS needs to be done BEFORE it gets to your router or you need to start dropping some TCP packets to slow those other bastards down. Some of my info may be inaccurate… It’s been awhile since I’ve had to manage networks… so this information is about 10 years out of the back of my head. Search ratelimiting or cisco traffic policing for more accurate information. In a perfect world t he congestion bit would be used better to proactively notify senders they were sending too fast… In the mean time, dropping packets is the most effective way to handle this from an end receiver.
44. Sen March 24th, 2010 at 18:50 | #44 Reply | Quote …. so how would you fix it? am i the only one not getting it?
45. Greg March 29th, 2010 at 10:53 | #45 Reply | Quote I use Vuze and this has always been a terrible problem for me especially as i’m sharing the internet connection with 5 other guys. In your article You point out that this seems to be a problem of too much buffering. So i looked around the Vuze settings, and found entries called ―Socket SO_SNDBUF‖ and ―Socket SO_RCVBUF‖ (Located in Connection->Extended Settings). I set them both to 3000 (2x the MTU), and now it seems to run much more smoothly. Haven’t done any
