The fact that ISPs have oversold their services and painted themselves into a corner with misleading, if not downright false advertising is entirely their problem. Get a law passed and let them whine about it and then deal with it.

So, what would you prefer? Would you prefer fixed lower bandwidths; everyone gets exactly 1 Mbps all the time, for the same price you're paying now for 50 Mbps? Then you're going to have a ton of bandwidth going unused all the time, while everything will load slower.

Or would you prefer the famously unpopular bandwidth caps and overage charges?

Saying "unlimited bandwidth" but selling it as a residential (ie, non-commercial) plan seems like a fairly reasonable solution to the problem. It gives actual non-commercial residential customers what they want, the ability to have spikes of high bitrate transfers without having to worry about overage charges or being throttled, while actually allowing that pool of bandwidth to be shared reasonably fairly.

The problem is if you divide shared resources like upstream bandwidth (or in the case of hosting, resource like disk or RAM) in precise amounts such that you could support 100% of the capacity you sold at any time, you would be wasting the vast majority of your resources. Most customers can't predict in advance exactly how much they need, and are quite averse to any kind of overage charges.

Now, for some (mostly business) customers and workloads, precise metering works well, and for some completely dedicated bandwidth works well. But for the average residential customer, who has extremely spiky bandwidth demands and is quite averse to overage charges, neither of those work well.

I would prefer minimum service guarantees rather than maximum usage allowances. If I want to maximize my use of my connection, the only thing my ISP should do is reduce my throughput when someone else is trying to check their email, so that I do not get in their way. Other than that Internet connections should be a free-for-all.

The real problem here is that ISPs are so grossly overselling their networks' capacities that they cannot provide any guarantee beyond dialup speeds unless people take turns. Rather than admit this, they try to assign blame to people who foolishly think that "Up to 20Mbps!!!" means "up to 20Mbps" when in fact the ISPs want it to mean "up to 20Mbps in occasional short bursts, and only for the things we think you should be doing."

I am not asking for commercial-grade service. I am asking that my neighbors be able to stream videos while I am seeding Debian torrents. Nobody expects some kind of Internet-connection-utopia, but it is not wrong or unreasonable to expect to be allowed to use my Internet connection as I wish, particularly when I am paying more and receiving less than my overseas friends.

Following the thread here, I think what you are asking for is unreasonable given the current infrastructure or massively unpopular.

In the current environment what you are asking for is completely possible. Comcast will give you your speeds, but if they see you are seeding your debian torrents for too long, they will throttle your connection. Now they could advertise this, but we know how unpopular connection throttling is. However in a way, a connection throttle is your minimum service guarantee - your speed will never go below the throttle.

Now I don't know anything about laying internet pipe, but lets say we gave everyone minimum service guarantees, how much do you think that would cost, and how much would you be willing to pay for a given speed? It may be completely possible that Comcast is fine with your seeding as long as you limit your speed to percentage of that number.

This doesn't really refute your point, but given these two constraints its easy to see why ISPs may choose to go the "up to XMbps" route rather than the "minimum allowed route." Would it really change anything for you if you found out that your minimum service guarantee was 768kbps at your current price point?

"Now I don't know anything about laying internet pipe, but lets say we gave everyone minimum service guarantees, how much do you think that would cost, and how much would you be willing to pay for a given speed? It may be completely possible that Comcast is fine with your seeding as long as you limit your speed to percentage of that number."

I think the cost of providing such a guarantee depends on the guarantee, and therein lies the problem. Right now, I suspect that a typical cable modem connection would only be able to guarantee dialup speeds, due to how massively oversold the bandwidth is. Of course, most people do not seed torrents or run web servers or whatever else on their home connection, and so we never see service so degraded.

"Would it really change anything for you if you found out that your minimum service guarantee was 768kbps at your current price point?"

Sure: I would have a much higher opinion of my ISP. Frankly, I would have a higher opinion if service classes were based on such minimum service levels rather than maximum levels: I would much rather pay for at least 768kbps than at most 20Mbps, particularly if "at least 768kbps" meant "at least 768kbps, and higher throughput when available."

If you want minimum service guarantees, buy a dedicated line. They generally run somewhere around $2500 a month for symmetric 50 Mbps.

If you don't want to pay that much, but you still do want the option of bursting to 50 Mbps (or more), shared bandwidth, like you get from your average residential plan, is your best bet. And I don't know about your experience, but more often than not, I do get the advertised speeds, at least when I'm connecting to servers that can support that sort of bandwidth.

I mean, I agree with you in theory, it would be great to be able to get dedicated symmetric 100 Mbps for $80 a month; but it's just not economically feasible.

And I agree that ISPs discriminating based on type of traffic (torrent vs VOIP vs HTTP, etc) is not good, but this whole thread is about Google changing from "you can't host servers" to "you can host servers but only for non-commercial purposes", which is, I think, a huge improvement and fairly reasonable given that commercial use tends to be very different than residential use and it doesn't make sense to treat them the same way.

It's not that I want a guarantee of 50Mbps. What I really want is for everyone to have a guarantee of, say, 768kbps, so that when I am sitting here seeding Debian torrents I do not have to worry that the old lady living next door will not be able to video chat with her grandchildren. I also find it somewhat difficult to accept the idea that no such guarantee will be made, yet I will still be punished if I reduce the quality of service for my neighbors -- either there is a guarantee, in which case I should be throttled in a way that ensures everyone enjoys at least that service level, or there is no guarantee in which case consuming lots of bandwidth is simply rude (but not something that should be punished).

Frankly, I think this solution is not so bad: I get lower priority as I consume more bandwidth, until I am getting only the minimum throughput for my service class. Thus people can still get their burst speeds, and I can still seed torrents or run a web server or whatever else.

My real point is that I do not see why people should be punished for using their connection continuously. Comcast did not offer me "burst service," they offered me "always on" service with "speeds up to X." I feel little sympathy for an ISP that massively oversold service and then claimed that it is abusive for customers to actually use what they are paying for.

It's actually pretty difficult, technically, to provide you with a minimum guaranteed amount of bandwidth, without simultaneously limiting everyone else.

The problem is that there are many layers in the networking stack which interact in complicated ways. TCP congestion control algorithms and TCP slow start, which happen at the endpoints. Buffering, which happens at pretty much every intermediate hop in the path. And traffic shaping, which can pretty much only drop packets to limit bandwidth used.

If I'm supposed to guarantee you 768kbps, that means that I need to drop any packets from everyone else that exceeds the capacity minus 768kbps. But if I do this for everybody, I'm back to the case where I'm giving everyone a dedicated 768kbps, and they aren't able to share excess bandwidth at all.

Now, you could imagine a scheme in which I don't limit other people's bandwidth until you actually start using some. But then if it's congested when you start your transfer, you won't actually be able to send data at your full 768kbps at first, so TCP congestion control will kick in and limit you to less than that. So when traffic shaping looks at your stream, it sees something less than being used. Should it then limit everyone else such that it guarantees you your full 768kbps? If it does that every time someone uses their connection, then services that trickle just a small amount of data will essentially guarantee that everyone is always being guaranteed their 768kbps by limiting everyone else, and you're back to the "OK, now everyone has a dedicated 768kbps connection that won't go faster".

Or should it limit everyone else based on the bandwidth that you are currently using? Say, when it notices that you are sending 10 kbps, it throttles everyone else down to allow you 15 kbps; then when you increase to sending 15 kbps, it throttles everyone else down to allow you to send 20 kbps, and so on, until you hit your guaranteed bandwidth. Based on the delays it takes for TCP congestion control to ramp up the speed it tries to send while traffic shaping ramps up the amount it's limiting everyone else to give you the bandwidth you need, and the latencies involved (generally tens to low hundreds of milliseconds round trips) it could be a few seconds before you are actually able to get up to your guaranteed bandwidth.

But those first few seconds are generally the most crucial to be delivered quickly. Most people are browsing the web, where there are lots of short page loads. You want those to be able to kick in quickly and then be done with.

Remember, this is an industry where we've only recently come to understand the effects of buffer bloat on performance. Effective quality of service guarantees, other than giving everyone X guaranteed bandwidth by limiting everyone else to the total minus X bandwidth (which, when the network approaches capacity, becomes not much different than giving everyone an X bandwidth dedicated line), is a hard problem.

  My real point is that I do not see why people should be 
  punished for using their connection continuously. Comcast 
  did not offer me "burst service," they offered me "always 
  on" service with "speeds up to X." I feel little sympathy 
  for an ISP that massively oversold service and then claimed 
  that it is abusive for customers to actually use what they 
  are paying for.

Yes. It's always on service (as in, you can be connected and have the service available 100% of the time) and have speeds up to X available. That's what they promised, and that's what they deliver. They also sold this as a "Residential" service. They have other services for commercial and dedicated lines, that they will sell you for greater utilization.

You are not paying for continuous use of X, you are paying for an always available line that gives you speed up to X, for personal residential use.

Restrictions on not using the network abusively or not running commercial services on a home network are not some creepy abusive plot, they are really the best that the networks can do to provide you with affordable, high-speed broadband.

Do I wish that there were more competition? Sure. Do I wish that I got higher speeds for cheaper? Sure. Do I wish that they would be network neutral and not block ports running on my network? Absolutely. But there's absolutely nothing wrong with selling a residential plan that is for residential purposes, and enforcing that, separately from their commercial plan, and separately from their dedicated plan. It's not that difficult; if you want to use it for commercial purposes, use their commercial plan. If you want guaranteed bandwidth, buy a dedicated line.

And remember, the ISPs have not "massively oversold" their services. If you look at the FCC's report (measured in 2011): http://web.archive.org/web/20130511164730/http://transition.... , you'll see several ISPs that actually exceed their advertised bandwidths even during peak hours. Now, some of them are clearly not doing so well by that metric; Century Link is not even hitting 80% off peak, and barely 50% on peak. I would say you have a good case for false advertising there. But if you notice, Comcast and Verizon FiOS manage to exceed their advertised bandwidth, on average, even on peak. This matches pretty well with my experience over the last 5 years or so. I've had both DSL and Cable. DSL routinely underperformed what was sold, which was already paltry. But cable has generally exceeded what was sold, when connecting to servers that could actually handle the load.

So, I'm not sure I'd call that "massively overselling", other than maybe Century Link (and the other DSL providers don't hold up too well either).

Think of it like a buffet. For efficiency and simplicity, it's an all-you-can eat plan; unlike food, people aren't very good at knowing how much they are going to use and so charging based on usage is much more unpopular, making buffet style plans much more populat. So they do "all you can eat". And the rules are, it's all you can eat, and in one sitting; you can't take a bunch with you and then sell it to someone else, nor can you horde food for takeout because "you're going to eat it later". Would you start complaining and railing that the all-you-can eat buffet massively overselling? No. That's all that they're doing with the internet plan; it's just that there are less social conventions and it's easier to accidentally (or deliberately) massively overuse your fair share of bandwidth than it is to massively abuse a buffet, since it all happens automatically.

I would prefer truthful advertisement. If they must cut down on minimum transfer rates, then that’s what the advertisement should say.

I have a general rule: Buy 250KB/s, and you get 100% of the bought speed.

Buy 1MB/s, and get 95% of the bought speed.

Buy 10MB/s, and get 80% of the bought speed.

Buy 100MB/s, and get in average around 50% of the bought speed. 25 at prime time, and 90 around 4am.

In the end, more expensive connection give you more, but each level give less proportionally bandwidth than advertised. This should be illegal. If they want to sell me 25MB/s, but at strange hours up to 100MB/s, than the advertisement should say so, plain and simple.

If you went to the gas station and only got 50% from paying for a full tank, you, me and everyone would be angry. It wouldn't matter if the gas station was "low" on gas, or if it meant that the gas station might end up with no gas left for the next customer.

Bandwidth is not like gas. It's not something that you buy, and consume it, and then go out and buy more.

Instead, there's an infrastructure that need to be built. That infrastructure has a certain capacity; at any given time, only so many bits per second can be flowing over it.

Now, most people don't use all of their capacity at a continuous, constant rate. Rather, their usage is quite bursty. You use a lot of bandwidth for a short amount of time when loading a video, and then none while you're watching it. Or you use lots of small bursts while surfing the web.

If they sold 100 Mbps (no one sells 100 MB/s) and guaranteed that to you, it would be astronomically expensive. But selling you service that gets 100 Mbps in bursts if people have average personal-use utilization patterns is perfectly possible.

Now, how exactly should they advertise that? That's a tough question. If they were going to advertise the minimum guaranteed bandwidth, it would be laughably low (and of course, they could never really guarantee any minimum, since how much bandwidth you get for a given transfer depends not only on your connection, but the backbones in between and server's connection; there is no way to guarantee any particular amount of bandwidth in any meaningful way). And you would almost always get more than that, so that would be a fairly meaningless number.

Perhaps they could advertise an aggregate average bandwidth? That would be pretty hard to define and test (what sample period are you measuring over? How do you know when a transfer has stalled due to congestion somewhere vs being finished?), and wouldn't really give you much meaningful. Are you really going to go out an canvas your neighbors to determine if your neigbhorhood really is getting the promised average aggregate bandwidth?

The only really meaningful thing they can give you is what the actual maximimum bandwidth they will allow over your connection. That is a hard and fast number, and can be tested fairly easily. Of course, that doesn't say much about your bandwidth accessing any particular site, but there's nothing you can really do about that without control of the entire network.

Now, actually in practice, it appears that when I buy a certain level from Comcast, they don't actually use that as a hard maximum. I've seen bursts that were higher than the advertised bandwidth; and of course, I've seen transfers that were lower.

I know that everyone likes to criticize ISPs, and there is definitely some behavior that I find odious (blocking particular protocols, attempts to subvert net-neutrality). But over-selling bandwidth is not one of them. I can't really imagine a system that would work better than over-selling bandwidth.

I don't have any problem with an ISP saying "up to #Mbps" as that's what they allow each customer to get (though this is 2013 and I'm sure with the amount of metrics that CPE can gather, we could measure average thruput and advertise that too).

What I have a huge issue with is selling the services as unlimited and then smacking people down when they walk over some invisible line, or worse selling an "up to #Mbps" line when they know damn well based on basic math that nobody's ever going to get anywhere near that speed in that area.

It's absolutely ripe for abuse. You, as an ISP, can legally sell someone a 100 meg line you know in advance won't reach that speed.

If they has such hard time to measure limits, they can always use first hop router as test base.

But lets take the look from the buyer. You see a 8MB line for $80 amount, and 10MB line for say for $100. Knowing what you pay for is the minimum requirement for informed purchase, or you end up with a lemon market.

ISP are lemon markets, and you get poor quality for more expensive price. No amount of "The infrastructure has a certain capacity" argument is going to change that.

How about this: for $50/mo you get a guarantee of at least 1Mbps of continuous throughput (or at least a best effort at that), with no restriction on your usage beyond that? Why does it need to be black or white? Why not just dynamically adjust the throughput for "heavy users" so that nobody sees speeds below some level regardless of how others are utilizing the network?

Oh, wait, these connections are so grossly oversold that the only such guarantee that could be made would be on the order of dialup speeds. Why should we feel pity for ISPs that create such a situation, when they are charging us more and providing us with less than what people pay and enjoy in other countries?

For what it's worth, this is exactly how we're planning to handle shared connections.

Our typical "small office" connection is advertised as "60x60 bursting on top of a dedicated T1 connection". Which comes out to most people getting around 50x50 about 85% of the time, but always having a dedicated 1.5mbps waiting for you (so it can never drop below that speed, even if every subscriber maxed out their line at the same time).

We also do 'dynamically adjust the throughput for heavy users'. The problem is the heavy users are typically us, the folks on Hacker News who complain when their torrent seeds get slowed down. These are the users who complain about internet being 'oversold' when in reality, they alone account for 80% of their entire neighborhoods backhaul costs.

- - -

I encourage you, if you feel ISP's connections are so oversold, please try to start your own ISP. I'd love to offer everyone awesome dedicated connections, and I'll be your best friend if you can show me a way to do things cheaper :)

But I suspect you'll find out that it's not exactly the easiest business in the world to run, especially if your bootstrapping it.

You are bringing transparency to your pricing. That is huge. What HN get annoyed about is not torrents slowing down. But torrents being slowed down for arbitrary reasons.

We know that a person's email and voice calls are more important than getting a Debian distro 4 seconds earlier. QoS shaping does not violate network neutrality the way I understand it.

That would be fine except for path dependence. You're talking about taking essentially the same service but making it sound much worse than it currently sounds.

I had always assumed the advertised speeds were likely (according to an unregulated definition), but not guaranteed. Even if the ISP gave me an SLA (maybe they did?), they couldn't make promises regarding the performance of any external network anyway.

Maybe they should be required to report a sharing ratio in addition to bandwidth and latency within the ISP's own network. Any thoughts on how that would look, or what formula might make sense?