Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

You still haven't refuted the point. This isn't about the relationship between Verizon and its customers, at all.

The agreement between Level3 and Verizon probably has some balanced traffic requirement, with penalties for asymmetry. If that is the case, then the nature of the imbalance would force Level3 to make payments to Verizon.

Verizon gave a very clear suggestion for Netflix: work with other providers as well. If Netflix worked with the overwhelming majority of third party providers, then you could argue that Verizon is not playing fair. But I don't see any commentary that Netflix is actually working with most of the third party providers.



ISPs charge me primarily to download. Literally I can't upload as much as I download. My uploads are contractually limited to 20% of my downloads.

So does that mean there's a downloader-pays rule?

If so that means that Verizon should be paying Level3 because it consumes 3x the bandwidth that it provides.

Verizon's needs Level3 3x as badly as Level3 needs Verizon on a bit-by-bit basis. So why isn't Verizon paying Level3 for the imbalance?

There are actually two sides to the ISP game. There's the consumer ISP side and there's the commercial ISP side.

Consumer ISPs are in business to get their customers hooked up to as much of the internet as possible to enable them to consume things. And they charge their customers (normal folk like us) for this service.

Commercial ISPs are in business to get their customers hooked up to as much of the internet as possible to provide services. And they charge their customers (big companies) for it.

When commerial ISPs meet consumer ISPs typically they say "let's trade traffic for free so that way we can both keep our money and not worry about accounting"

What's happened is that a consumer ISP is trying to charge more than just it's last-mile normal customers for data that they use and which it is ostensibly contractually obliged to provide to them without further fees.

What Verizon is saying is "look they need to pay us to upgrade this link" which on the surface doesn't sound too bad. It kinda makes sense. If I used more bandwidth they would charge me more. But the thing is that I am already paying them to make sure that the connection to the internet is uncongested. They are deliberately allowing it to get congested to try and bill both sides of the transit.


This. Completely this.

They may have a peer balancing agreement, which is not being held up, but they're also still being payed by their customers to provide access to content outside of their own network and Verizon is enforcing a limitation from another agreement (the peering agreement) that is affecting their ability to provide the content their customers want to have access to.

You can't have it both ways. Or... I guess you can, as Verizon has shown.


Commercial ISPs vs. Consumer ISPs is a useful way to frame the issue. From an economic perspective, the big difference is that the Consumer ISPs generally have a monopoly over the Consumer endpoints, while the Commercial ISPs do not have a monopoly over Commercial endpoints and must negotiate a competitive market.

This obviously gives the Consumer ISPs pricing power in all of their business dealings (including any peering contracts that they force the Commercial ISPs to sign). The fact that Verizon owns RedBox (a Netflix competitor) & sells TV packages is just further motivation for hampering the interconnection.


Isn't redbox owned by Coinstar? But has agreements with Verizon?



You realize that Verizon is both a consumer and commercial ISP, right?


Yes I'm well aware. A good friend of mine works on the commercial side. He is a programmer that is basically in charge of all their route-finding algorithms internal to the commercial side of the network. It is completely separate from the consumer side.

On the commercial side they provision a link from point A to point B (which is what his software figures out) and it's dedicated bandwidth. They are never over-subscribed.

I'm almost positive that this is not how they do things on the consumer side. Otherwise the Netflix debacle wouldn't be happening. But basically the consumer side is always over-subscribed and by a reasonably large factor. They're counting on consumer traffic to be bursty which it normally is. Netflix isn't bursty the way they're used to.

I sympathize with the idea of expecting one thing when building a model and seeing it play out differently. It sucks and it can cost a lot of money to make good on promises that you've made. But remember that Verizon is the one that made these promises. Their customers didn't put a gun to their head and say "promise me these crazy high download speeds or else!", Verizon did it willingly to try and steal customers from their competition.

Because they did it to themselves I have little sympathy.


Netflix debacle is Netflix choosing not to pay for CDN and instead asking ISPs to take thier CDN for free or we overload your interconnect ports and blame you for it on our site.

See first page about CDN for a description of how they moved from CDNs paying for access to demanding ISPs installed thier CDN for free. http://arstechnica.com/information-technology/2014/07/how-co...


Netflix has actually addressed this in one of their blog posts: http://blog.netflix.com/2014/03/internet-tolls-and-case-for-...

> But when we ask them if we too would qualify for no-fee interconnect if we changed our service to upload as much data as we download -- thus filling their upstream networks and nearly doubling our total traffic -- there is an uncomfortable silence. That's because the ISP argument isn't sensible. Big ISPs aren't paying money to services like online backup that generate more upstream than downstream traffic. Data direction, in other words, has nothing to do with costs.

> in other words, moving to peer-to-peer content delivery


I would love to see Netflix buy Backblaze and offer free computer backups to every customer, just to fuck with this specious argument.


Of course, it wouldn't fuck with anything since Verizon would just change their argument.


Verizon is charging me for access to the internet -- then charging the internet for access to me. That is double-billing, and it's egregious.

Verizon is selling me a 50Mbps connection, but if I use it, for even a little, they get upset and say they are over capacity. This is the same company that over-subscribes their network on purpose, as most ISP's do. They will come into a neighborhood with a 1Gbps link, then sell 50Mbps to 100 homes, in hopes that not everyone uses the connection at the same time, nor for long duration.


This is a weak argument. Internet connectivity simply is not measured as a 1-dimensional value. It is effectively infinite dimensionsal.

The marketing papers over this in misleading ways, and that is bad, but if you wanted to pay for your own dedicated bundle of wires, like a businesses office building does, you could get it.


>This is a weak argument.

>The marketing papers over this in misleading ways, and that is bad

It's not a weak argument at all and the misleading marketing is exactly the reason it is not. In fact, Verizon has oversold bandwidth (or at least offered it for less than they are willing to accept).

But, what you seem to be effectively saying is that customers are at fault for buying an advertised product which touted the benefits they desired at a price that was acceptable.


> Verizon is selling me a 50Mbps connection, but if I use it, for even a little, they get upset and say they are over capacity.

If your last mile network was built to allow you to saturate your 50Mbps connection 24/7 your bill would be much larger than it is now. So much that you probably wouldn't pay for internet service, so oversubscribing is a totally normal and rational process when building out a network.


If I accept your premise as true, then please explain:

1. Kansas City, KS

2. Chattanooga, TN

3. Provo, UT

4. Austin, TX

These places all have fiber to the house, and supposedly gigabit speeds. Two of them were built out with zero help from Google, so Google can't be making huge losses on the internet to prop up other parts of their business.


If everyone on those networks were saturating their connections 24/7, then yes, there would also be network performance issues.


>>> If your last mile network was built to allow you to saturate your 50Mbps connection 24/7 your bill would be much larger than it is now.

> If everyone on those networks were saturating their connections 24/7, then yes, there would also be network performance issues.

Okay, first it was the last mile that was expensive. Now it's the back-haul that's expensive.

You do realize that you're contradicting YOURSELF here, right?


I'm not sure how you are misunderstanding or purposefully ignoring what I'm writing in every post.

Your network no matter the bandwidth to your house is oversubscribed. If everyone was using their network connection at max capacity 24/7 there would be network congestion issues.

Thus the arguments that "I've been sold 50Mbps/down I should be able to use it at max capacity 24/7" without issue are incorrect.


> Your network no matter the bandwidth to your house is oversubscribed. If everyone was using their network connection at max capacity 24/7 there would be network congestion issues.

No, not necessarily. That's how Verizon Residential chose to build out their network but that's not the only way to do so. The commercial side of Verizon manages to provide their customers with 100% of SLA throughput 24/7 no problems. Technically this is feasible.

I would agree that I might not be able to get 100% utilization at the price I am currently paying. That is a point I will concede. BUT, SOMEHOW Google Fiber in three cities and Chattanooga is also supplying gigabit fiber. And they're not charging $10k/mo per person to do so. Could everyone use the full gigabit connection 24/7? I seriously doubt it. But if they're oversubscribed by 10x on gigabit and other places are oversubscribed by 10x on 100 Mbps then they are successfully delivering 10x the internet for the same price, or perhaps 50%-100% more. It's not linear that's for sure.

For your argument to hold water if Google Fiber was 10x faster than Verizon, then Google Fiber would also have to be 10x as oversubscribed as Verizon. I very, very seriously doubt that this is the case.

> Thus the arguments that "I've been sold 50Mbps/down I should be able to use it at max capacity 24/7" without issue are incorrect.

This is a straw man argument. You are not engaging in an honest dialog. You seem to be purposefully obtuse. People watching Netflix in NO WAY constitutes 100%, 24/7 utilization!

How about "I've been sold 50 Mbps service and I want to use 3 Mbps of it for an average of 3 hours per day, which is a measly 6% of the advertised speed and is only 12.5% of the 24/7"

There are two ways to do the math here. One is based on peak utilization, where we assume that everyone watches Netflix at the same time, 100% identically, say from 6pm-9pm. In that case Verizon can be over-subscribed by a factor of roughly 16x (50Mbps/3Mbps) and still keep their customers happy.

The other is to assume that everyone watches Netflix perfectly randomly. In that case let's assume 3 hours per day. Here Verizon can be over-subscribed first by the 16x (since people don't use all 50 Mbps) and further by another 8x (since people are using the service for 3/24 hours per day). That gives you an over-subscribed factor of 128x.

The truth of course would lie somewhere in between 16x and 128x. But what we're seeing is that Verizon is over-subscribed by a factor of more than 16x otherwise people wouldn't have any problems watching Netflix. Put as percentages instead of ratios, customers are asking Verizon to provide between 0.75% and 6% of what they promised to deliver, and Verizon is refusing to do even that.

Further we have evidence via a blog post from Verizon that their entire network is NOT oversubscribed by a factor of 16x as it's only at 40-60% utilization. What is happening is that Verizon is throttling their users at the peering point. It's a great gig if you don't get caught. http://publicpolicy.verizon.com/blog/entry/why-is-netflix-bu...

If we assume that 100% of all traffic on Verizon's network is from Netflix that means Verizon could double the peering bandwidth to L3 before their network got congested. Is that a valid assumption? I don't know, but I sincerely doubt it's 100%. During peak hours it might make up 50%. In that case doubling the peering bandwidth would increase utilization from roughly 50% to roughly 75% which is perfectly acceptable.

It must take some pretty fantastic mental gymnastics to convince yourself that a provider who is not able to make good on 6% end-user utilization is totally in the right and their upset customers are irrational and wrong.


Verizon is charging me for access to the internet -- then charging the internet for access to me. That is double-billing, and it's egregious.

Both me and my brother get billed when we call each other on our cellular phones! Help me!


Which is not how it works in the rest of the world. Only U.S. consumers are so used to getting screwed that they would defend it as normal. Verizon loves double-billing so much for cell service that they want to do the same with the internet.


There are other countries that bill for incoming calls. I know of at least Singapore that does it.

Also, your phone provider does get charged for each incoming call. They just eat/redistribute that cost, since for normal phone user it's likely to be less than a penny per month. But try running a conference service and you're definitely going to get billed for incoming calls (apart from some weird force-subsidised areas where you can get paid to receive calls)


The difference is that in Singapore, you can get charged for incoming calls depending on your contract, but you _don't_ get charged for incoming SMS.

From a Singaporean perspective, that's the most bizarre part of mobile service in the US. I need to get an unlimited texting plan so that I don't get billed for people sending me texts? Whoa!


In India, when we were younger, we'd sometimes all call someone on their birthday. Before you know it, there'd be eight people at a time (with people dropping in and out to wish) on a conference call held at the birthday person's number. That person would pay nothing for this.

I'm pretty sure you can still do this.


U.S. consumers think it's extremely unfair that a poor guy with a landline should subsidize the rich guy with the cell phone or the satellite phone. It might cost me $1 a minute to call my neighbor? No thanks!


Well, that is a damned strange US custom that isn't the case in many other places. It is certainly not the case here in Europe, for example. The receiver pays.


Don't you mean caller pays? Otherwise I could make someone else spend money just by calling them.


Caller pays, indeed.


No one pays for incoming calls in India. This is one of the reasons why even very poor have cell phones. Unfortunately american cell phone users aren't that lucky. Both the caller and callee have to pay. Verizon being cell phone company probably want to charge both sides. Most likely they will win.


This is a very bad analogy. When you sign up for mobile phone service, you're also agreeing to usage fees or paying extra to get "unlimited", which case, you pay nothing for those calls.

When I sign up with an internet provider, I am agreeing to pay a fee for open access to the internet at large. I'm not signing up to pay for each piece of content I access or each site I connect to.

I've never agreed, to the best of my knowledge to pay for access only to content served at a rate limited by the reverse utilization of the peering connection that content provider's content can be accessed across.


To use your own analogy -- it would be as-if your phone provider charged you to call me (they do), my phone provider charged me to receive the call (they do), then my phone provider charged you for making the call (they don't).


Actually, they do (on that last point). Which is why Google Voice won't route calls to certain numbers. It's called a termination fee, which works like this: I pay my local phone company, and also pay a long distance carrier. When I call you, your local phone company charges the long distance carrier a fee, which that carrier then passes on to me. (Well, it used to work this way, but almost everything is flat rate now).


In UK when Ann calls Bob it is Ann who pays for the call. Bob doesn't pay anything. When Bob calls Ann it'll be Bob paying for the call.

Are you saying that in the US when Ann calls Bob that both Ann and Bob will be charged?


For POTS, only the caller gets charged.

If you are using a cell phone or a satellite phone, you pay for your connection to that network, whether or not you started the call.

You certainly don't make it so that grandma using her POTS has to pay for the guy with the more expensive connection. That is viewed as extremely unfair.

OTOH, using the cell network generally has free calling to anywhere in the lower 48. It's not the old physical wires that are the expensive part, it's maintaining and upgrading all the cell towers, and so people pay for that usage. Once you've paid to connect your call to the wired network, you can do whatever within very wide geographic borders.


It's actually a historical issue. In the US cellphones orginaly used standard POTS numbers while in other countries cellphones often had an easily distinguishable format. This means that in the US granny wouldn't know she was calling a cellphone so charging her extra would be an unpleasant suprise, while in other countries she knew and was thus implicitly accepting to pay the charge.


But if you answer on your land line you don't get billed by his mobile provider.


I know the answer to this one! We need these guys to setup at the other end of a Level3 link, and then Verizon customers can push them one byte for each byte they get from Netflix:

http://devnull-as-a-service.com

Imbalance fixed, but I don't think that is what Verizon wants.


Seriously though: what if Netflix worked with Level3 to set up an efficient Level3-owned /dev/null server in the carrier hotel AND have every Netflix client that connects over Verizon upload random bits to that IP address whenever buffering is complete or the movie is paused? Then the rates would balance out, and Verizon would no longer have a contractual excuse to NOT hook up those cables to the Level3 endpoint. The only way for Verizon to mitigate this would be to deny their end users the upstream bandwidth they've paid for but are not using (cue user outcry), or to smear Netflix even more so than they're already doing (cue more chances for the PR to backfire). Verizon would hate Level3 for this, but they already do, and they couldn't do anything about it without denying their end users Netflix access altogether!

Of course, since the NSA would need to listen in on all that new random-bit traffic (since Netflix is obviously the new Skype in the terror playbook), taxes would need to rise significantly in order to pay for more NSA bulk-data-collection servers. But that's just the cost of living in America...


Then how would you explain Verizon's refusal to host Netflix content inside its network that's under-utilized?

Seems to me it's a win-win for everyone, including poor border routers. Except Verizon then wouldn't make an extra penny from that efficiency, hmm?


Verizon gave a very clear suggestion for Netflix: work with other providers as well.

Why should they have to do that? Level 3 has open ports now (so they claim). It doesn't matter which provider the traffic is coming from, either Verizon wants it or they don't (and apparently they don't, even though their core is no where close to saturated).

The balanced traffic requirement is bogus anyway. Its not as if Verizon offers their own video streaming service that I can subscribe to as a Comcast or AT&T customer. So they're never going to see balanced traffic in the age of video streaming.


It is entirely about the relationship between Verizon and their customers. Verizon's customers want these interlinks, they just don't know it. This whole thing started when Netflix posted a message to their customers explaining it. Verizon is trying to defend themselves against the complaints of their customers.


The concept of asymmetry as you're using it doesn't make sense in this situation.

- If you push traffic to my network it is because someone in my network requested it. - Incoming and outgoing traffic are technically no different. - It makes more sense to pay to receive traffic than get paid to receive traffic. That's actually what VZ customers pay for and that's why not many people complain about ADSL speeds.

If I were Level3 I would actively partner with VZ alternatives in the US, announce it to VZ customers so they can move away and then proceed to remove any peering with VZ. Enough bullshit already.


I don't think in the history of the internet there has ever been a "balanced" traffic arrangement. The majority of people are always going to be requesting more than they are uploading.


As a residential subscriber this is contractually the case. ISPs do not allow the operation of servers in residential contracts and they are almost always provisioned with asymmetric speed. 50 down, 5 up, etc.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: