The IP Development Network Blog

One advantage of being your own boss is that you can tell granny how to suck eggs without getting the sack. So at the risk of teaching an old dog old tricks, today's article is going to go back to basics and draw a picture of a baseline network.

Under normal circumstances we might just draw a cloud in place of the detail below because the detail, like the network itself, is only relevant when there are problems that need to be understood. Once those problems are solved, we can all go back to drawing clouds.

The time to examine those problems is now. This week, Ofcom launched its Next Generation Access (NGA) consultation which appears to run alongside the efforts of the Broadband Stakeholder Group (BSG) activities and Stephen Timms (MP)'s efforts to convene a summit. Whether all these acronyms offer parallel efforts or something more in keeping with the market trend for convergence, we shall see.

While the focus is on the local loop and the peak capacity that can deliver, the situation is a lot more complex than simply opening the gates yet further.

Transition
We are in transition from dial up to "true broadband". In fact we may always be in transition because after true broadband we should probably expect the next next-generation lobby to be pushing for something that they might have to call "true true supersuperfast broadband, honest". Drawing lines in the sand though - at say 100Mbps - gives a target that helps us plan for the next stages in the evolution, but it is worthwhile noting that there will never be enough to satisfy the high end users.

We know that video streams are an elephant in the room, which some have estimated will account for 98% of internet traffic within two years. This is a serious problem so that means getting into more detail about the network and the factors that are creating the problems.

The first point to make is that for all the fuss about Fibre to the Home (FTTH), that is only one part of the jigsaw in delivering video to people's homes. The local loop is one of four major choke points on the access network side that need to be considered. Sitting alongside those access issues are considerations of how the internet routes packets and how and where those packets are stored.

In fact, for all the hype, it may be that FTTH is one of the least pressing of the issues that stand in the way of the internet's ability to deliver the larger and much more lumpy video traffic on the horizon. At some point, the last mile will again be the most significant bottleneck - as it was when all we had was dial up - but right now, how many people would be able to use a 100Mbps local loop if it magically turned up on their doorstep?

Would the architecture support it? Would there we enough capacity in the home and on the core network to use it? How many knock-on issues would we need to solve before we spend the money on fibre in the local loop? A chain is only as strong as its weakest link...

The Home Network (x, in the diagram above)
A good place to start: it is undoubtedly the most complex issue because of the anarchy that exists in this space. There is no control over end points and how they are attached to the network, or indeed which network they are attached to. Security? It is best not to ask - denial is a wonderful thing...

Consumers are truly left high and dry to build the wireless / ethernet / homeplug network for themselves. If they are really lucky, they can get a friend / son / daughter to do it for them.

This makes the introduction of new hardware and new services that would use the 100Mbps a significant challenge. The customer may not have a network, or it may be "a bit flaky" such that when they come home with shiny new CE equipment, they are disappointed (or worse) to find that it doesn't work. So they make a call to the ISP but after a long wait, they find that their "service" provider isn't there to help.

Do you phone a friend every time you want to install a new device in your home? How soon before your DIY network starts to creak and your friend's generosity starts to get seriously tested? If everyone suddenly had 100Mbps to the home today, very little of it would be usable because the capacity of the last yard is significantly below that. Before FTTH, we need a solution that simplifies the home network and extends management of that to a real "service provider".

The Local Loop (y)
No-one is happy with the current state of affairs. That is not to say that everyone agrees that access networks are too old and slow and are in dire need of an upgrade - LLU has yet to be fully exploited, so perhaps we should start getting the best out of that? There are two conflicting priorities that need to be managed: ultimate speed is one of them, but at least as important is ultimate reach.

Looking first at speed, there seems to be a clear assumption made by many that we need more than ADSL2+. This point is worth explaining because this is not about headline speeds: 24Mbps for all would be enough for a fair few years. But ADSL 2+, like ADSL 1 speeds degrade with distance, so only anything substantial can only be delivered in real time over relatively short distances.

This table is from the BSG report, Pipe Dreams. Links to articles covering that and other articles can be found on the Digital Divide section of this site.

In summary, only 30-40% of the population are close enough to their exchange to get 8Mbps or more on copper. You can get more on cable but cable also covers less than half the population, similarly concentrated into densely populated areas. For some therefore, there is a vibrant market and the local loop is no barrier at all. Certainly not one requiring life support from a quango or two.

The role of the quango should be to concern itself with the areas where the market does not have an answer. In the local loop, this includes a significant number where there are signs emerging that the market for connectivity beyond LLU will fail. This failure will occur because the market needs huge investments by Openreach to shorten the copper loops but for the monopoly it is hard to see any extra revenue to pay for the new investment.

So 60% of us might be stuck with the speed we have today - it doesn't matter whether we use ADSL1 or 2+, the result is the same because the line length is the problem, not the technology at the exchange. And because the copper replacement case is so weak, you might have to move to a new build estate to get fibre to your home...

BT IPStream and ADSL1 (1)
There are very clear signs here and now of market failure in the provision of basic broadband access. Fortunately this only impacts a very small minority who cannot get 512k or more - a rare enough occurrence that some even appear as "news" these days.

This market failure today is very small indeed, but there is the prospect of many more (perhaps 15-20% of the population) getting left behind in the rollout of LLU. Although there is no doubt that competition here has led to cheaper products for all, those price reductions came at the expense of the digitally divided for whom competition in the local loop is a double whammy.

Competition means that investment from all players, including BT, has focused on the denser locations where the business case is best. For most, that means higher speeds and lower prices but the money taken out of the value chain through price competition, is money that was once used to cross-subsidise services where the business case didn't make sense. For the minority on the other side of the divide, LLU enshrines a two tier system.

Two Tier Pricing
A two tier system means two tier pricing, but it is worthwhile understanding what that two tier system means. It does not mean people miss out on broadband: almost everyone can get affordable broadband connectivity if they want it - 99.x% have access to some form of broadband and prices are universally below £20.

Two tier pricing may mean that the basic product is available for free on LLU exchanges and for £10-£15 more on IPStream, but even that is not the problem. The problem of the two tier pricing system as it is evolving, is the impact that it is having on the affordability of broadband capacity once you have the basic connectivity.

This manifests itself as usage caps and fair use policies because broadband capacity (as distinct from broadband connectivity) is hundreds of times more expensive on IPStream than on LLU. For these consumers on the wrong side of the digital divide, competition in the market means that the cost of actually using the service is prohibitive.

Every action has an equal & opposite reaction
This two tier system is the direct product of "managed competition". IPStream's prices are maintained artificially high to allow room for competitors to build their own infrastructure at a cheaper rate than they can lease capacity from BT.

Solving the two tier pricing problem may distort the competition that has been so carefully created because it means BT selling IPStream at rates comparable to LLU. This would undoubtedly stop future LLU investments and throw into doubt the commercial viability of many existing deployments. It also requires that BT have an incentive to cut prices for the least profitable exchanges in an environment where there is no competitive pressure demanding that they do so.

There is a significant difference between now and 2004 when BT held back enabling the least profitable exchanges with ADSL1 because the promise of returns was non-existent. The difference is that now BT has to compete with LLU; in 2004 they were a monopoly and could cross-subsidise more effectively.

The key question that we need to be clear on is who are we trying to deliver fibre to? Is it the top x% where a little shove makes the business case work? Or are we going to let the market work on that while aligning regulation and politics to deal with the bottom y%?

Backhaul (z)
Backhaul is an issue that is best summarised quickly here. There are more details in a previous series of articles written by Keith McMahon and I a few months back.

Backhaul has been a severe inhibitor to the development of broadband in the UK for the past few years but it appears that BT have been quietly upgrading capacity of even some of the long tail of exchanges to fibre (I have heard anecdotes of exchanges on the 95th percentile being glassed up). This, combined with their new BNS product for LLU operators described by Keith in the above article means that we are much closer to removing capacity constraints in backhaul.

That is not to say that backhaul is universally cheap though, as the model is heavily distance dependent and profitability is reliant on customer density. The pricing scheme is built to deliver service to those with their own core networks close to the exchanges being unbundled. The model is designed to clearly benefit the decreasing number of larger players.

Backhaul competition exists but the BNS introduction certainly took the price floor down a few notches. Additionally, there is a subset of exchanges colocated with the core network itself but these have a much easier life because there, core networks are cheap and plentiful and the backhaul circuits are simply internal wiring.

Would backhaul survive an overnight upgrade of local loops to 100Mbps? For the vast majority of users, the answer would have to be yes but what would break would be the business model because backhaul pricing is based on today's usage and not what you would see with 100Mbps in the last mile.

Backhaul Pricing
This final point deserves explanation because the way that prices are set is a self fulfilling prophesy. In simple terms, there is a "budget" for backhaul - ISPs and even consumers buy as much as they can for that budget. They will expand their usage gradually to fill it and then throttle back use so that they fall within the budget, until the price falls and the cycle starts again.

Dropping prices means more capacity would be available within the budget, but it does not often lead to absolute gains in total revenue because people still spend the budget. Of course it is recurring revenue so you need to keep cutting prices to keep the business - most assets require between 3 (hardware), 5 (system) and 15 (infrastructure) years of use at a recurring fee to pay for itself.

The problem for those selling capacity is that when you drop the price, it takes time to recover the revenues you have given away in the reduction and even when you do, you often find yourself back to square 1 as the throttling caps the upside. So it makes most sense to hold tight and wait for someone else to make the first move.

This point is clearer working through an example... Say you have 100Mbps of used capacity at £5 per Mbps and are charging £10 per Mbps to your customers. Your total cost is £500, your revenue is £1,000. Say that you then upgrade that circuit to 1Gbps at £1.25 per Mbps (total cost £1,250).

At that point in time you are making a £250 loss - what do you do? If the market is saturated you face a problem because you somehow need to be able to get users to pay more than their budget (£1,000). Even if there is still some growth room from new users, do you hold on and sell slowly at £10 per Mbps? Cut the price by the same proportion as the cost to £2.50 per Mbps (total losses now £1,000 and a breakeven point of 5 times your existing sold capacity)? Or something in between? Does this change if I tell you that your competitor is selling at £4 per Mbps...? £3.95, perhaps?

The problem is that prices are increasingly lumpy with ever larger upgrade steps (100Mbps to 1Gbps is 10x as is the next step to 10Gbps). Such steps cause problems because available capacity increases far in excess of demand. Pricing on the basis of availability would leave the owner with pennies in comparison to pricing on the basis of usage, although the latter has the effect of stagnating growth.

Backhaul Competition
Extending the core networks to get increasing numbers of exchanges on-net is the only way to take the recurring cost off your books, if operators want to. Putting their own fibre into exchanges sounds attractive, but it is even more attractive to wait until someone else does and then needs to sell the new capacity. At that point the wholesale customer can start to drive the price down aggressively at the expense of the facilities-based carriers who undercut each other progressively downwards.

There is an incentive problem for operators who may be considering investing in their own backhaul builds. They are better off waiting for some other idiot to make the first move...

If competition is going to stretch into the provision of local loops, it must first address the much simpler issue of backhaul competition. It is simpler because it is a fraction of the cost, but the issues are the same: protection of existing assets, build cost, site access, asset sharing, equivalence, price fixing, price regulation, period of regulation, certainty, etc. Perhaps it is a safer place to experiment with various solutions?

Content Issues
In simple terms, hosting content on your own servers is cheapest, next comes content on peer networks that can be reached through internet exchange points while Transit is the most expensive.

Transit originated as a way to get access to US content, but more and more of the big US properties are now hosted on caches that can be reached through in country peering (from my ISP, you can get to google.com through LINX). Transit still plays a big part because it is the only way to reach everything else (youtube.com goes through transit). The difference between transit and peering is that you can't transit peer networks as a general rule.

Transit networks themselves host a lot of content but their value primarily lies in that you can go over one of these networks to reach something the other side. So instead of maintaining thousands of smaller circuits with everyone else, Transit takes care of that in one interface.

How would content hosting be impacted by 100Mbps in the last mile? It might be ugly for a while as the shock of an overnight upgrade kicks in, but as we are unlikely to wake up tomorrow and find the tooth fairy has given us all fibre, we have some time to consider the impact on the electrical grid.

Space is not a problem: in the late 1990s data centres the size of football pitches were constructed which are still being filled now. Network connections are not the problem as most are on multiple fibre rings.

Power on the other hand is a real concern, particularly given climate concerns and the ever increasing cost of energy. We really do not understand the power consumption increases driven by fibre to the home - this cannot be ignored as delivering new electrical capacity may be even more problematic than laying the fibre.

Bigger Lumps of Data
Video is not necessarily going to be the most popular internet application but it doesn't have to be to cause the predicted impact. It is not where people spend the most time that necessarily drives the traffic: a second of HD video is 65 times as much data as a second of high quality music. Put another way, 1 hour of video is 65 hours of music or 315,000 page views on google.com...

For video files of that size, there are storage implications, but storage capacity is far more advanced than network capacity so it becomes more a question of where do you keep it to minimise the distance travelled and subsequently the cost you incur. If you can control it there are suggestions of charges for premium delivery to help monetise the downstream access network assets.

Whoa! Network Neutrality alert - but looking at how this is being played out, there is a question whether the content applications will cooperate to allow the ISP to exert such control. P2P is an example of how content companies are trying to work their way over the top of ISP platforms.

P2P vs Client Server
The concern with video applications is understanding the direction the market will develop. Will it be the wild west all over again with P2P data everywhere (forcing much of traffic onto transit networks) or will the video market evolve to work with the networks (much of the content locally hosted). At stake is the bill that ISPs pay transit providers for global access.

The choice of application is as much political as it is technical. If P2P wins, it will be increasingly hard for ISPs to do anything about monetising the increasing volumes of content but it may deliver an inferior user experience - something the ISPs can comfort themselves with. ISPs would be much happier with client server as that is something their networks have been built around and something they can control the quality and cost of.

Fibre and 100Mbps access certainly plays into the P2P corner as it blows away one of the fundamental limits of P2P - upstream bandwidth. In a DSL environment there is only the capacity to create perhaps a 10th of the capacity there is the potential to consume. In a fibre environment, it can all be P2P.

I believe that we need to look at how and where the networks route P2P and move routing closer to the edge to reduce tromboning. This is because applications would perform much better and network demands may well be lower, even allowing for the additional Layer 3 technical and operational overhead. Geo-aware P2P might work for everybody, but that is a story I have written up before.

Busy Hour Planning
There is a huge difference between how computers are used on the internet and how TVs are used. Watching television is much more heavily concentrated: peak audience (of all channels) is around 2.8 times the average audience over a 24 hour period, whereas for web surfing this is nearer 2.1. P2P actually generated very good peak load efficiency because the ratios of download applications that use P2P is around 1.4.

What on earth does this mean...? In simple terms, you need to provide 33% more capacity for watching TV as you would for viewing the same volume of data on the web because you have to build for the peak unless you want congestion on the network. Furthermore, congestion for streamed services like TV is far more serious than for web access (where building to the 95th percentitle was commonplace).

The chart shows how the usage of various applications varies and where the peak loads are on the respective networks. There is no weighting for file size - the area under each line has been rebased to 1,000 units. The aim is merely to show the peak to mean traffic profile of video is significantly higher than for web and P2P.

This reiterates the point above that for the same volume of data, you need more network for video than you do for other internet applications.

Which makes a situation which is already very bad, even worse - the capacity that we actually use today is only a fraction of what is available on existing local loops. Average usage of around 5GB per month on a 2Mbps circuit uses 0.8% of the connection's maximum capacity. There are over 8 Exabytes (8,912 Petabytes) per month of unused capacity on existing networks.

A 2Mbps link is enough capacity to deliver 146 hours of 1080p programming per month - the average household watches just over 100 hours per month. What we have today could deliver what we need tomorrow.

It highlights the inefficient use of the total available resource... The problem is "on-demand".

Conclusion
Video is the only application that looks remotely like driving demand for fibre. Assuming for today that we need to move video over from its existing broadcast platform - a case worth exploring in detail in another thread - it is clear that there are a number of key areas where we are not ready for fibre to the home.

We do not have the ability to deliver service because of networking issues in the home and commercial models in backhaul and hosting are going to have to change in light of the new traffic demand. But these are functions of evolution that will follow the technical capability as it grows.

Where there are serious questions to be asked are in the supply of power for the next generation capabilities and in the efficient use of the resource that is in place today. Every routing hop is another drop of oil gone forever and do we need to build nuclear power stations next to data centres to support demand there?

It is also clear to me that we are not using what we have in place today. Perhaps we should stop to think about that too before ploughing huge sums into delivering yet more peak capacity?

The removal of the bottleneck in the backhaul means that it is only the commercial model preventing full-time wirespeed usage of connections. For 70%, this is 2Mbps plus. Even if you need 10Mbps for the video itself, technology is evolving that predicts what a user might want "on-demand" and pre-loads it for viewing at 10Mbps.

This offers the network provider a way of maximising resource usage. If you can fill the unused capacity on the network today instead of pushing the headline speed, you don't need the expensive infrastructure upgrades.

There are clearly areas where there is a vibrant market for connectivity because of recent regulatory efforts to encourage competition. But this risks leaving a subset of the population behind with access speeds below what might be necessary - 30% cannot get 2Mbps. This is the area where lobbying and regulation should concern itself, not with the drive to 100Mbps.

Labels: BT, digital divide, FTTH, next generation networks, Ofcom, timeshift TV, video on demand

# posted by Jeremy Penston @ 9/28/2007 12:39:00 PM

Friday, 28 September 2007

Back to Basics