« Summary Slides: How to make money from video | Main | Comment: BSG report lost in translation? »
Joost: analysis of a bandwidth hog
By jpenston | April 16, 2007 |
|
If you own networks, you will be worried. Even before the Viacom content deal, nervous glances were being cast in the direction of what was then known as The Venice Project. Because of the owners’ history (Skype & Kazaa), their aim to revolutionise TV had to be taken very seriously from the outset. At the very least, they could certainly afford to give it a whirl…
So now that Joost is here, what do we know about the application? Here are some key facts:
- it pulls around 700 kbps (320 MB per hour) off the internet and onto your screen
- it sends a lot of that data on to other users - about 220 kbps (105 MB per hour) upstream
- it is ad-funded. Viacom pay Joost and provide content
- the content is filmed to broadcast standard
- the picture quality for the videos themselves is quite low - it is really quite blocky
- their PR has claimed “near-high definition quality” - something they are a long way from at present. Look for this to come as the service grows
- it has a very slow zap time (channel hopping is not easy)
Once I had been able to investigate the traffic flows, all of this made a lot more sense. The reason Joost is distributed at such low resolutions must surely have something to do with the number of peers on their network - or more specifically the lack of them. In my sample, 26% of UDP bytes (the media itself) came straight off the Level3 network while a further 21% came from a very small IP range on the Infonet network in Belgium. I strongly suspect that these are centrally hosted nodes which offer very high bandwidth (at a commercial rate) much like any other content owner would buy with a hosting package.
In other words, it seems that the service is relying much more on client server architecture and much less on P2P than you might expect at present. It makes sense if you consider the need to give a reasonable quality to the early adopters, but all those L3 bandwidth charges can’t be cheap. Maybe that’s why they aren’t sending at higher resolution - because they would have to pay for it? I wonder if such considerations will be carried forward when this burden can be offloaded onto peers? Will this give the developers a conscience for the ISPs that are delivering their services? I wonder…
There were 299 IP addresses that I captured in a 5 minute Ethereal probe into Joost. 17 of them delivered the 47% of bytes that I have already referred to - those that look very much like a traditional client server architecture. The other 282 addresses look like other Joost users just like me - the peer to peer that you would expect from the founders of Kazaa.
I did this analysis because I wanted to see how intelligent the routing of the application data is. I was curious whether developers had made efforts to limit the impact on ISPs transit and interconnect bandwidth. I hoped to find that some consideration had gone into reducing tromboning of traffic…
To some degree they seem to have done so: just under 41% of bytes appear to have come from other UK ISPs, but what I really wanted to see was the majority of the traffic coming from other subscribers on my ISP (Zen). This would have meant that the cost of the application was squarely in the hands of the ISPs to fix - being high only because of their inefficient routing architectures. Alas, there was no such luck: less than 0.4% of UDP bytes stayed within Zen’s network.
If I’m losing the non-techies with all this, please bear with me. The important point is that Joost’s traffic profile shows a highly inefficient use of bandwidth, with packets flying around all over the world in order to reach me. I even received a small amount of content from subscribers in countries such as Japan, Lithuania, Portugal and Romania. In all my 5 minute sample showed packets coming from 247 IP addresses, on 82 ISP networks in 20 countries. 90% of bytes came from the top 30 IP addresses.
So what about the upstream? One thing that struck me was that it was a very different collection from those that were sending to me. I noticed that 19% of downstream packets came from Telewest and a further 7% from NTL (all now Virgin Media to marketing folks of course, although the networks will retain their individuality for some time to come I suspect). So I was getting 26% in total from UK cable users and yet I was sending only 3% of my uploads back to the same providers. My upload data was concentrated much more heavily (14 IPs receieved 90% of data sent), but with a slightly longer tail too. In the 5 minutes sampled, I sent to 266 IPs on 90 networks in 22 countries.
So who was I sending stuff to…? 74% stayed in the UK and 18% went to Spain (how very English) but if I was hoping that the stuff I was sending would stay on Zen’s network, I would be disappointed: only 8% of my uploads stayed on Zen’s infrastructure.
The point of this article is to highlight how disconnected applications are from the networks they need to reach their users. This is particularly the case with applications that come from “the web”. I do not know whether this is because network companies have not shared their issues with the wider market or whether it is because of the anarchy that prevails online. If you had to pay for all the capacity you use, you would not design an application that works like Joost does!
ISPs need to share the responsibility for the evolution of the market. We are where we are now due to a historic reluctance to address previous behaviour online (eg. spammers, pornography, copyright abuse). Network providers have previously argued that they were bit pipes with no control over (or responsibility for) content, which suited them at the time given that such uses drove traffic volumes and revenues and were expensive to address. Now they are starting to see that content control is needed for their own preservation, they need to understand that this has much wider implications: if you can control Joost, you can control allofmp3 and child pornography as well perhaps…?
Content distributors also need to appreciate that they cannot keep up the pretence that the cost of bandwidth is not their problem. They have the most to gain from the incremental use of bandwidth (and also the most to lose in the event of a bandwidth war).
Topics: Uncategorized |



April 16th, 2007 at 1:50 pm
Hey there, I’m Joost’s network architect. If you want, you can read my presentation from UKNOF. It’s online at;
http://www.uknof.org.uk/uknof7/MacCarthaigh-Joost.pdf
I have an updated version here too;
http://www.stdlib.net/~colmmacc/slides/Joost-network.html
We have many gigs of transit, and are adding more. I’m not sure who claimed it’s near HD quality, I like to think it’s about NTSC, sometimes better, never quite PAL.
We have some efforts in the code to save transit costs, there is very very basic prefix awareness, and we’re adding AS-level awareness using live BGP data. I have looked at adding AS adjacency information, ie prefer AS-adjacent peers, but it’s a lot of work and the US internet is relatively poorly mapped, so I don’t think this will come soon.
April 16th, 2007 at 2:13 pm
Thanks for posting this colmmacc. It is good to hear that Joost is thinking about how to solve these problems.
Clearly, there are serious architectural issues on the network side that need to be overcome. It may be that Joost is the incentive the network industry needs to get this right?
May 25th, 2007 at 11:32 am
Update… It turns out that the stdlib.net files contained a little more info than was at first obvious, including details of Joost’s pre-launch plans.
The uknof slides are still up there (although the download is 18MB), without the secrets, but the stdlib files have been taken down.
The Joost site contains a forum discussion indicating that these will be reported, but there is (now at least) a 403 error when I try and get the files.
For more on Joost from me, the latest post can be found here.
June 7th, 2007 at 9:26 am
Hi,
Thanks for the interesting info.
I am being bombarded with UDP packets on port 13775 from a belgian IP address 91.86.103.252.
every 20 seconds for the last 12 hours…..is that a possible consequence of running Joost?
H
June 7th, 2007 at 10:15 am
I’m sure an engineer could give you a better answer, but my suggestion would be to turn Joost off and see if you are still getting the data. If you are, it’s not Joost. If it stops, it is.
You may need to exit it by right clicking the icon on the tray on the bottom right corner, as when you put it on standby, Joost is still active - unless you can find the checkbox on the preferences that alters this.
You can also check the RIPE database that shows you that the IP you give is on the Mobistar network. I’m not aware of any Joost hosted servers there (but could be wrong), but it could be a peer. The main Joost servers are on Level3’s global network and Infonet in Luxembourg.
June 9th, 2007 at 10:42 pm
I can now confirm Joost is “spamming” you after use with UDP requests after use.
Now 3 days after latest use of Joost, it is wearing off, and the only IP active is 89.251.0.64, which is in fact Joost!
My firewall log ran full after a good hour when it was worst, as 7 different IP’s sent UDP requests each every 20 seconds to port 13775.
I now takes a good 5 hours to run full, and Joost is now deleted from my system!
Joost may not only take network capacity while working for you, it also keeps your line partly busy after you shut it down.
Thanks again for your information.
H
October 2nd, 2007 at 4:37 pm
colmmacc, tx for your network architecture slides, but the link to the updated set you gave returns “403 Forbidden”. Any chance of fixing it?
October 2nd, 2007 at 4:55 pm
I’m not sure that colmmac wants to relive the experience because of all the shenanigans that followed, but there is still a YouTube video of him presenting the slides here.
The FIA did the same thing and gave away a ton of Ferrari technical secrets in the McLaren spying case too, so colmmac is certainly in good company.