Friday, May 2. 2008
Twitter derailed?
News today that Twitter is going to use its funding to get off the Ruby on Rails system, followed by the standard harrumphing from Railistas that Ruby on Rails is not the problem, but "those dorks at Twitter" are - etc etc
Two observations, if we may*
Firstly, speaking to a lot of startup entrepreneurs, we reckon many (most) of them don't "get" scaling issues early enough. I wrote this after attending the first ever Open Coffee club in London last March:
What we found was that the younger startups were in the main (i) pretty naive about scalability and (ii) quite fervent in support of various "sexy" development environments, Ruby on Rails was high on the Sexy Reverence list.
Secondly, Ruby on Rails. I'm sorry to say this, but the correlation of the terms "Scaling Problems" and Ruby on Rails" does seem higher than usual. I suspect part of it is that the language is young and very powerful, and part of it is that the sort of startups who were religiously "into" RoR were maybe younger and less experienced, but right now its probably worth avoiding it for serious scaling and using something more proven, if only because more experienced people probably use these other languages.
All this does not help Twitter of course, but a total rewrite is a very hard thing to do while you are also supporting an existing system that is growing (in users and complexity) very rapidly. I would advise against it. What I would suggest is rewriting the pieces of the system that are most bottlenecked in older, lower level languages - takes longer, but they are faster and more robust. I also don't know enough about how they interface to the databases, but this can cause - or save - endless hassle if done well. Some databases are just better at scale than others, so that is also an area to look at.
Update 1. - Twitter has issued a response, denied moving off RoR, also notes that a lot of the code is not in RoR:
Update 2 - Andy Wise, one of my Broadsight colleagues, emailed a few extra thoughts after this was posted:
(* Context - we are ex Big Telco, Web 1.0 internet and Real Time OSS guys - we (think) we know a thing or two about scaling - or at least know where the bodies are
)
Two observations, if we may*
Firstly, speaking to a lot of startup entrepreneurs, we reckon many (most) of them don't "get" scaling issues early enough. I wrote this after attending the first ever Open Coffee club in London last March:
Some general observations from the conversations mixed with our experience, all to do with the Boring Bits - that's right, the Infrastructure Behind the Sexy Stuff.
1. Infrastructure matters, especially the scalability bit. Too many Web 2.0 betas are built to launch, not to scale - causes big problems down the line
2. Forget about the whizzy bits, make the basics work first - and work fast
3. Many people we spoke to were going to have to build the same basic (SocNet) functionality, on their own - clearly an opportunity for a platform play.....
Had a chat after with Mike Butcher re software development productivity - probably needs a blog on its own, but in our experience anyway there is a very direct relationship between productivity, closeness of location and quality of team. Also don't get religious (or fashionable) about languages, or snotty about using open libraries of Other People's Code.
What we found was that the younger startups were in the main (i) pretty naive about scalability and (ii) quite fervent in support of various "sexy" development environments, Ruby on Rails was high on the Sexy Reverence list.
Secondly, Ruby on Rails. I'm sorry to say this, but the correlation of the terms "Scaling Problems" and Ruby on Rails" does seem higher than usual. I suspect part of it is that the language is young and very powerful, and part of it is that the sort of startups who were religiously "into" RoR were maybe younger and less experienced, but right now its probably worth avoiding it for serious scaling and using something more proven, if only because more experienced people probably use these other languages.
All this does not help Twitter of course, but a total rewrite is a very hard thing to do while you are also supporting an existing system that is growing (in users and complexity) very rapidly. I would advise against it. What I would suggest is rewriting the pieces of the system that are most bottlenecked in older, lower level languages - takes longer, but they are faster and more robust. I also don't know enough about how they interface to the databases, but this can cause - or save - endless hassle if done well. Some databases are just better at scale than others, so that is also an area to look at.
Update 1. - Twitter has issued a response, denied moving off RoR, also notes that a lot of the code is not in RoR:
FWIW: Twitter currently has no plans to abandon RoR. Lots of our code is not in RoR, already, though. Maybe that's why people are confused.
Update 2 - Andy Wise, one of my Broadsight colleagues, emailed a few extra thoughts after this was posted:
The most important thing a start-up has to do is to get out there & generate a brand identity. They need the twiddley bits early on and to build users as rapidly as possible in order to attract funding and momentum.
Its also quite difficult in the early stages of a venture to see where the bottle necks will be - this is somewhere our expertise could help.
So I would not criticise anyone too much when they find themselves in the position of being successful - but in need of a re-write to boost performance.
Without being successful they wont have a performance problem
Without being successful they wont have the cash to do a rewrite
The great mantra is of course Moore's Law - so long as your computing power can double every 1-2 years then it does not mater if the dev. environment is poorly performing - just get more hardware.
This argument breaks down as scalability issues are not necessarily linear. No amount of additional computing power can resolve concurrent access to a single record in a database. The issue comes down to the design of the solution.
In some examples the start-ups have actually been very successful at building scalable systems - all the work being done on peer to peer networks is a great example of new ways to approach scalability.
I suspect the difference is in the goal of the start-up - those developing media players know they have a scalability problem - their innovation is to fix it. Those developing SocNet applications like Twitter have a different goal - they are experimenting with the way people behave. Its only their success that has exposed them to scalability issues.
As you allude to - what is needed is a Social Net core that many players can licence - and that core needs to address the scalability issue as a matter of architecture from day one.
(* Context - we are ex Big Telco, Web 1.0 internet and Real Time OSS guys - we (think) we know a thing or two about scaling - or at least know where the bodies are
)
Trackbacks
Trackback specific URI for this entry
No Trackbacks
Comments
Display comments as
(Linear | Threaded)
To add to what you are saying above, I don't think twitter's situation is a clear-cut case of "RoR is bad".
Firstly, chat is INCREDIBLY resource-intensive - especially the way they do it. Secondly, from what I understand they are indeed already writing some modules in a lower-level language (can't find a reference to that info, but I'm pretty sure it's the case).
These two points are perhaps a better explanation of why the RoR community is up in arms; RoR offers such a low barrier to entry to the web game that, naturally, comparatively few RoR developers have enough insight to understand twitter's decisions and their issues. That's a good thing though, because it means there's plenty of room for consultants like us
Of course I'm not impartial: give me @benjiegillam's memcached PHP Kitteh framework over RoR any day.
Firstly, chat is INCREDIBLY resource-intensive - especially the way they do it. Secondly, from what I understand they are indeed already writing some modules in a lower-level language (can't find a reference to that info, but I'm pretty sure it's the case).
These two points are perhaps a better explanation of why the RoR community is up in arms; RoR offers such a low barrier to entry to the web game that, naturally, comparatively few RoR developers have enough insight to understand twitter's decisions and their issues. That's a good thing though, because it means there's plenty of room for consultants like us
Of course I'm not impartial: give me @benjiegillam's memcached PHP Kitteh framework over RoR any day.
The Twitter folks had denied (via Twitter) the TechCrunch story within minutes of its appearance. Not that this takes away from your points about scaling issues, but let's not spread the rumor any further.
@ Jof - that rings true - and if anybody knows this stuff its you guys. In Twitter's own comment today they do say that a lot of the code is not in RoR anyway.
@ Mike - I have updated my post - but I do note that TC is saying that they've heard the story from lots of sources.
@ Mike - I have updated my post - but I do note that TC is saying that they've heard the story from lots of sources.
Think that Andy and Jof have pretty much nailed it here (in terms of argument ‘for’), in identifying that RoR is a pragmatic choice because of low barriers-to-entry and good speed-to-market (since it’s open source and is fast to develop in, encouraging innovation and a ‘get stuff out there’ mentality).
Most web apps don’t get big, so making a big upfront (time/money) investment in scaling is not a good choice, as a) it means the app isn’t getting out there and being used/iterated, and b) it’s not always apparent where the scaling issues/bottlenecks are going to occur. So when resource is limited (as it invariably is in most start-ups) focus on where the return is going to be made (getting traction). This is essentially what Andy’s said, and it’s the line that’s articulated in the 37signals book “Getting Real” (http://gettingreal.37signals.com/), the 37signals RoR guide/philosophy for building web apps (written by the guys behind the rails framework), and where they say don’t worry about scaling until you have to.
And if you find that you’ve got a scaling issue then congratulations, since it means you’re getting lots of usage (assuming you built sensibly in the first instance), and you can then worry about the bottlenecks that have become apparent. There is performance stuff that can be deployed early, e.g. memcached, cachefu, etc, so there’s no reason to not do some basic optimisation stuff at the outset, which should help you get over initial humps and surges. And if you haven’t got that knowledge inhouse then if you’re working with the right hosting company you can get some invaluable advice on best-practise in this respect (we work with Engine Yard, and they do exactly that, since the team there is some of the best minds in the community).
Also, the fact that Benchmark Capital (also invested MySQL, amongst other things open source), is making investments in rails-oriented businesses is a good sign that in addition to a vibrant community there’s some serious institutional/commercial firepower that’s starting to be directed on the area. (They’ve invested in New Relic (rails performance stuff, see http://tinyurl.com/5h6a55) and Engine Yard (rails hosting, see http://tinyurl.com/24jzxo)).
So the summary from my perspective is that the pros (agility) outweigh the cons (baked-in scalability), and if you experience scaling problems then rejoice, and then get some shit-hot performance people on-the-case (in light of twitter’s challenges, possibly easier said than done). In parallel the knowledge that there’s a good deal of effort going into solving performance/optimisation both from the community and commercial entities, and that there are a growing number of well-trafficked sites that are using RoR, is re-assuring. So we’re optimistic (but are focussed on the early stage success side of things at the moment!).
Context: I’m a business guy (learning to code on the side), not a techie (so I’m not evangelising on behalf of the community; I’m actually pretty agnostic, and it’s down to what allows us to get the job done). We’re using RoR, and it’s allowed us to get things out of the door pretty fast. As mentioned above, we use Engine Yard for hosting, and they’ve been invaluable in relation to operational support/performance and optimisation advice. We’re also currently looking for a lead RoR person to join the boxedup team (with founder-type status) so give me a shout (@ me on twitter - chrsoz) if you’re interested (sorry for the plug Alan)!
Most web apps don’t get big, so making a big upfront (time/money) investment in scaling is not a good choice, as a) it means the app isn’t getting out there and being used/iterated, and b) it’s not always apparent where the scaling issues/bottlenecks are going to occur. So when resource is limited (as it invariably is in most start-ups) focus on where the return is going to be made (getting traction). This is essentially what Andy’s said, and it’s the line that’s articulated in the 37signals book “Getting Real” (http://gettingreal.37signals.com/), the 37signals RoR guide/philosophy for building web apps (written by the guys behind the rails framework), and where they say don’t worry about scaling until you have to.
And if you find that you’ve got a scaling issue then congratulations, since it means you’re getting lots of usage (assuming you built sensibly in the first instance), and you can then worry about the bottlenecks that have become apparent. There is performance stuff that can be deployed early, e.g. memcached, cachefu, etc, so there’s no reason to not do some basic optimisation stuff at the outset, which should help you get over initial humps and surges. And if you haven’t got that knowledge inhouse then if you’re working with the right hosting company you can get some invaluable advice on best-practise in this respect (we work with Engine Yard, and they do exactly that, since the team there is some of the best minds in the community).
Also, the fact that Benchmark Capital (also invested MySQL, amongst other things open source), is making investments in rails-oriented businesses is a good sign that in addition to a vibrant community there’s some serious institutional/commercial firepower that’s starting to be directed on the area. (They’ve invested in New Relic (rails performance stuff, see http://tinyurl.com/5h6a55) and Engine Yard (rails hosting, see http://tinyurl.com/24jzxo)).
So the summary from my perspective is that the pros (agility) outweigh the cons (baked-in scalability), and if you experience scaling problems then rejoice, and then get some shit-hot performance people on-the-case (in light of twitter’s challenges, possibly easier said than done). In parallel the knowledge that there’s a good deal of effort going into solving performance/optimisation both from the community and commercial entities, and that there are a growing number of well-trafficked sites that are using RoR, is re-assuring. So we’re optimistic (but are focussed on the early stage success side of things at the moment!).
Context: I’m a business guy (learning to code on the side), not a techie (so I’m not evangelising on behalf of the community; I’m actually pretty agnostic, and it’s down to what allows us to get the job done). We’re using RoR, and it’s allowed us to get things out of the door pretty fast. As mentioned above, we use Engine Yard for hosting, and they’ve been invaluable in relation to operational support/performance and optimisation advice. We’re also currently looking for a lead RoR person to join the boxedup team (with founder-type status) so give me a shout (@ me on twitter - chrsoz) if you’re interested (sorry for the plug Alan)!
Secondly, Ruby on Rails. I'm sorry to say this, but the correlation of the terms "Scaling Problems" and Ruby on Rails" does seem higher than usual.
Correlation as compared to what? What's high? And what on earth does this mean anyway? I bet there's more sites out there with scaling issues in java, does that mean java has scaling issues? Or that people don't know what they are doing?
Correlation as compared to what? What's high? And what on earth does this mean anyway? I bet there's more sites out there with scaling issues in java, does that mean java has scaling issues? Or that people don't know what they are doing?
Correlation / comparison - I have seen people develop seriously scalable systems using Perl and PHP and they by and large hold up, whereas there is a lot of anxiety about Ruby - just Google it!
Java has got scaling issues - or more accurately speed issues at scale - but we have built some serious real time iron using Java so I'm not too bothered with it.
As to not knowing what people are doing, my point is more that a lot of Web 2.0 stuff is done by people not familiar with network infrastructure - in fact the Twitter guys admitted as much
Java has got scaling issues - or more accurately speed issues at scale - but we have built some serious real time iron using Java so I'm not too bothered with it.
As to not knowing what people are doing, my point is more that a lot of Web 2.0 stuff is done by people not familiar with network infrastructure - in fact the Twitter guys admitted as much
Quicksearch
For More Information
Contact us
Broadsight website
Articles
Broadstuff - the Twitter edition
Broadstuff - the Jaiku edition
Broadstuff - the FriendFeed edition
Subscribe to Broadstuff via email
Shameless Plug - vote Broadstuff for Best UK Web 2.0 / Business Blog
(we like the big red button anyways.....)
Books we are reading:
Contact us
Broadsight website
Articles
Broadstuff - the Twitter edition
Broadstuff - the Jaiku edition
Broadstuff - the FriendFeed edition
Subscribe to Broadstuff via email
Shameless Plug - vote Broadstuff for Best UK Web 2.0 / Business Blog
(we like the big red button anyways.....)
Books we are reading:
Syndicate This Blog
Bookmark
Archives
Blog Administration
Creative Commons Licence
Categories
- About Us
- Advertising
- B2B Media 2.0
- Blogstuff
- Business Models
- Dis - Aggregation
- DRM / IP
- Enterprise 2.0
- Gaming / Virtual Worlds
- Hacks and Startups
- Identity / Profiles / Trust
- Media Web 2.0
- Mobile Multimedia
- Mobile TV
- MyPCTV
- New Search
- Odds and Sods
- Open Source
- Quad Play
- Social Networks
- User Devices
- Web Services
- Web TV / IPTV / DTV