Agile is a great methodology, and applied right it can make your development process less like a black box, and more open to the stakeholders. It can do wonders for improving the reputation of your dev team with the rest of your company.

However:

We have taken down the main fav.or.it service while we upgrade to the new version. What is included? Well I will not ruin the surprise but we believe it is a generation shift for social news and the way we all interact with blogs. Although a selected few have had the chance to preview the reader what we are about to release is the next step in our evolution. This will be the culmination of all our efforts over the last year and we are very excited that it is now very near to launch. <!– more –>

Not Closed

The other important aspect is that when this goes live, it will be live for everyone! The product will still be in beta (and likely to stay that way for several months) but we hope that it is going to have something for everyone.

Keep an eye on this blog for updates or follow me on twitter to get the latest news on the new release.

I am getting sick of talk about twitter and it’s scalability problems and also frankly unqualified people slagging the service for it’s unreliability and also coming up with stupid ignorant answers to how it should be fixed.

As part of our future we do see a many-to-many problem arising so I put some thought to how we would tackle it (like any of our other problems) by doing some ‘planning’.

In a recent post Robert Scoble tries to explaining how twitter works by saying that twitter is using some form of ‘pivot table‘ – (my terminology for what he explains) and says that a model that others have put forward (i.e. a de-normalized system of inserting messages into everyones queues) was akin to microsoft exchange, now these two examples are so horribly not connected – and I won’t rant about how BAD exchange is efficiency wise, but please Robert do not get into any technical arguments please.

How it should be done

So what did I come up with for solving the many-to-many problem?

Firstly let me explain my terminology, I use the term ’shard’ because a shard could be a physical server or a daemon running alongside other daemon’s running on a server or even a grid platform.

The basic idea is to build a system in which no one particular element is unscalable. So the design takes into account that however big things get just larger numbers of shards would be required.

First a diagram (which I hope makes some sense of the complexity described below)

Queue Shard

The queue shard stores the message queue for a set number of users the shard would be found by a hashing mechanism that would map the user id to the particular user. The queue system would use a purely in-memory storage that would allow the for the really fast insertion speeds that would required.

The other really important point to make is that this server is only dealing with dates and the id of the tweet that it references (tweet id’s covered below)

Friend Shard

The friend shard stores the friends list for a number of users, this is likely to be a larger number of users per shard than the queue shard purely because its job is a lot easier. So lets say each shard can work with 100,000 users. When this shard gets told about a new message for a user it looks up its list of friends.

Then using the hashing algorithm that maps each user id to a specific queue shard it builds up a list of friends that live on each of those queue shards.

So in the extreme example of Robert Scoble of 25k friends we hope to split up the task so that no queue shard would have to deal with more than 1000 inserts.

Joiner Shard

Because we are only storing the id’s of each tweet in the queue when a query comes in for a stream we fetch the list of the 20 (or however many) tweets from the queue shard for that particular user then we need to go get each actual text tweet from a tweet shard. The joiner shard would very likely use a memcache (across a lot of servers) so that the tweet storage would get hit as little as possible.

The second thing to note is that as described the queue shards are in-memory only – they are storing data very inefficiently (e.g. totally de-normalized) but it would be a serious overhead for them to store to disk. The joiner shard would need to be able to cope with only getting partial results back from the queue shard – and then have to look it up from the ‘normalized’ version of the database (described below)

Tweet Shards

When new tweets come in they are stored in a tweet shard, we use a hash algorithm to spread them across a set number of servers. The algorithm should take into account that over time tweets get moved onto archive servers which are designed more for mass storage rather than quick retrieval.

Normalized Queue Storage

Because the in-memory queue shards are very fast but very inefficient it would be not sensible to store more than a weeks worth of data in them (and maybe even less) – as the many-to-friendlist-to-queue system even when stored just using id’s is still a large storage overhead compared with storing the data in a standard pivot/normalized style. So for every request that comes in it is also passed into this shard to be archived and also for when you do get the rare request for older data.

Web Shard

This is the easy bit, the other shards described so far have all done the complex database work, the web shard is responsible for dealing with the actual requests be they API / HTTP / whatever, and then pass on the request into the relevant other shards.

I will describe below the order in which the shards are called for particular actions (the two that are most common)

Sending a Tweet
The following is a rough step list of how a new tweet gets insert into the database.

1. Web Shard – Receives the request – First it sends a request off to the Tweet shard to store the message itself which then returns an ID that will then be used for all the other requests. Then it looks up which friend shard stores the particular user that the request is coming from.
   1. Tweet Shard – Receives message to store the tweet and returns the new ID
2. Friend Shard – Receives the message – then sends out signals to all the relevant queue shards for all the friends of that user.
3. Queue Shard – Gets a list of friends (that it deals with) and inserts the id of the message into each of their queues.
4. Normalized Queue Shard – We also store the data in this archive database for long term storage (and slow retrieval)

Building the Stream
This is a few short steps of how we build up a stream for a particular user.

1. Web Shard – Receives the request – sends a request to a joiner shard asking for the stream for that user
2. Joiner Shard – Looks up which queue shard has the data it is looking for – gets the list of id’s – then looks up those messages within the tweet shards – it then puts them together and returns them back to the web shard
3. Tweet Shard – Returns the actual text tweets

The above could be more complex in circumstances when the data is not available within the queue shards – or within the tweet shards – at which point they both seamlessly go look up the data in the archive instead.

Conclusion

This looks complex – (well it is..) but in general terms of complexity of building large scale web applications this is pretty simple stuff (fav.or.it has far more complex problems!). I understand that the twitter people now have the extremely hard task of slowly improving each module rather than a complete redesign. Whatever the future I wish them well. btw – if you want you can follow me (and fav.or.it’s exploits) via my twitter feed

(just as an aside – the fav.or.it blog will be going through a major overhaul soon and more of our wonderful employees we start posting (they already have their own individual persona blogs) so we have had to cater for this in our new design, but fear not we will also be splitting up the feed so if you only want product updates you can still have that, if you want development/tips/programmer/design stuff you will also have the option to subscribe to an overall feed or to individual topics.)

The fav.or.it API is designed to allow access to many of it’s core services. The first round of exposed requests allow comments to be distributed from external clients through fav.or.it back to the host of the specified blog. It is our desire that by opening up our core platform that we start a new flow of comments around the blogosphere and that by doing so everyone will benefit from a ‘better conversation’. The API will become available for testing sometime this week, we will make a further announcement when it is ready. We hope that this documentation will allow you to get started on developing exciting applications.

For full documentation on the API we have a Google Group which has detailed description of each request and more detailed information on all our services.

Suggested uses:

• Give commenting functionality to PC/Based RSS Readers
• Extending web based RSS Readers
• Send comments from embedded widgets
• Supply comments from external blogs (that fav.or.it cannot aggregate)

The API

fav.or.it exposes many of it’s data services via an application programming interface (API). Developers and power users may sign up for access the API by registering with fav.or.it and agree to our terms of service to use fav.or.it services and data in their applications.

RESTful Resources

The fav.or.it API tries to conform as much as possible to the design principles of Representational State Transfer (REST). You’ll find that you can simply change the file extension on almost any request to get results in the format of your choice. This document notes which formats are available for each method.

fav.or.it presently supports the following data formats: XML, JSON, and PHP (serialized)

Identity Management

fav.or.it has an internal identity system that allows the mapping of multiple identities from external providers onto a single fav.or.it user. Within fav.or.it a user may identify themselves across multiple services which then links any incoming comments/posts as being owned by that user.

The following services can be authenticated by fav.or.it – we will be announcing further services in the near future.

• openid
• twitter
• blogger (blogspot)
• wordpress.com
• bloglines

Partner Services

A partner service is any external site that supplies fav.or.it with authenticated data. In most cases this means supplying authenticated comments from their own user base. fav.or.it has an internal identity management system that allows the mapping of multiple identities across comment systems. If you are interested in becoming a partner service please get in touch.

Parameters

Some API methods take optional or requisite parameters. Where applicable, we’ve documented those parameters. Remember to convert to UTF-8 and URL encode parameters that take complex strings.
HTTP Requests

Unless otherwise noted, methods exposed by the fav.or.it API require a GET request. Methods that post new comments require a POST. Where noted, some API methods will return different results based on HTTP headers sent by the client.

Rate Limiting

Public clients are allowed 500 requests in a 24 hour time period, starting from their first request. Notification that a client has exceeded the rate limit will be sent as JSON,XML or PHP when either is the requested format, and otherwise will be sent in plain text.

Example XML failure

  failure
  missing param 'comment_id'

Spam

fav.or.it takes spam very seriously and will not tolerate spam sent through it’s API. Our community + technologies identify spam before it is forwarded onto external blogs.

In the case of persistent spamming an API Key will be disabled along with the account that it is associated with.

Requests

• sendcomment
• sendcommentas
• commentstatus

Talented Web Developer

It is the new year and we are looking for talented web developers to join our development team working on fav.or.it. We are preferably looking for PHP 5 developers but will cross-train from Perl/Ruby/Python. We are looking for pure programming skills first and foremost so if your not an uber-geek who can do hex calculations in your head then don’t bother applying. The winner gets all the monitors they desire, the best development tools and a great start-up environment.

This a fantastic opportunity to get involved with an exciting project and make a real difference to the landscape of blogosphere.

Essential skills

• Triple ‘A’ Programming skills in any of the following (PHP, Ruby, Perl)

Desirable skills

• PHP 5 (OO)
• Experience with one of the following PHP Frameworks
• CakePHP
• Zend Framework
• Symphony
• XHTML, CSS, JavaScript, DHTML/AJAX, XML
• MySQL relational DB design
• Linux, Apache, MySql administration
• XML-RPC
• Code versioning tools (Subversion, CVS)

Role Involves

• Development, maintenance and support of new functionalities
• Analysis of business and user requirements and production of functional specifications;
• Project implementation from inception through development and support;
• Testing of web functionality;
• Resolving problems on specific platforms.

Location: Reading (UK)

Salary based upon experience

Send CV’s + Sample of code to

Find A Geek – Get Paid!

We are also paying a £500 finders fee if you introduce someone to us that we then hire.

Looking for a new job? Based in the UK? Want to work for one of the hottest start-ups in the web arena? Count yourself as one of the top Ruby/PHP/Perl/Python programmers? We are looking for a Senior Web Developer to work on fav.or.it and help shape the future of the blogosphere.

If you know someone who is looking for a job and thinks fits the description we will give you a £250 finders fee if they accept the job (and stick around long enough to work out where the toilets are e.g. 8 weeks)

Senior Web Developer

We are looking for very experienced web developer, if you cannot name ten web technologies off the top of your head you’re not the right person for us. Building large structured code must be
your bread and butter and being at the cutting edge of web development your goal.

You will be working on a very exciting project with massive global potential, your contributions will help shape the future of the blogosphere and how the mass market interacts with it.

We are preferably looking for PHP 5 experience but are happy to work with anyone who understands the web programming space and is willing to quickly learn a new language.

Essential skills

• Triple ‘A’ Programming skills in any of the following (PHP, Ruby, Perl)

Desirable skills

• PHP 5 (OO)
• Experience with one of the following PHP Frameworks
  • CakePHP
  • Zend Framework
  • Symphony
• XHTML, CSS, JavaScript, DHTML/AJAX, XML
• MySQL relational DB design
• Linux, Apache, MySql administration
• XML-RPC
• Code versioning tools (Subversion, CVS)

Role Involves

• Development, maintenance and support of new functionalities
• Analysis of business and user requirements and production of functional specifications;
• Project implementation from inception through development and support;
• Testing of web functionality;
• Resolving problems on specific platforms.

Location: Reading
Salary £35,000 to £50,000 + share options

To apply for this role send your CV to or use my contact form.