iWeb Communities: Blog

When starting with a website, most people don’t generally plan their infrastructure architecture; and won’t try to imagine what to do when their traffic will grow to millions of connections every month. Here’s how a webmaster generally sees it:

• Install a web application on a dedicated server, with everything running on the same server
• If usage outgrows capacity, then consider switching to a bigger server
• Maybe put the database server on another server if there is even more success
• and then ??

The problem is even more acute for web startups or for large companies setting highly-trafficked websites, since they’ll hit limits much more quickly than the case depicted above.

Solutions range from hiring a consultant expert in complex infrastructures, or getting the services of an operations engineer, a database administrator. Some go through the easy solution, which loading servers with gigabytes of RAM and faster CPUs.

It’s better though to educate yourself about solutions well before hitting those walls; as such, here are a few pointers which will help you scale your setup:

• The most valuable read is Building Scalable Websites, by Cal Henderson, lead developer at hugely successful photo-sharing website flickr.com. Unsurprisingly, Cal Henderson writes that the most reliable and cost-effective recipe for scaling is horizontal scaling, and to be able to distribute load on commodity servers. Other interesting books include High-Performance MySQL and Scalable Internet Architectures.
• The Flickr Code blog, Digg Technology blog and Facebook Engineering page have interesting reports and articles on challenges brought on those teams, facing rapid growth with massively successful websites
• The Highly Scalability website has a section dedicated to “Real-Life Architectures“, with details on software and hardware used by known websites such as YouTube, PlentyOfFish, Google, Amazon, Ebay, LiveJournal etc. It’s undoubtedly a resource you would want to bookmark and go through, if there’s a configuration you find inspiring for your website.

Of course, if you only have a blog with mild growth, spending your time reading those resources might be premature. However, web developers might learn valuable lessons reading those, by learning about “recipes which work” on a large scale.

In either case, you can also contact us for advice and let our experts architect find a solution for you.

Here are also other interesting articles you can read:

• Hosting Guide: How to Choose a Dedicated Server
• Top iPhone Applications for a Sys-admin
• Web Application Launch Checklist