Facebook's Engineering Project

A mere 15 engineers dealt with Facebook’s ‘largest ever engineering project’ – namely its new messaging system. Taking a little over a year the project rolled out a new distributed database platform that uses the open source version of Google’s BigTable platform – its name is Hbase.

Andrew Bosworth, director of engineering, explained how Facebook needed a new database infrastructure in order to provide a single interface for email, IM, text messages and on-site Facebook messages – plus one archive for all the above. Hbase was able to do this.

Bosworth goes on to say, “"In order to support this really cool and very deep scenario...we needed to rebuild our [messaging] infrastructure.” Areas looked at – included, Cassandra, the open source platform for inbox searching.

“We weren't sure of the trade-offs [with Cassandra]. We tested MySQL, but we weren't sure it could perform with long-tail data. So we invested in HBase."

HDFS, the Hadoop File System, is an open source incarnation of the old Google File System - Hbase sits on this platform. This log-based storage system made the most sense in order to, “expose the long-tail of your conversation history really easily,” said Bosworth. “We want this to be real-time — so you always know what messages you’re getting.”

Any intention of standardising on a single platform? Bosworth was clear with regards to this area. “You can get into trouble for over-standardising the technology,” he said. Separate tasks, separate platforms is the way it will stay for now.

Perhaps the “small, really quick engineering teams” simply “use the right technology for the problem (we’re) solving.”