3 articles in this selection

2009/07/27

Google released whitepapers on their distributed filesystem called GFS and a parallel computing architecture called MapReduce. Since then, a number of open-source projects have started that implement these ideas. Of note, though, is the Apache project Hadoop which implements a distributed filesystem and the MapReduce framework in Java. So I decided to build a cluster of 10 nodes and try it out....

2009/07/27

Apache Hadoop is a free Java software framework that supports data intensive distributed applications. It enables applications to work with thousands of nodes and petabytes of data. Hadoop was inspired by Google's MapReduce and Google File System (GFS) papers....

2009/07/27

With the rise of Social Media and the decreasing cost of storage, very small companies have a need for processing massive quantities of data. Furthermore, it's easier than ever to write software to generate/output/process data, thanks to languages like Ruby, frameworks like Spring, and scalability best practices. A few days' work and a handful of engineers can net you a bare-bones Twitter clone, or a crawler to get the link graph of the entire Internet. This data growth can be far from linear. You simply can't analyze this much data in an RDBMS - but these small startups can't spend millions on DWs, either. Since the analysis is a core of the business, internal hacked-together tools cobbled together from SQL boxes often emerged. What results is a temporary solution, not a platform. With Hadoop, HBase, and Hive, there’s now a free, scalable Data Warehousing platform that makes it possible to migrate a considerable portion of DW analytics....