http://simonwillison.net/2009-11-30T11:30:12+00:00Simon Willison2009-11-30T11:30:12+00:002009-11-30T11:30:12+00:00https://simonwillison.net/2009/Nov/30/hive/#atom-tag

<blockquote cite="http://www.facebook.com/note.php?note_id=114588058858"><p>Today, Facebook counts 29% of its employees (and growing!) as Hive users. More than half (51%) of those users are outside of Engineering. They come from distinct groups like User Operations, Sales, Human Resources, and Finance. Many of them had never used a database before working here. Thanks to Hive, they are now all data ninjas who are able to move fast and make great decisions with data.</p></blockquote> <p class="cite">&mdash; <a href="http://www.facebook.com/note.php?note_id=114588058858">Facebook Data Team</a></p> <p>Tags: <a href="https://simonwillison.net/tags/facebook">facebook</a>, <a href="https://simonwillison.net/tags/hadoop">hadoop</a>, <a href="https://simonwillison.net/tags/hive">hive</a></p>

2009-10-21T18:48:36+00:002009-10-21T18:48:36+00:00https://simonwillison.net/2009/Oct/21/cloudera/#atom-tag

<p><strong><a href="http://www.cloudera.com/blog/2009/10/01/introducing-cloudera-desktop/">Introducing Cloudera Desktop</a></strong></p> It’s a GUI for Hadoop, and under the hood is a whole stack of open source software, including Python, Django, MooTools, Twisted, lxml, CherryPy, Mako, Java and AspectJ. <p>Tags: <a href="https://simonwillison.net/tags/aspectj">aspectj</a>, <a href="https://simonwillison.net/tags/cherrypy">cherrypy</a>, <a href="https://simonwillison.net/tags/cloudera">cloudera</a>, <a href="https://simonwillison.net/tags/django">django</a>, <a href="https://simonwillison.net/tags/hadoop">hadoop</a>, <a href="https://simonwillison.net/tags/java">java</a>, <a href="https://simonwillison.net/tags/lxml">lxml</a>, <a href="https://simonwillison.net/tags/mako">mako</a>, <a href="https://simonwillison.net/tags/mootools">mootools</a>, <a href="https://simonwillison.net/tags/open-source">open-source</a>, <a href="https://simonwillison.net/tags/python">python</a>, <a href="https://simonwillison.net/tags/twisted">twisted</a></p>

2009-04-07T09:19:38+00:002009-04-07T09:19:38+00:00https://simonwillison.net/2009/Apr/7/amazon/#atom-tag

<p><strong><a href="http://developer.amazonwebservices.com/connect/entry!default.jspa?categoryID=265&amp;externalID=2294&amp;fromSearchPage=true">Finding similar items with Amazon Elastic MapReduce, Python, and Hadoop streaming</a></strong></p> Tutorial for running Hadoop jobs on Elastic MapReduce using Python and the 2005 Audioscrobbler dataset. <p>Tags: <a href="https://simonwillison.net/tags/amazon">amazon</a>, <a href="https://simonwillison.net/tags/amazon-web-services">amazon-web-services</a>, <a href="https://simonwillison.net/tags/audioscrobbler">audioscrobbler</a>, <a href="https://simonwillison.net/tags/elasticmapreduce">elasticmapreduce</a>, <a href="https://simonwillison.net/tags/hadoop">hadoop</a>, <a href="https://simonwillison.net/tags/mapreduce">mapreduce</a>, <a href="https://simonwillison.net/tags/python">python</a></p>

2009-04-02T10:25:37+00:002009-04-02T10:25:37+00:00https://simonwillison.net/2009/Apr/2/mapreduce/#atom-tag

<p><strong><a href="http://aws.amazon.com/elasticmapreduce/">Amazon Elastic MapReduce</a></strong></p> Hadoop as a service. Basically a web based GUI around Hadoop—you could roll this yourself on EC2 but for a small markup on regular EC2 prices you get to avoid the extra work setting everything up. Data processing scripts can be written in Java, Ruby, Perl, Python, PHP, R, or C++ and are loaded in to S3 before firing off the job. <p><small></small>Via <a href="http://joedrumgoole.com/blog/2009/04/02/amazon-web-services-adds-map-reduce/">Joe Drumgoole</a></small></p> <p>Tags: <a href="https://simonwillison.net/tags/amazon">amazon</a>, <a href="https://simonwillison.net/tags/amazon-web-services">amazon-web-services</a>, <a href="https://simonwillison.net/tags/cloud-computing">cloud-computing</a>, <a href="https://simonwillison.net/tags/ec2">ec2</a>, <a href="https://simonwillison.net/tags/hadoop">hadoop</a>, <a href="https://simonwillison.net/tags/mapreduce">mapreduce</a>, <a href="https://simonwillison.net/tags/s3">s3</a></p>

2008-10-01T13:22:19+00:002008-10-01T13:22:19+00:00https://simonwillison.net/2008/Oct/1/about/#atom-tag

<p><strong><a href="http://www.cascading.org/about.html">Cascading</a></strong></p> A Java API abstraction layer over Hadoop that lets developers think in terms of pipes and filters rather than map/reduce. The Cascading developers claim that this model is easier to understand and less error prone. <p>Tags: <a href="https://simonwillison.net/tags/cascading">cascading</a>, <a href="https://simonwillison.net/tags/hadoop">hadoop</a>, <a href="https://simonwillison.net/tags/java">java</a>, <a href="https://simonwillison.net/tags/mapreduce">mapreduce</a>, <a href="https://simonwillison.net/tags/pipesfilters">pipesfilters</a></p>

2008-07-07T14:15:23+00:002008-07-07T14:15:23+00:00https://simonwillison.net/2008/Jul/7/bill/#atom-tag

<p><strong><a href="http://www.dehora.net/journal/2008/07/06/3-12-minutes-to-sort-a-terabyte-hadoops-code-structure/">3 and 1/2 minutes to sort a Terabyte, and a look at Hadoop&#x27;s code structure</a></strong></p> Bill de hÓra uses some clever static analysis tools to explore Hadoop’s 100,000+ lines of code. <p>Tags: <a href="https://simonwillison.net/tags/bill-de-hora">bill-de-hora</a>, <a href="https://simonwillison.net/tags/hadoop">hadoop</a>, <a href="https://simonwillison.net/tags/java">java</a>, <a href="https://simonwillison.net/tags/static-analysis">static-analysis</a></p>

2008-05-31T14:14:56+00:002008-05-31T14:14:56+00:00https://simonwillison.net/2008/May/31/lastfm/#atom-tag

<p><strong><a href="http://blog.last.fm/2008/05/29/python-hadoop-flying-circus-elephant">Python + Hadoop = Flying Circus Elephant</a></strong></p> Last.fm have released Dumbo, a Python module that lets you easily write Hadoop map/reduce tasks using Python and generators. <p>Tags: <a href="https://simonwillison.net/tags/dumbo">dumbo</a>, <a href="https://simonwillison.net/tags/generators">generators</a>, <a href="https://simonwillison.net/tags/hadoop">hadoop</a>, <a href="https://simonwillison.net/tags/lastfm">lastfm</a>, <a href="https://simonwillison.net/tags/mapreduce">mapreduce</a>, <a href="https://simonwillison.net/tags/python">python</a></p>

2007-10-09T11:33:58+00:002007-10-09T11:33:58+00:00https://simonwillison.net/2007/Oct/9/writing/#atom-tag

<p><strong><a href="http://www.michael-noll.com/wiki/Writing_An_Hadoop_MapReduce_Program_In_Python">Writing An Hadoop MapReduce Program In Python</a></strong></p> Hadoop (the open source map/reduce framework) can interact with any program that reads from stdin and outputs on stdout—so it’s trivial to drop in Python scripts for the map and reduce steps. <p>Tags: <a href="https://simonwillison.net/tags/hadoop">hadoop</a>, <a href="https://simonwillison.net/tags/mapreduce">mapreduce</a>, <a href="https://simonwillison.net/tags/python">python</a></p>

2006-08-23T08:36:14+00:002006-08-23T08:36:14+00:00https://simonwillison.net/2006/Aug/23/hadoop/#atom-tag

<p><strong><a href="http://lucene.apache.org/hadoop/">Hadoop</a></strong></p> Open-source Google File System / map-reduce equivalent. Apparently scales amazingly well. <p>Tags: <a href="https://simonwillison.net/tags/hadoop">hadoop</a></p>