To state the obvious, Hadoop is a disruptive technology. One way this might play out is as a replacement for ETL and data warehousing setups in big companies. Picture a pipeline of (1) DB2 tables (2) VSAM and other structured files, (3) Oracle OLTP databases, (4) Informatica/Ab Initio/Data Stage/whatever jobs filling up (5) Oracle data warehouses, and finally (6) SQL Server cubes connected to front-end applications in the hands of analysts. There are a lot of variations on this idea out there, but let’s call it an example of a common pattern. 1, 2, 3 and 6 are hard to dislodge, because they’re actual operations and high-level-user-facing apps, respectively, but 4 and 5 are pretty ripe, in many cases, to be moved from the special-purpose clusters they tend to run on to a general-purpose Hadoop cluster, on commodity hardware, with probable increased parallelization and massive cost savings. There’s a lot of Oracle in that space, and Oracle now has a commanding position relative to the fate of java. So let’s work this angle, but not make it too obvious, or go to far, or maybe java will start languishing like MySQL. I jest. Sort of.

One theme that played out through many presentations was abstraction layers on top of raw Map Reduce. The two biggest are Pig and Hive, which are Yahoo’s and Facebook’s solutions to the same basic problem, of how to write less code for repetitive Map Reduce tasks. There’s a lot of good commentary out there on those. Hive is more like a sql shell, and if you want to extend it, I think you’re going to be writing, say, Python mappers/reducers and streaming them into/out-of your Hive setup. With Pig, you’re operating, as they put it in the training/documentation/O’Reilly book, which collectively document Pig very well, more at the level of a SQL query optimizer. You have some iteration facilities, and you can extend it with java. Pig does more exactly what you tell it to do, and Hive is something you ‘hint’ at. These are general-purpose tools.

In the more specialized area of web analytics, eBay has a very interesting internal tool, called Mobius Query Language, on which Neel Sundaresan gave a fascinating talk. I’ll update with a link if Cloudera posts the presentation, but it helps you model visits with landmarks, duration, and some other concepts I didn’t take notes on. It clearly helps them wrap their code around the maddeningly amorphous user visit: participating in an auction, bidding, winning, abandoning, etc. The language seemed general-purpose enough for application to any user-behavior modeling. The interface is a SQL-like query language that seems, like Hive, to generate Map Reduce jobs based on nicely abstracted view of exactly the sorts of questions you want to ask your web analytics system. For the moment, I’m doing what web analytics I’m doing by extending Pig, but I hereby declare the Movement to Get eBay to Opensource the Mobius Query Language. Who’s with me?

On the conference in general, there is some good commentary out there, from Dan Milstein, Steve Laniel, Hilary Mason, and no doubt others.