I’m speaking at this event on Wednesday, November 12th: http://dataatscaleboston.splashthat.com/. The talk is called “Scaling Redis and Memcached at Wayfair”.
Me: Information Week says I’m the Don Draper of Wayfair.
Wife: I don’t think that’s what they actually said.
Me: Maybe not, but I don’t think you can avoid that conclusion. Jeff Bertolucci wrote a piece saying marketing now lives and dies on good math, not good creative, and data scientists are the new Mad Men. Before Don Draper became a partner at that advertising firm in the TV show, he was the head of the creative department. I took a couple of math classes in college, I can do linear regressions in R, and more importantly I manage a band of data scientists, so….
Wife: For a data scientist, you’re not too logical.
Me: Fuzzy logic. It’s the next big thing.
Wife: Don’t get any ideas.
No marriages were harmed in the making of this blog post.
Posted in Uncategorized.
– October 26, 2012
I joined Wayfair as a full-time employee on August 1st, 2011, to run internal search and customer recommendations, after a brief stint as a consultant. I have other responsibilities now as well, but over at the company tech blog, http://engineering.wayfair.com, I’ve written mostly about the search and recommendations systems. We’ve done a lot over there in a year. Let’s recap a few highlights:
- We wanted to deploy a Python web service, so we figured out a few things about web operations with Python, which we expressed in both comic-book and serious/detailed form.
- We set up a KISS-method recommender system based on simple correlation metrics, as a proof of concept.
- That worked, so we started down the road of more involved recommendations, at first with Markov clusters.
- Jeff Bertolucci of Information Week picked up that piece, did an interview with me, and wrote an article about it: http://www.informationweek.com/big-data/news/big-data-analytics/240007850/online-retailer-uses-dna-research-to-connect-with-customers. The crack team of data scientists is delighted to see their work highlighted in this way!
- We improved internal search in various ways, including hacking Solr and contributing our patch back to the open source community.
- We wrote some classifiers and other machine-learning programs for various purposes, and then linked one of them up to the search capability.
It’s hard work, I suppose, but mostly it just feels exhilarating and fun.
Posted in technology.
– October 24, 2012
From 2009 to mid 2011 I was running Dalton, Clark & Associates LLC full time, and consulting to some awesome clients, especially boston.com/Boston Globe, and StyleFeeder (now retired), which was acquired by Time, Inc., where I did some things for Stylefind (now retired). But in May of 2011, I did a small consultation for Wayfair (then CSN Stores, rebranded as Wayfair in September) and by August 1st I had joined full time to be the manager/architect of the search and recommendations group within Wayfair Engineering. My joke name for the group is the department of
Applied Calculus and Linear Algebra SOLR and Hadoop NumPy/SciPy Hackery Search and Recommendations Finding Useful Stuff. Anyway I’ve handed off the day-to-day operations of Dalton Clark & Associates LLC to the other owner, Abbe Dalton Clark. The views expressed in this blog are my own, and do not reflect the views of my employer.
Posted in technology.
– January 8, 2012
Why did I want to update my old setup? Simple: new and updated tools. A lot of people are using Hadoop 0.20.2 or higher, Hive and Pig have come a long way lately, and there was this new R-for-hadoop thing called Rhipe that I heard about at Hadoop World in December. It sounded like (and I can now say is) one of the best Hadoop tools ever. Just as Hive gives you a kind of mysql shell for Hadoop, so Rhipe gives you an R shell for Hadoop. A thousand thanks to Saptarshi Guha for this awesome tool. The hadoop-lzo package, thanks to Kevin Weil and Todd Lipcon, is more solid and easier to use than ever. There’s a new cluster launcher called Whirr. Dumbo just works on this platform, so no more patching my own version like before, which was a bit of a pain, as I described in one of those previous howtos. I’ve now got all this working with the Cloudera hadoop 0.20.2 distribution, CDH3B3, to be specific, and it seems to be all kinds of awesome.
Step one: launch hadoop clusters.
This used to work great with Tom White’s python scripts. Now it has all changed, and the replacement for those scripts, Whirr, is still in the process of emerging as a useful product. It’s much more powerful and flexible than the old scripts, but at least for now, you need to obtain the not-yet-released version 0.4 from subversion, to get the most out of it. In versions prior to that, you had to put your customized scripts in a publicly readable S3 location, and that doesn’t work for me. Documentation is evolving because the project is changing quickly, but the responsiveness of the committers on the user list is superb. I recently reported an issue with overriding a type of hadoop configuration property: it took about 24 hours of back and forth to confirm it was a problem, and then Andrei Savu had a patch up within a couple of hours, and it was discussed, reviewed by Tom White and committed in very short order. This is how opensource software should always be!
With Whirr 0.4+, the basic procedure is to untar (for now, check out) the distribution, set a WHIRR_HOME environment variable to the root of it, copy the vanilla customization scripts from ‘services/cdh/src/main/resources/functions’ to a directory where you can version them or whatever you want, and symlink from $WHIRR_HOME/functions to that directory. Then set up a configuration file. There are ‘recipes’ for various types of clusters in the distribution. I started with a plain one, and hacked on it a bit until I got this:
whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,6 hadoop-datanode+hadoop-tasktracker
# Ubuntu 10.04 LTS Lucid, 64-bit. See http://alestic.com/
# Ubuntu 10.04 LTS Lucid, 32-bit. See http://alestic.com/,
# Amazon linux, S3-backed, 64-bit:
# Amazon linux, S3-backed, 32-bit
Notice the four flavors of linux that I have tried out: Amazon linux 32-bit and 64-bit, and Ubuntu images from Alestic. The 64-bit Alestic one was the default in the whirr recipe. I usually roll a bit more Redhat/yum, so I wanted to try ‘Amazon linux’, which gives you a stripped down, yum-managed linux with python 2.6. This was a bit tricky. Amazon has two kinds of ‘Amazon linux’: ebs-backed and S3-backed. The ebs-backed ones have a small root partition and no extra disk space at all. I guess they assume you will just spin up an ebs volume and mount that if you need more space–fair enough. The S3-backed ones have a big extra partition. However, it’s not mounted on /mnt, as it is on every other Amazon machine image in the world, but rather on /media/ephemeral0. Amazon linux is eccentric that way. Or maybe, since it’s Amazon, it’s the rest of the world that’s eccentric. They explain it all here. Suffice it to say there are a *lot* of scripts out there that assume that /mnt will be big enough for a truckload of data. So in your cdh install script you’ll need something like
rm -rf /mnt
ln -s /media/ephemeral0 /mnt
and you’ll want that to happen before the script starts installing hadoop.
My script installs a few things. On ubuntu it looks like this:
apt-get install -y hadoop-hive
apt-get install -y hadoop-pig
apt-get install -y hadoop-hbase
apt-get install -y s3cmd libxaw7-dev libcairo2-dev r-base liblzo2-dev lzop python-setuptools git-arch subversion
tar zxf protobuf-2.3.0.tar.gz
R CMD INSTALL Rhipe_0.65.3.tar.gz
and, on Amazon linux, like this:
yum install -y hadoop-hive
yum install -y hadoop-pig
yum install -y hadoop-hbase
yum -y install gcc python-devel
cd /etc/yum.repos.d && wget http://s3tools.org/repo/RHEL_6/s3tools.repo
i386count=`uname -a | egrep "i386|i686" | wc -l`
if [ $i386count = '1' ]; then
rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm
rpm -Uvh http://download.fedora.redhat.com/pub/epel/6/x86_64/epel-release-6-5.noarch.rpm
yum install -y s3cmd screen readline-devel texinfo gcc-c++ gcc-gfortran cairo-devel lzo-devel lzop subversion protobuf
There are a few more things I put in the script, but they are very much in the personal preference category. In particular, I set some users, groups and permissions, so that the user I’m doing all this as can move files around to appropriate locations on the slaves. You can do that stuff after the cluster is up too, so it’s not supremely important to get everything happening on startup.
This takes care of some of the tools we want, and all of the prerequisites for the remaining ones.
Notes on R and Rhipe:
Rhipe’s documentation says it needs protocol buffers version 2.3 and above, but at least for now you need exactly version 2.3.
In order to get Rhipe to work, you’ll need to set LD_LIBRARY_PATH. Put something in the install script that appends a line to /usr/lib/hadoop/conf/hadoop-env.sh. In my case that’s this for Amazon Linux, where both R and Rhipe are compiled from source:
echo "export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib/R/lib:/usr/local/lib/R/ library/Rhipe/libs:\$LD_LIBRARY_PATH" >> /usr/lib/hadoop/conf/hadoop-env.sh
On Ubuntu, where R-base is a deb, and only Rhipe is compiled from source, it’s /usr/local/lib:/usr/lib/R/lib:/usr/local/lib/R/site-library/Rhipe/libs.
Step 2: Tweak configuration post-launch.
There are a few things you should put in place to make your life easy. First, you need to be able to run ‘slaves.sh’ to run commands on all the slaves: ‘slaves.sh uptime’ or ‘slaves.sh sudo yum install -y puppet’, for example. This means you need the ip addresses or host names of the slaves to be in /usr/lib/hadoop/conf/slaves, one per line. I used ‘whirr list-cluster’, the ‘cut’ command, and some hackish unix tricks to make this happen. This should ideally be a whirr command, as it was in the old python scripts, but I’m so grateful for all the rest of the whirry goodness that I’m not really complaining about that.
What type of thing might you want to do, in this category? On the Ubuntu images, I have the install script download Rhipe and install it, since R is already there. On Amazon linux, though, R won’t rpm-install, so I download it from source, compile and install it after the cluster is up. The compilation takes a long time, and I don’t want things to time out too badly on startup. For that, you definitely want to be able to do something like (pseudo-code alert): ‘slaves.sh wget R && tar zxf R.tar.gz && cd R && ./configure –with-x=no –with-cairo=yes –enable-R-shlib && make && sudo make install” or something like then, and *then* install Rhipe. Also, if you want to change any Hadoop configuration properties, you’ll change a file such as /usr/lib/hadoop/conf/core-site.xml, and then you’ll want to do something like ‘for i in `cat /usr/lib/hadoop/conf/slaves`; do scp /usr/lib/hadoop/conf/core-site.xml $i:/usr/lib/hadoop/conf; done’ and then restart all the data nodes, and then the name node. Those commands are
slaves.sh sudo /etc/init.d/hadoop-0.20-datanode restart
slaves.sh sudo /etc/init.d/hadoop-0.20-tasktracker restart
#pause to make sure the data nodes have had a chance to start.
#Otherwise you may encounter
#"java.io.IOException: /tmp/hadoop....jobtracker-info file could
#only be replicated to 0 nodes instead of 1"
#when you try to run your first map-reduce
sudo /etc/init.d/hadoop-0.20-namenode restart
sudo /etc/init.d/hadoop-0.20-jobtracker restart
Step 3: Run a hello-world test.
Don’t skip this step. The one I use takes a text file and uses ‘cat’ as the mapper and ‘wc -l’ as the reducer. I run this on a plain text file and a .lzo file, properly indexed, to make sure my compression is working and it’s not counting the .lzo_index file as data.
Step 4: Process some data
We get data from a lot of sources. If I had a hadoop cluster running constantly, I would probably do this very differently, but for me the whole point of using Amazon is to be able to fire up a big cluster, process some data, produce some results, store them some handy place like S3, and shut down the cluster. How best to make things available to hadoop? I came up with the following expedient:
First, fire up an ec2 instance and a big ebs volume, mounting the latter on the former. Grab your raw files-up-to-yesterday and store them on the volume. Then write a script or three that pre-process the raw files for consumption by hadoop. There are a few things you may want to accomplish in this step. For example, you might take small files and cat them together, then lzop the resulting big file, so you have your data in the large chunks Hadoop prefers. Store these ready-for-hadoop files in another part of the volume. Then, set up a procedure that can grab everything between yesterday and today, or everything back three days, because your previous load might not have final results, or something like that. Finally, make a snapshot of this ebs volume.
When you want to process some data, fire up a hadoop cluster, create a volume from the snapshot, mount the volume on the name node, run the ‘get-me-the-latest’ procedure, and load the lzop files into hdfs. The compression reduces the file size here by a lot, and saves a lot of time at this point. Run a quick script along the lines of “for f in `ls`; do hadoop jar /usr/lib/hadoop/lib/hadoop-lzo-0.4.9.jar com.hadoop.compression.lzo.LzoIndexer myhdfsinputdir/$; done” (supposing you are in a directory where you have all the files you just loaded into hdfs). Do your processing, store your results, and shut everything down.
One last thing: you might ask, why not Elastic Map Reduce? This setup is a lot like EMR, but I think I have more freedom to install the tools I want and automate jobs. Somebody please correct me if I’m wrong about that. I’m certainly in a better position to repurpose anything I develop for non-Amazon use, and I want the freedom to do that.
YMMV, but I’m getting a lot of cost-effective data processing, and useful information, out of this procedure these days.
Posted in technology.
– March 30, 2011
I went to Hadoop World in NYC last week. There was a lot of big talk about how this was where the alpha geeks had gathered. Self-serving, sure, but not false.
There were a lot of excellent talks. I’ll divide the highlights into two categories.
First, the practical items.
- I followed all of @kevinweil of Twitter’s suggestions from last year, and they were very helpful, so I went back for more this year. He spoke about file formats. Sexy! But all my experience of the last 1.5 years confirms this is very important in Hadoop world. He’s using protocol buffers + lzo compression, and has a project called Elephant Bird for protobuf-handling code generation and other things. As usual he puts it all up on github, and I’ll be using it. It’s interesting that they’re not using Avro or Thrift, the other versions of the same binary-format idea as Google’s protocol buffers (Avro is the Hadoop version, Thrift is from Facebook). Twitter does not otherwise fear to use the always-poorly-documented but sometimes excellent Facebook offerings, including Scribe for log collection. They “work closely with Facebook” to make that work. Hmmm.
- I will not be using Scribe, because the Flume project (an agent-based collector for log files) seems to have achieved critical mass. That’s, at least, what Otis Gospodnetić was saying at his excellent talk about search analytics. His reputation, company, and awesome on-topic demo site Search Hadoop, made for a packed room. He’s using Flume to collect everything, so we’ll give that a try at some point. For the moment, I will no doubt continue to get good results on ec2-based clusters, by pushing files to S3 with cron or a daemon of some kind, and using SQS to manage consumption by Hadoop and other things. (Who needs an Enterprise messaging infrastructure any more, now that we have SQS! Heh.) If I start running an all-internal cluster, I would think Flume would help a lot.
- Jonathan Gray from Facebook, who is an HBase committer, spoke about HBase. A lot of good stuff using Hadoop is going live soon at FB. He was secretive about which parts of Facebook were involved, but not about the core technology. Hive + HBase is growing quickly, as a way not to have to stream everything you have, when you want to look at some little thing in a Hive query. I’ll be doing more of that this year.
- Chris Gillette of Visible Measures took us all to school on how to manage your own Hadoop cluster on a budget, through hairy things like major version upgrades. All I can say is that with more control than you have on ec2, comes more responsibility.
- The MPP database vendors were there: Asterdata, Greenplum (now part of EMC), Vertica, Netezza. An emerging pattern is something like this:
- Install a Hadoop cluster.
- Realize that the Hive and Pig shells, although useful, don’t quite give your analysts everything they need. They might want a full-featured SQL interface, perhaps with some SAS-like features. Maybe if you’re very advanced with Hive+HBase+R you could keep up with their demands, but without a decent-sized team of crack people, this could easily lead to expertise bottlenecks, delay and disappointment.
- Install a Vertica/Asterdata/Greenplum/Netezza appliance/cluster on the side, into which your Hadoop results, less fully baked than they would have to be without such a thing, are being fed. This might work pretty well with Infobright as well. At least in the case of Greenplum, you can do an interesting variant of this idea by installing a Greenplum cluster, side by side with your Hadoop cluster, on the same nodes. That would give you a full-featured Postgres-style sql interface (or whatever the vendors are giving you) to sharded data in column stores, on which you could do joins without waiting very long for results. And that would free the analysts, perhaps to a significant degree. YMMV. Make sure you’re getting your file formats and storage engines right, because more data size in these MPP DBs will cost you a lot in license fees.
Secondly, the more theoretical stuff.
- RHIPE is an R + Hadoop thing by Saptarshi Guha of Purdue University, that allows you to define Map-Reduce jobs as list mapping operations without leaving the R shell. Spooky. You can’t exactly just install it from CRAN yet, but this could be awesome for the modeler crowd.
- My prize for the most fascinating talk goes to Daniel Abadi of Yale University, expanding on a blog post from last year. He humorously revisited the whole Stonebraker/DeWitt blog post that stoked an infamous Hadoop/DBMS controversy a while ago. Asking for a show of hands, he surveyed the room as to whether Hadoop and the MPP DBs were cooperative or competitive technologies. The sense of the room was, ‘cooperative’. See the last of the practical suggestions above: there’s a lot of momentum in the field around that cooperation. But Professor Abadi is a contrarian, and, in fact, a contrarian with benchmarks! He was asserting that Hadoop and the MPP DBs are actually competitive technologies, even if there are successful to highly varying degrees in different areas at the moment (and so are usefully being made to cooperate for now). In particular, Hadoop does not do well on performance of certain kinds of SQL-query-like MR jobs, and the MPP DBs do not do as well at scaling past a certain point. Enter Hadoop DB. The idea is to integrate Hadoop and a good column-store engine at a deeper level than what the MPP DB vendors have done thus far through superficial systems integration with Hadoop. The researchers (Professor Abadi’s students, if I have this right) tried a couple of column stores, and got very good performance results with an Ingres-based one, while preserving the linear scalability of Hadoop. IMHO this is potentially very disruptive, and very threatening to the MPP DB vendors. We’ll be keeping an eye on this effort.
Thanks to Cloudera for putting this on.
Posted in technology.
– October 20, 2010
This post on the StyleFeeder tech blog is a HOWTO for taking a Cloudera Hadoop distribution in the 0.20 series, patching it for yourself, and running a Hadoop cluster on EC2 based on it.
– January 22, 2010
My heartfelt congratulations to my clients and colleagues at StyleFeeder, which is being acquired by Time, Inc. Time is getting a tremendous asset, technology that will give them an edge, and top talent.
– January 18, 2010
My colleagues and clients at StyleFeeder are good enough to let me post on their tech blog from time to time. I’m exploring Hadoop on their behalf, as partially described here: http://seventhfloor.whirlycott.com/2010/01/14/hadoop-for-the-lone-analyst/. That’s basically a HOWTO for Hadoop 0.20 + Apache logs + MySQL on EC2, with tips on streaming, compression, Pig, Redhat/CentOS and the Cloudera Python scripts for EC2.
– January 14, 2010
After a couple of virtual machine upgrades that worked pretty well, I decided to upgrade my main workhorse laptop (a 15″ Lenovo W500) from Windows Vista 64-bit Ultimate to Windows 7 64-bit Ultimate. You can only upgrade to an identical subtype, without 3rd-party tools or a lot of hacking. The result? More disk space than when I started, and suddenly all my devices work. Skype is no longer helpless at finding the built-in video camera. It had been complaining that the camera was in already in use. When I plug in headphones, the speakers go silent without my having to find that really obscure control panel check box. The ATI Radeon HD 3650, which Lenovo is calling the ATI Mobility FireGL V5700, seems to have discovered the DisplayPort connection to my Dell 2408WFP, and the external monitor picture is really crisp now. You have to change the display settings twice: first have it duplicate the display on both monitors, then extend it to the other monitor. Now it can find the external monitor without having to reboot, and without getting a link failure at some point (usually when you’re working on something really interesting). I upgraded to VMWare 7 while I was at it, and instead of doing this: (split across two screens), which was totally useless, and in a way comical, it’s allowing me to run Centos or Ubuntu across two monitors. I think it’s the better drivers/OS combo, not the VMware, that’s working now. Come to think of it, it’s pretty outrageous what all was broken before. Better late than never.
Posted in technology.
– November 14, 2009