How to Build Ruydoop Yourself

Recently I come working with Hadoop and the better Gem I’ve seen is Rubydoop, but it has an issue with Hadoop 2.0.x, basically does not support it because has been changed its API.

I created an issue and few days late received a patch to fix it :D, but with a condition: I should build the .jar myself (and gem).

I’ll expect that you have set up hadoop 2.0.x.

1 – Clone repository

Very simple, you need clone the repository and switch to branch that has fix.

1
2
3

git clone git@github.com:iconara/rubydoop.git && \
cd rubydoop && \
git checkout hadoop_204

Branch hadoop_204 has commit.

2 – Export MY_RUBY_HOME

The Rakefile expects that you have an environment variable called MY_RUBY_HOME

1

export MY_RUBY_HOME=/path/home/jruby

The above path depends on your SO.

3 – Create .classpath

This file was created me more problems, because there is not nothing clean about what is the right content. In fact, I followed suggestion made by @iconara, but did not work for me.

This file must be created in root dir where you cloned repository.

As I mentioned earlier, this post expect that you have set up hadoop 2.0.x

What path I should put into .classpath file?

1

hadoop classpath

The previous command to show you the HADOOP_CLASSPATH, basically where Hadoop has their jars files.

I needed to put two jars into .classpath to everything worked as expected.

Here my .classpath

1
2

/path/hadoop/common/hadoop-common-2.0.4-alpha.jar
/path/hadoop/mapreduce/hadoop-mapreduce-client-core-2.0.4-alpha.jar

Once you set the right paths the rest it’s very simple, you need run the following command.

1

rake build

Whether everything is fine, the previous command should build lib/rubydoop.jar (reason of this post).

1

gem build rubydoop.gemspec

The above command will produce an output like:

1
2
3
4

Successfully built RubyGem
Name: rubydoop
Version: 1.1.0.pre7
File: rubydoop-1.1.0.pre7.gem

Finally, You need install your gem that recently has been built. Just put it in the working directory and run gem install rubydoop Rubygems picks .gem files in the working directory over gems on rubygems.org.

Last but not least, I suggest change the gem version, by this way you can add a specific version in your Gemfile and avoid conflict with current release version. In fact, as you can see I changed Rubydoop version to 1.1.0.pre7.