Tuesday, July 30, 2013

Building Hadoop from source

Hadoop building process from Source Tree

STEP 1:  Install all required packages in respective OS, I am using Ubuntu 13.04
                 * Unix System
                 * JDK 1.6
                 * Maven 3.0
                 * Findbugs 1.3.9 (if running findbugs)
                 * ProtocolBuffer 2.4.1+ (for MapReduce and HDFS)
                 * CMake 2.6 or newer (if compiling native code) 
                 * Internet connection for first build (to fetch all Maven and Hadoop dependencies)

STEP 2: Download source from Git repo as it most suitable for nightly build
              Git Repo: git://git.apache.org/hadoop-common.git

STEP 3: Build hadoop-maven-plugin
               cd hadoop-maven-plugin
               mvn install
               cd ..
STEP 4:  Optional to Build native libs 
               mvn compile -Pnative
STEP 5:  Optional findbugs (need in case want to create site)
                mvn findbugs:findbugs
STEP 6:  Building distributions:

               Create binary distribution without native code and without documentation:
               $ mvn package -Pdist -DskipTests -Dtar

               Create binary distribution with native code and with documentation:
               $ mvn package -Pdist,native,docs -DskipTests -Dtar

               Create source distribution:
               $ mvn package -Psrc -DskipTests

               Create source and binary distributions with native code and documentation:
               $ mvn package -Pdist,native,docs,src -DskipTests -Dtar

               Create a local staging version of the website (in /tmp/hadoop-site)
               $ mvn clean site; mvn site:stage -DstagingDirectory=/tmp/hadoop-site


Reference :  GitHub

No comments:

Post a Comment