Dnfox269

The following are my notes on how to setup a freshly installed Centos 6.5 machine to compile the Hadoop native library. My reason for doing this was to add a modified version of the native library which utilized hardware compression. In order to accomplish that, I needed to recompile the native library. In a previous post, “Integrating Hardware Accelerated Gzip Compression into Hadoop”, I described the process for modifying the Hadoop native library to use a hardware compression accelerator.

How to resolve Hadoop warning 'WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform using builtin-java classes where. Feb 19, 2005 While setting up a new cluster with Hadoop (3.1.1) and Spark (2.4.0), I encountered these warnings when running spark: 19/02/05 13:06:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform. Using builtin-java classes where applicable. To debug this issue, I used a useful trick explained on stackoverflow. Added Hadoop native library to LDLIBRARYPATH on.bashrc file and reload the library into the current session using source /.bashrc export LDLIBRARYPATH =$HADOOPHOME /lib/native export LDLIBRARYPATH = /usr/lib/hadoop/lib/native If this doesn’t resolve, please try next step below. Solution 4: Copy the Native Libraries.

In that previous post I utilized Hadoop 2.2. In this post I’m using Hadoop 2.4 because that’s the version of Hadoop that matches what is utilized by Ambari 1.6.0, the management tool I’ve chosen to administer my current cluster. Additionally, I was using Ubuntu 12.04 in the previous post, whereas in this post I’m using Centos 6.5. Ubuntu is typically my default Linux flavor, but because it is not compatible with Ambari, I installed Centos 6.5.

Hadoop Native Library For Pc

All of these steps assume you are logged in as “root”. If you aren’t, prefix the commands below with “sudo”.

Hadoop Native Library Linux

1. Install the development tools required to build the native library via the package manager. (Note: Some of these may already be installed)

2. Optionally install these packages. (Note: These steps aren’t required to build Hadoop’s native library, I just find myself having to install them eventually)

   • Install the kernel development package, vi, and git using the package manager
   • Install hexedit manually
   • If you have an AHA Hardware Compression Accelerator installed, install/load the driver, and build the hardware accelerated zlib variant. See Steps 1-2 of this post.

3. Install Oracle Java JDK 1.7
   You might already have this JDK installed, otherwise download Oracle Java JDK 7.1; The file should be named something similar to “jdk-7u71-linux-x64.tar.gz”. On my machine which is also a datanode (installed via Ambari) the JDK was located in /usr/jdk64/jdk1.7.0_67/.

   • Without the JDK already installed do this:
   • With the JDK already installed to /usr/jdk64/jdk1.7.0_67/ do this:
4. Install Maven 3.0.5
5. Install Ant 1.9.4
6. Install Protobuf 2.5.0
7. Download Hadoop 2.4.0 source
8. Compile the Hadoop native library. This step takes a while. On my machine it takes around 50 minutes to complete. (Note: Make any changes to the native library, such as adding support for hardware compression, before this step).
9. Backup Hadoop’s native library, and then replace it with the version compiled in step 8 (Note: The path to the hadoop libraries below are the default location that Ambari installs to. Your installation path will vary depending on how you went about your Hadoop installation)

   That’s it!

Hadoop Native Library 64 Bit Download