The following is a repost of my answer to a question on LinkedIn, but I thought it may prove useful to people evaluating Hadoop distributions. The following is a substantially over simplified set of choices (in alphabetical order): Amazon: Apache Hadoop provided as a web service. Good solution if your data is collected on Amazon…saves …
Once you start to use Hadoop in your day-to-day business operations, you’ll quickly find that uptime is an important consideration. No one wants to explain to the CEO why a report is not delivered. While most of Hadoop’s architecture is designed to work in the face of node failure (such as the DataNodes), other components …
Introduction Start the HBase Shell All subsequent commands in this post assume that you are in the HBase shell, which is started via the command listed below. hbase shell You should see output similar to: 12/08/12 12:30:52 WARN conf.Configuration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available HBase Shell; enter ‘help<RETURN>’ for list of supported commands. Type …
Introduction I ran into an annoying error in HBase due to the localhost loopback. The solution was simple, but took some trial and error. Error I was following the HBase logs with the following command: tail -1000f /var/log/hbase/hbase-hbase-master-freshstart.log The following error kept poping up in the log file. org.apache.hadoop.hbase.master.AssignmentManager: Unable to find a viable location …
Introduction We’re going to start with a very simple Pig script that reads a file that contains 2 numbers per line separated by a comma. The Pig script will first read the line, store each of the 2 numbers in separate variables, and will then add the numbers together. Create the Sample Input File cd …
Introduction Installing Pig is drop dead simple. Installation sudo apt-get install pig Check the Pig version. pig –version Setup the Environment We’re going to set the environment variables system-wide for Pig programming. sudo vi /etc/environment Paste the following environment variables into the environment file. HADOOP_MAPRED_HOME=”/usr/lib/hadoop-mapreduce” PIG_CONF_DIR=”/etc/pig/conf” source /etc/environment That’s it. You can now start to …
Introduction The hadoop fs -ls command allows you to view the files and directories in your HDFS filesystem, much as the ls command works on Linux / OS X / *nix. Default Home Directory in HDFS A user’s home directory in HDFS is located at /user/userName. For example, my home directory is /user/akbar. List the …
Introduction hdfs dfsadmin -metasave provides additional information compared to hdfs dfsadmin -report. With hdfs dfsadmin -metasave provides information about blocks, including> blocks waiting for replication blocks currently being replication total number of blocks hdfs dfsadmin -metasave filename.txt Run the command with sudo -u hdfs prefixed to ensure you don’t get a permission denied error. CDH4 …
Introduction hdfs dfsadmin -report outputs a brief report on the overall HDFS filesystem. It’s a userful command to quickly view how much disk is available, how many datanodes are running, and so on. Command Run the command with sudo -u hdfs prefixed to ensure you don’t get a permission denied error. CDH4 runs the namenode …
Introduction HBase is a tabular-oriented database that runs on top of HDFS. It is modeled on Google’s BigTable. In this post, I’m going to install HBase in Pseudo mode, so please use these instructions for setting up a developer’s workstation, not for a production cluster. When should you use HBase HBase should be used when …
AkbarAhmed.com 