{"id":57,"date":"2012-05-24T10:20:38","date_gmt":"2012-05-24T14:20:38","guid":{"rendered":"http:\/\/lichun.cc\/blog\/?p=57"},"modified":"2012-05-24T10:20:38","modified_gmt":"2012-05-24T14:20:38","slug":"build-local-haoop-cluster-on-linux","status":"publish","type":"post","link":"https:\/\/www.lichun.cc\/blog\/2012\/05\/build-local-haoop-cluster-on-linux\/","title":{"rendered":"Build Local Single Node Hadoop Cluster on Linux"},"content":{"rendered":"

This post shows how to build a local single node Hadoop cluster on Linux.<\/p>\n

Prerequisite:<\/h2>\n

(1) Install JDK , Download Link<\/a><\/p>\n

(2) install ANT, Download Link<\/a><\/p>\n

Use the bin version, and add the following lines in your
\n~\/.bash_profile<\/strong><\/p>\n

export ANT_HOME=YOUR_ANT_PATH\nexport JAVA_HOME=YOUR_JDK_PATH\nexport PATH=${PATH}:${ANT_HOME}\/bin<\/pre>\n
 <\/p>\n
Install Hadoop:<\/h2>\n
(1) Download Hadoop, Download Link<\/a>
\n
\n(2) uncompress the file, move the folder to wherever you want, I use ~\/hadoop\/<\/p>\n
(3) enter ~\/hadoop\/, run<\/p>\n
ant<\/pre>\n
(4)\u00a0Put localhost in two files, conf\/masters and conf\/slaves:<\/p>\n
echo localhost > conf\/masters;\necho localhost > conf\/slaves;<\/pre>\n
(5) In conf\/core-site.xml:<\/p>\n
<?xml version=\"1.0\"?>\n<?xml-stylesheet type=\"text\/xsl\" href=\"configuration.xsl\"?>\n<configuration>\n  <property>\n    <name>fs.default.name<\/name>\n    <value>hdfs:\/\/127.0.0.1:9000<\/value>\n  <\/property>\n<\/configuration><\/pre>\n
(6) In\u00a0conf\/mapred-site.xml:<\/p>\n
<?xml version=\"1.0\"?>\n<?xml-stylesheet type=\"text\/xsl\" href=\"configuration.xsl\"?>\n<configuration>\n  <property>\n    <name>mapred.job.tracker<\/name>\n    <value>localhost:9001<\/value>\n  <\/property>\n\n  <property>\n    <name>dfs.datanode.socket.write.timeout<\/name>\n    <value>0<\/value>\n  <\/property>\n<\/configuration><\/pre>\n
(7) edit conf\/hadoop-env.sh, uncomment the JAVA_HOME line, and put the right path for JDK<\/p>\n
export JAVA_HOME=YOUR_JDK_PATH<\/pre>\n
(8) Run Hadoop !<\/p>\n
.\/bin\/start-all.sh<\/pre>\n
 <\/p>\n
After Running:<\/h2>\n
(1)\u00a0Check that the daemons are running using jps (shows java processes):<\/p>\n
jps<\/pre>\n
Should show 5 processes and Jps (and maybe more if you are running e.g. Eclipse):<\/p>\n
17400 SecondaryNameNode\n17172 NameNode\n17599 TaskTracker\n17279 DataNode\n17493 JobTracker\n17699 Jps<\/pre>\n
Check the local NameNode:\u00a0http:\/\/localhost:50070\/dfshealth.jsp<\/a>\u00a0should show one live node.<\/p>\n
Check the local JobTracker:\u00a0http:\/\/localhost:50030\/jobtracker.jsp<\/a>\u00a0should show ‘State: RUNNING’, not ‘INITIALIZING’.<\/p>\n
If the Namenode is not running, format the Namenode, and run start-all.sh again:<\/p>\n
rm -r \/tmp\/hadoop-*;\nbin\/hadoop namenode -format;\n.\/bin\/start-all.sh<\/pre>\n
This can be caused by switching Hadoop versions.
\n(2) Sanity test<\/p>\n
hadoop fs -ls \/<\/pre>\n
(3) Word Count Example:<\/p>\n
Please Look Here<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
This post shows how to build a local single node Hadoop cluster on Linux. Prerequisite: (1) Install JDK , Download Link (2) install ANT, Download Link Use the bin version, and add the following lines in your ~\/.bash_profile export ANT_HOME=YOUR_ANT_PATH export JAVA_HOME=YOUR_JDK_PATH export PATH=${PATH}:${ANT_HOME}\/bin   Install Hadoop: (1) Download Hadoop, Download Link<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[19],"tags":[15,16,17,18],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p2s9sh-V","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.lichun.cc\/blog\/wp-json\/wp\/v2\/posts\/57"}],"collection":[{"href":"https:\/\/www.lichun.cc\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.lichun.cc\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.lichun.cc\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lichun.cc\/blog\/wp-json\/wp\/v2\/comments?post=57"}],"version-history":[{"count":0,"href":"https:\/\/www.lichun.cc\/blog\/wp-json\/wp\/v2\/posts\/57\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.lichun.cc\/blog\/wp-json\/wp\/v2\/media?parent=57"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.lichun.cc\/blog\/wp-json\/wp\/v2\/categories?post=57"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lichun.cc\/blog\/wp-json\/wp\/v2\/tags?post=57"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}