How Do You Troubleshoot the HDFS Error “failed on connection exception: java.net.ConnectException: Connection refused;”?

Problem scenario
You have a multi-node Hadoop cluster running Hadoop version 3. You run this command: hdfs dfsadmin -report

You receive an error that includes this message: “failed on connection exception: java.net.ConnectException: Connection refused; “

What should you do?

Potential Solution
Run these three commands:

bash /usr/local/hadoop/sbin/stop-dfs.sh
hdfs namenode -format
bash /usr/local/hadoop/sbin/start-dfs.sh

 » Read more..

How Do You Troubleshoot “Error: Could Not Create the Java Virtual Machine”?

Problem scenario
You ran a Hadoop command but you receive this error:

Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

What do you do?

Solution
Run hadoop help. This error can happen when you have an incorrect flag.

 » Read more..

How Do You Troubleshoot the Message “ERROR: but there is no HDFS_DATANODE_USER defined.”?

Problem scenarios
One of the following apply to you.

Situation 1:
You run “start-dfs.sh” and it seems to work, but the “jps” command does not show that “DataNode” is running.

OR

Situation 2:
You run “sudo bash start-dfs.sh” but you receive this message:

ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined.

 » Read more..

How Do You Get hdfs or Yarn to Start When You Get an Error Such As “Permission denied (publickey,gssapi-keyex,gssapi-with-mic)”?

Problem scenario
You try to use start-dfs.sh or start-yarn.sh.  You received this message:  Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

What do you do?

Solution
You need to be able to ssh into the node without any Hadoop components.  To help you troubleshoot, consider the following items on the server that is causing the problem (e.g., the DataNode server, but it could be the NameNode server itself):

1. 

 » Read more..

How Do You Get hdfs or Yarn to Start the Jps Process You Expect to Start?

Problem scenario
You use start-dfs.sh and start-yarn.sh and there are no errors.  They seem to work, but when you use the jps command, you do not see the service you expect.  Why isn’t a jps process starting when you run one of these scripts?

Solution
The root cause could be a variety of root causes.  Here are some potential solutions.

1.  In these files, is there an all uppercase stanza like one or both of the following?

 » Read more..

How Do You List the Hostnames of the Nodes in Your HDFS Cluster?

Problem scenario
You want to see what servers are in your HDFS cluster (e.g., the DataNodes and the NameNodes).  What command can display the FQDNs of such servers?

Solution
Run this command from the NameNode:

hdfs dfsadmin -report | grep Hostname

 » Read more..

Announcement: Big Data Quiz Now Available!

We worked very hard to write an original twelve-question Big Data Quiz.  Please do not expect another posting (like we normally post) for several days.  Please be sure to check out the Big Data Quiz!

 » Read more..

How Do You Troubleshoot an Empty Multi-node Hadoop Cluster?

Problem scenario
One or more of the following is happening:
1) There are 0 DataNodes in your Hadoop cluster according to an error message
2) There is 0 B configured as capacity (as shown from a “hdfs dfsadmin -report” command).
3)  There is one fewer DataNode in your Hadoop cluster than you expect.
4)  You run “hdfs dfsadmin -report | grep Hostname” and do not see a node that has its DataNode service (as seen with the jps command) started and stopped from the NameNode with corresponding start-dfs.sh and stop-dfs.sh script runs.

How Do You Add a New Node to a Hadoop Cluster?

Problem scenario
You have a multi-node cluster of Hadoop.  You want to add a new data node.  What do you do?

Solution
1. a)  Log into the server that will be the new DataNode.  Do these things until you get to step 2.

b)  Install Hadoop on the new DataNode.  If you do not know how, see this posting.

c) 

 » Read more..

How Do You Install Hadoop with a Script for Any Type of Linux Server?

Updated on 1/22/19

Problem scenario
You want to install open source Hadoop.  You may want a single-node or multi-node deployment with CentOS/RedHat/Fedora, Debian/Ubuntu, and/or SUSE Linux distributions.  You want to have most of it scripted and have the same script work on any variety of Linux.  How do you install Hadoop quickly with a script that works on almost any type of Linux?

Solution
1. 

 » Read more..