How Do You Troubleshoot the Message “ERROR: but there is no YARN_RESOURCEMANAGER_USER defined.”?

Problem scenario
You run sudo bash start-yarn.sh but you receive this message:

ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.

What should you do?

Solution
1. Modify start-yarn.sh. Underneath the last section of comments, place three lines with the following text:

YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root

2. Modify stop-yarn.sh. Underneath the last section of comments,

 » Read more..

How Do You Solve This Problem “Error: Could not find or load main class org.apache.hadoop.util.VersionInfo”

Problem scenario
You run “hadoop version” but you receive this message “Error: Could not find or load main class org.apache.hadoop.util.VersionInfo”. What do you do?

Possible Solution #1
Use “sudo ” before the “hadoop version” command.

Possible Solution #2
Use “sudo -i ” before the “hadoop version” command.

Possible Solution #3
Use a different user.

 » Read more..

How Do You Get Hadoop Commands to Work from Any Directory without Using the Full Path?

Problem scenario
Hadoop is installed on Linux. But hadoop version and other hadoop commands are not working. What should you do?

Solution
Find the hadoop executable in a directory named bin. It is often “/usr/local/hadoop/bin/hadoop”. Ultimately you need to find the directory that houses this “bin.” has a subdirectory with “bin” and “hadoop” inside, run these two commands:

sudo find / -name hadoop -type f
whereis hadoop

Run these commands interactively where “/usr/local/hadoop” is the directory that is the parent of the subdirectory named “bin” that is the parent of the hadoop executable.

 » Read more..

How Do You Know if Hadoop is Installed (and the version if it is installed) on Linux SUSE?

Problem scenario
You are administering Linux SUSE machines. You want to see if Hadoop is installed on them. The command hadoop version does not work.

Solution
Run this command:

sudo find / -name hadoop -type f

From the results above, you can probably find the file and path of the executable. It will likely not be in /var/ or /tmp/.

 » Read more..

How Do You Troubleshoot the HDFS Error “failed on connection exception: java.net.ConnectException: Connection refused;”?

Problem scenario
You have a multi-node Hadoop cluster running Hadoop version 3. You run this command: hdfs dfsadmin -report

You receive an error that includes this message: “failed on connection exception: java.net.ConnectException: Connection refused; “

What should you do?

Potential Solution
Run these three commands:

bash /usr/local/hadoop/sbin/stop-dfs.sh
hdfs namenode -format
bash /usr/local/hadoop/sbin/start-dfs.sh

 » Read more..

How Do You Troubleshoot “Error: Could Not Create the Java Virtual Machine”?

Problem scenario
You ran a Hadoop command but you receive this error:

Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

What do you do?

Solution
Run hadoop help. This error can happen when you have an incorrect flag.

 » Read more..

How Do You Troubleshoot the Message “ERROR: but there is no HDFS_DATANODE_USER defined.”?

Problem scenarios
One of the following apply to you.

Situation 1:
You run “start-dfs.sh” and it seems to work, but the “jps” command does not show that “DataNode” is running.

OR

Situation 2:
You run “sudo bash start-dfs.sh” but you receive this message:

ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined.

 » Read more..

How Do You Get hdfs or Yarn to Start When You Get an Error Such As “Permission denied (publickey,gssapi-keyex,gssapi-with-mic)”?

Problem scenario
You try to use start-dfs.sh or start-yarn.sh.  You received this message:  Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

What do you do?

Solution
You need to be able to ssh into the node without any Hadoop components.  To help you troubleshoot, consider the following items on the server that is causing the problem (e.g., the DataNode server, but it could be the NameNode server itself):

1. 

 » Read more..

How Do You Get hdfs or Yarn to Start the Jps Process You Expect to Start?

Problem scenario
You use start-dfs.sh and start-yarn.sh and there are no errors.  They seem to work, but when you use the jps command, you do not see the service you expect.  Why isn’t a jps process starting when you run one of these scripts?

Solution
The root cause could be a variety of root causes.  Here are some potential solutions.

1.  In these files,

 » Read more..

How Do You List the Hostnames of the Nodes in Your HDFS Cluster?

Problem scenario
You want to see what servers are in your HDFS cluster (e.g., the DataNodes and the NameNodes).  What command can display the FQDNs of such servers?

Solution
Run this command from the NameNode:

hdfs dfsadmin -report | grep Hostname

 » Read more..