A Technical I.T./DevOps Blog
Problem scenario
You run sudo bash start-yarn.sh but you receive this message:
ERROR: Attempting to operate on yarn resourcemanager as root
ERROR: but there is no YARN_RESOURCEMANAGER_USER defined. Aborting operation.
What should you do?
Solution
1. Modify start-yarn.sh. Underneath the last section of comments, place three lines with the following text:
YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root
2. Modify stop-yarn.sh.
…
Problem scenario
You run “hadoop version” but you receive this message “Error: Could not find or load main class org.apache.hadoop.util.VersionInfo”. What do you do?
Possible Solution #1
Use “sudo ” before the “hadoop version” command.
Possible Solution #2
Use “sudo -i ” before the “hadoop version” command.
Possible Solution #3
Use a different user.
…
Problem scenario
Hadoop is installed on Linux. But hadoop version and other hadoop commands are not working. What should you do?
Solution
Find the hadoop executable in a directory named bin. It is often “/usr/local/hadoop/bin/hadoop”. Ultimately you need to find the directory that houses this “bin.” has a subdirectory with “bin” and “hadoop” inside, run these two commands:
sudo find / -name hadoop -type f
whereis hadoop
Run these commands interactively where “/usr/local/hadoop” is the directory that is the parent of the subdirectory named “bin” that is the parent of the hadoop executable.
…
Problem scenario
You are administering Linux SUSE machines. You want to see if Hadoop is installed on them. The command hadoop version does not work.
Solution
Run this command:
sudo find / -name hadoop -type f
From the results above, you can probably find the file and path of the executable. It will likely not be in /var/ or /tmp/.
…
Problem scenario
You have a multi-node Hadoop cluster running Hadoop version 3. You run this command: hdfs dfsadmin -report
You receive an error that includes this message: “failed on connection exception: java.net.ConnectException: Connection refused; “
What should you do?
Potential Solution
Run these three commands:
bash /usr/local/hadoop/sbin/stop-dfs.sh
hdfs namenode -format
bash /usr/local/hadoop/sbin/start-dfs.sh
…
Problem scenario
You ran a Hadoop command but you receive this error:
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
What do you do?
Solution
Run hadoop help. This error can happen when you have an incorrect flag.
…
Continue reading “How Do You Troubleshoot “Error: Could Not Create the Java Virtual Machine”?”
Problem scenario
You want to know if a MapReduce job succeeded. What do you do?
Solution
Run one or both of these pairs of commands:
hadoop job -list all
hadoop job -list all | grep SUCCEEDED
mapred job -list all
mapred job -list all | grep SUCCEEDED
…
Continue reading “How Do You Know if a MapReduce Job Succeeded?”
Problem scenarios
One of the following apply to you.
Situation 1:
You run “start-dfs.sh” and it seems to work, but the “jps” command does not show that “DataNode” is running.
OR
Situation 2:
You run “sudo bash start-dfs.sh” but you receive this message:
ERROR: Attempting to operate on hdfs datanode as root
ERROR: but there is no HDFS_DATANODE_USER defined.
…
Problem scenario
You try to use start-dfs.sh or start-yarn.sh. You received this message: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
What do you do?
Solution
You need to be able to ssh into the node without any Hadoop components. To help you troubleshoot, consider the following items on the server that is causing the problem (e.g., the DataNode server, but it could be the NameNode server itself):
1.
…
Problem scenario
You use start-dfs.sh and start-yarn.sh and there are no errors. They seem to work, but when you use the jps command, you do not see the service you expect. Why isn’t a jps process starting when you run one of these scripts?
Solution
The root cause could be a variety of root causes. Here are some potential solutions.
1. In these files,
…
Continue reading “How Do You Get hdfs or Yarn to Start the Jps Process You Expect to Start?”