Problem scenario
You downloaded and installed Hadoop core. But you cannot find the core-site.xml file. What should you do?

Possible Solution #1
Run this: sudo find / -name core-site.xml

Possible Solution #2
Did you download the installation media with a file “-site” in its name?

Try again with a .tar.gz file without “-site” (and without “-src”) in its name:
https://dlcdn.apache.org/hadoop/core/stable/

(This is the for open source version and not a specific vendor’s implementation of Hadoop.)

…

Continue reading “Where Is the core-site.xml File in a Hadoop Installation?”

Problem scenario
You get a “Segmentation fault” or “Segmentation fault (core dumped)” error when you run any “hdfs” command. What should you do?

Solution

Root cause
There is probably an infinite loop/recursion problem. (Segmentation faults involve writing to the stack. Eventually the memory gets filled up.) There is some configuration problem with your Hadoop / hdfs installation.

…

Continue reading “How Do You Troubleshoot “Segmentation fault” Errors in Hadoop/HDFS?”

Problem Scenario
You run an HDFS command, but you get this message: “ERROR: Hadoop common not found”

What should you do?

Solution
As the hduser or user that runs hdfs, log in. Run “echo $HADOOP_HOME”

That directory should have a libexec directory with a file called hadoop-config.sh.

Run this: ls -lh $HADOOP_HOME/libexec

One way to create it is this: 1) find a hadoop-config.sh file (e.g.,

…

Continue reading “How Do You Troubleshoot “ERROR: Hadoop common not found” when Running Hadoop?”

Problem scenario
You are running a Hadoop command, but you get this message:

java.net.ConnectException: Your endpoint configuration is wrong; For more details see: http://wiki.apache.org/hadoop/UnsetHostnameOrPort, while invoking ApplicationClientProtocolPBClientImpl.getApplicationReport over null after 3 failover attempts. Trying to failover after sleeping for 22088ms.
2020-12-20 18:54:36,679 INFO ipc.Client:

What should you do?

Possible Solution
Is Resource Manager running? Start a new terminal and run “jps” to find out.

…

Continue reading “How Do You Troubleshoot the Hadoop Error “ApplicationClientProtocolPBClientImpl.getApplicationReport”?”

Problem scenario
You run a Hadoop command, but you get this error:

2020-12-20 18:19:33,706 INFO client.DefaultNoHARMFailoverProxyProvider: Connecting to ResourceManager at /0.0.0.0:8032
2020-12-20 18:19:36,068 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

You tried restarted the start-dfs script with this command: bash /usr/local/hadoop/sbin/start-dfs.sh

It did not help. You ran “jps” to see if Resource Manager was running.

…

Continue reading “How Do You Troubleshoot the Hadoop Error “Connecting to ResourceManager”?”

Problem scenario
Hadoop is not working correctly. You can get either a mapreduce job to work or a “hadoop” CLI command to work. But neither work unless you change an .XML file in between each operation.

Map Reduce jobs, when they are failing (and the Hadoop commands are working), there may be an error like this:

2021-01-01 22:47:42,337 INFO mapreduce.Job: Task Id : attempt_1609558624072_0001_m_000003_1,

…

Continue reading “How Do You Get Both MapReduce Jobs and hadoop CLI Commands to Work Simultaneously without Alternately Changing an XML File Before Each One?”

Problem scenario
You have run start-dfs.sh and start-yarn.sh. You have stopped all the Hadoop services too. When you run “jps”, the NameNode is not showing up. You have tried a variety of different troubleshooting methods (including rebooting the NameNode). The NameNode has never worked correctly. You can delete all the data in the cluster because it never really worked. What should you do?

Solution
Run this but remember it will delete all your data:

hdfs namenode -format # Warning: this command will delete all your data in Hadoop …

Continue reading “How Do You Get the NameNode Process to Start in Hadoop?”

Problem scenario
When you run hadoop commands, you get an error like this:

Exception in thread “main” java.lang.NullPointerException
at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:784)
at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:769)
at org.apache.hadoop.mapreduce.tools.CLI.listAllJobs(CLI.java:697)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:428)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1277)

What should you do?

Solution
Find the mapred-site.xml file. Make sure it has these stanzas (within the configuration and /configuration tags):

<property> …

Continue reading “How Do You Troubleshoot ‘Exception in thread “main” java.lang.NullPointerException org.apache.hadoop.mapreduce.tools.CLI.displayJobList’?”