Problem scenario
Hadoop is not working correctly. You can get either a mapreduce job to work or a "hadoop" CLI command to work. But neither work unless you change an .XML file in between each operation.
Map Reduce jobs, when they are failing (and the Hadoop commands are working), there may be an error like this:
2021-01-01 22:47:42,337 INFO mapreduce.Job: Task Id : attempt_1609558624072_0001_m_000003_1, Status : FAILED
Container launch failed for container_1609558624072_0001_01_000008 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
When the Hadoop jobs are failing (and the Map Reduce jobs are working), there may be an error like this:
Exception in thread "main" java.lang.NullPointerException
at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:784)
at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:769)
at org.apache.hadoop.mapreduce.tools.CLI.listAllJobs(CLI.java:697)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:428)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1277)
You make changes to a yarn-site.xml file and try one of the jobs. They work alternately. You see that the changes take effect when you run a map reduce job and separately a hadoop command to view the jobs. Either the map reduce job will work or the hadoop command will work. Both will not work unless you make changes to the yarn-site.xml. You have to change the yarn-site.xml file each time before issuing one of the commands. You know you don't need to restart anything because you get different behavior as soon as you make changes to the yarn-site.xml. How do you get both to work simultaneously?
Solution
Restart DFS and YARN servcices. Use stop-dfs.sh and stop-yarn.sh scripts. Then use start-dfs.sh and start-yarn.sh scripts.
You must restart the DFS and YARN services. Yes, you see different behavior immediately after changing the yarn-site.xml or mapred-site.xml files. However, for the changes to take full effect, you must restart the services. Then both the mapreduce jobs and the hadoop CLI commands will work.