How Do You Resize a Server in Google Cloud Platform?

Problem scenario
You want to increase the amount of RAM that a VM instance has in GCP.  How do you do this?

Possible Solution #1
1.  Log into Google Cloud Platform
2.  In the "Cloud Engine" screen click on VM "instances."
3.  You should see a list of VM instances.  Under the "Recommendation" tab you will see "Increase perf."  Click on this. If you do not see "Increase perf", stop the machine. Then click on its hyperlinked name. Then click on "Edit". In the "Machine type" section click the drop down menu to resize the server to a different combination of CPU and memory.
4.  A pop-up will appear called "Resize instance."  Click "APPLY".
5.  Note that when the server is resized it may have a new external IP address.

Possible Solution #2
This solution was tested in January of 2019.
1. In the GCP console go to VM instances.
2. Check the box on the left for the server.
3. Click the "STOP" button near the top.
4. Click on the name of the VM instance which should be hyperlinked.
5. In the "VM instance details" page, click "EDIT".
6. In the "Machine type" section, click "Customize."
7. Select the number of cores and the amount of Memory you desire.
8. Scroll to the bottom and click "Save".

How Do You Get the Hadoop Yarn Web UI to Work?

Problem scenario
You want to get Hadoop's web UI to work.  You have access to the back-end of a Linux server.  How do you get the front-end of Hadoop (or hdfs) to work?

Solution
1.  Deploy Hadoop.  See this link for directions for any type of Linux. 
2.  Start the Hadoop services.  For a single-node deployment of Hadoop use sudo find / -name start-dfs.sh to find it; use bash /path/to/start-dfs.sh to run it.  For a multi-node deployment of Hadoop, log into the NameNode and use sudo find / -name start-all.sh to find it; use bash /path/to/start-all.sh to run it.
3.  If it is a single-node cluster, find the external IP address of the server with curl http://icanhazip.com.  If you have a multi-node Hadoop cluster, find the external IP address of the NameNode with curl http://icanhazip.com.
4.  Open a web browser from a workstation and go to http://<externalIPaddress>:8088

If it does not work, remember that the back-end of the server running yarn should show "ResourceManager" in the results of a command like this: sudo jps  # get ResourceManager to show first if there is a problem.  To install jps, see this link.

How Do You Get the HDFS Web UI to Work?

Problem scenario
You want to get Hadoop's web UI to work.  You have access to the back-end of a Linux server.  How do you get the front-end of Hadoop (or hdfs) to work?

Solution
1.  Deploy Hadoop.  See this article if you need help with how to do it. 
2.   From the NameNode server in a multi-node deployment or from the single Hadoop server, run the start-dfs.sh script (use sudo find / -name start-dfs.sh to find it; use bash /path/to/start-dfs.sh to run it).  
3.  If it is a single-node cluster, find the external IP address of the server with "curl icanhazip.com".  If you have a multi-node Hadoop cluster, find the external IP address of the NameNode with "curl icanhazip.com".
4.  Open a web browser from a workstation and go to http://<externalIPaddress>:50070
(The port could be different if you changed it from the default one.)

How Do You See the Storage Usage on a Datanode in an hdfs System?

Problem scenario
You want to see which datanodes are active underlying a given hdfs system.  You also want to know statistics about the storage usage.  If you are regularly adding data to your hdfs system, you want to stay below 70% utilization.  If you want your hdfs system to perform well but you are not regularly adding new files, you want to stay under 80%.  How do you find out about the storage usage of your datanodes that support your hdfs system?

Solution
Run this command:

hdfs dfsadmin -report
# or this command below
sudo /usr/local/hadoop/bin/hdfs dfsadmin -report

How Do You Troubleshoot the Hadoop Error “Could not find or load main class jar”?

Problem scenario
You are trying to run a Hadoop operation.  You issue your "hadoop jar" or "hdfs jar" command.  You get one of these errors:

"Error: Could not find or load main class jar"

or this error

"Error: Could not find or load main class hadoop-streaming-2.8.1.jar"

What is wrong?

Solution
1.  Use "hadoop..." instead of "hdfs".

2.  Make sure you have the hadoop-streaming*.jar file.  You could run this command to find it:

sudo find / -name hadoop-streaming-*.jar

If you cannot find it, you can download it from here.

3.  Remember to put the full path after the "hadoop jar ".  Here is an example:

hadoop jar /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.8.1.jar -file /home/hduser/mapper.py -mapper /home/hduser/mapper.py -file  /home/hduser/reducer.py -reducer /home/hduser/reducer.py -input /user/hduser/contint/* -output /user/hduser/gooddir-output

Alternatively you could use "cd /to/the/path/with/hadoop-streaming-2.8.1.jar" (where 2.8.1 is just an example).

What is MapReduce in Hadoop?

Problem scenario
You want to learn more about MapReduce.  You want to learn what it does so you can grasp Hadoop more thoroughly.  What is MapReduce?

Solution
MapReduce is a core process of Hadoop.  With multi-node deployments of Hadoop, MapReduce distributes data to different datanodes (servers that are controlled by a master server).  This data is retrievable thanks to MapReduce.  Conceptually there are two main components to MapReduce: mapper and reducer.  The mapper function creates temporary key-value pairs.   The reducer is composed of three phases: shuffle, sort and reduce.  Shuffle and sort happen simultaneously.  Together these components leverage commodity hardware in multi-node deployments.  If you want to learn more you can see this link.

After you run a MapReduce job, you'll see output that is a summary of the job.  The summary will include various quantifiable aspects of the job in the "Map-Reduce Framework" section of the  text output.

It is not hard to set up Hadoop and run a MapReduce job.  You can learn more about the process if you do this.  To deploy Hadoop on any type of Linux, see this link.  You can follow this link to run a MapReduce job for learning.

How Do You Cancel an hdfs Command That Appears to Be Making No Progress?

Problem scenario
An hdfs command hangs (or appears to be hung).  To interrupt the command, you try ctrl-c.  But this does nothing.  How do you cancel out?

Solution
Use ctrl-z.  This will suspend the job.  You can then kill it with kill, kill 9, or some other means.

How Do You Solve the Problem of Being Prompted for Credentials after You Run the start-dfs.sh Script?

Problem scenario
You try to start Hadoop's dfs in a multi-node deployment.  All the Hadoop nodes are running Linux.  You run this:

bash /usr/local/hadoop/sbin/start-dfs.sh

You see this:

Starting namenodes on [hadoopmaster]
hadoopmaster: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-hadoopmaster.out
root@hadoopdatanode's password: localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-hadoopmaster.out
hadoopmaster: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-hadoopmaster.out
localhost: ulimit -a for user root
localhost: core file size          (blocks, -c) 0
localhost: data seg size           (kbytes, -d) unlimited
localhost: scheduling priority             (-e) 0
localhost: file size               (blocks, -f) unlimited
localhost: pending signals                 (-i) 2544
localhost: max locked memory       (kbytes, -l) 64
localhost: max memory size         (kbytes, -m) unlimited
localhost: open files                      (-n) 1024
localhost: pipe size            (512 bytes, -p) 8

You notice that DFS is not running properly on the datanode (aka slave node).  You think the prompts for "root@hadoopdatanode" (the Hadoop slave server) are a sign of the  problem.  What do you do?

Solution
If you want to passwordlessly SSH as a given user from one server to another, the remote server's authorized_keys must have the public key from the local server (server from which you attempt to initiate the SSH connection).

In this case you are being prompted for a password.  The remote server would not do this if its /root/.ssh/authorized_keys had the id_rsa.pub file for the root  user on the starting (local) server.  The starting server is the Hadoop master server.

From the Hadoop namenode (aka master) server, do this:

cat /root/.ssh/id_rsa.pub

Copy the output to the last line of /root/.ssh/authorized_keys on the destination (non-master) Hadoop datanode server.

How Do You Set up a Multi-Node Cluster of Hadoop with Linux Servers?

Updated 1/30/18

Problem scenario
You want to deploy open-source Hadoop as a multi-node cluster to some Linux servers (e.g., two more more servers as opposed to a single-server deployment).  You want to use AWS, Azure and/or Google Cloud Platform Linux servers, possibly in a cross-cloud configuration (using two or three public cloud services).  What do you do to install and configure Hadoop to leverage two or more computers in a potentially hybrid cloud environment?

Solution
These directions work with all the servers being in AWS, Azure, Google Cloud Platform or a mix of each of them!  (These directions should work for a hybrid cloud deployment without much extra configuration for you.  We recommend using nmap and being methodical with checking ports very early in the process if you are deploying a hybrid cloud Hadoop multi-node cluster.)  These directions work regardless of what type of Linux you are using (e.g., CentOS/RedHat/Fedora, Debian/Ubuntu, or SUSE).  It can take less than 15 minutes if there are not too many nodes and the nodes are not too big.

You need just two Linux servers: one for the master server (referred to as the NameNode) and one for the slave node (referred to as the DataNode).  These directions were designed for open-source Hadoop.  The relevant security groups (for AWS, or NSGs for Azure, or firewall rules for Google Cloud Platform) need to allow connectivity between the two clouds.  If you are going to deploy it using two clouds, see the "Tips for deploying a multi-node cluster" at the bottom.

1.  Install Hadoop on at least two different Linux servers.  If you need directions, there is an article for each the three major distributions of Linux (CentOS/RedHat/Fedora, Debian/Ubuntu and SUSE).  For these directions below you will need a group called 'hadoop' and a user called 'hduser' on every NameNode and DataNode.  (These names could be different depending on your specific needs.  If you follow the hyperlinked article above, you will create this group and user so that the rest of these directions below will work.)  Remember your hduser account passwords.  Mentally designate one Linux server as the NameNode (previously referred to as the "master").  The other server(s) will be the DataNodes (previously known as slave(s)).

2.a.  Configure the /etc/hosts file that is on each NameNode (master) server and DataNode (slave) server(s).  A NameNode server's /etc/hosts file  must have an entry of each DataNode.  Datanodes' /etc/hosts files must have one entry for the NameNode server.  The IP address you use depends on  whether or not the server you are adding to the /etc/hosts file is in the same cloud.  If the NameNode and DataNode are in the same cloud, you  would use the internal IP address of the server for which you are entering.  For hetergeneous clouds (e.g., Azure having a NameNode and AWS having  a DataNode), make sure you use corresponding external IP addresses in the /etc/hosts file if the entry is referring to server in a different cloud.

2.b.  Ensure that the AWS Security Group, or Azure NSG, or Google Cloud Platform firewall rule does not restrict traffic between the NameNode and DataNode.  For AWS Security Group configuration, the IP addresses will be internal if they are all in the same AWS VPC and governed by the same Security Group.  If NameNode is in a heterogeneous public cloud relative to the DataNode, then external IP addresses will be needed for Rules to allow the servers to communicate.

3.  The NameNode must be able to one, passwordlessly ssh to itself with hduser; two, passwordlessly ssh to the DataNode(s) with hduser.  For the second one, from the NameNode server, run these commands:

su hduser
cat /home/hduser/.ssh/id_rsa.pub
# If the above is not found, you may need to run this: ssh-keygen -t rsa -P ""  # press enter to the next prompt

4.  Copy the content from the last command above and do on each DataNode server, append it to /home/hduser/.ssh/authorized_keys.  After you save the file, ensure that this file /home/hduser/.ssh/authorized_keys has rw------- permissions.  Use chmod 600 authorized_keys if necessary.

5.  On the Hadoop NameNode server, modify this file /usr/local/hadoop/etc/hadoop/workers
Add a new line with the domain name of each DataNode server.  It will look like this:

coolnameofserver

Remember to save the file.

6.   Modify the core-site.xml file (e.g., often found at /usr/local/hadoop/etc/hadoop/core-site.xml). Find the <property> tag and the tag inside it named  <value>.  Change the "localhost" to the domain name of the NameNode server itself.

7.  Modify /usr/local/hadoop/etc/hadoop/mapred-site.xml

In between the <configuration> and </configuration> tags, place this text with "NameNode" being replaced by the domain name of your NameNode server:

<property>
 <name>mapred.job.tracker</name>
  <value>NameNode:54311</value>
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
</property>

8.  Does the hdfs-site.xml (e.g., /usr/local/hadoop/etc/hadoop/hdfs-site.xml) have a dfs.replication value element that is correct?  The value is computed by adding the number of NameNode and DataNode servers that will be in the Hadoop cluster once the new node has been added.

Find the hdfs-site.xml file (e.g., /usr/local/hadoop/etc/hadoop/hdfs-site.xml).  Inside it find the "dfs.replication" element.  Underneath it will be a <value> number. Modify it to be "2" (with no quotes) if you are configuring one NameNode and one DataNode Hadoop server.  The number should be equivalent to total number of nodes that are your NameNode and DataNode servers in the Hadoop cluster you are configuring. 

If you find no such element, copy this text inside the <configuration> and </configuration> tags (but replace the "2" with the sum of your NameNode and DataNodes in your desired Hadoop cluster):

<property>
  <name>dfs.replication</name>
  <value>2</value>
  <description>Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.
  </description>
</property>

9.  Repeat steps 7 through 9 for each remaining DateNode server. 

10.  Start the multi-node instance. Log into the NameNode as hduser (e.g., su hduser).  Then run this command:  
bash /usr/local/hadoop/sbin/start-all.sh

Tips for deploying a multi-node cluster of Hadoop with both AWS and Azure
By default you will not be able to ping Azure instances.  But this does not mean SSH will not work.  ICMP packets are turned off by default in Azure.  

The datanodes can be in AWS as long as the AWS instances are governed by a Security Group which allows inbound connections originating from the Azure Hadoop NameNode.  To find its external IP address, go to the back end of the NameNode and run "curl http://icanhazip.com" (with no quotes).  The other requirement is that the Network Security Group in Azure for the Hadoop NameNode must allow communication from the AWS datanodes.

Tips for cluster deployments to hybrid clouds involving both AWS and Google Cloud Platform
The datanodes' DataNode service (as shown with the jps command) may be controlled by the NameNode, but the node may not be in the cluster unless the NameNode accepts connectivity over the port configured in the core-site.xml file.

You may want to see this posting for various troubleshooting when adding a node to the cluster.

How Do You Run a MapReduce Job (with Python)?

Problem scenario
You have studied what a MapReduce job is conceptually.  But you want to try it out.  You have a Linux server with the Hadoop namenode (aka master node) installed on it.  How do you run a MapReduce job [to understand what it is all about]?

Solution
This is just an example.  We tried to make this as simple as possible.  This assumes that Python has been installed on the Hadoop namenode (aka master node) running on Linux.

1.  Deploy Hadoop if it has not been installed.  See this link to deploy Hadoop to a single server.  If you want to set up a multi-node cluster of Hadoop, see these directions. These directions can work with any distribution of Linux, but we found the with CentOS, it was unreliable. (With CentOS we saw ResourceManager stopping inexplicably, excessive CPU consumption, or some other undefined problem that makes the MR jobs take too long. We do not know why because CentOS is a great distribution of Linux.)

2.  Log into the Hadoop namenode (aka master), and then run this Bash program (to get files ready and prepare two Python programs).

#!/bin/bash
mkdir /tmp/contint/

curl http://www.gutenberg.org/files/2701/2701.txt > /tmp/contint/2701.txt
curl http://www.gutenberg.org/files/2694/2694.txt > /tmp/contint/2694.txt
curl https://www.gutenberg.org/files/74/74-0.txt > /tmp/contint/74-0.txt

echo "#!/usr/bin/env python3
# This file is mapper.py
# This was taken from https://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/
# There were slight modifications to it.

import sys

# input comes from STDIN (standard input)
for line in sys.stdin:
        # remove leading and trailing whitespace
        line = line.strip()
        # split the line into words
        words = line.split()
        # increase counters
        for word in words:
                # write the results to STDOUT (standard output);
                # what we output here will be the input for the
                # Reduce step, i.e. the input for reducer.py
                #
                # tab-delimited; the trivial word count is 1
                print ('%s\t%s' % (word, 1))
" > /tmp/mapper.py

chmod +x /tmp/mapper.py

echo "#!/usr/bin/env python3
# This file is reducer.py
# This was mostly taken from https://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/
from operator import itemgetter
import sys

current_word = None
current_count = 0
word = None #this is a test

# input comes from STDIN
for line in sys.stdin:
        # remove leading and trailing whitespace
        line = line.strip()

        # parse the input we got from mapper.py
        word, count = line.split('\t', 1)

        # convert count (currently a string) to int
        try:
                count = int(count)
        except ValueError:
                # count was not a number, so silently
                # ignore/discard this line
                continue

                # this IF-switch only works because Hadoop sorts map output
                # by key (here: word) before it is passed to the reducer
        if current_word == word:
                current_count += count
        else:
                if current_word:
                        # write result to STDOUT
                        print('%s\t%s' % (current_word, current_count))
                        current_count = count
                        current_word = word

# do not forget to output the last word if needed!
if current_word == word:
        print('%s\t%s' % (current_word, current_count))
" > /tmp/reducer.py

chmod +x /tmp/reducer.py

echo "Script finished."
echo "***********************************************************************************"
echo ""
# Scripting the below is problematic on Ubuntu/Debian if the script is run via a "sudo" command.
# Plus the below is just an example. Some people may want to call the directory something else.
echo " Run these commands (or commands like these) to prepare a directory to run the hadoop commands:

su hduser
cd ~
mkdir placetowork
cp /tmp/reducer.py placetowork/
cp /tmp/mapper.py placetowork/
chown -R hduser:hadoop placetowork

# Then run these commands manually as the hduser in the placetowork subdirectory:
bash /usr/local/hadoop/sbin/start-dfs.sh
bash /usr/local/hadoop/sbin/start-yarn.sh
# run the above two commands again just in case. (sometimes it is necessary; you could use "jps" to verify if they need to be run once again)
bash /usr/local/hadoop/sbin/start-dfs.sh
bash /usr/local/hadoop/sbin/start-yarn.sh

hdfs dfs -mkdir -p /usr/hduser/contint
hdfs dfs -copyFromLocal /tmp/contint/* /usr/hduser/

# The command below maybe needs to use /home/hduser instead of the /user/hduser
hadoop jar /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-3.3.0.jar -file /tmp/mapper.py -mapper /tmp/mapper.py -file /tmp/reducer.py -reducer /tmp/reducer.py -input /usr/hduser/* -output /usr/hduser/contint-output -verbose
*************************************************************************************
"

3.  Find the hadoop-streaming*.jar file.  Run this command if you need to find it:

sudo find / -name hadoop-streaming-*.jar

4.  Optional step.  Run this command after your replace "/user/hduser" with the directory in your hdfs that you will write to:   hdfs dfs -ls /user/hduser

# This just tests you are able to use hdfs with the user you are logged in as.

5.  Run the command below after you replace "/usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-3.3.0.jar" with the path and file name of the hadoop-streaming*.jar file you have and after you replace "/user/hduser/" with the HDFS path your user has permissions to write to:

hadoop jar /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-3.3.0.jar -file /tmp/mapper.py -mapper /tmp/mapper.py -file /tmp/reducer.py -reducer /tmp/reducer.py -input /user/hduser/* -output /user/hduser/contint-output -verbose

6. Optional step. You can run this command to see the status of your job: 

mapred job -list all
# if you have an old version of hadoop, try this command: hadoop job -list all

FYI
The above directions were tested on Ubuntu 20.x, SUSE 15.x and CentOS 8.x Linux servers (as well as other types including different versions of RHEL). With CentOS we saw some unreliable problems. We do think CentOS is a great OS however.