How Do You Create Servers Using .JSON Files and Resource Manager in Azure?

0 Comments

Problem scenario
You want to automate the creation of servers in Azure.  You want to try ARM templates to learn more about them.  How do you use .json files and Azure PowerShell to do this (without using the web UI)?

Solution
Prerequisites
i. You have a way of running Azure PowerShell commands.  If you need assistance with this, see this posting if your workstation is running Windows 7 or this posting if your workstation is running Windows 10. 

…

Continue reading “How Do You Create Servers Using .JSON Files and Resource Manager in Azure?”

How Do You Resize a Server in Google Cloud Platform?

0 Comments

Problem scenario
You want to increase the amount of RAM that a VM instance has in GCP.  How do you do this?

Possible Solution #1
1.  Log into Google Cloud Platform
2.  In the “Cloud Engine” screen click on VM “instances.”
3.  You should see a list of VM instances.  Under the “Recommendation” tab you will see “Increase perf.”  Click on this.

…

Continue reading “How Do You Resize a Server in Google Cloud Platform?”

How Do You Get the Hadoop Yarn Web UI to Work?

0 Comments

Problem scenario
You want to get Hadoop’s web UI to work.  You have access to the back-end of a Linux server.  How do you get the front-end of Hadoop (or hdfs) to work?

Solution
1.  Deploy Hadoop.  See this link for directions for any type of Linux. 
2.  Start the Hadoop services.  For a single-node deployment of Hadoop use sudo find / -name start-dfs.sh to find it;

…

Continue reading “How Do You Get the Hadoop Yarn Web UI to Work?”

How Do You See the Storage Usage on a Datanode in an hdfs System?

0 Comments

Problem scenario
You want to see which datanodes are active underlying a given hdfs system.  You also want to know statistics about the storage usage.  If you are regularly adding data to your hdfs system, you want to stay below 70% utilization.  If you want your hdfs system to perform well but you are not regularly adding new files, you want to stay under 80%.  How do you find out about the storage usage of your datanodes that support your hdfs system?

…

Continue reading “How Do You See the Storage Usage on a Datanode in an hdfs System?”

How Do You Troubleshoot the Hadoop Error “Could not find or load main class jar”?

0 Comments

Problem scenario
You are trying to run a Hadoop operation.  You issue your “hadoop jar” or “hdfs jar” command.  You get one of these errors:

“Error: Could not find or load main class jar”

or this error

“Error: Could not find or load main class hadoop-streaming-2.8.1.jar”

What is wrong?

Solution
1.  Use “hadoop…” instead of “hdfs”.

2. 

…

Continue reading “How Do You Troubleshoot the Hadoop Error “Could not find or load main class jar”?”

What is MapReduce in Hadoop?

0 Comments

Problem scenario
You want to learn more about MapReduce.  You want to learn what it does so you can grasp Hadoop more thoroughly.  What is MapReduce?

Solution
MapReduce is a core process of Hadoop.  With multi-node deployments of Hadoop, MapReduce distributes data to different datanodes (servers that are controlled by a master server).  This data is retrievable thanks to MapReduce.  Conceptually there are two main components to MapReduce: mapper and reducer. 

…

Continue reading “What is MapReduce in Hadoop?”

How Do You Solve the Problem of Being Prompted for Credentials after You Run the start-dfs.sh Script?

0 Comments

Problem scenario
You try to start Hadoop’s dfs in a multi-node deployment.  All the Hadoop nodes are running Linux.  You run this:

bash /usr/local/hadoop/sbin/start-dfs.sh

You see this:

Starting namenodes on [hadoopmaster]
hadoopmaster: starting namenode, logging to /usr/local/hadoop/logs/hadoop-root-namenode-hadoopmaster.out
root@hadoopdatanode’s password: localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-hadoopmaster.out
hadoopmaster: starting datanode, logging to /usr/local/hadoop/logs/hadoop-root-datanode-hadoopmaster.out
localhost: ulimit -a for user root
localhost: core file size         

…

Continue reading “How Do You Solve the Problem of Being Prompted for Credentials after You Run the start-dfs.sh Script?”