How Do You Create an Ubuntu Linux VM in Azure Using PowerShell Azure?

Updated 1/12/19

Problem scenario

You want to create an Ubuntu Linux server with one vCPU and one GB of RAM in Azure using PowerShell.  You want to use Azure Resource Manager (an ARM template).  You do not want to use the Azure Portal for 95% of the work involved.  What do you do to automate the process (including the deployment of a resource group, a network security group, a subnet) so you can have a VM that is ready to use?

Solution
Warning: This will create a VM in your Azure account.  It will cost you money to do this.

Prerequisites
You have a way of running Azure PowerShell commands.  If you need assistance with this, see this posting if your workstation is running Windows 7 or this posting if your workstation is running Windows 10.  You can log into the Azure Portal at the end (to set the password of the administrator user that you create when you create the VM with this process).

Procedures
1.  Save this PowerShell script to your desktop (with its first line being "<#").

<# 
 .SYNOPSIS
    Deploys a template to Azure

 .DESCRIPTION
    Deploys an Azure Resource Manager template

 .PARAMETER subscriptionId
    The subscription id where the template will be deployed.

 .PARAMETER resourceGroupName
    The resource group where the template will be deployed. Can be the name of an existing or a new resource group.

 .PARAMETER resourceGroupLocation
    Optional, a resource group location. If specified, will try to create a new resource group in this location. If not specified, assumes resource group is existing.

 .PARAMETER deploymentName
    The deployment name.

 .PARAMETER templateFilePath
    Optional, path to the template file. Defaults to template.json.

 .PARAMETER parametersFilePath
    Optional, path to the parameters file. Defaults to parameters.json. If file is not found, will prompt for parameter values based on template.
#>

param(
 [Parameter(Mandatory=$True)]
 [string]
 $subscriptionId,

 [Parameter(Mandatory=$True)]
 [string]
 $resourceGroupName,

 [string]
 $resourceGroupLocation,

 [Parameter(Mandatory=$True)]
 [string]
 $deploymentName,

 [string]
 $templateFilePath = "template.json",

 [string]
 $parametersFilePath = "parameters.json"
)

<#
.SYNOPSIS
    Registers RPs
#>
Function RegisterRP {
    Param(
        [string]$ResourceProviderNamespace
    )

    Write-Host "Registering resource provider '$ResourceProviderNamespace'";
    Register-AzureRmResourceProvider -ProviderNamespace $ResourceProviderNamespace;
}

#******************************************************************************
# Script body
# Execution begins here
#******************************************************************************
$ErrorActionPreference = "Stop"

# sign in
Write-Host "Logging in...";
Login-AzureRmAccount;

# select subscription
Write-Host "Selecting subscription '$subscriptionId'";
Select-AzureRmSubscription -SubscriptionID $subscriptionId;

# Register RPs
$resourceProviders = @("microsoft.compute","microsoft.storage","microsoft.network");
if($resourceProviders.length) {
    Write-Host "Registering resource providers"
    foreach($resourceProvider in $resourceProviders) {
        RegisterRP($resourceProvider);
    }
}

#Create or check for existing resource group
$resourceGroup = Get-AzureRmResourceGroup -Name $resourceGroupName -ErrorAction SilentlyContinue
if(!$resourceGroup)
{
    Write-Host "Resource group '$resourceGroupName' does not exist. To create a new resource group, please enter a location.";
    if(!$resourceGroupLocation) {
        $resourceGroupLocation = Read-Host "resourceGroupLocation";
    }
    Write-Host "Creating resource group '$resourceGroupName' in location '$resourceGroupLocation'";
    New-AzureRmResourceGroup -Name $resourceGroupName -Location $resourceGroupLocation
}
else{
    Write-Host "Using existing resource group '$resourceGroupName'";
}

# Start the deployment
Write-Host "Starting deployment...";
if(Test-Path $parametersFilePath) {
    New-AzureRmResourceGroupDeployment -ResourceGroupName $resourceGroupName -TemplateFile $templateFilePath -TemplateParameterFile $parametersFilePath;
} else {
    New-AzureRmResourceGroupDeployment -ResourceGroupName $resourceGroupName -TemplateFile $templateFilePath;
}
# Last line of PowerShell script.

2.  Save these template.json and parameters.json files to the same folder on your workstation. 

parameters.json has the content starting below with "{" and ending with "}"

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "location": {
            "value": "eastus2"
        },
        "virtualMachineName": {
            "value": "nov16"
        },
        "virtualMachineSize": {
            "value": "Standard_B1s"
        },
        "adminUsername": {
            "value": "cooluser"
        },
        "virtualNetworkName": {
            "value": "changethis-vnet"
        },
        "networkInterfaceName": {
            "value": "nov1646"
        },
        "networkSecurityGroupName": {
            "value": "nov16-nsg"
        },
        "adminPassword": {
            "value": "c00lp@ssW0rd3
"        },
        "diagnosticsStorageAccountName": {
            "value": "changethisdiag604"
        },
        "diagnosticsStorageAccountType": {
            "value": "Standard_LRS"
        },
        "diagnosticsStorageAccountId": {
            "value": "Microsoft.Storage/storageAccounts/changethisdiag604"
        },
        "addressPrefix": {
            "value": "10.1.0.0/24"
        },
        "subnetName": {
            "value": "default"
        },
        "subnetPrefix": {
            "value": "10.1.0.0/24"
        },
        "publicIpAddressName": {
            "value": "nov16-ip"
        },
        "publicIpAddressType": {
            "value": "Dynamic"
        },
        "publicIpAddressSku": {
            "value": "Basic"
        }
    }
}

template.json starts with below "{" and ends with "}"

{
    "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "location": {
            "type": "string"
        },
        "virtualMachineName": {
            "type": "string"
        },
        "virtualMachineSize": {
            "type": "string"
        },
        "adminUsername": {
            "type": "string"
        },
        "virtualNetworkName": {
            "type": "string"
        },
        "networkInterfaceName": {
            "type": "string"
        },
        "networkSecurityGroupName": {
            "type": "string"
        },
        "adminPassword": {
            "type": "securestring"
        },
        "diagnosticsStorageAccountName": {
            "type": "string"
        },
        "diagnosticsStorageAccountId": {
            "type": "string"
        },
        "diagnosticsStorageAccountType": {
            "type": "string"
        },
        "addressPrefix": {
            "type": "string"
        },
        "subnetName": {
            "type": "string"
        },
        "subnetPrefix": {
            "type": "string"
        },
        "publicIpAddressName": {
            "type": "string"
        },
        "publicIpAddressType": {
            "type": "string"
        },
        "publicIpAddressSku": {
            "type": "string"
        }
    },
    "variables": {
        "vnetId": "[resourceId('changethis','Microsoft.Network/virtualNetworks', parameters('virtualNetworkName'))]",
        "subnetRef": "[concat(variables('vnetId'), '/subnets/', parameters('subnetName'))]"
    },
    "resources": [
        {
            "name": "[parameters('virtualMachineName')]",
            "type": "Microsoft.Compute/virtualMachines",
            "apiVersion": "2016-04-30-preview",
            "location": "[parameters('location')]",
            "dependsOn": [
                "[concat('Microsoft.Network/networkInterfaces/', parameters('networkInterfaceName'))]",
                "[concat('Microsoft.Storage/storageAccounts/', parameters('diagnosticsStorageAccountName'))]"
            ],
            "properties": {
                "osProfile": {
                    "computerName": "[parameters('virtualMachineName')]",
                    "adminUsername": "[parameters('adminUsername')]",
                    "adminPassword": "[parameters('adminPassword')]"
                },
                "hardwareProfile": {
                    "vmSize": "[parameters('virtualMachineSize')]"
                },
                "storageProfile": {
                    "imageReference": {
                        "publisher": "Canonical",
                        "offer": "UbuntuServer",
                        "sku": "16.04-LTS",
                        "version": "latest"
                    },
                    "osDisk": {
                        "createOption": "fromImage",
                        "managedDisk": {
                            "storageAccountType": "Standard_LRS"
                        }
                    },
                    "dataDisks": []
                },
                "networkProfile": {
                    "networkInterfaces": [
                        {
                            "id": "[resourceId('Microsoft.Network/networkInterfaces', parameters('networkInterfaceName'))]"
                        }
                    ]
                },
                "diagnosticsProfile": {
                    "bootDiagnostics": {
                        "enabled": true,
                        "storageUri": "[reference(resourceId('changethis', 'Microsoft.Storage/storageAccounts', parameters('diagnosticsStorageAccountName')), '2015-06-15').primaryEndpoints['blob']]"
                    }
                }
            }
        },
        {
            "name": "[parameters('diagnosticsStorageAccountName')]",
            "type": "Microsoft.Storage/storageAccounts",
            "apiVersion": "2015-06-15",
            "location": "[parameters('location')]",
            "properties": {
                "accountType": "[parameters('diagnosticsStorageAccountType')]"
            }
        },
        {
            "name": "[parameters('virtualNetworkName')]",
            "type": "Microsoft.Network/virtualNetworks",
            "apiVersion": "2017-08-01",
            "location": "[parameters('location')]",
            "properties": {
                "addressSpace": {
                    "addressPrefixes": [
                        "[parameters('addressPrefix')]"
                    ]
                },
                "subnets": [
                    {
                        "name": "[parameters('subnetName')]",
                        "properties": {
                            "addressPrefix": "[parameters('subnetPrefix')]"
                        }
                    }
                ]
            }
        },
        {
            "name": "[parameters('networkInterfaceName')]",
            "type": "Microsoft.Network/networkInterfaces",
            "apiVersion": "2016-09-01",
            "location": "[parameters('location')]",
            "dependsOn": [
                "[concat('Microsoft.Network/virtualNetworks/', parameters('virtualNetworkName'))]",
                "[concat('Microsoft.Network/publicIpAddresses/', parameters('publicIpAddressName'))]",
                "[concat('Microsoft.Network/networkSecurityGroups/', parameters('networkSecurityGroupName'))]"
            ],
            "properties": {
                "ipConfigurations": [
                    {
                        "name": "ipconfig1",
                        "properties": {
                            "subnet": {
                                "id": "[variables('subnetRef')]"
                            },
                            "privateIPAllocationMethod": "Dynamic",
                            "publicIpAddress": {
                                "id": "[resourceId('changethis','Microsoft.Network/publicIpAddresses', parameters('publicIpAddressName'))]"
                            }
                        }
                    }
                ],
                "networkSecurityGroup": {
                    "id": "[resourceId('changethis', 'Microsoft.Network/networkSecurityGroups', parameters('networkSecurityGroupName'))]"
                }
            }
        },
        {
            "name": "[parameters('publicIpAddressName')]",
            "type": "Microsoft.Network/publicIpAddresses",
            "apiVersion": "2017-08-01",
            "location": "[parameters('location')]",
            "properties": {
                "publicIpAllocationMethod": "[parameters('publicIpAddressType')]"
            },
            "sku": {
                "name": "[parameters('publicIpAddressSku')]"
            }
        },
        {
            "name": "[parameters('networkSecurityGroupName')]",
            "type": "Microsoft.Network/networkSecurityGroups",
            "apiVersion": "2017-06-01",
            "location": "[parameters('location')]",
            "properties": {
                "securityRules": [
                    {
                        "name": "default-allow-ssh",
                        "properties": {
                            "priority": 1000,
                            "protocol": "TCP",
                            "access": "Allow",
                            "direction": "Inbound",
                            "sourceAddressPrefix": "*",
                            "sourcePortRange": "*",
                            "destinationAddressPrefix": "*",
                            "destinationPortRange": "22"
                        }
                    }
                ]
            }
        }
    ],
    "outputs": {
        "adminUsername": {
            "type": "string",
            "value": "[parameters('adminUsername')]"
        }
    }
}

While these files (parameters.json and template.json) need to be in the same folder as each other, they need not be in the same folder as the PowerShell script (in step #1 above).  If you want the administrator account to have a name different from "azubuntu", change it in the parameters.json file.  You will set the password later, so you do not need to worry about that.  This solution will create the VM in Azure region "East US 2."  If you want to change this, replace "eastus2" with the region of your choice in the parameters.json file.***  The resource group that will subsume the VM can be in a different region from the VM's region.  The resource group's location will be configured in step #8.

3.  Open PowerShell ISE.

4.  Open the PowerShell script referred to in step #1.

5.  From the PS prompt (e.g., PS C:\Users\jdoe>), change to the directory path where you saved the template.json and parameters.json files (referred to in step #2).

6.  Have these things handy: credentials to your Azure Portal account and your Azure subscription ID.*

7.  Run the script by pressing the green arrow button that looks like a "Play" button.

8.  Respond to the interactive prompts.

    i.   Enter the Subscription ID.  
  ii.  The resource group name you enter must be "changethis" unless you went into the template.json and parameters.json and replaced "changethis" with a new string.**  If this is your first time running this PowerShell script and the name of the resource group (e.g., "changethis") is new, you will be prompted for a location in Azure to place this resource group. When you are prompted for "resourceGroupLocation" there are options in a different formats that will work.***  
  iii. For the deploymentname, you can make something up (e.g., acoolname).

9.  You may have to wait five minutes while it creates.  When the deployment is finished you will have a VM named "nov16."  You should now change the password of the administrator account of the VM.  Log into the Azure Portal. 

Open the CLI by clicking on the ">_" symbol in the upper right hand corner.  The lower part of the screen will become a dark blue.  Click "PowerShell (Windows)", and then click "Create storage".  Wait a few minutes for the initialization process to complete.  The upper left hand portion of the lower screen should say "PowerShell."

Prepare a draft of this command below by doing the replacing "azubuntu" with the username (if you changed the username in step #2), replacing "changethis" with the name of the resource group (unless you never changed it in step #8), replacing "coolPassword" with the password you want the user to have:

az vm user update --resource-group changethis --name nov16 --username azubuntu --password coolPassword

#Run this command in the PowerShell CLI prompt in the Azure Portal.

To quote Azure PowerShell "The supplied password must be between 6-72 characters long and must satisfy at least 3 of password complexity requirements from the following: 1) Contains an uppercase character 2) Contains a
lowercase character 3) Contains a numeric digit 4) Contains a special character 5) Control characters are not allowed."

10.  Now you should be able to log into the new machine "nov16" with the username and password there were used in step #9.

11.  If you want to re-use this procedure,  Modify the parameters.json file in two ways as described in i. and ii. below.

     i. Replace "nov16" in with the name you want to give to the VM (the hostname).
  ii. Change the "diag604" values to something else in the diagnosticsStorageAccountName's value and the diagnosticsStorageAccountId's value.  Save the file.  Here is an excerpt from the file as an example after the values have been changed:

 "diagnosticsStorageAccountName": {
            "value": "changethisdiag605a"
        },
        "diagnosticsStorageAccountType": {
            "value": "Standard_LRS"
        },
        "diagnosticsStorageAccountId": {
            "value": "Microsoft.Storage/storageAccounts/changethisdiag605a"

12.  You are now done.

* If you need assistance with determining your subscription ID, see this posting.

**  If you do not want to create a new resource group named "changethis", stop the PowerShell script and do these two things:

  1.  Go to the template.json file and replace all instances of "changethis" with the name of your choice.  
  2.  Go to the parameters.json file and replace all instances of "changethis" with the name of your choice.  

*** To find out potential regions (and the VM sizes that are correspondingly available), run these Azure PowerShell command:
# Use Login-AzureRmAccount if you have not logged in yet.  Then run these commands:
$list = Get-AzureRmLocation
echo $list.DisplayName

In the future we want to re-write these directions to use write-verbose instead of write-host. We want to warn people that write-verbose is recommended instead of write-host (according to this posting).

How Do You Create Servers Using .JSON Files and Resource Manager in Azure?

Problem scenario
You want to automate the creation of servers in Azure.  You want to try ARM templates to learn more about them.  How do you use .json files and Azure PowerShell to do this (without using the web UI)?

Solution
Prerequisites
i. You have a way of running Azure PowerShell commands.  If you need assistance with this, see this posting if your workstation is running Windows 7 or this posting if your workstation is running Windows 10. 
ii. You can log into the Azure Portal at the end (to set the password of the administrator user that you create when you create the VM with this process).

Procedures
This example will create a Windows server. 

1.  Create a file named CreateVMcontint.json with the contents below, but do two things first.  The first thing is to replace variables that start with "cont" or "my" as desired.  The second thing is to replace the user "contint" and its password "neatPassword5" with credentials you want to use.  Keep in mind that this .JSON file will have administrator credentials of the servers you create with it.

Here it is (CreateVMcontint.json):

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "adminUsername": { "type": "string" },
    "adminPassword": { "type": "securestring" },
    "contintuser": {     "type": "string",
        "defaultValue": "contintuser",
       "metadata": {
        "description": "Username for the Virtual Machine."           
       }
       },
     "neatPassword": {     "type": "securestring",
       "defaultValue": "neatPassword5",
       "metadata": {
        "description": "Password for contintuser."           
       }
       },
       "contint": { "type": "string",
          "defaultValue": "contint",
            "metadata": {
        "description": "Name of the Virtual Machine."           
           }
         }
     },
  "variables": {
    "vnetID": "[resourceId('Microsoft.Network/virtualNetworks','continualNET')]",
    "subnetRef": "[concat(variables('vnetID'),'/subnets/contIntSubnet')]",     
  },    
  "resources": [
    {
      "apiVersion": "2016-03-30",
      "type": "Microsoft.Network/publicIPAddresses",
      "name": "myPublicIPAddress",
      "location": "[resourceGroup().location]",
      "properties": {
        "publicIPAllocationMethod": "Dynamic",
        "dnsSettings": {
          "domainNameLabel": "contintresourcegroupdns1"
        }
      }
    },
    {
      "apiVersion": "2016-03-30",
      "type": "Microsoft.Network/virtualNetworks",
      "name": "continualNET",
      "location": "[resourceGroup().location]",
      "properties": {
        "addressSpace": { "addressPrefixes": [ "10.0.0.0/16" ] },
        "subnets": [
          {
            "name": "contIntSubnet",
            "properties": { "addressPrefix": "10.0.0.0/24" }
          }
        ]
      }
    },
    {
      "apiVersion": "2016-04-30-preview",
      "name": "contint",
      "type": "Microsoft.Compute/virtualMachines",
      "location": "[resourceGroup().location]",
      "dependsOn": [
        "[resourceId('Microsoft.Network/networkInterfaces/', 'contIntNic')]"
      ],
      "properties": {
        "hardwareProfile": { "vmSize": "Standard_DS1" },
        "osProfile": {
          "computerName": "[parameters('contint')]",
          "adminUsername": "[parameters('contintuser')]",
          "adminPassword": "[parameters('neatPassword')]"
        },
        "storageProfile": {
          "imageReference": {
            "publisher": "MicrosoftWindowsServer",
            "offer": "WindowsServer",
            "sku": "2012-R2-Datacenter",
            "version": "latest"
          },
          "osDisk": {
            "name": "myManagedOSDisk",
            "caching": "ReadWrite",
            "createOption": "FromImage"
          }
        },
        "networkProfile": {
          "networkInterfaces": [
            {
              "id": "[resourceId('Microsoft.Network/networkInterfaces','contIntNic')]"
            }
          ]
        }
      }
    },    
    {
      "apiVersion": "2016-03-30",
      "type": "Microsoft.Network/networkInterfaces",
      "name": "contIntNic",
      "location": "[resourceGroup().location]",
      "dependsOn": [
        "[resourceId('Microsoft.Network/publicIPAddresses/', 'myPublicIPAddress')]",
        "[resourceId('Microsoft.Network/virtualNetworks/', 'continualNET')]"
      ],
      "properties": {
        "ipConfigurations": [
          {
            "name": "ipconfig1",
            "properties": {
              "privateIPAllocationMethod": "Dynamic",
              "publicIPAddress": { "id": "[resourceId ('Microsoft.Network/publicIPAddresses','myPublicIPAddress')]" },
              "subnet": { "id": "[variables('subnetRef')]" }
            }
          }
        ]
      }
    }
  ]
}

2.  Create a second file named Parameters.json with the contents below, but replace the user "contint" and its password "neatPassword" with credentials you want to use.  Keep in mind that this .JSON file will have administrator credentials of the servers you create with it.

Here is the contents of Parameters.json:

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json",
  "contentVersion": "1.0.0.0",
  "parameters": {
  "adminUserName": { "value": "contintuser" },
    "adminPassword": { "value": "neatPassword" }
  }
}

3.  Run this PowerShell script, but remember two things: first it will require an interactive login to the Azure account.  Secondly you will want to change the paths to the two .json files from "C:\path\to\" to the location they reside in on your server (and you may want to change the ResourceGroupName from "contIntGroup" to something else as well as the the location from "East US" to "West US"):

Login-AzureRMAccount
$storageName = "st" + (Get-Random)
New-AzureRmStorageAccount -ResourceGroupName "contIntGroup" -AccountName $storageName -Location "East US" -SkuName "Standard_LRS" -Kind Storage
$accountKey = (Get-AzureRmStorageAccountKey -ResourceGroupName contIntGroup -Name  $storageName).Value[0]
$context = New-AzureStorageContext -StorageAccountName $storageName -StorageAccountKey  $accountKey
New-AzureStorageContainer -Name "templates" -Context $context -Permission Container

Set-AzureStorageBlobContent -File "C:\path\to\CreateVMcontint.json" -Context $context -Container "templates"
Set-AzureStorageBlobContent -File "C:\path\to\Parameters.json" -Context $context -Container templates

$templatePath = "https://" + $storageName + ".blob.core.windows.net/templates/CreateVMcontint.json"
$parametersPath = "https://" + $storageName + ".blob.core.windows.net/templates/Parameters.json"
New-AzureRmResourceGroupDeployment -Name "contintDeployment" -ResourceGroupName "contIntGroup" -TemplateUri $templatePath -TemplateParameterUri $parametersPath

How Do You Install UiPath for Free to Try out RPA?

Problem scenario
You want to install UiPath Studio CE to test out RPA on a Windows 2016 Server.   How do you get a free version of UiPath (e.g., Studio CE)?

Solution
Prerequisite #1
You will have to agree to this agreement on UiPath's website if you want to use the product.

Prerequisite #2
For this test, we recommend at a minimum 1 CPU at 2.3 Ghz and 2.25 GB of RAM.  This will not be a high performing machine, but it will minimally help you with a proof of concept.  It is strongly recommended to have 2 CPUs and more memory.

Procedures
1.  Open I.E.  If you are prompted, you may want to accept the recommended settings.

2.  Go to the gear icon in the upper righthand corner.  Go to "Internet Options."

3.  Go to Security tab and then click on "Local Intranet."

4.  Click on the "Sites" button.

5.  Enter this text with no quotes "http://download.uipath.com/" and click "Add".

6.a.  In I.E., browse to this URL: http://download.uipath.com/UiPathStudioSetup.exe

6.b.  To the option that comes up, choose "Run"

7.  You will eventually see "Welcome to UiPath Studio."  Click "Start Free".  

8.  Enter an email address in the prompt that asks for it.  Click "Activate".

9.  To the prompt about "Content from the website listed below is being blocked..." click Close.

11.  You are done.  The following steps are optional to harden your Windows Server.  Go to the gear icon in the upper righthand corner.  Go to "Internet Options."

12.  Go to Security tab and then click on "Local Intranet."

13.  Find the "http://download.uipath.com/" URL and remove it.

How Do You Resize a Server in Google Cloud Platform?

Problem scenario
You want to increase the amount of RAM that a VM instance has in GCP.  How do you do this?

Possible Solution #1
1.  Log into Google Cloud Platform
2.  In the "Cloud Engine" screen click on VM "instances."
3.  You should see a list of VM instances.  Under the "Recommendation" tab you will see "Increase perf."  Click on this. If you do not see "Increase perf", stop the machine. Then click on its hyperlinked name. Then click on "Edit". In the "Machine type" section click the drop down menu to resize the server to a different combination of CPU and memory.
4.  A pop-up will appear called "Resize instance."  Click "APPLY".
5.  Note that when the server is resized it may have a new external IP address.

Possible Solution #2
This solution was tested in January of 2019.
1. In the GCP console go to VM instances.
2. Check the box on the left for the server.
3. Click the "STOP" button near the top.
4. Click on the name of the VM instance which should be hyperlinked.
5. In the "VM instance details" page, click "EDIT".
6. In the "Machine type" section, click "Customize."
7. Select the number of cores and the amount of Memory you desire.
8. Scroll to the bottom and click "Save".

How Do You Get the Hadoop Yarn Web UI to Work?

Problem scenario
You want to get Hadoop's web UI to work.  You have access to the back-end of a Linux server.  How do you get the front-end of Hadoop (or hdfs) to work?

Solution
1.  Deploy Hadoop.  See this link for directions for any type of Linux. 
2.  Start the Hadoop services.  For a single-node deployment of Hadoop use sudo find / -name start-dfs.sh to find it; use bash /path/to/start-dfs.sh to run it.  For a multi-node deployment of Hadoop, log into the NameNode and use sudo find / -name start-all.sh to find it; use bash /path/to/start-all.sh to run it.
3.  If it is a single-node cluster, find the external IP address of the server with curl http://icanhazip.com.  If you have a multi-node Hadoop cluster, find the external IP address of the NameNode with curl http://icanhazip.com.
4.  Open a web browser from a workstation and go to http://<externalIPaddress>:8088

If it does not work, remember that the back-end of the server running yarn should show "ResourceManager" in the results of a command like this: sudo jps  # get ResourceManager to show first if there is a problem.  To install jps, see this link.

How Do You Get the HDFS Web UI to Work?

Problem scenario
You want to get Hadoop's web UI to work.  You have access to the back-end of a Linux server.  How do you get the front-end of Hadoop (or hdfs) to work?

Solution
1.  Deploy Hadoop.  See this article if you need help with how to do it. 
2.   From the NameNode server in a multi-node deployment or from the single Hadoop server, run the start-dfs.sh script (use sudo find / -name start-dfs.sh to find it; use bash /path/to/start-dfs.sh to run it).  
3.  If it is a single-node cluster, find the external IP address of the server with "curl icanhazip.com".  If you have a multi-node Hadoop cluster, find the external IP address of the NameNode with "curl icanhazip.com".
4.  Open a web browser from a workstation and go to http://<externalIPaddress>:50070
(The port could be different if you changed it from the default one.)

How Do You See the Storage Usage on a Datanode in an hdfs System?

Problem scenario
You want to see which datanodes are active underlying a given hdfs system.  You also want to know statistics about the storage usage.  If you are regularly adding data to your hdfs system, you want to stay below 70% utilization.  If you want your hdfs system to perform well but you are not regularly adding new files, you want to stay under 80%.  How do you find out about the storage usage of your datanodes that support your hdfs system?

Solution
Run this command:

hdfs dfsadmin -report
# or this command below
sudo /usr/local/hadoop/bin/hdfs dfsadmin -report

How Do You Troubleshoot the Hadoop Error “Could not find or load main class jar”?

Problem scenario
You are trying to run a Hadoop operation.  You issue your "hadoop jar" or "hdfs jar" command.  You get one of these errors:

"Error: Could not find or load main class jar"

or this error

"Error: Could not find or load main class hadoop-streaming-2.8.1.jar"

What is wrong?

Solution
1.  Use "hadoop..." instead of "hdfs".

2.  Make sure you have the hadoop-streaming*.jar file.  You could run this command to find it:

sudo find / -name hadoop-streaming-*.jar

If you cannot find it, you can download it from here.

3.  Remember to put the full path after the "hadoop jar ".  Here is an example:

hadoop jar /usr/local/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.8.1.jar -file /home/hduser/mapper.py -mapper /home/hduser/mapper.py -file  /home/hduser/reducer.py -reducer /home/hduser/reducer.py -input /user/hduser/contint/* -output /user/hduser/gooddir-output

Alternatively you could use "cd /to/the/path/with/hadoop-streaming-2.8.1.jar" (where 2.8.1 is just an example).

What is MapReduce in Hadoop?

Problem scenario
You want to learn more about MapReduce.  You want to learn what it does so you can grasp Hadoop more thoroughly.  What is MapReduce?

Solution
MapReduce is a core process of Hadoop.  With multi-node deployments of Hadoop, MapReduce distributes data to different datanodes (servers that are controlled by a master server).  This data is retrievable thanks to MapReduce.  Conceptually there are two main components to MapReduce: mapper and reducer.  The mapper function creates temporary key-value pairs.   The reducer is composed of three phases: shuffle, sort and reduce.  Shuffle and sort happen simultaneously.  Together these components leverage commodity hardware in multi-node deployments.  If you want to learn more you can see this link.

After you run a MapReduce job, you'll see output that is a summary of the job.  The summary will include various quantifiable aspects of the job in the "Map-Reduce Framework" section of the  text output.

It is not hard to set up Hadoop and run a MapReduce job.  You can learn more about the process if you do this.  To deploy Hadoop on any type of Linux, see this link.  You can follow this link to run a MapReduce job for learning.

How Do You Cancel an hdfs Command That Appears to Be Making No Progress?

Problem scenario
An hdfs command hangs (or appears to be hung).  To interrupt the command, you try ctrl-c.  But this does nothing.  How do you cancel out?

Solution
Use ctrl-z.  This will suspend the job.  You can then kill it with kill, kill 9, or some other means.