How Do You Write a Python Program That Can Run Linux Bash Commands?

Problem scenario
You want to use Python to run Linux commands.  You have been told to not use the "import os" for this task.  You want to manipulate the text and output of Bash commands for sophisticated processing and automation with Python.  How can a Python program run Bash commands?

Solution
Use "from subprocess import check_output" as the first line.  Then encapsulate the Bash command inside double quotes, square brackets and parentheses like these two lines of Python code:

out = check_output(["date"])
sn = check_output(["hostname"])

Later in your Python program, you will be able to use "out" and "sn" as the date and servername respectively.  Here is an example (test.py) of how to use the "date" and "hostname" Bash commands inside a Python script:

from subprocess import check_output
out = check_output(["date"])
sn = check_output(["hostname"])
for i in sn:
   print(i)
print(sn)
print(out)

If you are not constructing the Bash commands from an untrusted source, or you have a sanitation system for such input, you can use "shell=True".  This will enable you to invoke a wider range of Bash commands (including those with pipes "|") with greater complexity. Here is a Python 2 example of how to use "shell=True":

from subprocess import check_output
out = check_output(["cat /etc/*-release | grep NAME | grep PRETTY"], shell=True)
sn = check_output(["hostname"])
print out

The above in Python 3 is here:

from subprocess import check_output
out = check_output(["cat /etc/*-release | grep NAME | grep PRETTY"], shell=True).strip()
out = str(out)
out = out[1::]
sn = check_output(["hostname"])
print(out)

How Do You Troubleshoot the Problem “ImportError: Entry Point (‘console_scripts’, ‘Parquet’) Not Found”?

Problem scenario
You are trying to run Apache Parquet commands.  But each command gives this error:

Traceback (most recent call last):
  File "/usr/local/bin/parquet", line 11, in <module>
    load_entry_point('parquet==1.2', 'console_scripts', 'parquet')()
  File "/home/ubuntu/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 570, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2750, in load_entry_point
    raise ImportError("Entry point %r not found" % ((group, name),))
ImportError: Entry point ('console_scripts', 'parquet') not found

How do you fix this problem?

Solution
1.  Remove pip  (e.g., with Ubuntu run "sudo apt-get remove python-pip")

2.  Reinstall pip (e.g., with Ubuntu run "sudo apt-get install python-pip")

3.  Upgrade pip (e.g., with Ubuntu run "sudo easy_install --upgrade pip")

How Do You Learn More about Quality Assurance Automation Tools?

Problem scenario
You want to learn more about QA automation tools. What do you do?

Solution
The DevOps Zone web page has an article entitled "Top 10 Testing Automation Tools for Software Testing."  For people who are unfamiliar with QA-related technologies, they may not recognize the names of the companies behind the tools except HP.  Many companies rely on HP's QA tools despite the fact that HP may sell its software development tools business unit to a different company.  To learn more about QA tools themselves you may be interested in purchasing a book or two.  You may also want to go Selenium's website.  Another good article is this TechTarget one.

How Do You Deploy Apache Mesos and Apache Marathon to an Ubuntu Linux Server in AWS?

Problem scenario
You want to install Apache Mesos and Apache Marathon to an Ubuntu 16.x Linux server in AWS.  How do you do this?

Prerequisites
You need two Linux instances with relevant Security Group rules added to allow for connectivity between the two.  One Ubuntu Linux server will be for the Mesos master and the other server will be for the Mesos slave.  You also need to be able to use a web browser to the Mesos master server.  You can use Apache Mesos and Marathon with other distributions of Linux.  This solution happens to use Ubuntu.

Solution
Steps 1 through 8 will all be done on the server that will become the Apache Mesos master server.
1.  On the server that will be the Apache Mesos master server, run these commands:

sudo apt-get -y update
sudo apt-get -y install zookeeperd

2.  Run this command:
sudo service zookeeper status
# Enter these characters without the quote marks ":q" to escape

3.  Run this command:
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv DF7D54CBE56151BF

4.a.  Create this file /etc/apt/sources.list.d/mesosphere.list
4.b.  This file above should have this line:
deb http://repos.mesosphere.com/ubuntu xenial main

5.  Run these commands:
sudo apt-get -y update
sudo apt-get -y install mesos

6.  Create this file /etc/mesos-master/advertise_ip
It should have the external IP address of the server:
x.x.x.x

To find what x.x.x.x should be if you do not know it, run "curl http://icanhazip.com" with no quotes from the master.

7.  Create this file /etc/mesos-slave/advertise_ip
It should have the external IP address of the server:
y.y.y.y

To find what y.y.y.y should be if you do not know it, run "curl http://icanhazip.com" with not quotes from the slave.

8.  Run this command:
sudo service mesos-master start

Steps 9 through 16 should be done on the server that will be the Apache Mesos slave server.

9.  Go to the server that will be the Mesos slave.  Run these commands:

sudo apt-get -y update
sudo apt-get -y install zookeeperd

10.  Now run this command:
sudo service zookeeper status
# Enter these characters without the quote marks ":q" to escape

11.  Run this command:
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv DF7D54CBE56151BF

12.a.  Create this file /etc/apt/sources.list.d/mesosphere.list
12.b.  The file above should have this line:
deb http://repos.mesosphere.com/ubuntu xenial main

13.  Now run these commands:
sudo apt-get -y update
sudo apt-get -y install mesos

14.  Create this file /etc/mesos-master/advertise_ip
It should have the external IP address of the server:
x.x.x.x

To find what x.x.x.x should be if you do not know it, run "curl http://icanhazip.com" with not quotes from the master.

15.  Create this file /etc/mesos-slave/advertise_ip
It should have the external IP address of the server:
y.y.y.y

To find what y.y.y.y should be if you do not know it, run "curl http://icanhazip.com" with not quotes from the slave.

16.  Now run this command:
sudo service mesos-slave start

17.  Go back to the Mesos master server.  Run these commands:
sudo apt-get -y install marathon
sudo service marathon start

You are done.

18.  Optional step:  Now you can open a web browser knowing what the external IP address of the Mesos master server is.  To see the Mesos web UI, go here (where x.x.x.x is the external IP address of the Mesos master): x.x.x.x:5050
To see the Marathon web UI, go here (where x.x.x.x is the external IP address of the Mesos master): x.x.x.x:8080

How Do You Use PowerShell on Your Desktop to Manage Azure?

Problem scenario
You want to use PowerShell on your local machine to create, delete, and restart servers in Azure.  You want to do other things with Azure using PowerShell on your workstation.  What do you do?

Solution
1.  To run Azure commands from your desktop, you need to connect your PowerShell ISE with your Azure instance.  If you are using Windows 10, see this posting and ignore this set of directions here entitled "How Do You Use PowerShell on Your Desktop to Manage Azure?"  If you are using Windows 7, proceed to step #2 and remember that you will need to reboot your workstation.

2.  Install the Azure tools; you can download them here.

2.  When the installer above is open click "Add" next to "Microsoft Azure PowerShell."

3.  Click "Install."

4.  You can read the license terms.  It appears that the licenses are the Apache License and MIT License.  If you agree click "I accept."

5.  Reboot your computer.

6.  Open PowerShell.  Run this command: Get-AzurePublishSettingsFile

# Be prepared for a web browser to open up with the Azure Portal website.

7.  Log into the Azure Portal via the web browser as you normally would.

8.  You'll be prompted to open or save a file.  Save it wherever you like on your desktop.

9.  Draft a command like this:

Import-AzurePublishSettingsFile -PublishSettingsFile "C:\Users\jdoe\downloads\mysettings.publishsettings"

# where C:\users\jdoe\downloads\ is the path to the file you just downloaded.

Run this command once you've changed the location.

10.  Optional step. Run this command to verify it worked:

Get-ChildItem -Recurse cert:\ | Where-Object {$_.Issuer -like '*Azure*'} | select FriendlyName, Subject

# Above command was taken from this article.

11.  Now you can run Azure commands.  Optionally you can try this one: Get-AzureLocation

How Do You Troubleshoot a Problem with Adding a DataNode to a Hadoop Cluster?

Problem scenario
You are trying to add a DataNode to an existing Hadoop cluster.  There are numerous problems.  What do you do to troubleshoot the process?

Possible solutions
1.  New versions of Hadoop use a "workers" file -- not a "slaves" file.

2.  Do you have a DNS solution in place for your DataNode and NameNode to resolve each other?  If you do not have a DNS server, does the /etc/hosts file of the DataNode server have an entry for the NameNode?  Can the NameNode resolve the domain name of the DataNode servers individually via the NameNode itself?

3.  Can the "hduser" (or whichever user starts the Hadoop cluster) passwordlessly SSH into the DataNode from the NameNode?  This is generally necessary for a multi-node cluster.  If you need directions for setting up passwordless SSH, see this article.

4.  Warning: This will delete all the data from your cluster. If you are having a problem adding a DataNode to a cluster, you may want to try deleting all directories and files in /app/hadoop/tmp/ on the individual DataNodes that are not being added when you run start-dfs.sh on the NameNode.  After you delete the data in /app/hadoop/tmp/ on the DataNodes, you may want to run hdfs namenode -format on the NameNode.  This will delete all the data in your Hadoop cluster; however it can help with troubleshooting.  You can then run the start-dfs.sh script to see if the DataNodes will be added to the cluster.

5.  When you run the jps command on the NameNode and DataNode, do you see a Hadoop node component running?  You may need to shut down all node services first and restart them.  To install jps, see this article.

6.  Check the /usr/local/hadoop/etc/hadoop/core-site.xml files on the DataNodes.  Do they have "localhost" or do they have the hostname of the NameNode?  They need the hostname of the NameNode.  If you forgot to modify these files, the respective DataNode will not join the cluster as normal.  The "jps" command will still show the "DataNode" service starting and stopping as you control it via the NameNode.  But the DataNode will not make its storage capacity available.

7.  If the DataNode service starts and stops with the NameNode's run of start-dfs.sh and stop-dfs.sh but you are not seeing the DataNode in the cluster with an "hdfs dfsadmin -report" command, the problem could be that there is a firewall rule protecting the NameNode that blocks port 54310.

You may want to use nmap on the DataNode to determine if the port defined in /usr/local/hadoop/etc/hadoop/core-site.xml (the path may be different) is not blocked.  If this port is blocked you can still start and stop the DataNode service from the NameNode, but there will be a problem with the DataNode actually working in the cluster. 

8.  Are you using the start-all.sh script on the NameNode?  Some Hadoop administrators use combinations of start-dfs.sh on both the NameNode and the DataNode.  It may be easier if you use the start-all.sh script from the NameNode.  The start-all.sh script is not advisable in production environments.  This is just for troubleshooting.

9. Search for a dfs.hosts.exclude file. Is the server that will not join the cluster in there?

10. Do you have a dfs.include or dfs.hosts file? Could you try adding the server to one of these files and restarting the services on the name node?

11.  Warning: This will delete all the data from your cluster. You may want to read these step-by-step directions to start over with your deployment (i.e., you may want to reinstall and reconfigure from the beginning).

12. Optional reading on Apache's website.

How Do You Create a Chef Automate Server in Azure?

Problem scenario
You want to have a Chef Automate server in Azure.  How do you create one?

Solution
1.  In the Azure portal click the "New" button on the left.
2.  Search for "Chef Automate" with no quotes.
3.  Click "Create".
4.  Fill out the required configuration settings.  Make a mental note of the Chef Automate FQDN DNS Label that you provide (e.g., contint.eastus.cloudapp.azure.com).  For the selection of a Chef Automate license file, you can use none and be provisioned with a 30 day trial license.
5.  If you can agree to the Terms of Use and various conditions, click "Purchase."
6.  While the process is completing, install on your Windows, Mac, or Linux workstation the Chef Development Kit.  To learn how, use this link, but you do not need to do the section on "Setting up the Chef repo."
7.  When the "Chef Automate" server has been provisioned, compose a URL like this (but do not use the quotes) but replace "contint.eastus.cloudapp.azure.com" with the FQDN you chose in the process above:

"https://" + "contint.eastus.cloudapp.azure.com" + "/biscotti/setup"

8.  Go to the URL (the one you composed in the previous step) via a web browser.  You may want to confirm the security exception to proceed.
9.  In the web UI, enter the info as it is required.  If you can agree to the EULA and Master Agreement, click "Setup Chef Automate & Download Starter Kit."
10.  Save the .zip file when prompted.
11.  Click the web UI button for "Login to Chef Automate".  Your are now done.

How Do You Create an Ubuntu Linux VM in Azure Using PowerShell Azure?

Updated 1/12/19

Problem scenario

You want to create an Ubuntu Linux server with one vCPU and one GB of RAM in Azure using PowerShell.  You want to use Azure Resource Manager (an ARM template).  You do not want to use the Azure Portal for 95% of the work involved.  What do you do to automate the process (including the deployment of a resource group, a network security group, a subnet) so you can have a VM that is ready to use?

Solution
Warning: This will create a VM in your Azure account.  It will cost you money to do this.

Prerequisites
You have a way of running Azure PowerShell commands.  If you need assistance with this, see this posting if your workstation is running Windows 7 or this posting if your workstation is running Windows 10.  You can log into the Azure Portal at the end (to set the password of the administrator user that you create when you create the VM with this process).

Procedures
1.  Save this PowerShell script to your desktop (with its first line being "<#").

<# 
 .SYNOPSIS
    Deploys a template to Azure

 .DESCRIPTION
    Deploys an Azure Resource Manager template

 .PARAMETER subscriptionId
    The subscription id where the template will be deployed.

 .PARAMETER resourceGroupName
    The resource group where the template will be deployed. Can be the name of an existing or a new resource group.

 .PARAMETER resourceGroupLocation
    Optional, a resource group location. If specified, will try to create a new resource group in this location. If not specified, assumes resource group is existing.

 .PARAMETER deploymentName
    The deployment name.

 .PARAMETER templateFilePath
    Optional, path to the template file. Defaults to template.json.

 .PARAMETER parametersFilePath
    Optional, path to the parameters file. Defaults to parameters.json. If file is not found, will prompt for parameter values based on template.
#>

param(
 [Parameter(Mandatory=$True)]
 [string]
 $subscriptionId,

 [Parameter(Mandatory=$True)]
 [string]
 $resourceGroupName,

 [string]
 $resourceGroupLocation,

 [Parameter(Mandatory=$True)]
 [string]
 $deploymentName,

 [string]
 $templateFilePath = "template.json",

 [string]
 $parametersFilePath = "parameters.json"
)

<#
.SYNOPSIS
    Registers RPs
#>
Function RegisterRP {
    Param(
        [string]$ResourceProviderNamespace
    )

    Write-Host "Registering resource provider '$ResourceProviderNamespace'";
    Register-AzureRmResourceProvider -ProviderNamespace $ResourceProviderNamespace;
}

#******************************************************************************
# Script body
# Execution begins here
#******************************************************************************
$ErrorActionPreference = "Stop"

# sign in
Write-Host "Logging in...";
Login-AzureRmAccount;

# select subscription
Write-Host "Selecting subscription '$subscriptionId'";
Select-AzureRmSubscription -SubscriptionID $subscriptionId;

# Register RPs
$resourceProviders = @("microsoft.compute","microsoft.storage","microsoft.network");
if($resourceProviders.length) {
    Write-Host "Registering resource providers"
    foreach($resourceProvider in $resourceProviders) {
        RegisterRP($resourceProvider);
    }
}

#Create or check for existing resource group
$resourceGroup = Get-AzureRmResourceGroup -Name $resourceGroupName -ErrorAction SilentlyContinue
if(!$resourceGroup)
{
    Write-Host "Resource group '$resourceGroupName' does not exist. To create a new resource group, please enter a location.";
    if(!$resourceGroupLocation) {
        $resourceGroupLocation = Read-Host "resourceGroupLocation";
    }
    Write-Host "Creating resource group '$resourceGroupName' in location '$resourceGroupLocation'";
    New-AzureRmResourceGroup -Name $resourceGroupName -Location $resourceGroupLocation
}
else{
    Write-Host "Using existing resource group '$resourceGroupName'";
}

# Start the deployment
Write-Host "Starting deployment...";
if(Test-Path $parametersFilePath) {
    New-AzureRmResourceGroupDeployment -ResourceGroupName $resourceGroupName -TemplateFile $templateFilePath -TemplateParameterFile $parametersFilePath;
} else {
    New-AzureRmResourceGroupDeployment -ResourceGroupName $resourceGroupName -TemplateFile $templateFilePath;
}
# Last line of PowerShell script.

2.  Save these template.json and parameters.json files to the same folder on your workstation. 

parameters.json has the content starting below with "{" and ending with "}"

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "location": {
            "value": "eastus2"
        },
        "virtualMachineName": {
            "value": "nov16"
        },
        "virtualMachineSize": {
            "value": "Standard_B1s"
        },
        "adminUsername": {
            "value": "cooluser"
        },
        "virtualNetworkName": {
            "value": "changethis-vnet"
        },
        "networkInterfaceName": {
            "value": "nov1646"
        },
        "networkSecurityGroupName": {
            "value": "nov16-nsg"
        },
        "adminPassword": {
            "value": "c00lp@ssW0rd3
"        },
        "diagnosticsStorageAccountName": {
            "value": "changethisdiag604"
        },
        "diagnosticsStorageAccountType": {
            "value": "Standard_LRS"
        },
        "diagnosticsStorageAccountId": {
            "value": "Microsoft.Storage/storageAccounts/changethisdiag604"
        },
        "addressPrefix": {
            "value": "10.1.0.0/24"
        },
        "subnetName": {
            "value": "default"
        },
        "subnetPrefix": {
            "value": "10.1.0.0/24"
        },
        "publicIpAddressName": {
            "value": "nov16-ip"
        },
        "publicIpAddressType": {
            "value": "Dynamic"
        },
        "publicIpAddressSku": {
            "value": "Basic"
        }
    }
}

template.json starts with below "{" and ends with "}"

{
    "$schema": "http://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "location": {
            "type": "string"
        },
        "virtualMachineName": {
            "type": "string"
        },
        "virtualMachineSize": {
            "type": "string"
        },
        "adminUsername": {
            "type": "string"
        },
        "virtualNetworkName": {
            "type": "string"
        },
        "networkInterfaceName": {
            "type": "string"
        },
        "networkSecurityGroupName": {
            "type": "string"
        },
        "adminPassword": {
            "type": "securestring"
        },
        "diagnosticsStorageAccountName": {
            "type": "string"
        },
        "diagnosticsStorageAccountId": {
            "type": "string"
        },
        "diagnosticsStorageAccountType": {
            "type": "string"
        },
        "addressPrefix": {
            "type": "string"
        },
        "subnetName": {
            "type": "string"
        },
        "subnetPrefix": {
            "type": "string"
        },
        "publicIpAddressName": {
            "type": "string"
        },
        "publicIpAddressType": {
            "type": "string"
        },
        "publicIpAddressSku": {
            "type": "string"
        }
    },
    "variables": {
        "vnetId": "[resourceId('changethis','Microsoft.Network/virtualNetworks', parameters('virtualNetworkName'))]",
        "subnetRef": "[concat(variables('vnetId'), '/subnets/', parameters('subnetName'))]"
    },
    "resources": [
        {
            "name": "[parameters('virtualMachineName')]",
            "type": "Microsoft.Compute/virtualMachines",
            "apiVersion": "2016-04-30-preview",
            "location": "[parameters('location')]",
            "dependsOn": [
                "[concat('Microsoft.Network/networkInterfaces/', parameters('networkInterfaceName'))]",
                "[concat('Microsoft.Storage/storageAccounts/', parameters('diagnosticsStorageAccountName'))]"
            ],
            "properties": {
                "osProfile": {
                    "computerName": "[parameters('virtualMachineName')]",
                    "adminUsername": "[parameters('adminUsername')]",
                    "adminPassword": "[parameters('adminPassword')]"
                },
                "hardwareProfile": {
                    "vmSize": "[parameters('virtualMachineSize')]"
                },
                "storageProfile": {
                    "imageReference": {
                        "publisher": "Canonical",
                        "offer": "UbuntuServer",
                        "sku": "16.04-LTS",
                        "version": "latest"
                    },
                    "osDisk": {
                        "createOption": "fromImage",
                        "managedDisk": {
                            "storageAccountType": "Standard_LRS"
                        }
                    },
                    "dataDisks": []
                },
                "networkProfile": {
                    "networkInterfaces": [
                        {
                            "id": "[resourceId('Microsoft.Network/networkInterfaces', parameters('networkInterfaceName'))]"
                        }
                    ]
                },
                "diagnosticsProfile": {
                    "bootDiagnostics": {
                        "enabled": true,
                        "storageUri": "[reference(resourceId('changethis', 'Microsoft.Storage/storageAccounts', parameters('diagnosticsStorageAccountName')), '2015-06-15').primaryEndpoints['blob']]"
                    }
                }
            }
        },
        {
            "name": "[parameters('diagnosticsStorageAccountName')]",
            "type": "Microsoft.Storage/storageAccounts",
            "apiVersion": "2015-06-15",
            "location": "[parameters('location')]",
            "properties": {
                "accountType": "[parameters('diagnosticsStorageAccountType')]"
            }
        },
        {
            "name": "[parameters('virtualNetworkName')]",
            "type": "Microsoft.Network/virtualNetworks",
            "apiVersion": "2017-08-01",
            "location": "[parameters('location')]",
            "properties": {
                "addressSpace": {
                    "addressPrefixes": [
                        "[parameters('addressPrefix')]"
                    ]
                },
                "subnets": [
                    {
                        "name": "[parameters('subnetName')]",
                        "properties": {
                            "addressPrefix": "[parameters('subnetPrefix')]"
                        }
                    }
                ]
            }
        },
        {
            "name": "[parameters('networkInterfaceName')]",
            "type": "Microsoft.Network/networkInterfaces",
            "apiVersion": "2016-09-01",
            "location": "[parameters('location')]",
            "dependsOn": [
                "[concat('Microsoft.Network/virtualNetworks/', parameters('virtualNetworkName'))]",
                "[concat('Microsoft.Network/publicIpAddresses/', parameters('publicIpAddressName'))]",
                "[concat('Microsoft.Network/networkSecurityGroups/', parameters('networkSecurityGroupName'))]"
            ],
            "properties": {
                "ipConfigurations": [
                    {
                        "name": "ipconfig1",
                        "properties": {
                            "subnet": {
                                "id": "[variables('subnetRef')]"
                            },
                            "privateIPAllocationMethod": "Dynamic",
                            "publicIpAddress": {
                                "id": "[resourceId('changethis','Microsoft.Network/publicIpAddresses', parameters('publicIpAddressName'))]"
                            }
                        }
                    }
                ],
                "networkSecurityGroup": {
                    "id": "[resourceId('changethis', 'Microsoft.Network/networkSecurityGroups', parameters('networkSecurityGroupName'))]"
                }
            }
        },
        {
            "name": "[parameters('publicIpAddressName')]",
            "type": "Microsoft.Network/publicIpAddresses",
            "apiVersion": "2017-08-01",
            "location": "[parameters('location')]",
            "properties": {
                "publicIpAllocationMethod": "[parameters('publicIpAddressType')]"
            },
            "sku": {
                "name": "[parameters('publicIpAddressSku')]"
            }
        },
        {
            "name": "[parameters('networkSecurityGroupName')]",
            "type": "Microsoft.Network/networkSecurityGroups",
            "apiVersion": "2017-06-01",
            "location": "[parameters('location')]",
            "properties": {
                "securityRules": [
                    {
                        "name": "default-allow-ssh",
                        "properties": {
                            "priority": 1000,
                            "protocol": "TCP",
                            "access": "Allow",
                            "direction": "Inbound",
                            "sourceAddressPrefix": "*",
                            "sourcePortRange": "*",
                            "destinationAddressPrefix": "*",
                            "destinationPortRange": "22"
                        }
                    }
                ]
            }
        }
    ],
    "outputs": {
        "adminUsername": {
            "type": "string",
            "value": "[parameters('adminUsername')]"
        }
    }
}

While these files (parameters.json and template.json) need to be in the same folder as each other, they need not be in the same folder as the PowerShell script (in step #1 above).  If you want the administrator account to have a name different from "azubuntu", change it in the parameters.json file.  You will set the password later, so you do not need to worry about that.  This solution will create the VM in Azure region "East US 2."  If you want to change this, replace "eastus2" with the region of your choice in the parameters.json file.***  The resource group that will subsume the VM can be in a different region from the VM's region.  The resource group's location will be configured in step #8.

3.  Open PowerShell ISE.

4.  Open the PowerShell script referred to in step #1.

5.  From the PS prompt (e.g., PS C:\Users\jdoe>), change to the directory path where you saved the template.json and parameters.json files (referred to in step #2).

6.  Have these things handy: credentials to your Azure Portal account and your Azure subscription ID.*

7.  Run the script by pressing the green arrow button that looks like a "Play" button.

8.  Respond to the interactive prompts.

    i.   Enter the Subscription ID.  
  ii.  The resource group name you enter must be "changethis" unless you went into the template.json and parameters.json and replaced "changethis" with a new string.**  If this is your first time running this PowerShell script and the name of the resource group (e.g., "changethis") is new, you will be prompted for a location in Azure to place this resource group. When you are prompted for "resourceGroupLocation" there are options in a different formats that will work.***  
  iii. For the deploymentname, you can make something up (e.g., acoolname).

9.  You may have to wait five minutes while it creates.  When the deployment is finished you will have a VM named "nov16."  You should now change the password of the administrator account of the VM.  Log into the Azure Portal. 

Open the CLI by clicking on the ">_" symbol in the upper right hand corner.  The lower part of the screen will become a dark blue.  Click "PowerShell (Windows)", and then click "Create storage".  Wait a few minutes for the initialization process to complete.  The upper left hand portion of the lower screen should say "PowerShell."

Prepare a draft of this command below by doing the replacing "azubuntu" with the username (if you changed the username in step #2), replacing "changethis" with the name of the resource group (unless you never changed it in step #8), replacing "coolPassword" with the password you want the user to have:

az vm user update --resource-group changethis --name nov16 --username azubuntu --password coolPassword

#Run this command in the PowerShell CLI prompt in the Azure Portal.

To quote Azure PowerShell "The supplied password must be between 6-72 characters long and must satisfy at least 3 of password complexity requirements from the following: 1) Contains an uppercase character 2) Contains a
lowercase character 3) Contains a numeric digit 4) Contains a special character 5) Control characters are not allowed."

10.  Now you should be able to log into the new machine "nov16" with the username and password there were used in step #9.

11.  If you want to re-use this procedure,  Modify the parameters.json file in two ways as described in i. and ii. below.

     i. Replace "nov16" in with the name you want to give to the VM (the hostname).
  ii. Change the "diag604" values to something else in the diagnosticsStorageAccountName's value and the diagnosticsStorageAccountId's value.  Save the file.  Here is an excerpt from the file as an example after the values have been changed:

 "diagnosticsStorageAccountName": {
            "value": "changethisdiag605a"
        },
        "diagnosticsStorageAccountType": {
            "value": "Standard_LRS"
        },
        "diagnosticsStorageAccountId": {
            "value": "Microsoft.Storage/storageAccounts/changethisdiag605a"

12.  You are now done.

* If you need assistance with determining your subscription ID, see this posting.

**  If you do not want to create a new resource group named "changethis", stop the PowerShell script and do these two things:

  1.  Go to the template.json file and replace all instances of "changethis" with the name of your choice.  
  2.  Go to the parameters.json file and replace all instances of "changethis" with the name of your choice.  

*** To find out potential regions (and the VM sizes that are correspondingly available), run these Azure PowerShell command:
# Use Login-AzureRmAccount if you have not logged in yet.  Then run these commands:
$list = Get-AzureRmLocation
echo $list.DisplayName

In the future we want to re-write these directions to use write-verbose instead of write-host. We want to warn people that write-verbose is recommended instead of write-host (according to this posting).

How Do You Create Servers Using .JSON Files and Resource Manager in Azure?

Problem scenario
You want to automate the creation of servers in Azure.  You want to try ARM templates to learn more about them.  How do you use .json files and Azure PowerShell to do this (without using the web UI)?

Solution
Prerequisites
i. You have a way of running Azure PowerShell commands.  If you need assistance with this, see this posting if your workstation is running Windows 7 or this posting if your workstation is running Windows 10. 
ii. You can log into the Azure Portal at the end (to set the password of the administrator user that you create when you create the VM with this process).

Procedures
This example will create a Windows server. 

1.  Create a file named CreateVMcontint.json with the contents below, but do two things first.  The first thing is to replace variables that start with "cont" or "my" as desired.  The second thing is to replace the user "contint" and its password "neatPassword5" with credentials you want to use.  Keep in mind that this .JSON file will have administrator credentials of the servers you create with it.

Here it is (CreateVMcontint.json):

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "adminUsername": { "type": "string" },
    "adminPassword": { "type": "securestring" },
    "contintuser": {     "type": "string",
        "defaultValue": "contintuser",
       "metadata": {
        "description": "Username for the Virtual Machine."           
       }
       },
     "neatPassword": {     "type": "securestring",
       "defaultValue": "neatPassword5",
       "metadata": {
        "description": "Password for contintuser."           
       }
       },
       "contint": { "type": "string",
          "defaultValue": "contint",
            "metadata": {
        "description": "Name of the Virtual Machine."           
           }
         }
     },
  "variables": {
    "vnetID": "[resourceId('Microsoft.Network/virtualNetworks','continualNET')]",
    "subnetRef": "[concat(variables('vnetID'),'/subnets/contIntSubnet')]",     
  },    
  "resources": [
    {
      "apiVersion": "2016-03-30",
      "type": "Microsoft.Network/publicIPAddresses",
      "name": "myPublicIPAddress",
      "location": "[resourceGroup().location]",
      "properties": {
        "publicIPAllocationMethod": "Dynamic",
        "dnsSettings": {
          "domainNameLabel": "contintresourcegroupdns1"
        }
      }
    },
    {
      "apiVersion": "2016-03-30",
      "type": "Microsoft.Network/virtualNetworks",
      "name": "continualNET",
      "location": "[resourceGroup().location]",
      "properties": {
        "addressSpace": { "addressPrefixes": [ "10.0.0.0/16" ] },
        "subnets": [
          {
            "name": "contIntSubnet",
            "properties": { "addressPrefix": "10.0.0.0/24" }
          }
        ]
      }
    },
    {
      "apiVersion": "2016-04-30-preview",
      "name": "contint",
      "type": "Microsoft.Compute/virtualMachines",
      "location": "[resourceGroup().location]",
      "dependsOn": [
        "[resourceId('Microsoft.Network/networkInterfaces/', 'contIntNic')]"
      ],
      "properties": {
        "hardwareProfile": { "vmSize": "Standard_DS1" },
        "osProfile": {
          "computerName": "[parameters('contint')]",
          "adminUsername": "[parameters('contintuser')]",
          "adminPassword": "[parameters('neatPassword')]"
        },
        "storageProfile": {
          "imageReference": {
            "publisher": "MicrosoftWindowsServer",
            "offer": "WindowsServer",
            "sku": "2012-R2-Datacenter",
            "version": "latest"
          },
          "osDisk": {
            "name": "myManagedOSDisk",
            "caching": "ReadWrite",
            "createOption": "FromImage"
          }
        },
        "networkProfile": {
          "networkInterfaces": [
            {
              "id": "[resourceId('Microsoft.Network/networkInterfaces','contIntNic')]"
            }
          ]
        }
      }
    },    
    {
      "apiVersion": "2016-03-30",
      "type": "Microsoft.Network/networkInterfaces",
      "name": "contIntNic",
      "location": "[resourceGroup().location]",
      "dependsOn": [
        "[resourceId('Microsoft.Network/publicIPAddresses/', 'myPublicIPAddress')]",
        "[resourceId('Microsoft.Network/virtualNetworks/', 'continualNET')]"
      ],
      "properties": {
        "ipConfigurations": [
          {
            "name": "ipconfig1",
            "properties": {
              "privateIPAllocationMethod": "Dynamic",
              "publicIPAddress": { "id": "[resourceId ('Microsoft.Network/publicIPAddresses','myPublicIPAddress')]" },
              "subnet": { "id": "[variables('subnetRef')]" }
            }
          }
        ]
      }
    }
  ]
}

2.  Create a second file named Parameters.json with the contents below, but replace the user "contint" and its password "neatPassword" with credentials you want to use.  Keep in mind that this .JSON file will have administrator credentials of the servers you create with it.

Here is the contents of Parameters.json:

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json",
  "contentVersion": "1.0.0.0",
  "parameters": {
  "adminUserName": { "value": "contintuser" },
    "adminPassword": { "value": "neatPassword" }
  }
}

3.  Run this PowerShell script, but remember two things: first it will require an interactive login to the Azure account.  Secondly you will want to change the paths to the two .json files from "C:\path\to\" to the location they reside in on your server (and you may want to change the ResourceGroupName from "contIntGroup" to something else as well as the the location from "East US" to "West US"):

Login-AzureRMAccount
$storageName = "st" + (Get-Random)
New-AzureRmStorageAccount -ResourceGroupName "contIntGroup" -AccountName $storageName -Location "East US" -SkuName "Standard_LRS" -Kind Storage
$accountKey = (Get-AzureRmStorageAccountKey -ResourceGroupName contIntGroup -Name  $storageName).Value[0]
$context = New-AzureStorageContext -StorageAccountName $storageName -StorageAccountKey  $accountKey
New-AzureStorageContainer -Name "templates" -Context $context -Permission Container

Set-AzureStorageBlobContent -File "C:\path\to\CreateVMcontint.json" -Context $context -Container "templates"
Set-AzureStorageBlobContent -File "C:\path\to\Parameters.json" -Context $context -Container templates

$templatePath = "https://" + $storageName + ".blob.core.windows.net/templates/CreateVMcontint.json"
$parametersPath = "https://" + $storageName + ".blob.core.windows.net/templates/Parameters.json"
New-AzureRmResourceGroupDeployment -Name "contintDeployment" -ResourceGroupName "contIntGroup" -TemplateUri $templatePath -TemplateParameterUri $parametersPath

How Do You Install UiPath for Free to Try out RPA?

Problem scenario
You want to install UiPath Studio CE to test out RPA on a Windows 2016 Server.   How do you get a free version of UiPath (e.g., Studio CE)?

Solution
Prerequisite #1
You will have to agree to this agreement on UiPath's website if you want to use the product.

Prerequisite #2
For this test, we recommend at a minimum 1 CPU at 2.3 Ghz and 2.25 GB of RAM.  This will not be a high performing machine, but it will minimally help you with a proof of concept.  It is strongly recommended to have 2 CPUs and more memory.

Procedures
1.  Open I.E.  If you are prompted, you may want to accept the recommended settings.

2.  Go to the gear icon in the upper righthand corner.  Go to "Internet Options."

3.  Go to Security tab and then click on "Local Intranet."

4.  Click on the "Sites" button.

5.  Enter this text with no quotes "http://download.uipath.com/" and click "Add".

6.a.  In I.E., browse to this URL: http://download.uipath.com/UiPathStudioSetup.exe

6.b.  To the option that comes up, choose "Run"

7.  You will eventually see "Welcome to UiPath Studio."  Click "Start Free".  

8.  Enter an email address in the prompt that asks for it.  Click "Activate".

9.  To the prompt about "Content from the website listed below is being blocked..." click Close.

11.  You are done.  The following steps are optional to harden your Windows Server.  Go to the gear icon in the upper righthand corner.  Go to "Internet Options."

12.  Go to Security tab and then click on "Local Intranet."

13.  Find the "http://download.uipath.com/" URL and remove it.