How Do You Configure Maven to Use an Apache Parquet Plugin?

Problem scenario
You want to use Maven's Apache Parquet plugin with Hadoop.  How do you use these Apache technologies together?

Solution
1.  Install HDFS.  See this link if you are using Ubuntu.  See this link if you are using a RedHat distribution of Linux. If you have more than one server and want a multi-node cluster of Hadoop, see this link for directions on how to deploy and configure it.

2.  Install maven (e.g., "sudo apt-get -y install maven" or see this post if you are using a RedHat derivative)

3.  Run this command (but feel free to substitute "contint" and "cont-int" with the names of your choice):
mvn archetype:generate -DgroupId=com.contint.app -DartifactId=cont-int -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false

4.  Run this command (you'll need the location for steps #5 and #6): sudo find / -name pom.xml

5.  Modify the file found above.  In pom.xml under the lowest </dependency> tab, add these five lines (to create additional dependency element):
   <dependency>
      <groupId>org.apache.parquet</groupId>
      <artifactId>parquet-avro</artifactId>
      <version>1.9.0</version> <!-- or latest version -->
   </dependency>

6.  Assuming you are in the directory with the pom.xml file modified above, run mvn package (with no quotes). If you were not in the directory with the pom.xml file, change directories, then run mvn package.

How Do You Get to See Settings and The Help Menu of Atlassian HipChat?

Problem Scenario
You are logged into the web UI of HipChat.  You expect to hover over either your name or the question mark icon in the upper right hand corner and see drop down options.  But this is not what happens.  When you single-click on your name or the question mark icon, you briefly get a glimpse of menu options.  You want to click on the menu options.  How do you do this?

Solution
Double-click your name or the question mark.  Now the drop down menu should remain long enough for you to click on an option.

Roughly 1.5% of Workstations Run Linux and Linux vs. Windows at the Workplace

Various companies such as Google use Linux desktops instead of Windows (according to ZDNet). According to TechReublic other big names that use it are NASA, the French military police, CERN (which is also known as the European Organization for Nuclear Research) and the U.S. Department of Defense. While an estimated 1.5% of desktops use Linux, it is a good choice for staying safe from viruses and spyware.  

Another case for using Linux is that employees can benefit from the experience with an increase in earning potential.  See this article for more information.

To see how Macs play into the desktop market and compete with Linux, here is an article from 2012.

Development in a Windows environment is not like Linux.  As a recommended practice in Linux, running things as administrator should be done on an ad hoc basis.  Some people who develop in Windows recommend that Visual Studios always be run as administrator.  To learn more about this, see this link.

Should more people use Linux for everyday tasks and/or software development?  What is your opinion?

How Do You Resume a DVD Movie Where You Stopped Watching It?

Problem scenario
In Windows you sometimes watch DVDs but have to stop the movie mid-way through.  You cannot always watch an entire movie in one setting from start-to-finish.  You stop the DVD by closing Windows Media Player while the movie was still playing or paused.  Later after you have shut off your computer, you want to resume the DVD at the exact hour, minute and second where you were last watching it.  How do start the movie at the point where you left off?

Solution
Go to My Computer.  Find the drive letter for the DVD player.  Right click it and choose "Open AutoPlay..."

How Do You Learn More About “Low Latency Programming”?

Problem scenario
You want to learn more about low latency programming for working at a hedge fund company.  Where do you begin?

Solution
An Amazon search for "low latency programming" brings up books that could be very useful for other high-paying I.T. jobs in big data such as an Apache Thrift book or a book for Building a Columnar Database on RAMCloud.  These technologies are relevant to the Hadoop ecosystem.  If you want to learn more about "low latency programming" you may want to read one of the following such as High-Performance Computing Using FPGAs, High-Speed Decoders for Polar Codes, or Low Latency, High Performance Data Stream Processing: Systems Architecture, Algorithms and Implementation.  You may also want to learn more about distributed computing and Parquet.

If you want to learn more about RAM and how to deal with it as a programmer for high-performance computing or optimization, you may want to read this paper by a professional at Red Hat, Inc.

You may also want to view these links:

How Do You Turn off OS Firewalls in an AWS Instance of Linux SUSE?

Problem scenario
You are running Linux SUSE in AWS.  You want to have no firewalls running on the OS.  How can you be sure that there is no OS-level firewall running in Linux SUSE?

Solution
#1  Become root with this command: sudo su -
#2  Enter this command:  yast firewall
#3  You will see either a screen that warns you about another firewall running or you'll see the YaST2 screen for a firewall.  If the situation is the former, then you know another firewall is running.  If the latter is the case in this screen you will see "Stop Firewall Now" in aqua.  Other other letters will be in light gray; this means that no firewall is running on your Linux OS.  If you can put a cursor on "Stop Firewall Now", then the Yast Firewall is running. 

If you want to turn off a firewall from your AWS instance, but the Yast Firewall menu above did not help you, follow these steps.  First be sure you know what Security Groups that your server is in.  An AWS Security Group can simulate an OS firewall for many purposes.  Secondly, do these things:

Run this command:

$ sudo iptables-save > /root/firewall.rules

Then run this script that was taken from a cyberciti.biz posting to disable the Linux OS firewall:

#!/bin/bash
# reset.fw - Reset firewall
# set x to 0 - No reset
# set x to 1 - Reset firewall
# ---------------------------------------------------------------------------------------------------------------
# Added support for IPV6 Firewall
# ---------------------------------------------------------------------------------------------------------------
# Written by Vivek Gite <vivek@nixcraft.com>
# Modified by continualintegration.com.  We found it worked better with Linux SUSE in AWS with the modifications.
# ---------------------------------------------------------------------------------------------------------------
# You can copy / paste / redistribute this script under GPL version 2.0 or above
# =============================================================
x=1

# set to true if it is CentOS / RHEL / Fedora box
RHEL=false

### continualintegration.com edited a few lines here  ###
IPT=/usr/sbin/iptables
IPT6=/usr/sbin/ip6tables

if [ "$x" == "1" ];
then
    if [ "$RHEL" == "true" ];
    then
          # reset firewall using redhat script
        /etc/init.d/iptables stop
        /etc/init.d/ip6tables stop
    else
        # for all other Linux distro use following rules to reset firewall
        ### reset ipv4 iptales ###
        $IPT -F
        $IPT -X
        $IPT -Z
        for table in $(</proc/net/ip_tables_names)
        do
            $IPT -t $table -F
            $IPT -t $table -X
            $IPT -t $table -Z
        done
        $IPT -P INPUT ACCEPT
        $IPT -P OUTPUT ACCEPT
        $IPT -P FORWARD ACCEPT
        ### reset ipv6 iptales ###
        $IPT6 -F
        $IPT6 -X
        $IPT6 -Z
        for table in $(</proc/net/ip6_tables_names)
        do
            $IPT6 -t $table -F
            $IPT6 -t $table -X
            $IPT6 -t $table -Z
        done
        $IPT6 -P INPUT ACCEPT
        $IPT6 -P OUTPUT ACCEPT
        $IPT6 -P FORWARD ACCEPT
    fi
else
        :
fi
# End of script

If you changed your mind, you can revert the firewall changes by running this command without the leading "#" symbol:
# sudo iptables-restore < /root/firewall.rules

How Do You Optimize SQL to Get Away from Disk Contention, to Reduce the I/O Activity?

Problem scenario
Let's say you want to rearrange the columns a of SQL table because a special order would make subsequent SQL commands easier.  Or you just want to avoid disk contention on the underlying storage device.  How do you optimize SQL to do less input/output activity on the disk while being able to rearrange the order of columns?

Solution
There are other ways such as avoiding "select *", changing the frequency that indexes are updated, normalizing tables or partitioning tables.  For this answer we recommend using a view.  A view can accomplish the rearrangement of columns and put the results of a SELECT statement into memory as opposed to leaving a table on the disk.  This does take more RAM.  Here is an example of a view based on a table named "goodTable":

create view continualView AS select a.column3, a.column1, a.column2 * from goodTable a

Now continualView can be queried like goodTable.  You would want to avoid views if your environment was memory constrained.  But if there was ample memory and there I/O activity on the storage device (e.g., NAS, SAN, or hard disk of the server) was a bottleneck for the performance of your SQL database, then a view may help.

At the OS level if you are running Linux, you may want to enable hugepages.

How Do You Troubleshoot the Hadoop NameNode Service Not Running Properly?

Problem scenario
The Hadoop NameNode service won't start.  You run start-dfs.sh and it starts the SecondaryNameNode and the DataNode.  It won't start the NameNode.

Solution
Assumption: This solution only works if you are running the start-dfs.sh as a sudo user or the root user itself.

Procedures
Verify you can ssh with root to the localhost.  Run this command twice without exiting from the first session:  ssh root@localhost

If you cannot do that, verify the SSH service is running.  Verify you set up SSH keys for passwordless authentication for root.  You will need to be able to run these in succession without exiting them:

ssh root@localhost
ssh root@localhost

What Is an ARM in Modern Information Technology Vernacular?

Question
What Is an ARM in Modern Information Technology Vernacular?

Answer / Disambiguation
In I.T. jargon the term ARM can refer to a number of different things.  Here are some potential definitions of this acronym in random order:

#1  Azure Resource Manager is a component of Azure.  It can be interacted with in the console if you log into a Azure via a web browser.  ARM Templates are JSON files that can programmatically create servers, NICs, subnets etc.

#2 a) ARM can stand for an Advanced RISC Machine.  In I.T. RISC stands for Reduced Instruction Set Computing.  RISC technology is not limited to mobile devices.  IBM's AIX Unix servers leverage RISC architecture.  Sources include TechTarget and Wikipedia. 

#2 b) ARM could refer to the global technology company referred to as ARM Limited or ARM Holdings.

#2 c) Acorn RISC Machine.

#3  Application Response Measurement which is "an open standard published by the Open Group for monitoring and diagnosing performance bottlenecks within complex enterprise applications that use loosely-coupled designs or service-oriented architectures."  This quote was taken from Wikipedia's article on the subject.

#4  Asynchronous Response Mode.  This term is rare however.

#5  Adaptive and Reflective Middleware.   An annual workshop on this topic has been happening since 2004.  To learn more about the workshop itself see the Cyber Physical Systems Virtual Organization website.

How Do You Create a Linux Server in Azure with PowerShell When You Keep Getting an Error “Status Code: 400…Bad Request”?

Problem scenario
You are trying to create a Linux server in Azure with the Azure Tools for PowerShell.  You receive this message:

"Destination path for SSH public keys is currently limited to its default value due to a known issue in Linux provisioning agent."

You see  "StatusCode: 400 ... ReasonPhrase: Bad Request."

What do you do to create an Ubuntu 16.x server and not get this error?

Solution
Assuming you were using a .pub file with SSH when you received the error above, this solution below will work.  Create a VM without using a .pub file.  You have to enter the username and password interactively to get around the need for a .pub file being in a .ssh folder.

Here is a script that will create an Ubuntu 16.x Linux machine with 4 GB of RAM.  Just save the script then run it and answer the prompts.  You will need to manually enter your Azure credentials when prompted with a pop up after the script starts.

# Usage instructions
# Do five things before running this script:
# 1.  Change the password "ch@ngePassword" below
# 2.  Change the $resourceGroup name from "contIntGroup" to the one you want.  
# 3.  Change the location "eastus" unless that is what you want.
# 4.  Change the vmName variable from "contIntVM" to the name you want.
# 5.  Save the script locally.  It cannot run as an untitled draft.
# 6.  Optional: Change the IP addresses associated with the subnet mask and separately virtual network to something you want.
# 7.  Optional: You may want to change the names of the virtual network and subnet too.

# This script was modified from a copy taken
# from https://docs.microsoft.com/en-us/azure/virtual-machines/scripts/virtual-machines-windows-powershell-sample-create-vm

# This script creates a new resource group, a subnet mask and a NIC for the Linux server too.

write-host "*****************************"
write-host " "
write-host "ALERT: This script will require interaction.  It was designed that way."
write-host 'First you will be prompted to enter your Azure credentials to interactively login with a GUI.  This script therefore launches a web browser.'
write-host " "
write-host 'The subsequent GUI prompt will have the header "cmdlet Get-Credential at command pipeline..."'
write-host 'For this prompt, enter the credentials you want your Linux server to have.'
write-host "*****************************"
# Variables for common values
Login-AzureRmAccount
$resourceGroup = "contIntGroup"
$location = "eastus"
$vmName = "contIntVM"

# Definer user name and blank password
#$securePassword = ConvertTo-SecureString 'ch@ngePassword' -AsPlainText -Force
#$cred = New-Object System.Management.Automation.PSCredential ("contintuser", $securePassword)

$creda = Get-Credential

# Create a resource group
New-AzureRmResourceGroup -Name $resourceGroup -Location $location

# Create a subnet configuration
$subnetConfig = New-AzureRmVirtualNetworkSubnetConfig -Name contIntSubnet -AddressPrefix 192.168.1.0/24

# Create a virtual network
$vnet = New-AzureRmVirtualNetwork -ResourceGroupName $resourceGroup -Location $location `
  -Name ContIntNET -AddressPrefix 192.168.0.0/16 -Subnet $subnetConfig

# Create a public IP address and specify a DNS name
$pip = New-AzureRmPublicIpAddress -ResourceGroupName $resourceGroup -Location $location `
  -Name "mypublicdns$(Get-Random)" -AllocationMethod Static -IdleTimeoutInMinutes 4

# Create an inbound network security group rule for port 22
$nsgRuleSSH = New-AzureRmNetworkSecurityRuleConfig -Name ContIntNetworkSecurityGroupRuleSSH  -Protocol Tcp `
  -Direction Inbound -Priority 1000 -SourceAddressPrefix * -SourcePortRange * -DestinationAddressPrefix * `
  -DestinationPortRange 22 -Access Allow

# Create a network security group
$nsg = New-AzureRmNetworkSecurityGroup -ResourceGroupName $resourceGroup -Location $location `
  -Name myNetworkSecurityGroup -SecurityRules $nsgRuleSSH

# Create a virtual network card and associate with public IP address and NSG
$nic = New-AzureRmNetworkInterface -Name contIntNic -ResourceGroupName $resourceGroup -Location $location `
  -SubnetId $vnet.Subnets[0].Id -PublicIpAddressId $pip.Id -NetworkSecurityGroupId $nsg.Id

# Create a virtual machine configuration
$vmConfig = New-AzureRmVMConfig -VMName $vmName -VMSize Standard_D1 | `
Set-AzureRmVMOperatingSystem -Linux -ComputerName $vmName -Credential $creda | ` #-EnablePasswordAuthentication
Set-AzureRmVMSourceImage -PublisherName Canonical -Offer UbuntuServer -Skus 16.04-LTS -Version latest | `
Add-AzureRmVMNetworkInterface -Id $nic.Id

# Create a virtual machine
New-AzureRmVM -ResourceGroupName $resourceGroup -Location $location -VM $vmConfig

In the future we want to re-write these directions to use write-verbose instead of write-host. We want to warn people that write-verbose is recommended instead of write-host (according to this posting).