Cloud Computing Is Changing the Global Economy

The Economist, specifically the August 27th - September 2nd 2016 issue, on page 47 and 48, in the article "Linux and AWS," reports four fascinating facts:

1) The world's I.T. budget is approximately $3,400,000,000,000 (almost 3.5 trillion) dollars.   
2) As of mid-2016, Google's market share for cloud services increased 162% from the previous year!  
3) Roughly 65% of the over one million AWS customers are startups.
4) AWS has computing capacity that is ten times greater than its closest 14 competitors combined! (As of early September 2016.)

Update as of February 2021: If you combine IaaS and PaaS, AWS accounts for 32% of the market according to this website.

Update in early 2022: Worldwide spending on I.T. is predicted to be $4.5 trillion dollars for 2022 (according to Gartner). To put that in perspective, the GDP of the U.S. was estimated to be roughly $23 trillion dollars in 2021 (according to the Bureau of Economic Analysis).

How Do You Unzip Files Using Ubuntu (Using the Command Line Interface)?

Goal:  You want to use unzip with Ubuntu Linux without using the GUI desktop.
Background:  The unzip character-based utility is useful for many reasons.  At the command line you can access compressed files that normally occupy little space on the disk.  Transferring .zip files is great for keeping network bandwidth utilization low.

Try this command to see if unzip is installed: man unzip

Part 1:  Installing Unzip
If the command cannot be found, use this command:  sudo apt-get install unzip

If the above command does not work, (for example, you do not have access to the Internet, and your network has no Debian package repositories configured), get the file from here:  https://packages.debian.org/wheezy/unzip

To find the correct installation media (and choose the correct link from the website above), you need to determine the architecture.  To do this, use this command: dpkg --print-architecture

Once you get the correct file on to Ubuntu, install it with this command: dpkg -i unzip.*.deb

Part 2:  Using Unzip
Now that unzip is installed, you can use it with a command like this (assuming you have a file named foobar.zip):

unzip foobar.zip

The above command will decompress the contents while retaining a copy.  The extracted contents will be decompressed in the same directory you run the command in.  The extracted files will not go to the directory where foobar.zip is if you use the path of foobar.zip in the invocation of the unzip command (e.g., if you are in /root/ and run unzip /tmp/foobar.zip the extracted files will go into the /root/ directory).

How Do You Zip Files Using Ubuntu Linux From The Command Line Interface?

Goal:  You want to use zip with Ubuntu Linux without using the GUI.  You want to use the character prompt exclusively.

Background:  The zip character-based utility is useful for many reasons.  At the command line you can free up space while retaining files for future usage.  Try this command to see if zip is installed: man zip

Try this command to see if zip is installed: man zip

Part 1: Installing Zip
If the command cannot be found, use this command:

sudo apt-get install zip

If you do not have access to the Internet, and your network has no Debian package repositories configured, get the file from here:  https://packages.debian.org/wheezy/zip

To find the correct installation media (and choose the correct link from the website above), you need to determine the architecture.  To do this, use this command: dpkg --print-architecture

Once you get the file over, install it with this command: dpkg -i zip.*.deb

Part 2  How to Use Zip
Now that zip is installed, you can use it with a command like this (assuming you have three regular files named pub.txt, pub1.txt, pub2.txt):

zip target.zip pub.txt pub1.txt pub2.txt

The above command will create a zip file called target.zip.  The files will be compressed (that is, the .zip file will be smaller than the sum of the three .txt files' sizes).  This can help reduce disk I/O when fetching the file, moving it around, and reduce network congestion with fewer packets over the network when it is in transit.

How do you unzip files using SUSE?

Goal  You want to use unzip when you are using SUSE (either openSUSE or SUSE Linux Enterprise).

Background
The unzip character-based utility is useful for many reasons.  At the command line you can access compressed files that normally occupy little space on the disk.  Transferring .zip files is great for keeping network bandwidth utilization low.

Try this command to see if unzip is installed: unzip --version
AWS SUSE servers have unzip installed by default, but they have no man pages.

Part 1:  Installing Unzip
If the command cannot be found, use these commands:

sudo su -
zypper install unzip

If the above command does not work, (for example, you do not have access to the Internet, and your network has no Debian package repositories configured), go here to get a file:

http://software.opensuse.org/download.html?project=Archiving&package=unzip

To find the correct installation media (and choose the correct link from the website above), you need to determine the architecture.  To do this, use this command: cat /etc/*-release

The above command will tell you if you are using SUSE Linux Enterprise or openSUSE.  The buttons can be right clicked to copy the location (e.g., to use wget http://software.opensuse.org/ymp/Archiving/SLE_12_SP1/zip.ymp).  Otherwise you click "Grab binary packages directly" and then download the appropriate file to your workstation and transfer it over to the Linux machine (e.g., with SFTP or SCP).

Once you get the file over, install it with this command: zypper install /path/to/unzip.rpm

Part 2:  How to Use Unzip
Now that unzip is installed, you can use it with a command like this (assuming you have a file named foobar.zip):

unzip foobar.zip

The above command will decompress the contents while retaining a copy.  The extracted contents will be decompressed in the same directory you run the command in.  The files will not necessarily go to the directory where foobar.zip is (e.g., if you are in /root/ and run unzip /tmp/foobar.zip the extracted files will go into the /root/ directory).

How do you zip files using SUSE Linux?

Goal:  You want to use zip when you are using SUSE (openSUSE or SUSE Linux Enterprise).  You want to use the character (or command) prompt.  You do not want to use the GUI.
Background:  The zip character-based utility is useful for many reasons.  At the command line you can free up space while retaining files for future usage.  Try this command to see if zip is installed: man zip

Part 1 Installation Process
If the command cannot be found, use these commands:

sudo su -
zypper install -y zip

If you do not have access to the Internet (so a .ymp file could not help you), and your network has no .rpm package repositories configured, get the file from here:

http://software.opensuse.org/download.html?project=Archiving&package=zip

If you are using a free SUSE Linux with AWS, then your version is SLE (not openSUSE).  To find the exact version of SUSE you are using, issue this command: cat /etc/*-release
AWS SUSE servers have zip installed by default.

The buttons can be right clicked to copy the location (e.g., to use wget http://software.opensuse.org/ymp/Archiving/SLE_12_SP1/zip.ymp).  But .ymp files are only useful if your server has Internet access.  You click "Grab binary packages directly" and then download the appropriate file to your workstation and transfer it over to the Linux machine (e.g., with SFTP or SCP).    

Once you get the file over, install it with this command: zypper install /path/to/zip*rpm

Part 2  How to Use Zip

Now that zip is installed, you can use it with a command like this (assuming you have three regular files named pub.txt, pub1.txt, pub2.txt):

zip target.zip pub.txt pub1.txt pub2.txt

The above command will create a zip file called target.zip.  The files will be compressed (that is, the .zip file will be smaller than the sum of the three .txt files).  This can help reduce disk I/O when fetching the file, moving it around, and reduce network congestion with fewer packets over the network when it is in transit.

How do you unzip files using RHEL (or CentOS or Fedora Linux)?

How do you unzip files using Red Hat Enterprise Linux (or CentOS or Fedora Linux)?

Goal:  You want to use unzip when you are using Red Hat Enterprise Linux (or CentOS or Fedora Linux).
Background:  The unzip character-based utility is useful for many reasons.  At the command line you can access compressed files that normally occupy little space on the disk.  Transferring .zip files is great for keeping network bandwidth utilization low.

Try this command to see if zip is installed: man zip

Try this command to see if unzip is installed: man unzip

Part 1:  How to Install Unzip
If the command cannot be found, use these commands:

sudo su -
yum install -y unzip

If you cannot install unzip that way (either because you have no Internet connection or rpm package repository server configured on your network), go to this URL:  http://rpmfind.net

Search for "unzip."  To determine the exact version of RedHat Linux you are using, issue this command: cat /etc/*-release

While rpmfind.net has no RHEL packages, CentOS rpms should work. They definitely work for RHEL 7.2 for unzip.  

Once the .rpm is downloaded, you can use this command: rpm -ivh unzip-6.0-15.el7.x86_64.rpm

(where unzip-6.0-15.el7.x86_64.rpm is the exact name of the rpm package you downloaded)

Part 2: How to Use Unzip

Now that unzip is installed, you can use it with a command like this (assuming you have a file named foobar.zip):

unzip foobar.zip

The above command will decompress the contents while retaining a copy.  The extracted contents will be decompressed in the same directory you run the command in.  The files will not necessarily go to the directory where foobar.zip is (e.g., if you are in /root/ and run unzip /tmp/foobar.zip the extracted files will go into the /root/ directory).

How do you zip files using Red Hat Enterprise Linux (or CentOS or Fedora Linux)?

Goal:  You want to use zip when you are using Red Hat Enterprise Linux (or CentOS or Fedora Linux).
Background:  The zip character-based utility is useful for many reasons.  At the command line you can free up space while retaining files for future usage.  Try this command to see if zip is installed: man zip

Part 1:  How to install zip?
If the command cannot be found, use these commands:

sudo yum install -y zip

If you do not have access to the Internet, and your network has no rpm  configured, get the file from here:  http://rpmfind.net

Search for "zip."  To find the exact version of RedHat you are using, issue this command: cat /etc/*-release

While rpmfind.net has no RHEL packages, CentOS rpms should work. They definitely work with RHEL 7.2 for zip.  

Once the .rpm is downloaded, you can use this command: sudo rpm -ivh zip-3.0-10.el7.x86_64.rpm

(where zip-3.0-10.el7.x86_64.rpm is the exact name of the rpm package you downloaded)

Part 2:  How to Use Zip
Now that zip is installed, you can use it with a command like this (assuming you have three regular files named pub.txt, pub1.txt, pub2.txt):

zip target.zip pub.txt pub1.txt pub2.txt

The above command will create a zip file called target.zip.  The files will be compressed (that is, the .zip file will be smaller than the sum of the three .txt files).  This can help reduce disk I/O when fetching the file, moving it around, and reduce network congestion with fewer packets over the network when it is in transit.

How Do You Assign a Password to a Variable In Linux Without Making It Visible?

Goal:  You want to issue a command (e.g., an export command) that has a password in it.  But you do not want anyone looking over your shoulder to see it.  You do not want the password to be visible in the history of the shell.
Solution:  Type this 'read -s foobar'
Then press enter.  The cursor will go to the next line.  Type in the password.  The characters will not echo to the screen.  The cursor will not advance.  Press enter.  Now you can use the variable like this: $foobar
For example, you can use export special_setting=connectionString@format:jdoe:$foobar
You will now have exported a setting that will be interpret "$foobar" as the password you entered.  The content of the variable will never be displayed.  It is important to remember that the variable name in the first "read -s foobar" statement has no dollar sign "$".  While the variable's reference, for evaluation, uses a dollar sign "$".  This is a convention in bash scripts as well as interactive bash commands.

How to Troubleshoot PowerShell Downloading a File From the Internet

Problem scenario:  When trying to use PowerShell to download a file, get you this error 'Error calling "DownloadFile" with "2" argument(s): "An Exception occurred during a WebClient request."'

Solution:  Make sure you have ample space on the disk and make sure the destination folder exists.  A typo in the path or if the folder not existing could cause this error.  Insufficient disk space could also cause the problem.  Is the file you are trying to download very big?

How to Install Hadoop on an AWS Instance of RedHat Linux or an Azure Instance of CentOS

Updated 1/5/18

THESE DIRECTIONS ARE OUTDATED.  They are here as a reference for legacy purposes only.  For directions on how to install Hadoop on a RedHat or CentOS server, see this article.

Problem scenario

You want to install an open source version on a RedHat derivative distribution of Linux in a public cloud.  How do you do this?

Solution
These directions will allow you install Hadoop on a RedHat derivative (e.g., CentOS or RHEL) in a public cloud.  These directions are for a single node deployment.  (For a multi-node deployment, see these directions.) They include a script, directions how to run the script, and other necessary commands.  These directions have been tested to work on RedHat 7.4 in AWS and CentOS 6.8 in Azure.  The script was designed to install the open source version of Hadoop 2.8.1. 

1.  Log into CentOS or RHEL.    
2.  Run these interactively as they are not easily or safely scripted:

sudo adduser hadoop
sudo passwd hadoop  #respond with the password of your choice
ssh-keygen -t rsa -P "" # press enter to the prompt
sudo cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
sudo chmod 0600 ~/.ssh/authorized_keys
ssh 127.0.0.1   # respond with 'y' to accept the fingerprint
cd /tmp

3.  Create a script with the content below from "#!/bin/bash" to "Proceed with the manual steps".  Beware that this script could overwrite some files.  This script was intended for a new OS with no data or special configuration on it.

#!/bin/bash
# Written by continualintegration.com
yum install -y java-1.7.0-openjdk-devel wget
cd /home/hadoop
wget http://apache.claz.org/hadoop/common/hadoop-2.8.1/hadoop-2.8.1.tar.gz
tar xzf hadoop-2.8.1.tar.gz
mv hadoop-2.8.1 hadoop
echo '
export HADOOP_HOME=/home/hadoop/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin' >> ~/.bashrc
source ~/.bashrc
sed -i '/export JAVA_HOME/c\export JAVA_HOME=/usr/.' /home/hadoop/hadoop/etc/hadoop/hadoop-env.sh
echo '<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/app/hadoop/tmp</value>
  <description>A base for other temporary directories.</description>
</property>

<property>
  <name>fs.default.name</name>
  <value>hdfs://localhost:54310</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri has a scheme that determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri has authority that is used to
  determine the host, port, etc. for a filesystem.</description>
</property>
</configuration>
' > /home/hadoop/hadoop/etc/hadoop/core-site.xml
echo "Proceed with the manual steps"

4.  sudo bash nameOfscript.sh  # You must run script above as a sudoer or the root user.

5.  To get all the hdfs commands to work, one way would be to allow root to ssh to the local server and run "sudo bash" to kick off scripts to start the Hadoop daemons.  There are other ways, but here is how to do it with a reliance on the sudo user:

sudo su -
vi /etc/ssh/sshd_config
 #To allow root user to log in

#For Centos with Azure, change the "PermitRootLogin no" to "PermitRoot Login yes"

#For RHEL with AWS, change the "#PermitRootLogin yes" to "PermitRoot Login yes"

service sshd stop
service sshd start

ssh-keygen -t rsa -P "" #press enter to the default prompt.  

If you are using RedHat in AWS, use this command:
cat /home/ec2-user/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys

If you are using CentOS in Azure, use this command (but substitute centos with the username):
cat /home/centos/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys

Run these regardless of what OS or public cloud provider you are using:
cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys

ssh root@localhost
ssh root@localhost 
#do it again
exit # the second SSH session
exit # the first SSH session

6.  Complete the configuration and start the NameNode service while logged in as root:

hdfs namenode -format
bash /home/hadoop/hadoop/sbin/start-dfs.sh


THESE DIRECTIONS ARE OUTDATED.  They are here as a reference for legacy purposes only.  For directions on how to install Hadoop on a RedHat or CentOS server, see this article.