How Do You Install and Configure the AWS CLI on a Linux Server (without the pip Command)?

Problem scenario
Using different package managers can lead to dependencies being met while systems administrators will encounter error messages that detect, find or determine something to the contrary (e.g., false unmet dependency errors or conditions).  Some enterprise policies disallow the pip command.  How do you set up the AWS CLI on a Linux server without the pip command?

Solution (Part 1 and Part 2)
Prerequisites
i.  You must have Python installed.  Run this command:  python --version
Make a mental note of the version if it is installed. You may want to try this too (if the command did not work): python3 --version

ii.  This only applies if you are using CentOS/RHEL/Fedora.  (Skip this prerequisite (ii and all of its sub-steps) if you are using a different distribution of Linux):
ii.a.  Do this if you don't have Python:  sudo yum -y install python3
ii.b.  You need pip or pip3 to be installed.  If you need help installing pip, see this posting.
ii.c.  Run this command:  sudo pip3 install colorama rsa s3transfer ruamel.yaml botocore  # Use "pip3" instead of "pip" if you have "pip3". You may want to try sudo /usr/local/bin/pip if pip3 cannot be found.

iii.  If Python is not already installed, run one of the following if you have Debian or Ubuntu Linux:
sudo apt -y install python3
or
sudo apt -y install python

Procedures
Part 1:  Install AWS CLI
Step #1:  If the AWS CLI has already been installed, skip to step #5.  Otherwise using a web browser, go to the GitHub site to retrieve the .tar.gz file with the installation media. You may want to use curl -L <URL found above> > /tmp/nameoffile

Step #2:  Click on the desired version (e.g., 1.18).

Step #3:  Right click the hyperlink for "Source code (tar.gz)" and copy that hyperlink.

Step #4.a:  If you are running Debian or Ubuntu Linux, do this step, otherwise skip to step 4.b:
If you are running Python version 2.x, run this command:  sudo apt-get -y install python-setuptools
If you are running Python version 3.x, run this command:  sudo apt-get install python3-setuptools

Step #4.b:  Run these commands:
curl -L https://.../1.16.foobar.tar.gz > /tmp/1.16.foobar.tar.gz   #where https://.../1.11.foobar.tar.gz  is the hyperlink copied in the step above.
sudo mv /tmp/1.16.foobar.tar.gz /bin
cd /bin
sudo tar -zxvf 1.16.foobar.tar.gz  # where foobar.tar.gz is the name of file downloaded from the above curl
cd aws-cli.version  #where aws-cli.version is the name of the directory created in the above step
sudo python3 setup.py build   #This command should be with "python" instead of "python3" if you have Python 2.x only.
# *** (See below.)
sudo python3 setup.py install  #This command should be with "python" instead of "python3" if you have Python 2.x only.
# If you are running RHEL 8.x, run this command: sudo python3 setup.py install ../..
Run this: sudo ln -s /usr/local/bin/aws /bin/aws
sudo aws --version or aws --version  # The response should indicate a version.  If the command was not found, aws cli was not installed.

Part 2:  Configure AWS CLI
If AWS CLI is already installed, start with step #5.

Step #5: You must have your default region name, AWS Access Key ID, and AWS Secret Access Key available.  If you have them, skip the next steps and start working at Step #9.  To retrieve the region name, log into the AWS console from a web browser.  By default when you log in, the URL should have what you need.  Find the portion of the URL that is to the right of the "=" sign.  For example, "us-east-1" is to the right of the "=" sign: https://console.aws.amazon.com/console/home?region=us-east-1

Step #6: To find the AWS Access Key ID and AWS Secret Access Key, in the AWS console, click on your name in the upper right hand corner.  Then click on "My Security Credentials."

Step #7:  Click "Create New Access Key"

Step #8:  Click "Show Access Key."  Keep these values handy for the next step.

Step #9:  Log into the Linux server with AWS CLI.  Type this command followed by "Enter":  aws configure

Respond to the first three prompts with something; the last of the four prompts can be blank and you can just press "enter."

AWS Access Key ID [None]: 
AWS Secret Access Key [None]: 
Default region name [None]: 
Default output format [None]:

Step #10:  To test, try this command followed by "Enter": aws ec2 describe-instances
This will test that you have connectivity to your AWS account, the AWS Access Key ID has permissions to list the EC2 (virtual) servers, and the AWS Secret Access Key was valid.

*** If commands above or below the three asterisks "***" throw an error about the version of setuptools being too low, try the command again.  It may work a second time.  This is only advisable if you are setting up a proof-of-concept or configuring an environment without sensitive data.  For production or hardened environments, you may want to investigate as normal.  If you need assistance installing the latest version of setuptools (and it may not be necessary, repeating the command may work as a kludge), see this posting.  You may need to follow that posting's directions, then run the commands to build and install setup.py for setuptools in a repetitive way.  We only recommend repeating the commands that failed the first time to make it work if you are not using sensitive data.  There may be a more serious issue at hand.

In Case You Want Something Different
If you are using Debian/Ubuntu Linux and can install pip (or have pip installed), you can install the AWS CLI with this alternative directions.

Using Bash How Do You Replace a Line with a String with a Forward Slash in a File by Its Line Number?

Problem scenario
How do you insert a string with forward slashes using a perl command like this?

perl -i -pe 's/.*/Hello World/ if $.==15' foobar.txt

This command would insert "Hello World" on the 15th line of the text file named foobar.txt.  Instead of "Hello World" you want something else inserted that happens to include forward slashes ("/").  You want to identify a line in a file by its line number.  You want this line to be overwritten with a string that includes, among other characters, a forward slash.  How do you do this?

Solution
Use "\" backslashes right before the forward slashes.  Here is an example to insert cfg_dir=/usr/local/someprog/etc/hadoop into foobar.txt (line 15 with full deletion of line 15's previous content):

perl -i -pe 's/.*/cfg_dir=\/usr\/local\/someprog\/etc\/hadoop/ if $.==15' foobar.txt

# For your reference:
# The "/" to the left of the "cfg_dir" is for the command itself.  
# The last "/" is for the perl command itself (not the input).

Using Bash How Do You Replace a Line in a File by Its Line Number?

Problem scenario
On your Linux server you know one line in a file needs to be replaced.  You can identify the file by its line number.  You have a string or pattern you want to be inserted where this line is. How do you do this with Bash?

Solution
To replace all the content of line 15 of foobar.txt with "Hello World" run this command:

perl -i -pe 's/.*/Hello World/ if $.==15' foobar.txt

For something more advanced, you may want to inject a string with forward slashes.  Unix uses forward slashes in directory paths.  Perl does not parse these forward slashes like other alphanumeric symbols.  You have to use a back slash to have Perl place a forward slash in the manner described above.   This example below will replace whatever is on line 34 of /tmp/contint.txt with "/subdirectory/path/to/file" with no quotes.

perl -i -pe 's/.*/ \/subdirectory\/path\/to\/file/ if $.==2' blah

How Do You Deploy Puppet Agent to an Ubuntu 16 Linux Server in AWS?

Updated on 4/15/18

Problem scenario
You installed Puppet Master on an AWS instance of Linux.  (See this link if you want to install Puppet Master on Ubuntu.  If you want to install Puppet Master on a Red Hat Enterprise Linux server, see this link). You now want an AWS instance of Ubuntu to be configured as a Puppet Agent node with open source Puppet version 5. You want to run a manifest to make sure that this new Puppet Agent node (i.e., server) is working with your Puppet Master server.  What do you do to install Puppet Agent on an Ubuntu server and configure it to work with Puppet Master?

Solution
This solution is for Ubuntu 16 Linux instances in AWS.  You could easily adapt this solution to other Ubuntu servers in different environments.  It explains how to deploy Puppet and apply a manifest (in combination with these directions) from scratch.  These directions have been tested to work with a Puppet Master server on either an AWS RHEL server or an AWS Ubuntu Linux server.

Prerequisite
Using AWS change the Security Group so that the inbound rules will allow connections from the Puppet Agent server.  One way of doing this is to find the internal IP addresses of the Puppet Agent server.  This Linux command should help you if ran on an Ubuntu Linux server in AWS:

ip addr show | grep inet | grep -v 127.0.0.1 | grep -v inet6 | cut -c 10-24 | awk -F "/" '/1/ {print $1}'

Using the AWS console configure the relevant AWS Security Group to allow an inbound connection from the IP address in the result above.  Disregard a trailing backslash and any CIDR or numbers to the right of the backslash (e.g., ignore any trailing "/xx").

Here is a detailed explanation of how to create an inbound connection in an AWS security group:

Go to Security Groups.  Find the relevant security group and click the "Inbound" tab.  Then click "Edit."  Click "Add Rule."  Then choose for "Type" in the dropdown menu "Custom TCP Rule."  For "Port Range" choose 8140.  For the "Source" drop down option, choose "Custom."  Enter the internal IP address of the Puppet agent like this:

x.x.x.x/32

Substitute x.x.x.x with the internal IP address (as found with the above "ip addr show" command).

Procedures
1.  On the Puppet Master server, run these four commands:

sudo puppet master
sudo puppet resource service puppet ensure=running enable=true
hostname -f
ip addr show | grep inet | grep -v 127.0.0.1 # remember the IP address result from this command

2.  On the Puppet Agent server, modify the /etc/hosts file.  It should have this stanza where x.x.x.x is the internal IP address from the Puppet Master server (the result of the last command ran in step #2) and "FQDNofPuppetMaster" is the FQDN of the Puppet Master server:

x.x.x.x puppet FQDNofPuppetMaster

3.  On the Puppet Agent node, create a file name puppetagent.sh in /tmp/.  Have the content of this file be the following:

#!/bin/bash
# Written by continualintegration.com
# This script will install Puppet agent on Debian 9 or Ubuntu 16.x
# This script will not work on Ubuntu 17.x

distro=$(cat /etc/*-release | grep NAME)

debflag=$(echo $distro | grep -i "ubuntu")
if [ -z "$debflag" ]
then   # If it is not Ubuntu, test if it is Debian.
  debflag=$(echo $distro | grep -i "debian")
  echo "determining Linux distribution..."
else
   echo "You have Ubuntu Linux!"
   curl -O https://apt.puppetlabs.com/puppetlabs-release-pc1-xenial.deb
   dpkg -i puppetlabs-release-pc1-xenial.deb
   apt -y update
   apt-get -y install puppet-agent
fi

if [ -z "$debflag" ]
then
  echo "You could do not have Debian Linux."
else
   echo "You are using Debian Linux."
 #  Commented out lines are provided as a reference in case you need a higher version.
 #  curl http://http.us.debian.org/debian/pool/main/p/puppet/puppet_5.4.0-2_all.deb > /tmp/puppet_5.4.0-2_all.deb
 #  dpkg -i /tmp/puppet_5.4.0-2_all.deb
 #  apt -y update
 #  apt-get -y install ruby-shadow ruby-deep-merge ruby init-system-helpers ruby-augeas puppet
   apt-get -y install puppet
fi

systemctl start puppet
systemctl enable puppet
ln -s /opt/puppetlabs/puppet/bin/puppet /usr/bin/puppet
puppet agent -t -d
#cd /tmp
#wget https://apt.puppetlabs.com/puppet5-release-xenial.deb
#dpkg -i puppet5-release-xenial.deb

4.  On the Puppet Agent node become root or assume sudoers privileges.  Run this command:

sudo bash /tmp/puppetagent.sh

5.  On the Puppet Agent node update this file: /etc/puppetlabs/puppet/puppet.conf
If you cannot find it there, use this command: sudo find / -name puppet.conf
The last line of the [main] section of this puppet.conf file should have this stanza (where FQDNofPuppetMasterserver with the result of the "hostname -f" command ran on the Puppet Master server (in step #1)):

server=FQDNofPuppetMasterserver

6.  Go to the Puppet Master server.  Run this command:  sudo puppet cert list

7.  Assuming the above had some output such as puppet.agent.continualintegration.com, run this command (but substitute puppet.agent.continualintegration.com with the FQDN that resulted from the command in step #6):

sudo puppet cert sign puppet.agent.continualintegration.com

Alternative step #7: Assuming that step #6 showed no other servers that you do not want signed, run this command:

sudo puppet cert sign --all

8.  Test it.  If you are using Puppet 5, see if you have /etc/puppetlabs/.  If the puppetlabs subdirectory was not created, your installation failed.  The "puppet" and "puppet master" commands may work however.  The command puppet -V will tell you what version you are using.  But the installation may be defective if you have not /etc/puppet/ or /etc/puppetlabs/ directory path.

a) On the Puppet Master server go to /etc/puppetlabs/manifests/.  (The directory path used to be /etc/puppet/manifests.)  Create site.pp with the following content:

  exec { 'somethingneat':
        command => '/bin/date > /tmp/continual.txt'
       }

b)  On the Puppet Agent node, run this:  sudo puppet agent -t -d

c)  Check the /tmp/ directory for the file named "continual.txt."

How Do You Have PHP Return an Informative Error Message When a SQL Command Returns No Results?

Problem scenario
You have a PHP web page that integrates with Postgres behind the scenes.  You find that when a SQL command returns nothing (e.g., when a table name is used and it is known to not exist in the underlying database), the PHP web page is blank.  You want the result to be more clear about what happened or potentially happened.  How do you handle an exception or commonly occurring SQL query with no results using PHP?  

Solution
This assumes you were using pg_query.  Remember that pg_query returns a boolean value if the query failed.  If credentials were not typed in correctly the result of the pg_query() will be a false.  

Successful pg_query executions will return non-false statements -- but the value will not necessarily be "TRUE."  Failed executions will return false.

The solution is to introduce logic by testing the result of the pg_query() with an if statement.

...
$cc = pg_query($query);
if($cc) {} else {echo  '<td>' . "Query failure.  Did you enter the correct credentials?  Was the relevant table name typed correctly?  Was the user the owner of the database or table involved?  Were certain variables hard-coded that prevent the operation you attempted?" . '</td>';}

How Do You Install Puppet Master on an AWS Instance of Ubuntu 16?

Updated 12/1/17

Problem scenario
You need to install open source Puppet version 5 on a server to become a Puppet Master.  You have an AWS Ubuntu Linux server.  You only have five minutes to do it.  How do you install Puppet Master immediately?

Solution
1.  Create puppetinstaller.sh in the /tmp directory.  It should have the following lines (stop before you get to step #2):

#!/bin/bash
# Written by continualintegration.com

apt-get update
apt-get -y install ntp
echo '
server 0.us.pool.ntp.org
server 1.us.pool.ntp.org
server 2.us.pool.ntp.org
server 3.us.pool.ntp.org' >> /etc/ntp.conf

service ntp restart
cd ~ && wget https://apt.puppetlabs.com/puppet5-nightly/puppet5-nightly-release-xenial.deb
dpkg -i puppet5-nightly-release-xenial.deb
apt-get update
apt-get -y install puppetserver puppet-agent

2. Run this command:  bash /tmp/puppetinstaller.sh

3.  Verify it worked by running this command:  puppet -V
      You are done with the installation when you see the numeric version from the above command.

4.  This is an optional configuration step.Mentally decide how much RAM you want Puppet Master application to have.  (In small development environments it is possible the role of Puppet Master competes with another purpose of the server.)  If you are not sure how much RAM you have, try run this command:  cat /proc/meminfo | grep Mem

Create this file or modify the existing one:  /etc/default/puppetserver
In this file put a stanza like one of the following, but if the file exists modify a single stanza you see that is like it:

JAVA_ARGS="-Xms512m -Xmx512m"

JAVA_ARGS="-Xms1g -Xmx1g"

JAVA_ARGS="-Xms2g -Xmx2g"

# The above stanzas would limit Puppet Master's RAM usage at 512 MB, 1 GB and 2 GB respectively.

#  If you want to increase the amount of virtual memory, you can use this posting for directions on how to create swap space.  If you want more RAM, you can upgrade the flavor of the AWS EC-2 instance (aka virtual server) by following this link.

If you want to see directions like these for installing Puppet Master on a Debian server in GCP, see this posting.

How Do You Create a Table inside a Specific Postgres Database?

Problem scenario
You want to create a table in a specific Postgres database.  You want to use a command line interface (not a GUI) for this task.  (The GUI version of these directions can be found here.)  What are the SQL commands to do this?

Solution
This assumes you have a role (a username and password) to connect to the Postgres database with permissions to create tables.

1.  Connect to the Postgres instance (e.g., either through a web front-end or the command line interface).  If you are unsure on how to do this, see this link. If you use an desktop application to connect to the instance, skip step #2.

2.  Connect to the database using a command like this:
\c nameofdatabase \\where nameofdatabase is the name of the database you want

3.  Run a SQL [DDL] command like this (where cool is the name of the table and column1 is the name of the first column in the table):  create table cool(column1 int);

If you are not sure what database you are connected to, run this command:
SELECT current_database();

The database you connect to with a GUI application is chosen at the time you configure the connection.

How Do You List the Tables of a Specific Postgres Database?

Problem scenario
You want to see the names of the tables of a specific Postgres database.  You know the Postgres database name.  How do you do list its tables?

Solution
This assumes you have a role (a username and password) to connect to the Postgres database.

1.  Connect to the Postgres database (e.g., either through a web front-end or the command line interface).  If you are unsure on how to do this, see this link.

2.  Connect to the database with a command like this:
\c nameofdatabase \\where nameofdatabase is the name of the database you want

3.  Run this SQL command:
\dt

How Do You Change a Firewall Rule in Google Cloud Platform?

Problem scenario
You want to create an exception to your GCP firewall to allow connectivity from a different workstation or server.  What should you do?

Solution
1.  In the upper right hand corner of the console, click the icon with three horizontal bars.  This is the "Products and Services" button.  Then go to "VPC Network" -> "Firewall Rules."

2.  Click "Create Firewall Rule."

3.  Fill out the required options.

4.  Click Create.

How Do You Install Npm and/or node.js on a AWS Instance of RHEL?

Problem scenario
Using a RHEL server in AWS, you try this command "npm -v", and it shows that npm is not installed.  You want to use the npm utility or you want Node.js to be installed on your RedHat version 7 Linux server in AWS.  You may have tried various yum commands but received errors about dependencies.  Various other scripts fail to get npm to be a recognized command.  How do you install npm and/or Node.js without changing yum repositories and without using the wget command?

Solution
Warning: This solution takes approximately 30 minutes, but it does not require interaction.  Do not try this solution on a sever with less than 1 GB of RAM. If you want extra swap space, you can see this posting. If you want to install npm on a different distribution of Linux, see this posting.

1.  Create a file named "npminstaller.sh" (with no quotes) with the following content:

version=8.1.3
#If version changes to 9, update the cd node-v8* command below
cd /opt/
curl http://nodejs.org/dist/v$version/node-v$version.tar.gz > node-v$version.tar.gz
tar xzvf node-v$version.tar.gz
cd node-v8*
yum -y install gcc gcc-c++
./configure
make
make install

2.  Run the script with sudo bash npminstaller.sh with no quotes.