How Do You Install Docker on Different Distributions of Linux in Different Public Clouds or on an On-Premise Server?

Problem scenario
You want to install Docker.  You want directions for installing it with and without making changes to how packages are installed on the Linux server.  You want directions that apply to Linux servers in AWS, Azure, or GCP.  You want directions that can apply to CentOS/RHEL, SUSE, or Ubuntu.  What do you do?

Solution
Find the link below that is the most appropriate for your needs.

How To Install Docker on an AWS Instance of RedHat Linux  (These directions use CentOS packages.)
How Do You Install Docker on an AWS Instance of RHEL?  (These directions use RHEL packages.)
How To Install Docker on an AWS Instance of SUSE Linux?
How To Install Docker on an AWS Instance of Ubuntu  (These directions do make changes to the Debian package repositories of your Linux server and use dockerproject.org and Ubuntu.com for the installation media.)
How Do You Install Docker on an AWS Instance of Ubuntu?  (These directions do not make changes to the Debian package repositories of your Linux server.)
How Do You Install Docker-Compose on a Linux Server?
How Do You Install Docker on a RHEL VM in Azure?
How Do You Install Docker on a GCP VM Running RHEL?
How Do You Install Docker on Debian 9 in GCP?



To run a Docker container as a non-root user, see this posting.

How Do You Troubleshoot the Cassandra Problem about “Connection refused”?

Problem Scenario
You have installed Apache Cassandra on Linux.  It is deployed as a single node configuration (not a cluster).  When you run "cqlsh" you get this error:

"Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})"

How do you troubleshoot cqlsh when you get this message ""Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})"?

Solution
Possible solution #1
Try running this (but substitute x.x.x.x with internal IP address of the Cassandra server): cqlsh x.x.x.x

Possible solution #2
Modify /etc/cassandra/default.conf/cassandra.yaml so the rpc_address stanza uses the FQDN of the Cassandra server (as found with a hostname -f command).  Restart the Cassandra services with these commands:

sudo systemctl daemon-reload
sudo systemctl enable cassandra
sudo systemctl start cassandra

Possible solution #3
If you get cqlsh to work sometimes and not others, check /var/log/cassandra for clues.  Search for the word "error" in the .log files.  You may need to increase the amount of virtual memory or add RAM to your server.  Click this link for more information on how to add virtual memory (aka swap space).  If the server is in AWS click here for directions on how to resize it.  If the server is in GCP click here for directions on how to resize it.

Problem solution #4
You may want to see if a software firewall such as ufw or firewalld is running.  You may want to check if iptables is blocking the port.  For further network diagnosing, see this posting.  In our experience we do not see activity on port 9042 when Cassandra is properly listening and ready for a cqlsh command.

Possible solution #5
Deploy Cassandra again.  Remove the files that were deployed and created, and start over.  If you need directions for deploying Cassandra to a new, fresh Linux server, click on the link that corresponds with your distribution:

Debian or Ubuntu
CentOS/RHEL/Fedora
SUSE

How Do You Install kubectl on a CentOS/RHEL/Fedora of Linux server?

Problem scenario
You want the kubectl command to work on a server running a Red Hat distribution of Linux.  How do you install kubectl on a CentOS/RHEL/Fedora Linux server?

Solution
#1  This assumes that you have access to the internet.  You can do it with these three commands:

cd /tmp

curl -Lo kubectl https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl && chmod +x kubectl

sudo mv kubectl /usr/local/bin/

(An alternative way with a .repo file that is more secure would be to use steps #1 through #3 of How Do You Install Kubernetes on an AWS Instance of RedHat Linux? and then run "sudo yum -y install kubectl". )

#2  For kubectl commands to work, you will need a .kube directory in your home directory.  You may want to run this: mkdir ~/.kube

#3  Inside this .kube directory, you will need a file following this naming convention: config-foobar
Replace foobar with the name of your cluster.

#4  The contents of this config-foobar file should be something like this if you are using AWS:

apiVersion: v1
clusters:
- cluster:
    server: <endpoint-url>
    certificate-authority-data: <base64-encoded-ca-cert>
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: aws
  name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: aws-iam-authenticator
      args:
        - "token"
        - "-i"
        - "<cluster-name>"
         - "-r"
         - "<role-arn>"

#5  If you are not using AWS, you will need to modify the above template to the best of your ability.  If you are using AWS, follow these four steps (each quoted from https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html ):

"Replace the <endpoint-url> with the endpoint URL that was created for your cluster."  (It will appear under the "API server endpoint" value.)

"Replace the <base64-encoded-ca-cert> with the certificateAuthority.data that was created for your cluster."  (It will appear under the "Certificate Authority" value.)

"Replace the <cluster-name> with your cluster name."

Replace the <role-arn> with the value for the "Role ARN" as the web UI shows.

#6  You will need to run this command (but replace foobar with the name of your cluster):

export KUBECONFIG=$KUBECONFIG:~/.kube/config-foobar

(You may want to do this: "cd ~/.kube" and run "ls -lh".)

You may want to see these directions too.

How Do You Deploy Graylog in a Docker Container?

Problem scenario
You want to set up (install and configure) Graylog from a Docker container.  What do you do?

Solution
1.  Verify you have a least two processors with this command:  cat /proc/cpuinfo | grep -ic processor

Was the output two or more?  If so, proceed.  If not, get a server with two processors.  If you are using an AWS EC-2 instance, see this link to add a processor.  If you are using a GCP virtual server, see this posting for adding a processor.

2.  Install Docker.  If you need help with this, see this posting.

3.  Run these four commands (where x.x.x.x in the final command below is the external IP address found with a "curl icanhazip.com" command):

docker pull graylog2/server

docker run --name contint-mongo -d mongo:2

docker run --name contint-elasticsearch -d elasticsearch:2 elasticsearch -Des.cluster.name="graylog"

docker run --link contint-mongo:mongo --link contint-elasticsearch:elasticsearch -p 9000:9000 -e GRAYLOG_WEB_ENDPOINT_URI="http://x.x.x.x:9000/api" -d graylog2/server

4.  Wait a few minutes, and then open a web browser.  Go to "http://x.x.x.x:9000/" (with no quotes), where x.x.x.x is the external IP address of the Linux server.  The default credentials to the web UI of Graylog are as follows:

username: admin
password: admin

What is DevOps and Why Do People Want to Adopt It?

What is DevOps?
DevOps, among other things, is a business cultural shift that affects how labor is divided at different companies that adopt it. At the enterprise level, a company that embraces DevOps culture may have teams of employees focused on builds and releases or automation. These teams may be integrated with software engineers, systems administrators, quality assurance analysts, database administrators and security experts. A reduction itself in departments within I.T. in an organization may be a movement toward DevOps practices. Departments and specializations may be seen as factors that isolate professionals rather than integrate (for collaboration and building security into technology products) from a DevOps consultant's perspective. Some people say DevOps is automating what you can program (having programs being designed by automatic processes such as those in operations) and programming what you can automate (creating automation that is sophisticated for operations tasks).

Why do people and businesses want to use DevOps?
DevOps is a force multiplier. Code can be generated dynamically that adheres to a number of quality guidelines in an efficient way. Code itself can be instrumented (e.g., with a metaclass in a DevOps-favorite language of Python) and have the ability to manipulate or leverage itself.* This can enable greater observability, measurability, and quality assurance.

One reason to use DevOps is to develop software or I.T. solutions that are designed securely and highly performant. A second reason is to use Agile, or to be efficient with software development in fulfilling frequently-changing business requirements; to bring software (as a service through a website) to the market as quickly as possible, DevOps can help. A third reason is to moralize the employees by giving them more interesting work.

Ultimately development involves security and performance concerns and timing is critical for the delivery a new product to the marketplace. By dividing labor into specific categories that are inches wide and miles deep, the costs associated with the constraints from other teams can go unappreciated. Technology professionals frequently do not have visibility into their teammates' productivity obstacles. There often is not enough time to learn of different work blockers. Processes can be dependent on one workers' knowledge if a company shuns integration in favor of isolation (as exemplified in the seminal book The Phoenix Project). If the employees can achieve an optimal amount of communication (neither excessive nor insufficient), sharing responsibilities has a multitude of advantages.

The traditional division of responsibilities of I.T. development and operations can make people not work together. Rather than have programmers programming and systems administrators administering servers, companies mature in their DevOps adoption have people doing some duties of both programming and systems administration.

There are several goals that an enterprise has. By sharing broad responsibilities, each person has a greater appreciation for product and platform security. By leveraging tools and methods across specialized I.T. disciplines, organizations can perform more effectively. Internal tools can be built as an investment to help the business maximize its productivity. Having individuals leverage (and repurpose) multiple, unisolated tools can lead to efficiencies and creative utilizations that a highly departmentalized businesses would be ignorant of.

An enterprise may want to adopt DevOps culture which would involve "full stack developers" to engineer platforms and leverage programming to automate infrastructure tasks. At the individual level a DevOps practitioner may fulfill duties as a professional I.T. generalist. DevOps professionals may engage in activities as diverse as coding, installing and configuring operating systems, optimizing containers, creating or fixing a CI/CD pipeline, extending configuration management tools to deploy code or infrastructure patches rapidly, testing code, automating processes related to quality assurance, managing databases, or adjusting networking devices to enhance security. Harley Shaiken said "[s]killed work can be satisfying because it is varied, challenging, and creative." (This quote was taken from on page 113 of Disabling Professions, a book by several authors.) In The Mythical Man-Month, or more specifically "No Silver Bullet", the author Fred Brooks indicates that job growth could be limited for functional specialists in a matrixed team (pages 222 and 305). Some DevOps advocates have pushed for adopting it because they personally dislike the overly-repetitive tasks of highly specialized positions. Automation and well-built internal tools can lead toward greater quality control of a company's product line.

To comply with confusing regulations, you may want to use DevOps. Sarbanes Oxley encourages utilization of DevOps as developers must live in a development environment (according to this external site).

Some sources are skeptical of generalists handling the work of specialists.** Arguably Sarbanes Oxley discourages DevOps as generalists cannot write code and have access to production if the application deals with financial reporting. Some companies error on the side of caution and design jobs based on the regulations from a risk-averse attitude.

"For software developers, the big problem comes in Separation of Duties. Basically, SarbOx says that a software developer should not have ANY kind of write access to production systems that can affect the financial reporting of the corporation." (This previous quote was taken from Toolbox.com.) Automation capabilities and scaling capabilities of DevOps help make compliance achievable (with regulations and laws beyond just Sarbanes Oxley).

DevOps engineers and consultants generally could not have learned their trade at school (as of 2019). Ivan Illich said (on page 66 of The Right to Useful Unemployment) "education turns into the major generator of a disabling division of labour." The DevOps movement enhances the productive potential of many professionals. DevOps engineers tend to earn more than software engineers or systems administrators.

Shareholders, governing boards, and executives of businesses usually determine the extent of DevOps adoption more than the employees do. The DevOps Handbook indicates that a business' profitability is directly proportional to the degree that it adopts DevOps.


*Many implementations of DevOps have nothing to do with Python's metaclasses. It is just an example.

The term metaprogramming refers to the potential for a program to have knowledge of or manipulate itself. Python supports a form of metaprogramming for classes called metaclasses.

Taken from https://realpython.com/python-metaclasses/

To learn more about metaclasses in Python, see page 173 of Expert Python Programming, Third Edition.
** This article says that generalists can make mistakes.

How Do You Install Golang on to Any Type of Linux?

Updated on 10/18/19

Problem scenario
You want to do some coding in the Go programming language.  You want a script to install Golang on any distribution of Linux (e.g., CentOS/Red Hat Enterprise Linux/Fedora, Debian/Ubuntu, or SUSE).  What do you do?

Solution
1.  Create a file in /tmp/ called go.sh with the following content:

version=1.10.8  # Change this version as needed
curl https://storage.googleapis.com/golang/go$version.linux-amd64.tar.gz > /tmp/go$version.linux-amd64.tar.gz
cp /tmp/go$version.linux-amd64.tar.gz /usr/local/go$version.linux-amd64.tar.gz
cd /usr/local/
tar xzvf go$version.linux-amd64.tar.gz
#echo "export PATH=""$""PATH:/usr/local/go/bin" >> ~/.profile
echo "export PATH=""$""PATH:/usr/local/go/bin" > /etc/profile.d/go.sh
echo "You will have to log out and log back in for go to work."
echo "You can use the 'go version' command to determine what version was installed"

2.  Run it like this:  sudo bash /tmp/go.sh

How Do You Demonstrate Recursion in a Sorting Algorithm Written in Python?

Problem scenario
You want to use recursion in Python in a sorting algorithm. How do you do this?

Solution
Save this program as contintsort.py and run it python contintsort.py. The code closer to the top is in Python 2, and the code closer to the bottom is in Python 3.

# Written by www.continualintegration.com
# Usage is self-explanatory. Just run the program. python contintsort.py
# No extra files are needed.
# This demonstrates a sorting algorithm.  It runs in O(n^2) time in worst case scenarios.
# It runs in O(n) time in best case scenarios (when the data is already sorted).
# It accepts user input as long as the input is an integer.

import random, re  # We need random to generate random integers.  We need re to do "regular expressions"
# Regular expressions are methods of searching one pattern of text with another.

print "If you just press enter with the prompts below, you will get random integers assigned."
print "Enter an integer or press enter to get one assigned at random."
print " "

first = raw_input("Enter the first number: ")
second = raw_input("Enter the second number: ")
third = raw_input("Enter the third number: ")
fourth = raw_input("Enter the fourth number: ")
fifth = raw_input("Enter the fifth number: ")
sixth = raw_input("Enter the sixth number: ")
seventh = raw_input("Enter the seventh number: ")
eighth = raw_input("Enter the eighth number: ")
ninth = raw_input("Enter the ninth number: ")
tenth = raw_input("Enter the tenth number: ")
coolList = [first, second, third, fourth, fifth, sixth, seventh, eighth, ninth, tenth]

for i in range(10):
  if coolList[i] == "":
    coolList[i] = random.randint(1,1001) # Assign if a blank was entered
  else:
    b = coolList[i] # Assign input to a variable.
    a = (re.search('[a-zA-Z]', b)) # Check for non-numerical characters that were inputted.
    if a: coolList[i] = random.randint(1,1001) # Assign random int if alphabet chars were entered
    else: coolList[i] = int(coolList[i]) # make input an integer (cast input to be an integer)

def swapper(counter, interListb, flag):   # Define a function that requires three parameters when called.
  if (flag == 'exitnow'):
     print "************************************"  # This function will end if the flag was set to exit.
  else:
     x = interListb[(counter)]
     y = interListb[(counter + 1)]
     if (y < x):   # swap and then verify previous pair in list is ordered after swap
       interListb[(counter + 1)] = x
       interListb[(counter)] = y
       counter = counter -1  # Now that there was a swap, let's see of the pair to the left of swapped number is in order.
       if (counter < 1):
           counter = 0 # Do not go outside the index.  Go to the left-most item in the list.
       swapper(counter, interListb, flag) # Make recursive calls to check adjacent pairs in the list moving to the left.
     else:          # do not swap. do move to the right in the list to continue checking pairs.
       var1 = (len(interListb) - 2) # Counter will be incremented later. Lists are 0 based. Length counts the 0th (first) item.
       if counter >= var1:  # The right-most item was reached.  Set a flag to exit the recursive calling of swapper.
           flag = 'exitnow' # Exit recursion with the next call.
       counter = counter + 1
       swapper(counter, interListb, flag) # Make recursive calls to check adjacent pairs in the list moving to the right.
  return interListb # Return the list outside of the recursion.  In Python you do not return something inside a recursive call.

qq = swapper(0, coolList, 'enter') # This invokes swapper for the first time.
print qq

Here is the program in Python 3:

# Written by www.continualintegration.com
# Usage is self-explanatory. Just run the program. python contintsort.py
# No extra files are needed.
# This demonstrates a sorting algorithm.  It runs in O(n^2) time in worst case scenarios.
# It runs in O(n) time in best case scenarios (when the data is already sorted).
# It accepts user input as long as the input is an integer.

import random, re  # We need random to generate random integers.  We need re to do "regular expressions"
# Regular expressions are methods of searching one pattern of text with another.

print("If you just press enter with the prompts below, you will get random integers assigned.")
print("Enter an integer or press enter to get one assigned at random.")
print(" ")

first = input("Enter the first number: ")
second = input("Enter the second number: ")
third = input("Enter the third number: ")
fourth = input("Enter the fourth number: ")
fifth = input("Enter the fifth number: ")
sixth = input("Enter the sixth number: ")
seventh = input("Enter the seventh number: ")
eighth = input("Enter the eighth number: ")
ninth = input("Enter the ninth number: ")
tenth = input("Enter the tenth number: ")
pre_list = [first, second, third, fourth, fifth, sixth, seventh, eighth, ninth, tenth]

for i in range(10):
  if pre_list[i] == "":
    pre_list[i] = random.randint(1,1001) # Assign if a blank was entered
  else:
    b = pre_list[i] # Assign input to a variable.
    a = (re.search('[a-zA-Z]', b)) # Check for non-numerical characters that were inputted.
    if a: pre_list[i] = random.randint(1,1001) # Assign random int if alphabet chars were entered
    else: pre_list[i] = int(pre_list[i]) # make input an integer (cast input to be an integer)

def swapper(counter, intermediate_list, flag):   # Define a function that requires three parameters when called.
  if (flag == 'exitnow'):
     print("************************************")  # This function will end if the flag was set to exit.
  else:
     x = intermediate_list[(counter)]
     y = intermediate_list[(counter + 1)]
     if (y < x):   # swap and then verify previous pair in list is ordered after swap
       intermediate_list[(counter + 1)] = x
       intermediate_list[(counter)] = y
       counter = counter -1  # Now that there was a swap, let's see of the pair to the left of swapped number is in order.
       if (counter < 1):
           counter = 0 # Do not go outside the index.  Go to the left-most item in the list.
       swapper(counter, intermediate_list, flag) # Make recursive calls to check adjacent pairs in the list moving to the left.
     else:          # do not swap. do move to the right in the list to continue checking pairs.
       var1 = (len(intermediate_list) - 2) # Counter will be incremented later. Lists are 0 based. Length counts the 0th (first) item.
       if counter >= var1:  # The right-most item was reached.  Set a flag to exit the recursive calling of swapper.
           flag = 'exitnow' # Exit recursion with the next call.
       counter = counter + 1
       swapper(counter, intermediate_list, flag) # Make recursive calls to check adjacent pairs in the list moving to the right.
  return intermediate_list # Return the list outside of the recursion.  In Python you do not return something inside a recursive call.

sorted_list = swapper(0, pre_list, 'enter') # This invokes swapper for the first time.
print(sorted_list)

What is an Azure Logic App?

Question
You have heard about Azure's Logic Apps.  What are they and what do they do?

Answer
Azure Logic Apps are a PaaS solution.  Logic Apps execute in response to a triggering event (e.g., recurring on a schedule based on time, receiving a REST API call, or a Tweet being posted).  They initiate a variety of different tasks as you can determine the OS architecture (either Windows or Linux) without having to manage that same OS.

Logic Apps are very flexible and powerful.  Logic Apps can do many things such as message queueing, monitor Twitter accounts (if you enter your Twitter account's credentials), integrate with SalesForce, or send REST API POST calls with authentication headers. 

How Do You Install Redis on RedHat 7?

Problem scenario
You want to install Redis on RedHat Enterprise Linux 7.x.  (Redis gets its name from "REmote DIctionary Server" according to this external link.  It is one of the most popular NoSQL solutions on the market according to G2 Crowd.)  What do you do to install Redis on a RHEL server?

Solution
Prerequisite
This solution requires wget.  You may need to install it: sudo yum -y install wget

Procedures
Run these three commands:
sudo wget -r --no-parent -A 'epel-release-*.rpm' http://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/
sudo rpm -Uvh dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-*.rpm
sudo yum -y install redis

To test the installation, run these two commands:

sudo service redis start
redis-cli ping  
# This should return the word "PONG"

How Do You Install Zookeeper on Any Type of Linux?

Updated on 5/14/20 with a minor update on 11/13/21

Problem scenario
You want to install Apache Zookeeper with a script.  You want the same script to work regardless of what distribution of Linux you are using (e.g., Ubuntu/Debian, RedHat (including CentOS/Fedora), or SUSE).  How do you write a script to install the latest version of Zookeeper?

Solution
The file that is attached called zki.txt is not needed but is attached as a reference copy of the below.  Create a file in /tmp/ and call it zookeeper.sh.  Below is the content of /tmp/zookeeper.sh:

#!/bin/bash
# Written by www.continualintegration.com

version=3.7.0 #change this as needed
debianjavaversion=11

distro=$(cat /etc/*-release | grep NAME)

debflag=$(echo $distro | grep -i "ubuntu")
if [ -z "$debflag" ]
then   # If it is not Ubuntu, test if it is Debian.
  debflag=$(echo $distro | grep -i "debian")
  echo "determining Linux distribution..."
else
   echo "You have Ubuntu Linux!"
fi

rhflag=$(echo $distro | grep -i "red*hat")
if [ -z "$rhflag" ]
then   #If it is not RedHat, see if it is CentOS or Fedora.
  rhflag=$(echo $distro | grep -i "centos")
  if [ -z "$rhflag" ]
    then    #If it is neither RedHat nor CentOS, see if it is Fedora.
    echo "It does not appear to be CentOS or RHEL..."
    rhflag=$(echo $distro | grep -i "fedora")
    fi
fi

if [ -z "$rhflag" ]
  then
  echo "...still determining Linux distribution..."
else
  echo "You have a RedHat distribution (e.g., CentOS, RHEL, or Fedora)"
  yum -y install java-1.8.0-openjdk* nc   # install nc for initial testing only.
  JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk
  echo 'export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk' >> ~/.bashrc
  source ~/.bashrc
  source /etc/environment
fi

if [ -z "$debflag" ]
then
  echo "...still determining Linux distribution..."
else
   echo "You are using either Ubuntu Linux or Debian Linux."
   apt-get -y update # This is necessary on new AWS Ubuntu servers.
   apt -y install openjdk-$debianjavaversion-jre-headless unzip
fi

suseflag=$(echo $distro | grep -i "suse")
if [ -z "$suseflag" ]
then
  if [ -z "$debflag" ]
  then
    if [ -z "$rhflag" ]
      then
      echo "*******************************************"
      echo "Could not determine the Linux distribution!"
      echo "Installation aborted. Nothing was done."
      echo "******************************************"
      exit
    fi
  fi
else
   zypper -n install java-1_8_0-openjdk
fi

curl https://dlcdn.apache.org/zookeeper/zookeeper-$version/apache-zookeeper-$version-bin.tar.gz > /tmp/apache-zookeeper-$version-bin.tar.gz 
cp /tmp/apache-zookeeper-$version-bin.tar.gz /bin/apache-zookeeper-$version-bin.tar.gz
cd /bin
tar -zxvf apache-zookeeper-$version-bin.tar.gz
sudo mkdir -p /usr/lib/jvm/java-$debianjavaversion-openjdk-amd64/conf/management/
sudo touch /usr/lib/jvm/java-$debianjavaversion-openjdk-amd64/conf/management/management.properties

echo 'tickTime=2000
dataDir=/var/zookeeper
clientPort=2181' > /bin/apache-zookeeper-$version-bin/conf/zoo.cfg

echo "To find the version that is installed, and confirm installation worked, run these two commands."
echo "cd /bin/apache-zookeeper-"$version"-bin/bin"
echo "sudo /bin/bash zkServer.sh start-foreground"
echo "Then start a second terminal session to the server"
echo "From the second terminal session, run this echo command:"
echo "echo status | nc 127.0.0.1 2181"
echo "Then use Ctrl-c on the first terminal to exit out and move on."