How Do You Deploy the WildFly Application Server in Docker?

Problem scenario
You want to prepare for WildFly Swarm to run Java EE programs.  You want to deploy Red Hat's WildFly application server in a Docker container.  You are running a Linux server.  What do you do?

Solution
Background
The regular JBoss Application server was renamed to Wildfly.  Do not confuse the JBoss application server with the JBoss web server.  The JBoss Enterprise Application server still exists.

Prerequisites
This assumes you have already installed Docker.  To install Docker on Ubuntu see this posting; to install Docker on a RedHat server see this posting.

Procedures
#1  Run these three commands from the Linux server:
docker run -it jboss/wildfly
docker run -p 8080:8080 -p 9990:9990 -it jboss/wildfly /opt/jboss/wildfly/bin/standalone.sh -bmanagement 0.0.0.0
curl http://icanhazip.com # to find the external IP address of the Docker host

#2  Open a web browser.  Go to this URL where x.x.x.x is the external IP address found above:  http://x.x.x.x:9990

How Do You Solve the Problem of the “docker login” Command Seeming to Work but Hanging for a Long Period of Time and/or Erroring out with “connection refused”?

Problem scenario
"docker login" is challenging you for a username and password.  But after you enter the credentials and press enter, it hangs.  You are not logging into the Docker registry.  What is wrong?

Solution
It may be surprising, but you can be challenged for a username and password when the Docker registry is turned off.  If a firewall blocks a necessary port to the Docker registry or if the Docker registry's server is off completely, you may get a slow response.  Eventually you may get "connection refused."  You may want to install nmap to test the relevant port.  You may want to verify the Docker registry server is turned on.

How Do You Troubleshoot “Permission Denied” When Trying to Use “Docker Login” to Authenticate with a Docker Registry?

Problem scenario
You are trying to log into a Docker registry but you get this message:

"Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.29/auth: dial unix /var/run/docker.sock: connect: permission denied"

You know the credentials are correct.  What is wrong?

Root cause
It could be that your user account is not a member of the "docker" group on the Linux system.  Despite being challenged for a username and password, this message is not about authentication with the Docker registry server.  It pertains to the user not being a member of the "docker" group. 

Possible Solution #1 (recommended)
Add the Linux user trying to run the "docker login" command to the "docker" group.  You may need to reinstall Docker to work without a "sudo " command; if you need assistance with this, see this posting.

Possible Solution #2 (very inadvisable and only acceptable in rare instances)
When you run "docker login https://x.x.x.x" use "sudo" in front of it like this:

sudo docker login https://x.x.x.x

You could be root and run docker login https://x.x.x.x but this is inadvisable like the sudo  alternative above. Assuming you are the jdoe user, you could run a command like this to add it to the "docker" group: sudo usermod -aG docker jdoe

Possible Solution #3
The credentials you were using were actually incorrect.  Get known working credentials and try again.

How Do You Troubleshoot the Problem “No package ‘Luajit’ Found” When Attempting to Install Snort?

Problem scenario
You install the Lua Just-In-Time package named luajit (e.g., with sudo apt-get -y install luajit).  You are trying to install Snort.  When you run the ./configure script, you get this error:

"
-- Checking for module 'luajit'
--   No package 'luajit' found
CMake Error at /usr/share/cmake-3.5/Modules/FindPackageHandleStandardArgs.cmake:148 (message):

/usr/bin/luajit-2.0.4
/usr/share/luajit-2.0.4/jit
"

You tried this:

./configure_cmake.sh --with-luajit-libraries=/usr/share/luajit-2.0.4/jit/ --with-luajit-includes=/usr/bin

But that command failed too.

Solution
This is a work around.  Installing snort from source can be difficult.  Run these two commands:
sudo apt-get -y update
sudo apt -y install snort

If you are using a RedHat derivative, you may want to get a binary file to install it (e.g, from rpmfind.net).

To confirm snort was installed, run this:

snort --version

How Do You Deploy Puppet Agent to a RedHat Linux Instance in AWS?

Problem scenario
You installed Puppet Master on an AWS instance of RedHat Enterprise Linux.  (See this link if you want to install Puppet Master.) You now want another AWS instance of RHEL 7.x to be configured as a Puppet Agent node. You want to run a manifest to make sure that this new Puppet Agent node (i.e., server) is working with your Puppet Master server.  What do you do to install Puppet Agent 5.x on a RedHat Linux server and configure it to work with Puppet Master?

Solution
This solution is for RHEL 7.3 instances in AWS.  You could easily adapt this solution to other RedHat servers in different environments.  It explains how to deploy Puppet and apply a manifest (in combination with these directions) from scratch.

Prerequisite
Using AWS change the Security Group so that the inbound rules will allow connections from the Puppet Agent server.  One way of doing this is to find the internal IP addresses of the Puppet Agent server.  This command should help you if run on a Linux server in AWS:

ip addr show | grep inet | grep -v 127.0.0.1 | grep -v inet6 | cut -c 10-24 | awk -F "/" '/1/ {print $1}'

Configure the relevant AWS Security Group allow an inbound connection from the IP address in the result above.  

Here is a detailed explanation of how to create an inbound connection:
Go to Security Groups.  Find the relevant security group and click the Inbound tab.  Then click "Edit."  Click "Add Rule."  Then choose for "Type" in the dropdown menu "Custom TCP Rule."  For "Port Range" choose 8140.  For the "Source" drop down option, choose "Custom."  Enter the internal IP address of the Puppet agent like this:

x.x.x.x/32

Substitute x.x.x.x with the internal IP address (as found with the above "ip addr show" command).

Procedures
1.  On the Puppet Master server run these three commands:

sudo puppet master restart
hostname -f
ip addr show | grep inet | grep -v 127.0.0.1 | grep -v inet6 | cut -c 10-24 | awk -F "/" '/1/ {print $1}'

# Remember the IP address from the last command above.  Disregard any trailing backslash "/" or numbers thereafter.

2.  On the Puppet Agent server, modify the /etc/hosts file.  It should have this stanza where x.x.x.x is the internal IP address from the Puppet Master server (the result of the last command ran in step #2):

x.x.x.x puppet

3.  On the Puppet Agent node, run these two commands:

sudo rpm -ivh https://yum.puppetlabs.com/puppet5/puppet5-release-el-7.noarch.rpm
sudo yum -y install puppet

4.  On the Puppet Agent node update this file: /etc/puppetlabs/puppet/puppet.conf
The last line of the [main] section of this puppet.conf file should have this stanza (where FQDNofPuppetMasterserver with the result of the "hostname -f" command ran on the Puppet Master server (in step #2)):

server=FQDNofPuppetMasterserver

5.  Run these two commands on the Puppet agent node:

sudo ln -s /opt/puppetlabs/bin/puppet /usr/bin/puppet
sudo puppet agent

6.  Go to the Puppet Master server.  Run this command:  sudo puppet cert list --all

7.  Assuming the above had some output such as puppet.agent.continualintegration.com, from the Puppet  Master server run this command (but substitute puppet.agent.continualintegration.com with the FQDN that resulted from the command in step #7):

sudo puppet cert sign puppet.agent.continualintegration.com

Alternative step #8: Assuming that step #7 showed no other servers that you do not want signed, run this command:

sudo puppet cert sign --all

8.  Test it.

a) On the Puppet Master server go to /etc/puppetlabs/code/environments/production/manifests/.  Create site.pp with the following content (replace the FQDN):

  exec { 'somethingneat':
        command => '/bin/date > /tmp/continual.txt'
       }

b)  On the Puppet Agent node, run this:

sudo puppet agent -t -d

c)  Check the /tmp/ directory for the file named "continual.txt."

How Do You Control, Manage, List, Upload and Download Files to and from S3 without Using the GUI?

Problem scenario
You are using a Debian distribution of Linux.  You want to upload and download files to S3 without using the GUI.  You want to be able to automate processes with scripts that interact with S3 from your Ubuntu Linux server.  How do you do this?

Solution
The procedures have two parts after you have the prerequisites met.

Prerequisites
i. You need to have python-setuptools to be installed. To test if it is installed, run these two commands:

python
from setuptools import setup

If you get a message like "ImportError" or it is not installed, see this posting.

ii. You need the clock on your server to be configured to be relatively accurate. For servers that deviate significantly, s3cmd will not work. To configure ntp, see this external posting.


Part 1 Set up s3cmd
1.  Download the installation media.  Get the latest .tar.gz file from here (at least 2.0).
2.  Save it to the /tmp/ directory initially because the write permissions are usually relaxed.  Then copy it to /opt/.
3.  Run this command (but substitute s3cmd-* if there is more than one file with this file name prefix to identify the file you just downloaded):

sudo tar -xvf s3cmd-*

4.  Run these three commands:

cd s3cmd*
sudo python setup.py build
sudo python setup.py install

5.  Run this command and read the steps below to help answer the prompts you will see:

sudo s3cmd --configure

# Respond to the prompts of the above command.  If you do not know how to get the Access Key ID or the Secret Access Key, go to this URL and look at "Part 2."

For the encryption password, make up any value you want.

You can accept the default for this prompt:

"Path to GPG program [/usr/bin/gpg]:"

For use "HTTPS Protocol" you can enter "No" unless you want a different option.

For this prompt, just hit enter:

"HTTP Proxy server name:"

For test access, choose "Y."

Then save the settings if it was successful.

Part 2  Use s3cmd to test
Run this command:
s3cmd ls

To learn more about the s3cmd commands, go to this site.

What is a Build?

Question
In DevOps engineering you hear the term "build", what is it?

Answer
A build is usually a binary file which is an executable application. In DevOps engineering a build is usually compiled from a specific version of source code. In the context of Jenkins it is the "[r]esult of a single execution" of a Jenkins project (previously known as a Jenkins job). In the context of the modern and popular 12 Factor Application methodology, the build process should be separated from the run process (taken from https://12factor.net/). There are instances when the build process is closely tied to an application's execution; this would diverge from the 12 Factor Application process.

For a definition of the "build process" see this posting. To read other definitions, see these external sites:
https://searchsoftwarequality.techtarget.com/definition/build
https://www.techopedia.com/definition/3759/build
https://en.wikipedia.org/wiki/Software_build

What is Apache Parquet?

Question
What is Apache Parquet?

Answer
Apache Parquet is columnar data representation/manipulation tool for a Hadoop ecosystem.  Data in a given column is largely uniform (e.g., a long string of characters, a single character, or an integer) in that it repeats a specific type and format of data as opposed to two cells in the same row (which may be very dissimilar types of data).  Columns may be in memory or disk -- just as relational rowset data is.  The columns will be more compressible (in RAM or on disk) due to the underlying data uniformity than rows themselves.  If you want to learn more about columnar databases, see this link.

The term parquet refers to "diminutive" (as in a compartment) in French. The open-source Apache program Parquet was indeed created by a French-speaking person. In English it is pronounced "par-kay" (like "par" for the course and "cay" as in the "Cayman Islands").  Also in English parquet can refer to long flooring strips (which physically resemble vertical columns).  If you want to learn more about Parquet, see this PDF file. 

Literature about Apache Parquet (like the link above or on Apache's website) refer to the Dremel paper.  What is the Dremel paper?  According to citations in a Wired article we believe that "the" Dremel paper is this one.  Dremel is not a person. "Dremel is a scalable, interactive ad-hoc query system for analysis of read-only nested data."  (This quote was taken from the 10-page PDF linked above.)  If you want to install the open source version of Apache Parquet, see this article.

How Do You Install Apache Parquet?

Problem scenario
You want to install Apache Parquet on the Hadoop namenode.  What do you do?

Solution
Prerequisite
This assumes that you have installed Hadoop.  For directions, see this posting.

Procedure
Run these commands:

sudo su -
apt-get -y install pip
pip install thriftpy
pip install snappy
exit

sudo apt-get -y install libsnappy-dev thrift-compiler

curl https://pypi.python.org/packages/74/b5/bc459aab0566fc3cf3397467922c37411ab6e3361bab9e0ca165e1089ce8/parquet-1.2.tar.gz#md5=05aacec0620ac63ecd7dd77bf7fb9fee > /tmp/parquet-1.2.tar.gz
sudo cp /tmp/parquet-1.2.tar.gz /opt/
cd /opt
sudo tar -xvf parquet-1.2.tar.gz
cd parquet-1.2
sudo python setup.py build
sudo python setup.py install