CONTINUAL INTEGRATION – Page 213 – A Technical I.T./DevOps Blog

How Do You Deploy Docker to Ubuntu Linux Servers Using Puppet?

Problem scenario
You want to deploy Docker to Ubuntu Linux using Puppet. How do you write a manifest to install Docker an Ubuntu server?

Solution
This solution demonstrates transferring a file with Puppet.

1. Install Puppet master on an Ubuntu Server. See this link if you want directions.
2. Install Puppet agent on a different Ubuntu server. See this link if you want directions.
3. On the Puppet Master server, run this command (taken from this link):

puppet module install puppetlabs-docker_platform --version 2.2.1

4. On the Puppet Master server, go to /etc/puppet/module/docker/. Create a directory named "files".

5. Still on the Puppet Master server, in the "files" directory, create a file named "docker_inst.sh" with the following content:

#!/bin/bash
apt-get update
apt-get install apt-transport-https ca-certificates
sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
echo 'deb https://apt.dockerproject.org/repo ubuntu-trusty main' > /etc/apt/sources.list.d/docker.list
echo 'deb http://cz.archive.ubuntu.com/ubuntu trusty main' >> /etc/apt/sources.list.d/docker.list
apt-get update
apt-cache policy docker-engine
apt-get --assume-yes install linux-image-extra-$(uname -r) linux-image-extra-virtual
apt-get --assume-yes install docker-engine
sudo service docker start
sudo groupadd docker
sudo usermod -aG docker ubuntu
docker run hello-world

6. Still on the Puppet master server, go to /etc/puppet/manifests/. Modify site.pp to have these stanzas:

file { '/tmp/docker_inst.sh':
   mode => '0644',
   source => 'puppet:///modules/docker/docker_inst.sh',
 }
exec { 'install_docker':
    command => "/bin/bash /tmp/docker_inst.sh"
}

7. On the Puppet Agent server, run this:

puppet agent -t -d

How Do You Deploy Docker to RedHat Linux Servers Using Puppet?

Problem scenario
You want to use Puppet to deploy Docker to a RHEL server in AWS. How do you do this?

Solution
Warning: This may not work for Puppet versions 5 or higher. It should work for Puppet version 4 or lower. This solution is not a best practice. It is quick way to deploy a version of Docker made for CentOS to RHEL instances. It demonstrates transferring a file with Puppet. This solution is for educational purposes only.

1. Install Puppet master on a RedHat Server. For directions, see this link.
2. Install Puppet agent on a different RedHat server. For directions, see this link.
3. On the Puppet Master server, run this command (taken from this link):
puppet module install puppetlabs-docker_platform --version 2.2.1

4. On the Puppet Master server, go to /etc/puppet/module/docker/. Create a directory named "files".
5. Still on the Puppet Master server, in the "files" directory, create a file named "docker.repo" with the following content:

[dockerrepo]
name=Docker Repository
baseurl=https://yum.dockerproject.org/repo/main/centos/7/
enabled=1
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg

6. Still on the Puppet master server, go to /etc/puppet/manifests/. Modify site.pp to have these stanzas:

file { '/etc/yum.repos.d/docker.repo':
   mode => '0644',
   source => 'puppet:///modules/docker/docker.repo',
 }
 exec { 'install_docker':
    command => "/bin/yum -y install docker-engine"
}

7. On the Puppet Agent server, run this: puppet agent -t -d

How Do You Deploy Puppet Agent 3.x to a RedHat Linux Instance in AWS?

Update on 12/4/17
If you want to install Puppet Agent 5.x (a much newer version), see this posting instead.

Problem scenario
You installed Puppet Master on an AWS instance of RedHat Enterprise Linux. (See this link if you want to install Puppet Master). You now want another AWS instance of RHEL 7.x to be configured as a Puppet Agent node. You want to run a manifest to make sure that this new Puppet Agent node (i.e., server) is working with your Puppet Master server. What do you do to install Puppet Agent 3.x on a RedHat Linux server and configure it to work with Puppet Master?

Solution
This solution is for RHEL 7.3 instances in AWS. You could easily adapt this solution to other RedHat servers in different environments. It explains how to deploy Puppet and apply a manifest (in combination with these directions) from scratch.

Prerequisite
Using AWS change the Security Group so that the inbound rules will allow connections from the Puppet Agent server. One way of doing this is to find the internal IP addresses of the Puppet Agent server. This command should help you if it is run on a Linux server:

ip addr show | grep inet | grep -v 127.0.0.1 | grep -v inet6 | cut -c 10-24 | awk -F "/" '/1/ {print $1}'

Configure the relevant AWS Security Group allow an inbound connection from the IP address in the result above.

Here is a detailed explanation of how to create an inbound connection:

Go to Security Groups. Find the relevant security group and click the Inbound tab. Then click "Edit." Click "Add Rule." Then choose for "Type" in the dropdown menu "Custom TCP Rule." For "Port Range" choose 8140. For the "Source" drop down option, choose "Custom." Enter the internal IP address of the Puppet agent like this:

x.x.x.x/32

Substitute x.x.x.x with the internal IP address (as found with the above "ip addr show" command).

Procedures
On the Puppet Master server, do the following two steps:

1. Mentally decide how much RAM you want Puppet Master application to have. If you are not sure, try
run this command: cat /proc/meminfo | grep Mem

2. Create this file: /etc/default/puppetserver
In this file put a stanza like one of the following:

JAVA_ARGS="-Xms512m -Xmx512m"

JAVA_ARGS="-Xms1g -Xmx1g"

JAVA_ARGS="-Xms2g -Xmx2g"

3. As root run these three commands:

puppet master restart
hostname -f
ip addr show | grep inet | grep -v 127.0.0.1 | grep -v inet6 | cut -c 10-24 | awk -F "/" '/1/ {print $1}'

# Remember the IP address from the last command above. Disregard any trailing backslash "/" or numbers thereafter.

4. On the Puppet Agent server, modify the /etc/hosts file. It should have this stanza where x.x.x.x is the internal IP address from the Puppet Master server (the result of the last command ran in step #2):

x.x.x.x puppet

5. On the Puppet Agent node, run these two commands:

rpm -ivh https://yum.puppetlabs.com/puppetlabs-release-el-7.noarch.rpm
yum -y install puppet

6. On the Puppet Agent node update this file: /etc/puppet/puppet.conf
The last line of the [main] section of this puppet.conf file should have this stanza (where FQDNofPuppetMasterserver with the result of the "hostname -f" command ran on the Puppet Master server (in step #2)):

server=FQDNofPuppetMasterserver

7. Run this command on the Puppet agent node: puppet agent start

8. Go to the Puppet Master server. Run this command: puppet cert list

9. Assuming the above had some output such as puppet.agent.continualintegration.com, from the Puppet Master server run this command (but substitute puppet.agent.continualintegration.com with the FQDN that resulted from the command in step #7):

puppet cert sign puppet.agent.continualintegration.com

Alternative step #8: Assuming that step #7 showed no other servers that you do not want signed, run this command:

puppet cert sign --all

10. Test it.

a) On the Puppet Agent server, run

puppet agent --enable

b) On the Puppet Master server go to /etc/puppet/manifests/. Create site.pp with the following content (replace the FQDN):

exec { 'somethingneat':
command => '/bin/date > /tmp/continual.txt'
}

c) On the Puppet Agent node, run this: puppet agent -t -d

d) Check the /tmp/ directory for the file named "continual.txt."

How Do You Install Puppet Master on a RedHat Linux Server (an AWS Instance)?

Problem scenario
You want to install Puppet Master on RHEL 7.x running in AWS. You only have five minutes to complete this task. How do you install Puppet Master immediately?

Solution
1. Create puppetinstaller.sh in the /tmp directory. It should have the following lines (stop before you get to step #2):

#!/bin/bash
# Written by continualintegration.com

yum -y install ntp
echo '
server 0.us.pool.ntp.org
server 1.us.pool.ntp.org
server 2.us.pool.ntp.org
server 3.us.pool.ntp.org' >> /etc/ntp.conf

systemctl start ntpd

yum -y update
rpm -ivh https://yum.puppetlabs.com/puppet5/puppet5-release-el-7.noarch.rpm
# rpm -ivh https://yum.puppetlabs.com/puppetlabs-release-el-7.noarch.rpm
# To install a legacy version of Puppet (3.x), comment out the first rpm stanza and uncomment out the second.

yum -y install puppetserver
ln -s /opt/puppetlabs/puppet/bin/puppet /usr/bin/puppet

2. Run this command: sudo bash /tmp/puppetinstaller.sh

3. Verify it worked by running this command: puppet -V

How Do You Troubleshoot the Error “Failed to restart puppetserver.service: Unit puppetserver.service not found.”?

Problem scenario
You try this command from the Puppet Master server:

service puppetserver restart

But you get this error:

"Failed to restart puppetserver.service: Unit puppetserver.service not found."

What should you do?

Solution
Do not use "puppetserver" in the command. The correct command is:

service puppetmaster restart

How Do You Get PHP to Have a Separate Line for an Input Field in a Submit Form?

Problem scenario
In a PHP "submit form" you have various fields. Two of the fields are on the same row. That is, when you open the page with a web browser, two input box fields are the same distance from the top and bottom of the page. You want each field to be on its own line. How do you do you get the PHP form to render the way you want?

Possible solution
This problem can be caused by a missing ">".

Field3 and Field4 in this example below (<html> to </html>) would appear on the same row when the below is saved as a .php file and viewed with a web browser:

<form action="cool.php" method="post">
Field1: <input type="text" name="field1"><br>
Password: <input type="password" name="password"><br>
Field3: <input type="text" name="field3"<br>
Field4: <input type="text" name="field4"<br>

</body>
</html>

Note the lack of a closing ">" on the "Field3" line. On the Field3 line before the "<br>", there is no ">". This code below will put Field3 on a different line from Field4:

</body>
</html>

How Do You Install and Configure the AWS CLI on a Linux Server (without the pip Command)?

Problem scenario
Using different package managers can lead to dependencies being met while systems administrators will encounter error messages that detect, find or determine something to the contrary (e.g., false unmet dependency errors or conditions). Some enterprise policies disallow the pip command. How do you set up the AWS CLI on a Linux server without the pip command?

Solution (Part 1 and Part 2)
Prerequisites
i. You must have Python installed. Run this command: python --version
Make a mental note of the version if it is installed. You may want to try this too (if the command did not work): python3 --version

ii. This only applies if you are using CentOS/RHEL/Fedora. (Skip this prerequisite (ii and all of its sub-steps) if you are using a different distribution of Linux):
ii.a. Do this if you don't have Python: sudo yum -y install python3
ii.b. You need pip or pip3 to be installed. If you need help installing pip, see this posting.
ii.c. Run this command: sudo pip3 install colorama rsa s3transfer ruamel.yaml botocore # Use "pip3" instead of "pip" if you have "pip3". You may want to try sudo /usr/local/bin/pip if pip3 cannot be found.

iii. If Python is not already installed, run one of the following if you have Debian or Ubuntu Linux:
sudo apt -y install python3
or
sudo apt -y install python

Procedures
Part 1: Install AWS CLI
Step #1: If the AWS CLI has already been installed, skip to step #5. Otherwise using a web browser, go to the GitHub site to retrieve the .tar.gz file with the installation media. You may want to use curl -L <URL found above> > /tmp/nameoffile

Step #2: Click on the desired version (e.g., 1.18).

Step #3: Right click the hyperlink for "Source code (tar.gz)" and copy that hyperlink.

Step #4.a:  If you are running Debian or Ubuntu Linux, do this step, otherwise skip to step 4.b:
If you are running Python version 2.x, run this command:  sudo apt-get -y install python-setuptools
If you are running Python version 3.x, run this command:  sudo apt-get install python3-setuptools

Step #4.b:  Run these commands:
curl -L https://.../1.16.foobar.tar.gz > /tmp/1.16.foobar.tar.gz   #where https://.../1.11.foobar.tar.gz  is the hyperlink copied in the step above.
sudo mv /tmp/1.16.foobar.tar.gz /bin
cd /bin sudo tar -zxvf 1.16.foobar.tar.gz  # where foobar.tar.gz is the name of file downloaded from the above curl
cd aws-cli.version #where aws-cli.version is the name of the directory created in the above step
sudo python3 setup.py build   #This command should be with "python" instead of "python3" if you have Python 2.x only.
# *** (See below.)
sudo python3 setup.py install  #This command should be with "python" instead of "python3" if you have Python 2.x only.
# If you are running RHEL 8.x, run this command: sudo python3 setup.py install ../..
Run this: sudo ln -s /usr/local/bin/aws /bin/aws
sudo aws --version or aws --version # The response should indicate a version. If the command was not found, aws cli was not installed.

Part 2: Configure AWS CLI
If AWS CLI is already installed, start with step #5.

Step #5: You must have your default region name, AWS Access Key ID, and AWS Secret Access Key available. If you have them, skip the next steps and start working at Step #9. To retrieve the region name, log into the AWS console from a web browser. By default when you log in, the URL should have what you need. Find the portion of the URL that is to the right of the "=" sign. For example, "us-east-1" is to the right of the "=" sign: https://console.aws.amazon.com/console/home?region=us-east-1

Step #6: To find the AWS Access Key ID and AWS Secret Access Key, in the AWS console, click on your name in the upper right hand corner. Then click on "My Security Credentials."

Step #7: Click "Create New Access Key"

Step #8: Click "Show Access Key." Keep these values handy for the next step.

Step #9: Log into the Linux server with AWS CLI. Type this command followed by "Enter": aws configure

Respond to the first three prompts with something; the last of the four prompts can be blank and you can just press "enter."

AWS Access Key ID [None]:
AWS Secret Access Key [None]:
Default region name [None]:
Default output format [None]:

Step #10: To test, try this command followed by "Enter": aws ec2 describe-instances
This will test that you have connectivity to your AWS account, the AWS Access Key ID has permissions to list the EC2 (virtual) servers, and the AWS Secret Access Key was valid.

*** If commands above or below the three asterisks "***" throw an error about the version of setuptools being too low, try the command again. It may work a second time. This is only advisable if you are setting up a proof-of-concept or configuring an environment without sensitive data. For production or hardened environments, you may want to investigate as normal. If you need assistance installing the latest version of setuptools (and it may not be necessary, repeating the command may work as a kludge), see this posting. You may need to follow that posting's directions, then run the commands to build and install setup.py for setuptools in a repetitive way. We only recommend repeating the commands that failed the first time to make it work if you are not using sensitive data. There may be a more serious issue at hand.

In Case You Want Something Different
If you are using Debian/Ubuntu Linux and can install pip (or have pip installed), you can install the AWS CLI with this alternative directions.

Using Bash How Do You Replace a Line with a String with a Forward Slash in a File by Its Line Number?

Problem scenario
How do you insert a string with forward slashes using a perl command like this?

perl -i -pe 's/.*/Hello World/ if $.==15' foobar.txt

This command would insert "Hello World" on the 15th line of the text file named foobar.txt. Instead of "Hello World" you want something else inserted that happens to include forward slashes ("/"). You want to identify a line in a file by its line number. You want this line to be overwritten with a string that includes, among other characters, a forward slash. How do you do this?

Solution
Use "\" backslashes right before the forward slashes. Here is an example to insert cfg_dir=/usr/local/someprog/etc/hadoop into foobar.txt (line 15 with full deletion of line 15's previous content):

perl -i -pe 's/.*/cfg_dir=\/usr\/local\/someprog\/etc\/hadoop/ if $.==15' foobar.txt

# For your reference:
# The "/" to the left of the "cfg_dir" is for the command itself.
# The last "/" is for the perl command itself (not the input).

Using Bash How Do You Replace a Line in a File by Its Line Number?

Problem scenario
On your Linux server you know one line in a file needs to be replaced. You can identify the file by its line number. You have a string or pattern you want to be inserted where this line is. How do you do this with Bash?

Solution
To replace all the content of line 15 of foobar.txt with "Hello World" run this command:

perl -i -pe 's/.*/Hello World/ if $.==15' foobar.txt

For something more advanced, you may want to inject a string with forward slashes. Unix uses forward slashes in directory paths. Perl does not parse these forward slashes like other alphanumeric symbols. You have to use a back slash to have Perl place a forward slash in the manner described above. This example below will replace whatever is on line 34 of /tmp/contint.txt with "/subdirectory/path/to/file" with no quotes.

perl -i -pe 's/.*/ \/subdirectory\/path\/to\/file/ if $.==2' blah