CONTINUAL INTEGRATION – Page 166 – A Technical I.T./DevOps Blog

How Do You Find the Instance Ids of Virtual Servers in AWS Using Python?

Problem scenario
You want to write a Python program to display the instance IDs of virtual servers in AWS. What do you do?

Solution

Prerequisites
This assumes you have installed Boto3. If you do not know how, see this posting. This assumes that you are running it on a server that has access to AWS (e.g., it has access to the internet).

Procedures
As a reference, you may go here to learn more about Boto's features. Use this program:

""" Usage instructions
1. replace "aaa111" with the aws_access_key_id of your account
2. replace "bbb222" with the aws_secret_access_key of your account
     If you are not sure how to find these credentials do the following:
     To find the AWS Access Key ID and AWS Secret Access Key, in the AWS console, click on your name in the upper right hand corner.
     Then click on "My Security Credentials." Click "Create New Access Key" Click "Show Access Key."
3. Replace us-west-1 with the region that you want to list Ec-2 instances for.
4. To run it, call it listids.py and run it with a command like this: "python listids.py"

"""

import boto3

ec2 = boto3.resource('ec2', aws_access_key_id='aaa111', aws_secret_access_key='bbb222', region_name='us-west-1')

print "Instance ID,             Instance Type"
instances = ec2.instances.filter(
    Filters=[{'Name': 'instance-state-name', 'Values': ['running']}])
for instance in instances:
    print(instance.id, instance.instance_type)

How Do You Fix the Problem in vi “E138: Can’t write viminfo file /home/cooluser/.viminfo!”?

Problem scenario
You save a file in vi. When you go to exit you get the error "E138: Can't write viminfo file /home/cooluser/.viminfo!"

The error appears to be benign. You can just press ENTER to continue. But you would rather not do this.

What should you do?

Solution
Find the group that this user should be associated with (e.g., "users"). Run these two commands but replace "jdoe" with the user having the problem and replace "users" with the group that the user should belong to:

cd /home
sudo chown -R jdoe:users jdoe

Now the problem should not happen any longer.

How Do You Troubleshoot SonarQube Not Starting Correctly and ce.Log Not Being Created Properly?

Problem scenario
You are running SonarQube. The service seems to start (based on the output of "sudo systemctl status sonar"). But based on an nmap command on your server's external IP address, you see no service is listening. You go to the logs directory and cannot find the ce.log file. You cannot reach the web UI for SonarQube. How do you fix this problem?

Possible Solution #1
Look at your sonar.properties file. Is the "sonar.web.host" stanza correct? Is there a typo in the hostname? Is the hostname configured in your DNS server or /etc/hosts file? A typo in this stanza, e.g., sonar.web.host=wrongservername, could cause the problem you experienced.

Possible Solution #2
An underpowered server may take time to rebuild indices. You may need to wait a few minutes. If you want to add RAM or CPU to your server, see this link if you are using AWS, or see this link if you are using GCP.

How Do You Troubleshoot the Kubernetes Error “Metrics not available or pod default/kubia-xxx-bmb”?

Problem scenario
You run "kubectl top pod" but you receive an error such as this:

'Metrics not available or pod default/kubia-xxx-bmb, age: 1hr10m... error: Metrics not available for pod default/kubia-xxx-bmb, age: 1hr10m"'

What should you do?

Updated on 9/2/19.

Solution
Wait five minutes. Then re-run this command: kubectl top pod

The source of this question and answer were taken from page 432 of Kubernetes in Action by Marko Luksa published in 2018 by Manning. For issues about metrics on pods, you may want to consider Prometheus or Datadog.

To learn more about Kubernetes in general, you may want to consider purchasing a book. Kubernetes is arguably the most in-demand skill with the greatest potential growth.

How Do You Troubleshoot the Ansible Message “ERROR! the role was not found”?

Problem scenario
You run ansible-playbook on a playbook (e.g., .yaml file) that uses a role. It fails with the message "ERROR! the role was not found". How do you correct this problem?

Solution
Find the ansible.cfg file. Look at the roles_path stanza. Add or modify the path that you want Ansible to look at for a role every time the Ansible playbook runs.

If you want a special ansible.cfg file (to not interfere with other users on your Ansible control host), copy one to the directory with the playbook. This will be used by default (when the ansible-playbook command runs). You can modify the roles_path as you desire without affecting other users. To learn more about the ansible.cfg file, see this link.

How Do You Install Memcached on an Ubuntu Server?

Problem scenario
You want to try out Memcached. How do you install memcached and php-memcached to test it?

Solution
(These directions were based on ServerMom.org directions.)

1. Run these commands:

sudo apt-get -y update
sudo apt -y install memcached php-memcached php7.0 apache2 # apache2 is not necessary but useful for testing purposes

2. Create this file:
sudo vi /var/www/html/info.php # add the three lines below

<?php
phpinfo();
?>

3. Run these commands:
sudo service memcached restart
sudo service php7.0-fpm restart

4. In /var/www/html/ create a file called contint.php. Use this as the content:

<?php
$mem = new Memcached();
$mem->addServer("127.0.0.1", 11211);

$result = $mem->get("blah");

if ($result) {
    echo $result;
} else {
    echo "This is the FIRST time that you downloaded this page!";
    $mem->set("This is not the first time you have downloaded this page.  This is held in memcached") or die("Couldn't save anything to memcached...");
}
?>

5. Test it out. Browse to the web page with a web browser (e.g., http://x.x.x.x/contint.php). Then click refresh. You should see the first message followed by the second message.

If you want to purchase a book on Memcached, click here.

How Do You Find the Most Recently Modified Files in Linux?

Problem scenario
Someone was doing a web conference and modified several files. You have forgotten what files she modified. How do you find the files that were modified most recently on a Linux server?

Solution
Run this command:

find $1 -type f -exec stat --format '%Y :%y %n' "{}" \; | sort -nr | cut -d: -f2- | head

# Taken from StackOverflow.com.

How Do You Troubleshoot a C Program That Prints a Warning Message like “expects argument of type ‘char*’, but argument 2 has type ‘int’ [-Wformat=]”?

Problem scenario
You compile a C program (with gcc foobar.cc) but you get an error message like this:

"foofbar1.cc:71...: warning: format '%s' expects argument of type 'char*', but argument 2 has type 'int' [-Wformat=]
printf ("y is a %s\n", y);
^ "

What do you do to not get this error?

Solution
While the code compiles, the executable may not run. A compilation error can be benign. For this case, integers need a special syntax to be printed out.

Here is a line that will produce this error (with a variable named "y"):

printf ("y is a %s\n", y);

Here is a similar but corrected line that will be error free:

printf ("y is a %d\n",y);

The "%s" is appropriate for printing string variables. For integer variables, use "%d" syntax.

For learning about the C programming language, you may want to purchase a book below:

How Do You Write an Ansible Playbook to Transfer a Compressed File and Unzip It without Using Linux Utilities?

Problem scenario
You have installed Ansible and configured it to work with other servers (managed nodes). You want to write an Ansible playbook that will work with Linux managed nodes with three requirements:

1. Transfer a compressed file from a web sever without wget.
2. Uncompress a .zip file without unzip.
3. Work without passwordless SSH authentication to the managed nodes.

How do you accomplish this with using Ansible-supported features?

Solution
1. Have the playbook use the get_url command to transfer the file. You do not need to use wget on the managed nodes. Here is an example of how the syntax should appear:

  get_url:
    url: https://www.continualintegration.com/good.zip
    dest: /tmp/good.zip

It can support cksum which is recommended for security reasons. If you want to learn more about this feature, see this posting.

2. Have the playbook use the "unarchive" command. You do not need to install unzip. Here is an example:

- name: Unzip a file on a remote server
  unarchive:
    src: /tmp/good.zip
    dest: /usr/local/bin
    remote_src: yes    # This tells Ansible the source file is on the remote server.  If the source file is elsewhere, remove this stanza.

To learn more about the unarchive feature, see this external link.

3. When running the "ansible-playbook" command, use the "--ask-pass" flag. For example if you want to run foo.yaml as the playbook, run a command like this:

ansible-playbook foo.yaml --ask-pass

This will prompt you for a password to SSH into the managed nodes.

What is GitOps?

Question
What is GitOps?

Answer
GitOps is a declarative method of managing infrastructure that is driven by committing code into a version control system. GitOps is an indirect, and usually imperative, approach to systems administration that gives a detailed history of past operations. GitOps is the practice of adding code to a Git repository to trigger an operational event (such as rebooting a server, creating a user account, or provisioning a new virtual server). Upon a successive file being uploaded to a specific location in Git, which may or may not require peer review to become complete and take effect, an action or set of actions will be taken. Each older version of the file will be saved in Git for future retrospection(s).

The term is a portmanteau of the software named Git and "Ops" which is short for [system] operations. Git itself was developed by Linus Torvalds and is the most popular distributed code versioning solution as of 2018. Some businesses may use GitOps with a different code versioning solution from Git itself. The term appears to have originated from Weaveworks.

GitOps was born of widespread automation and a need to audit systems administration (e.g., for security investigations or postmortems). As the popularity of the CI/CD pipeline has grown, triggering actions by uploading code has become a standard practice. GitOps is an extension of infrastructure-as-a-code -- using text to specify virtual servers' CPU, RAM, and hard disk capacities. Moreover GitOps is an outgrowth of event execution prompted by new files being uploaded into a code repository, often after one or two approvals of successful peer review. For root cause analysis, having detailed records in a centralized location can be of great value. Code versioning systems will continually track changes to configuration files as they are merged with the exact corresponding time and date. Pull-request detection can be set up to trigger actions and be the impetus of GitOps.

At the 18 minute and 46 second mark of this video presentation by Gene Kim, the DevOps guru states that the "top predictor of IT performance" was whether or not the Operations team was using a code version control system. In his book The DevOps Handbook says that in 2014 Puppet Labs found the same thing (on page 117). One factor in the "12 Factor App" methodology is having a single source code repository (https://12factor.net/). This methodology's adoption has increased lately, and this is right in line with ushering in adoption of GitOps with one exception: the twelve-factor app recommends environment variables never be stored in code repositories. It will be interesting to see how the industry reconciles GitOps and the twelve-factor app for environment variables.