How Do You Create a File on a Server with Terraform?

Problem scenario
You are trying to get user data to create a file on a server in Terraform. It is in a directory that requires sudo privileges. You use the "sudo" command in the Bash script. The Bash script executes except a file is never transferred. How do you get Terraform to copy a file to a new server?

Solution
Use the Terraform-supported cloud_config option instead of having a Bash script transfer the file.

Here is the content of the file:

provider "aws" {
  access_key = "AKIAabcd1234"
  secret_key = "secret/foobar"
  region     = "us-west-1"
}

resource "aws_instance" "example" {
  ami           = "ami-1234"
  instance_type = "t2.large"
  key_name      = "good_key"
  user_data     = local.cloud_config_config
}

locals {
  cloud_config_config = <<-END
    #cloud-config
    ${jsonencode({
      write_files = [
        {
          path        = "/etc/yum.repos.d/fun.repo"
          permissions = "0644"
          owner       = "root:root"
          content     = file("sourcefile.repo")
        },
      ]
    })}
  END
}

How Do You Do a regex Operation in Python without Importing a Module?

Problem scenario
You need a pattern matching function in a Python program, but you cannot use "import re". What should you do?

Possible Solution #1
Use the index() function.

Here is an example:

foobar = "abcdefghijklmnopqrstuvwxyz"
print(foobar.index('jk'))

Possible Solution #2
Use the .starswith() function.

foobar = "abcdefghijklmnopqrstuvwxyz"
print(foobar.startswith('abc'))

Possible Solution #3
Use the find() function.

foobar = "abcdefghijklmnopqrstuvwxyz"
print(foobar.find('z'))

Where Are The SSH Logs in Fedora?

Problem scenario
You are trying to find the SSH logs in Fedora. Where should you look?

Possible solution #1
/var/log/secure/

Possible solution #2
Check these locations:
/var/log/ssh/
/var/log/auth.log

Possible solution #3
/var/log/messages
(Many other applications write to this file besides SSH.)

Possible solution #4
They could be stored in binary format in /var/log/journal/ …

Run this command: journalctl -u sshd
Look at the output. Find an uncommon pattern/string in the output, say "foobar", and run a command like this:
grep -R "foobar" /var/log/

What is the Difference between a Data Structure Container and a Virtual Environment Container?

Question
You have heard about different types of containers and seek a disambiguation. Can containers be disambiguated?

Answer
One type of container is a virtual environment like Docker. Another type of container is a data structure of a programming language such as an array, set, list, tuple, or dictionary. These different types of collections of data are called containers.

What Are cgroups?

Question
You know cgroups provide isolation (along with namespaces in the context of containers). But what exactly are cgroups?

Answer
cgroups enforce hard limits of subsystems to allow efficient allocation of resources of the super-system.

Cgroups allow you to allocate resources — such as CPU time, system memory, network bandwidth, or combinations of these resources — among user-defined groups of tasks (processes) running on a system.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/resource_management_guide/ch01

cgroups apply limits to resources (according to https://www.linuxjournal.com/content/everything-you-need-know-about-linux-containers-part-i-linux-control-groups-and-process).

To read why it is inadvisable to change cgroups settings for Docker, see this posting.

Namespaces allow for identically named resources on a subsystem (e.g., a Docker container) to be used without interfering with other subsystems. To read more see this:

What Are Namespaces?

How Do You Troubleshoot Connectivity over Port 80 when You Know It Is Listening?

Problem scenario
A server is hosting a website. On the server, nmap -Pn x.x.x.x is showing port 80 is listening on the server itself. From another server this nmap -Pn x.x.x.x command is showing no ports (or a subset of the ports) are listening. What is the cause of this?

Possible Solution #1
There is a firewall on the webserver that is causing this. See the posting how to check if a firewall is on.

Possible Solution #2
See the posting How Do You Handle a Discrepancy between NMAP Results?

Possible Solution #3
There is an intermediate firewall between the two servers filtering traffic. Use traceroute for clues or ask a security/network person responsible for such a device.

Why is a Kubernetes Pod’s Status “Pending”, and What Can Be Done about It?

Problem scenario
You have a Pod that you want to run on a worker node. Its status is "Pending." What can you do to get the Pod to proceed (and what are some common root causes for this problem)?

Background
A common root cause is that the nodes have insufficient resources for the Pod (e.g., insufficient CPU or memory).

Possible Solution #1
Wait. It may eventually get scheduled to a worker node when there are sufficient nodes available.

Possible Solution #2
Can you change or remove a taint on a Pod? Can you add a toleration to the relevant YAML file for deploying this pod? Taints without tolerations can prevent nodes from being available to power a Pod. If the pod is a stateful set, it may need a certain type of disk (e.g., solid-state disk); therefore if the nodes have older hard drives, they won't be available for the scheduling. Is this possibly your problem? If a PersistentVolumeClaim is involved, the PVC may need to be deleted (according to this external posting).

Possible Solution #3
Can your other deployments/jobs have lower requests values for CPU and memory? Can your other deployments/jobs have lower limits values for CPU and memory?

Possible Solution #4
Are you using hostPort? Stackify.com says that this can contribute to a Pod reminaing in "pending" state. They say "[b]inding a pod to hostPort means limited areas for scheduling. However, it’s pointless using a service object to expose the pod.'

Possible Solution #5
Can you add a server as a worker node to the cluster? Or can you add CPU (see this pOsTiNg to resize a VM in the public cloud) or RAM (see this posting) to the node?

Possible Solution #6
Run this command:

kubectl get pods --field-selector=status.phase=Pending

Can you follow that up with a "kubectl describe pod foobar" (where "foobar" is the name of the Pod you want to investigate)? (This solution was adapted from this posting on the Datadog website.)

Possible Solution #7
Can you look in Splunk, Datadog or your centralized logging device to see if there is a "FailedScheduling" event that may provide details, hints or clues as the to problem.

Idea adapted from the Datadog website.

Possible Solution #8
When you "specify nodeSelector and no node matches label then pods are in pending state as they don't find node with matching label." (This quote was taken from https://www.bigbinary.com/blog/scheduling-pods-on-nodes-in-kubernetes-using-labels.)

You can remove a "nodeSelector" and "nodePool" stanza lines from the YAML file or you can label nodes with a "kubectl label nodes foo nodePool=barcluster" (where the node label is "foo" and the nodePool is named "barcluster"). This solution was adapted from StackOverflow.

Possible Solution #9
Are there affinity rules in place?

Anti-affinity rules can cause this problem because Pods will attempt to avoid certain nodes; to learn more about one such situation, see this posting: https://github.com/hashicorp/consul-helm/issues/243

Possible Solution #10
Could one of your nodes be not ready? Can you run "kubectl get nodes"? If some are not ready, see this posting.

Possible Solution #11
Is a deployment or rolling update happening? If one is and there a new PodDisruptionBudget was implemented, that may constrain resources available for scheduling. To learn more, you may want to see this:
https://kubernetes.io/docs/tasks/run-application/configure-pdb/#specifying-a-poddisruptionbudget

If you know the name of the deployment, to see the status of it and learn more details about how it is progressing, run this command (but substitute "foobar" with the name of the deployment in question):
kubectl describe deployment foobar

Possible Solution #12
Can you run "kubectl get events" or "kubectl logs"? These may help you determine the root cause.

Possible Solution #13
Are you 100% sure you are looking at the right pod? If you have multiple namespaces, the pod could be in an incorrect namespace. Have you run "kubectl get pods --all-namespaces" to be sure you are looking at the correct Pod?

Is It a Best/Recommended Practice to Use Branching with Version Control Systems?

Problem scenario
You are debating on using branching with your repositories or using trunk-based development (with virtually no other branches). Is it a best practice to develop on the mainline (with branching infrequently if ever)?

Solution
Maybe. It is not clear.

In Support of the "Yes" / You Should Use Branches
In practice many companies use Git Flow or other branches in their Git repositories.

Perforce is a very well-respected code versioning product. Its website says that a best practice for branching is to "protect your mainline." It says "…all commits have to be high quality." (See its source article for context.)

This qualifier "high quality" is a safe bet. But if all commits are of a "high quality", it is doubtful the your code would end up being merged to the mainline (or trunk) frequently. Moreover the existence of a best practice for branching strategies in an article that says "[u]se our version control branching best practices…" suggests branching can be a best practice. To learn more about Perforce's best practices for version control, see this posting.

This website says deciding on a branching policy is a best practice; therefore we interpret it as saying using branches is a best practice. Another source recommends using branching (for learning) is this website.

In Support of the "No" / You Should Not Use Branches
The book Continuous Delivery recommends developing on the mainline or trunk-based development. The DevOps Handbook also recommends not having branches in a code repository.

For an in-depth article on branching strategies and trunk-based development (that tends to shun branches), see this posting.