CONTINUAL INTEGRATION – Page 98 – A Technical I.T./DevOps Blog

What is the AWS CLI Command to Determine the VPC ID?

Problem scenario
You are now administering an AWS account. You have the AWS CLI installed and configured. How do you find out what the VPC IDs are?

Solution
Run this command:

aws ec2 describe-vpcs | grep VpcId

How Do You Use a New IAM User Account?

Problem scenario
You want to assume an IAM user from a command line of an EC-2 instance. What do you do?

Solution
Prerequisite
This solution requires you have the access key and its secret key in text.

Procedures

Run this command: aws configure
Respond to the next two prompts accordingly (so that it will be configured to use the IAM user account of your choice). You may want to view an article about creating a new IAM user.

How Do You Install Django on a CentOS/RHEL/Fedora server?

Problem scenario
You want to install Django and have it working for browsing from a desktop workstation. You do not want to leverage Django's API functionality. How do you install and configure Django from scratch?

Prerequisite
pip3 is probably installed. If you are not sure, you could run this: sudo find / -name pip3

You must have pip or pip3 installed. If you need assistance, see this posting.

Procedures

Run these two commands:

sudo yum -y install python3 mod_wsgi httpd
sudo python3 -m pip install Django

Find where the core subdirectory of Django is with this command:
sudo find / -name core -type d | grep django
The parent directory, of the "core" directory found above, should have permissions configured such that the user who will start and stop Django services has the ability to write to it. Here is an example (but your path and method may vary):

sudo chown -R ec2-user:ec2-user /usr/local/lib/python3.6/site-packages/django/

Run this command but change "contint" to the name your project:

django-admin startproject contint

Find the external IP address of the server with this command: curl icanhazip.com
Back up the settings.py file that was created from the django-admin command. Find it (e.g., with sudo find / -name settings.py | grep contint). After it is backed up, change the original.
Edit the file (with the details in the step below). Here is an example of how to edit it:
vi /usr/local/lib/python3.6/site-packages/django/mysite/mysite/settings.py
Find the ALLOWED_HOSTS stanza. Change it to look like this but replace "x.x.x.x" with the external IP address of the server:

ALLOWED_HOSTS = ['x.x.x.x']

Add or keep the single quotes inside the brackets. Save the changes and exit the file.

Start the Django server (you may or many not need to change the path to the manage.py file):

python3 /usr/local/lib/python3.6/site-packages/django/contint/manage.py runserver 0.0.0.0:4466

Draft this URL, but replace x.x.x.x with the external IP address of the server: http://x.x.x.x:4466
Open a web browser and go to the URL.

How Do You Install Erlang on RHEL 8.x?

Problem scenario
You are using RHEL 8.x. You want to install Erlang. What do you do?

Solution
Run these two commands:

curl -Ls https://github.com/rabbitmq/erlang-rpm/releases/download/v22.3.2/erlang-22.3.2-1.el8.x86_64.rpm > /tmp/erlang-22.3.2-1.el8.x86_64.rpm

sudo rpm -i /tmp/erlang-22.3.2-1.el8.x86_64.rpm

How Do You Troubleshoot “error occurred (AccessDeniedException) when calling the CreateCluster operation: User: … is not authorized to perform: iam:PassRole on resource:…”?

Problem scenario
You run this command:

aws eks create-cluster --name contint --role-arn arn:aws:iam::12345678910:role/contintrole --resources-vpc-config subnetIds=subnet-a123456,subnet-b77777777,securityGroupIds=sg-2e324234254

You get this error:

An error occurred (AccessDeniedException) when calling the CreateCluster operation: User: arn:aws:iam::12345678910:user/contintuser is not authorized to perform: iam:PassRole on resource: arn:aws:iam::12345678910:role/contintrole

What should you do?

Solution
1. Log into the AWS Console.
2. Go to IAM -> Users
3. Click on the user in question
4. Click on "Add Inline Policy"
5. Add this policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iam:GetRole",
                "iam:PassRole"
            ],
            "Resource": "arn:aws:iam::12345678910:role/contintrole"
        }
    ]
}

6. Click "Review Policy"
7. Enter a name
8. Click "Save policy". Wait 10 minutes for the changes to take effect.

How Do You Troubleshoot the AWS CLI Message “An error occurred (AccessDeniedException) when calling the ListClusters operation”?

Problem scenario
You run this command aws eks list-clusters, but you get this message:

"An error occurred (AccessDeniedException) when calling the ListClusters operation: User: arn:aws:iam::12345678910:user/jdoe is not authorized to perform: eks:ListClusters on resource: arn:aws:eks:us-west-1:12345678910:cluster/*"

What should you do?

Solution
1. Go to the AWS Console and log in.
2. Go to IAM -> Users
3. Click on the user's account.
4. Click on "Add Inline Policy"
5. Click on the JSON tab
Add this content:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "ekscontintadmin",
            "Effect": "Allow",
            "Action": "eks:*",
            "Resource": "*"
        }
    ]
}

6. Click "Review policy"
7. Give a name to it
8. Click "Save policy". Adding an inline policy to a role in AWS IAM can take 10 minutes to take effect. (It is not like Security Groups.)

How Do You Troubleshoot the EKS Error “AccessDeniedException…Clusters operation: Account …is not authorized to use this service”?

Problem scenario
In one region, but not another, you get this error with an "aws eks list-clusters" command:

"An error occurred (AccessDeniedException) when calling the ListClusters operation: Account 12345678910 is not authorized to use this service"

You know that IAM does not require region selection. What is causing this eks error?

Solution
Is your AWS CLI configured to use a region that eks does not support? To find out, try changing the region to us-east-1 (e.g., by running the "aws configure" command and responding to the prompts). If you change it to "us-east-1" and do not see the above error messages when you run the command(s) that created those messages, you should be ok.

How Do You Troubleshoot the Python Error “json.decoder.JSONDecode..”Expecting property name enclosed in double quotes…”?

Problem scenario
You are running a Python program that uses "import json". You get this error: "json.decoder.JSONDecode.."Expecting property name enclosed in double quotes…" You are not allowed to use bson, but you can use other Python packages. What should you do?

Solution
This solution only works if you can eliminate single quotes in the content to be serialized (or translated into JSON). This will get rid of single quotes indiscriminately.

Where "foobar" is the variable with the string you want to turn into JSON, do this:

import re, json
modifiedfoobar = foobar.replace("'", '"')   # (1)
latestagefoobar = json.dumps(modifiedfoobar)
finalfoobar = json.loads(latestagefoobar)

# (1) this eliminates single quotes. This may cause you problems if single quotes were part of the necessary content.
# If the string had single quotes instead of double quotes, this will save you from installing/importing bson.
# Often double quotes can be used instead of single quotes, but not always. If they are not interchangeable cannot, you may need bson.

How Do You Copy a File into an S3 Bucket using the AWS CLI?

Problem scenario
How do you copy a file from a Linux server with AWS CLI to an S3 bucket?

Solution
Draft a command like this but replace /tmp/good.txt with the path and name of the source file you are copying, and replace "foobar" with the name of the s3 bucket you are copying to (as the destination):

aws s3 cp /tmp/good.txt s3://foobar/

Run the command.

How Is Traffic Routed when a Browser Goes to a URL and the Website is Powered by a Kubernetes Cluster?

Question
You have been asked to explain how an external request for a website is routed and ultimately fulfilled by a pod in Kubernetes via an ingress controller. When a web browser downloads a website via HTTP and the website is running from a Kubernetes cluster, how does an individual container provide the HTML and/or data? How is external traffic routed to an underlying pod in a Kubernetes Cluster?

Answer
Assuming that external traffic can reach an ingress controller in Kubernetes, the short version of the answer may be the way ingress controllers are configured. (The long answer, or any answer, should account for variation in the process as there is not one way traffic is routed to a pod in a Kubernetes cluster.)

There are three types of HTTP routing: host-based, path-based and header-based (according to https://dzone.com/articles/the-three-http-routing-patterns-you-should-know). We think header-based routing with Kubernetes is exceptionally rare based on this SO posting. If you use an ALB with Kubernetes, you can use host-based routing (according to this Amazon posting).

A browser will make a GET request for a URL. The data will be retrieved via a DNS server based on the top-level domain (e.g., .com of the URL). The request will be routed to the appropriate domain (e.g., continualintegration). The next intermediate step would be to land on a load balancer or reverse proxy.*

External traffic will then go to either an Ingress or a Service of Kubernetes. Services pass the traffic to an Endpoints -- not to pods directly (according to page of 325 of Kubernetes in Action). (Technically the Endpoints resource should be plural.) Endpoints are separate resources (as opposed to being subcomponents of Services) according to page 133 of Kubernetes in Action.**

Ingress resources may use services without forwarding the traffic to the Service; Ingress controllers select pods to fulfill HTTP requests (according to page 145 of Kubernetes in Action). Ingress controllers can expose external traffic to underlying Kubernetes services themselves or send external traffic directly to pods and bypass Services (pages 143 through 146 of Kubernetes in Action and this). (An ambassador can also send traffic directly to pods and bypass kube-proxy on a worker node according to this external site.) Kubernetes Services can be available for external traffic without an Ingress forwarding such traffic (page 140 of Kubernetes in Action). To read about the advantages of a Service over an Ingress resource, you may want to view this posting.

Ingress controllers can be externally accessible with a public IP address (without a load balancer or reverse proxy). DNS can facilitate the resolution of an FQDN. During a request to a website over HTTP, once the request gets to the ingress controller, the traffic will be directed to a service either via a label selector or an IP address.

Assuming ingress controllers are used in Kubernetes, and assuming there was YAML in their creation, the "rules" stanzas in the "spec" section of the YAML will designate host stanzas (for routing to subdomains) and corresponding specific paths which map to backend services (a list of one or more) that are the target "services" that will fulfill the request.

Here is a YAML example for an ingress controller with label selectors that will route traffic destined for certain hosts (either continualintegration.foobar.com or weird.continualintegration.com) to corresponding services (either service1 or service2).

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-host
spec:
  rules:
  - host: "continualintegration.foobar.com"
    http:
      paths:
      - pathType: Prefix
        path: "/bar"
        backend:
          service:
            name: service1
            port:
              number: 80
  - host: "weird.contintualintegration.com"
    http:
      paths:
      - pathType: Prefix
        path: "/foo"
        backend:
          service:
            name: service2
            port:
              number: 80

Services can either have labels or IP addresses. It is possible to create a Service with no valid Endpoints resource (as explained here), but it would likely not be useful. Endpoints are separate Kubernetes resources (according to page 133 of Kubernetes in Action). Endpoints can be created manually if you do not create Services with labels. Routes to Endpoints can be done via IP addresses or label selectors that a Kubernetes service is configured with (pages 102 and 105 of Kubernetes Patterns by Bilgin Ibryam and Roland Huß (O'Reilly). Copyright 2019 Bilgin Ibryam and Roland Huß, 978-1-492-05028-5). Endpoints define what will be exposed in a pod (according to the front cover of Kubernetes in Action).

Internal networking in a Kubernetes cluster (e.g., among the worker nodes) is done via a Container Network Interface plugin (as opposed to Network Address Translation).

Load balancers can send traffic to different endpoints; load balancers require hardware to implement (according to page 114 of Kubernetes Patterns by Bilgin Ibryam and Roland Huß (O'Reilly). Copyright 2019 Bilgin Ibryam and Roland Huß, 978-1-492-05028-5). Pods can become available or unavailable based on readiness probes. kube-dns and kube-proxy are components that play a role in internal traffic routing inside a Kubernetes cluster. kube-proxy on each worker node helps balance the traffic load among the containers providing a given service (page 21 of Kubernetes in Action).

To read more about networking and Kubernetes, see this external page or this Dzone page. To learn more about how internal Kubernetes traffic, see this posting. For networking specific to GKE specifically see this Google documentation posting.