How Do You Troubleshoot the Kubernetes Error “no nodes available to schedule pods”?

Problem scenario
You run a kubectl command, but you receive one of the following error messages:

Warning FailedScheduling default-scheduler no nodes available to schedule pods
No resources found

What should you do?

Possible Solution #1
Did the control plane lose connectivity with the worker nodes? An intermediate network device such as a firewall may have been implemented. Did a data center become unavailable?

Possible Solution #2 (if using AWS)
1. Create a node.
2. You may need to create an IAM role for Kubernetes; if you need assistance with this, see this posting. Once you have identified or created an IAM role that is suitable, proceed to step 3.
3. Add these policies to the role: AmazonEKSClusterPolicy, AmazonEKSWorkerNodePolicy, AmazonEC2ContainerRegistryReadOnly.

How Do You Create an IAM Role in AWS to Allow for Nodegroups to Be Created in EKS?

Problem scenario
In the AWS Management Console, you cannot add a Node to an EKS cluster. The "Node IAM Role" never has any option. You click the "refresh" arrow, but all you see is "No roles found. Follow the link above to create a new role." What should you do?

Solution

1. Install and configure the AWS CLI. If you need assistance with this, see this posting.

2.a. Create Test-Role-Trust-Policy.json like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

2.b. Create special.json like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
          "iam:AmazonEKSClusterPolicy",
          "iam:AmazonEKSWorkerNodePolicy",
          "iam:AmazonEC2ContainerRegistryReadOnly"],
      "Resource": "arn:aws:iam::1234567891011:role/contintdelete-role"
    }
  ]
}

3. Run commands like these (but replace "contintdelete-role" with the role name of your choice, and "DELETEPOLICY" with the policy name of your choice):

aws iam create-role --role-name contintdelete-role --assume-role-policy-document file://Test-Role-Trust-Policy.json
aws iam put-role-policy --role-name contintdelete-role --policy-name DELETEPOLICY --policy-document file://special.json

How Do You Back Up Many Emails from a Web-based Email Quickly?

Problem scenario
You want to back up many different emails from a web-based email. You cannot select several and print them all at once. What should you do?

Solution
Prerequisite
This assumes your web-based email can be configured to work with a desktop email client (such as Outlook).

Procedures
Install and configure Thunderbird, Outlook or an email client. You should be able to select many and go to "Save As" or "Print".

How Do You Get Kubernetes Nodes to Be Ready?

Problem scenario
Your Kubernetes cluster is not working. The nodes are not ready. You see your Kubernetes nodes are not ready (with kubectl get nodes). You also see this error from a kubectl describe node foobar command:

runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

What should you do?

(If you get a different error, see this posting How Do You Troubleshoot a Kubernetes Cluster That is Not Working at the Node Level?.)

Possible Solution #1
Run these two commands (as we found them to work with an EKS cluster):

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml

kubectl -n kube-system apply -f https://raw.githubusercontent.com/coreos/flannel/bc79dd1505b0c8681ece4de4c0d86c5cd2643275/Documentation/kube-flannel.yml

Possible Solution #2
You may want to see this posting if there is a problem with the subnet.env file on the worker nodes.

Possible Solution #3 (if using AKS)
Reboot the node. If that doesn't work, reimage the node. (The error message likely would have been different in the problem scenario for this solution to work. This solution was adapted from this external page.)

Possible Solution #4 (if you are getting an error about a network plugin)
See the posting How Do You Get Nodes in Kubernetes to Be Ready with a Network Plugin Error?.

How Do You Find out What CNI Plugin Has Been Installed in Your Kubernetes Cluster?

Problem scenario
You want to know what CNI plugin your Kubernetes cluster is using (e.g., Flannel, Calico, Weave Net, Romana or another one). What do you do?

Possible solution
"You can install only one Pod network per cluster." taken from Kubernetes.io.

Go to a worker node and run these commands and look at the output:

ls -lh /etc/cni/net.d

ls -lh /opt/cni/bin | grep -i flannel

ls -lh /opt/cni/bin | grep -i calico

Use the sudo find / -name command to search for Romana or Weavenet vestiges.

Try running this command:

kubectl get pods --namespace kube-system
#The results may indicate Flannel, Calico, Romana or Weavenet

How Do You Tell If the .yaml File for a kubectl Command Will Work?

Problem scenario
You want to do some pre-testing on the .yaml file(s) you will use with kubectl. How do you validate a .yaml file has correct syntax for Kubernetes?

Possible Solution #1
Try this command:
kubectl apply --validate=true --dry-run=true --filename=nameofyourfile.yaml

Possible Solution #2
Try this website:
https://www.kubeyaml.com/

Possible Solution #3
Try kubeval: https://www.kubeval.com/

Possible Solution #4
Use Copper: http://copper.sh/

How Do You Create a .kube/config File with a “aws eks” Command?

One of the following apply:

Problem scenario #1
You are using EKS and you have no config file in the .kube directory. You want to run some kubectl commands.

Problem scenario #2
You are using EKS and run a kubectl command, but you get this error: "The connection to the server localhost:8080 was refused - did you specify the right host or port?"

Solution
Run a command like this (but replace "us-west-1" with your region and replace "foobar" with the cluster_name):

aws eks --region us-west-1 update-kubeconfig --name foobar

How Do You Troubleshoot the Error ‘”/usr/share/collectd/types.db” no such file or directory’?

Problem scenario
You are running an aws ssm command. You receive a message about an error when parsing amazon-cloudwatch-agent.toml that also says '"/usr/share/collectd/types.db" no such file or directory'. What do you do?

Solution
Install collectd. If you are using a RedHat derivative of Linux (e.g., Fedora or CentOS), run this: sudo yum -y install collectd

How Do You Get Presto to Start when Errors about Presto requiring an Oracle or OpenJDK JVM?

Problem scenario
Presto will not run because of an error about the JVM.

You get a message like this:
"Java version has an unknown format: 15" (from "C" below)
"Presto requires an Oracle or OpenJDK JVM (found Ubuntu)" (associated with the Java version shown in B or D below)
"Presto requires an Oracle or OpenJDK JVM (found Private Build)" (associated with the Java version in E below)
"Presto requires an Oracle or OpenJDK JVM (found IcedTea)" (associated with the Java version shown in F below)

Solution
For detailed background information, here are six outputs of "java -version":

A.
openjdk version "1.8.0_265"
OpenJDK Runtime Environment (build 1.8.0_265-b01)
OpenJDK 64-Bit Server VM (build 25.265-b01, mixed mode)

B.
openjdk version "11.0.8" 2020-07-14
OpenJDK Runtime Environment (build 11.0.8+10-post-Ubuntu-0ubuntu120.04)
OpenJDK 64-Bit Server VM (build 11.0.8+10-post-Ubuntu-0ubuntu120.04, mixed mode, sharing)

C.
java version "15" 2020-09-15
Java(TM) SE Runtime Environment (build 15+36-1562)
Java HotSpot(TM) 64-Bit Server VM (build 15+36-1562, mixed mode, sharing)

D.
openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment (build 14.0.1+7-Ubuntu-1ubuntu1)
OpenJDK 64-Bit Server VM (build 14.0.1+7-Ubuntu-1ubuntu1, mixed mode, sharing)

E.
openjdk version "1.8.0_265"
OpenJDK Runtime Environment (build 1.8.0_265-8u265-b01-0ubuntu2~20.04-b01)
OpenJDK 64-Bit Server VM (build 25.265-b01, mixed mode)

F.
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (IcedTea 3.16.0) (build 1.8.0_252-b09 suse-3.35.3-x86_64)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

The word "ubuntu" appears in three of the six. Some versions of Presto lend themselves, as of September 2020, to Python 2. We found issues with Java version 15, SUSE and Ubuntu/Debian packaged Javas. Be believe that the details of the "version" are taken from the binary Java file. We do not think they can be modified or faked easily.

We recommend using a Red Hat derivative of Linux for deploying Presto or a newer version of Presto (if you use Debian, Ubuntu or SUSE). As of late September 2020, there is a version 0.241 that seems quite stable and another version 0.3x that seems acceptable too.