How Do You Troubleshoot the Kubernetes Error “cluster unreachable”?

Problem scenario
You get a message "kubernetes cluster unreachable" when running a helm, kubectl, az, or eks command. What should you do?

Possible solution #1
Has a router been reconfigured? Has a new firewall rule been imposed? Has a data center gone down that housed the cluster? Did you receive an email about a maintenance window or a configuration change?

Possible solution #2
What are the permissions of the relevant .yaml file? Can you try changing them? (Idea taken from https://github.com/k3s-io/k3s/issues/1126.)

Possible solution #3
If you were running a kubectl command, can you try to run commands like these?

sudo cp ~/.kube/config ~/.kube/bak.config.bak
kubectl config view --raw >~/.kube/config

(Idea taken from https://github.com/k3s-io/k3s/issues/1126.)

Possible solution #4
Warning: this may install something undesirable. Use this with caution.
If you were using helm, Rancher and K3s, can you try running a command like this?

sudo helm install harbor/harbor --version 1.3.0 --generate-name --kubeconfig /etc/rancher/k3s/k3s.yaml

(Idea taken from https://github.com/k3s-io/k3s/issues/1126.)

Possible solution #5
Are you using Rancher and K3s? Try running this:

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

(Idea taken from https://github.com/k3s-io/k3s/issues/1126 )

Possible solution #6
Can you ping the remote Kubernetes server? Are you not on a server that has access to the remote network? Is the cluster available from the network you are on? Are you running the command from a workstation that is not connected to a VPN?

Possible solution #7
Are you using GKE? If so, read this posting: https://forum.rasa.com/t/rasa-x-kubernetes-cluster-unreachable-error-using-github-actions/37073/8

Possible solution #8
Verify configuration files. If there are two clusters configured, can you try to only use one cluster? Try each cluster one-at-a-time. This may help you pinpoint the problem.

Possible solution #9
Run commands like these:

kubectl config view
kubectl config use-context CONTEXT-CHOSEN-FROM-PREVIOUS-COMMAND-OUTPUT
kubectl config --help

(This was adapted from https://stackoverflow.com/questions/49260135/unable-to-connect-to-the-server-dial-tcp-i-o-time-out.)


If you want assistance troubleshooting a network problem, see this posting.

Which Is a Higher Position, a Senior Engineer or a Staff Engineer?

Question
Which is a higher position, a Senior Engineer or a Staff Engineer?

Answer
Companies do not always conform to a job title convention or standard. However we find that the Staff Engineer title denotes a step above Senior. A Staff Engineer could be below a Principal Engineer however.

For further reading, see these postings (from which this posting was adapted):


How Do You Troubleshoot String Manipulation with List Slicing Not Working as You Expect?

Problem scenario
You are taking a slice of a Python string. You know that syntax like var_x[-2:] signifies the penultimate character and the last character of a string (called var_x). But you are only seeing one character -- not two. You expect to see two (or a different number of characters than what you are seeing). Why is the snippet printing out fewer characters than you expect?

Possible Solution #1
There could be an invisible "\n" attached to the string (e.g., at the end of the line). If you are using "readlines" to read an input file, there could be a "new line" or "line break" being processed as a single character. You may be seeing an incorrect number of characters in your slice/portion. If you are reading a file, and processing it line by line, you may want to use the .strip() function. This can remove the "\n" character that is getting picked up and ruining otherwise sound logic for your string manipulation task.

Possible Solution #2
There is an "if" or "for" loop or some logic in your program that is causing a special case to not process the way you are thinking about it. Sometimes a condition is met where a portion of code is entered or skipped, but we humans forget about this scenario. Use some intermediate print statements; this may be tedious, but you may be able to pinpoint the problem.

How Do You Install gRPC for Python on a Linux Server?

Problem scenario
You want to install gRPC and configure it for Python. How do you do this?

Solution
Prerequisites
You have installed pipe and venv. If you need assistance installing pip, see this posting for Ubuntu/Debian or this posting for a Red Hat derivative (e.g., RHEL, Stream, CentOS, Fedora) or this posting for SUSE. If you need help installing venv, see this posting.

Procedures
Run these commands:

python3 -m venv to_test
cd to_test
source bin/activate

Run this:

pip3 install grpcio

How Do You Get a Multi-line Variable in Python to Be Treated as Lines and Not Characters?

Problem scenario
When a number of lines of text are read from a file, the Python program sees each line as such. When a number of lines of text are in the program itself, that is you use a multi-line variable assignment in a .py program, Python sees the characters individually and does not distinguish between the different lines. The len() function reflects these differences. You want the Python program to read the text in the program as if it were read from the "with open" and "readlines()" function line-by-line -- not character-by-character.

For example, your source text could be like this in a standalone .txt file:

foo bar 123 456
cat dog 999 888
horse cow 333 777

When you read the .txt file in the Python program, each line of text is treated as a separate item in a list. In a Python program, you could have an assignment like this:

data1 = """foo bar 123 456
cat dog 999 888
horse cow 333 777"""

But the three lines would be treated as a single string -- not a list. You want data1 to be three lines of text and the len(data1) function to return 3. You do not want to have separate .txt files for the .py program to open. You want everything in a self-contained .py file.

What do you need to do?

Solution
Use .splitlines() and list(). Here is an example:

list(data1.splitlines())

What Is The Time Complexity when There Are Four Variables Used Arithmetically in a Program?

Question
You have a program with four variables that are used in arithmetic, but no other operations happen. (There are no while or for loops at all.) What is the time complexity of such an algorithm?

Answer
O(1). An algorithm, as long as the number of operations is fixed or constant, can be considered to have O(1) time complexity. The source is page 39 of Elements of Programming Interviews in Python.

How Do You Rotate Certificates?

Problem scenario
You have load balancers, web servers, and other HTTP technologies that rely on SSL or TLS certificates. Some certificates will be revoked for security reasons. How do you provision new certificates to update them (because they will expire or you want to harden your environment by refreshing the certs as aged certs are more likely to be compromised than young ones)?

Solution
You may want to plan for having lower capacity than normal and therefore schedule the time to do this in production during an off-peak time.

Possible Solution #1
You could pay an third party company that is a certificate authority for new certs.

Possible Solution #2
Use AWS Certificate Manager
https://aws.amazon.com/certificate-manager/

Possible Solution #3
Use Google-managed SSL certs
https://cloud.google.com/kubernetes-engine/docs/how-to/managed-certs

Possible Solution #4
If you use Azure Active Directory, you could try AD certificate based authentication. See this Microsoft posting for more information.

Possible Solution #5
Azure App Service certificates. To use one, see this: https://docs.microsoft.com/en-us/azure/app-service/configure-ssl-certificate

The benefits that are explained here: https://www.azuretechguy.com/azure-app-service-certificate

Possible Solution #6
Letsencrypt.org

Possible Solution #7
Install openssl and use it. If you need assistance using openssl, see these postings:

Possible Solution #8
If you use IoT and AWS, see this posting as Lambda functions can be readily leveraged to rotate certificates at scale.

Possible Solution #9
Use Cloud Conformity.

Possible Solution #10 (for CloudFront certs)
See this Amazon article.

Possible Solution #11 (for RDS certs)
See this Amazon article.

Possible Solution #12 (for Kubernetes)


To see how to rotate IAM keys, see this posting.

Should You Ask What the Salary Pay Range Is?

Problem scenario
You have applied for a job. You are curious what it can pay. Would the interviewer be upset if you asked what the expected pay range is for the position?

Solution
In some jurisdictions private employers must disclose the pay range. If you live in one of those places and you are asking for a remote position, it may be legally required for the employer to disclose it. You can then have an idea of what the job can pay. This can help you negotiate a higher salary or not waste time.

What is an Apache Web Server Container?

Problem scenario
You are reading about Apache Web Server containers. These are not related to Docker containers. You often see them near or tags. What are Apache Web Server containers?

Answer
They are portions of text that contain Apache web server configuration. (The source of this is https://httpd.apache.org/docs/2.4/sections.html.) The configuration includes directives often related to the VirtualHost in Apache web server. (To read what a VirtualHost in Apache is, see this posting.)

There are two basic types of containers. Most containers are evaluated for each request. The enclosed directives are applied only for those requests that match the containers. The <IfDefine>, <IfModule>, and <IfVersion> containers, on the other hand, are evaluated only at server startup and restart. If their conditions are true at startup, then the enclosed directives will apply to all requests. If the conditions are not true, the enclosed directives will be ignored.

https://httpd.apache.org/docs/2.4/sections.html

The containers can be individual files if their names end with ".conf" (according to https://wiki.centos.org/TipsAndTricks/ApacheVhostDir).

Containers can be at the bottom of the httpd.conf file to support different website domains (according to page 819 of RHCSA/RHCE Linux Certification Study Guide Sixth Edition by Jang (McGrawHill)).

See also What Is a Container in I.T.?