In Kubernetes, What Happens when an API Server or Admission Controller Receives a kubectl Command and Updates etcd?

Problem scenario
You know that an admission controller in the API server has various stages to process a kubectl command in Kubernetes. You want to know how the API server or relevant admission controller works in the correct sequence. The sequence starts with a request (HTTP POST from a kubectl command) and ends with updating etcd. How does this happen in detail?

Solution
The API server's process of handling a kubectl command is very similar to what are called "admission controller phases."*

It is not clear what is the difference (if there is any).

In pure text, the successful process is as follows:

1) a kubectl command is run
2) an HTTP POST request is made to the API server
3) the API server receives the request and handles it
4) an authentication plugin receives the request and passes it on
5) an authorization plugin receives the request and passes it on
6) an admission control plugin receives the request and passes it on
7) object schema validation is done
8) a validating admission control phase may happen that could involve an external validating webook (possible step, but could be bypassed). No changes can happen at this stage; either acceptance or rejection could happen here*.
9) the changes are saved in the "distributed persistent storage" (page 310 of Kubernetes in Action) system (etcd).
10) a response is returned to the client terminal that ran the original kubectl command.

The above steps were adapted from page 316 (most of the above steps) and 318 (#10) of Kubernetes in Action and this external website (several steps were corroborated or borrowed).

* This external article refers to "admission controller phases."

What Does “Branch by Abstraction” Entail?

Problem scenario
You have read about "branching by abstraction," but you do not know what it means. You read that it is ideal when the coding you are doing cannot be done incrementally. How do you "branch by abstraction"?

Solution
It is a method of trunk-based development; instead of using a branch, you commit to the trunk (or main) branch of your repository. As you modify and develop a solution, you interact with an abstraction -- not the production code. To branch by abstraction, do the following:

  1. Create an abstraction over the part of the system that you need to change.
  2. Refactor the rest of the system to use the abstraction layer.
  3. Create a new implementation, which is not part of the production code path until complete.
  4. Update your abstraction layer to delegate to your new implementation.
  5. Remove the old implementation.
  6. Remove the abstraction layer if it is no longer appropriate.

(This solution was taken from page 350 of Continuous Delivery by Humble and Farley.)

See also these links:

To learn more about branching strategies in general, see this posting.

Is It a Best/Recommended Practice to Use Log4j?

Problem scenario
You are not sure if Log4j is acceptable to use. It is a best/recommended practice to use Log4j?

Answer
Maybe.

Log4j has been well-adopted by many of the most trusted companies in the I.T. industry. According to cybersecuritydive.com "Fortinet, IBM, Microsoft, Red Hat, Salesforce, and Siemens" use(d) Log4j. Log4j has been vulnerable since 2013 (according to this external website).

Below are some examples of credible sources referring to "best practices" and using Log4j:

Carbonite says it is a best practice to "stay current" (in this external posting). Many sources say it is a best practice to apply maintenance patches regularly and stay current; these sources include Faronics.com, The C2 Group, the U.K. National Cyber Security Centre, Forbes recommends covering "[a]ll [y]our [b]ases." Staying current was not enough for Log4j. Some companies used such an old version of Log4j -- in contradiction of the "best practice" of staying current. (Legacy software is widely used; if you want proof that Log4j version 1 was still being used after the major December 2021 vulnerability, see this thread https://github.com/apache/logging-log4j1/pull/18.) These companies using version 1 were not susceptible to the recent Log4j vulnerability. (Admittedly there is a weakness to version 1, but it appears to be less of a vulnerability according to https://www.slf4j.org/log4shell.html.)

As far as vulnerabilities are concerned, CVE-2021-44228 is probably as bad as it gets.
…
As log4j 1.x does NOT offer a JNDI look up mechanism at the message level, it does NOT suffer from CVE-2021-44228.

https://www.slf4j.org/log4shell.html

What is interesting is that https://www.slf4j.org/log4shell.html estimates that 10 times as many companies use version 1, which has been unmaintained since 2015, than version 2. Therefore very few companies use the "best practice" of switching to version 2. Arguably the market got it right by not adopting the Log4j version 2.x. (There is a book explained a story about investors guessing which company was responsible for the component that made the Challenger fail. The estimates were based on the share price of the possible contractor companies losing market value at the time of the explosion. At the conclusion of the investigation, the company that lost the largest percentage of value on the day when the Challenger blew up was the company that made the part that caused the failure. This book is called Wisdom of the Crowds. We are not necessarily endorsing the book, but it is an interesting concept.)

According to CRN, these security vendors were using Log4j version 2:
Broadcom, CyberArk, ForgeRock, F-Secure, Okta, SonicWall and Sophos. (This was taken from https://www.crn.com/slide-shows/security/12-cybersecurity-vendors-susceptible-to-the-log4j-vulnerability.)

Were these companies using best practices? Do not many people in the industry trust those companies? Some companies do not worry about peer-review and best practices. They are focused on multiple lines of defense.

Even Symantec products were affected:

Broadcom determined as of Tuesday that some or all versions of its CA Advanced Authentication, Symantec PAM Server Control, Symantec SiteMinder, and VIP Authentication Hub products are affected by the Log4j vulnerability.

https://www.crn.com/slide-shows/security/12-cybersecurity-vendors-susceptible-to-the-log4j-vulnerability

The vulnerability reminds us that the term "best practice" suggests there is a clear alternative toward something safe but some situations have no clear course of action even to the most well-respected companies in the world.

Is It a Best Practice to Normalize a SQL Database?

Problem scenario
There are sources that recommend database normalization as a best practice for relational databases. Is it always a best practice to normalize databases?

Answer
There is no clear answer as the sources vary.

A heavily voted-up answer on StackOverflow says that denormalization for OLAP performance is something to be avoided. We find recognition of this "best practice" on a venerated engineering blog:

Keeping data normalized is considered a best practice in MySQL. As we mentioned before, however, when using a NoSQL database like Cassandra, denormalizing data often improves query performance.

https://engineeringblog.yelp.com/2016/08/how-we-scaled-our-ad-analytics-with-cassandra.html

The above sources have legitimacy, but authoritative sources dispute that normalization is a best practice. No one should dispute that there are proponents and opponents of normalization of SQL/relational databases as a best practice.

The opponents of normalization as a best practice say that you need to have denormalized data for OLAP databases (for reporting to reduce the expensive operation of a JOIN in SQL, and to avoid aggregate queries which can be time-intensive) and for datawarehouses. Datawarehouses are usually not updated that frequently; it is common for batch jobs to happen monthly with datawarehouses. The work to normalize such a database may be economically prohibitive, and the benefits may be negligible or have a negative value.

Kimball cubes are widely used in the I.T. industry; they have changed the database and BI verticals. The man behind the namesake of Kimball cubes says this on his company's website:

In general, dimensional designers must resist the normalization urges caused by years of operational database designs and instead denormalize the many-to-one fixed depth hierarchies into separate attributes on a flattened dimension row.

https://www.kimballgroup.com/data-warehouse-business-intelligence-resources/kimball-techniques/dimensional-modeling-techniques/denormalized-flattened-dimension/

NetFlix and Meta (formerly known as Facebook) both use denormalization. A Microsoft MVP says that a datawarehouse best practice is to denormalize data.

In a dataware house the best practice is to denormalize the data to increase the performanze [sic] of ad-hoc reporting; with the prize of a higher data volume.

https://social.technet.microsoft.com/Forums/en-US/8c54d1d2-5fbd-4b62-a07f-16c34a863668/is-denormalization-best-practices?forum=transactsql

Other examples suggesting denormalization is beneficial (and indicated a "best practice") are the following:

If you want to read about the controversy of "best practices" in general, we recommend this posting.

What is the Difference between Yarn, the Package Manager, and YARN, the Hadoop Framework Tool?

Problem scenario
You are familiar with two different software tools called Yarn. You are not sure what they do or how they are different.

Solution / Yarn Disambiguation
There is a package manager that has some project manager functionalities called yarn. It often works with JavaScript.

This tool has an icon of a one-line drawing of a cat; the original Yarn website had other cat imagery, and there is the word "cat" in a yarn command. Our theory is that cats like to play with balls of yarn. This type of play can get out of control. Package managers make disparate and complex problems manageable. We do not think the etymology of this tool was that of a backronym.

We don't think resource negotiation is a factor or function of this Yarn. The product became quite big. This tool was released in 2016 (according to https://github.com/yarnpkg/yarn/releases/tag/0.2.0).

To learn more about it, see these postings:

Be advised we do not agree with the Wikipedia statement that this Yarn tool is an acronym.

To learn about the CLI, see this posting.

There is a separate tool called YARN that that is associated with Hadoop. Hadoop YARN is "[a] framework for job scheduling and cluster resource management" (as taken from https://hadoop.apache.org/).

The term YARN stands for "Yet Another Resource Negotiator" (according to TechTarget.com).

This tool was released in 2012 (according to Cloudera); it is part of the Hadoop framework. There is a GUI component to this tool as well as a command line component. To learn about the commands, see this external posting. To learn about the GUI, see this posting.

Products have been built on Hadoop YARN such as Tez; Tez is "[a] generalized data-flow programming framework, built on Hadoop YARN…" (according to https://hadoop.apache.org/).

To learn more about the sheer differences between the Yarn tools, see these postings:

https://stackshare.io/stackups/yarn-vs-yarn-hadoop
https://stackoverflow.com/questions/44739402/hadoop-yarn-vs-yarn-package-manager-command-conflict
https://superuser.com/questions/1193406/hadoop-yarn-yarn-npm-name-conflict
https://github.com/yarnpkg/yarn/issues/2337

Is It a Recommended Practice to Use Production Data when Testing Software?

Problem scenario
You want to test your application. You are considering using mock/fake data. But you are concerned that the tests will be insufficient. Is it a best practice to test with production data?

Answer
It depends. We prefer the term "recommended practice" because of questions like these.

Some sources recommend using production data:

…many production issues are due the lack of real(istic) test data…
To ensure software of the highest quality possible, you’ll need to keep the test environment as “in-sync” as possible with production.

https://www.datprof.com/blogs/using-production-data-for-testing/

Microsoft MVP recommends using production data for testing. This IBM document advises that someone use production data in a test environment. We would summarize this NimbleAMS document to suggest restoring production data in non-prod. This website essentially says that if you are following Agile best practices, then you are testing with production data.*

Some sources gived mixed recommendations on this topic
This U.K. source says it sometimes makes sense to use production data. This university thesis gives mixed treatment to using production data in a non-production environment. This thread gives mixed treatment to the practice of copying production data to lower environments. Another StackOverflow.com thread gives mixed treatment to the practice of copying production data to lower environments.

Continuous Delivery by Humble and Farley (on page 204 and 205) says that you spend more time trying to get a dataset than testing if you attempt to get data from production. In 2011 they did not think that production data should be used (but they did say for capacity testing it could be "occasionally" useful).

Some sources do not recommend using production data
These sources explicitly recommend you not use production data:

To not violate GDPR, you probably need to mask the production data (according to this source).

These sources also say you should not use production data in a test environment:

Red-gate.com also says you should not use production data in a test environment (see the term "safe"). For further reading on "best practices" versus "recommended practices, see this posting.

* See this quote:

Most of Agile development and product management’s best practices are forms of testing in development. We’re talking about very common practices like

CI/CD
A/B Testing
Phased Rollouts
Canary Deployments
Blue/green deployments
Usability Testing
Smoke & Sanity Testing

If you are following any of these practices—and many more like them—then you are already running tests with real-world users in a live production environment.

https://www.flagship.io/testing-in-production/

How Do You Install and Configure Terragrunt on a Linux Machine?

Prerequisite
Install make. If you need assistance, see this posting.

Procedures
Run these three commands:

curl https://github.com/gruntwork-io/terragrunt/releases/download/v0.35.18/terragrunt_linux_arm64 --output /tmp/terragrunt_linux_arm64

sudo mv -i /tmp/terragrunt_linux_arm64 /bin/terragrunt

sudo chmod u+x /bin/terragrunt

How Do You Troubleshoot the Kubernetes Error “cluster unreachable”?

Problem scenario
You get a message "kubernetes cluster unreachable" when running a helm, kubectl, az, or eks command. What should you do?

Possible solution #1
Has a router been reconfigured? Has a new firewall rule been imposed? Has a data center gone down that housed the cluster? Did you receive an email about a maintenance window or a configuration change?

Possible solution #2
What are the permissions of the relevant .yaml file? Can you try changing them? (Idea taken from https://github.com/k3s-io/k3s/issues/1126.)

Possible solution #3
If you were running a kubectl command, can you try to run commands like these?

sudo cp ~/.kube/config ~/.kube/bak.config.bak
kubectl config view --raw >~/.kube/config

(Idea taken from https://github.com/k3s-io/k3s/issues/1126.)

Possible solution #4
Warning: this may install something undesirable. Use this with caution.
If you were using helm, Rancher and K3s, can you try running a command like this?

sudo helm install harbor/harbor --version 1.3.0 --generate-name --kubeconfig /etc/rancher/k3s/k3s.yaml

(Idea taken from https://github.com/k3s-io/k3s/issues/1126.)

Possible solution #5
Are you using Rancher and K3s? Try running this:

export KUBECONFIG=/etc/rancher/k3s/k3s.yaml

(Idea taken from https://github.com/k3s-io/k3s/issues/1126 )

Possible solution #6
Can you ping the remote Kubernetes server? Are you not on a server that has access to the remote network? Is the cluster available from the network you are on? Are you running the command from a workstation that is not connected to a VPN?

Possible solution #7
Are you using GKE? If so, read this posting: https://forum.rasa.com/t/rasa-x-kubernetes-cluster-unreachable-error-using-github-actions/37073/8

Possible solution #8
Verify configuration files. If there are two clusters configured, can you try to only use one cluster? Try each cluster one-at-a-time. This may help you pinpoint the problem.

Possible solution #9
Run commands like these:

kubectl config view
kubectl config use-context CONTEXT-CHOSEN-FROM-PREVIOUS-COMMAND-OUTPUT
kubectl config --help

(This was adapted from https://stackoverflow.com/questions/49260135/unable-to-connect-to-the-server-dial-tcp-i-o-time-out.)

If you want assistance troubleshooting a network problem, see this posting.

Which Is a Higher Position, a Senior Engineer or a Staff Engineer?

Question
Which is a higher position, a Senior Engineer or a Staff Engineer?

Answer
Companies do not always conform to a job title convention or standard. However we find that the Staff Engineer title denotes a step above Senior. A Staff Engineer could be below a Principal Engineer however.

For further reading, see these postings (from which this posting was adapted):


How Do You Troubleshoot String Manipulation with List Slicing Not Working as You Expect?

Problem scenario
You are taking a slice of a Python string. You know that syntax like var_x[-2:] signifies the penultimate character and the last character of a string (called var_x). But you are only seeing one character -- not two. You expect to see two (or a different number of characters than what you are seeing). Why is the snippet printing out fewer characters than you expect?

Possible Solution #1
There could be an invisible "\n" attached to the string (e.g., at the end of the line). If you are using "readlines" to read an input file, there could be a "new line" or "line break" being processed as a single character. You may be seeing an incorrect number of characters in your slice/portion. If you are reading a file, and processing it line by line, you may want to use the .strip() function. This can remove the "\n" character that is getting picked up and ruining otherwise sound logic for your string manipulation task.

Possible Solution #2
There is an "if" or "for" loop or some logic in your program that is causing a special case to not process the way you are thinking about it. Sometimes a condition is met where a portion of code is entered or skipped, but we humans forget about this scenario. Use some intermediate print statements; this may be tedious, but you may be able to pinpoint the problem.