How Do You Troubleshoot a Puppet Manifest when The Errors and Logging Do Not Help You? (abridged version)

ABRIDGED VERSION (To see the unabridged version, see this posting.)

Problem scenario
A Puppet manifest is not working, but there are no obvious error messages. When running the puppet agent command, you use the -d flag for debugging. In your manifest, you use logoutput => true stanza. But still, you cannot figure out why your manifest is not working.

You tried this command: puppet parser validate nameOfManifest.pp

The above command had no output. On the Puppet Master server, you looked in /var/log/ to find clues. When the Puppet Agent runs and communicates with the Puppet Master server, on the back end of the Puppet Master, the activity should be somewhat logged to /var/log/puppet/puppetserver/puppetserver.log. You see no error messages anywhere that correspond to a puppet agent service running. You want to fix the manifest that is not producing the desired changes and/or effects that you want. Debugging can be difficult when Puppet appears to work perfectly. It seems hard to solve a manifest problem with no runtime errors and no errors in the logs.

How do you debug or otherwise troubleshoot a silently failing manifest that will not apply correctly?

Five Possible Solutions

#1. Are you using the ENC (External Node Classifier)? If you forget or fail to classify the Puppet agent and Puppet manifest correctly, you can observe a Puppet manifest not work with no error messages. For more details on how to remedy this issue, see "Possible solutions" #5 and #7 of this posting

#2. Are the modules directories or files subdirectories associated with the manifest not in their correct location? Or have such directories been given permissions that do not allow for files to be read from their locations? Puppet Master migrations or other systems administration operations on the Puppet Master server could cause dependent files to not be retrieved properly causing manifests to fail without warning or notice.

#3. Look at the Puppet manifest closely. Is the logic correct? Imperfect logic can be a big cause of errorless Puppet manifests not doing what is intended. For example, a node stanza could signify an incorrect server from what you expected. A class declaration could have prematurely terminating braces that could exclude an exec statement. This scenario (with the exec statement never running on a Puppet agent server) could be free from errors, even if you use the "puppet parser validate" feature. Be sure you did not forget to uncomment out a line in the manifest.

#4. Temporarily re-write a copy of the manifest purely for the sake of debugging. To diagnose what happens with the errorless run of a manifest, you may want to note the time at different stages as well as the user security context throughout the progression/failure of the manifest.

To determine the exact problem you could run exec commands that are specific to the OS receiving the configuration. For example, in Linux you could use date, whoami, or env. In Windows you could use date or whoami. These statements could be redirected (with the ">" symbol" to a file for viewing afterward.

Another way to learn more about a complex yet quietly failing manifest is to use Puppet's two DSL reserved words notify or notice. These can help you obtain more detailed messages without using regular Bash or PowerShell commands. To learn more about adding debugging details, see this external link. While the requires and require keywords can be confused, they can be helpful for ensuring the explicit execution order of a manifest that is failing in an elusive way.

#5. Make sure at least one Puppet manifest will work properly on a given Puppet agent server. Incorrectly configured Puppet master servers can cause an errorless run of a Puppet agent to actually not work. Restart the Puppet Master service. If that does not work, try rebooting the servers involved. If rebooting the servers involved does not work and you have time to do a great deal more, try to redeploy Puppet Master and Puppet Agent. When you try to apply the manifest again, it may work or there may be an error message.

Should Environmental Data Be Placed Into Version Control?

Problem scenario
You are not sure if environmental specific values should be placed into version control. What should do you?

Solution
Some people think that everything should go into version control. But others disagree. Here are examples of each philosophy:

"You need to get everything in version control. Everything. Not just the code, but everything required to build the environment." (This was taken from page 297 of The Phoenix Project.) This is very clear in how it disagrees with the twelve-factor app principles.

"The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally;" (This quote was taken from https://12factor.net/config.)

What Is The Difference between Unit Testing and Functional Testing?

Question
What is the difference between unit testing and functional testing in the context of professionals using one over the other?

Answer
A user wants to verify that the software behavior is correct from a black-box perspective; this more closely describes functional testing. A programmer is more concerned about the theoretical aspects of the code and its inner workings; this more closely describes unit testing. QA professionals blend user testing with programmer testing. See also What is an SDET?.

How Do You Use the “next” Function to Iterate Through an Iterable in Python?

Problem scenario
You want to use the reserved word "next" to parse through an iterable. What do you do?

Solution
Run this six-line program as an illustration:

contint = [ 15, 50, -100, 'foobar']
var1=iter(contint)
print(next(var1))
print(next(var1))
print(next(var1))
print(next(var1))

How Do You Set Up Apache Kafka on Any Distribution of Linux?

Problem scenario
You want to set up Apache Kafka with a single broker just to test out. How do you do this on any distribution of Linux?

Prerequisite
You must install Zookeeper. If you need assistance, see this posting.

You may want to install screen because this solution will rely on it; you could open duplicate terminals instead of using the screen command.

Have Zookeeper running before you run the [Kafka installation] script below.

Procedures

portion=2.8.0
kafkaversion=2.12-$portion
curl -Ls https://ftp.wayne.edu/apache/kafka/2.4.1/kafka_$kafkaversion.tgz > /tmp/kafka_$kafkaversion.tgz
cd /bin/
cp /tmp/kafka_$kafkaversion.tgz .
sudo tar -zxf kafka_$kafkaversion.tgz
cd kafka_$kafkaversion/bin
./kafka-server-start.sh -daemon /root/kafka/config/server.properties
./kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
./kafka-topics.sh -list --bootstrap-server localhost:9092
echo "Kafka should be installed now"
echo 'Run this command:
./kafka-console-producer.sh --broker-list localhost:9092 --topic test
echo "go to another terminal"
and then enter some text'
echo " "
echo 'Start a new terminal, and run this command:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning'

How Do You Create an AWS S3 Bucket via the AWS CLI?

Problem scenario
You want to create an AWS S3 bucket using the AWS CLI. What do you do?

Solution
Prerequisite
This assumes that you have installed the AWS CLI. If you need assistance, see this posting.

Procedures
Run a command like this but replace "foobar" with the unique name of an s3 bucket that other people will see.

aws s3 mb s3://foobar

If you want to create it in a non-default region, use the --region flag like this (but replace us-east-1 with the region of your choice):

aws s3 mb s3://foobar --region us-east-1

If you aren't sure about the default region (if you did not use the --region flag), see How Do You Determine the AWS CLI Default Region?.

What Are Namespaces?

Problem scenario
You have heard of namespaces in programming, DNS, AWS, operating systems, and in the context of managing containers. You want to see the disambiguation of of this term insofar as computing is concerned. What are namespaces in I.T.?

Overview
Names can be bound to different spaces (or subsystems), and this binding varies depending on the technology. Namespacing helps facilitate locally unique identification/resolution of names. Projects can scale with independent contributors re-using names that are not globally unique without interfering with each other because of namespaces.

Answer: Namespaces for Programming
"Namespaces in programming terminology are essentially "containers" that help you organize things like methods and classes. The usefulness of namespaces becomes more apparent as the programs you write become more and more complex and you start to make use of libraries and classes provided by other programmers--it becomes a useful tool to have objects and methods use descriptive and helpful naming schemes." This quote was taken from page 38 of Customizing Chef.

Namespaces are used in languages such as C++, Java, Python, and Ruby.

"A namespace is a set of names (identifiers) in which each name is unique." (Taken from page 1104 in a section on Python programming in A Practical Guide to Fedora and Red Hat Enterprise Linux.)

"A namespace is a self-contained package of variables, which are known as the attributes of the namespace object." (Taken from page 686 of Python Programming.)

In Python there is a lexical evaluation referred to as LEGB. This governs how identifiers/names are evaluated based on their scope or namespace.
"The LEGB rule is used to decide the order in which the namespaces are to be searched for scope resolution." (Taken from medium.com.) To learn more, see What is the Difference between global and nonlocal in Python?.

The LEGB identifier evaluation system for Python is a namcespace rule; in some ways this rule is different from namespace packages that Python uses via the "import" statement. When you use an "import" statement, words you use in your Python program can be matched/identified and evaluated when they otherwise wouldn't be.

If you try to run a statement with the word "requests" in a Python program without an "import requests" statement, you will get this error:

NameError: name 'requests' in not defined

You must import the requests namespace package first to avoid an error such as the one above.

Answer: Namespaces for DNS
DNS namespace or domain namespace is a reference to words separated by dots "." in a URL near the hostname. For example http://portal.continualintegration.com/ shows "portal" as a name in the continualintegration.com namespace. This concept is explained in detail here.

Answer: Namespaces for AWS
"A namespace is a way to group services for an application. When you create a namespace, you specify how you want to discover service instances that you register with AWS Cloud Map: using API calls or using DNS queries. You also specify the name that you want your application to use to discover instances. " (Taken from official AWS documentation.)

If you have the AWS CLI installed, you can view the namespaces with this command:
aws servicediscovery list-namespaces

Answer: Namespaces for Operating Systems
"A namespace is a set of names (identifiers) in which each name is unique." (Taken from page 37 of A Practical Guide to Fedora and Red Hat Enterprise Linux.)

Running processes can be separated at the Linux OS level. To see the inode number for a given name space on a Linux OS, use the lsns command. To learn about this command, see this posting.

If you want to list the namespaces in your Linux OS, see this posting (assuming you have installed the util-linux package).

To see the network namespaces on a Linux OS, see this this external page.

Answer: Namespaces for Containers
There are six types of namespaces for Docker containers: mount, UTS, IPC, PID, network and user. Namespaces and cgroups are components that do isolation for the containers.

"[N]amespaces allow for processes to use identically named resources and isolate them from each other's view of the system" according to page 156 of Docker Up and Running.

Mount namespaces are used by Docker "to make your container look like it has its entire own filesystem namespace." (This quote was taken from page 161 of Docker Up and Running.)

UTS namespaces were "[n]amed for the kernel structure they namespace, and ultimately from the "unix Timesharing System," UTS namespaces give your container its own hostname and domain name." (This quote was taken from page 161 of Docker Up and Running.)

Interprocess communication namespaces (better known as IPC namespaces) are created by Docker. Each Docker container gets a unique IPC namespace (according to https://dzone.com/articles/docker-in-action-the-shared-memory-namespace).

"PID namespace provides separation of processes. The PID Namespace removes the view of the system processes, and allows process ids to be reused including pid 1." (Taken from https://docs.docker.com/engine/reference/run/.)

Network namespace separates IP addresses and routing tables of Docker containers. To learn more, see this edureka.com posting or this platform9.com posting.

"User namespaces are used with containers to make it possible to setup a container without privileged operations, and so that a normal user can act as root inside a container to perform administrative tasks, without being root on the host outside." (This was taken from an external website.) User namespaces isolate users and groups in Docker containers from the Docker host server (according to page 162 of Docker Up and Running). To read more about them, see this posting.

Answer: Namespaces for Container Orchestration
"Using multiple namespaces allows you to split complex systems with numerous components into smaller distinct groups." (Taken from page 77 of Kubernetes in Action.)

"…there are actually three namespaces that Kubernetes ships with: default, kube-system (used for Kubernetes components), and kube-public (used for public resources)." (This quote was taken from https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-organizing-with-namespaces.)

The command kubectl get ns will list the namespaces.

Docker Swarm may not have namespaces, "Docker Swarm stacks are, in a way, similar to Kubernetes Namespaces. All the services in a stack are uniquely identified through a combination of a stack name and the names of services inside it. By default, all services within a stack can communicate with each other through the stack's default network. Services can speak with those from other stacks only if they are explicitly attached to the same network. All in all, each Swarm stack is separated from other stacks. They are, in a way, similar to Kubernetes Namespaces." This quote was taken from this site.

To learn more about different types of namespaces in general, you may want to see this Webopedia.com posting.

Answer: Namespaces for Puppet
A namespace in Puppet is defined by the scope of a module, class or other defined type.
In Puppet you have classes and defined types; these are in a namespace of the outermost class or definition (according to https://puppet.com/docs/puppet/7.4/lang_namespaces.html). Nested classes are not recommended however (according to https://puppet.com/docs/puppet/7.4/lang_namespaces.html).

The define key word enables you to define a type; to read more, see this posting.

Answer: Namespaces for XML Files
A namespace in XML allows for local names to be re-used like other namespaces. Some projects/solutions with XML will include independent contributors. Below are partial quotes and one paraphrase taken from w3.org that explain what an XML namespace is.

"Namespaces are a mechanism for managing names in a distributed way that greatly reduces the likelihood that two" people will create a conflicting name.

"An XML namespace has a namespace name (a ...[Uniform Resource Identifier]...) and a set of local names..."

How Do You Merge Three Files (.jpegs and PDFs) into One PDF?

Problem scenario
You have a non-business project (e.g., a non-commercial, non-professional, non-work-related project). You need to combine individual JPEGs and PDFs into one PDF. How do you do this?

Solution

  1. Print the JPEG(s) to Microsoft Print to PDF. Now the JPEG(s) has (or have) been converted into the PDF format.
  2. Use PDFmate.com to download and install "Free PDF Merger for Windows."
  3. Open PDFmate Merger. Choose the PDFs you want to merge (e.g., those created in step #1). It is user friendly.

Alternative Solution (that does not use PDFmate)
See this posting.

How Do You Set an Environment Variable for Every User Upon Rebooting a Linux Server?

Problem scenario
Without creating a hard or soft link of a file, how do you set an environment variable for every user after every reboot?

Solution

  1. Update the /etc/environment file so it has this stanza (but replace "JAVA_HOME" with the variable of your choice and replace /usr/bin/java with the path or value of the variable as you desire):

JAVA_HOME=/usr/bin/java

  1. Save the changes of the file. You are done. Log off as the user for the user you were logged in as to have the changes take effect.

If you want only one user to have the environment variable, see this posting.

Cryptocurrency News and Information

JP Morgan is viewing Bitcoin more favorably according to this Forbes article. If you would like to learn more about Bitcoin, you may want to see this article. If you want to buy Bitcoin or a variety of cryptocurrencies, try Changelly.

If you want to receive free cryptocurrency by just learning more, try Coinbase; it is ideal for Americans. For Europeans, the platform/company Iconomi.com can allow you to buy crypto or learn more.