A Technical I.T./DevOps Blog
Problem scenario
You want to use the reserved word "next" to parse through an iterable. What do you do?
Solution
Run this six-line program as an illustration:
contint = [ 15, 50, -100, 'foobar']
var1=iter(contint)
print(next(var1))
print(next(var1))
print(next(var1))
print(next(var1))Problem scenario
You want to set up Apache Kafka with a single broker just to test out. How do you do this on any distribution of Linux?
Prerequisite
You must install Zookeeper. If you need assistance, see this posting.
You may want to install screen because this solution will rely on it; you could open duplicate terminals instead of using the screen command.
Have Zookeeper running before you run the [Kafka installation] script below.
Procedures
portion=2.8.0
kafkaversion=2.12-$portion
curl -Ls https://ftp.wayne.edu/apache/kafka/2.4.1/kafka_$kafkaversion.tgz > /tmp/kafka_$kafkaversion.tgz
cd /bin/
cp /tmp/kafka_$kafkaversion.tgz .
sudo tar -zxf kafka_$kafkaversion.tgz
cd kafka_$kafkaversion/bin
./kafka-server-start.sh -daemon /root/kafka/config/server.properties
./kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test
./kafka-topics.sh -list --bootstrap-server localhost:9092
echo "Kafka should be installed now"
echo 'Run this command:
./kafka-console-producer.sh --broker-list localhost:9092 --topic test
echo "go to another terminal"
and then enter some text'
echo " "
echo 'Start a new terminal, and run this command:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning'Problem scenario
You want to create an AWS S3 bucket using the AWS CLI. What do you do?
Solution
Prerequisite
This assumes that you have installed the AWS CLI. If you need assistance, see this posting.
Procedures
Run a command like this but replace "foobar" with the unique name of an s3 bucket that other people will see.
aws s3 mb s3://foobar
If you want to create it in a non-default region, use the --region flag like this (but replace us-east-1 with the region of your choice):
aws s3 mb s3://foobar --region us-east-1
If you aren't sure about the default region (if you did not use the --region flag), see How Do You Determine the AWS CLI Default Region?.
Problem scenario
You have heard of namespaces in programming, DNS, AWS, operating systems, and in the context of managing containers. You want to see the disambiguation of of this term insofar as computing is concerned. What are namespaces in I.T.?
Overview
Names can be bound to different spaces (or subsystems), and this binding varies depending on the technology. Namespacing helps facilitate locally unique identification/resolution of names. Projects can scale with independent contributors re-using names that are not globally unique without interfering with each other because of namespaces.
Answer: Namespaces for Programming
"Namespaces in programming terminology are essentially "containers" that help you organize things like methods and classes. The usefulness of namespaces becomes more apparent as the programs you write become more and more complex and you start to make use of libraries and classes provided by other programmers--it becomes a useful tool to have objects and methods use descriptive and helpful naming schemes." This quote was taken from page 38 of Customizing Chef.
Namespaces are used in languages such as C++, Java, Python, and Ruby.
"A namespace is a set of names (identifiers) in which each name is unique." (Taken from page 1104 in a section on Python programming in A Practical Guide to Fedora and Red Hat Enterprise Linux.)
"A namespace is a self-contained package of variables, which are known as the attributes of the namespace object." (Taken from page 686 of Python Programming.)
In Python there is a lexical evaluation referred to as LEGB. This governs how identifiers/names are evaluated based on their scope or namespace.
"The LEGB rule is used to decide the order in which the namespaces are to be searched for scope resolution." (Taken from medium.com.) To learn more, see What is the Difference between global and nonlocal in Python?.
The LEGB identifier evaluation system for Python is a namcespace rule; in some ways this rule is different from namespace packages that Python uses via the "import" statement. When you use an "import" statement, words you use in your Python program can be matched/identified and evaluated when they otherwise wouldn't be.
If you try to run a statement with the word "requests" in a Python program without an "import requests" statement, you will get this error:
NameError: name 'requests' in not defined
You must import the requests namespace package first to avoid an error such as the one above.
Answer: Namespaces for DNS
DNS namespace or domain namespace is a reference to words separated by dots "." in a URL near the hostname. For example http://portal.continualintegration.com/ shows "portal" as a name in the continualintegration.com namespace. This concept is explained in detail here.
Answer: Namespaces for AWS
"A namespace is a way to group services for an application. When you create a namespace, you specify how you want to discover service instances that you register with AWS Cloud Map: using API calls or using DNS queries. You also specify the name that you want your application to use to discover instances. " (Taken from official AWS documentation.)
If you have the AWS CLI installed, you can view the namespaces with this command:aws servicediscovery list-namespaces
Answer: Namespaces for Operating Systems
"A namespace is a set of names (identifiers) in which each name is unique." (Taken from page 37 of A Practical Guide to Fedora and Red Hat Enterprise Linux.)
Running processes can be separated at the Linux OS level. To see the inode number for a given name space on a Linux OS, use the lsns command. To learn about this command, see this posting.
If you want to list the namespaces in your Linux OS, see this posting (assuming you have installed the util-linux package).
To see the network namespaces on a Linux OS, see this this external page.
Answer: Namespaces for Containers
There are six types of namespaces for Docker containers: mount, UTS, IPC, PID, network and user. Namespaces and cgroups are components that do isolation for the containers.
"[N]amespaces allow for processes to use identically named resources and isolate them from each other's view of the system" according to page 156 of Docker Up and Running.
Mount namespaces are used by Docker "to make your container look like it has its entire own filesystem namespace." (This quote was taken from page 161 of Docker Up and Running.)
UTS namespaces were "[n]amed for the kernel structure they namespace, and ultimately from the "unix Timesharing System," UTS namespaces give your container its own hostname and domain name." (This quote was taken from page 161 of Docker Up and Running.)
Interprocess communication namespaces (better known as IPC namespaces) are created by Docker. Each Docker container gets a unique IPC namespace (according to https://dzone.com/articles/docker-in-action-the-shared-memory-namespace).
"PID namespace provides separation of processes. The PID Namespace removes the view of the system processes, and allows process ids to be reused including pid 1." (Taken from https://docs.docker.com/engine/reference/run/.)
Network namespace separates IP addresses and routing tables of Docker containers. To learn more, see this edureka.com posting or this platform9.com posting.
"User namespaces are used with containers to make it possible to setup a container without privileged operations, and so that a normal user can act as root inside a container to perform administrative tasks, without being root on the host outside." (This was taken from an external website.) User namespaces isolate users and groups in Docker containers from the Docker host server (according to page 162 of Docker Up and Running). To read more about them, see this posting.
Answer: Namespaces for Container Orchestration
"Using multiple namespaces allows you to split complex systems with numerous components into smaller distinct groups." (Taken from page 77 of Kubernetes in Action.)
"…there are actually three namespaces that Kubernetes ships with: default, kube-system (used for Kubernetes components), and kube-public (used for public resources)." (This quote was taken from https://cloud.google.com/blog/products/gcp/kubernetes-best-practices-organizing-with-namespaces.)
The command kubectl get ns will list the namespaces.
Docker Swarm may not have namespaces, "Docker Swarm stacks are, in a way, similar to Kubernetes Namespaces. All the services in a stack are uniquely identified through a combination of a stack name and the names of services inside it. By default, all services within a stack can communicate with each other through the stack's default network. Services can speak with those from other stacks only if they are explicitly attached to the same network. All in all, each Swarm stack is separated from other stacks. They are, in a way, similar to Kubernetes Namespaces." This quote was taken from this site.
To learn more about different types of namespaces in general, you may want to see this Webopedia.com posting.
Answer: Namespaces for Puppet
A namespace in Puppet is defined by the scope of a module, class or other defined type.
In Puppet you have classes and defined types; these are in a namespace of the outermost class or definition (according to https://puppet.com/docs/puppet/7.4/lang_namespaces.html). Nested classes are not recommended however (according to https://puppet.com/docs/puppet/7.4/lang_namespaces.html).
The define key word enables you to define a type; to read more, see this posting.
Answer: Namespaces for XML Files
A namespace in XML allows for local names to be re-used like other namespaces. Some projects/solutions with XML will include independent contributors. Below are partial quotes and one paraphrase taken from w3.org that explain what an XML namespace is.
"Namespaces are a mechanism for managing names in a distributed way that greatly reduces the likelihood that two" people will create a conflicting name.
"An XML namespace has a namespace name (a ...[Uniform Resource Identifier]...) and a set of local names..."
Problem scenario
You have a non-business project (e.g., a non-commercial, non-professional, non-work-related project). You need to combine individual JPEGs and PDFs into one PDF. How do you do this?
Solution
Alternative Solution (that does not use PDFmate)
See this posting.
Problem scenario
Without creating a hard or soft link of a file, how do you set an environment variable for every user after every reboot?
Solution
JAVA_HOME=/usr/bin/java
If you want only one user to have the environment variable, see this posting.
JP Morgan is viewing Bitcoin more favorably according to this Forbes article. If you would like to learn more about Bitcoin, you may want to see this article. If you want to buy Bitcoin or a variety of cryptocurrencies, try Changelly.
If you want to receive free cryptocurrency by just learning more, try Coinbase; it is ideal for Americans. For Europeans, the platform/company Iconomi.com can allow you to buy crypto or learn more.
Question
What is a second generation configuration management tool?
Answer
Page xiii of The DevOps Handbook refers to John Willis characterizing Luke Kanies' thoughts on configuration mangaement as "second generation CM." But according to https://www.cmcrossroads.com/article/cm-generations-and-vision-future there are four generations of CM. This may or may not be compatible with what John Willis hand in mind, but we do not see how Luke Kanies' vision could be the second oldest of the four generations. We look forward to more sources defining different generations of configuration management tools (such as Docker, Puppet, Ansible etc.). Clearly configuration management has evolved over the years.
Configuration management is very important for managing codebases and for DevOps in general. Fred Brooks, the author of The Mythical Man-Month, envisioned a Configuration Manager as a code librarian (according to page 221 of The Cathedral and The Bazaar). GitOps is taking CM principles and making them a way that we communicate. To learn more about communication structures in business and how to reconcile Conway's Law with the DevOps movement, see this posting.
Problem scenario
You used a .json file and the AWS CLI to configure customized metrics. When you go to the GUI you do not see the graphs. The commands that were run with the .json file completed without errors or problems. How do you get the web console to show the metrics you configured for various AWS components?
Possible Solution #1
CloudWatch metrics are not all necessarily graphed. Some of the metrics may not be graphed. Verify the metrics you configured are graphed at all.
Possible Solution #2
Find the instance ID or resource ID of the resource you want to monitor. Go to the web console for AWS. Click on "Metrics" on the left. Search for the instance ID or resource ID. It should be there when it may not have been in a different AWS web console dashboard.
Problem scenario
You run an aws ec2 command and you get use the --filters flag with Name=VolumeId (consistent with the casing that you see in other AWS CLI contexts). You receive a message about "the filter VolumeId" being invalid. What should you do?
Solution
Use "volume-id" instead of "VolumeId".
In general we find that the syntax of the name itself, for an AWS CLI --filters flag, should be lowercase and hyphenated. We do not know why it is relatively undocumented. We find the hyphens are right before what would be medial captials of a given name.
You can use unfiltered output to see what is returned from a given AWS command. You can use the "." dot notation to drill into a given value that is nested in JSON with the nested key to the right of its parent.
Here you see VpcId nested inside of RequesterVpcInfo:
"RequesterVpcInfo": {
…
"VpcId": "vpc-123456",To refer to the VPC ID itself for filtering purposes, you would use the following (e.g., with an 'aws ec2 desribe-vpc-peering-connections' command) dot notation as illustrated:
--filters Name=requester-vpc-info.vpc-id,Values=vpc-123456
You can add more restrictions by adding additional Name-Values pairs with a space after the above snippet. Here is an example:--filters Name=requester-vpc-info.vpc-id,Values=vpc-123456 Name=accepter-vpc-info.region,Values=us-west-2
Output would have to meet both the requirements of the VPC ID as vpc-123456 and the accepter VPC would have to be in the region us-west-2.