A Technical I.T./DevOps Blog
Problem scenario
You are designing a Flask application with Python running some long-running processes. You want to handle the requests asynchronously. The individual requests will have a duration of more than one hour. What is the recommended way of handling this?
Solution
Use a task queue like Celery or RQ (Redis Queue).
Sources:
https://stackoverflow.com/questions/42790362/how-to-handle-long-sql-query-in-flask
Problem scenario
You want to extend Kubernetes' functionality. You want to use CustomResourceDefinitions. You can view API resources by running "kubectl api-resources". You want to add to the list. How do you use a CRD?
Solution
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: contintexamples.continualintegration.com
spec:
group: continualintegration.com
names:
kind: ContintExample
plural: contintexamples
listKind: ContintExampleList
singular: contintexample
scope: Namespaced
versions:
- name: v1
# Each version can be enabled/disabled by Served flag.
served: true
# One and only one version must be marked as the storage version.
storage: true
schema:
openAPIV3Schema:
type: object
properties:
spec:
type: object
properties:
owner:
type: string
description: The name of the owner of this Contint Example franchise
address:
type: string
description: The physical address of the Contint Example
currency:
type: string
description: The currency which this Contint Example accepts payments in
replicas:
type: integer
description: The number of instances of the Contint Example app to deploy
additionalPrinterColumns:
- name: Owner
jsonPath: .spec.owner
type: string
- name: Currency
jsonPath: .spec.currency
type: string
- name: Replicas
jsonPath: .spec.replicas
type: integer
- name: Age
jsonPath: .metadata.creationTimestamp
type: date(# The above file was adapted from https://www.tutorialworks.com/kubernetes-custom-resources/ )
2. Run this: kubectl apply -f contint.yaml
3. You have now created a CustomResourceDefinition. The below steps 3a, 3b, 3c, 3d, and 3e are optional.
3.a. Run kubectl api-resources to view what API resources are currently at your disposal. Run kubectl get crd to view what you already may exist.
3.b. Run kubectl get crd to view what you created among everything else that was already there.
3.c. Run kubectl describe crd contintexamples.continualintegration.com to view the details.
3.d. Run kubectl api-resources to view what API resources are currently at your disposal.
3.e. Test it by running this: kubectl api-resources --api-group=continualintegration.com
Question
When you have high availability of servers, you have seen diagrams with a "sink" server. What does the word "sink" signify?
Answer
A sink, homophonic with "sync", is a secondary server in a cluster or replication configuration. That is, there is a primary server that is actively processing transactions of databases or registering changes to files. This primary server may replicate changes over to a secondary or sink server. The sink server receives the changes secondarily from the primary server.
Problem scenario
From a Python command line, you get this message:
Traceback (most recent call last):
File "", line 1, in
ImportError: No module named 'pandas'
What should you do?
Solution
Prerequisite
You need pip to be installed. If you need assistance, see this posting.
Procedures
Run one of these commands:
pip install pandaspip3 install pandasYou scanned some documents into a Windows machine. One or both of the following apply:
Problem scenario #1
When you right click on them to print them, the print preview is showing them rotated 90 degrees from the orientation you want.
Problem scenario #2
When you right click on them to print them, they print rotated 90 degrees from the orientation that you would prefer.
You want to have them print in the way the .jpegs are viewed. How do you print them this way?
Solution
In the "Print Pictures" pop up, click on "Options" -> "Printer Properties" and go to the "Orientation" drop down. Choose "Portrait".
Problem scenario
You try this prog1.py:
book = {}
book["apple"] = 5
var1 = book["orange"]It returns this:
Traceback (most recent call last): File "prog1.py", line 3, in var1 = book["orange"] KeyError: 'orange'
You often see a couple lines of code check if a key exists in a dictionary before a specific action. You want to avoid this and write Python in a more elegant fashion. What do you do?
Solution
Use the .get() method.
Try this prog2.py:
book = {}
book["apple"] = 5
var1 = book.get("orange")It returns nothing. There are no errors. (You could then print var1 to view the contents.)
Question
You have heard of different stacks such as the call, kernel and user stacks.
How are they different?
Answer
"The call stack, or runtime stack, is an area of memory used to store function return information and local variables." (This was taken from https://developer.arm.com/documentation/101470/2021-2/Examining-the-Target/Examining-the-call-stack)
"The user stack contains the arguments, local variables, and other data for functions executing in user mode." (Taken from https://www.gotothings.com/unix/user-and-kernel-stack-for-copy-program.htm.)
"The user stack is only used while the process is running in user mode." (Taken from https://www.baeldung.com/linux/kernel-stack-and-user-space-stack.)
"The kernel stack is a per-process memory region maintained in kernel memory that is used as the stack for execution of the functions called internally during the execution of a system call." (This was taken from page 122 of Kerrisk's The Linux Programming Interface book.)
The user stack resides in a different location from the kernel stack (taken from https://stackoverflow.com/questions/12911841/kernel-stack-and-user-space-stack). What can access to the stacks differs too (also taken from https://stackoverflow.com/questions/12911841/kernel-stack-and-user-space-stack).
"In a Linux system, every user process has 2 stacks, a user stack and a dedicated kernel stack for the process." (Taken from https://www.quora.com/What-is-a-processes-kernel-stack-What-exactly-is-its-use-besides-keeping-the-thread_info.)
The call stack is the combination of the user stack and the kernel stack.
"The kernel stack and the user stack are implemented using the stack data structure and, taken together, serve as a call stack. " (Taken from https://www.baeldung.com/linux/kernel-stack-and-user-space-stack.)
The kernel stack (as defined by https://linux-kernel-labs.github.io/refs/heads/master/lectures/intro.html) is as follows:
"Each process has a kernel stack that is used to maintain the function call chain and local variables state while it is executing in kernel mode, as a result of a system call.
The kernel stack is small (4KB - 12 KB) so the kernel developer has to avoid allocating large structures on stack or recursive calls that are not properly bounded."
"The kernel stack is directly mapped to the physical memory, mandating the arrangement to be physically in a contiguous region." (Taken from https://subscription.packtpub.com/book/application-development/9781785883057/1/ch01lvl1sec10/kernel-stack)
"The call stack is a list of call stack entries, in a last-in-first-out (LIFO) order. A call stack entry is a call to a program or procedure." (Taken from https://www.ibm.com/docs/en/i/7.4?topic=overview-call-stack.)
See these articles for more information:
This may serve as a relevant illustration: https://stackoverflow.com/questions/48250382/how-to-display-user-mode-stack-of-current-kernel-mode-context
Question
You have read about user mode and kernel mode. Are they the different from userspace and kernel space respectively?
Answer
Yes. User space is not the same as user mode. Kernel space is not the same as kernel mode.
"Modern processor architectures typically allow the CPU to operate in at least two different modes: user mode and kernel mode (sometimes also referred to as supervisor mode)." Taken from page 23 of The Linux Programming Interface by Michael Kerrisk.
User space is defined here: What Is User Space and Is It Different from the User Stack?
Kernel space is defined here: What Is Kernel Space and Is It Different from the Kernel Stack?
Question
What is user space and is it different from the user stack? If the user stack is part of the user space, what else does user space have besides the user stack?
Answer
User space is a portion of memory of an Operating Systems that is not designated for kernel space. System calls enable user processes to enter kernel space. To read more about a system call, see this article.
System memory in Linux can be divided into two distinct regions: kernel space and user space. Kernel space is where the kernel (i.e., the core of the operating system) executes (i.e., runs) and provides its services.
http://www.linfo.org/kernel_space.html
Kernel space is mutually exclusive with user space (according to https://unix.stackexchange.com/questions/87625/what-is-difference-between-user-space-and-kernel-space, http://www.catb.org/jargon/html/U/userland.html, https://www.educative.io/answers/what-is-the-difference-between-the-kernel-and-user-spaces).
"Grossly simplifying, the kernel space is the memory area that is reserved to the kernel while user space is the memory area reserved to a particular user process. The kernel space is access protected so that user applications can not access it directly, while user space can be directly accessed from code running in kernel mode." (This quote was taken from https://linux-kernel-labs.github.io/refs/heads/master/lectures/intro.html.)
"When running in user mode, the CPU can access only memory that is marked as being in user space…" (according to page 605 of The Linux Programming Interface by Michael Kerrisk).
POSIX threads have IDs that are created and controlled in user space (according to page 605 of The Linux Programming Interface by Michael Kerrisk).
Yes, user space is different from the user stack. "In the user space, we can find the user stack that grows downward to lower addresses, whereas dynamic allocations (heap) grow upwards to higher addresses." (Taken from https://www.baeldung.com/linux/kernel-stack-and-user-space-stack.)
Definitely the user stack is only a sub-component of the user space. "The user space part of the virtual space is categorized into Stack and Heap, BSS, Data, Text." (Taken from https://medium.com/@shoheiyokoyama/understanding-memory-layout-4ef452c2e709.)
Here is a visual of what user space encompasses (beyond the user stack):
https://miro.medium.com/max/640/1*CtL4OiSQFvSrK8mIOkb5zQ.webp
BSS stands for Block Started by Symbol (according to http://dev.fyicenter.com/Interview-Questions/UNIX/What_is_BSS_Block_Started_by_Symbol_.html).
For further reading, see the following:
This explains the difference between the heap and the stack: https://stackoverflow.com/a/80113
This article refers to a user-space stack and to a user-space TCP stack.
This article also refers to a "userspace network stack."
This article refers to "user-space stack."
Do Either Kernel Space or User Space Play a Role in Operating Systems Networking?
Question
What is kernel space and is it different from the kernel stack?
Answer
"System memory in Linux can be divided into two distinct regions: kernel space and user space. Kernel space is where the kernel (i.e., the core of the operating system) executes (i.e., runs) and provides its services. "
Kernel space is mutually exclusive with user space (according to https://unix.stackexchange.com/questions/87625/what-is-difference-between-user-space-and-kernel-space, http://www.catb.org/jargon/html/U/userland.html, https://www.educative.io/answers/what-is-the-difference-between-the-kernel-and-user-spaces).
"Grossly simplifying, the kernel space is the memory area that is reserved to the kernel while user space is the memory area reserved to a particular user process. The kernel space is accessed protected so that user applications can not access it directly, while user space can be directly accessed from code running in kernel mode." taken from https://linux-kernel-labs.github.io/refs/heads/master/lectures/intro.html
Yes, there is a difference between the two concepts.
"The kernel stack is part of the kernel space. Hence, it is not directly accessible from a user process." (Taken from https://www.baeldung.com/linux/kernel-stack-and-user-space-stack)
See this for more information about a kernel accessing hardware:
https://www.continualintegration.com/miscellaneous-articles/can-the-kernel-access-hardware-directly/