What is DevSecOps (or DevOpsSec, SecDevOps, or rugged DevOps)?

Updated 2/10/22

Answer
DevSecOps is the integration of security practices with DevOps.  DevOps may be a culture of automation (as Stackify says), a blend of disciplines (development and operations), or a key word in a job title (e.g., DevOps Engineer or DevOps Architect).  DevSecOps seeks to rapidly integrate security measures into development and operations and avoid security specialists working in separate and relatively isolated teams.  By having security-minded professionals in the same team, the benefits of automation can help security, and security can protect rapidly-growing server environments with Agile development.

To learn more about DevSecOps, see the following five links:

  • The DevSecOps Manifesto is here.
  • An independent definition of DevSecOps is here.
  • SANS produced a whitepaper on DevSecOps.
  • Sonatype has a PowerPoint presentation here.
  • ZDNet has an article defining DevSecOps.
  • CA Technologies used to have an article defining DevSecOps here: https://automic.com/blog/just-what-devsecops-ara

In 2012 Gartner called what we think of DevSecOps as DevOpsSec (https://blogs.gartner.com/neil_macdonald/2012/01/17/devops-needs-to-become-devopssec/).  This term persists, and there are actually two more synonymous terms in use: SecDevOps and rugged DevOps.  To learn about the term Rugged DevOps (as it is sometimes capitalized), see this link.  A leading security website, CSOonline.com, describes three of the terms (DevOpsSec, DevSecOps, and SecDevOps, but not the rugged DevOps term).  A book published in 2016 clarifies that there are four different terms for the infusion of security practices into DevOps: "Whether you call it SecDevOps, DevSecOps, DevOpsSec, or Rugged DevOps..." (page 5 of DevOpsSec by Jim Bird, Oreilly Press 2016).  Another credible source confirms that there are four terms that mean the same thing (Sqreen.io).  Consider the following findings (as of 11/22/17):

  • A search for "devsecops" on Indeed.com found 192 jobs and on Dice.com found 31 jobs. 
  • A search for "secdevops" on Indeed.com found 38 jobs and on Dice.com found 3 jobs.
  • A search for "devopssec" on Indeed.com found 11 jobs and on Dice.com found 1 job.

Consider the updated findings (as of 9/25/20):

  • A search for "devsecops" on Indeed.com found 4,797 jobs and on Dice.com found 95 jobs. 
  • A search for "secdevops" on Indeed.com found 234 jobs and on Dice.com found 6 jobs.
  • A search for "devopssec" on Indeed.com found 16 jobs and on Dice.com found 0 jobs.

Consider the updated findings (as of 2/10/22):

  • A search for "devsecops" on Indeed.com found 19,002 jobs and on Dice.com found 110 jobs. 
  • A search for "secdevops" on Indeed.com found 536 jobs and on Dice.com found 3 jobs.
  • A search for "devopssec" on Indeed.com found 20 jobs and on Dice.com found 0 jobs.

Clearly varieties of the terms exist, and there has been a tremendous growth of this concept in the past few years. The term "rugged devops" is not discernibly searchable like the other three terms.  Which of the four options should be used for the infusion of security practices into DevOps?  Here is an excerpt from a third source, a relevant TechBeacon article, acknowledging all four terms:  "People call it DevSecOps, SecDevOps, DevOpsSec, and even rugged DevOps. How can we have so many different terms to describe the exact same thing?  This gives us a hint as to the disconnect that exists within security in DevOps. It’s still the wild west [sic]. There is no standard that defines security for DevOps, and the chances of a standard ever developing is small because different organizations are doing things their own way, and can’t even agree on a standard name. And while there is a standard for the secure development lifecycle (ISO/IEC 27034-1), few organizations are ever validated against it . [sic]"

To clarify, DevOpsSec, SecDevOps, DevSecOps, and rugged DevOps are each the same thing: they are the integration of security practices with DevOps.

Arguably DevOps necessarily incorporates security with quality assurance (or QA), development, and operations; DevSecOps or synonyms such as SecDevOps, DevOpsSec, or rugged DevOps, could all be considered redundant to the term DevOps.  One difference that is undeniable in three of these terms is the three letters Sec which explicitly reminds people about security. 

Gene Kim, who is a DevOps celebrity (if such a thing exists in September 2020), started to make a name for himself when he co-founded a security software company called Tripwire.  Tripwire's main product is a host-based intrusion detection system that is focused on finding changes in files themselves.  (Host-based intrusion detection systems, HIDSes, look for intrusions on a server, by monitoring files and processes.  They do not monitor a TCP/IP network like Snort or other network-based intrusion detection systems.)  Tripwire's system can also be called security information and event management, or SIEM.

This host-based intrusion detection system approach to security concerns itself with minute changes in files.  This scrutiny of files is characteristic of code versioning systems (such as Subversion or Git) used in CI/CD pipelines.  These pipelines are a significant part of what DevOps is -- integrating development with operations.  DevOps engineers can design CI/CD pipelines to have processes triggered on the act of a new file being uploaded or committed to a repository.  A typical CI/CD pipeline will initiate a build upon the event of an existing file being modified. 

In the mid 1990s when the Tripwire security product came out, no one could have known the role configuration management tools (such as Puppet, Chef, Ansible, and SaltStack) would play in changing files across large numbers of enterprise servers.  Similarly at this time no one could have known the importance of future tools such as Git, Jenkins, or the DevOps movement itself.  Gene Kim became an expert at file changes and automating an event (such as logging or notification) with his security software business.  By the time DevOps became a common word, he was incredibly well-prepared for the I.T. industry to begin a transformation toward focusing on automating processes upon a file being changed.  No one would disagree that Gene Kim is one of the most well-known figures in the DevOps realm today and will be one of the biggest names in the history of the DevOps movement.   His rise to fame came from expertise of a SEIM product.  This story of Gene Kim corroborates the idea that security is necessarily infused in regular DevOps culture (with no "sec" in the word).

The opinion that DevOps necessarily includes traditional practices of development, operations, QA, and security, is substantiated by the following list (when taking the first one into account with the other individual items):

  • Devops.com shows how QA, development and operations are part of DevOps.  Consider those three areas comprising DevOps with the four links below.
  • Wired.com explains how DevOps integrates security.
  • SANS has a whitepaper that explains how DevOps helps security work.  Specifically pages 5, 10, and 11 of this paper corroborate the view that DevOps inherently incorporates DevSecOps.
  • Ron Wilson who has authored an article on DevOps.com maintains that DevOps includes security in this link.
  • OWASP sponsored a discussion from Helen Bravo that was entitled "DevOps & Security: It's Happening. Right Now."  The slides were published in 2014, you can see them by going to this link.

If you want to read more on the topic of DevSecOps, see one of the following books:
DevSecOps All-Inclusive Self-Assessment
DevOpsSec by Jim Bird, Oreilly Press 2016
Hands-On Security in DevOps
Securing DevOps: Security in the Cloud

What Is a “data swamp”?

Question
What is a "data swamp"?

Answer
A data swamp is best defined as a severely degraded data lake.  The term data swamp connotes poor governance and negligent management that caused a data lake to gradually lose its value.  A data swamp is data lake that was once useful but through negligent utilization can no longer be used by even highly talented analytics professionals.  Data swamps can be improved and restored to regular, functional data lakes.  There is a gray area between a poor data lake and data swamp that is degraded with a low degree of severity. 

IBM suggests that there is a continuum from data swamp (least valuable) to data lake to data reservoir (most valuable).  Page 122 of this IBM document uses all three terms in a way that alludes to a continuous series of these three data concepts.  IBM's website refers to a "well-managed and governed data lake" being the same as a data reservoir.   Gartner makes a distinction between data lake and data reservoir too (infocus.emc.com).  Other companies besides IBM and Gartner refer to "data lake" and "data reservoir" interchangeably.

According to an article on TeraData's website that was published in late 2016, the biggest mistake that people are making  with data lakes is "poor governance."  Expect to hear or read the phrase "data swamp" as you continue to work with big data.  For further information about data swamps, data lakes, and data reservoirs, see the links below:

What Is the Difference between Erasure Coding and Negative Coding?

Question
Erasure coding versus negative coding: what is the difference?

Answer
In the verbose output of a Hadoop or mapred job, you may have seen "INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path:"

Erasure coding is a fault-tolerant storage mechanism.  Like replication the usable capacity of the disks participating in such a mechanism is not fully utilized as in a RAID 0 configuration.  (The aggregate amount of storage of disks in RAID 0 is the sum of the individual disks' capacity.)  Also like replication, one disk, and potentially more, could fail while the data written to the storage system with erasure coding is still safe.  To learn more about erasure coding, see this link from the Storage Networking Industry Association.

Negative coding is a programming technique where code is rewritten to preserve its functionality with fewer lines of code.  A benefit to programmers is that the code base is more concise.  With fewer lines of code, the program is potentially more readable and less intimidating to scrutinize.  A benefit from a computing hardware perspective is that the code base has a smaller footprint in storage (as it resides in a code versioning repository) and uses less RAM when in memory.  For more information on negative coding, see this external link.

"The real hero of programming is the one who writes negative code." -Douglas McIlroy
This quote was taken from here. McIlroy advanced the use of pipes at Bell Labs (the origin of Unix); to read more about this, see this link. He also championed Unix philosophy of having modular components in the 1960s (page 79 of this pdf).

Are There Similarities or Differences between Negative Coding and Throw-away Prototyping?

Question
What is the difference between negative coding and throw-away prototyping?

Answer
"Plan to throw one away..." -Frederick P. Brooks, Jr. (Taken from page 266 of The Mythical Man-Month).

Both negative coding and throw-away prototyping involve eliminating lines of code.  But this is the only similarity they have.  Negative coding is contrasted from throw-away prototyping in that the former is a good practice for production systems and long-term use whereas the latter is designed to be temporary.  Negative coding is the rewriting of code to make it more concise.  Rather than adding lines of code the programmer removes them (thus the word negative) while retaining a program's functionality. 

Throw-away prototyping is the development of a disposable program or programs to have a customer communicate feedback.  Throw-away prototyping is for proof-of-concept ideas and to work closely with the customer for crystallizing the requirements.  To take a customer's requirements and design a superficial solution with a high-level language such as Python while being aware that the code will never be used beyond a verification test is an example of a throw-away prototype.  Throw-away prototyping is the process of creating a symbolic application in the earliest stages of a software design project with the intent that this initial program is both disposable and merely symbolic (unsuitable for a production system due to performance or security reasons).  An illustrative representation of a program can be pithy in that it is the basis of communicating an understanding of many small requirements by a software designer to a customer to confirm or deny satisfaction with the prototype. 

Iterative development may involve a series of disposable experiments.  Planning on discarding a program allows developers to forgo time-consuming security, performance and "best practice" concerns initially.  It also invites greater candor from the customer as the illustrative "application" is intentionally abortive.  Developers will then invest time from the beginning of their real solution to handle the multitude of tedious obligations only after the feasibility of the project has been confirmed.  Thus they can begin a full software development lifecycle in a language such a C, replete with proper testing given the confidence bestowed by the customer's approval of the illustrative prototype.  The first line of code will have been written with the specific requirements in mind that will ultimately be able to handle a production load robustly and securely.  To learn more about throw-away prototyping, see page 136 of The Internet Encyclopedia, Volume 3 by Hossein Bidgoli published by John Wiley & Sons in 2004. It can be inappropriate from a business perspective for developers, in some instances, to present a throwaway to a customer (as described on page 116 of The Mythical Man-Month).

Given the rapid profusion of Docker containers, the disposable concept is very popular in the software world today. "[Docker's] architectural philosophy centers around atomic or throwaway containers." (This was taken from page 3 of Docker Up & Running by Matthias & Kane, O'Reilly Press, 2015.)

Throw-away prototyping is also referred to as revolutionary prototyping. (Slide 31 of this presentation confirms that these terms are synonymous; another source proving their equivalent meaning is here.)  It seems preferable to hyphenate the term "throw-away" compared to the unhyphenated variation "throwaway."  The reason is that two-word, or compound, adjectives that are not well-established nor frequently used should be hyphenated (Grammarbook.com).  Additionally a 2004 reference to "throw-away prototyping" uses the hyphenated version (Revolutionary vs. Evolutionary Rapid Prototyping: Balancing Software Productivity and HCI Design Concerns. Center of Excellence in Command, Control, Communications and Intelligence (C3I), by S.P. Overmyer, published by George Mason University, 4400 University Drive, Fairfax, Virginia).  This 2004 article also states that revolutionary prototyping is a synonym of throw-away prototyping.

Douglas McIlroy, head of research at Bell Labs when Unix was being created, said "[o]f course mass production, in the sense of limitless replication of a prototype, is trivial for software." (The sources of these statements are here and page 79 of this pdf). He also said "[t]he real hero of programming is the one who writes negative code." (This quote was taken from here.) McIlroy advanced the use of pipes at Bell Labs (the origin of Unix); to read more about this, see this link. He also championed Unix philosophy of having modular components in the 1960s (page 79 of this pdf).

How Do You Create Kubernetes Pods with a .YAML File That Use a Docker Image from a Google Container Registry Instead of Docker Hub?

Problem scenario
You have a .yaml configuration file that works with the kubectl command to create a deployment.  You want to change the source image from the hub.docker.com to a Google container source.  How do you configure the .yaml file to obtain the image from a Google container registry instead of Docker Hub?

Your .yaml file has these four lines (among other lines):

  spec:
      containers:
      - image: mysql
        name: mysql

Solution
You have to identify the URL from gcr.io first.  Once you have that, substitute the "mysql" on the "- image:" stanza to this new URL.  Here is how the .yaml file's four lines would look:

  spec:
      containers:
      - image: gcr.io/google_containers/mysql-galera:e2e
        name: mysql

For Future Reference
Assuming the .yaml file was complete and named contint.yaml, you could now create a Cluster with a command like this:

kubectl create -f contint.yaml

To delete the cluster created with such a file, just use a command like this:

kubectl delete -f contint.yaml

How Do You Enter a Docker Container?

Problem scenario
You have a Docker host that is running Linux (the server with Docker containers).  You want to run commands inside a Docker container.  How do you get into a Docker container?

Solution
Prerequisite
You need to know the container ID of a running container.  To find it run this: docker ps -a

Procedures
Run this command:
docker exec -it <container ID> bash

You may want to see this posting.  To view a list of Docker books, see this link.

How Do You Configure SSH Passwordless Authentication between an AWS EC-2 Instance and an Azure Virtual Machine?

Problem scenario
You have a Linux server in AWS and another Linux server in Azure. You want to use scp without a password to transfer files rapidly.  How do you configure passwordless SSH authentication between the two servers?

Solution
1.  Run this command on each server:

ssh-keygen -t rsa -P ""

# Accept the default prompt by pressing enter.

2.i.  Run this command on the AWS instance (replace "ubuntu" with "ec2-user" if the AWS instance is a RHEL server):

cat /home/ubuntu/.ssh/id_rsa.pub

2.ii.  Append the output (of the above command) to this file where "george" is a username on the Azure server:

/home/george/.ssh/authorized_keys

# Create the authorized_keys file if it does not yet exist.  RedHat servers do not necessarily have an authorized_keys file after ssh-keygen is run.   If you have to create it run this command (from the .ssh directory): sudo chmod 600 authorized_keys

3.i.  Run this command on the Azure instance where "george" is a username on the Azure server:

cat /home/george/.ssh/id_rsa.pub

3.ii.  Append the output (of the above command) to this file on the AWS server (replace "ubuntu" with "ec2-user" if the AWS instance is a RHEL server):

/home/ubuntu/.ssh/authorized_keys

# Create the authorized_keys file if it does not yet exist.

4.  Ensure that port 22 is not blocked via an Azure Network Security Group from the external IP address of the AWS EC-2 instance.  To find the external IP address, use the AWS Console and go to the EC2 Dashboard.  Alternatively from the backend you can run "curl http://icanhazip.com".  Do not be alarmed in nmap commands show port 22 is blocked.  Do not be alarmed if you cannot ping the Azure VM from the AWS instance.

5.  Ensure that inbound ports 22 and 80 are not blocked in an AWS Security Group from the external IP address of the Azure VM.  To find the external IP address, use the Azure Portal and go to resources and click on the VM.  Alternatively from the backend you can run "curl http://icanhazip.com".  

(Technically port 80 does not need to be opened.  For nmap results to work from the Azure VM, port 80 needs to be opened.  It is merely one port with one IP address.  You can keep it blocked if you do not need to use the nmap command for troubleshooting.)

6.  From the Azure instance, use this command:

ssh ubuntu@<IP address of server>    # replace "ubuntu" with "ec2-user" if the AWS instance is running Azure

# If you are prompted to continue connecting, choose "yes."

7.  From the AWS instance, use this command:

ssh george@<IP address of Azure instance>    # replace "george" with the username on the Azure instance

# If you are prompted to continue connecting, choose "yes."

# We hope you found these directions succinct yet thorough, and very clear.

How Do You Find Which Docker Images Your Kubernetes Cluster Pods Are Using?

Problem scenario
You want to know the Docker images of the pods running in your Kubernetes cluster(s).  How do you determine this?

Solution
From your Kubernetes server, run this single multi-line command:

kubectl get pods --all-namespaces -o jsonpath="{..image}" |\
tr -s '[[:space:]]' '\n' |\
sort |\
uniq -c

It was taken from this link.

How Do You Deploy a LAMP Stack Application Powered by Kubernetes?

Problem scenario
You want to deploy your own LAMP stack with the power of Kubernetes.  You do not want to rely on official Docker Hub images for the underlying Docker containers.  How do you do this?

Solution
1.  Deploy Kubernetes to AWS.  If you need help to deploy Kubernetes to AWS, see this link.  If you need assistance installing kubectl on any type of Linux (CentOS/RHEL/Fedora, Debian/Ubuntu or SUSE), see this link. 

If you want to deploy it to Azure or Google Cloud Platform with these directions, know that the end product will not be fully functional.  Kubernetes can be fully functional in Azure or GCP (but not with these directions that rely on an Elastic Load Balancer [in AWS]).  These directions below rely on AWS for full functionality.  Using these directions to deploy Kubernetes to Azure or GCP is acceptable for the basic testing of using a Docker image that is not from Docker Hub.  If you still want to use these directions and use a custom Docker image with Kubernetes in Azure, see this link to deploy Kubernetes (in Azure).  See this link to deploy Kubernetes in Google Cloud Platform.  These directions below were designed for AWS to create a load balancer therein. To create worker nodes in AWS (to work with EKS), see this posting.

2.a. Run this command: kubectl get cluster-info

If you see 'error: the server doesn't have a resource type "cluster-info"', do not be alarmed, and proceed with these directions; go directly to step 2b. If you “The connection to the server localhost:8080 was refused – did you specify the right host or port?” when using AWS and EKS, see this posting.

2.b.  From your server with kubectl, in a directory where you can write to (e.g., cd /home/ubuntu), download these four files:

The above files are used for creating a Kubernetes cluster.  They are configured to take a Docker image from a Google repository when creating the Kubernetes cluster.  (The files are modified versions of files originally taken from https://github.com/heptio/example-lamp.) 

One way to obtain the files if your Linux server has access to the internet, is to create this script.  You can call it g.sh, and run it (with bash g.sh):

curl -Lk https://raw.githubusercontent.com/ContinualIntegration/kubernetes/master/googsecrets.yaml > /tmp/googsecrets.yaml
curl -Lk https://raw.githubusercontent.com/ContinualIntegration/kubernetes/master/googphp.yaml > /tmp/googphp.yaml
curl -Lk https://raw.githubusercontent.com/ContinualIntegration/kubernetes/master/googmysql.yaml > /tmp/googmysql.yaml
curl -Lk https://raw.githubusercontent.com/ContinualIntegration/kubernetes/master/googdata-loader-job.yaml > /tmp/googdata-loader-job.yaml

3.  Run these commands if you deployed Kubernetes anywhere except Google Cloud Platform:

kubectl create -f googsecrets.yaml
kubectl create -f googmysql.yaml
kubectl create -f googphp.yaml
kubectl create -f googdata-loader-job.yaml

kubectl get service googweb -o wide  # See if the load balancer gets created after 25 minutes.

If you deployed Kubernetes to GCP, run these commands:

kubectl create -f googsecrets.yaml --validate=false
kubectl create -f googmysql.yaml --validate=false
kubectl create -f googphp.yaml --validate=false
kubectl create -f googdata-loader-job.yaml--validate=false

If you deployed this to a Kubernetes cluster in AWS, the external IP address will appear (e.g., after five minutes from the fourth command).  But if you deployed this to a Kubernetes cluster in Azure or GCP, do not expect to see an external IP address because it will not be created.  If you used Google Cloud Platform's Kubernetes Engine, you can click on "Workloads", "Discovery & load balancing", "Configuration", and "Storage" to see the results of the above commands.

How Do You Configure a Docker Host to Be a Client of a Docker Registry?

Problem scenario
Now that the Docker registry is set up and has been tested, you want the "docker login" command to be able to work from other Docker hosts.  How do you configure another server (a Docker host, as a client) to work with a Docker registry server?    

Solution
1.  Install Docker on the server that will be the client.  If you need directions, see this posting.  Once installed it will be referred to as the Docker host or the client.

2.  Copy the /etc/ssl/certs/ca-certificates.crt from the Docker registry server to the client server that is a Docker host.  For copying the .crt file with scp from the Docker registry server to new Docker hosts that will become the client servers of the registry, ensure that port 22 is initially open.  It can be closed later on.  

You may want to use this from the Docker registry server's back end: scp /etc/ssl/certs/ca-certificates.crt username@dockerclientFQDN:/tmp/ca-certificates.crt  
# where username is a username that can log into the Docker client server and dockerclientFQDN is the FQDN of the Docker client server.

You may not want to use sudo because that may cause scp to not work.  It is advisable to back up the /etc/ssl/certs/ca-certificates.crt on the Docker client first.  Then overwrite it with the file copied above.  It depends on how sensitive the client server is.  If it was just created or in a development environment, you likely would not need that .crt file backed up.

3.  Ensure that port 443 is open between the Docker registry server (inbound) and the client servers (outbound port 443).  You can use nmap to test.  If you are using AWS or Azure, you may need add a rule.  Your network engineer may need to open this port on a firewall that protects the servers.  This step is optional because a "docker login" command will fail if port 443 closed.  It is not that bad to find out late that port 443 is closed.

4.  If the FQDN of the Docker registry does not resolve from the Docker client (e.g., you cannot curl http://FQDNofDockerRegistry or ping FQDNofDockerRegistry because DNS is not configured) update the /etc/hosts file on the Docker client to map the IP address of the Docker registry server with the its FQDN.

5.  Make sure the Docker service is running on the client (e.g., sudo service docker start).  With Red Hat servers, it does not start automatically after it is installed.

6.  The basic way for a Docker client to connect to a registry is to use this command: "docker login http://FQDNofDockerRigstry" # with no quotes.  You will then need a username and password for the next prompts.