SaltStack Technology and Terminology

SaltStack provides for more complex configuration management than Ansible (another Python-based) configuration management tool.  Some people have criticized Salt for having too many new vocabulary words.  Like all complex technologies, they take time getting used to.  To help learn about Salt, I thought I'd provide an overview.

An SLS file is a SaltStack State file.  This file is the basis for determining the desired configuration of the client servers that are called Salt Minions.  A pre-written State file is called a formula in the world of SaltStack.  Just like sodium and chloride can be the basis of other compounds, formulas can be the basis of complex desired state configurations.  Grains, in SaltStack terminology, are data about a Salt Minion. A grain may include information such as an OS type of a Minion server.  The data is generated from the minion.  Pillars are data about Salt Minion servers too.  But pillars are are stored on the Salt Master server.  Pillars are encrypted, and they are ideal for storing sensitive data that should only go to certain Salt Minion servers.  Pillar sls files have data like a state tree (a collection of sls files) except that pillar data is only available for servers that have a given "matcher" type.  

Beacons, in the context of SaltStack, are constant listeners for a condition to be met.  If used properly the beacon can have a corresponding action to be taken from the "reactor system."  A reactor sls file will have a condition and trigger an action because of the beacon listener. 

The first two paragraphs were a combination of original content and content paraphrased from these two links:  Pillar and Highstate.  The final paragraph was paraphrased from this link on Reactors.

Containerization Has Its Advantages Over Virtualization

Containers, such as Docker, communicate to each other through a shared kernel.  Guest virtual machines communicate to each other through the hypervisor or host operating system.  Containers enjoy faster communication as staying within a shared kernel allows for more rapid communication than leaving a virtual machine and going out to a hypervisor (or host operating system) to communicate with another virtual machine.  Containers allow for sequestration of processes and fewer operating systems licenses compared to having a comparable solution with virtual machines.  Virtual machines can separate processes but require an operating system license for every virtual machine.

DevOps and ETL Quiz

Extract-Transform-Load workflows involve considerable architecture including a workflow over a network to take data from a flat file and ingest it into a database.  Automation is one way to manage the ETL support system.  DevOps Engineers commonly support database installations and configurations.  DevOps engineers commonly support continual delivery pipelines.  This automated process (involving automatic deployments) is often similar to automating an ETL process.  DevOps engineering, build and release engineering, automation development, and ETL design are all interdisciplinary fields of information technology.  This is a quiz related to both DevOps and ETL topics.

1.  What is the DevOps tool for databases?

a.  QuerySurge
b.  Beehive
c.  Stratos
d.  DBMaestro

2.  What does mung mean?


3.  What does idempotent mean?


4.  What is the name of the process of actively preparing data for serialization (e.g., data that was not otherwise logically contiguous on disk for a buffer) called?  This process may include modifying data from one programming language or interface so it is compatible with a different programming language or different interface.

a. Almquist variation
b. inmoning
c. scrum transition
d. marshalling

5.  How is an imperative process different from a declarative process?


6. What is a common tool that both ETL Developers and DevOps Engineers use?


7. Which of the following can you not create an AWS Data Pipeline with?

a.  AWS Management Console
b.  AWS Command Line Interface
c.  AWS SDKs
d.  AWS APIs
e.  None of the above

8.  Mesos Clusters cannot work with both HDFS and Digital Ocean?


9. Hadoop YARN cannot act as a scheduler for OpenShift?


10.  Which of the following Apache products can create ETL jobs?

a.  Accumulo
b.  Pig
c.  Stanbol
d.  Lucene

11.  Which of the following is not an ETL product?

a.     IBM InfoSphere Datastage
b.     Oracle Warehouse Builder
c.     Business Objects XI
d.     SAS Enterprise ETL server
e.     Stratos
f.     Informatica
g.     Apache Hadoop
h      Talend Big Data Integration

12.  In Informatica are mapplets only able to be used once without logic?


13.  Which of the tools below are tools designed to aide ETL process testing and validating data warehouses themselves?

a.  QuerySurge by Real-Time Technology Solutions
b.  DBMaestro
c.  Apache Cassandra
d.  Apache Stratos
e.  ETL Validator by datagaps inc.

14.  What is an example of cooked data in the context of ETL/Devops?

a.  Machine-corrupted data (e.g., from disk failure)
b.  Content that was corrupted maliciously
c.  Cleansed data
d.  Intentionally masked data (to hide identities)

15.  What is the technique that divides a table of a database into different subcomponents, such as partitioning columns, to improve read and write performance?

a.  data marting
b.  impedance matching
c.  sharding
d.  redis

16.  What tool allows you to designate when Docker containers process ETL jobs without manual configuration?

a.  Pachyderm
b.  Chronos
c.   Overwatch
d.  emerge-sync

17.  Which of the following can readily be used as a superior ETL platform?

a.  Hadoop
b.  Teradata
c.  Proxmor
d.  Note Beak

18.  There is consensus that small companies should use Informatica or a supported, proprietary ETL tool as opposed to an in-house developed tool.


19.  Which of the following has an open source version:

a.  Talend Integration Suite
b.  Pentaho Kettle Enterprise
c.  CloverETL
d.  All of the above
e.  None of the above

20.  What is a data lake?

a.  A synonym of data warehouse
b.  A buffer of streamed data
c.  An archive of metadata about previous real-time data streams
d.  A pool of unstructured data

21.  What is a data swamp?

a.  A dense data lake
b.  A severely degraded data lake
c.  A synonym of a data warehouse
d.  A pool of unstructured data
e.  An archive of metadata about previous real-time data streams

22.  Snappy is the name of which two concepts?

a.  The REST API for SnapChat
b.  A data compression and decompression library with bindings for several languages
c.  A Linux package management system
d.  An automation scheduler for Informatica
e.  An open source component to migrate SSIS packages to PostgreSQL

23. In a SQL database you have a left table with four rows and a right table with seven rows, what is the highest number of rows that can be returned with an inner join?

a. 0
b. 4
c. 11
d. More than 11

24. Which of the following provide Sqoop based connectors (choose all that apply)?

a. Teradata
c. Talend Open Studio
c. Informatica (modern versions)
d. Pentaho

25. What is a continuous application?

a. The namesake of CA traded on the Nasdaq as CA
b. An application that encompasses data streaming (e.g., ETL processes) from start to finish that adapts itself to the data stream(s) in real-time
c. An application that leverages ETL processing
d. An application receiving continuous integration (or continual integration)
e. An application receiving continuous delivery (or continual delivery)
f. An application receiving continuous deployments (or continual deployments)
g. An application that is always available through fault tolerance and load balancing

26. DevOps expert Gene Kim got his start with a security product called Tripwire, known for its emphasis on changes to files. There is a tool that keeps track of changes to a database. Which product below concerns itself with tracking changes of database schemas?

a. MongoDB
b. DBVersion
c. Databasegit
d. Liquibase

27. Which product enables you to quickly make copies of SQL Server databases for your Test, QA or development environments? Choose the most accurate answer.

a. Canonical's Juju
b. RedGate's SQL Provision
c. Apache Hamster
d. Apache Numa

28. The SQL Server database back ups are not working or you get false positives that your back up solution is successfully backing them up. What solution should you for a practical back up solution?

a. Write you own PowerShell script that backs up the database
b. Implement AlwaysOn Availability Groups
c. Implement RedGate's Toolbelt
d. Implement Apache Impala

29. Which AWS tool can perform ETL jobs? Choose two.

a. DMS (Database Migration Services)
b. DMS (Data Manipulation Service)
c. Glue
d. Cognito
e. Federation

30. Test Kitchen works for which of the following?

a. Chef
b. Terraform
c. PowerShell DSC
d. All of the above

*** See answers to quiz. ***
DevOps Books

How do you use the source keyword in Puppet’s DSL (when writing a manifest)?

When writing a Puppet manifest you can use the "content" reserved word.  You then have quotes around the actual text content of this file right in the manifest itself.  This works for a file that you want to create on a Puppet Agent server as long as the content is roughly one line of text.  But for a binary file, this will not work (as it cannot appear in the manifest).  The "source" reserved word allows you to point to a specific file on the Puppet Master server.  The Puppet constructor has three slashes after the colon.  Here is an example of the "source" reserved word and the Puppet constructor:

source => puppet:///modules/goodFolder/

What is important to know is the following non-obvious facts.  

1) The corresponding goodFolder must actually have a subdirectory named "files."  This directory "files" is not explicit in the source field declaration.  

2) Puppet.conf must have a main section that tells Puppet where to look for the "modules" subdirectory.  

default_modules = /etc/puppet/modules/

3) The path to the modules including the subdirectories (named goodFolder in this example and files itself) must have permissions that allow the Puppet process to access them.  This is true of the file too.

4) Some subdirectory besides "files" must be in "modules" to house "files."  The goodFolder in the example satisfies this.

Once you know these four facts, you can use the valuable source reserved word.  On a final note, if the destination of the file in the manifest is configured for a heterogeneous operating system relative to the Puppet Master OS (e.g., c:/temp/ is the destination of the manifest file transfer yet the Puppet Master is running on Linux or the destination is /tmp/ and the Puppet Master server is running on Windows), you may get an ignorable error when you compile the manifest.  But this is only true if your manifest doesn't specify nodes or classes to ensure Puppet does not attempt to apply a manifest for a non-applicable OS.  The caching of the catalog will find that the path "must" be fully qualified when it compiles.  Compilation will be successful and the manifest will run when the Puppet Agents connect.  So don't be surprised when this ignorable message is displayed.

Update on 12/28/16:  For troubleshooting manifests that are not doing what you expect despite no messages or few errors in the logs, see this posting.

Six Puppet Configuration Tips

Deploying Puppet Master and Puppet Agents for the first time can involve a significant amount of troubleshooting.  In this post, I want to review six miscellaneous points that may arise.  These are somewhat random, but they can serve in the rudimentary stages of quickly getting a proof of concept established.

1.  With a default configuration, Puppet Master on Linux will run manifests with only one name in only one location: /etc/puppet/manifests/site.pp

Many DevOps engineers do use manifests with different names.  However, absent special configuration, this the only file name and location that will work.

2.  A reality of efficient I.T., particularly in non-production environments with open source technologies, is to ignore certain error messages.  If you compile a manifest (e.g., with the puppet agent -t site.pp command), you may be able to ignore this subsequent error if it pertains to the Puppet Master FQDN:

"Error: Could not retrieve catalog from remote server: Error 400 on Server: Could not find default node or by name with ..."

3.  To find an error in a Puppet Manifest, try this command:  puppet parser validate nameOfManifest.pp
It will find errors such as upper case class names.  But it will not find an error such as a time when a resource declaration uses "requires => ..."  The correct Puppet DSL reserved word for a given resource declaration is "require" with no "s."

4.  Network Time Protocol (ntp) must be configured and running on the Puppet Master and Puppet Agent servers.  The time difference between a Puppet Master server and Puppet Agent node may seem insignificant to an individual person.  To see if ntp is running, try this:

sudo ps -ef | grep ntp

If ntp is not running on a Puppet Agent, manifests will appear to compile and run without errors on either the Puppet Master or Puppet Agent server.  Here is how to get ntp to automatically start. 

First, go into the /etc/crontab file.  Second, add this entry:  *  *  *  *  *  root service ntpd start
Third, save the file and exit.  Now ntp will start every minute regardless of who is logged in.

5.  /etc/puppet/puppet.conf can, by default, have the same content on the Agent nodes as the Master nodes.  One entry should be like this in the [main] section:

server = FQDNofPuppetMasterServer

This tip clarifies how multiple servers may have the same file and how it relates to the inter-server configuration of Puppet.

6.  Problem scenario:  Facter does not pick up the correct value from a Puppet Agent node with Windows Server.  
Solution:  Go to the Puppet Agent node.  Open PowerShell.  Run this: puppet facts

If the result says something like "no default action," go to the Control Panel -> Uninstall Programs.  See if Puppet is installed.  If it is, verify it says "Puppet Agent."  Puppet Master could be installed, but that will not give you facter.

Update on 12/28/16:  For troubleshooting manifests that are not doing what you expect despite no messages or few errors in the logs, see this posting.

Ansible Managing Windows Servers

While as of right now, there is not a great way to install Ansible on Windows servers (because you have to install cygwin).  Ansible running on Linux can readily configure Windows servers and push files down to them.  There are some things to look out for when setting this up.  It is not overly documented on Ansible's website.  Some documentation (on various websites) tells DevOps engineers (or the professional using Ansible) to use a windows.yml file in a group_vars directory.  This file is supposed to contain the authentication information.  There is scant reference to how this windows.yml file is later invoked or where the group_vars directory should be. What some devops engineers use is local user accounts on the Windows servers with an inventory file (on the Linux server) to configure communication to the Windows server.  They do this because Kerberos and other alternatives can be convoluted to set up.  SSH is often not used because it would require Cygwin on the Windows server or Moba SSH (software that costs over $40 per license).  (To install Cygwin for free on Windows 2016 Server, see this posting.)  To avoid using the windows.yml file in a group_vars directory, and to avoid using Kerberos, this article explains how to use Ansible on Linux with an inventory file that enables the management of Windows nodes via a local Windows user account.

This inventory file, named "hosts," is often located on the Ansible server in the /etc/ansible/ directory.  In this file (/etc/ansible/hosts) is a section called "windows:vars."  You may have to create such a stanza if it does not already exist.  The syntax of this section is very particular for local Windows accounts (with seemingly small errors causing unlogged errors and preventing Ansible from working properly). 

ansible_ssh_user = jdoe
ansible_ssh_pass = P@SSWORD
ansible_ssh_port = 5896
ansible_connection = winrm

Note the following five things: 
1.  Quotes around the username, password, port number, and connection type are optional.  Ansible playbooks can use the [windows:vars] section with or without quotes.  But be careful: this section is prone to error despite robustly handling quotes and no quotes.
2.  There are equals signs "=" and no colons ":".
3.  The second stanza is "ansible_ssh_pass."  There is no "password" in the above.  You may get a "401 Authorization" error if the word "word" is in the "ansible_ssh_pass" field.
4.  The last stanza "ansible_connection" has no ssh in it.  Ansible will try to use SSH by default.  You may get a "Banner exchange timeout" error if there is a "_ssh" in the ansible_connection field.
5.  The local account on the Windows machine (here jdoe) must be a member of the local server's "Administrators" group.

Remember that pywinrm needs to be installed on the Linux server.  To test if it is installed, enter the command prompt for python by typing python.  Then try this:  >>> import winrm
If you are taken to the next prompt, then pywinrm is installed.  If there is an error, it needs to be installed.

For Ansible to work on CentOS, you need to have these three Python modules installed: xmltodict, pywinrm, and isodate.  If you get the source files from the Internet, you can unpack them by doing this: tar -zxvf nameOfPackage.tar.gz
This command will create subdirectories where the file was.  You can choose where the subdirectory is place by adding the -C flag.  Here is an example:
sudo tar -zxvf nameOfPackage.tar.gz -C /bin/pythonPackage/
You can then change directories into the directory that was created from the above command.  Finally enter these two commands in sequence:
sudo python build
sudo python install

Python modules usually follow this model (with a file that takes a "build" parameter initially and an "install" parameter subsequently).

For Ansible playbooks to work against a Windows server, several PowerShell commands must be run on the Windows server one time.  Here is a link to the script that makes those changes.  You may find that two changes need to happen to this script for it to work properly:  One, the first five lines should be commented out.  Two, add this line of code to the script at the top without it being commented out: $verbosePreference = 'continue'

Here is what the first several lines of code should look like:

Param (
[string]$SubjectName = $env:COMPUTERNAME,
[int]$CertValidityDays = 365,
$CreateSelfSignedCert = $true )
$verbosePreference = 'continue'

If the script doesn't run, try using this command: get-executionpolicy
If the result is "Restricted," you may need to use this PowerShell command to run the script: set-executionpolicy remotesigned

These directions should generally apply to different Linux / Windows / Ansible / Python version combinations.  They are particularly relevant to CentOS 7.x, Windows Server 2012, Ansible 1.9.x, and Python 2.7.x.

How Do You Set Up Passwordless SSH to a Windows Server?

Problem scenario
Are are asking yourself "why am I being prompted for a password when my SSH keys were set up correctly?"  When the contents of the .pub file (the public key) are placed into the authorized_keys file (in the /home/jdoe/.ssh/ folder of a client machine), the user should be able to SSH over to the server with no password -- unless the SSH key was generated with a passphrase.  Assuming the SSH key was generated with an ssh-keygen command and no corresponding passphrase was entered at the time of creation, the login should happen with no password.  However, a password may be required if the authorized_keys file has excessively high privileges (e.g., chmod 777 authorized_keys or the file appears as having these settings "-rwxrwxrwx").  When an administrator is setting up SSH keys between two servers, he may find that being root makes his job easy.  When the users try to SSH without passwords, they may get prompted to enter a password.  An administrator may think to open up the authorized_keys file's permissions (as the users don't have as high of privileges).  Granting the user read, write and execute writes will not fix the problem.  Setting the permissions of the authorized_keys file to 400 (e.g., chmod 400 authorized_keys) will allow the file to provide passwordless authentication for the users.

For OpenSSH on Windows, the same principle applies.  Too permissive of permissions for an authorized_keys file or its parent .ssh folder will make the user enter his/her password. 


To avoid a prompt, do the following on a Windows server with OpenSSH:

1.  Right click the authorized_keys file (usually in C:\Users\jdoe\.ssh\) and go to Properties.

2.  Go to the Security tab.  Go to Advanced.

3.  Click "Disabled inheritance."

4.  A pop up menu with two options will appear.  Click the option for "Convert ..."

5.  Click a User and Group other than "Administrators"

6.  Click Remove

7.  Continue until you have removed everything but "Administrators"

8.  Move up in the directory so you can right click the .ssh folder.

9.  Go to the Security tab.  Go to Advanced.

10.  Click "Disabled inheritance."

11.  A pop up menu with two options will appear.  Click the option for "Convert ..."

12.  Click a User and Group other than "Administrators"

13.  Click Remove

14.  Continue until you have removed everything but "Administrators"

For a book on OpenSSH, you may want to try this one.

Moving a Windows .txt file to a Linux Server

Problem scenario:  When you move a Windows .txt file to a Linux server, new characters can be introduced.  For example, the content of the file can have a "^M" (with no quotes) at the end of every line.  Sometimes tr, sed, and awk won't work to remove this new jibberish (extraneous characters).  Moreover, sometimes the substitute command in vi will fail to do anything about these extraneous characters (^M).  How do you eliminate these extra characters at the end of every line? 

Solution:  Let's assume the file is named foo.txt.  Run this command:
perl -p -i -e "s/\r//g" foo.txt
It's as simple as that assuming you have write permissions to the file foo.txt.


Problem scenario:  When running a Bash script, you receive this message:

" /bin/sh^M: bad interpreter: No such file or directory"

Root cause:  The script was being housed on a Microsoft Windows file share or Windows folder.

Solution: First, install dos2unix.  Secondly, run it like this where is the script that is having the problem:


Apache Software Foundation Has Imperfect Websites

"Documentation plays a great role in the maintenance of a project."  (Page 19, Learn Apache Ant,, 2014) 

The open source movement is no exception.  As much as we admire The Apache Software Foundation, we are disappointed that they don't correct errors. We've submitted the first two errors below to them.  However, mistakes remain. Here are some errors to help other people who may be confused:

#1 was corrected.  (We removed the error from this page.)

#2 Error on web page for Apache Ant
Specific location:
What is wrong:  There are five broken links on this page.
The links to Antiplate, BuildMonkey, Leafcutter*, JAM - JavaGen Ant Modules, Savant are all broken.

* You have to confirm a security exception.  Professionals may not be allowed to do this from a corporate network.

#3  Error on web page for Apache Ant
Specific location:
What is wrong:  The "Jrun Ant Tasks" link ( redirects to ColdFusion. The "Surround SCM" link is broken.

#4  Web page on Apache Ant
Specific location:
Content: "Best-Practices Profile of Apache Ant at Sun's Dot-Com Builder
Sun has released an introductory article on Apache Ant on their Dot-Com Builder site on May 30 2001. See"
What is wrong:  The web page redirects to a generic Oracle web page.

#5 was corrected.

This post was last updated on January 1, 2018.

As of 9/2/16 there were two other problems.  We will post them below for posterity. 
#1 Error on web page for Apache Tomcat
Specific location:
Content: "Warning:  If Apache HTTP Server and Tomcat are configured to serve content from the same filing system location then care must be taken to ensure that httpd is not able to serve inappropriate content such as the contents of the WEB-INF directory or JSP source code."
What was wrong:  Filing system?  It is now fixed.

#5  Web page on Apache Maven
Specific location: used to have a broken link.
Content: Sonatype Nexus OSS (open source)
What was wrong:  There was a link that did not work.  It is now fixed.

What To Do When Linux Files Are Read Only

Sometimes you cannot edit and save a single file in Linux.  For example, if you boot into maintenance mode after the /etc/fstab is corrupt, you cannot edit the very file that is causing the problem.  To overcome this problem, you will need to log in as root and issue a command.  You will need to manually edit the /etc/fstab.  To do this in maintenance mode you need to issue this command from the prompt: 
mount -o remount,rw /
After this, you will be able to edit the /etc/fstab file.  Hopefully you will know what is wrong or you have a backup.  Be careful when running scripts that automatically update this file.  If the scripts haven't been tested or cannot handle exceptions, the /etc/fstab can be modified in unusual ways.