Troubleshooting DSC for Configuration Management (Problem and Solution)

Background:  DSC is a toolkit of PowerShell cmdlets that enable you to do configuration management.  It can allow you to do CM tasks with Windows and Linux machines (https://msdn.microsoft.com/en-us/powershell/dsc/LnxGettingStarted).

Problem/Scenario:  You are trying to use this PowerShell command (e.g., to test your Desired State Configuration tool):

Enter-PSSession -ComputerName goodServer -Credential jdoe

# where goodServer is a Windows server on the network and jdoe is a local user account on the goodServer machine.
# There may be a pop up for the password associated with jdoe.  You may enter the correct one.  But you are encountering an error like this:

"Enter-PSSession : Connecting to remote server goodServer failed with the following error message : WinRM cannot process the request.  The following error with the errorcode ... occurred while using Kerberos authentication: There are currently no logon servers available to service the logon request.  Possible causes are:
-The user name or password specified are invalid.
-Kerberos is used when no authentication method and no user name are specified.
-Kerberos accepts domain user names, but not local user names.
-The Service Principal Name (SPN) for the remote computer name and port does not exist.
-The client and remote computers are in different domains and there is no trust between the two domains.
-After checking for the above issues, try the following: -Check the Event Viewer for events related to authentication."

The event viewer has no leads for this problem.

Solution:
Prerequisite: You are logged on with local credentials to a Windows server that happen to be the same username and password as the credentials of the remote server ("goodServer" in this example).
1) Open PowerShell as an Administrator
2) Run this in PowerShell:

Set-Item wsman:\localhost\client\trustedhosts *

3) Respond "Yes" to the two following prompts.  The problem should now go away.
------------
See this: https://sqlbelle.com/2015/02/09/installing-sql-server-using-powershell-desired-state-configuration-dsc/

Attached files taken from: https://www.youtube.com/watch?v=bz3D5ciOdgo
Start-DscConfiguration -Path C:\Users\Mike\Documents\SqlServerInstall  -Wait -Verbose

What does the I.T. term “bootstrap” mean?

Bootstrap

Definition 1 (transitive verb) of bootstrap:  To turn on a computer so that the operating system is completely functional.  Source:  The fifth definition of "bootstrap" in Dictionary.com is specific to "computers." 

Definition 2 (transitive verb) of bootstrap:  To intentionally initiate multiple subprocesses (especially a primarily automatic sequence of often incremental subprocesses), as in a batch execution, from a single file or action.

Examples of definition 2: 
"Between the advent of handy tools, like Chef and Puppet, and virtual machine infrastructures, like VMWare and AWS, I feel like there has been a great debate about how to bootstrap machines." This quote was taken from Devops.com.

"In the following, I want to highlight how to create an Angular service that bootstraps the application with data defined in an ASP.NET MVC back-end."  This quote was taken from a blog that was up in 2016 (https://blog.mariusschulz.com/2014/03/25/bootstrapping-angularjs-applications-with-server-%20side-data-from-aspnet-mvc).

Definition 3 (noun) of bootstrap:  A software application for designing websites developed at Twitter.  Source: TechTarget.

Definition 4 (transitive verb): To provision a server and install an agent on the server.

Definition 5 (adjective): A type of DNS server that initially allows other DNS servers to be found. DNS will work after the initial resolution of a hostname of a regular DNS server identified without an IP address. See this posting for more information.


To elaborate on definition 4, a configuration management tool's master server may control the server with this agent. This previous sentence and definition 4 are according to page 26 of Terraform Up & Running.

Background Commentary and Research Notes of The I.T. Term "bootstrap" and "bootstrapping"

  • It could denote the process of turning on a server.  It would connote the details of booting a server involving the initiation of a self-sustaining process from the time it power is turned on until the server begins to function. (1) Relevantly, the bootstrap loader (which is executable code) of a server is stored in the MBR. (2)  The bootstrap loader loads the operating system during power up (and puts the kernel into memory according to page 1240 of A Practical Guide to Fedora and Red Hat Enterprise Linux by Mark Sobell).  According to Techopedia a bootstrap loader is a synonym for boot loader or boot manager.  Therefore one definition of the verbal phrase "to bootstrap a server" is "to turn on a server with a properly configured operating system."  You can power on servers with no hard drives.  The POST and BIOS will go through their processes.  The server will not function properly because it cannot boot without an operating system
  • A separate definition of "bootstrap a server" would be to "install an application with any necessary dependencies." When an installation process "bootstraps" something else it initiates on an as-needed basis a dependency-filling subprocess.  However it would be less likely that you would semantically "bootstrap a server" with this alternate definition and more likely that you would observe or run an .exe file that bootstraps an application installation.  To see an example of this separate definition of "bootstrapping" view this link from Microsoft that is for installing Office 2000.
  • In the past tense or as a adjective describing a server, "a bootstrapped server" could refer to a self-configured or automatically configured server. (3)  
  • Bootstrap can mean to configure a server to participate as a node in a cluster (e.g., a Consul cluster). (4)
  • To bootstrap a server in Chef is to make it a Chef client via installation and configuration of relevant media. (5)
  • For web applications there is a bootstrapping process.  This refers to a main file being a central point for joining together other files, initiating dynamic content (e.g., RSS feeds) connections and with other technologies (e.g., CSS). A definition from another source is 'In the context of PHP development, it also means funneling all web requests through a single script that performs the bootstrapping process, also called "front controller."'  To see an example of the term "bootstrapping" (in the context of web technologies) that can itself be manual or automatic in the context of AngularJS view this old link (https://blog.mariusschulz.com/2014/10/22/asynchronously-bootstrapping-angularjs-applications-%20with-server-side-data) or read this StackOverflow answer.  
  • "Bootstrapping" even in strictly I.T. vernacular has multiple definitions.  For further reading, you may want to see this link.

How Do You Troubleshoot the Docker problem “Error response from daemon: Cannot start container”?

Problem scenario
Sometimes you try to start a Docker container but there is a problem.  For example you try:
docker start <containerID>
But you receive this: "Error response from daemon: Cannot start container <containerID>: failed to create endpoint <name> on network bridge: ip tables failed: iptables --wait -t nat -A DOCKER -p tcp -d 0/0 --dport 80 -j DNAT --to-destination x.x.x.x:80 ! -i docker0: iptables: No chain/target/match by that name."

Solution
cd /var/lib/docker/network/files/
ls -lh > /tmp/forposterity
mkdir /tmp/backupdir
mv /var/lib/docker/network/files/ /tmp/backupdir
systemctl restart docker
 #change this command depending on your distribution and version of Linux to restart Docker services

How To Port Forward (redirect traffic destined for an IP address to a specific port)

Scenario:  On a Linux server, it can be useful to send traffic destined to a certain IP address to a different port on the server.  The listening service could be unique insofar as its port number has been designated.  The listening service could be a Docker container or a guest virtual machine.
Method:  iptables -t nat -A PREROUTING -i eth3 -p tcp --dport 80 -j DNAT --to 91.91.91.91:81
Explanation:  The interface receiving the HTTP requests (port 80) is eth3.  80 is the port that the server can listen on (for packets destined to eth3).  The routing that this command will produce will be to redirect the traffic to 91.91.91.91 over port 81.   Change the interface name (eth3), the d(estination) port value, the IP address or the final port number as needed.  This is an inbound rule so there are, in a way, two destination ports (80 for listening and 81 for somewhere else on the server).  For future reference, the --sport flag is a designation of a source port for IP tables commands.  NAT (network address translation) can work with mapping two different IP addresses or with mapping sockets (IP addresses bound to port numbers).

Saving a Docker Image and Using It On A Different Server

Problem Scenario
Sometimes the "docker save" command does not work as you would expect.  You run the command on a Docker host, but it does not work.  You might even try the -o flag or the --output=/tmp/destinationFileName.tar option.  But the response may be an surprising error "Cowardly refusing to save to a terminal. Use the -o flag or redirect."  You want to copy a Docker image from one machine to another computer but problems get in your way.  How do you solve this problem so you can transfer a flat file to another server and use the Docker image?  In other words, how do you do the most basic task with Docker, how do you bring a copy of a Docker container to another machine (another virtual server, a different Docker host)?

Solution
Prerequisites
You must have Docker installed.  If you need help installing Docker, see this posting.

Procedures
The redirect in the error is the right clue.  This solution assumes you have Docker installed on the different server. 

From the Docker host, use these three steps to copy a Docker container and place the container on another machine (server or host).

Step #1:  This command should work:
docker save repositoryName:versionName > /tmp/destinationFileName.tar

Alternatively, you could try this command:
docker save ImageName > /tmp/imagename.tar
# You could find the ImageName by running "docker ps -a" and looking at the results.

Step #2:  The above command will save the image as a regularly accessible flat file with the name destinationFileName.tar.  To use it on the destination Docker host, transfer it to that server.  Use scp or sftp to do the file transfer to a different server. 

Step #3:  Then use this command (assuming the location of the file on the destination server is /tmp/) on the different host:  
​docker load < /tmp/destinationFileName.tar

By the way, there are numerous books on Docker.

How To Import A Copy of An Existing GitLab project

Problem scenario:  You want to copy a GitLab project from one instance of GitLab to a new instance of GitLab.  The Git repository you want to copy to a new GitLab server is not presented via the git://, http://, nor https:// protocols.    

Prerequisites:  You have root access to the back end of the server with the GitLab.

Method of Solution:
1)  Go to /var/opt/gitlab/git-data/repositories/root/<nameOfProject>.git  

2)  Copy it to a staging area of the destination server.

3)  Make sure every user has logged off the destination GitLab server.  This avoids confusion of shutting things down when someone is trying to check code in.

4)  Log into the web UI of the GitLab instance that is the destination of this copy task. Create a new project with the name <nameOfProject> (with no .git extension).  

5)  Go to the back end of the GitLab server that is the destination of this copy task.  As root run this command:
rm -rf /var/opt/gitlab/git-data/repositories/root/<nameOfProject>.git  

6)  Copy the file in step 2 to /var/opt/gitlab/git-data/repositories/root/

7)  Run this command:  chmod git:git /var/opt/gitlab/git-data/repositories/root/<nameOfProject>.git

8)  Stop the GitLab service.  Restart it.

How To Enter the Web UI of Gitlab CE without Setting Up Its Backend Email

Problem scenario:  When you bring up the web UI for GitLab CE (Community Edition) for the first time, you are prompted to enter a new password twice.  This password will be for the admin@example.com username.  If someone else set it up and failed to provide you with the username, and the back end email has not been configured, follow these directions.
Prerequisite:  You must have root access.
Solution:  As root, enter an interactive console (another set of prompts) and change the default user's password:

1) gitlab-rails console production
2) user = User.where(id: 1).first
3) user.password = 'ciNewPassword'
4) user.password_confirmation = 'ciNewPassword'
5) user.save!

#Remember to change ciNewPassword in steps 3 and 4 to the new password you want.

Mostly taken from:
http://doc.gitlab.com/ce/security/reset_root_password.html

Having Two Docker Containers Share A Directory on the Host

Goal: You want two Docker containers to use the same file share on the Docker server from time to time.  

Problem/Error blocking goal:  In a Docker container, when you try to change directories to a directory that is on the Docker host (e.g., the docker container was created with the --volume flag), and you get an error "bash: cd <directoryname> permission denied," you have to run two commands to fix it.  An alternative problem is that you can cd into the directory, but when you try to list the files, you get "cannot open directory: Permission denied."  The same solution applies.

Solution:  From the Docker server, run these commands:
     su -c "setenforce 0"
     chcon -Rt svirt_sandbox_file_t </path/to/directoryname>

#where </path/to/directoryname> is the full path of the directory and its name

If you are running CentOS/RHEL/Fedora, you can avoid this problem after a reboot by doing the following on the Docker server itself:
1)  Create /etc/profile.d/custom.sh
2)  Provide these three lines as its contents:
#!/bin/bash
su -c "setenforce 0"
chcon -Rt svirt_sandbox_file_t </path/to/directoryname>

#where </path/to/directoryname> is the full path of the directory and its name

How To Fix a Docker Container That Is Giving An Error ‘exec “bash”: executable file not found in $PATH’

Normally you can enter a Docker container with this command:
docker exec -it <containerID> bash
It is possible you receive an error that says 'exec "bash": executable file not found in $PATH'
The root cause could be related to insufficient disk space on the Docker host.  Delete files or otherwise make room (e.g., add disk space).  Then stop the container with this command: docker stop <containerID>
Finally, restart the container with this command: docker start <containerID>
Now you should be able to enter the container.

How Do You Solve a Yum Install/Update Error That Says “Failed to connect to … connection refused”?

Problem:  When trying to use yum commands with a repository on your network, you receive "Failed to connect to x.x.x.x:80; Connection refused" (where x.x.x.x is the IP address of the yum repository server).  What are some possible solutions to this problem?

Potential solutions:
1)  Verify the Linux server with the repository has a valid repository created.  Use "createrepo <directoryToRPMfiles>"

2)  Verify Apache's httpd.conf file has a DocumentRoot value for the directory that was used with the createrepo command above.  This command can help you find it:
     cd /; find . -name httpd.conf | xargs cat * | grep DocumentRoot

3)  Verify that Apache is running: ps -ef | grep apache
These commands may help you start apache depending on your distribution of Linux:
     a) systemctl start httpd.service
     b) service apache2 restart
     c) apachectl start

4) Verify the firewall is not on.  Use ps -ef | grep firewalld to see if it is running.  If you are allowed to disabled the firewall, turn it off with "service firewalld stop".  Only do this if you know you have enough security measures in place for this to not cause a problem.  Turning off a firewall can be a bad idea; be sure other parties involved approve of this change.

5)  Use ping to establish connectivity.  This will not test port 80 or any port for that matter.  You could use "nmap -p 80 x.x.x.x" (where x.x.x.x is the IP address of the server with the yum repository).  If an intermediate firewall was installed that blocks port 80, that may explain the problem.  traceroute is another Linux utility that may be of some value here.