How Do You Troubleshoot a cronjob That Indirectly Uses Linux Programs (e.g., Executables)?

Problem scenario
You have a Python script that runs successfully when your user runs it.  You have a *nix crontab that calls this Python script at a scheduled time.  The script mostly works, but one part fails when the crontab launches it (automatically).  This problem does not happen when it has been run manually.

The part that fails involves a Linux command via a Python subprocess.check_output() invocation in the program.  That is, the part of the Python program that fails is the portion that invokes a Bash command.  You have tested the crontab to determine what user runs the jobs (by using "* * * * * /usr/bin/whoami > /tmp/log.txt").  You have found that the user is the same as the user who can run the script manually.  When that user runs it manually, there are no problems.  Why will the crontab not correctly run your Python script when it appears to be run in the context as the user who runs it manually and successfully?

Possible Solution #1 (For Maximum Learning if You Are Interested)
Find the line that does not work.  Examine if an executable file (or relevant file) relies on an environmental variable for PATH.  

For Detailed Analysis
One way to see the difference between the environment variables of the crontab execution (to fully appreciate the context of cronjobs) is to set up a cron job like this:

* * * * * /usr/bin/env > /tmp/logenvironment.txt

Compare /tmp/logenvironment.txt to the output displayed by your interactive execution of env (from the Linux command prompt).  The "PATH" variable may be different.  You may need to rewrite the Python script to call the full directory path of the executable (assuming calling an executable is the part where the problem is).  This should not be a problem when the user manually runs it interactively.  This should solve the problem of having different environment variables when the crontab runs the Python program contrasted from a manual run of the same program.

Possible Solution #2 (The Quick and Dirty Solution)
To fix the problem you may want to do the following "quick and dirty" solution. This is not a recommended practice as it may add excessive extra environment variables in a script, but it will resolve the problem. (It is fast because it can be difficult to ascertain which environment variable is necessary for the Python script to run successfully.) Another downside to this solution is that it makes the cronjob slightly more complex to maintain because you will have a Bash script to maintain just to run a Python program.

1. Identify the user account that can run the Python script interactively (or manually) without a problem.
2. Then have write a Bash script to run the Python program. You may want to write it like this:

env > pythonkicker.sh

3. Edit pythonkicker.sh to place at the very bottom a stanza like this:

/usr/bin/python yourprogram.py

This way you are sure that the environment variables are set whenever pythonkicker.sh is run.

4. Now have the cronjob call pythonkicker.sh instead of the Python script directly.

How Do You Use TLS with Nginx?

Problem scenario
You know how to use TLS with Node.js because of this article.  You want to use TLS with Nginx to test it out with a regular web browsing session.  You want to implement TLS without Node.js.  How do you set up TLS (without paying a certificate authority and without Node.js)?

Solution
This assumes that port 443 is not blocked from the workstation to the Nginx server.

1.  Install Nginx on Linux.  As an example, we'll install Nginx on an Ubuntu server in AWS (with a security group that allows inbound connectivity on ports 80 and 443 from the workstation).  Run this command from the Ubuntu server:
sudo apt-get -y install nginx

2.  Test it.  
  a)  From a Windows workstation open I.E. and go to the external IP address of the server.
  b)  In I.E. go to Page -> Properties. Click on Certificates.  You should see a pop up that says "This type of document does not have a security certificate."

3.  Create a certificate and key file.
   a) Go to the Nginx server on the backend.  
   b) From the command prompt, run these four commands:
cd /etc/nginx
sudo mkdir tlsfiles
cd tlsfiles
sudo openssl req -newkey rsa:2048 -nodes -keyout contint.key -x509 -days 9999 -out contint.crt

# Respond to the prompts however you see fit.  For the sixth (and penultimate) prompt you will be asked "Common Name (e.g. server FQDN or YOUR name) []:"  Enter the external IP address of the server.

4.  Modify the /etc/nginx/nginx.conf file.  You may want to make a back up first.  In the http {} block enter these lines directly under the "http {" stanza where x.x.x.x is the external IP address of the Nginx server:

  server {
    listen              80;
    listen              443 ssl;
    server_name         x.x.x.x;
    ssl_certificate     /etc/nginx/tlsfiles/contint.crt;
    ssl_certificate_key /etc/nginx/tlsfiles/contint.key;
    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2;
    ssl_ciphers         HIGH:!aNULL:!MD5;
    ssl                 on;
    }

5.  Restart Nginx.  To do this, run these two commands:

sudo service nginx stop
sudo service nginx start

6.  Import the certificate file into a workstation.  As an example, we'll use IE on Windows 7.  
  a)  Open the file /etc/nginx/tlsfiles/contint.crt
  b)  Copy the text into a Notepad on the Windows workstation.  Save it anywhere as "contint.crt".
  c)  Close the Notepad.  
  d)  Hold the Windows button and tap "r."  Enter "mmc" with no quotes and press enter.  Click "Yes" to the UAC prompt.
  e)  Go to File -> Add/Remove Snap-in.  Under "Available snap-ins" highlight Certificates and click "Add >".  For the pop-up choose "My user account" and click "Finish."  Click "OK."
  f) Expand "Certificates - Current User".  Right click "Trusted Root Certification Authorities" and choose "All Tasks" -> "Import".  Click "Next".  For "File name" browse to where you save "contint.crt" in sub-step "b" directly above.  Click "Next" and then click the following "Next".  Click "Finish."  
  g)  For the "Security Warning" about installing the certificate, just click "Yes".  Click "Ok" to the next message.

7.  Open a IE and go to this URL:
https://x.x.x.x where x.x.x.x is the IP address of the Nginx server (e.g., the external IP address of the AWS server).    

8.  Test it.  Open I.E. and go to the external IP address of the server. In I.E. go to Page -> Properties. Click on Certificates.  Click on Certification Path.  The certificate status should say "This certificate is OK."

How Do You Set up Passwordless SSH from One Server to Another?

Problem scenario
You want to configure passwordless SSH between two servers such that you can log in from server A to server B with no password.  You want to be able to use scp between the two servers without being challenged for a password.  How do you set up passwordless SSH between them?

Solution

Assumptions
Assuming both servers have a local user account named "ubuntu", the following will work.  This assumes that relevant firewalls allow port 22 connections between the two servers.  If one server is in AWS and another is in Azure, please see this posting.  If you have a Windows server, you will need to install Cygwin on it; see this posting if you do not know how.  If you have Linux servers, just follow the procedures below.

Procedures
1.  On server A, run this command:  ssh-keygen -t rsa -P ""
#  You will be prompted after you run the above command.  Press enter to accept the default location to save the key.

2.  Then run this command:  cat /home/ubuntu/.ssh/id_rsa.pub

3.  Append the output of the above command to a file on server B.  This file on server B to receive this appendage is this: /home/ubuntu/.ssh/authorized_keys

4.  You are done.  From server A you will now be able to use ssh ubuntu@x.x.x.x (where x.x.x.x is server B's IP address) and authenticate without a password.  If you want server B to be able to passwordlessly connect to server A, repeat the above steps but substitute the roles "server A" with "server B."

What is SNI?

Question
What is SNI?

Answer
SNI stands for server name indication (OpenSSL.org).  It is an extension of TLS (Transport Layer Security).  TLS is a mechanism for secure network communication (Microsoft).

When using openssl, and thus TLS, the SNI name can be a DNS name or an IP address (OpenSSL.org).  TLS needed the SNI extension capability for a client server to send the hostname in the first (a "client hello" message) of three parts of the TLS handhsake process (OpenSSL.org).  The second part is a "server hello" message from the server (Microsoft).  The final part of the handshake is the "server hello done" message from the server again (Microsoft).  To use TLS to learn more about how it works, see this link.

What Are Some Differences in the Two Code Versioning Systems Known as Git and Subversion?

Code versioning systems keep track of different versions of various programs and relevant flat files that the programs may use (e.g., initialization, configuration, parameter consumption, files etc.). Some people refer to them as SCMs (Source Control Managers) or VCSes (Version Control Systems).

Code from versioning systems can be read from a centralized repository and be written to a local drive.  This is considered a pull.  A programmer might pull code down to examine it, use it for testing, or modify it to enhance it or create new features.  Being able to pull code down to various servers allows for aggressive experimentation.  A developer can refactor code without a concern for the previous version(s).  If something fails, the professional simply leaves the code alone (intact) as it resides on the centralized server (in a repository).

Code can be read from a local drive and written to a centralized repository.  This is considered a push.  You can push code up to make it available to other developers.  The centralized repository thus acts as a backup mechanism for disaster recovery.  (If the server that originally had the code is lost, the repository will have a copy of the file.)

Subversion is centralized (with one repository) and Git tends to be decentralized (with numerous repositories).  For collaborative programming with multiple developers integrating their programs into a shared code base, Subversion may be more intuitive having a single source of truth (to control the code).  For individual programming efforts, Git may be an easy way to store the history of various programs and/or configuration files for a personal project.  Git does not require a web UI and is thus easier to set up.  Git lends itself to recording different versions of a program in a local way without a separate server. 

Both Git and Subversion archive code.  Git repositories can be ultimately merged for a single source of truth.  Conflicts of different check-ins for the same file from different developers can be reconciled.  To discuss the details and options for this process of reconciliation is beyond the scope of this article.  For beginners to understand version control, Subversion's web UI and its necessarily accompanying feature that requires it be centralized may be more understandable.  Other code versioning systems often resemble Subversion in corporate environments (without the decentralization aspect).

For projects where people must connect to a VPN to upload new code, Subversion will not be as easy to use.  Local Git repositories can be useful for programmers who work remotely.  If they will not be able to connect to the corporate VPN, they may have code to check in and will be able to do so.  Git will save the incremental changes locally when they cannot upload the code to a VPN repository.  Eventually this Git repository will merge the changes when the developer finally connects to the VPN.  Subversion does not have this system of recording changes for deferred synchronization while the remote development is in process.  Git easily has the ability to save version changes of code locally.  A centralized Git repository can be configured and used to later receive a replication of these changes from other remote, formerly disconnected Git repositories.

Some corporate environments disallow development from remote locations or off the corporate network.  In this case connectivity to the network is guaranteed for any development to take place.  Therefore Subversion would be equal to Git in this situation for this purpose.  This limitation has the pragmatic effect of preventing redundant work from individual programming efforts.

Other than the centralization versus the decentralization features of the two code versioning systems, there are other differences:

Their Histories / Created In Different Years
Subversion was created in the year 2000, and Git was created in the year 2005.

Different Adoption Rates and Utilization
MentorMate's website shows that in 2014, the primary code versioning system adopted by various companies was Git with Subversion being the second most commonly used solution.  From November 2016 until October 2017, the Subversion plugin for Jenkins has been downloaded more times than the Git plugin for Jenkins (according to this link for Git and this link for Subversion).  OpenHub.net shows that Git is used more extensively than Subversion.  As of 2016 zeroturnaround.com did a survey that showed that Git was much more widely used than Subversion.

In November of 2017 a search for jobs on Dice.com (with no location) with the keyword "svn" brought up 803 positions.  The same search on the same day with the keyword "subversion" brought up 434 jobs.  The same search on the same day with the keyword "git" brought up 3,364 jobs. 

Different Inventors

  • Git was created by Linus Torvalds famous for creating Linux.
  • Subversion was created by CollabNet and is now supported by the Apache Software Foundation.  

Subprojects Have Different Names
Both products lend themselves to disposable prototyping by allowing developers to pull code down and use a copy of a program.  Subprojects in Git are referred to as submodules whereas subprojects in Subversion are referred to as externals (GitHub.com).

Other Meanings of the Tools' Names

  • The word "git" used outside the context of I.T. in Great Britain means "a foolish or worthless person" (Merriam-Webster.com).
  • The word "subversion" used outside the context of I.T. means the act of overthrowing something.  To subvert is to undermine or destroy.  Subversion is the act of subverting.  In software (e.g., OS distributions) there are major versions (releases with new numbers in the ones place RHEL 6 vs. RHEL 7) and minor versions (decimal place changes) such as RHEL 7.3 vs. 7.4.  Some versions are never released into production.  Minor versions are subordinate in importance to major versions.  Subversion can be a word to mean "mini-version."

Is Python a Strongly-Typed Language?

Question
Some programming languages adhere to a discipline known as being "strongly typed."  This attribute governs variable assignments.  Is Python a strongly-typed language?

Answer
Technically, yes it is.  Some interviewers for jobs involving Python may ask this question and expect you to say "no it is not."

Detailed Explanation
In software development there are many people who have used Python and found its variables to be "dynamically typed" and not "statically typed."  They often characterize Python as a weakly-typed language.  Technically it is a strongly-typed language.  Guido van Rossum, the person who invented Python, explains that a weakly-typed language such as JavaScript handles the "+" operator differently from a strongly-typed language such as Python depending on a variable's value.

Here is what van Rossum says:

"... in some languages (like JavaScript) you can add strings to numbers 'x' + 3 becomes 'x3'."

...

"In a strongly typed language (like Python) you can't perform operations inappropriate to the type of the object - attempting to add numbers to strings will fail."

Both quotes were taken from this link.  This link explains that Van Rossum was the author.

To prove Van Rossum's point consider this Python program that prints "12":

x = 5
y = 7
z = x + y
print z

Now change the Python program to have the x variable receive the string "x" but keep the other three lines the same:

x = 'x'
y = 7
z = x + y
print z

The above modified program will print this (assuming you name the program itself a.py):

Traceback (most recent call last):
  File "/tmp/a.py", line 4, in <module>
    z = x + y
TypeError: Can't convert 'int' object to str implicitly

So you can see that Python could be more weakly-typed.  Some people would say there is a subjective zone between weakly-typed and strongly-typed (with some languages being examples of neither or both).

For further details see below
StackOverflow has a discussion about Python being strongly typed here.  Relevantly and more generally, StackOverflow also has a question about strongly-typed and weakly-typed attributes in programming languages here.  The news forum for Y Combinator has some information about Python being strongly-typed.  An external blog has some more analysis of this topic here.

How Do You Troubleshoot “curl: (52) Empty reply from server”?

Problem scenario
You are running a curl command.  But you get this error "curl: (52) Empty reply from server."  What is the problem?

Possible solution #1
You are using curl with a port assignment in the URL (e.g., http://coolname.com:5077).  Try a different port.

Possible solution #2  (This solution is only relevant if the URL endpoint is supported by a Docker container.)
The docker-compose.yml file could be configured incorrectly.  It may refer to files with incorrect paths.  You may need to change the docker-compose.yml file to refer to correct paths.  An alternative solution would be to create the necessary files that the docker-compose.yml file needs and the related subdirectories that the file is looking for.  Your "docker-compose up" command may work with no errors.  But the references to the URL path may produce the "Empty reply from server" if the docker-compose.yml file refers to directory paths (e.g., for nginx listening on a specific port) in subdirectories that do not exist.

How Do You Troubleshoot the Puppet Error “[/var/lib/puppet/facts.d]: Failed to generate additional resources using ‘eval_generate’: SSL_connect returned=1”?

Problem scenario
You run puppet agent -t -d and get this error:

"Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1"

What do you do?

Solution
This is more of a resolution rather than a diagnosis.  The error should go away if you follow these steps.

On the Puppet agent node run these commands:

sudo cd /var/lib/puppet/ssl
sudo rm -rf *
sudo puppet agent -t -d

If you cannot find the ssl directory in the above location, run this command: sudo find / -name ssl -type d

On the Puppet Master, do this:

sudo puppet cert list
sudo puppet sign <FQDN shown in output from above command>

How Do You Install Java Version 8 on a Windows 64 Bit Server with PowerShell?

Problem scenario
You need to install Java version 8 on a Windows 64 bit server.  How do you do this with PowerShell?

Solution
1.  Open PowerShell ISE as Administrator.

2.  Run this script:

#  This is a modified version of PowerShell code taken from this site:
#  https://skarlso.github.io/2015/06/30/powershell-can-also-be-nice-or-installing-java-silently-and-waiting/

$JDK_VER="1u151"
$JDK_FULL_VER="8u151-b12"
$JDK_PATH="1.8.0_151"
$source86 = "http://download.oracle.com/otn-pub/java/jdk/$JDK_FULL_VER/jdk-$JDK_VER-windows-i586.exe"
$source64 = "http://javadl.oracle.com/webapps/download/AutoDL?BundleId=230542_2f38c3b165be4555a1fa6e98c45e0808"
$destination86 = "C:\Program Files (x86)\$JDK_VER-x86.exe"
$destination64 = "C:\Program Files (x86)\$JDK_VER-x64.exe"
$client = new-object System.Net.WebClient
$cookie = "oraclelicense=accept-securebackup-cookie"
$client.Headers.Add([System.Net.HttpRequestHeader]::Cookie, $cookie)

Write-Host 'Checking if Java is already installed'
if ((Test-Path "c:\Program Files (x86)\Java") -Or (Test-Path "c:\Program Files\Java")) {
    Write-Host 'No need to Install Java'
    Exit
}

Write-Host 'Downloading x86 to $destination86'

$client.downloadFile($source64, $destination64)
if (!(Test-Path $destination64)) {
    Write-Host "Downloading $destination64 failed"
    Exit
}


try {
    Write-Host 'Installing JDK-x64'
    $proc1 = Start-Process -FilePath "$destination64" -ArgumentList "/s REBOOT=ReallySuppress" -Wait -PassThru
    $proc1.waitForExit()
    Write-Host 'Installation Done.'

} catch [exception] {
    write-host '$_ is' $_
    write-host '$_.GetType().FullName is' $_.GetType().FullName
    write-host '$_.Exception is' $_.Exception
    write-host '$_.Exception.GetType().FullName is' $_.Exception.GetType().FullName
    write-host '$_.Exception.Message is' $_.Exception.Message
}

if ((Test-Path "c:\Program Files (x86)\Java") -Or (Test-Path "c:\Program Files\Java")) {
    Write-Host 'Java installed successfully.'
}
Write-Host 'Setting up Path variables.'
[System.Environment]::SetEnvironmentVariable("JAVA_HOME", "c:\Program Files (x86)\Java\jdk$JDK_PATH", "Machine")
[System.Environment]::SetEnvironmentVariable("PATH", $Env:Path + ";c:\Program Files (x86)\Java\jdk$JDK_PATH\bin", "Machine")
Write-Host 'Done. Goodbye.'

In the future we want to re-write these directions to use write-verbose instead of write-host. We want to warn people that write-verbose is recommended instead of write-host (according to this posting).

How Do You Monitor a Server Being up with AWS or How Do You Use AWS Route 53 without a Domain Name?

Problem scenarios
One, or both, of the following apply to your situation:

Problem scenario #1
How do you use AWS (the SNS component) to set up an email alert to be notified of a server going down?

Problem scenario #2
You want to use Amazon Route 53 to see how this component works.  You do not want to register a domain name.  What do you do?

Solution
All steps are mandatory unless they say "Optional."

1.  Log into the AWS console.
2.  Go to this link: https://console.aws.amazon.com/route53
3.  On the left click "Health checks"
4.  Click "Create health check" button near the top
5.  For "Name" enter any arbitrary name to remember what this health check is (e.g., importantTester).
6.  Leave "What to monitor" set at "Endpoint".
7.  Under "Monitor an endpoint" make sure the following settings have the following values:

"Specify endpoint by" should be "IP address"
"Protocol" should be "TCP"
Enter the external IP address of the server you want to monitor.  (If you don't know it, go to the server and run the command "curl icanhazip.com".)
"Port" should be "80"

8.  Optional step:
If you want to test it out, expand "Advanced configuration" and set the "Failure threshold" to 1.  Then turn off the server or make the IP address unreachable.

9.  Click "Next"
10.  For "Create alarm" choose "Yes"
11.  For "Send notification to" choose "New SNS topic"
12.  For "Topic name" enter an arbitrary name that will be meaningful to you.
13.  For "Recipient email addresses" enter any email address you want to receive notification when the IP address is not reachable
14.  Click "Create health check"

15.  Outside of the AWS Console, go to the email account of one of the addresses (or the address) that you entered.  Confirm the subscription when you open the email from Amazon.

16.  Optional step:  Turn off or otherwise make unreachable the server that you entered the IP address for above.  You should get an email about it.  This will prove the directions above worked.