How Do You Fix a Python Program That Returns an Error “…subprocess.py … in check_output”?

Problem scenario
You are trying to use subprocess and check_output in Python to run raw Bash/shell/Linux commands.  

Your Python program has a line like this:
subprocess.check_output("/usr/local/bin/coolprog arg1 /path/to/file.txt arg3")

You get this error when you run the program:

 File "/usr/lib/python2.7/subprocess.py", line 566, in check_output

What do you do?

Solution
This solution only applies if the Bash command is constructed from known good input.  If the command that the subprocess invocation is constructed from is an external source, this solution would inadvisable; we do not recommend you adapt this solution to something that could be susceptible to malicious code injections.

The above subprocess.checkoutput line works for a single-word Linux command.  For commands with more involved syntax, you must change it to include "shell=True" after a comma and space; you also need a .split() at the end of the closing parenthesis above.  Here is an example of how to properly use check_output:

subprocess.check_output("/usr/local/bin/coolprog arg1 /path/to/file.txt arg3", shell=True).split()

# If you want to learn more about the safety concerns of subprocess and shell=True, see this link.

How Do You Get Raspberry Pi to Use a USB Drive?

Problem scenario
You are trying to use a USB stick with your Raspberry Pi.  But you get errors you receive say things like "operation not permitted" and "target is busy."  You want to create new directories and files on this thumb drive.  What do you do?

Solution
1. Remove the USB stick.
2.  Insert it.  Do not choose the GUI prompt to use File Manager.  
3.  With a command prompt, find the gid and uid of the user you want to be able to access to the USB drive.  If you know the username, run this command:

id jdoe # where jdoe is the username

4.  Use the following command where "foobar" is the name of the directory you want to refer to when you read/write to the USB drive, "1001" is the gid of the user above and "1000" is the uid of the user above.

sudo mount -o remount,gid=1001,uid=1000 /media/pi/foobar/

# Without modifying the fstab, this mounted directory may be available after a reboot of the Raspberry Pi!

How Do You Troublshoot the Error “snappymodule.cc:31:22: fatal error: snappy-c.h: No such file or directory” when Trying to Install Apache Parquet?

Problem scenario
When trying to install Apache Parquet on Ubuntu, you get this error:

"Running python-snappy-0.5.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-SIHL_T/python-snappy-0.5.1/egg-dist-tmp-O8UnkU
WARNING: '.' not a valid package name; please use only .-separated package names in setup.py
package init file '__init__.py' not found (or not a regular file)
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
snappymodule.cc:31:22: fatal error: snappy-c.h: No such file or directory
compilation terminated.
error: Setup script exited with error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
ubuntu@ubuntuf1:/opt/parquet-1.2$ find . -name snappymodule.cc
ubuntu@ubuntuf1:/opt/parquet-1.2$ sudo apt-get install python-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
python-dev is already the newest version (2.7.11-1).
0 upgraded, 0 newly installed, 0 to remove and 11 not upgraded."

What should you do?

Solution
Assuming you are running Ubuntu Linux, run this command to satisfy a dependency of Apache Parquet:
sudo apt-get -y install libsnappy-dev

How Do You Get to the First Character in a Linux Command Prompt?

Problem scenario
You are trying to run very long, complex Linux commands.  There are many case-sensitive flag options and convoluted DNS names.  It takes time to hold the left arrow to go to the beginning of the command.  Sometimes you want to comment out a line with a "#" at the beginning of the line as you draft these Linux commands.

Solution
Use ctrl-a to get to the first character of the command.

Conversely you may want to get to the end of the line immediately.  Use ctrl-e to get to the last character of the command.

Remember that "#" (with no quotes) will comment out whatever is to the right of it.  Therefore you can draft commands like this:

#Let me try to draft something with different flag options.  I can always use Ctrl-A to get to the first line and Ctrl-E to the end of this long line before I press enter to run the command.

Ctrl-c will allow you to cancel out of a command without running it.

How Do You Update the BIOS of an HP Computer When the Option You Need Is Grayed Out?

Problem scenario
You turn on an HP computer (e.g., a desktop or laptop) and press Esc to go to the Startup Menu.  You then press F10 or use the down key and press enter to get to the BIOS menu.  When you are there you see the option you need "Update BIOS Using Local Media" is grayed out.  You entered administrator-level credentials successfully.  You have not other credentials to log into this HP startup menu.  What do you do to update the BIOS?

Solution
Log in as a guest user.  For some reason a guest user may be able to have see clickable blue text "Update BIOS Using Local Media."  This solution can work.

How Do You Manage (List, Resume or Terminate) Suspended Linux Commands?

Problem scenario
You have started some commands then used Ctrl-z while they were still processing.  You want to see which jobs have been suspended.  You may want to terminate or run the jobs. What do you do?

Solution
Use the builtin Linux command "jobs" like this:

jobs -l

# Sample output may look like this:

[1]- 26830 Stopped                 ping -c 100 8.8.8.8
[2]+ 26831 Stopped                 ping -c 99 8.8.8.8

To resume the lowest job in the output of the above command, use this command with no arguments:

fg

# fg will resume the job with the "+" sign in the list from the "jobs -l" command.
# fg can be used to start specific jobs and not necessarily the bottom job when it has the correct argument.  

To terminate a job listed in "jobs -l" you can use this command:

disown <pid> # where <pid> is the number listed before the word "Stopped" in the output of "jobs -l"

You can also refer to the numbers such as [1] and [2] as they appear in the left hand column in the output of "jobs -l".  To do this use the %x format where "x" is the number that is in the brackets "[]" in the left hand column of the output of "jobs -l".

You can use fg or bg to resume the suspended jobs.  Based on this integer in the brackets "[]" in the left hand column of the output of "jobs -l", you can use the % syntax like this:

fg %1  #this will resume the "ping -c 100 8.8.8.8" job in the foreground

or

bg %1 #this will resume the "ping -c 100 8.8.8.8" job in the background

# The disown command is a builtin command. It can also refer to jobs with the %x syntax like the fg and bg examples above.

How Do You Enable IP Version 4 Forwarding on an Ubuntu Server?

Problem scenario
You want to enable IP version 4 forwarding on an Ubuntu server permanently.  How do you do this without rebooting?

Solution
1.  Run this command:sudo sysctl -w net.ipv4.ip_forward=1

2.  Modify the /etc/sysctl.conf file. Use "sudo vi /etc/sysctl.conf"

Uncomment out this stanza:

net.ipv4.ip_forward=1

3.  Save the changes.  You are done.

How Do You Use the $# Builtin Linux Variable?

Problem scenario
You want to test the $# built-in Linux variable. Every time you use it, it seems to return "0." How do you see the number of arguments in the most recent command?

Solution
Create a bash script with the following lines "echo... to fi":

echo "This script helps you learn about built-in variables in Linux."
if [ $# -eq 1 ]
   then
       echo "one argument was entered"
   else
       echo $# "arguments were entered."
fi

Run this script like this three times (with passing arguments in the last two):

bash coolscript.sh
bash coolscript.sh foo
bash coolscript.sh foo bar

# The output of the above three commands should demonstrate that $# keeps track of the number of arguments that were entered.

Why Can’t You Find the Linux Process for Apache Web Service That Appears to Be Running?

Problem scenario
You think the Apache web server is running.  You can see the web page from a web browser or your coworkers tell you that it is running.  You are not 100% sure however.  You want to see the Linux pid (process ID) for Apache web service that is running.  You run this:

ps -ef | grep httpd

You do not see any services running.  But you think that the Apache web service is in fact running.  Can Apache web service be running without an httpd process?

Solution
Run this:

ps -ef | grep apache

In Ubuntu Linux, Apache web server does not run with the httpd process.  It runs with a process with the string "apache".

If you still do not see an apache process running, it could be running in Docker.  Perhaps the web server you see is from a different server.  Sometimes with NAT or containers, IP addresses can become confused.

How Do You Upload Code to a GitLab Project from a Server That Does Not Have a Desktop UI?

Problem scenario
You have a Linux server with the command prompt but no GUI.  A separate server hosts GitLab.  How do you configure the first character-based server to use GitLab?

Solution
This assumes that you have deployed GitLab; if you need directions for this, see this posting.  This assumes that you have created a project in GitLab.  If you need directions for this, see this posting.

1.a.  From the web UI of the GitLab server, obtain the repository URL for the project.  It will look something like this:

git@goodserver/my-contint-project.git

If you cannot readily find this repository URL, you can log into the web UI of GitLab, go to Projects or "Projects" -> "Your projects".  Click on the project you want.  In the middle top of the screen there should be a field in between a "Fork"  button and an icon of a cloud/download button.  It will say "SSH" or "HTTP". Copy this string.  

1.b.  Go to the backend of the GitLab server.  Once you are at the character terminal, find the project on the backend.  To do this search for name of your .git file with a command like this:
sudo find / -name my-contint-project.git # but substitute the "my-contint-project.git" with the name of your git file (as the name should have been deteremined in step 5.a).

1.c.  Make a note of the full directory path found in step 1.b.

2.a.  Remain logged into the GitLab web UI.

2.b.  In the web UI for GitLab, click on the Project that has the repository you want (unless you are still at the "Details" screen where the Project's SSH or HTTP URL is visible).

2.c.  On the left click on "Settings".

2.d.  Click on "Repository" under "Settings."  

2.e.  Click on the "Expand" button for "Deploy Keys".

2.f.  Provide an arbitrary phrase in the "Title" field.

2.g.  Consider which server and which user will upload code to this GitLab project.  Go to that server and log in as that user even if it is a local user to that server.  Find the content of the id_rsa.pub file for this user (e.g., /home/jdoe/.ssh/id_rsa.pub).  If no id_rsa.pub file exists yet, go to the backend of the Linux server, log in as jdoe (or the user of your choice), run 'ssh-keygen -t rsa -P ""' and press enter to the next prompt.

Copy the contents of the id_rsa.pub file for this user on the server that will connect to the GitLab server and upload code to the GitLab project that you just created.  For the "Key" field input that is on the GUI of the GitLab server after step 2.f., paste the content of the id_rsa.pub file.

2.h.  Check the option for "Write access allowed"

2.i.  Click the green "Add key" button.

3.a.  From the server you want to upload code from, go to the Linux command prompt.  Log in as the user you want to upload  code as (e.g., jdoe).

3.b.  Verify git is installed (run "git --version").  Install it if necessary ("sudo apt-get -y install git" for a Debian distribution or "sudo yum -y install git" for a RedHat distribution).

3.c.  Draft a command like the one below (with "git clone" preceding the URL constructor obtained in step 1.a. above).    

git clone git@"<GitLabFQDN>":"<pathNotedin1.c>"/nameOfRepo.git

Substitute "<GitLabFQDN>" with either the FQDN of the GitLab server or the IP address.  Replace "<pathNotedin1.c>" with the  path you obtained in step 1.c above.  Substitute "nameOfRepo.git" with the name of the repository found in step 1.a.

Run this drafted command after you did the substitutions.

You may be prompted if you want to continue.  Choose "yes" with no quotes.  If you run the command above and you are prompted for a password, then the keys were probably not properly loaded to the GitLab GUI.  It is inadvisable, but you can continue with the git user.  If you do you may want to change the git user's password.  There are other ways of circumventing this problem (and this is extremely inadvisable to change the git user's password on the back end of the GitLab server because there are supported methods of using git commands). If you can change the git user's password (e.g, you are implementing a proof of concept or experimenting for the sake of learning with no sensitive data on a temporary server), log into the backend of the GitLab server.  Run this:  sudo passwd git

3.d. On the non-GitLab server, use this command:  cd nameOfRepo # where "nameOfRepo" is the name of the repository you just clones.  

3.e.  Create a file. For this example we'll use "fun.sh" with the content of these two lines:

var1=$(date)
echo $var1

3.f.  Run this command:  git add .

3.g.  Run this command but replace "jdoe" with the username that will interact with the GitLab project:  git config --global user.name "jdoe"

3.h.  Run this command:  git commit -m "Added a basic file."

3.i.  Run this command: git push origin master

Now you have added a file to the GitLab project.  Now you can add more files to the GitLab project via the Git command line.