How Do You Install a Pythonic Implementation of Apache Thrift?

Problem
You want thriftpy installed to deploy Apache Parquet.  How do you install thriftpy?

Solution
Prerequisite

Install pip.  If you need assistance, click on the link according to your distribution of Linux:
CentOS/RHEL/Fedora (most newer versions)
RHEL 6.x
Debian/Ubuntu
SUSE

Procedure
Run this command:  pip install thriftpy

How Do You Install These Kubernetes-Related Commands kubelet, kubeadm, kubectl, and etcd on an Ubuntu Server?

Problem scenario
You want to install kubelet, kubeadm, kubectl, and etcd on an Ubuntu server.  How do you do this?

Solution
1.  Run these five commands:

sudo apt-get -y update
sudo apt -y install docker.io
sudo su -
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
exit

2.  Use either Method A or Method B.

Method A

sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt -y install software-properties-common

Method B
Edit the kubernetes.list file so the correct .deb packages are installed.

sudo vi /etc/apt/sources.list.d/kubernetes.list

# add this line below as the sole entry

deb http://apt.kubernetes.io/ kubernetes-xenial main

3.  Run these two commands:

sudo apt-get -y update
sudo apt-get -y install kubelet kubectl kubeadm etcd

How Do You Get PowerShell’s Invoke-Webrequest to Work Consistently (Instead of “command not found”)?

Problem scenario
You use invoke-webrequest and it works fine with some websites. But sometimes you find that PowerShell says "command not found."  What is wrong?

Solution
invoke-webrequest will work to download html and htm files.  Depending on the source file on the web server, the command needs a flag to work properly.  For an image file with a .jpeg extension, the invoke-webrequest needs a flag "-ContentType 'image/jpg'" or something similar to work.  The flag depends on the file that you are downloading from a web server.  This command below is an example of a non-HTML file that you want to download with PowerShell (the file happens to be an image file named foobar.jpeg).

invoke-webrequest http://source/URL/foobar.jpeg -ContentType 'image/jpg'

If you do not need to do type checking when it downloads, try a wildcard:

invoke-webrequest http://source/URL/foobar.jpeg -ContentType *

How Do You Install node and npm on Any Distribution of Linux Server?

Updated on 5/1/20

Problem scenario
You have a Linux server with 1 GB of RAM.  You want to install node and npm because you want to try out the latest JavaScript technologies.  (You were persuaded to not use Gulp or Grunt to manage build scripts.  You want to use npm because of this link.)  You want to use the same copy of the bash script on CentOS/RedHat/Fedora, Debian/Ubuntu, and SUSE Linux to do this installation.  What do you do?

Solution
Prerequisite
Do not attempt to run this on a server with less than 1 GB of RAM.  We recommend you have 3 GB of memory, but 2 GB of that can be virtual memory.  If you need assistance creating 2 GB of swap space memory, see this posting.

Warning
If you use a RedHat derivative (e.g., CentOS, RHEL, or Fedora), this script will install Perl (in case you had a special version installed you can comment out this line).  This script will remove Perl if it is already installed.  You can comment out the "yum -y remove perl" command if you so desire. If you want to try an older, alternative set of directions for installing a RedHat derivative, this link may be of value, but we recommend this set below. Also, this installation can take over one hour to complete.

Recommendation for Debian/Ubuntu users
If you are using Ubuntu/Debian Linux, this command will do what you want easily:  sudo apt -y install nodejs npm
The command above will not work if you are behind a firewall or the repositories on your network do not have the dependencies involved.  If this is your problem, then proceed with the script below instead.

n.b. If you must use Python 2, and cannot install Python 3, use the attached file "npm install with python2.txt." For normal Linux servers, with recommend Python 3 and the newer version of npm as shown below.

1.  Create a file like this in /tmp/ called node.sh:

# Script written by www.continualintegration.com
# The apt-get, zypper, and yum commands will error out depending on the distribution.  
# Running them on a Linux distro that does not support them would not normally harm anything.
# This script for NPM verison 14.x works with Python 3.x

version="14.1.0"  # Change if desired.  This will be the version installed for the non-SUSE distros (e.g. Debian & RedHat)
distro=$(cat /etc/*-release | grep NAME)

debflag=$(echo $distro | grep -i "ubuntu")  # This will be empty if it is not Ubuntu Linux.
if [ -z "$debflag" ]  #If it is not Ubuntu, check if it is Debian.
then
    debflag=$(echo $distro | grep -i "debian")  # This will be empty if it is not Debian Linux.
fi

if [ -z "$debflag" ]
then   
     echo "...still determining Linux distribution..."  # At this point it could be SUSE or a RedHat distro
else
     echo "You are using either Ubuntu Linux or Debian Linux."
     apt-get -y update  # This is needed for AWS Ubuntu servers (it won't hurt to run on a GCP Debian as of February 2018)
     apt-get -y install gcc g++ make python3 # python is needed for AWS Ubuntu AMI (not GCP Debian as of February 2018)
     cd /opt/
     curl http://nodejs.org/dist/v$version/node-v$version.tar.gz > node-v$version.tar.gz
     tar xzvf node-v$version.tar.gz && cd node-v$version
     ./configure
     make
     make install
fi

rhflag=$(echo $distro | grep -i "red*hat")
if [ -z "$rhflag" ]
then   #If it is not RedHat, see if it is CentOS or Fedora.
  rhflag=$(echo $distro | grep -i "centos")
  if [ -z "$rhflag" ]
    then    #If it is neither RedHat nor CentOS, see if it is Fedora.
    echo "It does not appear to be CentOS or RHEL..."
    rhflag=$(echo $distro | grep -i "fedora")
    fi
fi

if [ -z "$rhflag" ]
   then
   echo "...still determining Linux distribution..."
else
   echo "You have a RedHat distribution (e.g., CentOS, RHEL, or Fedora)"
   yum -y install gcc gcc-c++ make python3 # make is included as a "just in case"
   yum -y groupinstall 'Development Tools'
   ln -s /usr/bin/python3 /usr/bin/python
   cd /opt/
   curl http://nodejs.org/dist/v$version/node-v$version.tar.gz > node-v$version.tar.gz
   tar xzvf node-v$version.tar.gz && cd node-v$version
   ./configure
   make
   make install
   sudoversion=$(man sudo | grep Sudo | awk '{print $2}')
   echo "If you see a number below that is lower than 1.7, you need to update sudo"
   echo $sudoversion  # Why it needs to be 1.7 or higher: CVE-2005-2959, CVE-2005-4158 & CVE-2006-0151
   echo "The number above should be greater than 1.7. If not, update your sudo command to 1.7 or higher"
   echo "Assuming that the version above was greater than 1.7 modify your /etc/sudoers file"
fi

suseflag=$(echo $distro | grep -i "suse")
if [ -z "$suseflag" ]
then
  if [ -z "$debflag" ]
  then
    if [ -z "$rhflag" ]
      then
      echo "*******************************************"
      echo "Could not determine the Linux distribution!"
      echo "Installation aborted. Nothing was done."
      echo "******************************************"
      exit
    fi
  fi
else
   echo "You have Linux SUSE."
   zypper -n addrepo http://download.opensuse.org/tumbleweed/repo/oss/ oss
   zypper --no-gpg-checks update
   zypper -n install make gcc cpp-7-2.9.x86_64 gcc-c++-7-2.9.x86_64  
   cd /opt/
   curl http://nodejs.org/dist/v$version/node-v$version.tar.gz > node-v$version.tar.gz
   tar xzvf node-v$version.tar.gz && cd node-v$version
   ./configure
   make
   make install
fi

npm i -g npm
sleep 1
finvar=$(/usr/local/bin/node -v)

if [ -z "$rhflag" ]
   then
   echo "It looks like Node version " $finvar " was installed!"
   echo "run these commands below to double check"
   echo "npm -v"
   echo "/usr/local/bin/node -v"
else
  linenumber=$(cat /etc/sudoers | grep -n secure_path | sed 's/[^0-9]*//g')
  if [ "$linenumber" -eq 80 ]
  then
    yum -y install perl
    cp /etc/sudoers /etc/sudoers.bak
    perl -i -pe 's/.*/Defaults    secure_path = \/sbin:\/bin:\/usr\/sbin:\/usr\/bin:\/usr\/local\/bin/ if $.==80' /etc/sudoers
    yum -y remove perl
    echo "We removed Perl.  Reinstall it if necessary."
  else
    echo "You will need to modify /etc/sudoers so that the secure_path setting includes /usr/local/bin"
    echo "If you do not modify it, you will not be able to use 'sudo node' or 'sudo npm' commands"
  fi
   echo "Node and npm seem to have been installed.  Use these commands below to check:"
   echo "npm -v"
   echo "node -v"
fi

2.  Run this command: sudo bash /tmp/node.sh

3.  The above script may take 30 minutes or more to complete.  When it is done, test it by running these commands:  npm -v
node -v

How Do You Get an Apache Web Server to Publish a Specific .PHP File on a CentOS/RHEL/Fedora Server?

Problem scenario
You are using a RedHat derivative distribution of Linux.  You want to install Apache Web Server on it so it can publish PHP content.  How do you install and configure the necessary dependencies to do this?

Solution
1.  Install Apache web server:  sudo yum -y install httpd

2.  Install PHP:  sudo yum -y install php

3.  Examine the httpd.conf file.  It is usually in this location: /etc/httpd/conf/

Find the DocumentRoot section.  It may look like this:

DocumentRoot "/var/www/html"

This means that the directory path where the web server will look for .html or .php files is /var/www/html.

4.  Place a PHP file in the DocumentRoot location found above (e.g., create /var/www/html/good.php) with the following content:

<?php
$var1="continualintegration";
?>

<html>
<body>

<form action="anotherPage.php" method="post">

Check this text field out: <input type="text" name="pk" value=<?php echo $var1; ?>>

<input type="submit"2>
</form>

</body>
</html>

5.  Start the web server with this command:  sudo service httpd start

6.  Open a web browser.  Construct a URL like this:

x.x.x.x/good.php

Replace x.x.x.x with the external IP address of the above server.  Replace good.php with the name you gave to the .php file in step #4.  Go to this URL in the web browser.  This tests your work above.

How Do You Use PHP Variables in HTML?

Problem scenario
You have a .php file with PHP and HTML code.  You want an HTML text field to be display a pre-populated text value by default.  The user can then modify this value from the web UI (i.e., front-end).  You want this text value to be based on a PHP variable (calculated in earlier in the code).  How do you get your HTML code to display the value of a PHP variable?

Solution
Prerequisites
For an Ubuntu or Debian distribution of Linux consider these bullets to install Apache web server and PHP:

  • To install a web server (so the .php pages can be published and thereby be visible in a web browser) on an Ubuntu or a Debian distribution of Linux, run this:  sudo apt-get -y install apache2-bin libapache2-mod-php
    • This link may also be helpful.  You may also want to see this link if you are new to Apache web server on Ubuntu.
  • To install PHP on an Ubuntu or a Debian distribution of Linux, run this: sudo apt-get -y install php

For a RedHat distribution of Linux (e.g., CentOS/Fedora), consider these bullets to install Apache web server and PHP:


Procedures

This example illustrates how you can inject (or transfer) a PHP value into HTML code.  A text field will have the value of a PHP variable if you use this code a .php file and view it in a web browser.

<?php
$var1="continualintegration";
?>

<html>
<body>

<form action="anotherPage.php" method="post">

Check this text field out: <input type="text" name="pk" value=<?php echo $var1; ?>>

<input type="submit"2>
</form>

</body>
</html>

How Do You Use Python to Process (and Thereby Clean) a Flat File That Is a List of Email Addresses with Extraneous Symbols?

Problem scenario
You have an input file (.txt) that is a list of email addresses.  The file is guaranteed to have one of the following: one or more spaces between each email address or there is certainly a new line between the each email address.  Beyond that there can be one or two commas after the email address.  There could also be one or two semicolons after the email address.  You know that this input file will not have a mix of the two punctuation marks.  That is you will not see both a comma and a semicolon after the each email address.

You want to write a Python program to produce a clean output with commas between the email addresses.  You want to eliminate duplicate email addresses, duplicate commas, and all semicolons.  You want to have the email addresses in all lowercase characters.  How do you use Python to manipulate (or process) a .txt file that has a list of email addresses and produce a new file (as output) that is a clean .csv?

Solution
Use this file csvmaker.py.  Follow the usage instructions in the comments at the top.

# Written by continualintegration.com.  
# Usage instructions and requirements.
# This Python program produces a clean .csv file of email addresses.  It requires an input file named inputfile.txt.
# inputfile.txt is in the same directory as this Python program.
# inputfile.txt can have email addresses on each line with or without a terminating semicolon or comma (but never a mix of those punctuation marks).
# The output file will be final.csv in the same directory as this .py file and the inputfile.txt.
#  Call this file "csvmaker.py".  Run it like this: "python csvmaker.py"
"""
To create an input file to demonstrate this program's features, create inputfile.txt with this as the content:

cool@cool.com
basic@basic.com;; another@another.com
good@good.com, whatever@whatever.com
fun@fun.com,,
try@try.com,

something@something.com;
example@example.com
"""
emaillist = [] # initialize a blank list
with open('inputfile.txt', 'r') as a:
        for b in a:
                if '@' in b:   #operate only on lines with "@"
                        c = b.split()  #split up words (that are separated by spaces) on line
                        for d in c:    #iterate through characters that make up words
                                if '@' in d:  #if email address,
                                        emaillist.append(d)  # This builds a list of raw email addresses that are unformatted.
e = list(set(emaillist)) # This eliminates duplicates in the list that is in memory called "emaillist"
e.sort() # Alphabetize the list of emails
fh = open('final.csv', 'w')
for c in e:  # Go through each line
        c = c.replace(";;", ",") # replace two semi-colons with a comma
        c = c.replace(",,", "") # replace two commas with a comma
        if ';' in c:
                c = c.replace(";", ",") # Replace one semi-colon with a comma
        else:
                if ',' in c:
                        var1 = 1; # do nothing. without an operation, Python will throw an error.
                else:
                        c = c + ","  # If there is no comma, add one.
#       print(c)
        fh.writelines(c)
fh.close()