CONTINUAL INTEGRATION – Page 183 – A Technical I.T./DevOps Blog

How Do You Install Boto 3 on a RHEL Server in AWS?

Problem scenario
You have an EC-2 instance running RedHat Enterprise Linux. You want to use the Python SDK for AWS called Boto. What should you do?

Solution
Prerequisites
Install pip. See this link if you do not know how.

Procedure
1. Run this command: sudo pip install boto3

2. You are done. The rest is optional.

3. You may want to install and configure the AWS CLI. But this is optional if you place the AWS access ID and secret access key in the Python programs themselves. For directions on installing and configuring the AWS CLI, see this posting.

4. Here is a sample program if you do not have AWS CLI installed and configured. This program merely lists S3 buckets that have been created. It is not destructive.

# Usage instructions
# 1. replace "aaa111" with the aws_access_key_id of your account
# 2. replace "bbb222" with the aws_secret_access_key of your account
# If you are not sure how to find these credentials do the following:
# To find the AWS Access Key ID and AWS Secret Access Key, in the AWS console, click on your name in the upper right hand corner.
# Then click on "My Security Credentials." Click "Create New Access Key" Click "Show Access Key."
# To run it, call it test3.py and run it with a command like this: "python test3.py"

import boto3 
s3 = boto3.resource('s3', aws_access_key_id='aaa111', aws_secret_access_key='bbb222')
 for bucket in s3.buckets.all():
    print(bucket.name)

5. As a reference, you may go to this external link to learn more about Boto's features.

A List of Elastic Stack (Elasticsearch, Logstash, Beat, and Kibana) Books

The Elastic Stack used to be called the ELK Stack; this link provides more information.

Applied Network Security Monitoring: Collection, Detection, and Analysis by Chris Sanders and Jason Smith
The Art of Monitoring by James Turnbull
ElasticSearch 5.0 Cookbook - Third Edition by Alberto Paro
Elasticsearch Blueprints by Vineeth Mohan
Elasticsearch: A Complete Guide by Bharvi Dixit, Rafal Kuc, Marek Rogozinski and Saurabh Chhajed
ElasticSearch Cookbook, Second Edition by Alberto Paro
Elasticsearch: The Definitive Guide: A Distributed Real-Time Search and Analytics Engine by Clinton Gormley and Zachary Tong
Elasticsearch Essentials by Bharvi Dixit
Elasticsearch for Hadoop by Vishal Shukla
Elasticsearch in Action by Radu Gheorghe, Matthew Lee Hinman and Roy Russo
ElasticSearch Indexing by Huseyin Akdogan
ElasticSearch Quick Start: An introduction to ElasticSearch in tutorial form. by Joel Abrahamsson
Elasticsearch Server - Third Edition by Rafal Kuc and Marek Rogozinski
Kibana Essentials by Yuvraj Gupta
Learning Elasticsearch by Abhishek Andhavarapu
Learning ELK Stack by Saurabh Chhajed
Learning Kibana 5.0 by Bahaaldine Azarmi
The Logstash Book by James Turnbull
Mastering ElasticSearch 5.0 - Third Edition by Bharvi Dixit
Mastering Elasticsearch, Second Edition by Rafal Kuc and Marek Rogozinski
Monitoring ElasticSearch by Dan Noble
NoSQL Injection for Elasticsearch by Gary Drocella
Relevant Search: With applications for Solr and Elasticsearch by Doug Turnbull and John Berryman

Announcement: Big Data Quiz Now Available!

We worked very hard to write an original twelve-question Big Data Quiz. Please do not expect another posting (like we normally post) for several days. Please be sure to check out the Big Data Quiz!

How Do You Find an IP Address That Is Mysterious and Elusive in Your Network?

Problem scenario
An IP address is pingable from one Linux server. On this server there are no Docker containers running. A traceroute reveals that this IP address is one hop away from your server. How do you find out what this IP address belongs to?

Solution
Use this command: ip addr show

You may also want to use this command: route -n

For a more thorough guide to troubleshooting network problems, see this posting.

Big Data Quiz

1. What does EDH stand for?

a. Enterprise Data Hub
b. Extract Develop Hadoop
c. Extract Decide Haul
d. Extract Data Hadoop

2. Gartner, Informatica and MapR think "data lakes" should be referred to as what?

a. data warehouses
b. data dams
c. data mills
d. data reservoirs

3. MapReduce is to Hadoop as ___________ is to Spark

a. Storm
b. Vertice Algorithm
c. Directed Acyclic Graph
d. RDD
e. Memory

4. RDD stands for what in Spark?

a. Really Different Data
b. Resilient Distributed Dataset
c. Real Developed Data
d. Reliable Data Distribution

5. Which three file systems are recommended to be used with HDFS on top?

a. cifs
b. ext3
c. ext4
d. gfs
e. hfs
f. JFS
g. nfs
h. reiserfs
i. vfat
j. XFS

6. If a Hadoop cluster had nodes that cost $15,000 each, would an HP Vertica or a Teradata solution cost more or less? Choose two.

a. HP Vertica would be cheaper
b. HP Vertica would be more expensive
c. Teradata would be cheaper
d. Teradata would be more expensive

7. What is "a scalable and fault-tolerant stream processing engine built on the Spark SQL engine."?

a. Structured streaming
b. Beam
c. Continual application
d. Storm

8. What is a framework that allows you to implement streaming and batch data processing jobs that can run on any execution engine?

a. Apache Apex
b. Apache Beam
c. Apache Cassandra
d. Apache Flink
e. Apache Storm

9. Which of the following does not need Hadoop (choose two)?

a. Apache Apex
b. Apache Flink
c. Apache Spark
d. Apache Tez

10. Which of the following is a "Hadoop YARN native platform" (thus dependent on Hadoop) and a type of "unified stream and batch processing engine"?

a. Apache Apex
b. Apache Beam
c. Apache Cassandra
d. Apache Delta
e. Apache Flink

11. What company provides a commercial version of Apache Spark that was founded by the people who invented Apache Spark?

a. Data Pipeline Gurus, LLC
b. Databricks
c. Hotfire Software
d. Zephyr Data

12. What is Microsoft's version of Hadoop?

a. MS Knowledge
b. BigTable
c. HDInsight
d. Datica
e. Kinesis

13. What are examples of a Directed Acyclic Graph?

a. A typical ETL process
b. The npm package manager
c. YARN
d. Spark operating on RDDs via stages which involves sub-tasks
e. Apache Airflow's pythonic schedule of phases for dynamic processing
f. All of the above
g. None of the above

14. At what stage in the MapReduce process does the "shuffle" phase happen?

a. Before the map stage
b. After the map stage and before the reduce stage
c. After the reduce stage
d. None of the above

15. How does Hadoop support high availability for your name node?

a. Via the secondary namenode
b. A standby namenode only in proprietary Hadoop versions
c. A standby namenode in open source or proprietary Hadoop versions
d. N/A. There is no native Hadoop support for highly available namenodes

For answers, see this posting.

How Do You Set up Ansible for the First Time?

Problem scenario
You have Ansible installed as a control server (a centralized server to push down configurations to other servers). You want to use it to manage another server. How do you configure the other server to be a managed node? In other words, how do you configure Ansible to push configuration changes down to servers?

Solution
Prerequisite
Ansible must be installed. If you need directions, see this posting if you are using RHEL. If you are using Linux SUSE and need directions for installing Ansible, see this posting.

Procedures
1. Configure passwordless SSH from the Ansible server to the managed node. If you are not sure how to do this, see this posting.

2. On the Ansible control server (not the managed node), modify the /etc/ansible/hosts file. (This directory path would normally never exist on a managed node. Only a server with Ansible would have /etc/ansible/.) If the directory /etc/ansible/ does not exist on the Ansible server, create it. Add these two lines to the hosts file in /etc/ansible/ but substitute x.x.x.x for the internal IP address of the managed node (change "group_name" to whatever you like as well as "alias" to whatever you like):

[group_name]
alias ansible_ssh_host=x.x.x.x

Each word in the first position (where the word "alias" is) should be unique. The "#" works as a comment in this file. You may comment out lines as you architect your environment.

3. Run this command as a test: ansible -m ping all

4. To be even more sure, run this command: ansible -m shell -a 'free -m' all

How Do You Install Jenkins 2.x on Ubuntu Linux in AWS?

Problem scenario
You want to install Jenkins on an Ubuntu Linux server without running an "apt-get upgrade" command. (You are ok with installing a "apt-transport-https" package.) What should you do?

Solution
See this posting because the directions work for Ubuntu in AWS and Debian in GCP.

(If you can run "apt-get upgrade", and you do not want to install "apt-transport-https", then see this posting to install Jenkins 2.x on Ubuntu Linux.)

How Do You Troubleshoot the Spark-Shell Error “A JNI error has occurred”?

Problem scenario
You run spark-shell in a Debian distribution of Linux (e.g., Ubuntu) but you receive this error:

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 64
        at java.util.jar.JarFile.match(java.base@9-internal/JarFile.java:983)
        at java.util.jar.JarFile.checkForSpecialAttributes(java.base@9-internal/JarFile.java:1017)
        at java.util.jar.JarFile.isMultiRelease(java.base@9-internal/JarFile.java:399)
        at java.util.jar.JarFile.getEntry(java.base@9-internal/JarFile.java:524)
        at java.util.jar.JarFile.getJarEntry(java.base@9-internal/JarFile.java:480)
at jdk.internal.util.jar.JarIndex.getJarIndex(java.base@9-internal/JarIndex.java:114)
        at jdk.internal.loader.URLClassPath$JarLoader$1.run(java.base@9-internal/URLClassPath.java:640)
        at jdk.internal.loader.URLClassPath$JarLoader$1.run(java.base@9-internal/URLClassPath.java:632)
        at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method)
        at jdk.internal.loader.URLClassPath$JarLoader.ensureOpen(java.base@9-internal/URLClassPath.java:631)
        at jdk.internal.loader.URLClassPath$JarLoader.<init>(java.base@9-internal/URLClassPath.java:606)
        at jdk.internal.loader.URLClassPath$3.run(java.base@9-internal/URLClassPath.java:386)
        at jdk.internal.loader.URLClassPath$3.run(java.base@9-internal/URLClassPath.java:376)
        at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method)
        at jdk.internal.loader.URLClassPath.getLoader(java.base@9-internal/URLClassPath.java:375)
        at jdk.internal.loader.URLClassPath.getLoader(java.base@9-internal/URLClassPath.java:352)
        at jdk.internal.loader.URLClassPath.getResource(java.base@9-internal/URLClassPath.java:218)
        at jdk.internal.loader.BuiltinClassLoader$3.run(java.base@9-internal/BuiltinClassLoader.java:463)
        at jdk.internal.loader.BuiltinClassLoader$3.run(java.base@9-internal/BuiltinClassLoader.java:460)
        at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method)
        at jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(java.base@9-internal/BuiltinClassLoader.java:459)
        at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(java.base@9-internal/BuiltinClassLoader.java:406)
        at jdk.internal.loader.BuiltinClassLoader.loadClass(java.base@9-internal/BuiltinClassLoader.java:364)
        at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(java.base@9-internal/ClassLoaders.java:184)
        at java.lang.ClassLoader.loadClass(java.base@9-internal/ClassLoader.java:419)
        at sun.launcher.LauncherHelper.loadMainClass(java.base@9-internal/LauncherHelper.java:585)
        at sun.launcher.LauncherHelper.checkAndLoadMain(java.base@9-internal/LauncherHelper.java:497)

How do you solve this?

Solution
Use Java version 8 not version 9.

Option 1
One option would be to uninstall Java (version 9) and reinstall Java (version 8). (You can view this posting for installing Java; make sure you make necessary changes so you install version 8.)

Option 2
If you have both installed and you are using Ubuntu, you can use this command:

sudo update-alternatives --config java

# To the prompt you will see, respond with the integer associated with Java 8 and press enter.

How Do You Install Apache Spark on Any Type of Linux?

Problem scenario
You want a generic script that can install open source Apache Spark on Debian/Ubuntu, CentOS/RedHat/Fedora or SUSE distributions of Linux. How do you do this?

Solution
1. Create a script such as this in /tmp/ (e.g., /tmp/spark.sh).

#!/bin/bash
# Written by www.continualintegration.com

sparkversion=2.2.1 # Change this version as necessary

distro=$(cat /etc/*-release | grep NAME)

debflag=$(echo $distro | grep -i "ubuntu")
if [ -z "$debflag" ]
then # If it is not Ubuntu, test if it is Debian.
debflag=$(echo $distro | grep -i "debian")
echo "determining Linux distribution..."
else
echo "You have Ubuntu Linux!"
fi

rhflag=$(echo $distro | grep -i "red*hat")
if [ -z "$rhflag" ]
then   #If it is not RedHat, see if it is CentOS or Fedora.
rhflag=$(echo $distro | grep -i "centos")
if [ -z "$rhflag" ]
    then    #If it is neither RedHat nor CentOS, see if it is Fedora.
    echo "It does not appear to be CentOS or RHEL..."
    rhflag=$(echo $distro | grep -i "fedora")
    fi
fi

if [ -z "$rhflag" ]
then
echo "...still determining Linux distribution..."
else
echo "You have a RedHat distribution (e.g., CentOS, RHEL, or Fedora)"
yum -y install java-1.8.0-openjdk*
JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk
echo 'export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk' >> ~/.bashrc
source ~/.bashrc
source /etc/environment
fi

if [ -z "$debflag" ]
then
echo "...still determining Linux distribution..."
else
   echo "You are using either Ubuntu Linux or Debian Linux."
   apt-get -y update # This is necessary on new AWS Ubuntu servers.
   apt -y install openjdk-8-jre-headless
fi

suseflag=$(echo $distro | grep -i "suse")
if [ -z "$suseflag" ]
then
if [ -z "$debflag" ]
then
    if [ -z "$rhflag" ]
      then
      echo "*******************************************"
      echo "Could not determine the Linux distribution!"
      echo "Installation aborted. Nothing was done."
      echo "******************************************"
      exit
    fi
fi
else
   zypper -n install java-1_8_0-openjdk
fi

curl http://ftp.wayne.edu/apache/spark/spark-$sparkversion/spark-$sparkversion-bin-hadoop2.7.tgz > /bin/spark-$sparkversion-bin-hadoop2.7.tgz

cd /bin
tar -zxvf spark-$sparkversion-bin-hadoop2.7.tgz
mkdir -p /usr/local/spark
mv /bin/spark-$sparkversion-bin-hadoop2.7/* /usr/local/spark

echo 'PATH="$PATH":/usr/local/spark/bin/' > /etc/profile.d/spark.sh

echo 'To test the installation, run "/usr/local/spark/bin/spark-shell"'

2. Run it like this:

sudo bash /tmp/spark.sh

How Do You Install OSSEC on Any Type of Linux?

Problem scenario
You have Debian/Ubuntu, RedHat (including CentOS and Fedora), and SUSE distributions of Linux. You want to install OSSEC on each server (to protect them with host-based intrusion detection systems, IDSes). You want to use the same script to install OSSEC on each server. How do you do this?

Solution
1. Create a script such as this /tmp/ossec.sh.

#!/bin/bash
# Written by www.continualintegration.com

ossecversion=3.1.0 # Change this version as necessary

filename=`basename "$0"`
numlines=$(wc -l $filename | awk '{print $1}')
if [ 83 -lt $numlines ] # change 83 to the correct number of lines if script grows
then
echo "line numbers too long. Were extra lines introduced accidentally during a cut and paste?"
echo "****************EXITING**********************************"
exit
else
echo "Script running..."
fi

distro=$(cat /etc/*-release | grep NAME)

if [ -z "$rhflag" ]
then
echo "...still determining Linux distribution..."
else
echo "You have a RedHat distribution (e.g., CentOS, RHEL, or Fedora)"
yum -y install httpd unzip wget gcc php sendmail python-inotify
fi

wget sendmail inotify-tools
fi

cd /tmp
wget https://github.com/ossec/ossec-hids/archive/$ossecversion.tar.gz
tar -zxvf $ossecversion.tar.gz
mv /tmp/ossec-hids-$ossecversion /bin/ossec #ossec-hids-$ossecversion
echo "go to /bin/ossec (with a cd) and run 'sudo bash install.sh'"
echo "Then follow the interactive text menu prompts after that. It should be self-explanatory."
echo "For a proof-of-concept, it is easiest if you do not configure email alerts or an SMTP notification server."

2. Run the script with this command: sudo bash ossec.sh