A Technical I.T./DevOps Blog
Problem scenario: On Ubuntu, after installing Hadoop, when you enter the command '/usr/local/hadoop/bin/hadoop namenode -format' and you get this error:
"Error: JAVA_HOME is not set and could not be found."
Solution: Verify Java is installed ('java -version'). If it is not installed, you can use 'apt-get install -y default-jre'.
Next, go to hadoop-env.sh. Find the export JAVA_HOME stanza. Change the '${JAVA_HOME}' value to '/usr' with no quotes.
Problem scenario
When you log into Ubuntu, you see a message like this:
8 packages can be updated.
5 updates are security updates.
You run this: sudo apt-get -y update
This completes. But next time you log in, you see the same message about 8 packages can be updated. What do you do?
Solution
Run this command: sudo apt-get -y upgrade
If you are looking for deploying Hadoop to Ubuntu, see this article.
Updated on 11/28/17
Problem scenario
You want to install Suricata on Ubuntu Linux. How do you do this?
Solution
These directions to install Suricata include a script and how to run it. The optional script in 2.c was based on these here. The optional script in 2.c script requires that your AWS Ubuntu Linux server is in a security group that has access to the Internet. The optional script takes approximately five minutes to run. But bandwidth and resources on your instance may vary. It is advisable to use step 2.a and skip steps 2.b, 2.c, and 3.
Step #1 Log in.
Step #2.a Run this command: sudo apt-get -y update; sudo apt-get -y install suricata
Now go to step #4. If for some reason you do not want to install it from the AWS .deb packages, skip step #2a and go to step #2.b
Step #2.b If you did not do step #2.a, run this command: vi /tmp/installer.sh
Step #2.c If you did not do step #2.a, do this step. This step involves copying the content below to be installer.sh. This script starts with the "#!/bin/bash" line. The final line of the script is "ldconfig"
#!/bin/bash
apt-get install -y libpcre3
apt-get install -y libpcre3-dbg
apt-get install -y libpcre3-dev
sleep 5
apt-get install -y build-essential
apt-get install -y autoconf
apt-get install -y automake
apt-get install -y libtool
apt-get install -y libpcap-dev
apt-get install -y libnet1-dev
apt-get install -y libyaml-0-2
apt-get install -y libyaml-dev
apt-get install -y zlib1g
apt-get install -y zlib1g-dev
apt-get install -y libcap-ng-dev
apt-get install -y libcap-ng0
apt-get install -y make
apt-get install -y libmagic-dev
apt-get update
sleep 5
apt-get install -y libpcre3
apt-get install -y libpcre3-dbg
apt-get install -y libpcre3-dev
apt-get install -y build-essential
apt-get install -y autoconf
apt-get install -y automake
apt-get install -y libtool
apt-get install -y libpcap-dev
apt-get install -y libnet1-dev
apt-get install -y libyaml-0-2
apt-get install -y libyaml-dev
apt-get install -y zlib1g
apt-get install -y zlib1g-dev
apt-get install -y libcap-ng-dev
apt-get install -y libcap-ng0
apt-get install -y make
apt-get install -y libmagic-dev
apt-get update
cd /bin
mkdir suricata
cd suricata
wget http://pkgconfig.freedesktop.org/releases/pkg-config-0.28.tar.gz
tar -xvzf pkg-config-0.28.tar.gz
cd pkg-config-0.28
./configure --with-internal-glib
# to not downgrade glib (ldd --version shows 2.19)
make
make install
cd /bin/suricata
wget http://www.openinfosecfoundation.org/download/suricata-1.4.6.tar.gz
tar -xvzf suricata-1.4.6.tar.gz
cd suricata-1.4.6
./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var
make install-full
ldconfigStep #3 Run the script with this command: sudo bash /tmp/installer.sh
Step #4 Run this command to confirm Suricata is installed: suricata -V
Problem Scenario: You want to use Ant to invoke your customized build.xml file. But when you run it, you get "
Target "build.xml" does not exist in the project ..."
Solution: Call ant with no other variables. Do not use "ant build.xml". Use "ant" by itself.
Miscellaneous: Most CentOS/RedHat repositories will have Ant available. As root, the command "yum install ant" should get it installed. You will have to respond "y" to some of the prompts. Here is a sample build.xml file that works with Ant.
<?xml version="1.0"?>
<project name="Continual Integration Project" default="info">
<target name ="info">
<echo>You correctly invoked Ant!</echo>
</target>
</project>
Question 1: What are the packages that must be installed on a CentOS server to have it serve as the Kerberos server?
Answer 1: These three packages: krb5-server krb5-libs krb5-auth-dialog
Question 2: What are the packages that must be installed on a CentOS server to have it serve as the Kerberos client?
Answer 2: These three packages: krb5-workstation krb5-libs krb5-auth-dialog
Problem scenario: When using Jenkins, PsExec never seems to work when called as a batch command in a .bat file or PowerShell script. Jenkins logs an error about the PsExec command not being found.
Solution / workaround: PsExec seems to never work if called by a PowerShell script or a batch script via Jenkins. The only exception seems to be if a Scheduled Task calls a .bat file or PowerShell script that uses PsExec. First, create a Scheduled Task that calls a .bat file or PowerShell script that invokes PsExec. Second, create a Jenkins job that has a Build Step of a Windows Batch command or Windows PowerShell command (as either will suffice). Design the PowerShell or batch command to invoke the Scheduled Task, use a command like this:
SCHTASKS /RUN /TN foobar
where "foobar" is the name of the Scheduled Task that you want to run. Now the Jenkins job will call the Scheduled Task which will in turn invoke PsExec.
Alternatively, you may want to go to Services and change the user for the Jenkins service. The service will need to be restarted after you enter the credentials. Finally, PowerShell scripts are known to potentially act differently if they are executed with a local user or from a remote server. Scheduled Tasks can be a workaround if the remote execution of a PowerShell script that fails but works when it is run locally. Don Jones has experienced this problem. This problem seems to happen mostly with very complex or involved PowerShell scripts (e.g., those that install big applications).
Updated 11/29/17
These directions will allow you to install Suricata. The script in step 2 was designed to install Suricata 4.0.1 on an AWS instance of RHEL 7.4. This script requires that your AWS RedHat Enterprise Linux server is in a security group that has access to the internet. You do not need a subscription to RedHat packages.
Step #1 Log into the Red Hat Enterprise Linux server.
Step #2 Create /tmp/installer.sh with the following content:
#!/bin/bash
yum-config-manager --enable rhui-REGION-rhel-server-extras rhui-REGION-rhel-server-optional
yum -y install wget libpcap-devel libnet-devel pcre-devel gcc-c++ automake autoconf libtool make libyaml-devel zlib-devel file-devel jansson-devel nss-devel
wget https://www.openinfosecfoundation.org/download/suricata-4.0.1.tar.gz
tar -xvzf suricata-4.0.1.tar.gz
cd suricata-4.0.1
./configure
make
make install
Step #3 Run the script with this command: sudo bash /tmp/installer.sh
Step #4 Run this command to confirm Suricata is installed:
suricata -V
Step #5. You are done.
FFR
The following 85 line script used to work to install Suricata 3.1 on an AWS instance of RHEL 7.2. It is here for reference (in case you are working on a legacy project).
This below script was based on these here. This script uses Fedora packages instead. Some shared objects and files are backed up before the installation of the Fedora rpms. The script restores these files. The script takes approximately five minutes to run. But bandwidth and resources on your instance may vary.
# Then input this script starting with the "#!/bin/bash" line below. The final line is "make install-full"
#!/bin/bash
mkdir /bin/suricata/
mkdir /tmp/contint
cd /bin/suricata
sudo yum -y install gcc libpcap-devel pcre-devel libyaml-devel file-devel \
zlib-devel jansson-devel nss-devel libcap-ng-devel libnet-devel tar make \
libnetfilter_queue-devel lua-devel
yum -y install wget
yum -y install bzip2
cp /usr/lib64/libyaml-0.so.2.0.4 /tmp/libyaml-0.so.2.0.4
wget ftp://rpmfind.net/linux/fedora/linux/releases/24/Everything/x86_64/os/Packages/l/libyaml-0.1.6-8.fc24.x86_64.rpm
rpm -ivh --force libyaml-0.1.6-8.fc24.x86_64.rpm
wget ftp://rpmfind.net/linux/fedora/linux/releases/24/Everything/x86_64/os/Packages/l/libyaml-devel-0.1.6-8.fc24.x86_64.rpm
rpm -ivh --force libyaml-devel-0.1.6-8.fc24.x86_64.rpm
cp /tmp/libyaml-0.so.2.0.4 /usr/lib64/libyaml-0.so.2.0.4
cp /usr/lib64/libpcap.so.1 /tmp/contint
cp /usr/share/man/man7/pcap-filter.7.gz /tmp/contint
cp /usr/share/man/man7/pcap-linktype.7.gz /tmp/contint
cp /usr/share/man/man7/pcap-tstamp.7.gz /tmp/contint
wget ftp://rpmfind.net/linux/fedora/linux/releases/24/Everything/x86_64/os/Packages/l/libpcap-1.7.4-2.fc24.x86_64.rpm
rpm -ivh --force libpcap-1.7.4-2.fc24.x86_64.rpm
wget ftp://rpmfind.net/linux/fedora/linux/releases/24/Everything/x86_64/os/Packages/l/libpcap-devel-1.7.4-2.fc24.x86_64.rpm
rpm -ivh libpcap-devel-1.7.4-2.fc24.x86_64.rpm
rpm -ivh --force libpcap-1.7.4-2.fc24.x86_64.rpm
cp /usr/lib64/libpcap.so.1 /tmp/contint
cp /usr/share/man/man7/pcap-filter.7.gz /tmp/contint
cp /usr/share/man/man7/pcap-linktype.7.gz /tmp/contint
cp /usr/share/man/man7/pcap-tstamp.7.gz /tmp/contint
rpm -ivh --force file-libs-5.25-6.fc24.x86_64.rpm
wget ftp://rpmfind.net/linux/fedora/linux/releases/24/Everything/x86_64/os/Packages/f/file-libs-5.25-6.fc24.x86_64.rpm
rpm -ivh --force file-libs-5.25-6.fc24.x86_64.rpm
cp /usr/bin/file /tmp/contint/file
cp /usr/share/man/man1/file.1.gz /tmp/contint/file.1.gz
wget ftp://rpmfind.net/linux/fedora/linux/releases/24/Everything/x86_64/os/Packages/f/file-5.25-6.fc24.x86_64.rpm
rpm -ivh --force file-5.25-6.fc24.x86_64.rpm
cp /tmp/contint/libmagic.so.1.0.0 /usr/lib64/libmagic.so.1.0.0
cp /tmp/contint/magic /usr/share/misc/magic
cp /tmp/contint/magic.5.gz /usr/share/man/man5/magic.5.gz
cp /tmp/contint/magic.mgc /usr/share/misc/magic.mgc
cp /tmp/contint/file /usr/bin/file
cp /tmp/contint/file.1.gz /usr/share/man/man1/file.1.gz
wget ftp://rpmfind.net/linux/fedora/linux/releases/24/Everything/x86_64/os/Packages/f/file-devel-5.25-6.fc24.x86_64.rpm
rpm -ivh file-devel-5.25-6.fc24.x86_64.rpm
wget ftp://ftp.netfilter.org/pub/libnfnetlink/libnfnetlink-1.0.1.tar.bz2
tar tvjf libnfnetlink-1.0.1.tar.bz2
./configure
make
make install
wget http://www.openinfosecfoundation.org/download/suricata-3.1.tar.gz
tar -xvzf suricata-3.1.tar.gz
cd suricata-3.1
./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --enable-nfqueue --enable-lua
./configure
make install-full
Not all directions are clear. Here is an example where Microsoft has less than perfect directions:
"Installation
Just copy PsExec onto your executable path. Typing "psexec" displays its usage syntax."
First off, psexec won't display usage syntax until after the installation. If you are like me you are asking what is the executable path?
The answer can be found by opening PowerShell and typing this:
Get-ChildItem Env:Path | ft -wrap -AutoSize
Any of those semi-colon delimited results can be the destination of the files inside the PsTools.zip file. The files must not be in a subdirectory of the results shown by the above PowerShell command. They must be directly in the folder shown by the above PowerShell command.
It is an easy install because no services need to be recycled. No reboot is needed either. Psexec is useful because the other Windows servers on the network need no special configuration if you have a common administrator account. The other Windows servers do not need to be part of a domain. The network logon will be a Type 3 (as seen in the Event Viewer). This is considered safe by SANS.
Problem scenario: Using a configuration management tool, you can modify the selections of servers for your configurations to exclude specific servers. With Puppet Master you can create classes of nodes (that exclude various Puppet Agent servers) and have manifests apply to the class. In Puppet you can also write manifests that specify individual servers. Chef recipes can target certain Chef clients and exclude others. SaltStack allows the Salt Master server to "call a highstate" (or run an action) on specific Salt Minion servers to push down files or execute commands while ignoring multiple Salt Minions. Ansible playbooks can be ran with an --extra-vars flag to limit the target servers that receive the configuration changes.
How do you do this in Jenkins?
Solution: The first part of the solution is to configure nodes so Jenkins master is configured to communicate with the servers you want to target. (There used to be an external page that was helpful here: https://wiki.jenkins-ci.org/display/JENKINS/Step+by+step+guide+to+set+up+master+and+slave+machines .)
Contrary to what the content of link says, you may need to install Jenkins on Windows servers if your Jenkins master is running on Linux. Homogeneous operating systems or having your Jenkins run on Windows will obviate the need for you to install Jenkins on slave machines. There are other ways, but installing Jenkins on the Windows servers can help you securely deploy from the Jenkins server running on Linux.
The second part of the solution happens when you are creating a new build. When you create a job in Jenkins (e.g., New Item -> Freestyle Project -> ...) click "Restrict where this project can be run." In the field that appears below it, enter the label of the node (e.g., it may be an IP address but it depends on what you entered when you configured Jenkins to manage the node).
Create the build as you normally would, and it will only run on the servers that you restricted it to run on.
Problem scenario: You are in the Jenkins master web UI and are trying to configure a Jenkins node (previously known as a slave) instance. You entered the correct hostname, username and password. You get this error:
"ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
java.lang.IllegalStateException: Connection is not established!"
Solution: Instead of using the hostname of the Jenkins node (previously known as a slave) server, use its IP address.
================
Problem scenario: You installed a new Jenkins instance. When you go to the IP address over port 8080 in a web browser, you get a blank white screen.
Solution: Go to Internet Explorer's tools (click the gear symbol if that is the version you have) and go to Options. Go to the Security tab. At the top, click the "Trusted Sites" icon. Click "Sites." Enter "http://x.x.x.x" with no quotes and replace x.x.x.x with the IP address of Jenkins. Click Add. Click Close. Click Ok. Now refresh your web browser. You should be able to go to Jenkins' UI.