A Technical I.T./DevOps Blog
Problem scenario
You use the mailq command. You see several emails have timed out. These emails are essentially in an "outbox" and a future delivery attempt is in the near future. How do you produce a list of the mail queue IDs by themselves?
Solution
If you want a list of the mail queue IDs in the queue, run this:mailq | awk '{print $1}' | grep -v '[[:punct:]]' | sort | uniq
Problem scenario
You have a sed command in a script, but you get an error like one of the following:
sed: -e expression #1, char 1: unknown command: '' sed: -e expression #1, char 1: unknown command:"'
How do you troubleshoot this?
Solution
You may need to assign variables like this:
var1=" foo""'"" bar" # Place quotes around any unusual characters (e.g., spaces or punctuation)
Use the escape character:
var2="info \'"
Combine the subvariables into one variable:
compvar=$var1$var2
See this posting if you have any further questions.
Problem scenario
You have a MySQL database. Rather than use a SQL front-end application, you want to write a Java program that will run a SQL statement against the database. What should you do?
Solution
1. Install the Java compiler if it is not already installed. If you need assistance, see this posting.
2. Run this command:curl -Lk https://www.javatpoint.com/src/jdbc/mysql-connector.jar > /tmp/mysql-connector.jar
3. Then run this command: sudo find / -name ext | grep jre
4. Place the mysql-connector.jar in the directory found above (e.g., with sudo mv).
5. Set the CLASSPATH variable in two steps. First set it to where the .class file is for the Java file you are running. If the .class file of the program you are running is in /home/jdoe/javaprograms run this command: export CLASSPATH=/home/jdoe/javaprograms
6. Draft this command but substitute "/path/to/mysql-connector.jar" with the path referred to in step #4.
export CLASSPATH=$CLASSPATH:/path/to/mysql-connector.jar
7. Create a .java program with the following content:
import java.sql.*;
class MysqlCon{
public static void main(String args[]){
try{
Class.forName("com.mysql.jdbc.Driver");
Connection con=DriverManager.getConnection(
"jdbc:mysql://auroraendpoint-us-west-2b.abcd1234.us-west-2.rds.amazonaws.com","jdoe","password");
Statement stmt=con.createStatement();
ResultSet rs=stmt.executeQuery("show databases;");
while(rs.next())
System.out.println(rs.getString(1) );
con.close();
}catch(Exception e){ System.out.println(e);}
8.a. Replace "auroraendpoint-us-west-2b.abcd1234.us-west-2.rds.amazonaws.com" with your Aurora MySQL database endpoint. It can be found in the AWS web console or with the AWS CLI. (If you need assistance installing and configuring the AWS CLI, see this posting.) With the AWS CLI, you can run this command to find it:
aws rds describe-db-instances | grep Address
(Unlike the traditional JDBC string for MySQL, the final word is not the database name if, and only if, you are using Aurora MySQL.) If you are not using an Aurora MySQL database, you will want to follow this format (for replacing "auroraendpoint-us-west-2b.abcd1234.us-west-2.rds.amazonaws.com"): "FQDN_of_host/database_name"
8.b. Replace "jdoe" with your MySQL database username.
8.c. Replace "password" with your MySQL database password.
9. Compile the program: javac MysqlCon.java
10. Run the program: java MysqlCon
Problem scenario
You have configured a Docker container with a web service (e.g., Apache web server or Nginx). You configured the listening, external port to be 80 or a different port number. You find the web server (either Apache or Nginx) is not working from a web browser. How do you find what is wrong?
Solution
#1 Use nmap to test the port and IP address. If you are using Windows, see this URL to use PowerShell to test the port and IP address. If there is no activity, one of the following tips may apply:
a) an intermediate firewall is blocking the connectivity. The results of an nmap command would show "filtered." The result "open" shows activity, and "closed" shows nothing is blocking the connection but nothing is listening on that port either. PowerShell would likely just show connectivity or a lack thereof. To turn off the firewall on an Ubuntu server (which is on by default on an AWS instance of Ubuntu), run this command: sudo ufw disable
To learn more about getting around a potential firewall, see this posting.
b) it could be that the host of the Docker container has IPv6 enabled. Disable ipv6. If you are running Ubuntu or a Debian distribution of Linux, use this link to disable IP version 6. If the server is a RedHat derivative, follow these steps to disable IP version 6:
i) sudo cp /etc/modprobe.d/disableipv6.conf /etc/modprobe.d/bak.disableipv6.conf.bak
(Do not be concerned if the source file does not yet exist. The above command is precautionary.)
ii) You need to become root to run a special echo command.sudo su -
echo "install ipv6 /bin/true" >> /etc/modprobe.d/disableipv6.conf
iii) reboot the server.
c) Verify the Docker service is running: docker ps -a
If it is not running, try this command if you are running a RedHat derivative: sudo systemctl start docker
Then start the Docker container with the web service.
#2 Verify that port forwarding is configured properly. If you are using a RedHat derivative, run this command from the OS: sudo sysctl net.ipv4.conf.all.forwarding
The result should end with "= 1." If it does not and it ends with an "= 0", run this command if the Linux server is a RedHat derivative: sysctl net.ipv4.conf.all.forwarding=1
#3 If there is no activity over the port (it isn't filtered/blocked by an intermediate firewall but there seems to be nothing listening on the port), try rebooting the server.
#4 Verify the Docker network IP address is within the range of the IP address on the Linux server. For example, run ip addr show. This should show a range of IP addresses near the word inet. If you misconfigured the Docker network to have an IP address too low or high for the range of IP addresses that can be seen by eth0 (or another known working NIC), then you may need to reconfigure the Docker network.
#5 If you notice that there is activity on the port that you expect for a small amount of time (e.g., three minutes) after the container has been started yet this activity goes away on its own, there could be two potential problems: one, there could be another Docker container or service competing for the port. Secondly there could be a misconfigured network interface. You may want to shut off other services and turn on only what you are testing. If the problem persists after a reboot with minimal services turned on, you may want to start over with your user-defined network. To deploy several Docker containers on one server with Nginx running in each of them simultaneously, try this link.
#6 You may want to examine Docker logs thoroughly. This link has information on how to do that. You may want to get a book on Docker; here is a good list.
Problem scenario
You cannot ping a server, but your nmap results show that port 22 on the server is open. You can reach the server with nmap, but not with ping. What is wrong?
Possible solution
Is the server in the AWS Security Group that you think it is? Go to the AWS console and verify the security group for the server is what you think is correct. You may want to see this posting.
Problem scenario
You want to create a Lambda function. What do you do?
Solution
Problem scenario
You like adding logging messages manually in Linux/Unix. You do it from the command prompt with "# write a cool message here". You review the messages with a "history" command. You also append messages to log files in /var/log/foorbar.d with 'echo "note this" >>'. How do you introduce your own messages into Windows event log?
Solution
Run a command like this but change "Application" to "Security" or whichever log category you want:
Write-EventLog -LogName Application -EventId 3 -Message "Continual Integration helps!" -Source "Windows Error Reporting"
# Note that "Windows Error Reporting" is just one of several valid sources. The EventID could be a different integer too.Problem scenario
You want to change the main configuration file for Apache web server (e.g., httpd.conf). But you do not want to drop current web surfers viewing the website. How do you change a security module or listening port for an Apache web server and have the changes take effect without interrupting the service by doing a regular restart?
Solution
Use the "graceful" directive. The exact command depends on the distribution of Linux that is running apache.
To learn more about it, see one of the links below:
https://www.cyberciti.biz/faq/apache-2-reload-httpd-config-file-unix-linux-command/
https://www.liquidweb.com/kb/restarting-services-from-the-command-line/
Explanation
Behind the scenes signals are sent to a [web service] process. SIGHUP would interrupt the service.* SIGUSR1 would be a signal that would facilitate a graceful reloading and prevent interruption.*
*Taken from https://www.liquidweb.com/kb/restarting-services-from-the-command-line/
For further context and knowledge you may want to read this related posting.
Problem scenario
You are trying to install open source Accumulo on Linux. You have two GB of swap space. You have installed Java, Hadoop, and Zookeeper. You have run the bootstrap_config.sh script for Accumulo 1.9.2.
You run this (and expected it to work): /bin/accumulo-1.9.2/bin/accumulo init
But you get this error:
OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N 2019-03-20 17:21:56,705 [start.Main] ERROR: Problem initializing the class loader java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.accumulo.start.Main.getClassLoader(Main.java:101) at org.apache.accumulo.start.Main.main(Main.java:48) Caused by: java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory at org.apache.commons.vfs2.impl.DefaultFileSystemManager.(DefaultFileSystemManager.java:136) at org.apache.accumulo.start.classloader.vfs.AccumuloVFSClassLoader.generateVfs(AccumuloVFSClassLoader.java:258) at org.apache.accumulo.start.classloader.vfs.AccumuloVFSClassLoader.getClassLoader(AccumuloVFSClassLoader.java:210) ... 6 more Caused by: java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at org.apache.accumulo.start.classloader.AccumuloClassLoader$2.loadClass(AccumuloClassLoader.java:294) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 9 more Exception in thread "Thread-0" java.lang.NoClassDefFoundError: org/apache/commons/io/FileUtils at org.apache.accumulo.start.classloader.vfs.AccumuloVFSClassLoader.close(AccumuloVFSClassLoader.java:425) at org.apache.accumulo.start.classloader.vfs.AccumuloVFSClassLoader$AccumuloVFSClassLoaderShutdownThread.run(AccumuloVFSClassLoader.java:75) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: org.apache.commons.io.FileUtils at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at org.apache.accumulo.start.classloader.AccumuloClassLoader$2.loadClass(AccumuloClassLoader.java:294) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 3 moreYou have modified this script purely for troubleshooting purposes: /bin/accumulo-1.9.2/bin/accumulo
You have it print out the $CLASSPATH variable when it runs. In two directories that you see when you run it (before it errors out), you have copied the commons-io-2.5.jar file. This way two directories definitely have the .jar file. You changed the permissions and ownership of these files to various permissive settings. You kept reproducing the problem after you did this. The problem even happens when you log in as root and start HDFS and Yarn and then run accumulo init. Why are you getting the error above when the commons-io-2.5.jar file is present in the classpath? How do you install Accumulo and run the accumulo init script?
Solution
Modify the accumulo-site.xml file. It may be here on your server: /usr/bin/accumulo-1.9.2/conf/accumulo-site.xml
Determine the version of Hadoop you are using (e.g., 2 or 3, or something else). This example will be for Hadoop 3. Add the third line in the below section (you'll see lines 1, 2, and 4 below in the file above; the third line needs to be added):
<!-- Hadoop 3 requirements -->
$HADOOP_PREFIX/share/hadoop/client/[^.].*.jar,
$HADOOP_PREFIX/share/hadoop/common/lib/.*.jar,
<!-- End Hadoop 3 requirements -->Problem scenario
You want to set up Zookeeper with three nodes in AWS. What do you do?
Solution
1. Install Zookeeper on each of the servers. If you need assistance with this, see this posting.
2. Modify the zoo.cfg file on each of the servers. Add stanzas like these but substitute foobarX.amazonaws.com with the Public DNS name of each server:
server.1=foobar1.amazonaws.com:2888:3888
server.2=foobar2.amazonaws.com:2888:3888
server.3=foobar3.amazonaws.com:2888:3888
initLimit=5
syncLimit=5
3. The assignment of server.1, server.2, and server.3 should each be the exact same. That is, zoo.cfg should be the same on each server.
4. Create a file /var/zookeeper/myid on each server. The value should be either a 1, 2, or 3 based on the server.X assignments above. So the server with the Public DNS name of foobar2.amazonaws.com should have a file /var/zookeeper/id with "2" (with no quotes) because it was assigned in zoo.cfg with the value server.2.
5. Make sure the Security Group governing the servers allow for connectivity based on the internal IP address of each server. The servers need to communicate with each other.
6. Go to your zookeeper directory (e.g., /bin/zookeeper-3.4.13/) and go to the bin subdirectory there in.
7. Run this command: sudo bash zkServer.sh start-foreground