CONTINUAL INTEGRATION – Page 173 – A Technical I.T./DevOps Blog

How Do You Install SonarQube on a Red Hat Distribution of Linux?

Problem scenario
You have a Red Hat distribution of Linux (e.g., CentOS/RHEL/Fedora). How do you install SonarQube?

Solution

Prerequisites
Install the Java Development Kit so you can later take advantage of plugins. If you need assistance with this, see this link.

Procedures
For future reference, at some point the "9.6"s you see below will have to be incremented to a newer version of PostgreSQL.

1. If you are running RHEL, run this command: sudo yum -y update

If you are using CentOS, run these two commands:
sudo yum -y install epel-release
sudo yum -y update

2. If you are running RHEL, run this command:
sudo rpm -Uvh https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-redhat96-9.6-3.noarch.rpm

If you are running CentOS, run this command:
sudo rpm -Uvh https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-centos96-9.6-3.noarch.rpm

3. Run these commands:
sudo yum -y install postgresql96-server postgresql96-contrib
sudo /usr/pgsql-9.6/bin/postgresql96-setup initdb

4.a. Enable MD5-based authentication by editing the /var/lib/pgsql/9.6/data/pg_hba.conf file.

4.b. Run this command: sudo vi /var/lib/pgsql/9.6/data/pg_hba.conf

4.c. Find the following stanzas; change the word "peer" to "trust" and change the word "ident" to "md5". Here is how it may appear before you make these changes:

# TYPE DATABASE USER ADDRESS METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     peer
# IPv4 local connections:
host    all             all             127.0.0.1/32            ident
# IPv6 local connections:
host    all             all             ::1/128                 ident

Here is what it will look like after the changes:
# TYPE DATABASE USER ADDRESS METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            md5
# IPv6 local connections:
host    all             all             ::1/128                 md5

5. Run these two commands:

sudo systemctl start postgresql-9.6
sudo systemctl enable postgresql-9.6

6. Change the postgres user's password: sudo passwd postgres

7. Assume the postgres user with this command: su - postgres

8. Create a user named "sonar" with this command: createuser sonar

9. Enter the PostgreSQL command prompt by running this command: psql

10. Prepare a command such as this, but replace "goodPassword" with the password you want for the sonar PostgreSQL database user:

ALTER USER sonar WITH ENCRYPTED password 'goodPassword';

11. Run the prepared command above (after you replaced goodPassword with the password of your choice).

12. Run this command from the SQL command prompt: CREATE DATABASE sonar OWNER sonar;

13. Leave the psql prompt by running this command: \q

14. Exit the postgres user to be a user that is a sudoer. Run this: exit

15. Run a script with these five lines (using sudo privileges when you run it):

version=7.0
curl -L https://sonarsource.bintray.com/Distribution/sonarqube/sonarqube-$version.zip > /tmp/sonarqube-$version.zip
yum -y install unzip
unzip /tmp/sonarqube-$version.zip -d /opt
mv /opt/sonarqube-$version /opt/sonarqube

# Call the file above "sonar.sh" and put it in /tmp. Run it with "sudo bash /tmp/sonar.sh"

16. Modify the SonarQube configuration file. To enter it, run this: sudo vi /opt/sonarqube/conf/sonar.properties

Find the following two stanzas:

#sonar.jdbc.username=
#sonar.jdbc.password=

"Uncomment and provide the PostgreSQL username and password of the database that we have created earlier. It should look like:" (Taken from Vultr.com.)

sonar.jdbc.username=sonar
sonar.jdbc.password=goodPassword

17. While in the same file, find this stanza (it may be on the 39th line):

#sonar.jdbc.url=jdbc:postgresql://localhost/sonar

Uncomment the line, and save the file. (Also taken from Vultr.com.)

18.a Create this file: /etc/systemd/system/sonar.service

18.b. The contents should be the following (with the first line being "[Unit]" and last line being "WantedBy = multi-user.target"):

[Unit]
Description=SonarQube service
After=syslog.target network.target

[Service]
Type=forking

ExecStart=/opt/sonarqube/bin/linux-x86-64/sonar.sh start
ExecStop=/opt/sonarqube/bin/linux-x86-64/sonar.sh stop

User=sonar
Group=sonargroup
Restart=always

[Install]
WantedBy = multi-user.target

19.a. Run these commands:

sudo useradd sonar
sudo groupadd sonargroup
sudo passwd sonar # respond with a new password
sudo usermod -a -G sonargroup sonar

Run a command like this but replace sonar with the user that will start the sonar service and replace sonargroup with the user's group:

sudo chown -R sonar:sonargroup /opt/sonarqube

# If you do not know how to find the user's group, log in as the user and run the command "groups" to list the user's membership(s) of any group(s).

19.b. Run these three commands:
sudo systemctl start sonar
sudo systemctl enable sonar
sudo systemctl status sonar

20. Configure the firewall to allow connections. If you are in a special environment, you can disable your OS firewall with this command:

sudo systemctl stop firewalld # Normally you would configure a special exception in the firewall.
sudo setenforce 0

21. Run this command to install Apache web server: sudo yum -y install httpd

22.a Run this command, but replace x.x.x.x with the external IP address of your server (or the URL that you will present with this server):

sudo vi /etc/httpd/conf.d/sonar.x.x.x.x.conf

22.b. Place these lines in the file (where the top and bottom lines have "VirtualHost" in them):

<VirtualHost *:80>
    ServerName sonar.x.x.x.x
    ServerAdmin admin@continualintegration.com
    ProxyPreserveHost On
    ProxyPass / http://localhost:9000/
    ProxyPassReverse / http://localhost:9000/
    TransferLog /var/log/httpd/sonar.x.x.x.x_access.log
    ErrorLog /var/log/httpd/sonar.x.x.x.x_error.log
</VirtualHost>

22.c. Replace x.x.x.x with the IP address or domain name of your server.

22.d. Replace "admin@continualintegration.com" with an email address of yours.

23. Run these two commands:

sudo systemctl start httpd
sudo systemctl enable httpd

24. Open a web browser. Go to the external IP address of the server. You should see the web UI for SonarQube.

How Is the Password Set for a Challenge for a Password from a “git clone git@…” command?

Problem scenario
A "git clone git@..." command prompts the user for a password. What is the password or how is it set?

Solution
The authentication for such a git command is usually done by the individual servers not through Git itself. You should go to the server with the source repository, and run this to change the password to something you know:

sudo passwd git

Alternatively you could change the owner and group of the .git file or the permissions of the git file. Then you could set up SSH with a user on the server that has the .git repo that has permissions to access the file. To set up SSH see this posting.

Why Do Linux Commands Not Work when They Work as Part of a Python Program with the subprocess.check_output Invocation?

Problem scenario
You are trying to debug a Python program that uses shell (or Bash) commands. The shell or Bash commands work when they are called with Python's subprocess.check_output. They do not work when called directly from the Linux command prompt. What is wrong?

Possible solutions
Normally I.T. professionals can get the Linux (Bash or shell) commands to work without as many moving parts. In some instances the Python script that executes the Bash commands seem to help the reliability of the commands. Sometimes you inherit code that you must maintain. In these instances you need to reverse engineer the code. These solutions below were made for the scenario where the Python program works, but when the individual Linux command is run separately without Python, the Linux command fails by itself.

1. Does the Python code have backslashes "\"? These are used as "escape" characters so the parsing of the Bash commands will be done properly. Some quotation marks are needed in Bash commands. The Python interpreter may not know if it should parse the punctuation itself or save it for the Bash command. It may be necessary to have these escape characters in the Python program. However these may cause the Bash commands to fail when run outside the context of a Python program (e.g., if they are lifted verbatim and run interactively for testing purposes).

2. Are there environment variables that are involved? It could be that manually running the Bash commands reveals that your user context is different from the Python invocation of such commands. As a debugging tool, compare the output of an "env" command run interactively to the same command run from the Python program. Some utilities in Linux may need their full subdirectory path to be called to work via Python subprocess.check_output invocations.

3. Does the Python code do some sort of authentication process? When trying to manually troubleshoot Linux commands used in a Python program, see if there are previous lines of code in the Python program that log into a system. Do not get ahead of yourself or make assumptions. Some Linux commands may only work if the sequence is exact.

How Do You Run a Groovy Program on a Debian or Ubuntu Server?

Problem scenario
You want to use a Java Virtual Machine language on an Ubuntu Linux server. You want to test out the language too. What should you do?

Prerequisites
Install Groovy version 1; this command should work: sudo apt-get -y update; sudo apt-get -y install groovy

Procedures
1. Create a file called hello.groovy with the following five lines:

class Example {
   static void main(String[] args) {
      println('Hello World');
   }
}

2. Run it like this: groovy hello.groovy

How Do You Troubleshoot the Message “/usr/bin/mongodb/bin/mongod: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory”?

Problem scenario
You are using a Red Hat derivative of Linux (e.g., CentOS/RHEL/Fedora). When trying to run a mongod command you receive this message: "/usr/bin/mongodb/bin/mongod: error while loading shared libraries: libcrypto.so.1.0.0: cannot open shared object file: No such file or directory".

What should you do?

Solution
Get different installation media. If you try to install a .tgz file for Ubuntu on CentOS/RHEL/Fedora, you will get this message. The solution is to get new installation media.

How Do You Troubleshoot a C Program Displaying the Error “Segmentation fault” at Run Time?

Problem scenario
Your C program compiles. But you get the error "Segmentation fault." You expect the value of a variable to be printed. How do you get the value to be printed without this error?

Solution
Does every print function print? If one print function seems to be displaying the "Segmentation fault" error, look closely at the syntax thereof. Here is a line that will produce this error:

printf ("y is a %s\n", y);

Here is a similar but corrected line that will be error free:

printf ("y is a %d\n",y);

The "%s" is appropriate for printing string variables. For integer variables, use "%d" syntax.

How Do You Fix a Non-Terminating (Hanging) Command Such as “git push origin master” ?

Problem scenario
The command "git push origin master" hangs. What should you do?

Solution
Cancel the job with ctrl-c. (Hold the "Control" key and tap the "C" key.)

For the target .git repository, does the file have the owner and group associated with the "git" user?

If you used the git user in the "git clone" command (as in "git clone @...") but then changed the owner and group of the .git repository to something else, you may see an indefinite hang after the "git push origin master" command.

Can the Kernel Access Hardware Directly?

Question
Can the kernel, or code running in it, access hardware directly?

Answer
Yes.

kernelspace lives in RAM (or memory) (1). Code that is processing in kernelspace can fully access other processes in memory or fully access hardware directly (1).

The kernel governs hardware utilization via drivers, memory, disk space, memory and CPU (1).

In RedHat distributions of Linux (including CentOS and Fedora) userspace exists in RAM (or memory) (2). Running applications live in userspace (2). "Code running in userspace cannot access hardware directly and cannot access memory allocated to other applications." (2)

Here is the guide:
(1) Page 1257 of 7th Edition of A Practical Guide to Fedora and RedHat Enterprise Linux by Mark G. Sobell
(2) Page 1279 of 7th Edition of A Practical Guide to Fedora and RedHat Enterprise Linux by Mark G. Sobell

How Do You Deal with the Python Error “IndexError: pop index out of range”?

Problem scenario
You have a Python list with n items. You are getting a "IndexError: pop index out of range" error. The order that these lists are being printed (used or displayed) is not what you expect. Your program prints the entire list initially, so you know what to expect. This IndexError and the order of the items being displayed is not what you expect. Here is an example:

>>> print(fun_list)
['item1', 'item2', 'item3', 'item4']

>>> print(fun_list.pop(0))
item1
>>> print(fun_list.pop(1))
item3

>>> print(fun_list.pop(2))
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
IndexError: pop index out of range

What is wrong?

Root cause
The .pop operation removes the item from the list each time the .pop operation is used. So the index becomes different because the list itself is different. Pop is destructive. The list is mutable.

Solution
Do not use pop. Print the items via the bracket syntax [] and the 0-based indexing.

Here is an example:

>>> fun_list = ['item1', 'item2', 'item3', 'item4']
>>> print fun_list[0]
item1
>>> print fun_list[1]
item2
>>> print fun_list[2]
item3
>>> print fun_list[3]
item4

Caveat: there are times when the pop operation is useful. If you are getting this error and do not know why, we would recommend avoiding pop.

If you want to learn more about Python, you may want to purchase a book on the subject.

How Do You Get hdfs or Yarn to Start the Jps Process You Expect to Start?

Problem scenario
You use start-dfs.sh and start-yarn.sh and there are no errors. They seem to work, but when you use the jps command, you do not see the service you expect. Why isn't a jps process starting when you run one of these scripts?

Solution
The root cause could be a variety of root causes. Here are some potential solutions.

1. In these files, is there an all uppercase stanza like one or both of the following?

HADOOP_SECURE_DN_USER=yarn

HDFS_NAMENODE_USER=root

If it has never worked, you may want to comment out such stanzas.

2. Are you using sudo to start these start-dfs.sh and start-yarn.sh scripts? Try not using sudo before you run the scripts.

3. You want want to set up Hadoop again. This link can help you with deploying open source Hadoop.