How Do You Connect to Your Apache Spark Deployment in AWS?

Problem scenario
You have recently deployed Apache Spark to AWS.  You see the EC-2 instances were created.  But you cannot access them over the web UI (even over ports 4140, 8088, or 50070).  You cannot access the instances via Putty.  You changed your normal Security Group to allow TCP communication from your work station's IP address.  What should you do to connect to your new Spark instance for the first time?

Solution
The root cause is the deployment of Apache Spark creates security groups behind-the-scenes.  Go to Security Groups and look for a new security group.  The "Group Name" will be something like "ElasticMapReduce-master" and "ElasticMapReduce-slave."  The "Description" will be something like "Master group for Elastic MapReduce..." or "Slave group for Elastic MapReduce..."  Change these security groups to allow for inbound connections from your workstation's IP address. Now you should be able to connect to your Spark instance.

How Do You Deploy an Apache Spark Cluster in AWS?

Problem scenario
You want to deploy Apache Spark to AWS.  How do you do this?

Solution
1.  Log into the AWS management console.  Once in, go to this link.

2.  Click "Create cluster" and then "Quick Create"

3. For Software Configuration, choose "Spark:..."

4. For "Security and access", for EC2 key pair, choose the key pair you desire.

5.  Click "Create Cluster"

How Do You Change the Password of the PostgreSQL Instance Account Named “Postgres”?

Problem scenario
You want to change the password at the SQL level -- not with the OS -- of the user named Postgres in your PostgreSQL install.  How do you change this password?

Solution
1.  Enter the SQL database to run a DDL command.
2.  Run this command to change the user named "postgres" to the password "abcd1234" with no quotes:

ALTER USER "postgres" WITH PASSWORD 'abcd1234';

How Do You Solve the Problem with Apache Web Server That Says “Invalid Mutex directory in argument file:${APACHE_LOCK_DIR}”?

Problem scenario
Your website is not working.  When you browse to the login page, the browser loads forever.  There seems to be a problem with the web server itself.

On the back end, you run this command from the operating system's command line:
netstat -anlp | apache2

It produces output like this:

"
[Sun Aug 06 21:03:22.088579 2017] [core:warn] [pid 30144] AH00111: Config variable ${APACHE_LOCK_DIR} is not defined
[Sun Aug 06 21:03:22.088682 2017] [core:warn] [pid 30144] AH00111: Config variable ${APACHE_PID_FILE} is not defined
[Sun Aug 06 21:03:22.088697 2017] [core:warn] [pid 30144] AH00111: Config variable ${APACHE_RUN_USER} is not defined
[Sun Aug 06 21:03:22.088701 2017] [core:warn] [pid 30144] AH00111: Config variable ${APACHE_RUN_GROUP} is not defined
[Sun Aug 06 21:03:22.088712 2017] [core:warn] [pid 30144] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Sun Aug 06 21:03:22.099496 2017] [core:warn] [pid 30144] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Sun Aug 06 21:03:22.099764 2017] [core:warn] [pid 30144] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Sun Aug 06 21:03:22.099775 2017] [core:warn] [pid 30144] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
AH00526: Syntax error on line 74 of /etc/apache2/apache2.conf:
Invalid Mutex directory in argument file:${APACHE_LOCK_DIR}
"

What should you do to fix this?

Solution
Did you really mean to pipe the output of "netstat -anlp" to "apache2"?  Kindly note that Linux administrators who are prone to mistyping commands (i.e., omitting the word "grep") may also mistype the URL or IP address in the web browser.  Make sure you typed in the correct IP address in your web browser.  To find the correct IP address, run this command:

curl icanhazip.com

The result should be the IP address in the web browser.  An incorrectly typed IP address in a web browser can make a web page appear to load forever.

How Do You Solve the Apache Web Server Problem That Says “AH00558: apache2: Could not reliably determine the server’s fully qualified domain name”?

Problem scenario
Your website is not working.  When you browse to the login page, the browser loads forever.  There seems to be a problem with the web server itself.

On the back end, you run this command from the operating system's command line:
netstat -anlp | apache2

You see this as the result of this command:
"AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 123.123.123.123. Set the 'ServerName' directive globally to suppress this message httpd (pid 793) already running"

What do you do to get your web server to work properly?

Solution
Did you really mean to pipe the output of "netstat -anlp" to "apache2"?  Respectfully, Linux administrators who are prone to mistyping commands (i.e., omitting the word "grep") may also mistype the URL or IP address in the web browser.  Make sure you typed in the correct IP address in your web browser.  To find the correct IP address, run this command:

curl icanhazip.com

The result should be the IP address in the web browser.  An incorrectly typed IP address in a web browser can make a web page appear to load forever.

How Do You Avoid Using the Root User When Doing Linux Administration?

Problem scenario
You've been given root credentials.  How do you create a new user and make it a sudoer so you can install and remove OS packages?

You know that it is best to use a non-root user for Linux administration. You have been given a root user.  How do you create a non-root user with full sudoer privileges?

Solution
This example assumes that the username is "mike."  Substitute "mike" with the username of your choice.

1. Run this command:

# useradd mike

Then respond to the following prompts.

2.  Run this command:

# visudo

3.  While in this text file (in the vi environment), enter this stanza:

mike    ALL=(ALL:ALL) ALL

4.  Save the changes to the file above.  Test the user by running this command:

sudo -i -u mike

5.  To full test it, run this:

sudo apt-get install apache2

Respond "n" to the prompt!  This is just to test it.

If you have issues with Docker, see this posting.

How Do You Troubleshoot the Error “Errors were encountered while processing: docker.io E: Sub-process /usr/bin/dpkg returned an error code (1)”?

Problem scenario
You want to remove Docker from your Ubuntu server.  You tried this command:

sudo apt-get remove docker

But you see this:

"
...
 Active: failed (Result: exit-code) since Sun 2017-08-06 21:59:11 EDT; 11ms ago
     Docs: https://docs.docker.com
  Process: 21981 ExecStart=/usr/bin/dockerd -H fd:// $DOCKER_OPTS (code=exited, status=1/FAILURE)
 Main PID: 21981 (code=exited, status=1/FAILURE)
   Memory: 14.0M
   CGroup: /system.slice/docker.service

Aug 06 21:59:10 server1 systemd[1]: Starting Docker Application Container E.....
Aug 06 21:59:10 server1 dockerd[21981]: time="2017-08-06T21:59:10.966158765-..."
Aug 06 21:59:11 server1 dockerd[21981]: time="2017-08-06T21:59:11.969170376-..."
Aug 06 21:59:11 server1 systemd[1]: docker.service: Main process exited, co...RE
Aug 06 21:59:11 server1 systemd[1]: Failed to start Docker Application Cont...e.
Aug 06 21:59:11 server1 systemd[1]: docker.service: Unit entered failed state.
Aug 06 21:59:11 server1 systemd[1]: docker.service: Failed with result 'exi...'.
Hint: Some lines were ellipsized, use -l to show in full.
dpkg: error processing package docker.io (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 docker.io
E: Sub-process /usr/bin/dpkg returned an error code (1)
"

The same result happens if you try to remove docker-engine.  You want to completely remove Docker from your Linux OS.

How do you remove Docker from an Ubuntu Linux server when you get an error such as "Errors were encountered while processing: docker.io E: Sub-process /usr/bin/dpkg returned an error code (1)"?

Solution
#1  Run this command:  sudo rm -rf /usr/bin/docker /etc/docker /usr/share/docker.io

#2  Reboot the server.

#3  Run these commands:

sudo rm -rf /lib/systemd/system/docker.service
sudo rm -rf /usr/share/man/man1/docker.1.gz

How Do You Configure the LAPP Stack to Authenticate and Allow Subsequent Pages Navigable Based on the Original Authentication?

Problem scenario
You are deploying a LAPP (Linux, Apache, PostgreSQL, and PHP) stack. You want the authentication to be done via a Postgres database behind-the-scenes.  You can control the credentials from the back end this way.  You want the user to be able to click around in the application once logged in.  You want only one challenge for user login credentials.  

How do you create a PHP login page (i.e., an authentication system) so the user can navigate through a series of multiple pages if the user enters a correct username and password on the first page?

Solution
Use session variables to allow a single login with unlimited web-page traversal thereafter. 

n.b.  This solution requires the user know a database name and table name that is within that database.  These two names (one of a table and one of the table's database) are requested on the same web page that requests the username and password.  Please note that this solution is more of an example and less of a step-by-step solution. 

This solution has not been peer-reviewed for best practice security measures (e.g., it has not been validated to be impervious from SQL injection attacks).  This solution below is ideal for testing on a development server or for learning purposes behind a firewall.  Do not use this "solution" for a production environment with sensitive data.

#1  If you already have a database created with a table and username and password to log into the database and view the table, skip to step #5.

#2  Install Apache, PostgreSQL and PHP on a Linux server.  See this link for details on how to do this.

#3  Once it is set up (you have deployed the LAPP stack), create a username and password in the PostgreSQL instance.  Here is a SQL command to create the user:  CREATE ROLE contuser WITH LOGIN PASSWORD 'integ3727';

#4  The role (the user itself) must be allowed to log into the database and run a SELECT statement on a known table in the database.  Create a database that is owned by the user above (to ensure you have the access to it).  One way would be to connect to the PostgreSQL instance as the user and run this SQL command with no quotes:

"CREATE DATABASE continual;"

Create a table inside this database.  To create a table inside of a specific Postgres database, see this link.

#5  Create three files like the following in the /var/www/html/ directory.

The first file is /var/www/html/initial.php.  It should have the following content:

<html>
<body>

<form action="step1.php" method="post">
UserName: <input type="text" name="username"><br>
Password: <input type="password" name="password"><br>
DatabaseWithTable: <input type="text" name="database"><br>
TableToView: <input type="text" name="table"<br>

<input type="submit">
</form>

</body>
</html>

The second file needed for this example is /var/www/html/step1.php and should have the following content:

<?php

session_start();

   $part1 = $_POST["username"];//'jenny';
   $part2 = $_POST["password"];//'P@$$W0rd!'
   $part3 = $_POST["database"]; //cooldb
   $part4 = $_POST["table"];//'neattable';
   $comp = 'user='.$part1.' password='.$part2 ;

   $host        = "host=127.0.0.1";
   $port        = "port=5432";
   $dbname      = "dbname=" . $part3;
   $credentials = $comp;
   $_SESSION['hostvar'] = $host;
   $_SESSION['portvar'] = $port;
   $_SESSION['dbnamevar'] = $dbname;
   $_SESSION['credentialsvar'] = $credentials;

   $db = pg_connect( "$host $port $dbname $credentials"  );
   $query = 'select * from ' . $part4 . ';';
   $cc = pg_query($query);

returner($cc);

function returner($cc)
{
/* This code below was copied from http://razorsql.com/articles/postgresql_column_names_values.html  It was modified somewhat.*/
$i = 0;
echo '<html><body><table><tr>';
if ($cc) {
  }
else {
echo  '<td>' . "Failure. Were the credentials entered correctly?  Was the database name entered correctly? Does that user entered have ownership of the table and access (ownership or some access rights) to the database name?  Something went wrong. " . "Is the name of the database after dbname= here correct? " . $dbname .
'</td>';
}
while ($i < pg_num_fields($cc))
{
        $fieldName = pg_field_name($cc, $i);
        echo '<td>' . $fieldName . '</td>';
        $i = $i + 1;
}
echo '</tr>';
$i = 0;
while ($row = pg_fetch_row($cc))
{
        echo '<tr>';
        $count = count($row);
        $y = 0;
        while ($y < $count)
        {
                $c_row = current($row);
                echo '<td>' . $c_row . '</td>';
                next($row);
                $y = $y + 1;
        }
        echo '</tr>';
        $i = $i + 1;
   }
pg_free_result($cc);
echo '</table></body></html>';

}

?>

<html>
<body>

<form action="recursive.php" method="post">
TableToView: <input type="text" name="table"<br>

        <input type="submit">
</form>

</body>
</html>

The third file needed for this example is /var/www/html/recursive.php and should have the following content:

<?php

session_start();

  $part4 = $_POST["table"];//'cooltable'

  $host = $_SESSION['hostvar'];
  $port =  $_SESSION['portvar'];
  $dbname = $_SESSION['dbnamevar'];
  $credentials = $_SESSION['credentialsvar'];
  $comp = $credentials;

  $db = pg_connect( "$host $port $dbname $credentials"  );
  $query = 'select * from ' . $part4 . ';';
  $cc = pg_query($query);

returner($cc);

function returner($cc)
{
/* This code below was copied from http://razorsql.com/articles/postgresql_column_names_values.html  It was modified somewhat.*/
$i = 0;
echo '<html><body><table><tr>';
if ($cc) {
  }
else {
echo  '<td>' . "Failure. Were the credentials entered correctly?  Was the database name entered correctly? Does that user entered have ownership of the table and

access (ownership or some access rights) to the database name?  Something went wrong. " . "Is the name of the database after dbname= here correct? " . $dbname .

'</td>';
}
while ($i < pg_num_fields($cc))
{
        $fieldName = pg_field_name($cc, $i);
        echo '<td>' . $fieldName . '</td>';
        $i = $i + 1;
}
echo '</tr>';
$i = 0;
while ($row = pg_fetch_row($cc))
{
        echo '<tr>';
        $count = count($row);
        $y = 0;
        while ($y < $count)
        {
                $c_row = current($row);
                echo '<td>' . $c_row . '</td>';
                next($row);
                $y = $y + 1;
        }
        echo '</tr>';
        $i = $i + 1;
   }
pg_free_result($cc);

echo '</table></body></html>';

}

?>

<html>
<body>

<form action="recursive.php" method="post">
TableToView: <input type="text" name="table"<br>

        <input type="submit">
</form>

</body>
</html>

#6  Go to x.x.x.x/initial.php in a web browser.  Enter info into the prompts and click submit.  You'll be able to enter a table name and view its content repeatedly as many times as you like.  The viewing will be in the web browser.  The content will be the equivalent of a SQL command such as "select * from tableName;".

For further info you may want to read this link.  For using PHP session variables more securely, you may want to view this GitHub link or this separate webpage.

How Do You Troubleshoot a 500 URL Return Code with a PHP Web Page That Is Using pg_connect?

Problem scenario
You have an Apache web server with PHP code.  You are trying to integrate web pages with Postgresql.  When browsing to a web page (.php), you are getting a 500 URL return code.  Your PHP web page file is using the reserved command "pg_connect."  How do you overcome this 500 URL return code error and have the web page work?

Solution
Possible solution #1:  Reboot the web server.

Possible solution #2:  Assuming other web pages work and the 500 URL return code only applies to a PHP file that is using pg_connect, there may be a blockage from the IP address of the client with the web browser. Another problem may be that the Postgres server is unreachable.  There could be a DNS issue (the FQDN of the Postgres server is not resolving to the IP address correctly) or a key port is blocked (TCP port 5432).  Try using the IP address instead of the FQDN of the Postgres server in the PHP page (e.g., for the $host value).