How Do You Use scp When You Get A “Permission Denied” Error?

Problem scenario
You want to copy files between RedHat Linux servers with adherence to security best practices.  You try to use scp but you get "Permission denied."  What do you do to copy files in a secure way?

Solution
1.  Verify you can SSH from one server to the other.  For directions for setting up SSH in a cross-cloud environment, you may want to see this posting.  If firewall rules prevent this, then scp will not work either.


2.  Ensure that you can create files in the destination directory.  You may want to use scp to try to copy a file to the /tmp/ directory first then try directory destinations with stricter write permissions.  Local server directory permissions could be the problem. You are done.


3.  This is purely optional and informational. For further information about scp, see this link. If you want to read more about technical security topics, see the many pages of this internal posting. If you are interested in buying a book on I.T. security, see this posting.

How Do You Troublshoot the Error “snappymodule.cc:31:22: fatal error: snappy-c.h: No such file or directory” when Trying to Install Apache Parquet?

Problem scenario
When trying to install Apache Parquet on Ubuntu, you get this error:

Running python-snappy-0.5.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-SIHL_T/python-snappy-0.5.1/egg-dist-tmp-O8UnkU
WARNING: '.' not a valid package name; please use only .-separated package names in setup.py
package init file '__init__.py' not found (or not a regular file)
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
snappymodule.cc:31:22: fatal error: snappy-c.h: No such file or directory
compilation terminated.
error: Setup script exited with error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
ubuntu@ubuntuf1:/opt/parquet-1.2$ find . -name snappymodule.cc
ubuntu@ubuntuf1:/opt/parquet-1.2$ sudo apt-get install python-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
python-dev is already the newest version (2.7.11-1).
0 upgraded, 0 newly installed, 0 to remove and 11 not upgraded.

What should you do?

Solution
Run this command to install libsnappy-dev (a dependency for installing Apache Parquet):
sudo apt-get -y install libsnappy-dev

How Do You Prepopulate Text Fields in a Web Page with Values from a SQL Database?

Problem scenario
You have a web page from an .php file that includes HTML code.  There are fields visible in the web page to the user.  These fields accept text. You want the fields to not be blank; you want the fields to have suggested values of text in them. You want the data in the fields to be taken from the result of a SQL statement.  How do you prepopulate text fields in a LAPP web page with values from SQL statements?

Solution
This assumes have you set up the LAPP stack.  If you do not know how to implement front-end authentication for your LAPP instance, you may want to see one of two blog posts.  This solution assumes you have configured authentication variables that you see in the pg_connect() statement.

Here is an example PHP web page (.php file) that will prepopulate text fields that the user may modify from his/her web browser:

<?php

   $db = pg_connect( "$host $port $dbname $credentials"  );

//The above line assumes the $host, $port, $dbname, and $credentials variables were properly assigned.

//With PHP code compose SQL statements like this:

   $query = 'select col1 from ' . $tn . ' where continual_col = ' . $id . ';';

// This assumes that col1 exists in the table defined in the variable $tn.
// This assumes that $id is the value of exactly one row in the table defined with variable $tn.
// This query will retrieve the value of one field in a table in the database.

//Now invoke Postgres commands via PHP to process the results of the query:

   $selectval = pg_query($query);
   $row1 = pg_fetch_row($selectval);
   $finalval = current($row1);

?>

<html>
<body>

<form action="nextpage.php" method="post">

good_field: <input type="text" name="suggestedval" value="<?php echo $finalval; ?>">

<input type="submit">
</form>

</body>
</html>

How Do You Configure Maven to Use an Apache Parquet Plugin?

Problem scenario
You want to use Maven's Apache Parquet plugin with Hadoop.  How do you use these Apache technologies together?

Solution
1.  Install HDFS.  See this link if you are using Ubuntu.  See this link if you are using a RedHat distribution of Linux. If you have more than one server and want a multi-node cluster of Hadoop, see this link for directions on how to deploy and configure it.

2.  Install maven (e.g., "sudo apt-get -y install maven" or see this post if you are using a RedHat derivative)

3.  Run this command (but feel free to substitute "contint" and "cont-int" with the names of your choice):
mvn archetype:generate -DgroupId=com.contint.app -DartifactId=cont-int -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false

4.  Run this command (you'll need the location for steps #5 and #6): sudo find / -name pom.xml

5.  Modify the file found above.  In pom.xml under the lowest </dependency> tab, add these five lines (to create additional dependency element):
   <dependency>
      <groupId>org.apache.parquet</groupId>
      <artifactId>parquet-avro</artifactId>
      <version>1.9.0</version> <!-- or latest version -->
   </dependency>

6.  Assuming you are in the directory with the pom.xml file modified above, run mvn package (with no quotes). If you were not in the directory with the pom.xml file, change directories, then run mvn package.

How Do You Get to See Settings and The Help Menu of Atlassian HipChat?

Problem Scenario
You are logged into the web UI of HipChat.  You expect to hover over either your name or the question mark icon in the upper right hand corner and see drop down options.  But this is not what happens.  When you single-click on your name or the question mark icon, you briefly get a glimpse of menu options.  You want to click on the menu options.  How do you do this?

Solution
Double-click your name or the question mark.  Now the drop down menu should remain long enough for you to click on an option.

Roughly 1.5% of Workstations Run Linux and Linux vs. Windows at the Workplace

Various companies such as Google use Linux desktops instead of Windows (according to ZDNet). According to TechReublic other big names that use it are NASA, the French military police, CERN (which is also known as the European Organization for Nuclear Research) and the U.S. Department of Defense. While an estimated 1.5% of desktops use Linux, it is a good choice for staying safe from viruses and spyware.  

Another case for using Linux is that employees can benefit from the experience with an increase in earning potential.  See this article for more information.

To see how Macs play into the desktop market and compete with Linux, here is an article from 2012.

Development in a Windows environment is not like Linux.  As a recommended practice in Linux, running things as administrator should be done on an ad hoc basis.  Some people who develop in Windows recommend that Visual Studios always be run as administrator.  To learn more about this, see this link.

Should more people use Linux for everyday tasks and/or software development?  What is your opinion?

How Do You Resume a DVD Movie Where You Stopped Watching It?

Problem scenario
In Windows you sometimes watch DVDs but have to stop the movie mid-way through.  You cannot always watch an entire movie in one setting from start-to-finish.  You stop the DVD by closing Windows Media Player while the movie was still playing or paused.  Later after you have shut off your computer, you want to resume the DVD at the exact hour, minute and second where you were last watching it.  How do start the movie at the point where you left off?

Solution
Go to My Computer.  Find the drive letter for the DVD player.  Right click it and choose "Open AutoPlay..."

How Do You Learn More About “Low Latency Programming”?

Problem scenario
You want to learn more about low latency programming for working at a hedge fund company.  Where do you begin?

Solution
An Amazon search for "low latency programming" brings up books that could be very useful for other high-paying I.T. jobs in big data such as an Apache Thrift book or a book for Building a Columnar Database on RAMCloud.  These technologies are relevant to the Hadoop ecosystem.  If you want to learn more about "low latency programming" you may want to read one of the following such as High-Performance Computing Using FPGAs, High-Speed Decoders for Polar Codes, or Low Latency, High Performance Data Stream Processing: Systems Architecture, Algorithms and Implementation.  You may also want to learn more about distributed computing and Parquet.

If you want to learn more about RAM and how to deal with it as a programmer for high-performance computing or optimization, you may want to read this paper by a professional at Red Hat, Inc.

You may also want to view these links:

How Do You Turn off OS Firewalls in an AWS Instance of Linux SUSE?

Problem scenario
You are running Linux SUSE in AWS.  You want to have no firewalls running on the OS.  How can you be sure that there is no OS-level firewall running in Linux SUSE?

Solution
#1  Become root with this command: sudo su -
#2  Enter this command:  yast firewall
#3  You will see either a screen that warns you about another firewall running or you'll see the YaST2 screen for a firewall.  If the situation is the former, then you know another firewall is running.  If the latter is the case in this screen you will see "Stop Firewall Now" in aqua.  Other other letters will be in light gray; this means that no firewall is running on your Linux OS.  If you can put a cursor on "Stop Firewall Now", then the Yast Firewall is running. 

If you want to turn off a firewall from your AWS instance, but the Yast Firewall menu above did not help you, follow these steps.  First be sure you know what Security Groups that your server is in.  An AWS Security Group can simulate an OS firewall for many purposes.  Secondly, do these things:

Run this command:

$ sudo iptables-save > /root/firewall.rules

Then run this script that was taken from a cyberciti.biz posting to disable the Linux OS firewall:

#!/bin/bash
# reset.fw - Reset firewall
# set x to 0 - No reset
# set x to 1 - Reset firewall
# ---------------------------------------------------------------------------------------------------------------
# Added support for IPV6 Firewall
# ---------------------------------------------------------------------------------------------------------------
# Written by Vivek Gite <vivek@nixcraft.com>
# Modified by continualintegration.com.  We found it worked better with Linux SUSE in AWS with the modifications.
# ---------------------------------------------------------------------------------------------------------------
# You can copy / paste / redistribute this script under GPL version 2.0 or above
# =============================================================
x=1

# set to true if it is CentOS / RHEL / Fedora box
RHEL=false

### continualintegration.com edited a few lines here  ###
IPT=/usr/sbin/iptables
IPT6=/usr/sbin/ip6tables

if [ "$x" == "1" ];
then
    if [ "$RHEL" == "true" ];
    then
          # reset firewall using redhat script
        /etc/init.d/iptables stop
        /etc/init.d/ip6tables stop
    else
        # for all other Linux distro use following rules to reset firewall
        ### reset ipv4 iptales ###
        $IPT -F
        $IPT -X
        $IPT -Z
        for table in $(</proc/net/ip_tables_names)
        do
            $IPT -t $table -F
            $IPT -t $table -X
            $IPT -t $table -Z
        done
        $IPT -P INPUT ACCEPT
        $IPT -P OUTPUT ACCEPT
        $IPT -P FORWARD ACCEPT
        ### reset ipv6 iptales ###
        $IPT6 -F
        $IPT6 -X
        $IPT6 -Z
        for table in $(</proc/net/ip6_tables_names)
        do
            $IPT6 -t $table -F
            $IPT6 -t $table -X
            $IPT6 -t $table -Z
        done
        $IPT6 -P INPUT ACCEPT
        $IPT6 -P OUTPUT ACCEPT
        $IPT6 -P FORWARD ACCEPT
    fi
else
        :
fi
# End of script

If you changed your mind, you can revert the firewall changes by running this command without the leading "#" symbol:
# sudo iptables-restore < /root/firewall.rules

How Do You Optimize SQL to Get Away from Disk Contention, to Reduce the I/O Activity?

Problem scenario
Let's say you want to rearrange the columns a of SQL table because a special order would make subsequent SQL commands easier.  Or you just want to avoid disk contention on the underlying storage device.  How do you optimize SQL to do less input/output activity on the disk while being able to rearrange the order of columns?

Solution
There are other ways such as avoiding "select *", changing the frequency that indexes are updated, normalizing tables or partitioning tables.  For this answer we recommend using a view.  A view can accomplish the rearrangement of columns and put the results of a SELECT statement into memory as opposed to leaving a table on the disk.  This does take more RAM.  Here is an example of a view based on a table named "goodTable":

create view continualView AS select a.column3, a.column1, a.column2 * from goodTable a

Now continualView can be queried like goodTable.  You would want to avoid views if your environment was memory constrained.  But if there was ample memory and there I/O activity on the storage device (e.g., NAS, SAN, or hard disk of the server) was a bottleneck for the performance of your SQL database, then a view may help.

At the OS level if you are running Linux, you may want to enable hugepages.