How Do You Troubleshoot a Python MapReduce Job That Returns “Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1”?

Problem scenario
Your Python MR (mapreduce) job is failing. You do not know why.

You may or may not get an error like this: "Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1"

Other symptoms you may experience is the MapReduce job takes a great deal of time. It seems to hang. What should you do?

Solution

  1. Determine which map .py file you are using and which reduce .py file that you are using. For this example, let's assume you are using /tmp/mapper.py and /tmp/reducer.py. Find the input file (or one input file). Let's say you are using /tmp/foobar.txt.
  2. Run a command like this: cat /tmp/foobar.txt | python3 /tmp/mapper.py

3. Did it fail with an error? Or did it run error free? If there was an error, change the syntax of mapper.py because the problem must be there. If there was no error, run a command like this:

cat /tmp/foobar.txt | python3 /tmp/mapper.py | python3 /tmp/reducer.py

If there was an error in the above command, change the syntax in reducer.py. If there was no error to the above command, use the MapReduce job on one input file at a time. Your mapper.py and reducer.py files seem good. Is the command you are using pointing to different map and reduce Python programs compared to what you think is being used?

How Do You Install SUSE when It Fails with a Kernel Error with Little Information?

Problem scenario
Linux SUSE won't install. When the installation process is happening it stops with an abortive error that exits the pretty GUI menu sequence. You get ">>> linuxrc 7.0.15 (Kernel 5.3.18-lp152.19-default) an error occurred during the installation." What should you do to install SUSE?

Possible Solution #1
Start over. When you see this window "YaST2 The system has an active network connection. Additional software is available online. Activate online repositories now?" Choose No.

Possible Solution #2

  1. Can you use Vagrant and Oracle VirtualBox? If you can use both of these, install them. Then use this Vagrantfile (some of the comments were borrowed from mwrock Vagrantfiles):
# -*- mode: ruby -*-
# vi: set ft=ruby :

# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|
  # The most common configuration options are documented and commented below.
  # For a complete reference, please see the online documentation at
  # https://docs.vagrantup.com.

  # Every Vagrant development environment requires a box. You can search for
  # boxes at https://vagrantcloud.com/search.
  config.vm.box = "toni03/opensuse-leap15"
  config.vm.box_version = "0.1"
  config.vm.provider "virtualbox"
  config.vm.network :public_network, :bridge => "enp1s0", adapter: "1"
  config.ssh.username = 'vagrant'
  config.ssh.password = 'vagrant'
  # Disable automatic box update checking. If you disable this, then
  # boxes will only be checked for updates when the user runs
  # `vagrant box outdated`. This is not recommended.
  # config.vm.box_check_update = false

  # Create a forwarded port mapping which allows access to a specific port
  # within the machine from a port on the host machine. In the example below,
  # accessing "localhost:8080" will access port 80 on the guest machine.
  # NOTE: This will enable public access to the opened port
  # config.vm.network "forwarded_port", guest: 80, host: 8080

  # Create a forwarded port mapping which allows access to a specific port
  # within the machine from a port on the host machine and only allow access
  # via 127.0.0.1 to disable public access
  # config.vm.network "forwarded_port", guest: 80, host: 8080, host_ip: "127.0.0.1"

  # Create a private network, which allows host-only access to the machine
  # using a specific IP.
  # config.vm.network "private_network", ip: "192.168.33.10"

  # Create a public network, which generally matched to bridged network.
  # Bridged networks make the machine appear as another physical device on
  # your network.
  # config.vm.network "public_network"

  # Share an additional folder to the guest VM. The first argument is
  # the path on the host to the actual folder. The second argument is
  # the path on the guest to mount the folder. And the optional third
  # argument is a set of non-required options.
  # config.vm.synced_folder "../data", "/vagrant_data"

  # Provider-specific configuration so you can fine-tune various
  # backing providers for Vagrant. These expose provider-specific options.
  # Example for VirtualBox:
  #
   config.vm.provider "virtualbox" do |vb|
  #   # Display the VirtualBox GUI when booting the machine
  #   vb.gui = true
  #
  #   # Customize the amount of memory on the VM:
     vb.memory = "1024"
   end
  #
  # View the documentation for the provider you are using for more
  # information on available options.

end

How Do You Harden a Website Application?

Problem scenario
You want to ensure you web application is protected from buffer overflows, injection attacks and other vulnerabilities that could reveal sensitive information. How do you harden a website application and follow security recommended practices?

Possible Solution #1
Endeavor to prevent injection attacks. Minify the website application by not allowing file uploads and limit POST requests to 2 MB (page 28 of Node.js Security by Liran Tal).

Possible Solution #2
Configure a global timeout for the website at 60 seconds (page 28 of Node.js Security by Liran Tal).

Possible Solution #3
This only applies if you are using JavaScript. Configure express-limiter (despite the fact that it requires Redis) to limit HTTP CRUD requests to the /login path of the URL. Such limitations should include 20 or fewer per hour from a single incoming IP address. (Both sentences were taken from page 26 of Node.js Security by Liran Tal).

Possible Solution #4
Avoid using eval() in of PHP, Perl, or JavaScript (page 60 of Node.js Security by Liran Tal). This allows an attacker to potentially issue system-level Linux commands if he/she can exploit your web application.

Possible Solution #5
Reduce data serialization as much as possible. Use programming languages with primitive data types (as opposed to pure object-oriented programming languages). The source of these two sentences was the OWASP website. Built-in data types from a pragmatic perspective can help control manipulating resources and keep your application and server secure.

Possible Solution #6
Try to use JSON instead of XML. If you must use XML, try to disable DTD processing. The source of the above two statements is this external page.

Possible Solution #7
Cleanse, filter and validate any user input before such values are used in the composition of a SQL statement (per this website).

Possible Solution #8
Ensure that code that interacts with the web application is threadsafe because penetration testing has its limitations in what it can reveal (page 427 of The Web Application Hacker's Handbook 2nd Edition). Thread safety is the isolative quality between two threads in a multi-processor environment (per MIT's website). When one thread can access memory that another thread was using independently, there is a breach to the principle of thread safety.

Possible Solution #9
View these pages if they are applicable:
What Are Some Ways to Prevent MITM Attacks or Other Session Exploitative Attacks with a Web Page That Uses JavaScript?

What Are Some Ways to Prevent XSS Attacks with a Web Page That Uses JavaScript?

Using Python How Do You Convert Each Line to Be an Item in a List?

Problem scenario
You have a multi-line variable in Python. You want to convert each line to be its own item in a list. How do you make a multi-line string variable to become a list with the same contents?

Solution
Use the .splitlines() method to manipulate the string in this way.

Here is an example:

list_of_input="""a line here
another line there.
123 abc"""

print(type(list_of_input))
print("Now we convert…")
list_of_input = list(list_of_input.splitlines())
print(type(list_of_input))
print(list_of_input)

How Do You Troubleshoot the Cargo Problem “error[E0658]: use of unstable library feature ‘str_strip’: newly added”?

Problem scenario
You run a cargo command. You get this error:

error[E0658]: use of unstable library feature 'str_strip': newly added
--> /home/jdoe/.cargo/registry/src/github.com-1ecc6299db9ec823/ethbloom-0.10.0/src/lib.rs:69:1
|
69 | / construct_fixed_hash! {
70 | | /// Bloom hash type with 256 bytes (2048 bits) size.
71 | | pub struct Bloom(BLOOM_SIZE);
72 | | }
| |_^
|
= note: see issue #67302 https://github.com/rust-lang/rust/issues/67302 for more information
= note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

error: aborting due to previous error

For more information about this error, try rustc --explain E0658.
error: could not compile ethbloom.

What should you do?

Possible solution
Run this command: rustc --version

Is it 1.45.0 or higher?

Remove rustc. If you are using a Debian/Ubuntu distribution of Linux, run this: sudo apt -y remove rustc

Reinstall rustc with this: sudo curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

(This solution was taken from https://github.com/ogham/dog/issues/26 .)


If you want to receive free cryptocurrency by just learning more, try Coinbase. For Europeans, the platform/company Iconomi.com can allow you to buy crypto or learn more.

Using a Debian/Ubuntu Distribution of Linux, How Do You Install kubeadm, kubectl, and/or kubelet?

Problem scenario
Using a Debian/Ubuntu distribution of Linux, how do you install kubeadm, kubectl, and/or kubelet?

You want to install kubeadm, kubectl or kubelet (without retrieving the binary of the file from a website and do a regular installation). What do you do?

Solution
Here is another way to install kubectl, kubelet and kubeadm:

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt-get -y install kubeadm kubelet kubectl

(If you can download a file from the internet and want to avoid using apt commands, see these links to install kubeadm, kubelet, and kubectl.)

How Do You Find the Menu Option, Button or Feature for an Application in MacOS?

Problem scenario
You are using a familiar application on a Mac. You cannot find a basic command button or menu option. What should you do?

Solution
At the very top of your screen, outside of the application itself, there is a ribbon. There should be drop down menu options specific for the application.

We find that some problems are caused because of time pressure. You may want to read about stress-management or patience. In some cases, you may need to find a new employer (that has reasonable expectations for productivity).

To browse Mac products, see this link.

How Do You Get Both MapReduce Jobs and hadoop CLI Commands to Work Simultaneously without Alternately Changing an XML File Before Each One?

Problem scenario
Hadoop is not working correctly. You can get either a mapreduce job to work or a "hadoop" CLI command to work. But neither work unless you change an .XML file in between each operation.

Map Reduce jobs, when they are failing (and the Hadoop commands are working), there may be an error like this:

2021-01-01 22:47:42,337 INFO mapreduce.Job: Task Id : attempt_1609558624072_0001_m_000003_1, Status : FAILED
Container launch failed for container_1609558624072_0001_01_000008 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

When the Hadoop jobs are failing (and the Map Reduce jobs are working), there may be an error like this:

Exception in thread "main" java.lang.NullPointerException
at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:784)
at org.apache.hadoop.mapreduce.tools.CLI.displayJobList(CLI.java:769)
at org.apache.hadoop.mapreduce.tools.CLI.listAllJobs(CLI.java:697)
at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:428)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1277)

You make changes to a yarn-site.xml file and try one of the jobs. They work alternately. You see that the changes take effect when you run a map reduce job and separately a hadoop command to view the jobs. Either the map reduce job will work or the hadoop command will work. Both will not work unless you make changes to the yarn-site.xml. You have to change the yarn-site.xml file each time before issuing one of the commands. You know you don't need to restart anything because you get different behavior as soon as you make changes to the yarn-site.xml. How do you get both to work simultaneously?

Solution
Restart DFS and YARN servcices. Use stop-dfs.sh and stop-yarn.sh scripts. Then use start-dfs.sh and start-yarn.sh scripts.

You must restart the DFS and YARN services. Yes, you see different behavior immediately after changing the yarn-site.xml or mapred-site.xml files. However, for the changes to take full effect, you must restart the services. Then both the mapreduce jobs and the hadoop CLI commands will work.

How Do You Solve “Could not find foobar in any of the sources” after a bundle Command?

Problem scenario
You run a bundle command (e.g., bundle exec foobar create). It fails with an error such as "Could not find foobar in any of the sources".

What should you do?

Solution
Remove the relevant Gemfile.lock file (e.g., in the directory you are running bundle) if you are allowed to. Are you using a variety of bundle commands? Each one could create a Gemfile.lock file so be aware of this. To learn more about the basics of bundle, see this.

If you cannot delete the Gemfile.lock file, try this external posting.

What Does the “^” Operator in Python Do?

Problem scenario
In Python you try operations like these two:

5 ^ 6
9 ^ 10

Both return "3". Why is this?

Solution
The exclusive or (aka XOR) symbol "^" "copies the bit if it is set in one operand but not both." (Taken from https://www.tutorialspoint.com/python/python_basic_operators.htm)

Here are some integers with their binary representation one space away:

3 11

5 101
6 110

9 1001
10 1010

5 is represented as 101. 6 is represented as 110. The 1 is set in the middle and right-most position of the binary values. Thus it returns a 3 which is associated with "11" in binary notation.

7 ^ 8 returns 15. 15 in binary notation is 1111.

7 111
8 1000

If we remember that ^ "copies the bit if it is set in one operand but not both." (taken from https://www.tutorialspoint.com/python/python_basic_operators.htm), we would expect it to return 1111. The integer equivalent of 1111 is "15".

To view more binary equivalents, see this https://www.convertbinary.com/numbers/.