How Do You Troubleshoot “java.io.IOException: Stream closed at java.base/java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:442)”?

Problem scenario
You are trying to run a Hadoop job. You get this error:
“java.io.IOException: Stream closed at java.base/java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:442)”

What should you do?

Solution
Is the “python” command recognized as such? You may need to install Python or link the python3 binary to be in a typical location where env variables would look for it (e.g., /usr/bin/python).

Here are commands that could help you:

whereis python3
sudo ln -s python3 /bin/python

If you need help installing Python,

How Do You Iterate Through Two Lists of Unequal Length in Python?

Problem scenario
You want to iterate through two lists and perform some operation. The lists are not equal length. You want to iterate through them in some type of step-by-step fashion despite their lack of equality. What should you do?

Solution
Use this as an example:

list_a = [1, 2, 3]
list_b = [”dog”, “cat”, “rat”, “chicken”]

i = 0
j = 0

while (i < …

How Do You Write Python Code to Test if a Dictionary Exists?

Problem scenario
You want to test if a dictionary exists or not. You know its name if/when it exists. What do you do?

Solution

# Suggested usage: 1) run it is as it is.
# 2) uncomment out the dictionary_name1 definition stanza.
# Then run this program again.

#dictionary_name1 = “good_dict”

if ‘dictionary_name1’ in locals():
print(“IT IS IN LOCALS”)
else:
print(“IT IS NOT IN LOCALS”)

if ‘dictionary_name1’ in globals():
print(“IT IS IN GLOBALS”)
else:
print(“IT IS NOT IN GLOBALS”) …

How Do You Troubleshoot a Python MapReduce Job That Returns “Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1”?

Problem scenario
Your Python MR (mapreduce) job is failing. You do not know why.

You may or may not get an error like this: “Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1”

Other symptoms you may experience is the MapReduce job takes a great deal of time. It seems to hang. What should you do?

Solution

  1. Determine which map .py file you are using and which reduce .py file that you are using.

Using Python How Do You Convert Each Line to Be an Item in a List?

Problem scenario
You have a multi-line variable in Python. You want to convert each line to be its own item in a list. How do you make a multi-line string variable to become a list with the same contents?

Solution
Use the .splitlines() method to manipulate the string in this way.

Here is an example:

list_of_input=”””a line here
another line there. …

What Does the “^” Operator in Python Do?

Problem scenario
In Python you try operations like these two:

5 ^ 6
9 ^ 10

Both return “3”. Why is this?

Solution
The exclusive or (aka XOR) symbol “^” “copies the bit if it is set in one operand but not both.” (Taken from https://www.tutorialspoint.com/python/python_basic_operators.htm)

Here are some integers with their binary representation one space away:

3 11

5 101
6 110

9 1001
10 1010

5 is represented as 101.

How Do You Troubleshoot the Python Problem “UnicodeEncodeError: ‘ascii’ codec can’t encode character”?

Problem scenario
You are trying to print out a variable in Python. But you get an error like this: “UnicodeEncodeError: ‘ascii’ codec can’t encode character ‘\xa0’ in position 70567: ordinal not in range(128)”

What should you do?

Solution
Root cause: A variable needs to be encoded as UTF instead of the default ASCII encoding.

Procedures
Assuming that var1 is a variable holding the results of a requests.get(url),

How Do You Get Python to Search a Web Page for a Specific Pattern?

Problem scenario
How do you write a Python 3 program to parse the text of a web page and determine if a string is on the web page or not?

Solution
Change the “yoururl” variable assignment and the “searchterm” assignment below to what you desire. Then run the program below:

import re
import requests
yoururl = ‘https://www.continualintegration.com/’
searchterm = “learned”
r = requests.get(yoururl)
found = re.search(searchterm, …

How Do You Write a Python Program to Test if Integers Are in Order?

Problem scenario
You need to write a Python program to see if a list of unique integers is in order. You can copy the list and sort it to compare the ideal state to the list provided. The program must print out a relevant message if a swap of two numbers can place those two numbers in the correct order. The program must print out if a given number is already in order.

What Does the “-> str” Syntax Signify in Python Functions?

Problem scenario
You see Python code with this “-str” syntax. What does it do?

Answer
It is purely for readability. To help your coworkers we recommend using the “-str” syntax if a function returns a string. Otherwise the way to find out is to read the code or temporarily change the code and use type(variable_returned) to show the data type of that which is/was returned.