A Technical I.T./DevOps Blog
Problem scenario
You want to see what servers are in your HDFS cluster (e.g., the DataNodes and the NameNodes). What command can display the FQDNs of such servers?
Solution
Run this command from the NameNode:
hdfs dfsadmin -report | grep Hostname
…
Continue reading “How Do You List the Hostnames of the Nodes in Your HDFS Cluster?”
We worked very hard to write an original twelve-question Big Data Quiz. Please do not expect another posting (like we normally post) for several days. Please be sure to check out the Big Data Quiz!
…
Continue reading “Announcement: Big Data Quiz Now Available!”
Problem scenario
One or more of the following is happening:
1) There are 0 DataNodes in your Hadoop cluster according to an error message
2) There is 0 B configured as capacity (as shown from a “hdfs dfsadmin -report” command).
3) There is one fewer DataNode in your Hadoop cluster than you expect.
4) You run “hdfs dfsadmin -report | grep Hostname” and do not see a node that has its DataNode service (as seen with the
…
Continue reading “How Do You Troubleshoot an Empty Multi-node Hadoop Cluster?”
Problem scenario
You have a multi-node cluster of Hadoop. You want to add a new data node. What do you do?
Solution
1. a) Log into the server that will be the new DataNode. Do these things until you get to step 2.
b) Install Hadoop on the new DataNode. If you do not know how, see this posting.
c)
…
Continue reading “How Do You Add a New Node to a Hadoop Cluster?”
Updated on 1/6/21
Problem scenario
You want to install open source Hadoop. You may want a single-node or multi-node deployment with CentOS/RedHat/Fedora, Debian/Ubuntu, and/or SUSE Linux distributions. You want to have most of it scripted and have the same script work on any variety of Linux. How do you install Hadoop quickly with a script that works on almost any type of Linux?
Solution
1.
…
Continue reading “How Do You Install Hadoop with a Script for Any Type of Linux Server?”
Question
What is Apache Parquet?
Answer
Apache Parquet is columnar data representation/manipulation tool for a Hadoop ecosystem. Data in a given column is largely uniform (e.g., a long string of characters, a single character, or an integer) in that it repeats a specific type and format of data as opposed to two cells in the same row (which may be very dissimilar types of data).
…
Problem scenario
You want to install Apache Parquet on the Hadoop namenode. What do you do?
Solution
Prerequisite
This assumes that you have installed Hadoop. For directions, see this posting.
Procedure
Run these commands:
sudo su –
apt-get -y install pip
pip install thriftpy
pip install snappy
exit
sudo apt-get -y install libsnappy-dev thrift-compiler
curl https://pypi.python.org/packages/74/b5/bc459aab0566fc3cf3397467922c37411ab6e3361bab9e0ca165e1089ce8/parquet-1.2.tar.gz#md5=05aacec0620ac63ecd7dd77bf7fb9fee >
…
Problem scenario
You want to learn more about low latency programming for working at a hedge fund company. Where do you begin?
Solution
An Amazon search for “low latency programming” brings up books that could be very useful for other high-paying I.T. jobs in big data such as an Apache Thrift book or a book for Building a Columnar Database on RAMCloud.
…
Continue reading “How Do You Learn More About “Low Latency Programming”?”
Problem scenario
The Hadoop NameNode service won’t start. You run start-dfs.sh and it starts the SecondaryNameNode and the DataNode. It won’t start the NameNode.
Solution
Assumption: This solution only works if you are running the start-dfs.sh as a sudo user or the root user itself.
Procedures
Verify you can ssh with root to the localhost. Run this command twice without exiting from the first session:
…
Continue reading “How Do You Troubleshoot the Hadoop NameNode Service Not Running Properly?”
Problem scenario
You are trying to start HDFS. But you get this error: “WARN conf.Configuration: bad conf file: element not <property>”
What should you do?
Solution
Look at mapred-site.xml, hdfs-site.xml, and core-site.xml files. Verify you have a <property> tag in each file.
cat /path/to/mapred-site.xml | grep -i property
cat /path/to/hdfs-site.xml | grep -i property
cat /path/to/core-site.xml | grep -i property
If there is no <property>
…