How Do You Delete a Directory on Linux Server That Is Not Easily Deleted?

Problem scenario
You are trying to remove a directory.  When you run a "sudo rm -rf" command you get this error:

rm: cannot remove '/path/to/': Device or resource busy

When you run a "sudo umount /path/to/" command you get this error:

"umount.nfs: device is busy"

What should you do to eliminate the directory?

Solution
1.  Is the "/path/to/" in the "sudo umount" command identical to the path in the error message?  If so, skip to step #2.  If not, remember that if you get an error like this "'/path/to/subdirectoryname': Device or resource busy", then you need to try the umount command *with* the subdirectory.  The root cause of this is you are trying to umount a parent directory of a subdirectory that is mounted.

2.  If the directory is not unmounting and you are using the "sudo umount" command, run these two commands:  

sudo umount -f -l /path/to

# The -f flag is for force.  The -l flag will clean up references to the file system.  Using these two flags at the same time is recommended by various Linux experts when a directory will not unmount as it normally should.

sudo rm -rf /path/to

How Do You Use a Jenkins Pipeline with a Scripted Syntax?

Updated and re-tested on 9/24/19
Problem scenario
You read about what Jenkinsfiles are.  Coveros.com defines them as 'Jenkinsfiles, using a domain specific language based on the Groovy programming language, are persistent files that model delivery pipelines “as code", containing the complete set of encoded steps (steps, nodes, and stages) necessary to define the entire application life-cycle.'

You want to use a Scripted Pipeline in Jenkins (via a Jenkinsfile).  What do you do to use the Scripted DSL syntax?

Solution
Prerequisites
i.  You must have Jenkins installed.  See the posting for your type of Linux below for assistance installing Jenkins:

CentOS/RHEL/Fedora
Debian/Ubuntu

Pipeline jobs require more memory than typical Jenkins jobs.  If you need to resize your server (i.e., give it more RAM), click here if you have a server in AWS, and click here if you have a server in GCP.

If you are not using a public cloud, you may want to add physical RAM.  You may want to create virtual memory to help with performance.  To do that, see this posting.

ii.  Install the Pipeline plugin.  Log into the web UI of Jenkins as an administrator.  Go to Manage Jenkins -> Manage Plugins -> Available.  Search for "Build Pipeline."  Check the box and click "Install without restart".  Next, click the option for "Restart Jenkins..." at the bottom of the screen.

Procedures
1.  Log into the web UI for Jenkins as an administrator.
2.a.  If you see "New Item" do 2.b, otherwise do 2.c.
2.b.  Click on "New Item".
2.c.  Click on "Dashboard" if you do not see "New Item."  Then click "New Item".
3.  Enter a name in the field and the top, then click on "Pipeline".  Then click "OK".
4.a.  Scroll down to the "Pipeline" section.  
4.b.  Enter this text (which is mostly taken from this external site):

node {
    stage('Example') {
        if (env.BRANCH_NAME == 'master') {
            echo 'I only execute on the master branch'
        } else {
            echo 'I execute elsewhere'
        }
    }
} //Taken from https://jenkins.io/doc/book/pipeline/syntax/#flow-control
node {
    stage('Example1') {
        if (env.BRANCH_NAME == 'slave') {
            echo 'I only execute on the slave branch'
        } else {
            echo 'I do not execute on the slave branch.'
        }
    }
}

5.  (You can uncheck the "Use Groovy Sandbox" or leave it checked.*)  Click "Save".

6.a.  You can test it by clicking on "Build Now."

6.b.  You are done.  The Jenkinsfile should be checked into a source control repository.

6.c.  Here is optional reading if you want to learn more.

*  This is information on using the Groovy sandbox:

"Jenkins limits the execution of any Groovy script by providing a sandbox.  The option "Use Groovy Sandbox", shown below, is available in the Pipeline tab, and it allows the scripts to be run by any user without requiring administrator privileges.  In this case, the script is run only by using the internal accessible APIs (that allow you to develop your script by using Groovy).
...
When unchecked, if the script has operations that require approval, an administrator will have to provide them.  This method is known as "Script approval".  By default, all Jenkins pipelines run in a Groovy sandbox.  If the option is checked and unauthorized operations are used, the script will fail when run.  Both the whitelist and the blacklist of functions can be checked at Script Security's built-in list. Please refer to In-process Script Approval for more information on this topic."  The above quoted paragraphs were taken from Dzone.com.

How Do You Indent a Block of Lines in vi The Same Amount?

Problem scenario
You want several lines to be indented by four more spaces automatically.  You do not want to do it manually line-by-line.  How do you indent multiple lines uniformly with the same number of spaces (regardless if they are indented similarly or dissimilarly)?

Solution
1.  Open the file in vi.
2.  Enter this command and press enter afterwards (but replace "4" with the number of spaces you want in your indentation):    :set shiftwidth=4
3.  Go to the highest line that you want indented.
4.  Press the "v" key
5.  Highlight the lines you want indented by pressing the down key.  Once all the lines are highlighted, go to the next step.
6.  Press the ">" key.  (On most keyboards it is pressed by holding shift and pressing the "." (period) key.  (To unindent, reverse indent, or shift to the the left, use the "<" key.)

How Do You Troubleshoot the Ansible Error “this task includes an undefined variable”?

Problem scenario
You are running an Ansible playbook, but you get the error "this task includes an undefined variable."  Every variable seems to be defined.  What is wrong?

Possible solution #1
Do one of the variables rely on an index of server names?  If the playbook orchestrates something that responds to Ansible values on other servers and one server is not reachable, you could get this error.  In the "Gathering Facts" stage, was each server connected to?  If one server was temporarily down, an indexing of server names and facts thereof may have not happened.  Thus you could have received an error about an undefined variable.

Possible solution #2
Use the "gather_facts: no" stanza under the module throwing the problem.  Variables derived from facts can be unnecessary.  Configuring the playbook to not gather facts is a workaround.  But this option reduces the number of potential problems that can keep the playbook from running.

How Do You Get Vagrant Working on a Debian/Ubuntu Server in AWS?

Problem scenario
You are using Ubuntu 16.x in AWS.  You want to get Vagrant working to create VMs in AWS.  What do you do?

Solution
1.  Install Vagrant.  If you need assistance see this link.

2.  Install gem and various other dependencies.  To do this run these three commands:  

sudo apt-get -y update
sudo apt-get -y install ruby-dev libffi-dev build-essential dh-autoreconf
sudo gem install ffi -v 1.9.14

3.  Install the vagrant-aws plugin:  vagrant plugin install vagrant-aws

4.  Retrieve a Vagrant box image with this command:

vagrant box add dummy https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box

5.  Create a Vagrant file with these three commands:

mkdir contint-demo
cd contint-demo
vagrant init

6.  Go to the ~/.ssh directory for the user that will run Vagrant.

7.  Create a file with the same name as the AWS key pair you will use with Vagrant.  This should be an existing key pair with AWS used to connect to your VMs as specified in the last step when you manually launch a new EC2 instance.  Let's say this key pair is called "foobar".  You should create a file named foobar in the ~/.ssh/ directory.  The content of foobar should be these three lines (but replace YourAWSAccessKeyID and YourAWSSecretAccessKey accordingly):

[default]
AWSAccessKeyId=YourAWSAccessKeyID
AWSSecretKey=YourAWSSecretAccessKey

To find the AWS Access Key ID and AWS Secret Access Key, in the AWS console, click on your name in the upper right hand corner. Click on "My Security Credentials."  Click "Create New Access Key." Then click "Show Access Key."

8.  The Vagrantfile must be replaced. Here is a template with six values to replace.

require 'vagrant-aws'
Vagrant.configure('2') do |config|
    config.vm.box = 'dummy'
    config.vm.provider 'aws' do |aws, override|
    aws.access_key_id = "value1"
    aws.secret_access_key = "value2"
    aws.keypair_name = 'value3'
    aws.instance_type = "t2.nano"
    aws.region = 'value4'
    aws.ami = 'value5'
    aws.security_groups = ['value6']
    override.ssh.username = 'ubuntu'
    override.ssh.private_key_path = '~/.ssh/value3'
  end
end

Change value1, value2, value3, value4, value5 and value6.  You may also want to change "t2.nano" and "ubuntu".

value1 and value2 would be determined in step 7 above.

value3 is the AWS Key Pair name.  For the ASS key pair name, this is a reference to the key pair to log into the instance OS with.  The last step of creating a VM in AWS gives you the opportunity to select the key pair.  If you want to test your machine's ability to connect with the Access Key ID and Secret Access Key, you may want to install the AWS CLI (sudo apt install awscli).  You can then run "aws configure".  This can test your network access, ensure that the time on your machine is configured properly etc.  This way if there is a problem with Vagrant connecting, you know the fundamentals are in place.  One advantage of testing like this gives you the ability to see the key pairs (with "aws ec2 describe-key-pairs").

value4 is the region name (e.g., us-west-1).  

value5 is the AMI ID.  
For the AMI (or image) ID, you could log into AWS and start to launch a new instance.  The AMI ID is visible in the first step when selecting an image.  

For value6, it should be fairly obvious what to change the Security Group name to.  With the AWS Console you can determine the Security Group name.  

9.  Run this command:  vagrant up --provider=aws --debug

10.  This will create VMs in the region that you provided for value4 in step #8.  You will want to make sure these are destroyed after you run the command to not incur an expense.  You can destroy them from the Vagrant host with this command:  vagrant destroy default

How Do You Clone a Git Repository when You Are Being Prompted for a Password and Do Not Know What It Is?

Problem scenario
You have a Git repository on one server.  You want to clone it to another server.  When you try to clone it, you are prompted for a password.  What do you do?

Solution
On the destination server, run this (where x.x.x.x is the IP address of the server with the Git repo):

git clone git@x.x.x.x:root/goodrepo.git

If you are prompted for a password, enter the password for the user git.  You could use a different user in your subsequent "git clone" command.  (Such a user must have permissions to read the .git file.  The user is specified immediately before the "@" symbol above.)

Alternatively if you can change the password for the git user, go to the back-end of the server with the git repository.  Run this:  sudo passwd git

How Do You Get awk to Use a Colon or a Semicolon as a Field Separator?

Problem scenario
You run a command such as this, with the goal of printing the number of occurrences of "foo":

grep -icR foo * | awk '{fs=":"; print $2}'

It does not do what you want it to do.  You want the field separator to be a colon.  It is not working.  How do you print the element after a ":" (colon) with an awk utility?

Solution
Technically awk is a programming language; but it is also a utility.  Use this command instead:

grep -icr foo *  | awk --field-separator=":" '{print $2}'

The syntax above uses awk in a different way to recognize the ":" (colon) as a field separator.  With the "--field-separator" syntax above, replacing the colon with a semicolon will allow you use to use a semicolon as a field separator too.

How Do You Set up AWS CloudFront?

Problem scenario
You want to test out a CDN in AWS.  What do you initially deploy CloudFront?

Solution

Prerequisite
This assumes that you have an S3 bucket with read permissions.  To do this, log into the Amazon S3 console and create a bucket.  Upload files and configure the permissions to grant read permissions to anyone.  

Procedures
AWS CloudFront has "Distributions."  You want to create a CloudFront Distribution.

1.  Log into the CloudFront console with this link.

2.  Click "Create Distribution"

3.  For this example, click "Get Started" under "Web" (as the delivery method for your content).

4.  Fill out the fields.  Not every field is mandatory.  If you are not sure click the "i" icon for more information on what to put in each field.  If you provide a basic S3 bucket of files (and not a website link hosted by an S3 bucket), CloudFront will take 15 minutes to deploy.  A website file (e.g., an AngularJS or HTML file) will make things faster.  But an S3 bucket is acceptable to use.

5.  Click "Create Distribution" in the lower right hand corner.

6.  You will see the "CloudFront Distributions" page with the Status "In Progress."  Eventually the Status will change to "Deployed."  If you configured CloudFront to use a regular S3 bucket by itself (and not a specific web page file), be prepared to wait 15 minutes.

7.  When the status is "Deployed", you can test your links.  Copy the domain name, e.g., d3lnudekzjqklzuwx.cloudfront.net and construct a URL to a file of your choice.  If you uploaded a file named foobar.txt to your S3 bucket (that is now part of the CloudFront CDN), construct the URL as follows:

http://d3lnudekzjqklzuwx.cloudfront.net/foobar.txt

8.  Open a web browser.  Go to the URL described in step #7.

How Do You Use the MongoDB Command Line?

Problem scenario
You want to enter the MongoDB shell to enter commands interactively.  What do you do?

Solution
1.  Run this command:  /usr/bin/mongo

If the command is not found, do two things:
  a)  Run this: sudo find / -name mongo | grep bin
  b)  Run the mongo file found above (as an executable).

2.  From the prompt ">" enter a command such as this:

db

How Do You Troubleshoot the Logstash (or Elastic Stack) Error “logging.log4j.core.appender.RollingFileAppender”?

Problem scenario
You try to start Logstash but you get this error: "main ERROR Could not create plugin of type class org.apache.logging.log4j.core.appender.RollingFileAppender for element RollingFile: "

What should you do?

Solution
Do one of the two options below.

#1  Solution (rather simple, for a one-time fix)
Do not start the process with the root or some other regular user.  Start the Logstash service with a user dedicated for Elastic Stack.  There should be a user that is dedicated for Elastic Search and Logstash.  Become that user:

sudo su -
su elasticuser

(Replace "elasticuser" with the user dedicated for Elastic Stack.)

Now start the logstash service.

#2  Solution (more complex, for a permanent solution)
1.  You may want to create a logstash.service file in /etc/systemd/system/.  Here is a template of the file; replace esu with the Elastic Stack user; replace esg with the Elastic Stack group; replace "/opt/logstash/config/" with the path to the startup.options file; and replace other options as needed:

[Unit]
Description=logstash

[Service]
Type=simple
User=esu
Group=esg

EnvironmentFile=/opt/logstash/config/startup.options
ExecStart=/opt/logstash "-f" "/opt/logstash/config/logstash-simple.conf"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

2.  You will need to run this command after you create it (or reboot the server): sudo systemctl daemon-reload

3.  Now you can run this command: sudo systemctl start logstash

Starting Logstash this way will always start with the user context of the dedicated ElasticStack user.