For Storing Data for with Few Transactions and Little Loading with a Need for Compression, Would a Column-Oriented Database Be Better than a Row-Oriented Database?

Problem scenario
You are trying to select a database that is right for your needs. You will not be doing many transactions. The database does not need to support regular ETL or OLTP. You are going to have a large database, and you want it to compress to the smallest size possible. In some cases a column-oriented database better than a row-oriented database. What type of database should you use for this situation?

Solution
A columnar database. Here are the three reasons:

  1. A single disk-read to retrieve a row is achievable with smaller tables. With very large amounts of data however, some tables in row-oriented databases need to be partitioned. Columnar databases scale to large sizes more easily *.
  2. Columns have uniform data types; this is optimal for compression. Column-oriented databases leverage run-length encoding **. Run-length encoding is like deduplication conceptually. Run-length encoding will store a sequence (or series) of data, and such sequences can be stored by their count.
  3. As there are no ETL jobs (per the problem scenario), a columnar database is superior. ETL jobs involve adding rows. Adding rows is inefficient for column-oriented databases because you have to write to many different columns. OLTP activity is better supported by row-oriented databases ***. If you will not be doing many ETL jobs or online transactional processing, column-oriented databases have benefits that row-oriented databases do not have.

* https://www.predictiveanalyticstoday.com/top-wide-columnar-store-databases/

** https://www.vertica.com/docs/9.2.x/HTML/Content/Authoring/AnalyzingData/Optimizations/UsingRunLengthEncodingRLEToImproveQueryPerformance.htm

*** https://www.omnisci.com/technical-glossary/columnar-database


FFR

  • To learn more about run-length encoding, see https://stackabuse.com/run-length-encoding/
  • Low cardinality in columns (meaning that values were often repeated) would increase the propensity to choose a columnar database because of run-length encoding.
  • ClickHouse is a column-oriented database, and it is very fast for many use cases (taken from clickhouse.com) . The company that maintains it recently (in 2021) obtained a large amount of funding (taken from this external website). As of January of 2022, Alibaba, Bloomberg, Microsoft, and Tesla have adopted ClickHouse (according to this external page).

How Do You Configure Thunderbird on Linux to Work with Web-Based Email?

Problem scenario
You want to configure Thunderbird to work with a web-based email account. What do you do?

Solution
Prerequisite

This assumes Thunderbird has already been installed. With CentOS/RHEL/Fedora, run this from the command line: sudo yum -y install thunderbird

With Debian/Ubuntu/Mint distributions of Linux with a GUI desktop, Thunderbird is usually already installed.

Procedures

  1. Launch Thunderbird.
  2. Set up your existing address.
  3. If there is a problem, you may need to click the "Configure manually" button and enter the web-based URL for the incoming and outgoing server.

How Do You Regain Access to Desktop Running in Oracle VirtualBox?

Problem scenario
You are using Oracle VirtualBox and a CentOS/RHEL/Fedora VM. You stepped away from your VM and now it appears the screen is locked. You do not see the desktop items you had open. You see there may be a new notification and you see the time. You tried to enter Ctrl-Alt-Del, Control-Alt-End, Control-Alt-Insert. None of the key combinations prompted you for your credentials. How do you log back in after the screen saver kicked in on your Red Hat distribution of Linux?

Solution
Click the mouse anywhere, then press Enter.

How Do You Build a Flask Application to Return the Price of a Cryptocurrency?

Problem scenario
You want to write a Flask program to display the high and low prices of a given cryptocurrency for a given day. You want the output to include the volume too. How do you do this?

Solution
Prerequisite

This assumes that you have installed Flask and pandas.

Procedures
Run this program. You may want to name it __crypto__.py
Then go to the IP address in a web browser or use curl. The way to compose the URL, see the usage instructions.

"""
Usage instructions:
Once running, the format of the URL to curl will look like one of these:

http://127.0.0.1:5000/crypto/BTC/2019-06-21
http://127.0.0.1:5000/crypto/ETH/2020-10-01

Change the dates to the date you want to see information for.
Change the currency symbol to the symbol you want to examine.

Written in March of 2021 by www.continualintegration.com

It was adapted from these postings:
https://towardsdatascience.com/cryptocurrency-analysis-with-python-buy-and-hold-c3b0bc164ffa
https://towardsdatascience.com/cryptocurrency-analysis-with-python-macd-452ceb251d7c?gi=12b983021ac2


"""

from flask import Flask, Response
import pandas as pd
import requests
from datetime import datetime


class Retriever:
    def get_filename(from_symbol, to_symbol,
                     exchange, datetime_interval, download_date):
        return '%s_%s_%s_%s_%s.csv' % (from_symbol, to_symbol,
                                       exchange, datetime_interval, download_date)

    def download_data(from_symbol, to_symbol, exchange, datetime_interval):
        base_url = 'https://min-api.cryptocompare.com/data/histo'
        url = '%s%s' % (base_url, datetime_interval)
        params = {'fsym': from_symbol, 'tsym': to_symbol,
                  'limit': 2000, 'aggregate': 1,
                  'e': exchange}
        request = requests.get(url, params=params)
        data = request.json()
        return data

    def convert_to_dataframe(data):
        df = pd.json_normalize(data, ['Data'])
        df['datetime'] = pd.to_datetime(df.time, unit='s')
        df = df[['datetime', 'low', 'high', 'open',
                 'close', 'volumefrom', 'volumeto']]
        return df

    def filter_empty_datapoints(df):
        indices = df[df.sum(axis=1) == 0].index
        df = df.drop(indices)
        return df

    def file_creator(from_symbol, to_symbol, exchange, datetime_interval):
        data = Retriever.download_data(from_symbol, to_symbol,
                                       exchange, datetime_interval)
        df = Retriever.convert_to_dataframe(data)
        df = Retriever.filter_empty_datapoints(df)
        latest_date = datetime.now().date().isoformat()
        filename = Retriever.get_filename(from_symbol, to_symbol, exchange,
                                          datetime_interval, latest_date)
        df.to_csv(filename, index=False)

    def read_dataset(filename):
        df = pd.read_csv(filename)
        df.datetime = pd.to_datetime(df.datetime)  # change to datetime
        df = df.set_index('datetime')
        df = df.sort_index()  # sort by datetime
        return df

    def processor(from_symbol, date_to_see):
        to_symbol = 'USD'
        exchange = 'Coinbase'  # "Bitstamp" is a valid option among others.
        datetime_interval = 'day'
        error_message = "INVALID FORMAT. The date must be in the format of YYYY-MM-DD. Use leading 0 for single digit months or days."
        if date_to_see[0:4].isnumeric():
            pass
        else:
            return "Letters found. " + error_message
        if date_to_see[5:7].isnumeric():
            pass
        else:
            return "Letters found. " + error_message
        if date_to_see[8:].isnumeric():
            pass
        else:
            return "Letters found. " + error_message
        if len(date_to_see) != 10:
            return error_message
        if date_to_see[4] != "-":
            return error_message
        if date_to_see[7] != "-":
            return error_message
        if int(date_to_see[0]) > 2:
            return "Year was too far in the future. "
        if int(date_to_see[0]) < 2:
            return "Year was too far in the past. "
        Retriever.file_creator(from_symbol, to_symbol, exchange,
                               datetime_interval)
        latest_date = datetime.now().date().isoformat()
        btc_date = open((Retriever.get_filename(from_symbol, to_symbol,
            exchange, datetime_interval, latest_date)), 'r')
        get_top = btc_date.read()[0:-1]
        for line in get_top.split():
            if line.split(",")[0] == date_to_see:
                return line


app = Flask(__name__)

@app.route("/")
def default():
    return Response("remember the path is /crypto/BTC/2020-01-01 "), 200

@app.route("/crypto/<string:symbol>/<string:day_var>", strict_slashes=False)
def crypto(symbol, day_var):
    res = Retriever.processor(symbol, day_var)
    portion = ", the format is as follows: date,low,high,open,close,volume_from,volume_to"
    compose = "For " + symbol + portion + "\n" + res
    return Response(compose), 200

if __name__ == "__main__":
    #app.run(debug=True)  # Uncomment out this line, comment out line below to run on 127.0.0.1
    app.run(host='192.168.1.15', debug=True)

If you want to receive free cryptocurrency by just learning more, try Coinbase; it is ideal for Americans. For Europeans, the platform/company Iconomi.com can allow you to buy crypto or learn more.

How Do You Keep a Windows 10 Hard Drive Using the Correct Time?

Problem scenario
You have two hard drives in your computer (e.g., desktop or laptop). You boot to each one from time-to-time. One hard drive is a Linux OS and the other is Windows 10. When you switch between the hard drives, your Windows OS has an incorrect time set. What should you do to stop needing to manually reset the time zone each time you boot to Windows?

Solution
1. Go to Control Panel -> Clock and Region -> Set the time and date
2. Go to the "Internet Time" tab
3. Click on "Change Settings"
4. Check the option "Synchronize with an Internet time server"
5. For the "Server" use "time.windows.com"
6. Click "Update now"
7. Click "OK"

If you want a non-GUI method (e.g., with PowerShell), see this posting.

How Do You Deal with Forearm, Hand and Wrist Pain from Keyboard Usage?

Problem scenario
You want to prevent hand, wrist or forearm pain from regular computer usage. What should you do to learn more about this and keep your hands/wrists/forearms from hurting?

Possible Solutions
Disclaimer: This is not medical advice. Repetitive motions can cause long-term health problems, and this website is purely for general information. Do not hesitate to consult with a real medical professional for your concerns or pain.

You may need to use the keyboard less. Carpal tunnel syndrome (which is often described as numbness rather than pure pain) and tendonitis are real concerns. There can be long term consequences of excessive computer usage.

Be sure to get plenty of exercise regularly. There are exercise videos and workout equipment available online.

A tablet computer can help with that. Voice recognition is another tool.

You may need to develop greater hand, wrist and forearm strength; to do this, you may want to see these very portable exercise devices. Some people would feel less pain with stronger hands, wrists and forearms.

You may need to use an ice pack.

You may need to purchase a more ergonomic mouse or a more ergonomic keyboard. Alternatively you may want to buy supportive equipment for your wrists.

You may need to call a licensed physician. A doctor can prescribe real treatment or refer you to physical therapy.

You may want to read more with these links:

How Do You Use an Ab Roller without Losing Your Grip?

Problem scenario
You have used an abdominal roller but hurt your wrist when you lost your grip. You like how it works your abs, but it feels precarious to use. What should you do?

Possible Root Cause
It could be that you were using a skinny ab roller. We find skinny ones to be more stressful and jerky to use -- but not in a way that enhances the workout. This type of ab roller may have caused the problem:

You do not want to lose your balance. You want the reps/extensions to be done with control throughout the range of the movement.

Possible Solution / Procedures
We recommend the wider Sports Research Ab Wheel Roller.

This thicker device seems to provide a more stable exercise than a thinner option.

How Do You Troubleshoot a Computer That Will Not Boot to a USB Drive?

Problem scenario
You have a thumb drive with an .iso on it. You want to boot to it to install a hard drive. When you turn your desktop/laptop/PC on, you get “BootDevice Not Found Please Install an operating system on your hard disk. Hard Disk (3F0).” Or you get an error like this “disk error press any key to restart.”

Solution
The USB drive may not be configured to be bootable. To do this, you will lose everything on your USB stick; be sure to back the files up. Then use this link.