modules

Command Line 3

Redirection & Piping

Redirecting output to file: >
command > filename

Takes the output of command and saves it in filename. This will overwrite the file if it already exists.

Redirecting output and appending to file: >>
command >> filename

Takes the output of command and appends it to the end of the content of filename. This will create the file if it does not yet exist.

Piping: |
command1 | command2

Pipes the results from command1 as input to command2, and then the results of command2 are printed to the console.

Lets see how it works

First, lets install a new command fortune

  1. Redirect: 
      fortune > wisething.txt
  2. Pipe 
      fortune | cowsay
  3. Pipe then Redirect 
      fortune | cowsay > wisecow.txt

❇️ Example: Redirection

  1. Redirect a fact about planet mars into the mars.txt.
echo "Mars is dusty." > ~/Development/universe/solar_system/planets/mars.txt
  1. Add another mars fact to mars.txt.
echo "Mars has an 687 day year." >> ~/Development/universe/solar_system/planets/mars.txt
  1. Cat the contents of mars.txt.
cat ~/Development/universe/solar_system/planets/mars.txt
  1. Make sure there is at least one newline at the end of mars.txt.

You can check if there is a newline at the end of the file if there is no % sign that appears at the end of the file.

In this screenshot, the first example with the % at the end has no newline. The latter example does.

❇️ Example: piping

Count the number of characters in the string “hello world” using wc.

echo "hello world" | wc -c

Count the number of lines in the file mars.txt

cat ~/Development/universe/solar_system/planets/mars.txt | wc -l

Count the number of characters in the first line of mars.txt

cat ~/Development/universe/solar_system/planets/mars.txt | head -n 1 | wc -c

➡️ Try It

Count the number of characters in mars.txt Count the number of characters in the last line of mars.txt

❇️ Example: piping

Count number of folders in the universe folder.

cd ~/Development/universe
find . -type d | wc -l

➡️ Try It

Count number of files in the universe folder.

Slackcat

Lets install a new command-line tool. On macOS use brew install slackcat. For linux, see the bottom of this page https://github.com/vektorlab/slackcat for instructions.

Configure slackcat by running this command and following the instructions in your web browser:

slackcat --configure

❇️ Example

Let’s use slackchat to send a simple message to the #scratchwork channel.

echo "hello world" | slackcat -c scratchwork

➡️ Try It

Notice how the message is being sent as a snippet. Figure out how to send a normal, non-snippet, message using slackcat.

➡️ Try It

  1. Count the total number of files and folders in the ~/Development/universe/ directory and send this to the person sitting next to you using slackcat. Use only one line and piping. You can use the --noop flag to first test it out without sending the message, then you can remove it to send the message. (hint: start with tree)
  1. In the ~/Development/universe directory, run ls, pipe the output of that into slackcat and send it to the #scratchwork channel. This time, make sure to send it as a snippet.

❇️ Example: piping and redirection

House Office Expenditure Data: https://projects.propublica.org/represent/expenditures

  1. Let’s start a new directory for the house expenditure data.

     cd ~/Development
 mkdir house-expenditure
 cd house-expenditure

  2. Download the Q2 2017 expenditure detail data and pipe it into a file.

     curl "https://projects.propublica.org/congress/assets/staffers/2017Q2-house-disburse-detail.csv" > 2017Q2-house-disburse-detail.csv

  3. Print the header (first line) of this file.

     head -n 1 2017Q2-house-disburse-detail.csv

  4. Print the last 12 lines of this file.

     tail -n 12 2017Q2-house-disburse-detail.csv

  5. Count the number of lines in this file.

     cat 2017Q2-house-disburse-detail.csv | wc -l

  6. Count the number of rows in this file that contains the word “technology” (case insensitive)

     cat 2017Q2-house-disburse-detail.csv | grep -i technology | wc -l

  7. Return only the rows containing the word “technology” and redirect the output into a file. Keep the header.

     head -1 2017Q2-house-disburse-detail.csv > technology.csv
 cat 2017Q2-house-disburse-detail.csv | grep -i technology >> technology.csv

  8. Grep a word of your choice and send the first 5 lines to #scratchwork channel on slack.

    cat 2017Q2-house-disburse-detail.csv | grep -i technology | head -n 5 | slackcat --filename technology.csv -c scratchwork