Although pbcopy and pbpaste can be used to do more than simply copy text data into and out of the general system clipboard, in practice, I never use those features. In fact, I sometimes find it annoying that there wasn’t just one tool provided that sensed if there was data on STDIN, copied it to the clipboard, and pasted data appropriately from the clipboard on STDOUT.

In fact, the more I thought about it, the more I thought it should do both.

This utility could probably be written in C or Go, but I couldn’t think of any reasons I’d want native speed for CLI-based clipboard manipulation, so I papered over the existing utilities with a bit of shell:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

pb() {
  if [[ -p /dev/stdin ]]; then              # STDIN is attached to a pipe
    pbcopy
  fi

  if [[ ! -t 0 && ! -p /dev/stdin ]]; then  # STDIN is attached to a redirect
    pbcopy
  fi

  if [[ -t 1 ]]; then                       # STDOUT is attached to a TTY
    if [[ -t 0 ]]; then                     # STDIN is attached to TTY
      pbpaste
    fi
  fi

  if [[ -p /dev/stdout ]]; then             # STDOUT is attached to a pipe
    pbpaste
  fi

  if [[ ! -t 1 && ! -p /dev/stdout ]]; then # STDOUT is attached to a redirect
    pbpaste
  fi
}

And here it is again with some composure metadata:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

pb() {
  author 'Erich Smith'
  about 'clipboard DWIM tool'
  example 'pb                    # paste'
  example 'echo hi | pb          # copy'
  example 'echo hi | pb | cat -  # copy and paste'
  example 'pb </tmp/file         # copy'
  example 'pb >/tmp/file         # paste'
  example 'pb </tmp/f1 >/tmp/f2  # copy and paste'
  group 'shell'

  if [[ -p /dev/stdin ]]; then              # STDIN is attached to a pipe
    pbcopy
  fi

  if [[ ! -t 0 && ! -p /dev/stdin ]]; then  # STDIN is attached to a redirect
    pbcopy
  fi

  if [[ -t 1 ]]; then                       # STDOUT is attached to a TTY
    if [[ -t 0 ]]; then                     # STDIN is attached to TTY
      pbpaste
    fi
  fi

  if [[ -p /dev/stdout ]]; then             # STDOUT is attached to a pipe
    pbpaste
  fi

  if [[ ! -t 1 && ! -p /dev/stdout ]]; then # STDOUT is attached to a redirect
    pbpaste
  fi
}

Credit to Dejay Clayton for his very informative SO post on the topic of STDIN, STDOUT and STDERR detection.

Sometimes you’ll want to view some output in less, the default PAGER, and you’ll realize that the output was colorized with ANSI escape sequences.

To demonstrate the issue, I’ll run a ruby file (example taken from docopt) through pygmentize:

1

$ pygmentize -l ruby naval.rb | less

which produces:

ESC[36mrequireESC[39;49;00m ESC[39;49;00mESC[33m"ESC[39;49;00mESC[33mdocoptESC[39;49;00mESC[33m"ESC[39;49;00m
docESC[39;49;00m ESC[39;49;00m=ESC[39;49;00m ESC[39;49;00m<...ESC[39;49;00m
ESC[33m  #{__FILE__} ship  move   [--speed=]ESC[39;49;00m
ESC[33m  #{__FILE__} ship shoot  ESC[39;49;00m
ESC[33m  #{__FILE__} mine (set|remove)   [--moored|--drifting]ESC[39;49;00m
ESC[33m  #{__FILE__} -h | --helpESC[39;49;00m
ESC[33m  #{__FILE__} --versionESC[39;49;00m

ESC[33mOptions:ESC[39;49;00m
ESC[33m  -h --help     Show this screen.ESC[39;49;00m
ESC[33m  --version     Show version.ESC[39;49;00m
ESC[33m  --speed=  Speed in knots [default: 10].ESC[39;49;00m
ESC[33m  --moored      Moored (anchored) mine.ESC[39;49;00m
ESC[33m  --drifting    Drifting mine.ESC[39;49;00m

ESC[31mDOCOPTESC[39;49;00m

ESC[34mbeginESC[39;49;00m
  ESC[39;49;00mESC[36mrequireESC[39;49;00m ESC[39;49;00mESC[33m"ESC[39;49;00mESC[33mppESC[39;49;00mESC[33m"ESC[39;49;00m
    ESC[39;49;00mppESC[39;49;00m ESC[39;49;00mESC[33mDocoptESC[39;49;00m:ESC[39;49;00mESC[33m:docoptESC[39;49;00m(ESC[39;49;00mdocESC[39;49;00m)ESC[39;49;00m
    ESC[34mrescueESC[39;49;00m ESC[39;49;00mESC[33mDocoptESC[39;49;00m:ESC[39;49;00mESC[33m:ExitESC[39;49;00m ESC[39;49;00m=ESC[39;49;00m>ESC[39;49;00m ESC[39;49;00meESC[39;49;00m
      ESC[39;49;00mESC[36mputsESC[39;49;00m ESC[39;49;00meESC[39;49;00m.ESC[39;49;00mmessageESC[39;49;00m
      ESC[34mendESC[39;49;00m
      (END)]]]]]]]]]]]]]]]]]]]]]]]]])]]]]]]]]]"]]]]"]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]"]]]]"]]]]

Keine Probleme!

Of course, you could have invoked less with -r: display raw control characters, but what to do after the fact, when you’ve got a steaming pile of escape sequences staring back at you?

Less supports entering flags from the : prompt. So just type the following literal key sequence to enter raw character mode:

:--r

That’s colon, dash, dash, r.

Now your less output looks like:

require "docopt"
doc = <<DOCOPT
Naval Fate.

Usage:
  #{__FILE__} ship new <name>...
  #{__FILE__} ship <name> move <x> <y> [--speed=<kn>]
  #{__FILE__} ship shoot <x> <y>
  #{__FILE__} mine (set|remove) <x> <y> [--moored|--drifting]
  #{__FILE__} -h | --help
  #{__FILE__} --version

Options:
  -h --help     Show this screen.
  --version     Show version.
  --speed=<kn>  Speed in knots [default: 10].
  --moored      Moored (anchored) mine.
  --drifting    Drifting mine.

DOCOPT

begin
  require "pp"
  pp Docopt::docopt(doc)
rescue Docopt::Exit => e
  puts e.message
end

Effect

A grizzled neckbeard hands you a shell script, and tells you to run it, and that it will install the application you’ve been wanting to try out.

“What?” you say. “Does it download it from the Internet?”

“No. No network required.”

“What, do I run this with Java, or something?”

“No, it is self-contained.”

“Do I need to type a cryptic command sequence to get this to run?”

Look, please just run the script. It will install your application.”

Then you look at the file, and see that it is several megabytes long. And, sure enough, it installs your shiny new app. But how?

Preparation

You will need:

• a tared and gzipped file containing the source or binary distribution you want to bundle
• knowledge of which additional commands to run, if any, to compile, install or otherwise deal with the unpacked files
• an open mind

Method

You will be creating a self-extracting executable. The fact that the ‘executable’ portion is really interpreted shell script, is irrelevant. This technique hinges on the ability of the shell script to reference itself, ignore the shell script portion, and feed the remainder to the shell script as binary data:

I actually don’t prefer to use this version above, but the magic is easy to spot on line 3: tail -n +8 $0: $0 is a reference to this, the current script file, so this command dumps line 8 through the end of the file through a pipeline to a tar extraction command. A careful count of the source reveals that line 8 is where the binary tar data was appended.

I don’t like this version because you have to manually and carefully count the lines of code in your script and change line 3 accordingly. I much prefer the following snippet which is more robust, but perhaps a bit harder to follow:

This version uses file redirection on $0 to put the contents of the script through the whole shell block between curly braces, and uses a while read loop to pre-emptively throw away the lines of STDIN that correspond to the script preamble, before passing the remaining, binary data through tar. The while loop also cleverly uses the string matching capability of case to treat the exit 0 as a sentinel value. Thus, exit 0 serves as both the start-of-data marker, and the end-of-script command. Wild.

Someone asked what I used to generate the slides. I used mdpress, which in turn uses impress.js for the Prezi-like transitions.

REPLs ❤ Pipes

The shell command prompt is great for quick REPL feedback: run a command, watch the output, tweak the command, and repeat until you have the output you desire. This is a great way to build up complex command sequences. Feedback is immediate, and the ability to recall and directly edit recent command history encourages experimentation.

The prompt is no replacement for a full text editor, however, and there are times that you’ll start a command at the prompt and wish you had done so in an editor.

While available for short-term recall in terminal history, command sequences entered at the prompt have a short shelf-life. Shell functions defined (albeit painfully) at the prompt, for example, do not persist across shell invocations.

Composing a complex logic construct or shell function at the prompt is, at best, ephemeral and awkward.

Scripts ❤ Functions

Script files are the natural habitat for functions and larger programmatic constructs: after all, you have a full 2D editor environment in which to draft, save and revise your code. What might be a struggle and a chore to compose at the command prompt becomes easy inside a text editor.

Persisting to the filesystem means that you can bring all the tools for dealing with files to bear on your script, including version control—critical to maintaining code of any real complexity.

Script files have downsides, of course. Chief among them that you lose all the feedback from the command prompt—if you are creating a complex command sequence, you’re flying blind inside the editor. Also, you’ve just inherited all the file-related baggage required to get your script to actually run: shebang lines, executable bits, and PATH considerations.

Composing a complex logic construct or shell function in a script file feels natural, but loses the immediacy of the prompt and can sometimes feel too heavyweight: both factors which discourage the proliferation of lots of small shell functions, which as we’ve seen can vastly improve the readability and maintainability of our command sequences.

Best of both worlds?

I care about readability in my shell code. If you’re reading this, perhaps you do too. I wanted the best of both worlds, prompt and editor, so I wrote composure, a set of opinionated shell functions that help you write shell functions.

In a future post, I’ll highlight a few of the features and benefits of writing shell functions with composure, and a few unexpected use cases I’ve found.

The fc utility

First introduced via the Korn Shell, and a standard Unix utility since at least the mid-90’s, fc is short for “fix command”. Its purpose is to open up a previous command in the editor of your choice, and re-run the revised command once you are finished editing it.

With no arguments, fc will open the most previous command in the editor for editing and re-evaluation. In pure shell terms, it is roughly equivalent to:

1

echo !! > /tmp/cmd && $EDITOR /tmp/cmd && eval $(cat /tmp/cmd); rm /tmp/cmd

Most implementations of fc will use an editor specified in the FCEDIT environment variable, followed by EDITOR, then vi, and finally ed. You could of course launch an external editor like Sublime Text or Eclipse, but I find that keeping the editor inside the terminal feels most natural for this purpose.

I want to talk now about the experience of using a command like this. If you’ve not used this utility before, go ahead and try it out a few times. Yes, right now. Seriously, I’ll wait.

Context Switching

So, kinda cool, but a little jarring, right? There you were happily plunking commands down at a prompt, and all of a sudden you’re in your familiar editor, but perhaps in a slightly new context. Now you are editing your last command, say ls -l, as if it were a script:

1
2
3

#!/bin/sh

ls -l

Furthermore, the act of saving and quitting ran the script, back at your terminal.

Two Environments, Two Adaptations

I think the feeling of slight disorientation that occurs when using fc for the first time stems from inherent tension between the two modes of operation, or execution environments of the shell interpreter: the shell is at once the familiar REPL prompt we use all the time, and the shell is also the interpreter of script files.

The Prompt

We naturally adapt our shell behaviors, as users, to these different execution environments: the interactive, REPL-nature of the prompt means that we can run a quick command, and immediately see the output displayed to the terminal. Based on that output, we might decide to try a different command altogether, consult a man page, or perhaps just hit the up arrow and make a quick revision to our command.

In particular, using the shell prompt to build up complex command pipelines is invaluable: you get to see the output of each filter command:

At the prompt, this feels natural. You try a command, examine the output or result, add an appropriate filter command or commands until you have massaged the output into a desired format. You have the benefit of understandability as a writer of the command because the output of all the intermediate commands are available to you while you are writing.

The Script

Contrast that experience with encountering something like this in a script:

1
2

#!/usr/bin/env bash
 netstat -p tcp | grep -i established | awk '{print $5}' | awk -F'.' '{ $NF="" } 1' | sed 's/ /./g' | sed 's/\.$//' | sort | uniq

The above is simply unintelligible. Even if you are a regex master and know the syntax for each of the above commands, you do not have the benefit of having written that command to begin with. The intent of all those pipeline filters is not communicated to the reader, and unless you actually played with that command, one step at a time, at the prompt, it is unlikely you would ever know what that command does.

Pipes, as a design feature, are central to the Unix way. They are Unix’s killer app. Unfortunately, they mask the intent of a command sequence. They are extremely convenient to use at a prompt, but end up being write-only.

The maintainer of a script with the above command sequence in it would be well-advised to break that command into sections, or at least encapsulate the pipeline in a descriptive function, like:

1
2
3
4
5
6
7

#!/usr/bin/env bash

function unique_remote_connected_hosts {
  netstat -p tcp | grep -i established | awk '{print $5}' | awk -F'.' '{ $NF="" } 1' | sed 's/ /./g' | sed 's/\.$//' | sort | uniq
}

unique_remote_connected_hosts

or perhaps:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

#!/usr/bin/env bash

function unique_remote_connected_hosts {
  remote_connections | strip_port_number | sort | uniq
}

function remote_connections {
  netstat -p tcp | grep -i established | awk '{print $5}'
}

function strip_port_number {
  awk -F'.' '{ $NF="" } 1' | sed 's/ /./g' | sed 's/\.$//'
}

unique_remote_connected_hosts

This functional composition is easy to do in a text editor, the natural habitat of a script. It is awkward and difficult to do at the prompt, which is too bad.

More on resolving this fundamental tension, next time.

Really, though—isn’t There More Than One Way To Skin A Cat? If cat with a single file argument gets the job done, who besides Randal Schwartz cares?

I care, and I think you should too.

The Broken Windows Theory suggests that by continuing to use a broken mental model of the cat utility, when you know of a better way, you may be subtly devaluing your surrounding code, and inviting future abuse. Left unchecked, this kind of thinking forms habits that tend to spider outward and, weed-like, slowly choke the life out of a project.

Saying “Who cares?” to fixable issues within your sphere of control and influence is, in effect, choosing laziness.

Laziness can be a virtue, but it has a dark side, and must be employed intelligently.

Laziness in the form of putting off much-needed refactoring leads to design spaghetti and unmaintainable code: Fail. This is self-defeating behavior that saps your morale, your momentum needed to accomplish the next task, and your motivation to continue learning and growing. This path ultimately leads to the place programmers go to die: maintaining some ancient application that everyone lives in fear of changing, hoping they are too valuable to be fired. Don’t be a Wally.

Laziness employed in service of, say, Test-Driven Design leads to deferring design decisions and coding the simplest thing that could possibly work: Win. This is wise coding that stems from an experiential belief that you will know more about your problem domain tomorrow than you do today. Implicit in this belief is an internal drive to be constantly learning, and a promise, to yourself at least, that tomorrow you will clean up any messes you unintentionally make today.

Don’t devalue your own worth, sense of accomplishment, and ability to grow. Fix that broken window today.

More on fixing your shell commands as you go, next time.

1

cat myfile | grep mypattern

Of course, this kind of abuse isn’t limited to the “pipe to grep” variety. Here’s the general form:

1

cat <file> | <command> arg1 arg2 argN

Indeed, flaming other Unix users for the Useless Use of Cat has long been the ‘national sport’ of the Unix tribe, since Usenet days.

What’s wrong with this pattern?

One thing, certainly. Two things, probably.

1. cat is meant to concatenate two or more files together

If you are using cat with only one filename argument, then by definition you aren’t concatenating anything.

So, here’s the proper use of cat:

1

cat blacklion.txt greenlion.txt redlion.txt bluelion.txt yellowlion.txt > VOLTRON.txt

2. many commands accept STDIN, but prefer a filename argument

This falls into a related anti-pattern of “Useless Use of Pipe”. Instead of:

1

cat internet_memes.txt | grep -i "chuck norris"

Consider:

1

grep -i "chuck norris" internet_memes.txt

Why is this such a common anti-pattern?

I still find myself occasionally falling into this pattern, despite two decades’ worth of keyboard time on Unix variants. I think this habit likely persists for two reasons:

1. The habit is learned early

Early bad habits are often hard to break. One of the first things a CLI user wants to do is to read a file, and until the use of pager commands becomes second nature, this will likely lead to:

1

cat somefile

For small ASCII files, this may even be an appropriate use of cat. Usually though, this early habit leads to a second, more subtle, and persistent mental model.

2. cat is a conceptual pipeline source

When building up a complex filter, often cat starts things off:

1

cat myfile | grep pattern | tr 'A-Z' 'a-z' > outfile

The thought process behind this command is: start with some text, then filter it thus, and filter it so, then write it to an output file. Simple. In some cases, I even give myself a pass on this if it makes my intention clearer. Usually, though, I try to fix the habit.

The Fix?

Refactoring. No, really. More on this next time.

Bonus: Useless use of wc

Often I will do the following to get a matching pattern count:

1

cat file1 file2 | grep pattern | wc -l

So, first let’s clean up the UUoP:

1

grep pattern file1 file2 | wc -l

Turns out, grep has a -c flag, which displays the count of matched lines, so:

1

grep -c pattern file1 file2

cats.each do |cat|
  cat.kill if cat.curious?
end