If your shell always get’s you wrong - quote the right way

Here is another question that often comes up by my students and clients:

” … how can I solve the problem with these special-characters? The shell always gets me wrong …”

The fast and simple answer to this is: You have to quote!

… and you have to to it the right way.

But let’s start from the beginning:

As you know, there are characters at the command line, that simply has special meanings for the shell.

You use them for instance, for referencing the content of a variable (“$”)

robert@demo:~$ echo Hello $NAME

… or for redirecting a datatream to a file (“>”):

robert@demo:~$ ls -l /etc > $TEMPFILE

But everytime you wanna use these special characters without their special meaning for the shell, you have to take special care about it.

(huh - three times “special” in one sentence. this must be really special ;-) )

So instead of writing

robert@demo:~$ echo Buy this book for $9 now
Buy this book for now

you have to write something like

robert@demo:~$ echo 'Buy this book for $9 now'
Buy this book for $9 now

In this way, the shell won’t try to interprete $9 as a variable. Instead it would take the “$”-sign just like what it is: a $-sign.

This mechanism is called “quoting” and technically explained for instance in depth in the Bash-Documentation.

Let me show you, how the quoting works. And your shell never gets you wrong again … ;-)

What special-characters you should care about

Special characters that need special handling in the shell are obviously the following:

<space> ; , & | > <.

But also the lesser known meta-characters ( and ) and the placeholders for the filename globbing ( * ? [ ] ) need special handling when used at the command-line.

If you use history expansion (why would you not?), then the character ! has also to be included in the “TakeSpecialCare”-list.

And because the backslash (\) is used for quoting, you have to quote it if you like to use it as really only a backslash.

The same goes for the " and the ' characters. You often use them if you quote intuitively, but they need special care if you want to use them as the characters they are.

So, the following characters need special care when used in the shell:^*

& ; ( ) < > space tab * ? [ ] ! “ ‘ ` # -

^*Note: Why the dash ‘-‘?

Typically a dash (‘-‘) doesn’t have a special meaning for the shell. But for commands started from the commandline it sometimes has - espacially if it is the first character of a parameter. Think about for instance ls -l, where “-l” is be the name of a file …

How to do this special handling

Generally, you have three options to let the shell ignore characters with special meanings:

• single-character-quoting: Prepend the special character with a backslash \

  echo By this book for \$9 only

• “weak-quoting”: Surround the string containing one or more special characters by "

  echo "This is an ampersand: &"

• “hard-quoting”: Surround the string containing one or more special characters by '

  echo 'By this book for $9 only'

Perhaps you’ve heard about all or some of these quoting-options.

But what are the differences?

Use \ to quote only one single character

Obviously, the first variant is only usefull for single characters.

Use this method, if you only want to quote a single known character or you want to quote the “quote-markers” (\, " and ').

Hint: What happens, if the shell sees the quote-markers (\, " or ') on your command-line?

It thinks “oh ok - let’s ignore those following special characters” and … removes the quote-markers itself before interpreting the commandline further.

Here are some examples:

spaces in filenames:

~$ mkdir my\ pictures	# --> creates a directory named "my pictures"

Instead of creating the two directories “my” and “pictures” (as it would without the backslash before the space), mkdir creates only one with a space inside the name (“my pictures”).

do not expand variables:

~$ echo hello $USER
hello robert
~$ echo hello \$USER
hello $USER

output the “quote-helpers”:

~$ echo try to use "string" or \n for quoting
try to use string or n for quoting
~$ echo try to use \"string\" or \\n for quoting
try to use "string" or \n for quoting

output a single asterisk (*):

if the shell “sees” an asterisk at the commandline, it tries to interpret the resulting string as a search-pattern for filenames.

If there are files that correspond to the pattern, then the shell substitutes the pattern by a list of that files. If no files where found, the shell ignores the pattern and leaves the pattern alone.

~$ echo *.txt
list.txt error.txt	<-- list with filenames matching the pattern
~$ echo *.text
*.text            <-- no filename matches the pattern, no substitution
~$ echo \*.txt
*.txt

“Weak-quoting” with double quotes “ … “

The second variant - often called “weak-quoting” - is the way you usually quote intuitively.

You use a filename containing spaces?

… surround it with double-quotes!

~$ cp financial report.doc /tmp
cp: cannot stat ‘financial’: No such file or directory
cp: cannot stat ‘report.doc’: No such file or directory
~$ cp "financial report.doc" /tmp

sure - you could quote the single space with a backslash, but surrounding the whole string with double-qoutes is more easy to read I think.

You want to print out a string with a sequence of spaces?

… surround the whole thing with double-quotes!

~$ echo hello        world
hello world
~$ echo "hello        world"
hello         world

• at the first try: the shell takes the sequence of spaces between “hello” and “world” as only one separator at the commandline and the echo-command gets two parameters for output: “hello” and “world”

• at the second try: we give the echo-command only one parameter for output containing all the important spaces.

… ok Robert - I’ve got it. But why the hell is this called weak-quoting?

Glad you ask ;-)

Why it is called “weak-qouting”

Well - it’s called “weak-quoting”, because there are characters, which are nevertheless interpreted by the shell.

Imagine, you want to output a string (with lot’s of spaces) and replace a variable-name by it’s content:

~$ echo "hello          $NAME"
hello          Robert

That’s great: The shell preserves the spaces and at the same time interprets the “$”-sign and expands the variable.

Between double-quotes, the following special characters will be interpreted: $, \, ! and the “backticks” (`).

Especially the backslash is handy if you want to quote something within your weak-quoted string:

~$ echo "I know you are $NAME, because of \$NAME"
I know you are Robert, because of $NAME

“Hard-quoting” with single quotes ‘ … ‘

But what if you want nothing to be interpreted by the shell? The strings shall be used as they are? Use the hard-quoting with single quotes:

~$ echo 'Hi $NAME, eat this: "; <> |"'
Hi $NAME, eat this: "; <> |"

That’s it

As you could see, you can use nearly any special character in any combinations in the shell without the fear of being misunderstood.

I hope you enjoyed this lesson. See you next time

Robert

Here is what to do next

If you followed me through this article, you certainly have realized that knowing some internals about how things are working at the Linux command line, can save you a lot of time and frustration.

And sometimes it’s just fun to leverage these powerful mechanics.

If you wanna know more about such “internal mechanisms” of the Linux command line - written especially for Linux beginners

have a look at “The Linux Beginners Framework”

In this framework I guide you through 5 simple steps to feel comfortable at the Linux command line.

This framework comes as a free pdf and you can get it here.

Wanna take an unfair advantage?

If it comes to working at the Linux command line - at the end of the day it is always about knowing the right tool for the right task.

And it is about knowing the tools that are most certainly available on the Linux system you are currently on.

To give you all the tools for your day-to-day work at the Linux command line, I have created “The ShellToolbox”.

This book gives you everything

• from the very basic commands, through
• everything you need for working with files and filesystems,
• managing processes,
• managing users and permissions, through
• hardware analyses and
• simple shell-scripting to the tools you need for
• doing simple “networking stuff”.

Everything in one single, easy to read book.

With explanations and example calls for illustration.

If you are interested, go to shelltoolbox.com and have a look (as long as it is available).