If your shell always get’s you wrong - quote the right way

This is another topic I’m often questioned about by my students: quoting when working with the shell:

…What are the problems with these “special-characters”?…

As you know, there are characters at the command line, that simply has special meanings for the shell.

You use them for instance for referencing the content of a variable (“$”) or for redirecting a datatream to a file (“>”):

~$ ls -l /etc > $TEMPFILE

Everytime you use such special characters but you don’t want that special meaning for the shell, you have to care about it accordingly.

This mechanism is called “quoting” and technically explained for instance in depth in the Bash-Documentation.

In this post I show you how the quoting works. And your shell never gets you wrong again … ;-)

What special-characters you should care about

Special characters that need special handling in the shell are obviously the following: <space> ; , & | > <. But also the lesser known meta-characters ( and ) and the placeholders for the filename globbing ( * ? [ ] ) need special handling when used at the command-line.

If you use history expansion (why should you not?), then the character ! has also to be included in the “TakeSpecialCare”-list.

And because the backslash (\) is used for quoting, you have to quote it if you like to use it really only as a backslash.

The same goes for the " and the ' characters. You often use them if you quote intuitively, but they need special care if you want to use them as the characters they are.

Summary:

the following characters need special care when used in the bash:*

& ; ( ) < > space tab * ? [ ] ! “ ‘ ` # -

Why the dash ‘-‘?

Generally a dash “-“ doesn’t have a special meaning for the bash - but for commands started from the commandline it sometimes has - espacially if it is the first character of a parameter. Think about listing a file with the name “-l” with the ls command.

How to do this special handling

Generally, you have three options to let the bash ignore characters with special meanings:

  • single-character-quoting: Prepend the special character with a backslash \ example: \c

  • “weak-quoting”: Surround the string containing one or more special characters by " example: "string"

  • “hard-quoting”: Surround the string containing one or more special characters by ' example: 'string'

I think you’ve heard already about these variants. But what are the differences?

Quoting only a single character

Obviously, the first variant is only usefull for single characters.

Use this method, if you only want to quote a single known character or you want to quote the “quote-markers” (\, " and ').

Hint: What happens, if the bash sees the quote-markers (\, " or ') on your command-line? It thinks “oh ok - let’s ignore those special characters” and … removes the quote-markers before interpreting the commandline further.

Here are some examples:

spaces in filenames:

~$ mkdir my\ pictures	# --> creates a directory named "my pictures"

Instead of creating the two directories “my” and “pictures” (as it would without the backslash before the space), mkdir creates only one with a space inside the name (“my pictures”).

do not expand variables:

~$ echo hello $USER
hello robert
~$ echo hello \$USER
hello $USER

output the “quote-helpers”:

~$ echo try to use "string" or \n for quoting
try to use string or n for quoting
~$ echo try to use \"string\" or \\n for quoting
try to use "string" or \n for quoting

output a single asterisk (*):

if the shell “sees” an asterisk at the commandline, it tries to interpret the resulting string as a search-pattern for filenames. If there are files that correspond to the pattern, the bash substitutes the pattern by a list of that files. If no files where found, the bash ignores the pattern and leaves the pattern alone.

~$ echo *.txt
list.txt error.txt	<-- list with filenames matching the pattern
~$ echo *.text
*.text				<-- no filename matches the pattern, no substitution
~$ echo \*.txt
*.txt

“Weak-quoting” with double quotes “ … “

The second variant - often called “weak-quoting” - is the way you usually quote intuitively. You see a filename containing spaces? Surround it by double-quotes. You want to print out a string with a sequence of spaces? Surround the whole thing with double-quotes:

~$ cp financial report.doc /tmp
cp: cannot stat ‘financial’: No such file or directory
cp: cannot stat ‘report.doc’: No such file or directory
~$ cp "financial report.doc" /tmp

Sure - you could quote the space with a backslash, but this variant is more common.

~$ echo hello        world
hello world
~$ echo "hello        world"
hello         world

At the first try, the bash takes the sequence of spaces between hello and world as only one separator for the commandline and the echo-command gets two parameters for output. At the second try we give the echo-command only one parameter for output containing all the important spaces.

… ok - I’ve got it. But why the hell is this called weak-quoting?

Glad you ask ;-)

Well - it’s called “weak-quoting”, because there are characters, which are nevertheless interpreted by the bash.

Imaging, you want to output a string (with lot’s of spaces) and replace a variable-name by it’s content:

~$ echo "hello          $NAME"
hello          Robert

That’s great: The shell preserves the spaces and at the same time interprets the “$”-sign and expands the variable.

The characters interpreted between double-quotes are$, \, ! and the “backticks” (`).

Especially the backslash is handy if you want to quote something within your weak-quoted string:

~$ echo "I know you are $NAME, because of \$NAME"
I know you are Robert, because of $NAME

“Hard-quoting” with single quotes ‘ … ‘

But what if you want nothing to be interpreted by the shell? The strings shall be used as they are? Use the hard-quoting with single quotes:

~$ echo 'Hi $NAME, eat this: "; <> |"'
Hi $NAME, eat this: "; <> |"

That’s it

As you could see, you can use nearly any special character in any combinations in the shell without the fear of being misunderstood.