linux - Default field separator for awk

Question

Sorry for this stupid question, searched but not confident is the right answer is found, so the default separator is only space for awk?

Answer 1

Here's a pragmatic summary that applies to all major Awk implementations:

• GNU Awk (gawk) - the default awk in some Linux distros
• Mawk (mawk) - the default awk in some Linux distros (e.g., earlier versions of Ubuntu ^{crysman reports that version 19.04 now comes with GNU Awk - see his comment below.})
• BSD Awk - a.k.a. BWK Awk - the default awk on BSD-like platforms, including OSX

^{On Linux, awk -W version will tell you which implementation the default awk is.
BSD Awk only understands awk --version (which GNU Awk understands in addition to awk -W version).}

Recent versions of all these implementations follow the POSIX standard with respect to field separators^[1] (but not record separators).

Glossary:

• RS is the input-record separator, which describes how the input is broken into records:

  • The POSIX-mandated default value is a newline, also referred to as below; that is, input is broken into lines by default.
  • On awk's command line, RS can be specified as -v RS=<sep>.
  • POSIX restricts RS to a literal, single-character value, but GNU Awk and Mawk support multi-character values that may be extended regular expressions (BSD Awk does not support that).

• FS is the input-field separator, which describes how each record is split into fields; it may be an extended regular expression.

  • On awk's command line, FS can be specified as -F <sep> (or -v FS=<sep>).
  • The POSIX-mandated default value is formally a space (0x20), but that space is not literally interpreted as the (only) separator, but has special meaning; see below.

---

By default:

• any run of spaces and/or tabs and/or newlines is treated as a field separator
• with leading and trailing runs ignored.

^{The POSIX spec. uses the abstraction <blank> for spaces and tabs, which is true for all locales, but could comprise additional characters in specific locales - I don't know if any such locales exist.}

Note that with the default input-record separator (RS), , newlines typically do not enter the picture as field separators, because no record itself contains in that case.

Newlines as field separators do come into play, however:

• When RS is set to a value that results in records themselves containing instances (such as when RS is set to the empty string; see below).
• Generally, when the split() function is used to split a string into array elements without an explicit-field separator argument.
  • Even though the input records won't contain instances in case the default RS is in effect, the split() function when invoked without an explicit field-separator argument on a multi-line string from a different source (e.g., a variable passed via the -v option or as a pseudo-filename) always treats as a field separator.

---

Important NON-default considerations:

• Assigning the empty string to RS has special meaning: it reads the input in paragraph mode, meaning that the input is broken into records by runs of non-empty lines, with leading and trailing runs of empty lines ignored.

• When you assign anything other than a literal space to FS, the interpretation of FS changes fundamentally:

  • A single character or each character from a specified character set is recognized individually as a field separator - not runs of it, as with the default.
    • For instance, setting FS to [ ] - even though it effectively amounts to a single space - causes every individual space instance in each record to be treated as a field separator.
    • To recognize runs, the regex quantifier (duplication symbol) + must be used; e.g., []+ would recognize runs of tabs as a single separator.
  • Leading and trailing separators are NOT ignored, and, instead, separate empty fields.
  • Setting FS to the empty string means that each character of a record is its own field.
• As mandated by POSIX, if RS is set to the empty string (paragraph mode), newlines ( ) are also considered field separators, irrespective of the value of FS.

---

<sup>[1] Unfortunately, GNU Awk up to at least version 4.1.3 complies with an obsolete POSIX standard with respect to field separators when you use the option to enforce POSIX compliance, -P (--posix): with that option in effect and RS set to a non-empty value, newlines ( instances) are NOT recognized as field separators. The GNU Awk manual spells out the obsolete behavior (but neglects to mention that it doesn't apply when RS is set to the empty string). The POSIX standard changed in 2008 (see comments) to also consider newlines field separators when FS has its default value - as GNU Awk has always done without -P (--posix).
Here are 2 commands that verify the behavior described above:
* With -P in effect and RS set to the empty string, is still treated as a field separator:
gawk -P -F' ' -v RS='' '{ printf "<%s>, <%s> ", $1, $2 }' <<< $'a b'
* With -P in effect and a non-empty RS, is NOT treated as a field separator - this is the obsolete behavior:
gawk -P -F' ' -v RS='|' '{ printf "<%s>, <%s> ", $1, $2 }' <<< $'a b'
A fix is coming, according to the GNU Awk maintainers; expect it in version 4.2 (no time frame given).
(Tip of the hat to @JohnKugelman and @EdMorton for their help.)</sup>