regex - Unicode equivalents for w and  in Java regular expressions?

Question

Many modern regex implementations interpret the w character class shorthand as "any letter, digit, or connecting punctuation" (usually: underscore). That way, a regex like w+ matches words like hello, élève, GO?_432 or gefr??ig.

Unfortunately, Java doesn't. In Java, w is limited to [A-Za-z0-9_]. This makes matching words like those mentioned above difficult, among other problems.

It also appears that the word separator matches in places where it shouldn't.

What would be the correct equivalent of a .NET-like, Unicode-aware w or in Java? Which other shortcuts need "rewriting" to make them Unicode-aware?

Answer 1

Source code

The source code for the rewriting functions I discuss below is available here.

Update in Java 7

Sun’s updated Pattern class for JDK7 has a marvelous new flag, UNICODE_CHARACTER_CLASS, which makes everything work right again. It’s available as an embeddable (?U) for inside the pattern, so you can use it with the String class’s wrappers, too. It also sports corrected definitions for various other properties, too. It now tracks The Unicode Standard, in both RL1.2 and RL1.2a from UTS#18: Unicode Regular Expressions. This is an exciting and dramatic improvement, and the development team is to be commended for this important effort.

---

Java’s Regex Unicode Problems

The problem with Java regexes is that the Perl 1.0 charclass escapes — meaning w, , s, d and their complements — are not in Java extended to work with Unicode. Alone amongst these, enjoys certain extended semantics, but these map neither to w, nor to Unicode identifiers, nor to Unicode line-break properties.

Additionally, the POSIX properties in Java are accessed this way:

POSIX syntax    Java syntax

[[:Lower:]]     p{Lower}
[[:Upper:]]     p{Upper}
[[:ASCII:]]     p{ASCII}
[[:Alpha:]]     p{Alpha}
[[:Digit:]]     p{Digit}
[[:Alnum:]]     p{Alnum}
[[:Punct:]]     p{Punct}
[[:Graph:]]     p{Graph}
[[:Print:]]     p{Print}
[[:Blank:]]     p{Blank}
[[:Cntrl:]]     p{Cntrl}
[[:XDigit:]]    p{XDigit}
[[:Space:]]     p{Space}

This is a real mess, because it means that things like Alpha, Lower, and Space do not in Java map to the Unicode Alphabetic, Lowercase, or Whitespace properties. This is exceeedingly annoying. Java’s Unicode property support is strictly antemillennial, by which I mean it supports no Unicode property that has come out in the last decade.

Not being able to talk about whitespace properly is super-annoying. Consider the following table. For each of those code points, there is both a J-results column for Java and a P-results column for Perl or any other PCRE-based regex engine:

             Regex    001A    0085    00A0    2029
                      J  P    J  P    J  P    J  P
                s    1  1    0  1    0  1    0  1
               pZ    0  0    0  0    1  1    1  1
            p{Zs}    0  0    0  0    1  1    0  0
         p{Space}    1  1    0  1    0  1    0  1
         p{Blank}    0  0    0  0    0  1    0  0
    p{Whitespace}    -  1    -  1    -  1    -  1
p{javaWhitespace}    1  -    0  -    0  -    1  -
 p{javaSpaceChar}    0  -    0  -    1  -    1  -

See that?

Virtually every one of those Java white space results is ? ?w?r?o?n?g?? according to Unicode. It’s a really big problem. Java is just messed up, giving answers that are “wrong” according to existing practice and also according to Unicode. Plus Java doesn’t even give you access to the real Unicode properties! In fact, Java does not support any property that corresponds to Unicode whitespace.

---

The Solution to All Those Problems, and More

To deal with this and many other related problems, yesterday I wrote a Java function to rewrite a pattern string that rewrites these 14 charclass escapes:

w W s S v V h H d D  B X R

by replacing them with things that actually work to match Unicode in a predictable and consistent fashion. It’s only an alpha prototype from a single hack session, but it is completely functional.

The short story is that my code rewrites those 14 as follows:

s => [u0009-u000Du0020u0085u00A0u1680u180Eu2000-u200Au2028u2029u202Fu205Fu3000]
S => [^u0009-u000Du0020u0085u00A0u1680u180Eu2000-u200Au2028u2029u202Fu205Fu3000]

v => [u000A-u000Du0085u2028u2029]
V => [^u000A-u000Du0085u2028u2029]

h => [u0009u0020u00A0u1680u180Eu2000-u200Au202Fu205Fu3000]
H => [^u0009u0020u00A0u1680u180Eu2000u2001-u200Au202Fu205Fu3000]

w => [pLpMp{Nd}p{Nl}p{Pc}[p{InEnclosedAlphanumerics}&&p{So}]]
W => [^pLpMp{Nd}p{Nl}p{Pc}[p{InEnclosedAlphanumerics}&&p{So}]]

 => (?:(?<=[pLpMp{Nd}p{Nl}p{Pc}[p{InEnclosedAlphanumerics}&&p{So}]])(?![pLpMp{Nd}p{Nl}p{Pc}[p{InEnclosedAlphanumerics}&&p{So}]])|(?<![pLpMp{Nd}p{Nl}p{Pc}[p{InEnclosedAlphanumerics}&&p{So}]])(?=[pLpMp{Nd}p{Nl}p{Pc}[p{InEnclosedAlphanumerics}&&p{So}]]))
B => (?:(?<=[pLpMp{Nd}p{Nl}p{Pc}[p{InEnclosedAlphanumerics}&&p{So}]])(?=[pLpMp{Nd}p{Nl}p{Pc}[p{InEnclosedAlphanumerics}&&p{So}]])|(?<![pLpMp{Nd}p{Nl}p{Pc}[p{InEnclosedAlphanumerics}&&p{So}]])(?![pLpMp{Nd}p{Nl}p{Pc}[p{InEnclosedAlphanumerics}&&p{So}]]))

d => p{Nd}
D => P{Nd}

R => (?:(?>u000Du000A)|[u000Au000Bu000Cu000Du0085u2028u2029])

X => (?>PMpM*)

Some things to consider...

• That uses for its X definition what Unicode now refers to as a legacy grapheme cluster, not an extended grapheme cluster, as the latter is rather more complicated. Perl itself now uses the fancier version, but the old version is still perfectly workable for the most common situations. EDIT: See addendum at bottom.

• What to do about d depends on your intent, but the default is the Uniode definition. I can see people not always wanting p{Nd}, but sometimes either [0-9] or pN.

• The two boundary definitions, and B, are specifically written to use the w definition.

• That w definition is overly broad, because it grabs the parenned letters not just the circled ones. The Unicode Other_Alphabetic property isn’t available until JDK7, so that’s the best you can do.

---

Exploring Boundaries

Boundaries have been a problem ever since Larry Wall first coined the and B syntax for talking about them for Perl 1.0 back in 1987. The key to understanding how and B both work is to dispel two pervasive myths about them:

1. They are only ever looking for w word characters, never for non-word characters.
2. They do not specifically look for the edge of the string.

A boundary means:

    IF does follow word
        THEN doesn't precede word
    ELSIF doesn't follow word
        THEN does precede word

And those are all defined perfectly straightforwardly as:

• follows word is (?<=w).
• precedes word is (?=w).
• doesn’t follow word is (?<!w).
• doesn’t precede word is (?!w).

Therefore, since IF-THEN is encoded as an and?ed-together AB in regexes, an or is X|Y, and because the and is higher in precedence than or, that is simply AB|CD. So every that means a boundary can be safely replaced with:

    (?:(?<=w)(?!w)|(?<!w)(?=w))

with the w defined in the appropriate way.

(You might think it strange that the A and C components are opposites. In a perfect world, you should be able to write that AB|D, but for a while I was chasing down mutual exclusion contradictions in Unicode properties — which I think I’ve taken care of, but I left the double condition in the boundary just in case. Plus this makes it more extensible if you get extra ideas later.)

For the B non-boundaries, the logic is:

    IF does follow word
        THEN does precede word
    ELSIF doesn't follow word
        THEN doesn't precede word

Allowing all instances of B to be replaced with:

    (?:(?<=w)(?=w)|(?<!w)(?!w))

This really is how and B behave. Equivalent patterns for them are

• using the ((IF)THEN|ELSE) construct is (?(?<=w)(?!w)|(?=w))
• B using the ((IF)THEN|ELSE) construct is (?(?=w)(?<=w)|(?<!w))

But the versions with just AB|CD are fine, especially if you lack conditional patterns in your regex language — like Java. ?

I’ve already verified the behaviour of the boundaries using all three equivalent definitions with a test suite that checks 110,385,408 matches per run, and which I've run on a dozen different data configurations according to:

     0 ..     7F    the ASCII range
    80 ..     FF    the non-ASCII Latin1 range
   100 ..   FFFF    the non-Latin1 BMP (Basic Multilingual Plane) range
 10000 .. 10FFFF    the non-BMP portion of Unicode (the "astral" planes)

However, people often want a different sort of boundary. They want something that is whitespace and edge-of-string aware:

• left edge as (?:(?<=^)|(?<=s))
• right edge as (?=$|s)

---

Fixing Java with Java

The code I posted in my other answer provides this and quite a few other conveniences. This includes definitions for natural-language words, dashes, hyphens, and apostrophes, plus a bit more.

It also allows you to specify Unicode characters in logical code points, not in idiotic UTF-16 surrogates. It’s hard to overstress how important that is! And that’s just for the string expansion.

For regex charclass substitution that makes the charclass in your Java regexes finally work on Unicode, and work correctly, grab the full source from here. You may do with it as you please, of course. If you make fixes to it, I’d love to hear of it, but you don’t have to. It’s pretty short. The guts of the main regex rewriting function is simple:

switch (code_point) {

    case 'b':  newstr.append(boundary);
               break; /* switch */
    case 'B':  newstr.append(not_boundary);
               break; /* switch */

    case 'd':  newstr.append(digits_charclass);
               break; /* switch */
    case 'D':  newstr.append(not_digits_charclass);
               break; /* switch */

    case 'h':  newstr.append(horizontal_whitespace_c