web crawler - Wildcards in robots.txt

Question

If in WordPress website I have categories in this order:

-Parent
--Child
---Subchild

I have permalinks set to: %category%/%postname%

Let use an example. I create post with post name "Sport game". It's tag is sport-game. It's full url is: domain.com/parent/child/subchild/sport-game Why I use this kind of permalinks is exactly to block some content easier in robots.txt.

And now this is the part I have question for. In robots.txt:

User-agent: Googlebot
Disallow: /parent/*
Disallow: /parent/*/*
Disallow: /parent/*/*/*

Disallow: /parent/* Is meaning of this rule that it's blocking domain.com/parent/child but not domain.com/parent/child/subchild and not domain.com/parent/?

Disallow: /parent/*/*/* Is meaning of this that it's blocking domain.com/parent/child/subchild/, that it's blocking only subchild, not child, not parent, and not posts under subchild?

Answer 1

Note that the * wildcard in Disallow is not part of the original robots.txt specification. Some parsers support it, but as there is no specification, they might all handle it differently.

As you seem to be interested in Googlebot, have a look at Google’s robots.txt documentation.

In the examples it becomes clear that * means

any string

"Any string" may, of course, also contain /.

So your first line Disallow: /parent/* should block every URL whose path starts with /parent/, including path segments separated by slashes.

Note that this would be the same as Disallow: /parent/ in the original robots.txt specification, which also blocks any URL whose paths starts with /parent/, for example:

• http://example.com/parent/
• http://example.com/parent/foo
• http://example.com/parent/foo.html
• http://example.com/parent/foo/bar
• http://example.com/parent/foo/bar/
• http://example.com/parent/foo/bar/foo.html