elasticsearch - Issues with regex in Kibana

Question

I am having a hard time using a regex pattern inside Kibana/Elasticsearch version 6.5.4. The field I am searching for has the following mapping:

"field": {
          "type": "text",
          "analyzer": "custom_analyzer"
        },

Regex searches in this field return several hits when requested straight to elasticsearch:

GET /my_index/_search
{
    "query": {
        "regexp":{
            "field": "abc[0-9]{4}"
        }
    }
}

On the other hand, in Kibana's discover/dashboard pages all queries below return empty:

original query - field:/abc[0-9]{4}/

scaped query - field:/abc[0-9]{4}/

desperate query - field:/.*/

Inspecting the request done by kibana to elasticsearch reveals the following query:

  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "field:/abc[0-9]{4}/",
            "analyze_wildcard": true,
            "default_field": "*"
          }
        }

I expected kibana to understand the double forward slash syntax /my_query/ and make a ′regexp query′ instead of a ′query_string′. I have tried this with both query languages: "lucene", "kuery" and with the optional "experimental query features" enabled/disabled.

Digging further I found this old issue which says that elastic only runs regex into the now deprecated _all field. If this still holds true I am not sure how regex work in kibana/elastic 6.X.

What am I missing? Any help in clarifying the conditions to use regex in Kibana would be much appreciated

All other stack questions in this subject are either old or were related to syntax issues and/or lack of understanding of how the analyzer deals with whitespaces and did not provide me any help.

Answer 1

So I don't exactly have the answer on how to make Lucene work with Regexp search in Kibana. But I figured out a way to do this in Kibana.

Solution is to use Filter with custom DSL

Here is an example of what to put in Query JSON -

{
    "regexp": {
        "req.url.keyword": "/question/[0-9]+/answer"
    }
}

Example Url I have in my data - /questions/432142/answer

Additional to this, you can write more filters using Kibana search (Lucene syntax)

It does the appropriate search, no escaping issue or any such thing.

Hope it helps.