elasticsearch - Index life cycle policy

Question

I would like to create a Hot-warm policy , and the index should rollover when the index is 20Gb of size or max_age equal to 30days BUT, if the size condition occur before the age condition, the index should rollover but the data have to stay in the hot node till the max_age condition occur. and then the data should be in warm data for 5 months and then deleted.

Example : if after 15days the index is 20gb, the index rollover but don't leave the hot data node till it's age is 30 days, so should stay other 15 days in the hot data before if goes into the warm data (Hope I explain it well :sweat_smile:)

SO I created this policy

PUT _ilm/policy/hot-warm-cold-delete-6months-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size":"20gb",
            "max_age":"30d"
          },
          "set_priority": {
            "priority": 50
          }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1
          },
          "shrink": {
            "number_of_shards": 1
          },
          "allocate": {
            "require": {
              "data": "warm"
            }
          },
          "set_priority": {
            "priority": 25
          }
        }
      },
      "delete": {
        "min_age": "150d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

but if I understand well, this means that the index will be sent to the warm data after 30 days of the rollover and not from the creation date, and it doesn't work as I want exactly

Could you tell me please if what I am trying to do is possible with ILM ?

Thanks for your help

Answer 1

You can work around the rollover date with index.lifecycle.origination_date:

If specified, this is the timestamp used to calculate the index age for its phase transitions. Use this setting if you create a new index that contains old data and want to use the original creation date to calculate the index age. Specified as a Unix epoch value.

You could either set that value manually or use index.lifecycle.parse_origination_date from the index name:

Set to true to parse the origination date from the index name. This origination date is used to calculate the index age for its phase transitions. The index name must match the pattern ^.*-{date_format}-\d+, where the date_format is yyyy.MM.dd and the trailing digits are optional. An index that was rolled over would normally match the full format, for example logs-2016.10.31-000002. If the index name doesn’t match the pattern, index creation fails.

So with the right index names, this should be doable (though it is IMO more of a workaround than a feature).