amazon web services - How can I use persisted storage for elasticsearch in AWS?

Question

I am deploying my container to AWS EKS which is running elasticsearch. I need to pick up a storage service to store all the data. I searched a few articles and people mentioned EBS. Is this a good storage for data source like elasticsearch? Is there any better solution?

I know AWS provides hosted elasticsearch solution but I don't want to stick on AWS. By using EKS, it is easy for me to migrate to other cloud later.

question from:https://stackoverflow.com/questions/65626692/how-can-i-use-persisted-storage-for-elasticsearch-in-aws

Answer 1

AWS provides a managed Elasticsearch service called Amazon Elasticsearch Service which is based on the OpenDistro Elasticsearch. Which means you are not tied into AWS. AWS just manages the cluster on your behalf. But because it is an AWS managed service it supports additional services such as Amazon CloudWatch and Amazon S3. If you are using Amazon Elasticsearch service you can store data on S3. See the blog post "Use Amazon S3 to Store a Single Amazon Elasticsearch Service Index". Amazon Elasticsearch service has a new cost effective way to store large amounts of read-only data by moving the data from EBS to S3. This new service is known as Ultrawarm. More information here.

To answer your question.

What do you want to store? As there are a few options. But as you mentioned EBS (Amazon Elastic Block Storage). I assume you want to store the Elastic Search data. In this case I would recommend you look at Amazon Elastic File System (EFS).

EBS is block storage bound to an EC2 instance. In your case the EC2 instance running as the EKS node. Generally EBS is bound to one EC2 instance only. While it does support multiple EC2 instances too with Amazon EBS Multi-Attach. But think of EBS as a hard drive.

On the other hand EFS is network storage that is scalable and can be linked to multiple containers (EKS, ECS, and Fargate), EC2 instances, and Lambda functions.

The following blog post explains the new feature - AWS Fargate for Amazon EKS now supports EFS. In summary AWS released a driver called the Amazon EFS CSI driver. The documentation is here. You can use this driver to mount and EFS volume within your container. Because it is network storage, you can share that volume with other instances.

The EKS Workshop has a section on how to link EFS to EKS. You can follow the workshop here.

Alternatively please visit the support response for "How do I use persistent storage in Amazon EKS"