Monday, 11 July 2016

Amazon EFS - sharing files painlessly

One of the most common problems, when it comes to scaling up, is how to share files between multiple web servers (nodes, instances). There are several options available but each of them has certain disadvantages.
After long time Amazon promoted EFS (Elastic file storage) from beta to available service (production-ready). Needles to say for now only in regions US East, US West, EU West. But EFS finally looks like a solution how to quickly, painlessly and securely share files.

Many devops are scratching their heads with question like this Shared File Systems between multiple AWS EC2 instances

 There were couple of options available how to achieve this:
  • Amazon S3
  • NFS server
  • Rsync
Amazon S3 is object storage, not file system, but there is a project called s3fs which allows you to mount S3 bucket as a volume. But there are some limitations and also higher latency might be a problem.

Usage of NFS is proved and reliable option, but initial setup takes a time and requires quite a few steps - see How To Set Up an NFS Mount on Ubuntu.

Rsync or to be precise Lsyncd, which can be described as scheduled Rsync, is effective way how to synchronise files across fleet of servers and store them on EBS (Elastic Block Store). Problem is that there needs to be one server which is master, from where files are being pulled and where are uploaded.

Amazon response to this is EFS. EFS is easy to setup, you pay only for used space per GB and works as NFSv4 volume with low latency.

Usage is really easy, in AWS console select "Elastic file system system" service.

Follow the wizard, it's straight forward. Only what you should remember is security group, this is how you can restrict access to EFS. What is best is to create own security group for instances accessing EFS. EFS uses NFSv4 which works over TCP and requires only port 2049 to be open.

On your EC2 instances you need to mount EFS. As AWS documentation suggest you need to install NFS client libraries :
  • On an Amazon Linux, Red Hat Enterprise Linux, or SuSE Linux instance:
    sudo yum install -y nfs-utils
  • On an Ubuntu instance:
    sudo apt-get install nfs-common
Mounting is straight forward:
sudo mount -t nfs4 -o nfsvers=4.1 XXXXDNS:/ /where/to/mount

where XXXXDNS is DNS name matching availability zone which you can find under DNS in EFS console, will look similar to  something 

To have EFS mounted after reboot, don't forget to add record to /etc/fstab which will have syntax:
XXXXDNS:/ /where/to/mount nfs defaults,vers=4.1 0 0

Then by running df command you can verify mounted volume availability



Post a Comment