Fargate and EFS

One of the awesome features that we’ve been looking out for is native support for EFS in AWS Elastic Container Service (ECS) and specifically the serverless offering of Fargate. This was announced early April 2020 and it opens up a whole lot of possibilities for those containerised applications requiring a shared filesystem.

While this was achievable with a bit more code and some extra work in ECS, having native support means less moving parts, simpler code, easier to manage and just a better overall solution.

Before we move onto a working example lets have a quick overview of these services.

And finally some key points and traps to be aware of:

  • Platform version of 1.4.0 of Fargate is required.
  • LATEST tag defaults to 1.3.0 so you have to make sure you use version 1.4.0
  • LATEST tag will be updated to point to 1.4.0 in May 2020 according to AWS.

I’ve based my example from this AWS tutorial.

We’ll be using the aws cli here as CloudFormation support is still a work in progress. I’ve put together a script to orchestrate all these commands so if you if you just want to quickly get a working version scroll to the bottom and see the details in the repo.

Firstly we need to set up some environment variables as we’ll be using these in our commands. We need variables for VPC_ID, PUBLIC_SUBNET_ID and PRIVATE_SUBNET_ID. We’ll be creating our Fargate container in a public subnet while our EFS will live in a private subnet.

<span class="hljs-keyword">export</span> VPC_ID=<your vpc <span class="hljs-keyword">id</span>>
<span class="hljs-keyword">export</span> PUBLIC_SUBNET_ID=<your public subnet <span class="hljs-keyword">id</span>>
<span class="hljs-keyword">export</span> PRIVATE_SUBNET_ID=<your private subnet <span class="hljs-keyword">id</span>>

Lets start by creating our security group which we’ll use on both the container and EFS.

aws ec2 create-security-<span class="hljs-keyword">group</span> --<span class="hljs-keyword">group</span>-name fargate-efs-sg \
  --description <span class="hljs-string">"Security group for fargate efs example"</span> --vpc-id <span class="hljs-string">"$VPC_ID"</span>

Add some ingress rules to our security group. Please substitute or set the variable SG_ID to the security group created above. eg. sg-123456. As our container runs in the public subnet I am locking down access to only my public ip with “curl -s checkip.amazonaws.com”. We also need to open up 2049 so that the container can talk to EFS.

<span class="hljs-preprocessor"># Authorise your ip only.</span>
aws ec2 authorize-security-<span class="hljs-keyword">group</span>-ingress \
  --<span class="hljs-keyword">group</span>-id <span class="hljs-string">"$SG_ID"</span> \
  --protocol tcp \
  --port <span class="hljs-number">80</span> \
  --cidr <span class="hljs-string">"$(curl -s checkip.amazonaws.com)/32"</span>

<span class="hljs-preprocessor"># Ingress to efs via sg only</span>
aws ec2 authorize-security-<span class="hljs-keyword">group</span>-ingress \
  --<span class="hljs-keyword">group</span>-id <span class="hljs-string">"$SG_ID"</span> \
  --protocol tcp \
  --port <span class="hljs-number">2049</span> \
  --source-<span class="hljs-keyword">group</span> <span class="hljs-string">"$SG_ID"</span>

Now we create our EFS filesystem.

aws efs create-<span class="hljs-keyword">file</span>-<span class="hljs-built_in">system</span>

Once that is done, it takes a minute or so to be available we need to create mount targets. Grab the filesytem id from the above command and substitute for variable FILE_SYSTEM_ID.

<span class="hljs-comment"># Create a mount target in a private subnet.</span>
aws efs create-mount-target \
--file-<span class="hljs-keyword">system</span>-id <span class="hljs-string">"<span class="hljs-variable">$FILE_SYSTEM_ID</span>"</span> \
--subnet-id <span class="hljs-string">"<span class="hljs-variable">$PRIVATE_SUBNET_ID</span>"</span> \
--security-group <span class="hljs-string">"<span class="hljs-variable">$SG_ID</span>"</span>

We can now create our ECS Cluster.

aws ecs <span class="hljs-operator"><span class="hljs-keyword">create</span>-cluster <span class="hljs-comment">--cluster-name fargate-cluster</span>
</span>

Now we need to create our task definition. Copy the following code and paste into a file called fargate-task.json. Substitute the FILE_SYSTEM_ID variable in the file for your actual filesystem id.

The task is a simple Apache container which has an index.html page containing some details of efs with “df -h /mount/efs > /usr/local/apache2/htdocs/index.html”.

{
    "<span class="hljs-attribute">family</span>": <span class="hljs-value"><span class="hljs-string">"fargate-efs"</span></span>,
    "<span class="hljs-attribute">networkMode</span>": <span class="hljs-value"><span class="hljs-string">"awsvpc"</span></span>,
    "<span class="hljs-attribute">containerDefinitions</span>": <span class="hljs-value">[
        {
            "<span class="hljs-attribute">name</span>": <span class="hljs-string">"fargate-app"</span>,
            "<span class="hljs-attribute">image</span>": <span class="hljs-string">"httpd:2.4"</span>,
            "<span class="hljs-attribute">portMappings</span>": [
                {
                    "<span class="hljs-attribute">containerPort</span>": <span class="hljs-number">80</span>,
                    "<span class="hljs-attribute">hostPort</span>": <span class="hljs-number">80</span>,
                    "<span class="hljs-attribute">protocol</span>": <span class="hljs-string">"tcp"</span>
                }
            ],
            "<span class="hljs-attribute">essential</span>": <span class="hljs-literal">true</span>,
            "<span class="hljs-attribute">entryPoint</span>": [
                <span class="hljs-string">"sh"</span>, <span class="hljs-string">"-c"</span>
            ],
            "<span class="hljs-attribute">command</span>": [
                <span class="hljs-string">"/bin/sh -c \"df -h /mount/efs > /usr/local/apache2/htdocs/index.html && httpd-foreground\""</span>
            ],
            "<span class="hljs-attribute">mountPoints</span>": [
                {
                    "<span class="hljs-attribute">sourceVolume</span>": <span class="hljs-string">"fargate-efs"</span>,
                    "<span class="hljs-attribute">containerPath</span>": <span class="hljs-string">"/mount/efs"</span>,
                    "<span class="hljs-attribute">readOnly</span>": <span class="hljs-literal">false</span>
                }
            ]
        }
    ]</span>,
    "<span class="hljs-attribute">volumes</span>": <span class="hljs-value">[{
      "<span class="hljs-attribute">name</span>": <span class="hljs-string">"fargate-efs"</span>,
      "<span class="hljs-attribute">efsVolumeConfiguration</span>": {
         "<span class="hljs-attribute">fileSystemId</span>": <span class="hljs-string">"$FILE_SYSTEM_ID"</span>,
         "<span class="hljs-attribute">rootDirectory</span>": <span class="hljs-string">"/"</span>
      }
    }]</span>,
    "<span class="hljs-attribute">requiresCompatibilities</span>": <span class="hljs-value">[
        <span class="hljs-string">"FARGATE"</span>
    ]</span>,
    "<span class="hljs-attribute">cpu</span>": <span class="hljs-value"><span class="hljs-string">"256"</span></span>,
    "<span class="hljs-attribute">memory</span>": <span class="hljs-value"><span class="hljs-string">"512"</span>
</span>}

Once we have the file saved with the correct filesystem id we can now register the task definition with.

<span class="hljs-comment"># Register the task definition</span>
<span class="hljs-title">aws</span> ecs register-task-definition --requires-compatibilities FARGATE \
  --cli-input-json <span class="hljs-url">file://fargate-task.json</span>

Now we can create the service. You will need to substitute your TASK_ARN from above along with PUBLIC_SUBNET_ID and SG_ID.

# <span class="hljs-operator"><span class="hljs-keyword">Create</span> the service.
aws ecs <span class="hljs-keyword">create</span>-service <span class="hljs-comment">--cluster fargate-cluster \</span>
  <span class="hljs-comment">--service-name fargate-service --task-definition "$TASK_ARN" \</span>
  <span class="hljs-comment">--desired-count 1 --launch-type "FARGATE" \</span>
  <span class="hljs-comment">--network-configuration \</span>
    <span class="hljs-string">"awsvpcConfiguration={subnets=[$PUBLIC_SUBNET_ID],securityGroups=[$SG_ID],assignPublicIp=ENABLED}"</span> \
  <span class="hljs-comment">--platform-version 1.4.0 \</span>
  <span class="hljs-comment">--tags key=Name,value=fargate-efs \</span>
  <span class="hljs-comment">--enable-ecs-managed-tags</span>
</span>

The service takes a minute or two to startup so after a little wait we need to get the containers public ip. You can log on to the console and get it from there or run the following script.

aws ec2 describe-network-interfaces \
    --filters Name=<span class="hljs-keyword">ta</span><span class="hljs-variable">g:</span><span class="hljs-string">"aws:ecs:serviceName"</span>,Values=fargate-service

Once we have the public ip we can now test to see if it all works. Substitute PUBLIC_IP from the value above. We put in some retries for our curl command as the container might not be completely running.

<span class="hljs-title">curl</span> --retry-connrefused --retry <span class="hljs-number">10</span> <span class="hljs-url">http://<span class="hljs-variable">$PUBLIC_IP</span></span>

And finally we should see some output like.

<span class="hljs-title">Filesystem</span>                                      Size  Used Avail Use% Mounted <span class="hljs-built_in">on</span>
fs-abc123.efs.ap-southeast-<span class="hljs-number">2</span>.amazonaws.<span class="hljs-url">com:/</span>    <span class="hljs-number">8</span>.0E     <span class="hljs-number">0</span>  <span class="hljs-number">8</span>.0E   <span class="hljs-number">0</span>% /mount/efs

You can see most of the code in our Github repo. Its been orchestrated into a run.sh script.

Finally that’s it. EFS inside Fargate. Pretty easy and an exciting feature that will be another great option for applications in AWS.