s3fs-fuse and AWS Elastic Beanstalk

If you’re using WordPress or another web application that utilizes uploads in a load balanced and autoscaled environment, you know what a pain it can be to manage the uploads so they don’t disappear when an instance shuts down. Likewise, you need to make sure that every instance would have to have a copy of the resources because a request could go to any of them.

We use WordPress, and use media uploads quite a bit. We’ve been looking for a nice solution to the problem but until recently we didn’t have one. As we recently changed our autoscaling configuration and we recycle instances daily now, we found ourselves needing a better solution to this problem, and quickly.

The obvious choice for centralized resource storage is AWS S3. The unobvious choice is how to go about it. We tried using a few different WordPress extensions to do the task, however all of them had faults in one way or another that make them not well suited for a load balanced and autoscaled environment. Then we stumbled upon s3fs-fuse. In short, it’s a FUSE filesystem extension to mount an S3 bucket as a local folder on the file system. This seemed like the perfect solution. I checked into some other S3 File System FUSE extensions and this one seemed to be the most mature and actively maintained, despite it’s high open issue count, some of which are more questions about the product than actual issues.

Fortunately, getting s3fs-fuse setup to work with AWS Beanstalk is as easy as adding a .config file to your .ebextensions folder and adding your Access Keys to your Elastic Beanstalk configuration and that’s it! No changes to WordPress! Now, if you have the need to use CloudFront for your WordPress resources, then you’ll probably need an additional custom plugin (ie: you’ll have to write it, I haven’t found anything that will do this) that will inspect content of your posts and rewrite the URLs to a CloudFront url. But you can still use this s3fs-fuse extension as a backend to ensure files are uploaded to S3.

So without further delay, here is the config file. Simply add an Access Key to your Elastic Beanstalk configuration (one that has access to the bucket you’re referencing), replace the bracketed options with your specific details and drop this file into your .ebextensions folder and deploy it to your Beanstalk environment.

  • Brett

    This looks fantastic! Exactly what I need. However, I’m confused as to why you need a patch in step 01 of the commands. Why are you patching the cpp file?

    • Matt Zuba

      The default environment variables used by s3fs (AWSACCESSKEYID and AWSSECRETACCESSKEY) do not match the default environment variables that the Beanstalk stack provides. Since I’m already using those for other purposes, it seemed silly to have to create two more environment variables that would contain the same values and was easier to modify s3fs before compiling it.

      • Brett

        Ah, ok. Smart thinking!

      • Brett

        Do you simply add AWS_ACCESS_KEY_ID simply and AWS_SECRET_KEY to the ‘Environment Tags’ when setting up a new application in EB? Or would I do a new command in .ebextensions to do something like: export AWS_SECRET_KEY=MY_AWS_KEY?

        • Matt Zuba

          After you launch the application, you have to go back into the configuration to the environment section and the keys are the very first ones in the list in the bottom section.

          • Brett

            Ok, I knew about that route. The issue with that (I think at least) is that the mounting won’t happen until I go after the application has launched – which is after the mounting command is executed. Or am I missing something?

          • Matt Zuba

            Unfortunately that’s the best way I’ve found to do it. Last time I tried setting up Environment Tags in the setup process, I added around 8 variables, and none of them stuck, I had to re-enter them all on the configuration page.

            When launching new apps, that hasn’t really been an issue for us, since it normally takes a few iterations of deployments to verify that everything runs and deploys smoothly and our go-live day is typically the next day or a few days out. My very last step when I’m sure everything is configured properly is to completely restart the environment so it relaunches with a fresh EC2 instance to verify it all does startup as I expect, since this is the same behavior that would happen when it autoscales.

          • Matt Zuba

            And as far as putting them in an .ebextensions file, the downfall with that (for us anyway) is that it would end up going in version control, which would reduce some of the security aspects of it.

          • Brett

            Ah, yes, that makes sense now. Thanks so much for your pointers.

          • Matt Zuba

            Sure thing, good luck!

  • Jonay López

    Hello, First of all, thanks for this config setup. Im getting this error when I tried to deploy with your .config: s3fs could not determine how to establish security credentials
    How can I setup the security credentials?

    • Matt Zuba

      Hi Jonay!

      Did you add your AWS Access Key and Secret Key to your Elastic Beanstalk configuration in your AWS Control panel?

      • Jonay López

        Hi Matt, Thanks for the answer. No, sorry but I dont know how to do that. I’m in the config panel of my environment, I can see: Scaling, Instances, Software Configuration etc etc…

        • Matt Zuba

          It’s under the ‘Software Configuration’ section. You’ll see a sub-section there for Environment Variables, two of which are where you fill in an Access Key and Secret Key for a IAM user that has read/write access to the bucket.

          • Jonay López

            You are right! Thanks you very much!

  • Christian Reyer

    I get Error : Command failed on instance. Return code: 1 Output: s3fs: could not determine how to establish security credentials. Hook /opt/elasticbeanstalk/hooks/appdeploy/enact/02_mount_s3fs.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.

    • PeYoTlL

      Same here :

      [CMD-AppDeploy/AppDeployStage1/AppDeployEnactHook/02_mount_s3fs.sh] : Activity execution failed, because: s3fs: could not determine how to establish security credentials (ElasticBeanstalk::ExternalInvocationError)

      caused by: s3fs: could not determine how to establish security credentials (Executor::NonZeroExitStatus)

      Just if I try to mount a bucket subfolder. If I mount the root of the bucket the error appears later when trying to access the subfolder :

      cd: download/: Operation not permitted

      • Matt Zuba

        Have you added your AWS Access and Secret Keys in your Elastic Beanstalk environment configuration? That’s the only reason I can think of that it wouldn’t work.

        • Christian Reyer

          yes but i have add you to facebook

        • PeYoTlL

          Yes. I still haven’t found what the problem…

          Since this morning I have a new issue :
          02_mount_s3fs.sh] : Activity execution failed, because: s3fs: could not determine how to establish security credentials (ElasticBeanstalk::ExternalInvocationError)
          caused by: s3fs: could not determine how to establish security credentials (Executor::NonZeroExitStatus)

          I just added a second bucket mounting instruction in the 03_s3fs.config and even if I revert the change I still get that error each time I deploy.

          Any idea?

  • Lorel Ipsum

    Hi! We are getting the below error
    Command failed on instance. Return code: 1 Output: s3fs: could not determine how to establish security credentials.

    We have created environment variables in Beanstalk for AWS access and secret key. The IAM user do have the S3 full access.

    Could you please help why we are getting this error.

  • Juan Caicedo

    We originally deployed http://www.spanishdict.com/blog using this approach, but we were finding it lead to a lot of image uploads failing after we upgraded to ghost v0.7. I’d recommend using a custom ghost storage module instead, that ended up working really well for us an reduced the complexity of our deployment! We followed https://github.com/muzix/ghost-s3