Unlimited Storage with WP Engine’s Proprietary LargeFS
Recently, we snuck something new to our pricing page and many of you began to take notice on Twitter and in emails. What we snuck in is a pretty sweet feature called LargeFSTM Storage. LargeFS is a new feature that we developed to store and transfer huge amounts of data, and integrate it into the standard WordPress experience.
We built LargeFS using Amazon S3, the industry-standard mechanism for storing and serving any number of files of any size, from 10 million icons to a million movies. It’s inexpensive and bills per-use. And because S3 is already redundant, it also solves the issue of backup. S3 really is the perfect tool for storing and serving large files or large quantities of files.
Basically, LargeFS is a way to expand your server storage from your normal server onto S3, where storage is practically infinite.
We had to spend some time coding away to solve the issue of integrating WordPress with S3. We wanted to get everything just right.
Historically it was challenging to cleanly integrate the standard WordPress UI with the scale of S3. For example, there were often problems uploading images through the media uploader and also retaining support for other plugins and themes which depended on those files being on the local filesystem.
LargeFS retains the advantages of S3 while mitigating the downsides, in many respects simply because it’s running at WP Engine. Just like the rest of our infrastructure, LargeFS is fully managed by WP Engine — all you have to do is create an S3 account and set us up with credentials to read and write in it.
How LargeFS works:
- We transparently move content off the server’s filesystem onto S3. This happens during low-traffic periods and is fully automated.
- We move files only once they’re sufficiently old, so it doesn’t interfere with other server operations and plugins.
- Our fast, front-end caching load balancers understand which paths are controlled by LargeFS, and automatically serve content either from the server’s filesystem or from S3, depending on where the file is.
- Uploading new files or modifying files are available instantly; there’s no lag and no caching delay.
- Larger files are 301’ed directly to S3 for better throughput and scalability under traffic loads, and also those URLs in HTML are automatically rewritten to go straight to S3, so under normal conditions this is as fast as possible for the end user.
- Smaller files which are accessed frequently are automatically cached by our caching load-balancers and will also be placed on our CDN automatically, so popular content is as fast as possible and doesn’t have any of the speed issues with S3.
- All of the above is tunable for specific customers, and even for specific paths within an installation.
Unlimited Means Unlimited Again
What this means for you is that any hosting plan at WP Engine literally has unlimited space. “Unlimited” has become a hosting buzzword that doesn’t mean anything because in many cases, and on many hosts, “unlimited” simply wasn’t. There was a file limit to your plan and you hoped you never ran into it. With LargeFS, we’re “re-claiming” the word unlimited, and adding yet one more feature to our already industry-leading Managed WordPress Hosting Platform.
Another great addition Austin..
I’m keeping an eye on your hosting plans and hoping to grab sometime when your price falls or an amazing offer comes in…
Hey Harsh!
Let me know if you would like to have a conversation about our hosting packages, I’m here to help! There are a lot of benefits you could be taking advantage of by hosting with WP Engine…let me know if you’d be open to discussing.
Sergio
I am developing a site for a (small) tv channel.
The requiremens would be:
* Storage for (almost) unlimited data.
* Upload files up to 1GB, so the upload max size should be much bigger than it is right now 50 MB).
* A (very) easy way to upload these media files, and then insert them in the site’s page/posts. The usual wordpress process to upload/insert media files would be great. If an external tool is needed it should be the easiest possible.
I have already signed for Amazon s3 and installed the plugin http://wordpress.org/extend/plugins/tantan-s3/ recommended by you guys. I also see you have this new tool which seems pretty good for my purposes.
However, I don’t seem to find the proper way to connect all these tools and get the three requirements above (actually I have no idea how to set this LargeFS tool).
Anyone would give a hand on that?
For future articles my suggestion would be a good one on how to approach this situation (I think it will be useful for a lot of people).
By the way, I am a WPEngine customer, glad with the service you give.
Thanks!
I don’t think it explains anywhere how to actually get this up and running?
Thanks for your question Joshua!
To set it up you need to create a S3 account and provide us with the credentials.
This is what we normally ask for from customers-
Access key:
Secret key:
Bucket Name:
Please specify the file path that needs to be offloading via LargeFS to the S3 bucket, and ensure the bucket is generated for the US region.
Hope that helps. If you have any questions, feel free to contact our Support Team—you can open a live chat or support ticket in our User Portal.
– Kirby
This is really one of the best features you guys have with WP Engine! I have compared your competitors and unfortunately nobody else was offering something as easy to use and scalable as you. Well done and keep up the good work! 🙂
Large is a new feature that we developed to store and transfer huge amounts of data.
how i will upload on my wordpress site
I was excited to find this gem today! Just what we needed for large video files.
When will this feature apply to copies of installs? I had to jump through a number of hoops because a large portion of my content in wp-content just didn’t exist there anymore (it’s been moved to S3). So my new instance appeared heavily broken, and there was no clear way to reconcile the missing content without a lengthy chat w/ support (thank you for the help, by the way). It was lengthy because the support tech had to rediscover what was wrong prior to being able to help.
No others provide features like you do . Great work it adds value.
This is awesome and a great option to upgrade to even for those with two or three installs.