Day 17 covered S3 and EBS — object storage and block storage for a single instance. Today we round out storage with four services that each solve a specific gap those two leave open: sharing storage across instances, moving huge volumes of data physically, archiving cheaply, and bridging on-premises storage with the cloud.

Amazon EFS (Elastic File System)

EBS has one hard limitation: a volume attaches to only one EC2 instance at a time. If you need multiple instances to read and write the same shared storage — a common web content directory, shared application state — EBS simply can't do it. That's exactly the gap EFS fills.

EFS is a shared file system that multiple EC2 instances can mount and access simultaneously, even across different Availability Zones. It works on the NFS (Network File System) protocol — the same standard on-premises teams have used for decades to share a file system across multiple laptops or servers from one central store. In AWS, EFS effectively replaces that on-premises file server, fully managed by AWS.

A few defining characteristics: EFS is file-based storage, unlike EBS's block-based model or S3's object-based model. It's only for Linux EC2 instances — Windows instances use a separate service (FSx) instead. It offers unlimited storage and, unlike EBS, requires no pre-provisioning — it automatically grows or shrinks based on the data you actually put in it, so you're not guessing a size upfront. To use it, you mount EFS to a directory inside each EC2 instance, the same way you'd mount any file system. EFS is Regional, but can be replicated to other Regions if you need that.

Worth knowing beyond the basics: EFS offers two storage classes — Standard, for actively accessed files, and Infrequent Access (EFS-IA), which costs less for files you don't touch often, with Lifecycle Management able to move files between them automatically based on actual access patterns. There are also two throughput modes: Bursting (throughput scales with the amount of data stored, suitable for most workloads) and Provisioned (you set a fixed throughput independent of storage size, useful for latency-sensitive or bursty workloads that don't have much data stored yet). And if you're on Windows and need something similar, Amazon FSx is the equivalent family of managed file systems — FSx for Windows File Server for SMB-based Windows workloads, and FSx for Lustre for high-performance computing workloads that need very fast, parallel file access.

AWS Snow Family — Physical data transfer

Sometimes the amount of data you need to move between on-premises and AWS is simply too large to transfer efficiently over a network connection — even a fast one. Moving petabytes of data over the internet could take weeks or months. The Snow Family solves this with physical data transfer: AWS ships you a rugged, secure storage device, you load your data onto it locally, and ship it back to AWS, who then uploads it directly into S3 on their end (or, in reverse, loads AWS data onto the device and ships it to you).

Three devices, scaled for different volumes:

• Snowcone — up to 8TB, small and portable, can even be used as an edge compute device in the field
• Snowball Edge — up to 100TB, the workhorse of the family for large migrations
• Snowmobile — literally a shipping container on a truck, moving petabytes of data — used for truly massive migrations, like an entire data center's worth of storage

All three devices are encrypted (256-bit) during transit for security, and they're used for both directions: migrating existing data into AWS, and in some cases, exporting data out of AWS back to a physical device.

Amazon S3 Glacier

Glacier is S3's archival counterpart — built specifically for data you need to retain but rarely, if ever, access: compliance records, old backups, historical logs. It's significantly cheaper than S3 Standard, in exchange for slower retrieval times.

The structure mirrors S3's bucket/object model but with different terminology: a vault is a container for archives (think of it like a bucket, but for archived data, often literally zip files), and an archive is the actual stored data — a single archive can be up to 40TB, and you can store an unlimited number of them. You can create up to 1,000 vaults per account.

Retrieval isn't instant, which is the trade-off for the lower cost, and Glacier actually offers a few retrieval speed tiers: Expedited (minutes, at a premium cost), Standard (a few hours), and Bulk (up to half a day or more, the cheapest option for retrieving large amounts of archived data at once). There's also Glacier Deep Archive, an even cheaper tier for data you essentially never expect to touch, with retrieval times measured in hours rather than minutes.

AWS Storage Gateway

Storage Gateway is AWS's answer to a very specific hybrid problem: what if you want to keep using local, on-premises storage for day-to-day work, but back it with the cloud instead of (or alongside) physical infrastructure? It's a hybrid storage service — a virtual appliance you run on-premises that connects your local environment to AWS storage services (S3, EBS, Glacier) behind the scenes.

Instead of company laptops writing directly to local private storage with no cloud tie-in, Storage Gateway sits between them and AWS: you get a local cache for low-latency access to frequently used data, while the bulk of your data actually lives durably in the cloud. It comes in a few flavors depending on what you're replacing: File Gateway presents cloud storage as a standard file share (backed by S3), Volume Gateway presents it as iSCSI block storage volumes (backed by EBS/S3, in either cached or fully-stored modes), and Tape Gateway presents it as a virtual tape library for organizations still running tape-based backup software, quietly backing those "tapes" with S3 and Glacier instead of physical tape.

Which one do you actually reach for?

Quick Recap Questions

1. Why can't EBS handle a scenario where 5 EC2 instances all need to read and write the same shared files?
2. Why would a company ship a physical Snowball device instead of just transferring data over the internet?
3. What's the trade-off Glacier makes to be cheaper than S3 Standard?
4. What problem does Storage Gateway solve that pure cloud storage doesn't?

Where to read & follow

• Hashnode: https://sr-palatasingh.hashnode.dev/series/aws-devops-blog
• GitHub: https://github.com/sr-palatasingh/AWS-DevOps-Blog/tree/main/posts
• LinkedIn: https://www.linkedin.com/in/soumyaranjan-palatasingh/

Coming up next

aws #devops #cloudcomputing #learning