FSx for Lustre for persistent storage solution provided by Hitachi Systems
By HSI | April 6, 2023
Challenge
Few problems that our customer faced having a on-premises high performance Lustre-based persistent file system solution are.
- Complex Backup solution
- High Data Backup Cost
The customer wanted a solution to store massive amount of data and highly reliable storage to persist data.
FSx for Lustre persistent file system
FSx for Lustre persistent file system provides highly available and durable storage for workloads that run for extended periods, or indefinitely. FSx for Lustre persistent file system is considered for processing heavy workloads that need durable and highly available storage.
The FSx for Lustre persistent file system can we used in 4 use cases
- SAS Grid
- High Performance Computing
- Persistence storage for containers
- Stateful Amazon EC2 Spot
What is an Open-source Lustre file system?
The open-source Lustre file system is designed for applications that require fast storage—where you want your storage to keep up with your compute. FSx for Lustre eliminates the traditional complexity of setting up and managing Lustre file systems, enabling to spin up and run a battle-tested high-performance file system in minutes. It also provides multiple deployment options to optimize the cost according to the needs.
Various storage options that are designed for various data processing needs are available with Amazon FSx for Lustre, including solid state drives (SSDs) and hard disk drives (HDDs):
SSD storage options – Pick one of the SSD storage options for low-latency, IOPS-demanding applications that often include quick, small-file operations.
HDD storage options – Pick one of the HDD storage options for throughput-demanding workloads that frequently include large, sequential file operations.
Solution using Persistent Deployment Type
For longer-term workloads and storage, as a part of solution we created persistent file systems which are used for long term storage and throughput-focused workloads that run on for long period so that data automatically gets duplicated inside the same availability Zone in which the file system is housed, and the file servers become highly available. We created data volumes which were associated and duplicated separately from the attached file server.
If a file server becomes unavailable on a persistent file system, it is automatically replaced within the minutes of failure. During that time, the client requests for data on that server transparently retry and eventually succeed after the file server is replaced. The data on the persistent file systems is replicated on the disk and any failed risks are automatically and transparently replaced.
We recommended the parallelizing of your workload. Parallelizing workloads by increasing the number of threads per file system client enables to drive higher throughput to the file system because FSx for Lustre bundles writes to disks. If clients running the workload are fully utilized, adding additional clients also enables you to drive higher throughput to the file system for the same reason.
Architecture
The below diagram shows the architecture for an Amazon FSx for Lustre persistent file system, with replicated, highly available file servers and data volumes within a single Availability Zone.
Benefits
- Better Redundancy: If a file server becomes unavailable on a persistent file system, it is automatically replaced within few minutes of failure.
- Cost Optimization: A wide range of storage options and features were offered from Amazon FSx for Lustre to optimize the price and performance. Storage on Solid-State Drives (SSDs) is optimized for latency-sensitive applications, whereas storage on Hard Disk Drives (HDDs) is intended for throughput-focused workloads.
- Security: It is highly secured because of the ability to encrypt the data
- Ease of Accessibility: Amazon FSx for Lustre provided the centralized user interface on the AWS console.