What happens when you mix massive data sets, regulations and GenAI?
Cold storage has breezed in to the Gartner® Hype Cycle for Storage Technologies, 20241 as an emerging trend due to the future implications of the unprecedented data growth we’re currently witnessing. This article will explore the importance of having a cold storage strategy and ways in which Tiger Technology and AWS Cloud can effectively support it.
What is Cold Storage?
Cold storage is a purpose-built system designed for the retention of inactive data.2 This is used to enable outcomes such as data resilience and redundancy, which are mission-critical priorities for organizations facing rising long-term retention mandates and regulatory compliance such as healthcare institutions and governmental agencies. Organizations in such sectors require long-term preservation of archive data rather than expensive, low-latency access to cold data sets.
Why is Cold Storage Important?
Data is expected to continue to grow at rates higher with each passing year. Research indicates that by 2025 global data creation is projected to grow to more than 180 zettabytes3 (that is 180 billion terabytes!). Further, the increase in the volume of unstructured data (think of any non-SQL database/file data such as videos, images, documents, application logs, etc.) has been particularly explosive, so much so that Gartner® end users report between 30% and 60% growth for their file data.4 In the meantime, global, local and corporate regulations governing the preservation of data for legal, research, risk management or auditing purposes are prompting organizations to retain large amounts of inactive data for years, decades, and in some cases forever. Recent advancements in the field of generative AI open the door to using data in new ways to provide business-driven insights, enhance customer experience, automate processes to name but a few GenAI applications. This prompts organizations to consider how they can preserve raw data long-term for future use by AI services.
Cold storage is becoming a necessity in addressing these technological challenges associated with spiking costs. A cold storage system is ideally suited to provide a cost-effective, durable, secure and hassle-free management and retention of archival data, all essential attributes when it comes to storing vast amounts of data that no longer requires high-performance, low-latency access.
Realizing Tangible Cost Optimization with Tiger Technology and AWS Cloud
Tiger Technology and AWS Cloud have been partnering together for years building cold storage infrastructure for customers ranging from oil & gas companies, manufacturing giants, law enforcement, educational institutions and governmental facilities. The combined power of Tiger Bridge and Amazon S3 enables organizations to optimize their archive costs and meet ever-increasing compliance mandates. When calculating ROI, there are three important factors to consider: the price of the storage component, its maintenance, and the cost of data life cycle management based on actual data access patterns. Typically, inactive data comprises >90% of the total size of data, and fast enterprise-grade storage can cost between $100-200/TB a month, bringing in a low ROI. The Tiger Bridge and Amazon S3 solution allows organizations to keep just a fraction of on-premises storage and automatically tier the data to less expensive and much more resilient cloud storage. A customer can lower their total storage cost to less than $20/TB a month and keep the performance of their enterprise storage used for hot data.
Let’s take a deeper look at each component in this intelligent cold storage solution.
Understanding Tiger Bridge
1. Bridging File Storage and Object Storage
Tiger Bridge is data management software that unlocks multiple outcomes for organizations with large data sets ranging from cost and resource optimization, data resilience and portability, storage capacity management, and operational efficiency. Tiger Bridge creates a global namespace across local and cloud storage, local-to-local storage or cloud-cloud storage to address storage standardization, capacity requirements, DR and business continuity, data orchestration and processing by cloud AI services.
2. Tiger Bridge Features
Tight Amazon S3 Support
Tiger Bridge works with all Amazon S3 storage classes and supports cloud-native functionality such as object immutability, Versioning in S3 buckets, and Delete markers for data recovery. The software enables a fully automated archiving workflow with data replication and restoration without the need for external solutions.
The Go-between for Applications and Cloud Objects
Tiger Bridge speaks the language of both local, often highly specialized applications as well as cloud object storage, two concepts that are not built to understand each other. Tiger Bridge enables transparent application access to data in the cloud and ensures data is AI-ready with its ability to directly serve files to object-based cloud services.
Smart Stub File Technology
The smart stub file technology by Tiger creates zero-byte files locally that contain all metadata and point to the files’ location in Amazon S3. Users and applications still see files and folders as local even when stored in the cloud. This enables constant access to data and on-demand downloads while optimizing the amount of storage used and minimizing cloud operation charges.
File System Level Integration
Tiger Bridge works with NTFS, ReFS and NAS (SMB, NFS) as a source. Integrated at Windows kernel level, Tiger Bridge natively supports Windows OS functionalities: Active Directory, DFS etc.
Data Integrity Capabilities
When Tiger Bridge replicates a file to the cloud, it preserves its original file format as well as all metadata. This means that if it ever needs to be downloaded from the cloud, the file remains unchanged, thus maintaining its integrity.
To find out more about how Tiger Bridge works with AWS cloud, skip ahead to the Benefits of the Tiger Bridge Integration with Amazon S3 section.
The Role of Amazon S3 Glacier Storage Classes in Cost-Effective Cold Storage
AWS Cloud currently offers seven storage classes – four hot tiers and three cold tiers, which are all S3 compatible and use the same API. For the purposes of this article, we will focus on Amazon S3 Glacier classes, purpose-built for cold storage.
Understand Amazon S3 storage classes with this infographic.5
S3 Glacier Instant Retrieval, S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive are the three cold storage classes, designed for archive data that rarely needs to be accessed. Storage costs are $4, $3.6 and $1 per TB/month, respectively. S3 Glacier Instant Retrieval delivers the fastest access to archive storage, with the same throughput and milliseconds access as the S3 Standard and S3 Standard-IA storage classes. With S3 Glacier Flexible Retrieval and S3 Glacier Deep Archive, data is only accessible as standard within up to 12 hours and between 12 and 48 hours, respectively. Minimum contract lengths of 3 and 6 months per object.
Amazon S3 offers all the benefits of public cloud including durability, security, availability and performance. Many organizations that need to store data for compliance reasons and are looking to slash long-term storage costs should consider Amazon S3 archive storage classes as a viable option.
Click here to see a comparison of public cloud storage tiers.
Benefits of the Tiger Bridge Integration with Amazon S3
Tiger Bridge integrates seamlessly with AWS, replicating data from on-premises storage to Amazon S3 or across EBS and Amazon S3 environments for data life cycle management, global file access, disaster recovery, file data migration and data insights. It supports all Amazon S3 storage classes, optimizing costs by automatically tiering data based on access patterns. The solution also provides continuous data protection, safeguarding against data loss and ransomware. Integration does not require any additional hardware or changes to existing workflows.
Get the Tiger Technology for AWS Brief
The AWS and Tiger Bridge integration brings a number of critical benefits to aid an organization’s long-term storage strategy:
1. Policy-based, hassle-free data life cycle management
Tiger Bridge runs on efficient built-in policies that allow users to automatically move data from hot/cool storage classes to archive classes based on size and frequency of file access thus enabling low-latency access to production data and cost-effective archival of inactive data.
When choosing Amazon S3 storage as a target, users can select their default storage class as well as archival retrieval type (standard, bulk or expedited) directly within the Tiger Bridge interface for each source folder or drive on their local storage.
Tiger Bridge also allows users to manually select a file for archival or the option to default to an Amazon S3 Glacier storage class for an entire source location, which means all archival data will be automatically tiered to the right Amazon S3 Glacier class based on the specific application or workflow needs and automatically rehydrated from Glacier on demand.
Straight-to-cold-storage with Tiger Bridge is particularly cost-effective in industries that produce vast amounts of data that is not modified/accessed after generation, for example video surveillance, as this allows the organization to skip the spending on hot/cold storage classes stage.
2. No vendor lock-in solution – migrate on demand
Tiger Bridge eliminates dependence on storage vendors by preserving data in its native file format and managing data at the native file system level. While data is replicated to Amazon S3 storage, users still see their original files and folder structure and can always access and retrieve their data. No vendor lock-in means that Tiger Bridge provides the freedom to migrate data between disparate storage solutions, e.g. between different public cloud providers, or between a cloud provider and a private data center or even tape storage, in a truly frictionless way. To illustrate, a large governmental entity is currently employing Tiger Bridge to migrate data from private cloud to AWS, which allows them to maintain 100% of their mission-critical operations in the meantime.
3. Pay-as-you-go pricing model
To bring the best value to customers, Tiger Bridge uses a cloud-like, subscription-based model. As a total cold storage solution, Tiger Bridge and Amazon S3 enable organizations to only pay for the storage and services they are consuming, thus removing the need for overprovisioning for on-premises hardware or carrying the operational burden of hardware refresh cycles and maintenance. Tiger Bridge’s licensing is not user-based and can include pre-paid capacity, which further adds to flexibility. Tiger Bridge does not require purchasing additional hardware or virtual appliances, further reducing the maintenance burden.
Implementing Tiger Bridge and Amazon S3
Setting Up Tiger Bridge
Installing and setting up Tiger Bridge software is quite straightforward. Before you begin, please make sure you have the following in place:
- Create a separate S3 bucket for each source configured on the same computer
- Provide IAM user credentials to be used by Tiger Bridge.
Then proceed to installation and configuration:
- Install Tiger Bridge on your Windows OS Server.
- Add a local volume source – this connects Tiger Bridge to your existing on-premises storage.
- Pair the local source with Amazon S3 target.
- In Default storage class, select the storage class to which Tiger Bridge will replicate data directly, omitting any intermediate tiers.
- In the Existing cloud data action dialog, select what to do with data already existing in the bucket.
- Select and set the needed policies - Replication, Reclaim Space, Archive.
A Deeper Dive into Setting up Cold Storage with Tiger Bridge and Amazon S3
With Tiger Bridge, users can set up archival directly within the Tiger Bridge UI with a few clicks. Tiger Bridge offers extended support for Amazon-native functionality.
1. Configure Archive Policy
Select when your files should be moved to Amazon S3 Glacier. This can be done based on access (files not accessed for more than + insert a threshold in seconds, minutes, hours, or weeks) and optionally based on file size (files bigger than + insert a threshold). Tiger Bridge will scan your source and start moving inactive data that meets the criteria you’ve set. The Amazon S3 bucket will display which files are cold and therefore stored in Glacier. Within the Tiger Bridge UI, users can select the archive retrieval option that best suits their needs and budget. Tiger Bridge uses Standard retrieval as default. If at the time of data retrieval there is insufficient capacity to process an Expedited request, Tiger Bridge automatically switches to Standard. The Expedited retrieval option is not available for the S3 Glacier Deep Archive storage class.
2. [Optional] Direct-to-cold-storage for a specific data set
As mentioned, the Default storage class option allows users to select the storage class Tiger Bridge will replicate data to. If you do not select a specific storage class, Tiger Bridge uses S3 Standard. However, if their specific workflow requires it, users can choose any of the S3 Glacier classes as a default tier, which means that Tiger Bridge will archive all data on the local source configured. It is important to become familiar with Amazon S3 pricing when it comes to storage, write operations and data retrieval when choosing the right Glacier class and retrieval option for your organization. Generally, the deeper the archive, the lower the storage cost, the longer the retrieval period and the higher the cost for expedited retrieval.
Download the Tiger Bridge for AWS deployment guide
Best Practices
How to transition existing data to Amazon S3 using Tiger Bridge
- Plan your Tiger Bridge Use Case
Tiger Bridge can address a number of hybrid cloud use cases. Determine the needs of your organization or a particular workload.
- Consider generating a Tiger Technology Storage Assessment Report.
How much do you know about your data access patterns? When asked, IT departments would estimate around 15-20% of their data is active/production data when in reality storage assessments we perform for our customers indicate that only around 3-7% of a typical organization’s data is live/active.
Learn more about how you can optimize your storage here
- Consider the cache storage capacity
Calculate the amount of high-performance cache storage necessary to support the volume of your frequently accessed data. Tiger Bridge will work in the background to move cold data to Amazon S3 Glacier.
- Consider long-term planning for your future data workflow.
As compliance requirements are increasing, evaluate how your data needs would change in 5-10 years and whether you would like to have archival data readily available in cloud storage so you can run cloud AI services. To make an informed financial decision, compare what your total cost of ownership would look like if you kept investing in your current storage solution against adopting cold storage in the cloud.
Real-Life Customer Scenarios
The Irwin S. Chanin School of Architecture at Cooper Union Protects Decades of Digital Archives
The Archive Department at The Cooper Union for the Advancement of Science and Art archives student and professor work annually, and so their storage demands exponentially increase every couple of years. Tiger Bridge is now running on their on-premises server to intelligently manage the life cycle of data across local and Amazon S3 storage. Tiger Bridge also reclaims space on the local server as soon as data is no longer needed for daily access, so users can download data from AWS on demand without running out of space.
The biggest outcome of using Tiger Bridge was being able to migrate to Amazon S3 where the customer can benefit from endlessly scalable, cost-effective storage and still preserve file metadata. Integrating Tiger Bridge for AWS into their tech stack meant that they no longer had to invest in on-premises storage up front or carry the burden of maintaining it. In addition, the transparency of the solution and the easy, self-service access to archived files contributes to reduced hands-on involvement from IT administrators in the day-to-day work of Architecture Archives staff.
Find out more about how Cooper Union uses Tiger Bridge and Amazon S3 for their archival data.
Oil & Gas Company Archives Seismic Data for Analysis
An energy company has performed a large number of seismic data surveys over the years amounting to several PB of data. The company determined that these large data sets are of huge value and need to be securely archived for future potential analysis. The customer is leveraging the Tiger Bridge and Amazon S3 integration to archive seismic data once the production analysis has been completed. In particular, the customer required that they archive directly to Amason S3 Glacier Deep Archive, a highly cost-effective storage class that Tiger Bridge supports seamlessly.
Tiger Bridge is running on the customer’s on-premises server that is connected to their local enterprise-grade storage. Tiger Bridge automatically replicates data in designated archival folders to AWS, after which the space is reclaimed locally.
The financial impact of introducing this cold storage solution is significant, as the Tiger Bridge and Amazon S3 integration allowed for a reduced on-premises storage footprint, shorter backup cycles and therefore significant cost savings for the customer over the short and long term. This highlights the opportunities for oil exploration and environmental management companies to enhance operational efficiency, derive insights from scientific data, implement cost-saving measures, and improve resource allocation strategies.
Conclusion
The importance of having a cold storage system in place is set to only rise with faster data generation rates, tighter governmental and organizational regulations, and increasing needs to extract more value from data. Organizations dealing with large amounts of data and longer retention requirements should assess their cold storage strategy now in order to build a cost-optimized infrastructure that can support their future data challenges. Tiger Technology and AWS combine seamless hybrid cloud file data services with highly durable, lower-cost cloud storage to offer a total solution that manages production data and cold data and ensures that archives are AI-ready today for the organizational GenAI needs of tomorrow.
Start designing your cold storage strategy with one of our experts in AWS cloud storage.
References:
- Gartner, Inc., “Hype Cycle for Storage Technologies, 2024”, Analyst Julia Palmer, 10 July 2024. GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.
- www.techtarget.com
- www.statista.com
- Gartner, Inc., “Modernize Your File Storage and Data Services for the Hybrid Cloud Future”, Analyst Julia Palmer, 28 February 2023. GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.
- www.aws.amazon.com