10 October 2022

With worldwide data generation increasing exponentially, enterprises are facing the pressing need to find and deploy solutions for data storage that are secure, reliable and scalable. Amy Saunders delves into the world of storage and explores several of the enterprise storage solutions available

Data is produced day in, day out, from billions of sources across the world. While the volume of data generated has been growing exponentially throughout the digital era, with more people operating more devices, and devices operating themselves, data volumes have begun to exceed even the wildest expectations off the back of the COVID-19 pandemic. With a colossal shift in workplace practises to limit physical contact and stop the spread of the virus, remote working has helped fuel a boom in demand for storage solutions, particularly highly secure cloud native products.

Why do we store so much data? One of the largest producers and users of data in the UK is the NHS, which holds the medical records for 65 million people. Those records must be updated and reviewed frequently for the patient’s full lifetime in order to provide the best possible healthcare, thus the data must be securely stored and easily accessed, nationwide. Moreover, with an aging population that by definition generates more medical data, the amount of data is only going to increase, and data storage pressure will grow.

Another huge data generator is the Internet of Things (IoT). Billions of connected devices are already in play around the world, monitoring everything from pressure at an offshore oil rig to rainfall levels at a particular agricultural field. This data can be stored, analysed, and used to provide meaningful and actionable intelligence, for example, for how often a valve is likely to need to be replaced at said oil rig, or which crop will grow best in the natural rainfall conditions at said field.

Much of the data being produced today can be monetised, either by internal use to improve processes, realising new innovation opportunities, or by sale to third parties. It would be breathtakingly wasteful then to leave that data unutilised, making storage and processing absolutely necessary.

TMI (too much information)

Data is being produced at an unprecedented rate. According to Statista, global data production hit 64.2Zb in 2020, and is expected to almost treble to 180Zb by 2025. This projection has been upgraded since the pandemic, when growth exceeded expectations due to uptake of remote and hybrid working and heavier use of home entertainment options.

Despite this production deluge, just 2% of all data created in 2020 was retained into 2021. However, even if only a tiny proportion of data is saved, global storage demands could prove immense in the years to come. Statista reported that in 2020, the installed base of storage capacity was 6.7Zb, which is expected to expand at a compound annual growth rate (CAGR) of 19.2% over 2020-2025.

Looking at revenues, Million Insights reported earlier this year that the global storage market size was valued at US$42.57 billion back in 2020 and is expected to expand at a CAGR of 5.4% over 2021-2028. Growth is being driven by improvements in storage systems which are redefining the consumption and deployment of storage solutions; the emergence of private and public cloud infrastructure; rising adoption of mobile devices by both private users and enterprise - the latter of which is finding that mobile devices deliver cost-efficiency, increased performance, energy savings and enhanced reliability; and, of course, the continually growing data volumes.

The large enterprises segment dominates the market, with a revenue share of more than 75% in 2020. However, it is the SMEs that are expected to expand at the highest CAGR from 2021-2028, with 7% growth expected. This is being driven in part by tax exemptions and government support.

The cloud segment dominated the market in 2020 with a share of more than 35%, and it is this same segment which is expected to achieve the highest CAGR for 2021-2028. Cloud storage is remarkably low cost, easy access, and is being rapidly adopted by enterprises across the world. Not only does cloud storage save the cost of maintaining an on-site data centre, but it also enables faster data retrieval irrespective of the data centre location. On the other hand, the traditional storage segment, which includes external storage devices, is expected to grow at a CAGR of 4.5% over 2021-2028.

While North America held the largest storage revenue share in 2020 at 35%, Europe comes in at a close second owing to the rising penetration of mobile devices, with the UK, Germany and France all playing a major part in said growth. The APAC region is expected to see the highest CAGR, at 7%, during 2021-2028, with countries such as China and India helping boost demand. Enterprises are investing heavily in these areas, which have large customer bases and a high rate of digital technology adoption.

So, what does that mean for enterprises? We’re already seeing small, medium and large enterprises upgrading their storage systems to more user-friendly, scalable and secure solutions to meet growing business needs. We can expect to see this pattern continue in the years to come, with a heavy focus on compression and deduplication of data in order to cut both OPEX and CAPEX. The bottom line is that enterprises need smart solutions that can grow with them in order to face the data deluge.

What are some of the options?

Amazon S3 –Fully scalable object, block and file storage

Amazon Web Services (AWS) offers several cloud storage solutions, and users can pick from object, block and file storage services.
The company’s top product, Amazon Simple Storage Service (Amazon S3), offers object storage with full scalability, and promises to store and protect any amount of data for a wide range of use cases, including data lakes, cloud-native applications, websites, backups, archive, machine learning and analytics. Designed for 99.999999999% (11x9s) durability, Amazon S3 leads the storage market in terms of revenue and data quantity.

Users can store data in Amazon S3 by working with an organisation from the AWS Partner Network (APN), which includes a variety of partners offering solutions for archive, primary storage, disaster recovery, backup and restore, and migration. To keep data secure, Amazon S3 allows users to block public access to all data with S3 Block Public access. Amazon S3 maintains compliance programmes such as PCI-DSS, HIPAA/HITECH, FISMA, and EU Data Protection Directive to help the user meet regulatory requirements.

AWS also offers Amazon Elastic Block Store (Amazon EBS), a block storage service offering an easy-to-use, scalable solution for Amazon Elastic Compute Cloud (Amazon EC2), delivering 99.999% durability.

There’s also AWS Backup, a fully managed, easy-to-use backup service that centralises and automates data backup across AWS in the cloud and on premises. The cost-effective solution helps support regulatory compliance and business policies for data protection.

Cloudian Hyperstore – Cloud native object storage with robust security

Cloudian offers simple, modularly scalable, cloud native object storage. S3 compatible Hyperstore enables organisations to store data across sites, on containers, servers, virtual machines or the cloud, within a single, unified platform. With HyperIQ, a single screen overview of the entire data environment is provided, making it easy to monitor user behaviour and spot performance or compliance problems.

Hyperstore seamlessly integrates with AWS, Microsoft Azure and Google Cloud, and uses the provider’s tools to copy and move data, a useful feature for archival storage, content distribution and disaster recovery.

Key features include its robust security – Hyperstore complies with US DoD SEC Rule 17a-4(f), FINRA Rule 4511, and CFTC Rule 1.31(c)-(d) security requirements – and data immutability for ransomware protection and compliance. By utilising S3 Object Lock, cybercriminals can be prevented from encrypting data, enabling the fast recovery of unencrypted backup copy. Hyperstore allows advanced metadata tagging for accelerated data searches and easier integration with AI and machine learning analytics applications. Data is stored locally to meet governance regulations and accessed from Cloudian’s all-flash or HDD-based storage with low latency. Cloudian offers appliances with pre-integrated Hyperstore software in 1U or 4U form factors, in all-flash and HDD configurations, ranging from 100TB to more than 1PB.

DataCore Swarm – Software-defined highly scalable object storage

DataCore’s SWARM is an intriguing software-defined object storage solution for on premise, and private, public or hybrid cloud environments. Swarm delivers a highly scalable solution for accessing, storing and distributing data with continuous protection, boasting 25 times less downtime than traditional RAID storage, 95% storage utilisation and five times faster throughput for S3 read and write operations.

Swarm simplifies the management, storage and protection of data while enabling S3/HTTP access to any device, application or end user. Custom metadata with ad hoc search and query, and policy-based replication for synchronous, asynchronous and stretch cluster uses are included. It utilises a web-based UI and API and automates storage and infrastructure management, providing simplified data management that can be overseen via the unified web console, where content usage quotas and performance trends can be monitored.

Data is secured through encryption in transit and at rest, and storage access and activity can be tracked with audit trails. Data immutability is assured with WORM integration, S3 object locking and Legal Hold, while content integrity is established using Integrity Seals.

Capacity can be added, and hardware refreshed by pooling any mix of HDDs, SSDs and x86 servers, with Swarm’s self-managing and self-healing architecture. Data can be searched for quickly, metadata customised, and files accessed across a single namespace via S3/HTTP, NFS and SMB. Cold data can be backed up to Wasabi, S3 Glacier and object cloud and tape storage solutions.

Dell Unity XT – Converged infrastructure for block and file storage

Dell EMC offers the Dell Unity XT family to deliver hybrid cloud and big data solutions utilising data centre infrastructure to combine storage, servers, and cybersecurity as converged infrastructure.

Dell Unity XT is a full block and file unified environment contained in a single 2U enclosure, using the same pool to host LUNs, NAS servers, file systems, virtual columns and consistency groups. It uses both storage processors to serve I/O and run data operations efficiently in an active/active manner to optimise performance, cost and density in the user data centre. The solution is managed via a simple interface called Unisphere which uses browser native HTML5.

Dell states that the Dell Unity XT family is extremely customisable. The purpose-built system can be configured as all-flash with only SSDs, or as a hybrid system with both SSD and spinning media. Dell Unity XT utilises inline data reduction to enable storage administrators to achieve more with less; Dell Unity Data Reduction works to reduce the amount of physical storage needed, providing space savings through data deduplication and compression.

Unified Snapshots provide point-in-time copies of block and file data that can be used for backup and restoration, while asynchronous replication offers an IP-based replication strategy within a system or between systems.

Scality RING – Software-defined ‘unbreakable’ file and object storage

Scality’s RING is a software-defined solution deployed in the user data centre, where it delivers resilient, petabyte-scale storage for archive, backup, web, video, analytics and custom applications. Deploying on any standard x64 servers, RING can scale linearly over thousands of servers, multiple sites, and an unlimited number of objects. Connectors can be co-located with customer application servers, on storage servers or on virtual machines.

Data becomes ‘unbreakable’ through replication, immutable object locking, erasure coding and a multi-geo distribution – which protects against data centre outages - in order to achieve 99.999999999999% (14x9s) of durability and 100% availability. RING comes with air-gapped, tamper-proof backup data that stays immune to ransomware, offering a swift recovery from cyber-attacks.

RING supports native file, object and AWS S3 and IAM APIs interfaces, providing extremely high performance at up to 90% lower total cost of ownership than legacy storage solutions. Data-rich custom and packaged applications can both aggregate multiple workloads into a single storage environment, eliminating storage silos, and expanding utilisation with cloud-like economies of scale. RING supports all-flash storage servers too.

With hybrid cloud support, Scality’s RING offers metadata-driven, policy-based data orchestration for data mobility across public and private clouds. Global metadata namespace, visibility and search is enabled across RING and public clouds, with support granted for both user and system-defined metadata. Supported storage and clouds include Amazon S3, Azure Blob Storage, Google Cloud Storage and Scality RING. 

HPE SimpliVity – Hyperconverged architecture for SMEs

A unique inclusion in our review due to its recommended use cases - smaller enterprise situations, for example, retail, ROBO, small office and VDI, where applications are typically less latency sensitive - HPE SimpliVity from Hewlett Packard Enterprise (HPE) delivers hyperconverged storage by combining the entire IT stack in each node, thus consolidating up to 10 devices and applications in a building block for virtualised workloads. A hyperconverged Ai-driven architecture, HPE SimpliVity delivers self-optimising, self-managing and self-healing infrastructure.

HPE SimpliVity offers a 69% savings cost and requires no specialists to operate. It enhances its rapid backup and recovery capabilities with a centralised HPE StoreOnce long-term backup appliance and seamless cloud backup via HPE Cloud Volumes Backup.

All data is stored on a least two local hyperconverged nodes for disaster recovery protection and backup, while always-on deduplication and compression reduces capacity utilisation up to tenfold. Up to 16 nodes/cluster and 96 nodes/federation makes HPE SimpliVity ultra-flexible, and it’s also possible to mix clusters within the same federation. Global VM-centric management and mobility, and AI with HPE InfoSight deliver easy-to-use, intelligent management.

Two iterations exist: HPE SimpliVity 325; and HPE SimpliVity 380 - both of which can be integrated with HPE Composable Fabric, an intelligent networking fabric – and which offer varying capabilities at different price points. HPE SimpliVity 380 enables storage-intensive workloads, multiple all-flash configurations (XS, S, M, L and XL), backup and archival node with hybrid flash storage, as well as hardware-accelerated or software-optimised for always-on deduplication and compression.

Something for everyone

As we’ve outlined, data generation is expanding exponentially, with no sign of slowing down in the foreseeable future. As the sources of those data become increasingly diverse, having the tools in place to store, process and ultimately use that data are vital.

Each enterprise will have their own needs and demands when it comes to shaping their storage environment, but with so many storage products available on the market today, a solution can be found for every use case. Whatever form that solution takes; scalability is of the essence. Ignoring the need for ever increasing data storage could find an enterprise in a sticky situation when its systems shout “storage full!”