31 October 2023
Wim Stoop, senior director, hybrid data platform, Cloudera
Compliance is a challenging topic for many data leaders, and it isn’t going to get any easier as global data volumes increase rapidly. While this growth means more opportunity to extract insight and value, it also puts teams under greater pressure to ensure information is stored securely and access is controlled. The maturation of AI technologies will add further pressure – this time to ensure underlying datasets and AI outputs can be trusted – as more use cases for AI come to light.
All this data transformation is happening during an unprecedented period of regulatory activity across the globe. In the EU alone, organisations will pass new compliance milestones next year in the AI Act, the NIS 2 Directive and Digital Services Act to name but a few. These regulations are wide-ranging, impacting both companies within the EU and global organisations that have operations in the region.
To manage this compliance complexity, organisations need a single set of governance policies that are always-on and run everywhere.
The diversity of data
Most organisations have set their sights on data-driven decision making as a critical competitive differentiator. And the scale of data volumes we’re talking about today makes cloud infrastructure and software vital to driving this kind of value.
Enterprises hold both structured and unstructured data, all with varying levels of sensitivity. Some will house Personally Identifiable Information (PII) on their network, and others may use Payment Card Industry data (PCI). Much of this data must remain on-premises to deliver enhanced security and control, or to satisfy regulatory and data sovereignty requirements. But some data will be sent to the cloud for cost and performance reasons. This is where compliance can get tricky, especially when data becomes fragmented across multiple public clouds and on-premises environments.
It’s easy to see why three-quarters (79%) of IT decision makers feel compliance is their main data management concern, according to recent research. Especially when considering that GDPR fines alone soared 168% annually to reach an estimated €2.9bn in 2022.
Disparate data has also resulted in organisations using a myriad of point solutions for data analytics and management. However, these solutions often lead to yet more fragmentation and compliance complexity, causing 79% of data leaders to believe point solutions have made compliance more challenging.
Always-on compliance
Getting governance right is essential to streamlining compliance. It should be consistent across all environments to deliver the right outcomes. In this context, effective governance means truly understanding data and its business relevance, having policies to authenticate and control who has access to data, and to secure it in other ways, such as with robust encryption. But it’s also critical to have continuous visibility into all enterprise data, and to apply key compliance principles like minimisation, purpose limitation and accountability. In an AI context, governance should be used to ensure models deliver trustworthy output. That means underlying datasets must be as comprehensive and clean as possible.
However, in a world where each cloud provider or on-premises environment has a distinct architecture, framework, and data security/privacy approach, applying governance is a challenge. By considering a modern data architecture that sits across disparate environments, organisations can move towards a more effective hybrid cloud governance model. Such an approach could use an integrated set of security and governance technologies built on metadata, to ensure consistency across clouds. With one set of globally defined policies in place, organisations can replicate their security approach across all environments, reducing risk and human error, whilst saving time and resources.
Truly effective data governance must be flexible, and it must preserve data mobility to be fit for purpose. By managing compliance in this way, data leaders can also remain compliant even as regulations change.
The path to growth
Above all, governance cannot be left to the last minute. It must be proactively implemented in advance of cloud initiatives, to avoid potentially damaging incidents. Yet it’s not just about mitigating the risk of serious financial or regulatory damage, which can stem from data breaches and non-compliance. By opening up new markets and helping to build loyalty and trust with customers, good governance is a business enabler in its own right.
With AI innovation marching on, and the volume of data organisations have available growing by the day, getting governance in place now will not only unlock more and better insights now, but also save trouble down the line.
