21 May 2019

Big data sets have become everyday workloads for many data centres and resource-challenged network managers must gain new skills to go with the flow, writes JAMES HAYES

Big data’s entrée into business computing was its promise that the analysis of very large volumes of data would deliver actionable insights from which quantifiable business value could be derived. It’s taken some time for that to happen, but many enterprises the world over now believe that big data analytics will deliver proven return on investment.

Now big data’s remit has expanded. Its potential is being applied to a broader range of data sets, from marketing and CRM to risk management and product development. Furthermore, big data tools are now being applied to real-time operations across applications like cyber security monitoring and customer interactions, air traffic control – and even data centre traffic control.

Cisco’s latest Global Cloud Index confirms that big data is a key driver of overall growth in stored data (see Networking+, March 2018 issue). Big data will reach 403Eb by 2021, up from 51Eb in 2016. Big data alone will represent 30 per cent of data stored in data centres by 2021 (up from 18 per cent in 2016). This ramp-up also reflect the fact that big data is getting bigger. Cisco defines ‘big data’ as data deployed in a distributed processing and storage environment (such as Apache Hadoop). However, Cisco notes that big data has also become a significant driver of traffic within data centres. While much of big data traffic is ‘rack-local’, enough of it exits the rack that big data will account for 20 per cent of all traffic within data centres by 2021 (up from 12 per cent in 2016).

Tidings of cheer for big data solutions vendors, maybe, but news that also brings profound challenges for the data centres tasked with shouldering this substantial workload expansion – or, more specifically, the IT practitioners tasked with the effective management of those facilities.

“Enterprises are under unprecedented pressures to manage this data in an effective manner,” says Alex Henthorn-Iwane, VP product strategy at ThousandEyes. “Given the data volumes involved, this is not always possible, and a 2018 survey [by Alteryx] estimated that IT inefficiencies cost European enterprises €49m a year.” IT architectures focused on big data are extremely complex, Henthorn-Iwane adds, so companies “cannot rely on mere assumptions” when they design them.

“Big data workloads are overwhelming network management teams – network speeds, data volumes and business application numbers are all increasing, so network and security analytics tools are reaching their capacity faster than expected,” agrees Marc Sollars, CTO at Teneo. “Such an information overload is creating a ‘monster’ of escalating costs, slower application performance, and less efficient security controls. And a lack of visibility of critical business applications as they are moved into different virtual and cloud environments also makes it harder for IT teams to optimise and secure them.”

“Enterprises are under unprecedented pressures to manage this data in an effective manner.” Alex Henthorn-Iwane, VP product strategy, ThousandEyes

With the increased ability to analyse the volume of data now available to asses business performance and/or consumer interaction, the network manager’s role has shifted significantly, says Kevin Drinkall, EMEA wireless & cloud market development manager at Zyxel. “The challenges involved with being a network manager no longer just involve looking after a network,” he explains, “but also ensuring data accessibility is a given across an organisation efficiently and securely.”

Distributed visibility

“When techies think of big data, they tend to think of those three Vs: volume, variety, and velocity – and [next they think of] the strain [big data workloads] put on legacy, fixed architectures,” says Sascha Giese, head geek at SolarWinds. “The most significant attribute is big data’s ability to break larger jobs into lots of smaller ones, distributing resources to work in parallel on a single task. As a result, IT managers should adopt a nuanced approach when it comes to managing these distributed networks.”

Senior solutions engineer at DataStax, Patrick Callaghan, points out that big data “comes in a lot of shapes and sizes, so different implementations can have different impact on your applications and on enterprise networks. For large-scale applications, the volume of traffic will be high and continuous, so you will have to have the scale of network to cope. For many of these applications, you should be looking at distributed deployments – so running across multiple services and locations for both availability and resiliency – and this can include running across hybrid or multi-cloud environments, too.”

“Network speeds, data volumes and business application numbers are all increasing, so network and security analytics tools are reaching their capacity faster than expected.” Marc Sollars, CTO, Teneo

Monitoring is essential, insists Giese. “Big data applications are sizeable, causing huge spikes in traffic, which can cause congestion leading to queuing delays, packet loss, and retransmissions. And each of those applications – possibly sitting in different partitions – will vary,” he says. “Some might be particularly bandwidthheavy, while others might be latencysensitive. As IT is increasingly faced with big data workloads, and networks become more distributed, it’s vital that network managers have full, granular visibility of the entire network if they are to guarantee performance and scale.”

The primary tool to bring that visibility is analytics, Drinkall says: “As big data evolves, organisations increasingly want and need greater data analytics. Network managers can address these challenges by expanding their knowledge into database infrastructure and storage solutions. They must up their game in skillsets, not just in terms of looking after networks, but in terms of analytics and business intelligence.” It might seem like so much extra graft and longer hours at labour over a hot workstation (and it is), but such IT skills acquisition can result in gains for the IT professionals involved.

Data drives analytics

According to Peter Walker, VP EMEA (north) at Information Builders, big data analytics now “drives the introduction of infrastructure network analytics platforms”. These examine the flow of data across the network in order to monitor the ‘health’ of the network, pinpoint areas where transmission is being delayed or disrupted, and suggest ways to optimise data transmission. Indeed, managing the network here involves understanding how distributed computing implementations work as a whole, agrees Callaghan: “For example, is there a master node that is in control, or is it a full masterless deployment?” (For big data, by the way, masterless systems should be preferred, Callaghan adds.)

‘Data-driven networking’ is a buzzterm that sometimes emerges when the topics of big data and networking are married. The underlying concept is that analysis of big volumes of real-time network data can be used to optimise the mechanisms for network packet forwarding: it’s like applying big data analytics to network data traffic, and then seeing what the results reveal about how network configuration could be tuned to improve performance, etc. This is an advanced approach to what network managers have been doing for years, although use of sophisticated analytics tools can mean upskilling for network specialists involved.

“Much has been written about datadriven networking,” says Walker. “Without data, organisations cannot become digital. All digitally-transformed enterprises must become ‘data-driven’ in every aspect of their operations in order to compete and survive as part of the larger data ecosystem. Organisations have realised that data is an asset that can be monetised – therefore, they need to optimise network performance to cope with additional devices, Operating Systems, workloads and processing requirement so that data from a multitude of sources can be integrated, augmented and served in the form of real-time intelligent insights.”

The combination of IT knowhow and capability with business context and requirement means the future of network management is key to unlocking the potential of data to help drive the future of enterprises, Walker believes.

“The concept of the ‘data-driven network’ will have real resonance for the future of network management, simply because everything we do is about traffic management within a network,” explains Drinkall. “Whether that’s how much traffic is passing through a switch, or how much traffic is going over the wireless – it is now extremely relevant for how organisations manage their networks.”

For Giese, network managers grappling with big data should understand that it is another form of the new norm: “Intelligent, automated networks are 100 per cent the future of our industry. Data-driven networking uses automation to apply big data analysis to existing, real-time network data to optimise network performance in the future. The days when a network admin would be stuck swivelling from monitor to monitor to guarantee performance, or wasting time manually configuring storage QOS, are gone.” The reality is, “many of the metrics that once took a team of IT managers to keep tabs on are now easily monitored from a single dashboard”, he thinks.

Automation is freedom

Meanwhile, the concept of the data-driven network does have “real resonance for the future of network management”, insists Walker at Information Builders. “The Internet of Things, Industry 4.0, Machine Learning (ML) and blockchain – all are technologies utterly dependent on reliable data capture, storage, processing, transmission and access.” Data consumption is becoming a fundamental part of every industry, from manufacturing to modern healthcare: “The key aspect for network managers is to ensure that data can be accessed,” says Walker.

“The challenges involved with being a network manager no longer just involve looking after a network.” Kevin Drinkall, EMEA wireless & cloud market development manager, Zyxel

“We are seeing the next evolution of this shift as those monitoring solutions leverage automation and ML to predict anomalies before they occur and ensure efficiency,” says Giese. “These new capabilities free-up IT teams, faced with shrinking budgets and increasingly fewer resources, so they have more time to address strategic business priorities.”

So, despite the challenges big data brings, it is important for network managers who find themselves in its path to remember that the same technology can be deployed in the service of network management itself, says Sollars. “Big data analytics can drive network and security analytics performance improvements,” he says. “Enterprise IT teams are starting to use discovery sessions with external networking specialists to simplify their network and security architectures. This approach finds ways to route the right traffic to the right network and security analytics tools at the right time.”

Sollars adds: “This, in turn, delivers visibility of network performance and security threats – and accelerates threat detection and response. It also makes networks more efficient and economical to run. These discovery sessions are most effective when network and security teams are equally represented.”

“Big data provides a great understanding of the pressure the network is under and its impact upon the user experience,” agrees Drinkall. “We can also understand [more about the] issues related to operating in an ‘always-on’ society. The data allows network management to add real value in evolving, scaling and adapting around user experience in this constantlydeveloping technical world.” Ultimately, analysis of big data can improve network management, Drinkall argues, as it reveals the stress points and allows network managers to become more fluid in their approach to how infrastructure is configured, maintained and evolved.

Now big data’s remit has expanded. Its potential is being applied to a broader range of data sets, from marketing and CRM to risk management and product development

“Major changes are underway, which will have unprecedented impact on IT infrastructure in relation to Big Data,” predicts Henthorn-Iwane. “Enterprises are rapidly embracing new technologies that are underpinned by Artificial Intelligence (AI), particularly in an area referred to as AIOps.” (As defined by Technopedia, AIOps automates various aspects of IT and utilises the power of AI to create self-learning programs that help innovation in IT services. Note that it shares the same acronym as Algorithmic IT Operations, another digital transformation driver.) “AIOps can be applied to IT and big data issues and applications, with the goal of enhancing or replacing current operational processes.

Network professionals should be aware that increased adoption of automated management systems – AI-powered or not – are both inevitable and indispensable if data centres stand a chance of keeping on top of rapidly escalating data traffic. “I think that we are already seeing the change with IoT deployments – with so many connected devices coming on-line and sending data back, all that data has to go somewhere,” Callaghan observes. “These data sets scale-up rapidly, but the data growth itself should be predictable as well. Cloud environments can be a great fit for this use case – they can scale up in a managed way.”

An additional surge change here will follow the rollout of 5G services. The speed of the new networks to support these services, the capacity for more devices, the bandwidth available – all of this will support more IoT use-cases, particularly those where real-time data is important. This will boost the volume of data being created; and data sets and streams that will have to be analysed will grow exponentially.

“For large-scale applications, the volume of traffic will be high and continuous, so you will have to have the scale of network to cope.” Patrick Callaghan, Senior solutions engineer DataStax

“By employing [AI-enabled] technology, companies hope to get a data-driven, granular insight into their overall digital experience and how to continually improve it,” Henthorn-Iwane says. “The rise of AI-automation may seem like a long way off, yet last year 61 per cent of firms surveyed in a report [Outlook on AI in the Enterprise 2018 by Narrative Science] reported that they had implemented AI technology in some way.” Forecaster Research and Markets expects the global AIOps market to grow at a CAGR of 28.08%, to reach a market size of $9.907bn by 2023.

“The more efficient the foundation of the network is, the easier it is to accommodate big data, [so] the IT estate will need to change in several ways to accommodate [escalating numbers of] Big Data sets,” says Drinkall. “There is a definite need for increased network bandwidth and connectivity to allow for this. At the same time it will be important for businesses to leverage advances in cloud technology – if businesses aren’t implementing it now, then they must start using it to leverage and manage big data [if for no other reason]. In doing so, it will allow for network managers to take advantage of emergent and advanced tools, such as those which use some form of AI.”

Big data = bigger skills?

Yet even though AI automation will enable data centre networks to better cope with big data workloads and increased IP traffic workflows, somebody still must manage the resource-saving tools. Network professionals who also possess competencies in automated system, AI, and data analytics are not commonly to be found. The traditional team structure has favoured specialist practitioners, although this is changing as CIOs look to build the kind of multidisciplinary IT teams many believe will be needed to cope with the demands of enterprise ICT in the 2020s.

“Traditional big data analytics functions, such as integrating data from multiple sources, ensuring data quality through master data management, as well as data analytics and data science, are now dovetailing into the network manager’s role,” says Walker. “The data revolution presents network managers with a golden opportunity to add to their skills by supporting enterprises in managing the volume, storage, processing and transmission of one of the modern enterprises greatest assets: data.”

“All digitally-transformed enterprises must become ‘data-driven’ in every aspect of their operations in order to compete and survive as part of the larger data ecosystem.” Peter Walker, VP EMEA (north), Information Builders

“Securing and managing big data applications will require some specific skills, particularly around how to implement the infrastructure,” Callaghan says. “Deploying best practices – from securing deployments either on-premises or in the cloud, through to encryption of connections between sites and role-based access control – will be necessary for years to come.”

Callaghan adds: “Alongside the infrastructure side, looking into areas like analytics, search and graph, can provide new opportunities [for network managers] to expand their skills. The number of people with these skills is still low, so there is a lot of demand for people that can skill-up here and help deploy these technologies over time.”

Could this ‘demand’ in fact represent somewhat of a career opportunity for data centre stalwarts – a way to extend their professional scope into the field of big data analytics or AI, say? Yes, it can, says Giese: “For network managers, that [career opportunity] could very well be an option. Dealing with Big Data requires upskilling – the new skills of which will broaden any IT pro’s horizon. Having a sound understanding and knowledge of data handling combined with traditional network engineering is a solid foundation for all-rounder, generalist roles like IT project management and can even help on the path to a CIO position, if that IT professional so wishes.”

They should certainly be mindful of the fact that the Tech Cities Job Watch from Experis reports demand for big data professionals increased 78 per cent between Q1/2017 and Q1/2018 (see Networking+, June 2018 issue). Its study for Q1/2018 reveals that these specialists command the highest salary of £67,464 compared to the four other ICT disciplines the report tracks (cloud, security, mobile and web development).

Walker suspects that even if network managers do not actively pursue this career direction, it will pursue them. “I believe that the role of network managers will naturally evolve into the areas of data management, data science and analytics – in fact, this evolution has already started,” he reports. “Connecting structured and unstructured data together, having a view on all data from the edge – i.e., sensor data – to operational data, and being able to provide business and IT context to this data, in a continuous stream, gives network managers a strong advantage within current and future digital enterprises.”