Do you have a data noise problem?

Do you have a data noise problem?

Have you ever watched a press conference when a room full of reporters bark questions at the same time? Typically, the media event host will call on a particular reporter to repeat the question and then move on to the next person in the room. Without some ground rules, an actual conversation couldn’t take place. No one could hear the questions being asked, and few would get any answers.

Unfortunately, this same scenario often occurs with industrial data. With so much operational technology (OT) data generated on any given day, manufacturers risk losing critical information in the sea of “data noise” coming from their systems or having to expend vast resources to clean that data in the cloud.

Consider the following statistics:

• According to the World Economic Forum, industries generated and consumed 74 zettabytes of data in 2021. This year, that figure is expected to reach 130 zettabytes.

• Out of more than 1,000 industrial IT and operations professionals, nearly 40% stated they planned to store and process most of their new operational data in the cloud in 2022, up from only 26% of respondents in 2020, according to IDC’s 2022 Worldwide IT/OT Convergence Survey: Summary of Findings (Doc # US49666722, September 2022).

In 2021, the World Economic Forum published the report “Data Excellence: Transforming manufacturing and supply systems” in collaboration with Boston Consulting Group. The report aimed to identify top priorities for decision-makers to capture value from data and analytics in manufacturing and includes findings from a study of more than 1300 manufacturing executives. While nearly three‐quarters (72%) of manufacturing executives consider advanced analytics to be more important now than they were three years ago, only 39% of companies have managed to scale beyond a single use case.

Manufacturers are clearly struggling with the implementation of data and analytics applications both internally and at an ecosystem level. But why?

I believe the source of the problem is data noise and lack of digital infrastructure.

When I meet with manufacturing data and systems professionals, they often share that they are struggling to make sense of their data. Oftentimes, their “strategy” is to simply dump the data from each application into the cloud without a standardized format or coordination with other systems. The result is a barrage of data that makes little to no sense for the consuming application and, more importantly, key decision makers.

In a recent call with a major automotive OEM leader, he told me their plan was to send 750,000 data tags from the OPC layer to the cloud. From there, a data scientist would magically make sense of the information and make it usable. That’s a major undertaking and—based on all the conversations I’ve had—not a recipe for success. It’s costly to send this vast quantity of data to the cloud, (much of which will never even be used) and furthermore, this industrial data is raw and has very little context or relationship. The data is like words being yelled at that press conference. It is nearly impossible for Data Scientists who are not OT domain experts to understand it.

To create coherent “conversations,” we need to coordinate key information together into sentences—or in this case, logical, usable payloads of information streamed as data pipelines. With an Industrial DataOps solution, you can contextualize data at the edge before it’s published to the cloud. In other words, an Industrial DataOps solution pieces together a story—or conversation—before the data reaches your data lake or analytics platform. You create small, targeted conversations for the people who need to hear it rather than dumping massive volumes of data points into a server. You reduce the burden on your data scientists by sending coordinated, contextualized data payloads that you need at the frequency required for use. And you filter out the noise, so the required listeners can “hear” what their systems are saying and act on the information.

If you are struggling with data noise, I encourage you to take a closer look at how an Industrial DataOps solution can help you convert your data noise into actionable conversations. This white paper, “DataOps: The Missing Link in Your Industrial Data Architecture” explains the role of DataOps in achieving the vision of Industry 4.0 at scale and breaks down the five required components of an effective industrial DataOps solution.

More about IIoT Data Architecture Using MQTT and Kafka

About the author

This article was written by John Harrington, the Chief Product Officer at HighByte, focused on product management, customer and partner success, and company strategy. His areas of responsibility include market research, customer use cases, product priorities, go-to-market, and financial planning.