Data ingestion from the network edge: An essential pillar in the intelligent data journey

Connectivity -> Ingestion -> Enrichment -> Insights -> Outcome

In the previous article, we discussed how connectivity is the foundation of digital and the first step of the physical-to-digital juxtaposition that drives every successful technology and business outcome.

The next step of this process is Ingestion or the consuming of data from the Edge.

In order to do that at scale and to make it repeatable, we must have a consistent mechanism to consume the data being created at the edge so we can move to the next step, Enrichment, that is used to enrich data to drive the actionable Insights required for our Digital Outcomes.

Ubiquitous and Universal

A common conversation around well-architected applications is their abilities to be:

  • Deployed without refactoring (same deployment regardless of hardware or location)
  • Remotely managed (configuration changes, health validation, updates, etc.)
  • Microservices architected with containerization
  • Scalable and Extensible
  • Secure

These capabilities are often only associated with workloads that live in “the Cloud”.

In reality, the Edge is generally a much bigger benefactor of this architecture as it does not have the same benefits of the massive connectivity, storage, and hands-on access afforded to the Cloud.

For these reasons, having a well-architected edge consumer mechanism, Ingestion enables the data digital journey to be far more achievable.

The Diversity of Edge Data and Communications

In the Physical world, we are fortunate to live in, there is a fantastic diverse set of languages and cultures – all beautiful and interwoven in wonderful ways.  There are also universal ways to communicate, hand motions, gestures, drawings, etc.

In the Digital world, we also have many languages and protocols that different systems use that are generally well-aligned to each individual use case (these were described in detail in the previous article).  However, many if not most of these, do not have the benefit of leveraging a universal consumer translator.

To enable a method of creating a correlated “sense” of the Physical World at the Digital Edge, we must be able to consume the diverse data streams that are created by a myriad of sensors and systems that communicate through different mediums, protocols, and message types.

Creating Sense from Diverse Senses

Automating the consumption of diverse streams of data at the edge, from deployment through ingestion, requires the knowledge of and experience with the sensors and systems that create data at the edge.

By enabling the correct software layer that consumes the information as well as having standards-based hardware for components that require a physical connection (described in detail in the previous article), we are able to build a “flow” of the data from the different sources to our consumer aggregation layer.

The templated “flows” that are constructed allow the universal edge consumer to gain access to the data, understand the message types, and ingest the data in a structured way that enables near real-time enrichment, the next step in the C-I-E-I-O process.

Michael Knight
Chief Technology Officer