aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • Programming
  • Software Engineering

Edge Stream Data Processing Based On KubeEdge And Kuiper

  • aster.cloud
  • August 3, 2021
  • 4 minute read

Guest post by KubeEdge Maintainers

Introduction: KubeEdge is an open-source edge computing platform. Based on the native container orchestration and scheduling capabilities of Kubernetes, KubeEdge achieves functionalities such as cloud-edge synergy, edge computing, edge device management, and edge autonomy. KubeEdge is also widely implemented in scenarios such as 5G MEC and AI cloud-edge synergy by using plug-ins.


Partner with aster.cloud
for your next big idea.
Let us know here.



From our partners:

CITI.IO :: Business. Institutions. Society. Global Political Economy.
CYBERPOGO.COM :: For the Arts, Sciences, and Technology.
DADAHACKS.COM :: Parenting For The Rest Of Us.
ZEDISTA.COM :: Entertainment. Sports. Culture. Escape.
TAKUMAKU.COM :: For The Hearth And Home.
ASTER.CLOUD :: From The Cloud And Beyond.
LIWAIWAI.COM :: Intelligence, Inside and Outside.
GLOBALCLOUDPLATFORMS.COM :: For The World's Computing Needs.
FIREGULAMAN.COM :: For The Fire In The Belly Of The Coder.
ASTERCASTER.COM :: Supra Astra. Beyond The Stars.
BARTDAY.COM :: Prosperity For Everyone.

This article mainly shares practices of edge stream data processing based on KubeEdge and Kuiper.

Kuiper – A stream processing product at the edge

Initiated in early 2019 and released in October 2019, Kuiper has been iteratively developed to be a classic stream processing architecture.

This product is designed to enable the stream processing that runs in the cloud, such as Spark or Flink jobs, to run at the edge.

Architecture Diagram of Kuiper

1611021593(1)

The overall architecture diagram consists of three parts. Sources on the left indicate the data sources, which can be the MQTT macOS broker at a certain edge of KubeEdge, or files, windows, or databases.

Sinks on the right indicate the storage locations after data processing. The target locations can be MQTT, files, databases, or HTTP services.

The middle part is divided into several layers. The top layer is the data service logic processing layer, which provides SQL statements and rule parser. SQL processors process and convert them into SQL plans. The middle layer contains the streaming runtime and SQL runtime, which are used to run the final plan. The bottom is the storage layer, which is used to store outgoing messages.

Application Scenarios of Kuiper

Real-time stream processing at the edge

Read More  Here's What You Need To Know To Get Started With Ansible

Rule engine to customize rules for alarm and message forwarding

Data format and protocol conversion for edge and cloud data to integrate IT and OT

KubeEdge and Kuiper Integration

1611024820(1)

Parts of the architecture

Kuiper is installed at the downstream of the KubeEdge MQTT broker. Both of them run at the edge. There are different underlying mappers, that is, different types of protocols. The edge MQTT broker is used to exchange messages.

Data Processing Types:

  • Obtaining the type definition from the device model file
  • Converting data to the types supported by Kuiper
  • Using the schema-less flow definition for creating a stream

The supported data types are int, string, bool, and float.

KubeEdge Model File and Configuration

The following figure shows parts of the configuration file, including the device name, property, name, data type, and description.

Parts of the configuration file

  1. Save the device model file.
  2. Configure the model file information in ect/mqtt_source.yaml.
  • KubeEdgeVersion: not used yet, reserved for future model files of different versions
  • KubeEdgeModelFile: path of the model file
  1. Deliver the configurations as a ConfigMap and save them to the related directory.

Using Kuiper

  1. Define the stream, which is similar to defining a table in a database.

DATASOURCE=”$hw/events/device/+/twin/update” is a topic already defined in KubeEdge.

  1. Define and submit the rule.
1611026731(1)
  • Implement the service logic with SQL and send the results to the specified target.
  • The following SQL operations are supported:

SELECT/FROM/WHERE/ORDER

JOIN/GROUP/HAVING

4 types of time windows + 1 count window

More than 60 SQL functions

  1. Run Kuiper.

Deploying Kuiper in KubeEdge

Method 1: Using Kuiper-Kubernetes-tool

This tool independently runs in containers to execute the command configuration file delivered as a ConfigMap.

  • For specifying IP address and port number of Kuiper in the configuration file
  • The directory where the command file is located
Read More  APIS IT Modernizes Legacy Applications And Projects With Red Hat OpenShift

Run the file. The tool periodically scans the file and executes the command.

Method 2: Using Kuiper-manager and Cloud-Edge Synergy Management Console

You can also use the management console to manage nodes on which Kuiper runs.

For example, when Kuiper is running in the IoV box that connects many vehicles, you can use Kuiper-manager to connect all instances and update rules in a unified manner.

First, install plug-ins. For example, if you want to connect to an unsupported source, you can write a plug-in with our guides and install it. After that, you can use it on an Android plug-in interface.

Next, create a stream definition.

Specify a location or path for saving data to the file system.

Finally, compile the rule on a visualized editing page.

Use Case: Industrial Internet Big Data Center

This is a typical application scenario. Kubernetes and CloudCore are deployed in the cloud to deliver rules to Kuiper through a management channel. Kuiper is deployed on the MQTT broker to define and clean data. Currently, there are two channels. The first channel is used to send processed messages to the cloud MQTT broker. The second channel is used to store data to the Influxdb database for local data persistence. Some third-party applications at the edge can directly invoke data from Influxdb for visualization. At the bottom layer, mappers are used to connect different data.

Application Scenarios of the Rule Engine in Kuiper

The built-in rule engine LF EdgeX Foundry was officially released in the Geneva version in April 2020.

1611041439(1)
1611041512(1)

Use Case: Data Format Conversion for Interconnection between Heterogeneous Systems

To realize data exchange with IT systems such as ERP and MES, flexible extensions are provided to easily collect and process data, even the heterogeneous data, using SQL built-in functions or extended functions. After the data is processed, the analysis result can be converted using the data template of the sink to flexibly adapt to various target systems that require different data formats and protocols. For example, for sending an instruction to control the device when the temperature is higher than 30°C to both the device and your WeChat, though different APIs and data are required, the rules are the same. Therefore, you can specify different topics in the same rule in data to send data without complex programming. SAP NetWeaver RFC SDK is used to read data from SAP, process and convert the data, and send the data to other heterogeneous systems.

Read More  What Can Confidential Computing Do For The Kubernetes Community?

Performance Data

  1. Kuiper supports concurrent running of thousands of rules.
  • 8,000 rules x 0.1 messages/second/rule, with the total TPS of 800 messages/second
  • Rule definition

Source: MQTT

SQL: select temperature from source where temperature>20 (90% data is filtered)

Target: logs

  • Configuration

AWS: 2 cores x 4 GB

Ubuntu

  • Resource usage

Memory: 89% ~ 72%; 0.4 MB/rule

GPU: 25%

  1. AWS t2.micro has 10k+/s message throughput.

For more details about KubeEdge, visit

Github: https://github.com/kubeedge/kubeedge

Website: https://kubeedge.io/en/


For enquiries, product placements, sponsorships, and collaborations, connect with us at [email protected]. We'd love to hear from you!

Our humans need coffee too! Your support is highly appreciated, thank you!

aster.cloud

Related Topics
  • CloudCore
  • KubeEdge
  • Kubernetes
  • Kuiper
You May Also Like
aster-cloud-erp-bill_of_materials_2
View Post
  • Software
  • Software Engineering

What is an SBOM (software bill of materials)?

  • July 2, 2025
aster-cloud-sms-pexels-tim-samuel-6697306
View Post
  • Programming
  • Software

Send SMS texts with Amazon’s SNS simple notification service

  • July 1, 2025
aster-cloud-website-pexels-goumbik-574069
View Post
  • Programming
  • Software

Host a static website on AWS with Amazon S3 and Route 53

  • June 27, 2025
View Post
  • Software Engineering
  • Technology

Claude 3.7 Sonnet and Claude Code

  • February 25, 2025
View Post
  • Engineering
  • Software Engineering

This Month in Julia World

  • January 17, 2025
View Post
  • Engineering
  • Software Engineering

Google Summer of Code 2025 is here!

  • January 17, 2025
View Post
  • Software Engineering

5 Books Every Beginner Programmer Should Read

  • July 25, 2024
Ruby
View Post
  • Software Engineering

How To Get Started With A Ruby On Rails Project – A Developer’s Guide

  • January 27, 2024

Stay Connected!
LATEST
  • 1
    Building secure, scalable AI in the cloud with Microsoft Azure
    • July 5, 2025
  • 2
    Turns out OpenAI is the customer behind Oracle’s mysterious $30 billion cloud deal
    • July 3, 2025
  • aster-cloud-erp-bill_of_materials_2 3
    What is an SBOM (software bill of materials)?
    • July 2, 2025
  • aster-cloud-sms-pexels-tim-samuel-6697306 4
    Send SMS texts with Amazon’s SNS simple notification service
    • July 1, 2025
  • Camping 5
    The Summer Adventures : Camping Essentials
    • June 27, 2025
  • aster-cloud-website-pexels-goumbik-574069 6
    Host a static website on AWS with Amazon S3 and Route 53
    • June 27, 2025
  • 7
    A looming hyperscaler exodus? UK IT leaders are thinking of ditching US cloud providers – here’s why
    • June 26, 2025
  • Prioritize security from the edge to the cloud
    • June 25, 2025
  • 6 edge monitoring best practices in the cloud
    • June 25, 2025
  • Genome 10
    AlphaGenome: AI for better understanding the genome
    • June 25, 2025
about
Hello World!

We are aster.cloud. We’re created by programmers for programmers.

Our site aims to provide guides, programming tips, reviews, and interesting materials for tech people and those who want to learn in general.

We would like to hear from you.

If you have any feedback, enquiries, or sponsorship request, kindly reach out to us at:

[email protected]
Most Popular
  • 1
    There’s a ‘cloud reset’ underway, and VMware Cloud Foundation 9.0 is a chance for Broadcom to pounce on it
    • June 17, 2025
  • 2
    ‘This was a very purposeful strategy’: Pure Storage unveils Enterprise Data Cloud in bid to unify data storage, management
    • June 18, 2025
  • 3
    Pure Accelerate 2025: All the news and updates live from Las Vegas
    • June 18, 2025
  • Oracle adds xAI Grok models to OCI
    • June 17, 2025
  • What is cloud bursting?
    • June 18, 2025
  • /
  • Technology
  • Tools
  • About
  • Contact Us

Input your search keywords and press Enter.