Iot Stream processing and

analytics in the fog

Umakant Singh
b221063@iiit-bh.ac.in
Branch - CE
Introduction
● Fog computing is a decentralized computing infrastructure that extends
computing resources to the edge of the network .
● Increasing interest from both academia and industry in fog architecture due
to low-latency , resilient and cost-effective services from initiatives like
OpenFog consortium .
● It is relevant in addressing the challenges faced by traditional cloud
architecture such as latency , capacity and cost-effectiveness.
● It enhances the efficiency , reliability and security of Iot applications by
distributing computing resources strategically through the network .
Types of Fog Streaming Applications

● Iot stream Query and Analytics - Real -time analysis of sensor data at the
edge for immediate insights and decision making .
● Real-time Event Monitoring -Monitoring and alerting for events like traffic
congestion or machine performances issues in real time.
● Networked control Systems for industrial automation - localized control and
monitoring of industrial processes for improved efficiency and reduced
latency.
● Real-time Mobile crowdsensing - Processing sensor data from mobile devices
in real time for environmental and traffic monitoring .
 (a) Cloud (b} ud

kOg i I ecísion
-- i
i
ú1 i

rJimr‘nsinn tdt
CJoud Cloud
Furictiunal
' m I Or troll - - Orltroli FO’ partitton

,i
Life cycle of Fog Data streams

● Data creation - Data Streams are generated by sensors , devices , or

users within the application , producing the raw data for further
processing .
● Data collection - The raw data is gathered from diverse sources ,
aggregated , and prepared for processing in the fog computing
environment.
● Data processing - The collected data is analyzed , transformed and
enriched with contextual information to derive meaningful insights and
actions .
● Application Integration - processed data is integrated into the application
to enhance decision making, automate tasks , and improve user
Workflow of Fog stream processing using DAG

● Dag model - Represents stream processing tasks with vertices as

processing elements(PEs).
● Fog stream processing tasks - PEs execute tasks like
categorization , segmentation , etc , based on data and
application needs .
● Examples - Gigasight and event detection Systems model workflows with
DAGs for efficiency .
● Parallel computations - Apache storm uses Dag to enable parallel
processing , with spouts producing and bolts consuming streams for
efficiency .
 Steamquey É0üI1tÏI1g
6ggf9g6tÏ0I1g,r0t9ÜÏ0fl,
seledi0h, J0iF
CollflÖl0fi É6t9g0fÏZütÏ0I1, t9gITlghtdtÏOI1,
de-interlacing, filtering,
c0mprestÏ0F, füSÏ0F

****** An0maly deterti0F, clustering

ttream mining summarization, classificatÏ0h,
*!* regressi0n
DE SOlVing, err0r estimati0h,
Netr0flefi parameter turning, m0del
Pr0cessing ë0ntr0l
identifica{Ï0h
gualitye2imatiOn,üskpricing
Fog Data Streaming Architecture

It includes six layers :-

● Application layer - Defines the objectives and logic of fog streaming jobs .

● Processing Layer - Executes application- specific processing tasks using

real- time stream processing engines like apache storm or spark streaming
.

● Data management layer - Handles data storage and organization , including

file systems , databases and data caches .
continued…

● Resource management layer - focuses on the utilisation and scheduling

of virtualized system resources like network bandwidth , CPUs, memory ,
and storage.

● Virtualization layer - Manages the configuration and virtualization of

system hardware resources using techniques like openstack and SDN.

● Physical network layer - Represents the heterogeneous and dynamic

physical network Infrastructure supporting fog data streaming .
Applications

Data
management MySQL
Resource
management YART¥ #QFTWARE

openst Ck
penFlow

Aggregation switch ' 'r End devices Fog servers

CDre switch
router
Physical
netwnrk
infrastructure ’Coré
lnterne

*ToR: Top of rack

Design space of fog data Streaming

The design space of fog data streaming considers viewpoints of system ,

data , optimization , and human.New Design challenges and issues with
existing techniques are addressed .
System ViewPoint

Challenges
● In fog environment , scalability becomes complex due to the distributed
nature of resources .
● Reliability is crucial to ensure uninterrupted operation , especially in
dynamic and heterogeneous environments.
● Effective resource management is essential to optimize resource allocation
and utilization across fog nodes .
Continued…

Issues
● Integrating fog systems with existing infrastructure requires
seamless interoperability and compatibility .

● Managing dynamicity and heterogeneity poses challenges in adapting to

changing network conditions , device capabilities and user mobile
patterns .
Data Viewpoint

Challenges
● Data heterogeneity arises from diverse sources, formats, and protocols in fog
environments, requiring robust mechanisms for data integration and normalization.

● Ensuring data interoperability is crucial for seamless communication and

collaboration across fog nodes and with external systems. Protecting data privacy
and security becomes challenging in fog environments where data may be distributed
across multiple nodes and transmitted over wireless networks.
Issues

● Processing large volumes of streaming data in real-time requires efficient

data processing and analytics techniques to extract valuable insights.

● Maintaining data quality and consistency is essential for

reliable decision-making and ensuring the accuracy of analytics
results.

● Enabling effective data visualization and interpretation tools enhances

user understanding and facilitates decision-making processes.
Optimization viewpoint

Challenges
● Optimizing resource usage involves dynamically allocating resources based on workload
demands and environmental conditions to ensure efficient utilization.

● Minimizing latency requires reducing processing and communication delays

through optimization techniques such as task scheduling and data caching.

● Improving energy efficiency is critical for prolonging the battery life of edge devices
and reducing overall energy consumption in fog environments.
Issues

● Balancing trade-offs between performance, cost, and energy consumption

is challenging due to conflicting objectives and constraints.

● Adapting to dynamic environmental changes, such as variations in workload

intensity and network conditions, requires flexible optimization strategies
that can adjust resource allocation and configuration dynamically.
Human viewpoint
Challenges
● Enhancing user interaction involves designing intuitive interfaces and user experiences
that cater to diverse user needs and preferences.

● Ensuring usability requires considering factors such as accessibility,

learnability, and efficiency in interface design.

● Fostering trust and transparency involves providing users with visibility into how their
data is collected, processed, and used, as well as offering mechanisms for user control and
consent.
Issues

● Addressing concerns related to data ownership and privacy requires

implementing robust data governance and access control mechanisms
to protect user data from unauthorized access and misuse.

● Building trust with users involves establishing clear communication

channels and providing timely feedback and support to address user
concerns and inquiries.
Conclusion

● This study delves into data stream processing and analytics within fog
architecture, focusing on four typical fog streaming applications.

● Through these applications, we uncover the commonalities and multidisciplinary

nature of fog streaming research. Our exploration extends to discussing fog
streaming models, architecture, and the opportunities and challenges for future
design.

● With the growing significance of network edge and stream processing,

their integration is poised to catalyze the development of fog data streaming
in both academic and industrial realms.
References
● M. Satyanarayanan et al., “Edge Analytics in the Internet of Things,” IEEE Pervasive Comput., vol. 14, no. 2, 2015, pp. 24–31.
● M. Chiang and T. Zhang, “Fog and IoT: An Overview of Research Opportunities,” IEEE Internet of Things J., vol. 3, no. 6, 2016, pp. 854–64.
● O. Diallo et al., “Distributed Database Management Techniques for Wireless Sensor Networks,” IEEE Trans. Parallel Distrib. Sys., vol. 26, no.
2, 2015, pp. 604–20.
● L. Canzian and M. Van Der Schaar, “Real-Time Stream Mining: Online Knowledge Extraction Using Classifier Networks,” IEEE Network, vol.
29, no. 5, 2015, pp. 10–16.
● R. A. Gupta and M.-Y. Chow, “Networked Control System: Overview and Research Trends,” IEEE Trans. Ind. Electron., vol. 57, no. 7, 2010,
pp. 2527–35.
● H. Zhang et al., “In-Memory Big Data Management and Processing: A Survey,” IEEE Trans. Knowl. Data Eng., vol. 27, no. 7, 2015, pp.
1920–48.
● S. Yang et al., “Distributed Optimization in Energy Harvesting Sensor Networks with Dynamic Innetwork Data Processing,”
Proc. IEEE INFOCOM, 2016, pp. 1–9.
● L. Yang et al., “A Framework for Partitioning and Execution of Data Stream Applications in Mobile Cloud Computing,” ACM
SIGMETRICS Performance Evaluation Review, vol. 40,no. 4, 2013, pp. 23–32.
● S. Wang et al., “Dynamic Service Migration in Mobile EdgeClouds,” Proc. IFIP Networking, 2015, pp. 1–9.
● L. Wang et al., “CCS-TA: Quality-Guaranteed Online Task Allocation in Compressive Crowdsensing,” Proc. ACM Ubicomp, 2015, pp.
683–94.
● P. Chen, S. Yang, and J. A. McCann, “Distributed RealTime Anomaly Detection in Networked Industrial Sensing Systems,” IEEE
Trans. Ind. Electron., vol. 62, no. 6, 2015, pp. 3832–42.
● W. S. Lasecki, C. D. Miller, and J. P. Bigham, “Warping Time for More Effective Real-Time Crowdsourcing,” Proc. ACM SIGCHI,
2013, pp. 2033–36.
● I.-H. Hou et al., “Asymptotically Optimal Algorithm for Online Reconfiguration of Edge-Clouds,” Proc. ACM Mobihoc, 2016, pp. 291–
300.
● J. Ghaderi, S. Shakkottai, and R. Srikant, “Scheduling Storms and Streams in the Cloud,” ACM Trans. Modeling and Performance Evaluation
of Computing Systems, vol. 1, no. 4, 2016, pp. 1–28.