The

Internet of Things:
Sensor Data Management

Course website: h8p://www.cs.unibo.it/projects/iot/

Prof. Luciano Bononi Prof. Marco Di Felice

luciano.bononi@unibo.it marco.difelice3@unibo.it

MASTER DEGREE IN COMPUTER SCIENCE

DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY
IoT Sensor Data Management

IoT SENSOR DATA MANAGEMENT

2
L. BONONI, M. Di FELICE, DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY
IoT Sensor Data Management
The word big-data is currently used to idenSfy: (i) data-
sources with specific characterisScs, as well as (ii) novel
technologies to manage the data.
CHARACTERISTICS OF BIG-DATA

q Cannot be managed using convenSonal technologies of informaSon systems

q  Volume à order of Petabytes
q  Velocity à data produced at high rate
q  Variety à heterogeneous data (text, image, video, etc)

IoT SENSOR DATA MANAGEMENT

3
L. BONONI, M. Di FELICE, DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY
IoT Sensor Data Management
² IoT & Big-data are two sides of the same coin!
q  Large-scale IoT deployments can produce huge amounts of data
q  IoT is about data, services, connecSvity: data are gathered by
objects, transfered, analyzed, and traslated into services

IoT SENSOR DATA MANAGEMENT

4
L. BONONI, M. Di FELICE, DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY
 IoT Sensor Data Management
² IoT & Big-data are two sides of the same coin!
(DIGITAL) SYSTEM MODEL

DATA DATA MODEL

SENSING
FILTERING ANALYSIS BUILDING

Raw sensor data

REAL SYSTEM measure how the UNDERSTANDING
CorrelaSon PREDICTING
system is working System
among system
parameters behaviour PLANNING
and between at Sme What-If
Inputs/outputs t+1 analysis

IoT SENSOR DATA MANAGEMENT

5
L. BONONI, M. Di FELICE, DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY
IoT Sensor Data Management
² IoT & Big-data are two sides of the same coin!
EXAMPLE: ENERGY@HOME TELECOM ITALIA (research project)
h8p://www.energy-home.it ClassificaRon
(i.e. idenSfy
the appliance)
Profiling
(i.e. idenSfy
users’ habbits)
²  5 smart plugs for house PredicRng
²  Around 200 installaSons in RAW SENSOR (i.e. predict energy consumpSon)
private houses DATA TRACES
Scheduling
²  1 sampling every 2 minutes
(i.e. intelligent ON/OFF schedule)

IoT SENSOR DATA MANAGEMENT

6
L. BONONI, M. Di FELICE, DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY
IoT Sensor Data Management
² Time-series à Sequence of Rmestamp plus values

IoT SENSOR DATA MANAGEMENT

7
L. BONONI, M. Di FELICE, DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY
IoT Sensor Data Management
² Time-series à Sequence of Rmestamp plus values

q  Data are immutable.

q  WriSng in append.
q  Reading conRguous sequence of samples data.
q  Highly compressible data.
q  DeleRng usually across large Sme period
q  High precision for short period of Sme.
q  Single value is not so important
IoT SENSOR DATA MANAGEMENT
8
L. BONONI, M. Di FELICE, DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY
IoT Sensor Data Management
²  RelaRonal Database Management Systems (RDBMS)
q Based on the relaSonal model first proposed by
Edgar F. Codd (1970)
q Employ SQL language
q Support ACID properSes
q Scheme-based (structured) database.
q Components: Tables (relaSons), primary keys,
foreign keys, NULL values.
IoT SENSOR DATA MANAGEMENT
9
L. BONONI, M. Di FELICE, DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY
IoT Sensor Data Management
²  Time-series implemented on RDMBS

PROBLEMS

²  Scalability (i.e. need to store large
amount of Sme-series data)
²  Performance (e.g. support for range-
based operaSons)
²  Aymmetric CRUD operaSons

IoT SENSOR DATA MANAGEMENT

10
L. BONONI, M. Di FELICE, DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY
IoT Sensor Data Management
²  NOSQL Database Management Systems
q Set of tools and logic models, alternaSve or
complementary to RDBMS (noREL).
q Support BASE properSes.
q Do not employ the SQL language.
q Scheme-less (un/semi-structured) database.
q Families: Key-values DB, Document-based DB,
Column-based DB, Graph-based DB.
IoT SENSOR DATA MANAGEMENT
11
L. BONONI, M. Di FELICE, DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY
IoT Sensor Data Management
²  Time-series Database

IoT SENSOR DATA MANAGEMENT

12
L. BONONI, M. Di FELICE, DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY
IoT Sensor Data Management
²  Time-series Database à Dedicated DBMS opSmized
for managing large volumes of Sme-series data
q  OpSmized data storage and sharding
q  OperaSonal support (e.g. range-based queries)
q  Time-granularity management
q  Time-series analyScs and mining

IoT SENSOR DATA MANAGEMENT

13
L. BONONI, M. Di FELICE, DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY
IoT Sensor Data Management
²  InfluxDB (hYps://www.influxdata.com)
q  Open-source Sme-series database (InfluxData)
q  Wri8en in GO language
q  SQL-like query language (InfluxQL)
q  Command Line Interface (CLI) and HTTP APIs
q  Support for distributed deployments
q  IntegraSon with Sme-series tools for data analyScs and
visualizaSon (e.g. Grafana)

IoT SENSOR DATA MANAGEMENT

14
L. BONONI, M. Di FELICE, DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY
IoT Sensor Data Management
²  InfluxDB (hYps://www.influxdata.com)
q  Time-Structured Merge Tree (TSM) à data structure used to
contain sorted, compressed series data.

q  Time Series Index (TSI) à address millions of unique Sme

series, regardless of the amount of memory on the server
hardware.
q  AutomaSc downsampling and data rentenSon procedures.

IoT SENSOR DATA MANAGEMENT

15
L. BONONI, M. Di FELICE, DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY
IoT Sensor Data Management
²  InfluxDB (hYps://www.influxdata.com)
q  Timestamp à RFC3339 UTC format (yyyy-mm-ddThh:mm:ssZ)
q  Field keys à string metadata, similar to column name
q  Field values à actual measured data (any type)
q  Tag-sets à opSmal, extra-informaSon about the measurements
q  Tag keys à string meta-data, similar to field keys
q  Tag values à string values
q  Measurement à Container to hold the Smestamps, fields and
tags (similar to a table in a RDBMS)
IoT SENSOR DATA MANAGEMENT
16
L. BONONI, M. Di FELICE, DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY
 Source: h8ps://www.linkedin.com/pulse/introducSon-influxdb-kristof-bruylants

IoT Sensor Data Management

²  InfluxDB (hYps://www.influxdata.com)

q  Series à collecSon of data-points, containing:

²  Measurement
²  Tag-sets
²  RetenRon policy à period that datapoints are being stored
in InfluxDB, which is called DURATION but also the number of
versions that should be kept on the cluster, as REPLICATION.

IoT SENSOR DATA MANAGEMENT

17
L. BONONI, M. Di FELICE, DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY
 Source: h8ps://www.linkedin.com/pulse/introducSon-influxdb-kristof-bruylants

IoT Sensor Data Management

²  InfluxDB (hYps://www.influxdata.com)

IoT SENSOR DATA MANAGEMENT

18
L. BONONI, M. Di FELICE, DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY
 Source: h8ps://www.linkedin.com/pulse/introducSon-influxdb-kristof-bruylants

IoT Sensor Data Management

²  InfluxDB (hYps://www.influxdata.com)
q  Command-Line Interface (CLI)
user@mypc:$influx
Connected to http://localhost:8086 version 1.5.2
InfluxDB shell version: 1.5.2

q  HTTP-based API
curl -i -XPOST http://localhost:8086/query --data-urlencode "q=CREATE
DATABASE mydb”
curl -i -XPOST 'http://localhost:8086/write?db=mydb' --data-binary
'cpu_load_short,host=server01,region=us-west value=0.64 1434055562000000000'

IoT SENSOR DATA MANAGEMENT

19
L. BONONI, M. Di FELICE, DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY
 Source: h8ps://docs.influxdata.com/influxdb/v1.5/tools/shell/#write-data-to-influxdb-with-insert

IoT Sensor Data Management

²  InfluxDB (hYps://www.influxdata.com)
q  Basic Commands
² CREATE DATABASE name
² USE DATABASE name
² SHOW DATABASES
² CLEAR DATABASE name
² CREATE RETENTION POLICY name ON measurement
DURATION 1d REPLICATION 1
² INSERT treasures,captain_id=pirate_king value=2

IoT SENSOR DATA MANAGEMENT

20
L. BONONI, M. Di FELICE, DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY
 Source: h8ps://docs.influxdata.com/influxdb/v1.5/query_language/data_exploraSon/

IoT Sensor Data Management

²  InfluxDB (hYps://www.influxdata.com)
q  InfluxQL language for data exploraRon
SELECT <field_key>[,<field_key>,<tag_key>] FROM <measurement_name>[,<measurement_name>]
SELECT * FROM "h2o_feet"

SELECT_clause FROM_clause WHERE <conditional_expression> [(AND|OR) <conditional_expression> [...]]

SELECT * FROM "h2o_feet" WHERE "water_level" > 8

SELECT_clause FROM_clause [WHERE_clause] GROUP BY [* | <tag_key>[,<tag_key]]

SELECT MEAN("water_level") FROM "h2o_feet" GROUP BY "location"

IoT SENSOR DATA MANAGEMENT

21
L. BONONI, M. Di FELICE, DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY
IoT Sensor Data Management
²  Grafana (hYps://grafana.com)

IoT SENSOR DATA MANAGEMENT

22
L. BONONI, M. Di FELICE, DEPARTMENT OF COMPUTER SCIENCE AND ENGINEERING, UNIVERSITY OF BOLOGNA, ITALY