* CareNet 11

Final presentation CSD Fall 2011 Project owner Bjrn Pehrson Project coaches Bruce Zamaere Erik Eliasson Herv Ntareme Siraj Rathore Team Members Bowei Dai Elis Kullberg Hannes Junnila Nur Mohammed Rashed Siddharth Madan Vasily Prokopov

Team member overview

Elis Kullberg

Bowei Dai

Siddharth Madan

Hannes Junnila

Nur Mohammad Rashed

Vasily Prokopov

CareNet 11

Agenda
Introduction to CareNet fall 11 Network operations Residential Gateways Virtualization analysis Field testing Other contributions Management issues Further work

CareNet 11

Introduction: CareNet-SE

Gateway improvement

Field testing

Collaboration

Media / Bifrost wiki

Infrastructure operation Primary objectives:

Successful take-over, operation and hand-over of the CareNet network and services. Development and field testing of the CareNet Home Extension, including residential gateways, HDVC clients, sensor gateways, health portal access and associated usecases.

CareNet 11

NOC: development vectors

SSVL KTH-LAN CSC
.9 AS 8973

Edge Router 1

AS 2839

19

EBGP 130.237.0.0/29 2001:6b0:1:2::/64

Robustness

.1 ::1

Edge Router 2

.1

2.1

IB GP

6.1 2

4.8

/30
AS 8973

19

6.2 2.3

2 .0/ 50

eth3

Edge Network
AS 8973

eth
.10

.4 ::4

h7 et
.35

Availability

Bifrost router @ Valhallavgen

eth 4:1, Ventspils 192.16.126.57/29

Core Network

h2 et .1 ::1

VR OSPF area 0

eth
.5 ::1

0 /3 4 .0 :/6 26 1 : .1 32: 16 : 2 . b0 19 1:6 0 20

1 20 92.1 01 6 . :6b 12 0:3 6.4 2:3 /30 ::/6 4

Security
KR

h0 et

.2 ::2

et h0
.6 ::2 HR

Bifrost router @ Kista

.65 ::1 .177 ::1

Bifrost router @ Huddinge

2 eth

eth3

eth2

.161 ::1

eth4

eth

.145 ::1

.129 ::1

192.16.126.64/26 2001:6b0:32::/64

192.16.126.176/28 2001:6b0:32:7::/64

192.16.126.160/28 2001:6b0:32:6::/64

...

Servers

Scalability
Domain SIP Mgmt. VPN Gwupd. Portal Log OMRS HDVIPER devices and IP phones

192.16.126.144/28 2001:6b0:32:5::/64

192.16.126.128/28 2001:6b0:32:4::/64

Efficiency

Access Network Kista

Clients
.80 ::0000 .126 ::0fff

Access Network Huddinge

NOC: achievements

CareNet 11

NOC: IPv6
Internet IPv6 SUNET IPv6 KTH-LAN IPv6

OSPFv3 as IGP in the core network

DHCPv6 in access networks IPv6 EBGP peering with neighbors

AS 2839

::1
2001:6b0:1:2::/64

Edge Router 2 Announcing: 2001:6b0:32::/49

Receiving: ::/0

AS 8973

EBGP

eth3

::4

h2 et

VR

eth

Bifrost router @ Valhallavgen

1

::1 OSPFv3 ::1 area 0.0.0.0

:/6 1: 2: 4

20

20

b :6 01

3 0:

01 :6b

0:3

2:3

::/6 4

CareNet 11

NOC: network uptime

A measurable objective was set to 99,9% Has it been acheivied? Almost, the uptime was 99,6%

Gateway: Background

10

* CareNet

Gateway: Background

ISP

11

* CareNet

Gateway: Background

Backup ISP

Primary ISP

12

* CareNet

Gateway: Background

Backup ISP

Primary ISP

13

* CareNet

Gateway: Background

Backup ISP

Primary ISP

14

* CareNet

Gateway: Background

15

* CareNet

Gateway: Connectivity
Host Namespace CareNet Container
tun0 Mudp OpenVPN

br0 eth3

Internet Container

Iptables masquerade
eth0 eth1 eth2 eth3 eth4 eth5 ppp0 eth0 eth1 eth2 eth3 eth4 eth5

eth0 eth1 eth2 eth3 eth4 eth5 ppp0 HDVC Sensor GW Health Portal Home Network ISP 1 ISP 2 Wireless ISP

Kernel

16

CareNet 11

Gateway: WebGUI

Similar to web interface on home routers Visualizes all relevant configuration and debug information

Requires low technical literacy

17

CareNet 11

Gateway: Virtualization issue tree

LXC isolation

Resource isolation

Data isolation

Namespace isolation

Error isolation

CPU

Memory

Network

Disk

File system isolation

File isolation

PID isolation

Network isolation

Proc/API (kernel) isolation

Unintentional error

Intentional error

CPU rate limiting

CPU sharing for multi-core environment

Memory utilization limiting

Memory access isolation

Rate limiting

I/O utilization

Bus error

Resources inefficient error

Race conditions

Driver error

File system error

Synthetically with unintentional errors

Break namespaces

Access violation

Unaligned access

Driver programming error

Provide an invalid buffer address

Do Linux Containers provide the level of isolation needed for the Residential Gateways?
- Overall research question

18

CareNet 11

Gateway: Virtualization issue tree

LXC Resource isolation

CPU

Memory

Network

Disk

CPU rate limiting

CPU sharing for multi-core environment

Memory utilization limiting

Memory access isolation

Rate limiting

I/O utilization

Do Linux Containers provide the level of isolation needed for the Residential Gateways?
- Overall research question

19

CareNet 11

Gateway: Virtualization issue tree

LXC Resource isolation

CPU

Memory

Network

Disk

CPU rate limiting

CPU sharing for multi-core environment

Memory utilization limiting

Memory access isolation

Rate limiting

I/O utilization

Do Linux Containers provide the level of isolation needed for the Residential Gateways?
- Overall research question

20

CareNet 11

Gateway: Virtualization platform

Bifrost Host

Container 1 Packet generator eh0: 10.0.1.2 eth0:0 10.0.1.1 routes 10.0.3.0/24 nexthop 10.0.1.2 eh1: 10.0.3.2 eth0:0 10.0.3.1 routes 10.0.1.0/24 nexthop 10.0.3.2 Packet receiver

Physical interface (2 VLANS)

Container 2

Physical interface (2 VLANS)

eth0:1 10.0.2.1 routes 10.0.4.0./24 nexthop 10.0.2.2

eh0: 10.0.2.2

eh1: 10.0.4.2

eth0:1 10.0.4.1 routes 10.0.2.0/24 nexthop 10.0.4.2

21

CareNet 11

Gateway: Virtualization example results

CPU load (%) as a function of Time (s)
120

100

Scheduling limits are enforced!

80

CPU Core load %

60
%CPU Container 1

%CPU Container 2
40

20

0 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 101 105 109 113 117 121 125 129 133 Time (s)

22

CareNet 11

Gateway: Sensor gateway topology

23

CareNet 11

Gateway: Sensor gateway topology

24

CareNet 11

Gateway: Sensor gateway topology

Sensor 1 Sensor 2 Sensor N

JSON over TCP

server.py

Round Robin Databases API_xml.py API_png.py

HTTP API

25

CareNet 11

Gaetway: Sensor Gateway Client

Application overview

Functionality specification
Heart Rate monitoring SPO2 monitoring Graphical User Interface

Audio Alarms Fully Open Source

Meets Karolinska requirements

Platform overview

26

CareNet 11

Gaetway: Sensor Gateway Client

Application overview

Functionality specification
Heart Rate monitoring SPO2 monitoring Graphical User Interface

Audio Alarms Fully Open Source

Meets Karolinska requirements

Platform overview

27

CareNet 11

Gaetway: Sensor Gateway HTTP API

Parameters - Time Span - Data type

- Patient ID
Return Data - PNG output - XML output

28

CareNet 11

Infrastructure Testing

In-House Testing

Custodian Testing

Hospital Testing

Testing

29

CareNet 11

In House Testing: Overview

Where

In the fully controlled TSLab environment

How

Via well established scientific methodology

Outcome

See next slide

30

CareNet 11

In House Testing: Outcomes

Test name HDVC connectivity and service test
Residential gateway pull update test UMTS roll over IPv6 connectivity test M-UDP test Virtualization Test (CPU and Memory)

Result Achieved Yes

Yes Yes Yes Yes Yes

31

CareNet 11

Custodian Testing: Overview

Where

At homes of six custodians Located in different parts of Stockholm area Different internet setups and internal networks

How

Initially manual data-collection via SSH Later automated data-collection via MRTG & Cacti See next slide for example

Outcome

Automated data-collection deployed On average two active custodians Numerous bug-fixes and redesigns

32

CareNet 11

Custodian Testing:MRTG Data & Analysis

Figure 1 Temperature in the CareNet room

Figure 2 CPU usage for a Residential Gateway

33

CareNet 11

Hospital Testing: Overview

At the MT Analys lab at Karolinska in Huddinge Fully integrated into the Karolinska hospital IT infrastructure Semi-controlled environment

Where

How

Phase 1: RG installation and field testing Phase 2: HDVC installation and field testing Phase 3: Update and handover to Carenet Spring 2012 team

Outcome

Reassurance that CareNet infrastructure compatible with Karolinska infrastructure Numerous bug-fixes in HDVC systems and RGs

34

CareNet 11

Hospital Testing: Outcomes

35

CareNet 11

Future Work and Recommendation

The primary aspects of the Carenet system that need to be tested are Sensor Gateway HDVC The primary aspects of each system that should be focused are: Robustness Scalability Reliability

36

CareNet 11

Other contributions: Bifrost wiki screencap

bifrost.slu.se

CSD course documentation

Mailing list

37

CareNet 11

Project management: SCRUM routine

Overall adoptions for CareNet-team 1 week sprints 2x meetings per week Combined sprint review/plan meeting

Outcome Good experiences Used week 8 and forward More appropriate than Kanban?

38

CareNet 11

Project management: SCRUM burndown

SCRUM Burndown Chart, project week 8-19

1400 1200

Hours remaining

1000 800 600 400 200 Measured Burn Down Planned Burn Down

0 0 5 10 15 20 25
Time (days)

30

35

40

39

CareNet 11

Project management: Time spent per member

450 400
350 300 MonAMI 250 200 150 100 50 0 Boewi Dai Elis Kullberg Hannes Junnila Siddharth Madan Nur Mohammad Rashed Vasily Prokopov Media Field Testing NOC Gateway

Time per member and work package outcome until week #18

Management

40

CareNet 11

Project management: Best team ever!

41

CareNet 11

Project management: Expectations on Spring team

Item
Operate infrastructure

Priority
High

Tech. Complexity
Medium

Continue field testing RGs

Solve any found RG bugs

High
High

Medium
High

42

CareNet 11

Question and Answer

43

CareNet 11