Sunbird Components and its Architecture

by
FRG, IIT Bombay

April 2018

by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 1 / 22
Outline

1 Apache Cassandra 8 Sunbird Middleware

2 Elastic Search 9 Akka Actor Model
3 Logstash, Kibana, and Kafka 10 Akka Libraries and Modules
4 ELK 11 DUI Framework
5 Keycloak 12 Sunbird Architecture
6 Sunbird Core Services 13 DevOps Tools
7 Play Framework 14 References

by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 2 / 22
 Apache Cassandra

Apache Cassandra (3.10) - I

Based on combination of Architectures: Amazon Dynamo [6] and

Google BigTable [5]
Amazon Dynamo
Peer-to-peer key-value store architecture
Data replication using autosharding across multiple nodes.
Eventual Consistency
Gossip protocol to talk to its neighbors for failure detection
cluster membership management in an asynchronous way.
Google BigTable
Uses the master-slave architecture
Petabytes and real-time operations.
Data storage
1 commit log
2 memory store
Data persistence using Sorted String Table (SSTable).
Failure? commit logs can be replayed and merged with the SSTable.
by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 3 / 22
Apache Cassandra - II

Falls under the AP part of the CAP theorem

Each key-value data element is replicated across the cluster
gossip protocol
Hinted Handoff: allows the data to be managed by another node in
case of failure.
Solve inconsistency across the replicas: anti-entropy and read repair.
Anti-entropy: merging of SSTable.
Soft deletes, marked with a special flag
Deleted records: tombstone records.
Merkle data structure to represent the data state

by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 4 / 22
 Figure: Cassandra Write Path [7]

Figure: Cassandra Read Path [7]

by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 5 / 22
 Figure: Traditional RDBMS-based web-tiered architecture [1]

Figure: Cassandra-based web-tiered architecture [1]

by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 6 / 22
 Elastic Search

Elasticsearch (5.4.0)

Peer-to-peer architecture.
Real-time search capabilities
RESTful API
Uses Apache Lucene
Library to handle document indexing and searching
Scalable, light and highly performing
Uses inverted index mechanism
Elasticsearch document is a JSON object
Multi-node clustering mechanism (autosharding)
Fault tolerance using replicas.

by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 7 / 22
 Logstash, Kibana, and Kafka

Logstash, Kibana, and Kafka

Logstash
Data pipeline
Designed to efficiently process logs, events, and unstructured data
sources
Logs converted to a single common format
Kibana
visualizing any kind of structured and unstructured data stored in
Elasticsearch.
histograms, geomaps, pie charts, graphs, tables, etc.
Kafka
Distributed publish-subscribe messaging system
Persistent messaging designed with O(1) disk structures
High throughput (millions of messages per second)
Real time between producer and consumer
Critical to event-based systems.

by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 8 / 22
 ELK

ELK

ELK = Elasticsearch + Logstash + Kibana

Figure: ELK Stack [2]

by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 9 / 22
 Keycloak

Keycloak (3.2.1)

Protocols: OpenID Connect, OAuth 2.0, and SAML.

Identity and access management
Authenticates application using SSO (Single sign on and out)
Social login
Management of roles and permissions (Realm management)
Configuration in Sunbird [4]

by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 10 / 22
 Sunbird Core Services

Sunbird Core Services

Learner Service
Content Service
Actor Service
Player Service
Keycloak Service

by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 11 / 22
 Play Framework

Play Framework

Modular architecture
Stateless, fully RESTful
Uses Akka HTTP and Streams as its web server for asynchronous I/O
Built-in hot reloading
Native Scala support

by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 12 / 22
 Sunbird Middleware

Sunbird Middleware

Akka Actor Model Framework for scale and distribution

Components
badge
content
core
course
dashboard
organization
telemetry user
LMAX exchange Disruptor: high performance inter thread messaging
library.

by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 13 / 22
 Akka Actor Model

Akka Actor Model

Abstraction paradigm
Think about code in terms of communication
Stateful entities communicating (explicit message passing)
No low-level concurrency constructs like atomics or locks
Transparent remote communication between systems
Clustered High-availability architecture, scales in or out on demand
Characteristics of Actors
Asynchronous messaging
Each entity manages their own state. They can
create child actors,
send messages to other actors, or
stop child actors or themselves.

by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 14 / 22
 Akka Libraries and Modules

Akka Libraries and Modules

Actor Library
Remoting
Cluster
Cluster sharding
Cluster singleton
Cluster publish subscribe
Persistence
Distributed data
Streams
HTTP

by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 15 / 22
Split-Burn reslover
Configuration checker
Diagnostics recorder
Thead Stravation detector

by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 16 / 22
 DUI Framework

DUI Framework

Dynamic DUI Framework mobile app using javascript genie

by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 17 / 22
 Sunbird Architecture

Sunbird Architecture

Figure: Sunbird Architecture

by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 18 / 22
 DevOps Tools

DevOps Tools

Docker Swarm
Ansible
Jenkins
Github
Monitoring and
Operational tools like
cAdvisor, ELK stack,
Prometheus, Grafana

Figure: DevOps Tools [3]

by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 19 / 22
 References

References I

[1] Cassandra design patterns. available at

”http://apprize.info/php/cassandra_3/4.html.
[2] Logstash elasticsearch kibana investigation. available at
”http://accelazh.github.io/elasticsearch/
Logstash-Elasticsearch-Kibana-Investigation.
[3] Sunbird architecture. available at
”https://github.com/project-sunbird/sunbird-devops/wiki/
Deployment-Architecture-Overview.
[4] Sunbird architecture. available at
”http://www.sunbird.org/developer-docs/installation/
keycloak_realm_configuration/.

by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 20 / 22
 References

References II

[5] Sanjay Ghemawat Wilson C. Hsieh Deborah A. Wallach Mike Burrows

Tushar Chandra Andrew Fikes Robert E. Gruber Fay Chang,
Jeffrey Dean.
Bigtable: A distributed storage system for structured data. available at
”https://static.googleusercontent.com/media/research.
google.com/en//archive/bigtable-osdi06.pdf.
[6] Madan Jampani Gunavardhan Kakulapati Avinash Lakshman Alex
Pilchin Swaminathan Sivasubramanian Peter Vosshall Werner Vogels
Giuseppe DeCandia, Deniz Hastorun.
Dynamo: Amazons highly available key-value store. available at
”https://www.allthingsdistributed.com/files/
amazon-dynamo-sosp2007.pdf.

by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 21 / 22
 References

References III

[7] Anish Sneh.

Cassandra internals. available at ”http:
//www.anishsneh.com/2015/03/cassandra-architecture.html.

by FRG, IIT Bombay Sunbird Components and its Architecture April 2018 22 / 22