Professional Documents
Culture Documents
Project Slides
Project Slides
Project Slides
TECHNOLOGIES FOR
DISTRIBUTED SYSTEMS
Matteo Moreschini
A.A. 2018/2019
Alessio Russo Introito
Tommaso Scarlatti
Overview
Tommaso
10.0.0.6
exercise 895
cooking 808
theatre 744
music 731
animal 643
painting 614
drawing 578
culture 547
health 453
sport 453
history 323
photography 260
Key-value pairs (Akka)
topic text
Labeled
LDA
tweets
Kafka Compare
Tweets
tweets results
Vector K-means
Doc2Vec
tweets clustering
Literature review
github.com/tmscarla/kafka-twitter
Scope
● Python
(3.7.4)
● confluent-kafka
(0.11.6)
● Tkinter
(8.5)
● Flask
(1.1.1)
Zookeeper
Overview
Kafka
Cluster
Client
Application
Server
Write
PUBLISH
AS
Batch Reading
● AvroConsumer, con diversi brokers
● Get latest N messages: adjust offset from the one of
latest message to (latest-N)
HTTP GET
/tweets/{filters}/latest Kafka
READ
AS
Streaming
● 5 min window messages
● One request --> Stream of tweets
HTTP POST
Kafka
/tweets/streaming
STREAM
STREAM OF CHUNK
TWEETS ECONDED
AS RESPONSE
Filtering
HTTP POST
Kafka
/tweets/streaming
STRE
AM
FILTERING
STREAM OF CHUNK
TWEETS ECONDED
AS RESPONSE
FILTERING
Final Configuration
Kafka Broker
Id=6
Kafka Broker
Id=17
Tommaso Zookeeper
10.0.0.6 App WS
github.com/tmscarla/akka-big-data
Scope
Tommaso
10.0.0.6:8080
10.0.0.6:9000
Master
4 3 5
Worker 2 Mailbox 1
6
REST API
○ SUBMIT JOB
curl -d '{"id":"2"}' -H "Content-Type: application/json" -X
POST http://localhost:8080 /job
○ STATISTICS
curl -X GET http://localhost:8080 /stats
4.
PARALLEL K-MEANS
WITH OPENMP
AND MPI
github.com/tmscarla/k-means-parallel
Scope
Termination:
● No changes in two adiacent iterations
○ Flag in each processor
● Number of iterations > L
Workflow
MPI_Allreduce:
1 7 7 358
Node 1
2 9 9 255
Node 0 MPI
Datapoints
K, L, M
Node 2
Distance metrics
● Euclidean Distance:
● Cosine Similarity:
Results (1/2)
Results (2/2)
Overview
Client 1
Node 1
Tommaso
10.0.0.6
● N number of tweets
● k number of clusters
● ωk dominant class
● cj real class
Results
Purity = 0.21137750
Conclusions
Questions?
Matteo Moreschini
Alessio Russo Introito
Tommaso Scarlatti