Professional Documents
Culture Documents
Weed File System: Simple and Highly Scalable Distributed File System (Nofs)
Weed File System: Simple and Highly Scalable Distributed File System (Nofs)
Project Objectives
Yes
1
2
Not
Namespaces
POSIX compliant
Design Goals
1
2
3
4
Workflow
Master Node
Generate Unique Keys
Track volume status
<volume id, <url, free size>>
Maintained via heartbeat
Can restart
fid format
Sample File Key:
3,01637037d6
Volume Node
Keep several volumes
Each volume keep a map
Map<key, <offset, size>>
Compared to HDFS
HDFS
WeedFS
MasterNode only
stores volume location
MasterNode can be
restarted fresh
Easy to have multiple
instances (TODO)
Automatic Compression
Compress the data based on mime
types
Transparent
Works with browser if accept gzip
encoding
No replication.
1 replica on local rack
1 replica on local data center, but different rack
1 replica on a different data center
2 replicas, first on local rack, random other
server, second on local datacenter, random
other rack.
2 replicas, first on random other rack and same
data center, second on different data center
Future work
Tools to manage the file system