Creating a Raspberry Pi-Based Beowulf Cluster

Joshua Kiepert
Updated: May 22nd, 2013

Introduction
Raspberry Pis have really taken the embedded inu! "ommunity by storm# $or those un%amiliar, ho&ever, a Raspberry Pi 'RPi( is a small '"redit "ard si)ed(, ine!pensive sin*le+board "omputer that is "apable o% runnin* inu! and other li*ht&ei*ht operatin* systems &hi"h run on ,RM pro"essors# $i*ure 1 sho&s a %e& details on the RPi "apabilities#

$i*ure 1: Raspberry Pi Model - '.12M- R,M( /he RPi0luster pro1e"t &as started a "ouple months a*o in response to a need durin* my Ph2 dissertation resear"h# My resear"h is "urrently %o"used on developin* a novel data sharin* system %or &ireless sensor net&orks to %a"ilitate in+net&ork "ollaborative pro"essin* o% sensor data# 3n the pro"ess o% developin* this system it be"ame "lear that perhaps the most e!pedient &ay to test many o% the ideas &as to "reate a distributed simulation rather than developin* dire"tly on the %inal tar*et embedded hard&are# /hus, 3 be*an developin* a distributed simulation in &hi"h ea"h simulation node &ould behave like a &ireless sensor node 'alon* &ith inherent "ommuni"ations limitations(, and as su"h, intera"t &ith all other simulation nodes &ithin a ,4# /his approa"h provided true asyn"hronous behavior and a"tual net&ork "ommuni"ation bet&een nodes &hi"h enabled better emulation o% real &ireless sensor net&ork behavior# $or those &ho may not have heard o% a -eo&ul% "luster be%ore, a -eo&ul% "luster is simply a "olle"tion o% identi"al, 'typi"ally( "ommodity "omputer hard&are based systems, net&orked to*ether and runnin* some kind o% parallel pro"essin* so%t&are that allo&s ea"h node in the "luster to share data and "omputation# /ypi"ally, the parallel pro*rammin* so%t&are is MP3 'Messa*e Passin* 3nter%a"e(, &hi"h utili)es /0P53P alon* &ith some libraries to allo& pro*rammers to "reate parallel pro*rams that "an split

a task into parts suitable to run on multiple ma"hines simultaneously# MP3 provides an ,P3 that enables both asyn"hronous and syn"hronous pro"ess intera"tion# 6hile my simulation did not ne"essarily need a -eo&ul% "luster, sin"e it simply re7uired a number o% "omputers on a "ommon net&ork, a "luster environment provided a very "onvenient development plat%orm %or my pro1e"t thanks to its "ommon %ile system and uni%orm hard&are# 8ere at -oise 9tate University &e have su"h a "luster in the :Meta;eek ab< 'or =ny! ab(, &hi"h is run by the 0omputer 9"ien"e 2epartment# /he =ny! "luster "urrently utili)es 32 nodes, ea"h o% &hi"h has an 3#1;8) 3ntel >eon ?3+122. 7uad+"ore pro"essor and @;- o% R,M# 0learly, this setup has the potential to o%%er 7uite *ood per%orman"e %or parallel pro"essin*# 9o, &ith =ny! available to me, you may ask &hy 3 &ould build a -eo&ul% "luster usin* Raspberry PisA 6ell, there are several reasons# $irst, &hile the =ny! "luster has an e!"ellent uptime ratin*, it "ould be taken do&n %or any number o% reasons# 6hen you have a pro1e"t that re7uires the use o% su"h a "luster and =ny! is unavailable, there are not really any other options on "ampus available to students aside %rom &aitin* %or it to be"ome available a*ain# /he RPi0luster provides another option %or "ontinuin* development o% pro1e"ts that re7uire MP3 or Java in a "luster environment# 9e"ond, RPis provide a uni7ue %eature in that they have e!ternal lo&+level hard&are inter%a"es %or embedded systems use, su"h as 320, 9P3, U,R/, and ;P3=# /his is very use%ul to ele"tri"al en*ineers re7uirin* testin* o% embedded hard&are on a lar*e s"ale# /hird, havin* user only a""ess to a "luster is %ine i% the "luster has all the ne"essary tools installed# 3% not ho&ever, you must then &ork &ith the "luster administrator to *et thin*s &orkin*# /hus, by buildin* my o&n "luster 3 "ould out%it it &ith anythin* 3 mi*ht need dire"tly# $inally, RPis are "heapB /he RPi plat%orm has to be one o% the "heapest &ays to "reate a "luster o% 32 nodes# /he "ost %or an RPi &ith an @;- 92 "ard is CDE.# $or "omparison, ea"h node in the =ny! "luster &as some&here bet&een D1,000 and D1,.00# 9o, %or near the pri"e o% one P0+based node, &e "an "reate a 32 node Raspberry Pi "lusterB ,lthou*h an ine!pensive bill o% materials looks *reat on paper, "heaper parts "ome &ith their o&n set o% do&nsides# Perhaps the bi**est do&nside is that an RPi is no &here near as po&er%ul as a "urrent !@F P0# /he RPi has a sin*le+"ore ,RM11GF ',RMvF( pro"essor, runnin* at G00M8) 'thou*h over"lo"kin* is supported(# ,dditionally, sin"e the RPi uses an ,RM pro"essor, it has a di%%erent ar"hite"ture than P0s, i#e# ,RM vs !@F# /hus, any MP3 pro*ram "reated ori*inally on !@F must be re"ompiled &hen deployed to the RPi0luster# $ortunately, this issue is not present %or 1ava, python, or perl pro*rams# $inally, be"ause o% the limited pro"essin* "apability, the RPi0luster &ill not support multiple users simultaneously usin* the system very &ell# ,s su"h, it &ould be ne"essary to "reate some kind o% time+ sharin* system %or a""ess i% it ever needed to be used in su"h a "apa"ity# /he overall value proposition is pretty *ood, parti"ularly i% "luster pro*ram development is %o"used on distributed "omputin* rather than parallel pro"essin*# /hat is, i% the pro*rams bein* developed %or the "luster are distributed in nature, but not terribly 0PU intensive# 0ompute+intensive appli"ations &ill need to look else&here, as there simply is not enou*h :horse po&er< available to make the RPi a terribly use%ul "hoi"e %or "luster "omputin*#

Building the System

/here are really only %ive ma1or "omponents needed %or a &orkin* "luster: "omputer hard&are, inu! =9, an MP3 library, an ethernet s&it"h, and possibly a router# =% "ourse, &e have already "hosen the 2

RPi as the "omputer hard&are# $i*ure 2 'a( sho&s the overall net&ork ar"hite"ture# /he system desi*n in"ludes 32 RPi nodes, E@+port 105100 s&it"h, ,r"h inu! ,RM, and MP3083# /he bill o% materials "an be seen in $i*ure 2 'b(#

(a) RPiCluster Network Architecture

(b) Bill of Materials Figure 2: Cluster Design

/here are several operatin* systems available as pre+"on%i*ured inu! ima*es %or the RPi: Raspbian :&hee)y< 'based on Ubuntu 11#10( and ,r"h inu! %or ,RM 'aka :alarm<( 'raspberrypi#or*(# Raspbian is available in both hard and so%t %loat versions# :8ard+%loat< implies it is "ompiled &ith support %or the hard&are %loatin*+point unit in the ,RM pro"essor, &hereas so%t+%loat implies it &ill use so%t&are libraries to implement %loatin*+point operations# 0learly, hard %loat is desirable %or per%orman"e, but %or "ompatibility reasons so%t+%loat is available# 3n the "ase o% Raspbian, hard+%loat is not yet supported by 3

=ra"leHs Java 2evelopment Kit 'J2K(5Java Iirtual Ma"hine 'JIM( %or ,RM# 3t is &orth notin* that the =ra"le J2K is not the only Java implementation available, so there is no need to limit per%orman"e &ith so%t+%loat# /he beauty o% open sour"e is that you "an simply "ompile the pro*ram %or your plat%orm even i% the vendor has not taken the time to do so# 3n this "ase, =penJ2K is the open sour"e implementation o% Java, and it is available on all inu! distributions, in"ludin* Raspbian and ,r"h# ,r"h 3nu! %or ,RM is only available in the hard+%loat "on%i*uration# 9o, &hi"h inu! distribution makes sense %or a "luster o% RPisA /his depends *reatly on ho& "om%ortable you are &ith inu!# Raspbian provides a turnkey solution in that it "omes pre+installed &ith the >2? 'a %ull desktop environment(, and bein* 2ebian based, there is a hu*e set o% pa"ka*es pre"ompiled and available %or the RPi# /his is *reat %or *ettin* started &ith inu! and &ith the RPi in *eneral# /he do&nside to this distribution is its &ei*ht# 6ith support %or so many %eatures to be*in &ith, there is a sta**erin* amount o% daemons runnin* all the time# ,s su"h, boot time is mu"h lon*er than it has to be, and you have many pa"ka*es installed that you most likely &ill never need or use# ,r"h 3nu! on the other hand, takes the minimalist approa"h# /he ima*e is tiny at C1.0M-# 3t boots in around 10 se"onds# /he install ima*e has nothin* e!tra in"luded# /he de%ault installation provides a bare bones, minimal environment, that boots to a "ommand line inter%a"e '0 3( &ith net&ork support# /he beauty o% this approa"h is that you "an start &ith the "leanest, %astest setup and only add the thin*s you need %or your appli"ation# /he do&nside is you have to be &illin* to &ade throu*h the learnin* pro"ess o% a di%%erent 'but ele*ant( approa"h to inu!, &hi"h is %ound in ,r"h inu!# $ortunately, ,r"h inu! has some o% the best or*ani)ed in%ormation out there %or learnin* inu!# ,n e!"ellent pla"e to start is the ,r"h inu! -e*innerHs ;uide# 3 "ould *o on about all the thin*s 3 appre"iate about ,r"h inu!, but 3 di*ress# ,s you mi*ht have *uessed %rom my previous statements, and bein* a : inu! *uy< mysel% ' inu! Mint, primarily(, 3 "hose ,r"h inu! %or the RPi0luster# ,s %or the MP3 implementation, 3 "hose MP308 primarily due to my previous %amiliarity &ith it throu*h Parallel 0omputin* "lasses '=penMP3 is available also(# 3% you are runnin* Raspbian, MP3 "an be installed by e!e"utin* somethin* like:
$ sudo apt-get install mpich2

/hatHs it# 8o&ever, on ,r"h inu! MP308 must be "ompiled %rom sour"e, or installed %rom the ,UR database# 3 did some lookin* on ,UR, and only %ound an older version o% MP308 'built %or an old version o% *""(# 9o, 3 &ent &ith the "ompile+%rom+sour"e method# Jou "an *et the sour"e %rom mpi"h#or*# =penMP3 is available in the ,r"h inu! repositories, and it "an be installed as %ollo&s:
$ sudo pacman -Syy openmpi

3n the "ase o% MP308, "ompilin* lar*e amounts o% sour"e on a G00M8) ma"hine "an be tedious# 9o, &hy not utili)e our 0ore iG Kuad+0ore to do the 1obA ?nter: K?MU# K?MU is a 0PU ar"hite"ture emulator and virtuali)ation tool# 3t so happens that it is "apable o% emulatin* an ,RM ar"hite"ture very similar to the RPi# /his allo&s us to boot an RPi ima*e dire"tly on our !@F or !@FLFE systemB /here is one "aveat, you need an RP3 kernel %or K?MU that kno&s ho& to deal &ith the K?MU virtuali)ed hard&are &hi"h is not normally part o% the RPi kernel ima*e# /his involves do&nloadin* the inu! kernel sour"e, applyin* a pat"h to support the RPi, addin* the various kernel %eatures and drivers

needed, and "ompilin*# /he "ompile pro"ess %or the kernel only took about . minutes on my iG laptop# 2o&nloadin* the sour"e %rom the ;it repository, ho&ever, took mu"h lon*er 'C1#.;-(# /here are several tutorials available &hi"h provide details %or runnin* K?MU &ith an RP3 system ima*e# , "ouple re%eren"es 3 %ound use%ul are Raspberry Pi under K?MU and Raspberry Pi &ith ,r"hlinu! under K?MU# ,side %rom the need %or a "ustom kernel, there are a "ouple system "on%i*uration "han*es needed &ithin the RPi ima*e to allo& it to boot %la&lessly# /he "han*es primarily have to do &ith the %a"t that the RPi ima*es assume the root %ile system is on 5dev5mm"blk0p2 and the boot partition is on 5dev5mm"blk0p1# K?MU makes no su"h assumptions so you have to map 5dev5sda devi"es to mm"blk0 devi"es on boot# 6ith the RPi system ima*e ad1usted and the "ustom kernel built, startin* K?MU is somethin* like the %ollo&in*:
$ qemu-system-arm -kernel ./zImage -cpu arm1176 -m 256 - !ersatilep" -no-re"oot -serial stdio -append #root$/de!/sda2 panic$% r&# -hda archlinu'-h(-2%1)-%2-11.img

=n"e you have a kernel ima*e ')3ma*e( that is suitable %or K?MU you "an point it at the ne& kernel and the RPi system ima*e# Runnin* an RPi ima*e via K?MU may be si*ni%i"antly %aster than &orkin* on the RPi, o% "ourse, this depends on the "omputer bein* used to run K?MU# $i*ure 3 sho&s the boot s"reen o% ,r"h inu! ,RM under K?MU#

Figure : Arch !inu" ARM un#er $%M&

6ith a &orkin* ,r"h inu! ima*e under K?MU it is possible to update to the latest pa"ka*es, install the ne"essary pa"ka*es %or the "luster environment, and "ompile MP308 %or parallel "omputin*, all %rom the "om%ort o% you !@F inu! system# 3 used K?MU %or all o% the a%orementioned tasks# =n"e

"on%i*uration o% the ima*e &as "ompleted &ith K?MU, 3 &as able to simply &rite the resultin* ima*e to ea"h RPi 92 "ard 'a%ter modi%yin* hostname and 3P address, uni7ue to ea"h(# -elo& are a %e& o% the steps involved %or "on%i*urin* an ,r"h inu! ima*e %or use in a "luster: 0han*e root pass&ord %rom de%ault 'root(:
* pass&d

$ull system update:

# pacman -Syyu

3nstall typi"ally needed pa"ka*es, bold pa"ka*es are need %or "ompilin* MP308 and %or usin* 4$9 or =penMP3:
# pacman -S nfs-utils base-devel openmpi sudo adduser nano !im gd" "c minicom Set the local timezone+ # timedatectl set-timezone ,merica/-oise

,dds a ne& user###MuserN in this "ase 'be sure to add *roups &heel,audio,video,uu"p &hen prompted(:
# adduser .user/

,llo& users that are part o% the &heel *roup to use superuser permissions:
# chmod 6%% /etc/sudoers # nano /etc/sudoers

'Un"omment the line: O&heel P ### and save(

* logout

' o*in as user &ho &ill be usin* MP3(

0 ssh-keygen -t rsa 0 ssh-keygen -t dsa 0 ssh-keygen -t edsa

0luster setup is basi"ally 1ust "on%i*urin* a %e& %iles so that all nodes "an %ind ea"h other and have pass&ord+less 998 a""ess to one another# Key "on%i*uration %iles %or "luster setup under ,r"h inu!: 5et"5e!ports 5et"5idmapd#"on% 5et"5hosts 5et"5hostname 5et"5%stab 5et"5net&ork#d5ethernet+stati" 5et"5"on%#d5net"%*

Performance Comparison
3nitially, 3 "reated a %our+node test plat%orm &ith RPis# 9o, it &as possible to do some preliminary testin* o% per%orman"e be%ore "reatin* the %ull "luster# /he %irst test &as to determine the *eneral per%orman"e o% a sin*le node runnin* multiple threads, usin* the MP3 libraries# /he MP3 test pro*ram 3 used is one 3 developed durin* a parallel "omputin* "lass at -9U# /he pro*ram "al"ulates pi usin* the Monte 0arlo method# 3n this method, essentially the more random numbers you *enerate, the more a""urate your resultin* estimation o% pi# 3t is an embarrassin*ly parallel &ay to "al"ulate pi 'i#e#, s"ales easily to multiple independent pro"esses, near 100O per%orman"e *ain %or ea"h additional pro"ess added(# ,ll that is needed is a &ay to paralleli)e the *eneration o% random numbers so that ea"h pro"ess does not repeat the random se7uen"e 'remember, &e are dealin* &ith pseudorandom numbers here(# 3n this "ase, 3 use a library "alled 'ran#, developed by 2r# Jain and Jason Main at -9U# /able 1 sho&s the hard&are spe"i%i"ations o% the various plat%orms tested %or "omparison#
(able ): (est *ar#ware +'ecifications

$i*ure E sho&s the e!e"ution time %or Q00 Million iterations o% the Monte 0arlo pi "al"ulation pro*ram 'PM0P3(#

Figure ,: MP- +ingle No#e Perfor.ance1

1

/his is runnin* the MP3 pro*ram on a sin*le node &ith multiple threads, not in parallel &ith other nodes in a "luster#

3% you "an not tell %rom $i*ure E, that is a ma!imum o% 1E#F se"onds %or a sin*le thread on an =ny! node# 6ith %our threads it *oes do&n to 3#@. se"onds thanks to %our pro"essin* "ores# 6ith ei*ht threads, it *oes up to 3#Q0 se"onds# 0ompare that to our lo&ly RPi, &hi"h re7uires E.F se"onds %or sin*le thread# ,dditional threads do not help due to its sin*le "ore desi*n# $i*ures . and F sho& the per%orman"e improvement as &e add additional RPi nodes to parti"ipate in the "al"ulation, in e!e"ution time and speed up, respe"tively#

Figure /: %"ecution (i.e Re#uction ()0 2 no#es)

Figure 1: +'ee#0&' as Co.'are# to ) No#e

,s is seen in $i*ures . and F, &e "learly *et the e!pe"ted per%orman"e improvement as &e add additional nodes to parti"ipate in the "al"ulations#

Rack and Power Design

=ne aspe"t o% the "luster desi*n that re7uired 7uite a lot o% thou*ht &as the ra"k mountin* system and po&er distribution method# 3n order to keep the "luster si)e to a minimum &hile maintainin* ease o% a""ess, the RPis &ere sta"ked in *roups o% ei*ht usin* P0-+to+P0- stando%%s &ith enou*h room in bet&een them %or a reasonable amount o% air %lo& and "omponent "learan"e# /his "on%i*uration suited our needs %or po&er distribution very &ell sin"e it allo&ed %or a po&er line to be passed verti"ally alon* ea"h sta"k# Usin* this orientation, %our RPi sta"ks &ere assembled and mounted bet&een t&o pie"es o% R< a"ryli"# /his "reated a solid stru"ture in &hi"h the "luster "ould be housed and maintain physi"al stability under the "ombined &ei*ht o% 32 ?thernet "ables# $i*ure G 'a( sho&s some initial e!perimentation usin* this method o% mountin*# /he ple!i*lass layout %or mountin* ea"h RPi sta"k &as desi*ned usin* ?a*le0,2, alon* &ith the po&er5 ?2 P0-# $i*ure G 'b( sho&s the layout#

(a) Rack Mounting RPis &sing PCB +tan#0offs

(b) Ple"iglass !a2out ((o'3Botto. of RPiCluster Rack) Figure 4: RPiCluster Rack Co.'onents

/here are t&o methods o% po&erin* an RPi# ?a"h RPi has a mi"roU9- port that enables it to be po&ered %rom a "onventional po&ered U9- port# Usin* this method &ould re7uire a mi"roU9- "able and a po&ered U9- port %or every RPi in the "luster, &hi"h &ould add a lot more "ablin* to deal &ith# 3t also means that there &ould be "ablin* on both sides o% the "luster# ,lternatively, there is an 35= header on the side o% the RPi &hi"h "ontains a .I pin that "an be used to po&er the board e!ternally# /he latter option &as "hosen as it &ould also allo& %or additional "ustomi)ation and "ontrol o% ea"h node# , "ustom P0- &as "reated to %it the 35= header to provide po&er and an R;- ?2 'sin"e ele"tri"al en*ineers must, o% "ourse, have ?2s to sho& that their pro1e"t is &orkin*(# $i*ure @ 'a( and 'b( sho& the "ir"uit and P0- layouts, respe"tively#

(a) RPiCluster Power3!%D Boar# +che.atic

10

(b) RPiCluster Power3!%D Boar# PCB !a2out Figure 5: RPiCluster Power3!%D Boar#

,s seen in $i*ure @ 'a(, aside %rom a R;- ?2 and some "onne"tors, there is also a poly %use 'P$1(# /his &as in"luded to maintain short+"ir"uit prote"tion in the event o% a board %ailure# /he RPi already has a poly %use bet&een the U9- po&er "onne"tor and the .I rail# /hat %use is bypassed &hen usin* the .I pin to po&er the board# JP1 provides a .I inter"onne"t verti"ally bet&een the RPis in ea"h sta"k# 6ith the po&er bein* dire"tly distributed via the Po&er5 ?2 board, it &as ne"essary to %ind a *ood sour"e o% .I &ith su%%i"ient ampera*e to drive the &hole "luster# ?a"h RPi dra&s about E00m, at .I '26(, thus &e needed a minimum o% 13, o% .I 'F.6( 'and more %or over"lo"kin*(# Jou "ould, o% "ourse, buy a dedi"ated hi*h output .I po&er supply, but a *reat option is to use a standard P0 po&er supply# P0 po&er supplies are already desi*ned %or hi*h loads on their .I rails, and they are relatively "heap# /he E306 /hermaltake 'E306 "ombined output %or .I, 12I, et"( &e sele"ted is rated to provide 30, at .I '1.06( and "ost D3F# 6e opted to pur"hase t&o supplies to keep the overall load very lo& on ea"h and allo& %or over"lo"kin* or %uture e!pansion#

Final weaks
,s mentioned earlier, the RPi pro"essor supports over"lo"kin*# ,nother note&orthy %eature is that the .12M- o% R,M is shared by both ;PU and 0PU, and the amount o% R,M reserved %or the ;PU is "on%i*urable# -oth o% these %eatures are "on%i*urable throu*h modi%i"ation o% parameters in 5boot5"on%i*#t!t# ,%ter some testin*, the &hole "luster &as set to run in :/urbo Mode< &hi"h over"lo"ks the ,RM "ore to 1;8) and sets the other "lo"ks '92R,M et"##( to .00M8)# /he result &as a proportional in"rease in per%orman"e 'C30O( and po&er dra&# 9in"e the "luster does not re7uire a desktop environment, the ;PU R,M reserve &as redu"ed to E@M-# E@M- seems to be the smallest amount o% R,M allo&ed %or the ;PU on the "urrent version o% ,r"h inu! ,RM %or the RPi 'it &ouldnHt boot &ith anythin* less(#

11

Power Consumption
=ne o% the *reat thin*s about "reatin* a "luster &ith ,RM+based pro"essors is lo& po&er "onsumption# ,s dis"ussed earlier, ea"h RPi uses about 26 o% po&er '&hen runnin* at G00M8)(# , number o% po&er measurements &ere made at the &all &ith the RPi0luster in various operational states# /his allo&ed the individual "omponent po&er usa*e to be determined &ithout takin* ea"h item o%%+line to measure po&er dra& individually# ,s 3 have over"lo"ked the "luster to 1;8) "ore %re7uen"y and .00M8) %or 92R,M et"#, the po&er "onsumption is hi*her# $i*ures Q 'a( and 'b( sho& the overall po&er use 'as measured at the &all( in relative proportion 'per"ent( and &atts, respe"tively 'over"lo"ked to 1;8)(#

(a) RPiCluster Co.'onent Power &seage ('ercent)

(b) RPiCluster Co.'onent Power &sage (6atts) Figure 7: RPiCluster Power Consu.'tion

,s is seen in $i*ure Q 'b(, there is about 1.O di%%eren"e in po&er "onsumption bet&een idle and %ull 0PU utili)ation# ,dditionally, about 12O o% the total po&er used is attributed to the "oolin* portion o% the "luster '%our 120mm %ans(# /he ma!imum total po&er usa*e o% the "luster at the &all is about 1FG 12

6atts# 3t is &orth notin* that a"tive "oolin* is te"hni"ally unne"essary# 2urin* my testin*, the 9o0 temperature did not e!"eed F0S 0 &hile under 100O load, and the 9o0 is rated to operate up to @.S 0# 6ith "oolin* a"tive, ho&ever, the 9o0 temperatures did not e!"eed 3.S 0# 0oolin* &as retained be"ause it seemed better %or the 9o0 to operate at lo&er temperatures 'and it looks ni"e(#

Conclusion
=verall, the RPi0luster has proved 7uite su""ess%ul# 9in"e "ompletin* the build 3 have moved my dissertation &ork e!"lusively to the RPi0luster# 3 have %ound per%orman"e per%e"tly a""eptable %or my simulation needs, and have had the lu!ury o% "ustomi)in* the "luster so%t&are to %it my re7uirements e!a"tly# 3 &ould like to thank all the *uys 3 &ork &ith in the 8artman 9ystems 3nte*ration aboratory %or all their help implementin* the "luster desi*n, espe"ially Mi"hael Pook, Iikram Patel, and 0orey 6arner# ,dditionally, this pro1e"t &ould not have been possible &ithout support %rom 2r# 9in Min* oo and the 2epartment o% ?le"tri"al and 0omputer ?n*ineerin*# ###and %or your vie&in* pleasure, 3 have a %e& more photos:

13

1E

1.

1F