HADOOP CLASS ROOM NOTES Q& Kelly Technologies Flat No: 212, 2nd Floor, Annapurna Block, Aditya Enclave, Ameerpet, Hyderabad, AP. Ph No: 040 6462 6789, 0998 570 6789 E-mail: info@kellytechno.com, www.kellytechno.com.(Hadoop) ; By Ur. GOPAL KRISHNA x Tngpired by Geogle gigioble and cnpreduca, papers cca 2004" *¥ created = by pougcutting- ecie\toutt ° * originally pant “to Support disti bution for Boke Ergin shwSf laghdint «44 “eee a a ude tr Aeaah ee fe ; He ye New . Se Bigbata we : Bir an a te Big DAY apt the grav! challerge tor coi ZasIons face os they deal with loge ond “es con OF «dota OF information that abo tok aan ingat, . camper pofrashacture é . ioe aes eed high volumes (Size ard for rate) oF vot.dart a vised dora (steucwured on unghacture) from * AS " multiple gourds % dealing = oth unpredictaple content = with 70 oppayer cher or grruckuye* 7 — real- time Ilect?on, * Geaplng ~eol- time okally Peehnotogien collection. 10. 212, 2nd Floor, and — WEY Anaguma Block, Arya Enctars, , Hyderabed-595 616, analysis Bigooto {ROR ROMEE: o eew genom ond arcnitecdkry » duigred 0 economica value from veg large volumes of @ varety of data, Py ewabling high -vclodty andlor analysis. of technolagies wide captire , discovers ,6 The dare wrasse oo . climate trformation, poses to Social Melia She gether sya, digital pictur * and videos purchase transaction ‘ecords and cel phore = GpS Sigmmk to mame feeo nis dara i Bigdoto- axee ve of Bighata ; - youme describes the amaint of data gencraiet by | organizations oa individual Vovdety describes stuctured and unstructured data , Such ay text, &Mor data, audio, vicko , click Stans, log files and = more: Velocity dwertb the Frequency at ushich dam is generoiet, capuned ond Shaved, sg Qua ~ Hedonp :~ Had is the Apache open Source Softeunye Pramecare py | corny eith ByData. TE wos derived from Gecgle techrolagey | & to practice ty yah ant others. But, Sypata } iy yoo varied «orth Compe ford Shee one- Size ~ fits— all solution: tured the greater ram, | tle wadeop «hs Surely capt om te A jest one of . three classes of technologie, cogritton , 2 s se suited to Storing ond managing Gigdata. Hadaop 3 Soft wane Framework , ehich manny it ¢ no: oF Compaen thet = were Speci Ficeally ceesignad 3. solve lange - Scale dishibutel dara Storage, analysis any atvieval . toaKs- ‘ 2 of Bigdota 4a 2 Geagk p7rorss 20 pB a day a unyba machine hed 3 B+ {00 TE/wBnth m Facebax hax &-5Ps of User dara + 15 T/day % ebay bor 65 pa of wer dara + 50 7B} aby. > 0 Oo fe} oO oO oO 9 oO oO 0 °o oO ° b ° ° 0 t f 0 oe pe \o oO oO ot ccoooo | ea Gtrivial versio seen) oss maps ine @faretla to the TEP Froressors* Hadeop "runs 20 nodes: > eutht soto ee ‘ PP pega 212, 2 ug, Catlin a teva ENT pees ance OLS Haaeop praiect GBD TETERO standalone Saveloprnent of raped ord perch ar (1060) n0d8) 749 on 188 ces in-41-9h hadeop reseastch chustey — 300 noes 1 yaheo! setup > on 500 POMS Ga] ours [better306 i- Research Clursr reaches 600 odes. 006 | - Ly, 20S! sort benchraxk yun on 20 nedey in 8 hes 100 nad iN Bah 6a rade in 5-hy ; aco nodes in 7-8 hm jan, BOF Reseaxch clustey veaches G00 MOMs. dean clusters of 1600 nodes. we 2007 :- Reseanch clusters, - ‘ im 209 eure 1 .gort — bencherayh apt 2B: Won. the on God née. seconds lo Tera Gyr oF data per day onto the 2CE 2B :- Leading reseaxch Clusts: arch 209 i- VF clusters’ “with a total of ay ,0e0 “Ode aps! 2009 :- won the minute, Sore by Sorting 0G én sq secords (on mo node) ond the (co - terabyle sore in [on 3,40 vodkes] - ytd eioutes wompe fos 3 Siaifetical Anoly gone WERT Reet sean - ~~ Googe Recaives over 2,00 ,c00 | Seavich Queries © foceteoK Fe@ive sy 722 “Kes”, © consumes — Sper $212,070 ON Ub Shoppiry- © Apple recaives UF 100 Apps dewonicads. : minute OF cath on Skype; 2 /370,0 : ee 8 9B,cap POSES ST RCOREES » © 20,000 posts-0n Tumblr, 2 1g 00 hous OF music Streamixg on fandora ads on & Craigslist , vploaded to FicKr © yaycoo ee 2 6,600 prctares © 1,500, new blog Pests, 2 Gee men yoriTUbe videos a DAR oOHadeop |S History :- By Mr. Gi a By lr. GOPAL KRiShNA —> Hadep was Created ky Ooug Gutting & Michel - Doug , who way working ar yoneo at te after D6 son'S toy elephant rally developed to Support dist bution J. @faxella - the time, . tre ws OI 7 fos cthe® Nateh Seaxch engine project i 5 padeop taxed on wan done EY Google i the fe 1aqo4| ly 2060 - rh vA ea Seino Bt spectfically , on pre gn 2003 , aetly ‘ Cars) pavtere? Bio. bE Aya / ponapura eee ED O16 2004 - home ays 0 oT + ¥| | | i 1 @ Lage dara 0° the wed: | @® Nuteh ute f° craw! this Ueb dora Boo @® tose valume of dota had 0 of s to Sowed - HORS toh @ wee ese this dot 2 Report © MapReduce Fromersors butt for codig § rormty onalyti a.© unstructared dota, weblogs , click Streams , Apache logs Server logs ~ fige, uebdov, chekwa | Fiusene ark sertbe @ se ant Hiho fer loading dota into HDFS- RDBMS data. ® High level trkerfaces wepired over low Level map reduces programming - Hive , PID , yeqh: Dy Br tok ewith advanced UE Reporting. BD workfiow tools over sap-Reduce prccesss “and High level t ~ Ooze D D wonttar & monage hodexp , wan qob%/hive , vfeeo HPS -~ high \eve) view - Hue, Karmasphere , eclipse plugin , cacti, govgfia. D Sopport Framecomis ~ Ayro(Sertalization), Zookeeper(Coosds =) B® Moe Hg level Tlerfaces/ ase — Mathout , Elastic TapRediids B. ortp abo possiHe In HBOHe. | B® luane w ter Search engine Ebrony eowrtten 77 | qoup.- y Hodeop 4% best Krexon for MapReduce ord it's Diutyitutel file Syste (HORS , gram from OFS) [nore ors abo wed “fora projects that fall under are ure elo of — infrashuctare efor dist buted Computing and large Seale dara proassirg | ADES :- Tf yd wank Yor + Carnputers to work on yr you'd better Spread yur data aeross 4ouo+ computers wors dex this for gore HOFS hos a few moving whe Dararncdes Store YOUF dove, ord the Nase ode ochere stud & Stored. There are Othey gos | = Powe enough to get Staxied. jor, theo aeeps OOK Of pieces, tout ‘ ce esMapReduce :- sos Wepkecncg i this is the programm ; yadcop. "There are tO phase, rot J mane| for op ar Reduce. TO Impress ycor foprsisly Caled tell them here 8 sort & Shuffle — bekeween — the Qauce proses The gob Tracker manages the yan Corny — . mropReduce gob: The tantackdrs take Orders from Tf you Whe — gaua /then code in . she TOTIAKET og yor THE SOO other non ~qyua languages gp Gre gut) rn LUCK you con we 9 utility Calle ttadeop streaming - g | By Mt. GOPAL KRishaay, Hadesp sreaming - A atiltty to anable trap Reduce | language: c, perl, Python, crt, Bash ete qhe enarnges pec: a python mapper and on Any yeducer Bye ond Hue tf you Une sak, yor evil be delighted yo rear that gor cP write sar and Hive Convert TE a MapReduce sjob- No, you don't get a fall +0 yooo nots and Avst - sab environment, wwe you do get pou ttt peta byte scalability: Hue gives you 2 browser bose graphical qnterface to JO gor Hive Work: pig :- A Higher - level prograeoming environment todo wap Reduce coorg: e"y hangasige ww called Pig latin. wand Y the rat Conventiams — Some cohat yor «roy fi "9 - uncaaverrttarad ’ bat yee get ond high anatlability - BER provides bi -directforal dota -hans-fer bey Hedoop and = yur favorite velatfonal database. cozie:- Manages Hedeop uarkficco. This dos meee ? ag eur Sebeduler OF Bem tooling peat it Rept: Ihde $$ —then- else aaa ark — control 4a a pradcop TOPS* ws e at “ | pocredtble paice ~performanga super - Scalable -value Store. Eb works persistent yash-rap (for python £ “te i Tot a yelatfenal database Very much Uke @ think ater onary) the more Hense- despite Fuse A veal Ume loader for Seana gn data, sto wodeop: qr stores dota in HoesY and BAe - want to get Stexdted with flume , cohich origi ool Fume. ‘eich -\eatining fer Hadteop « used fey predicts and’ other adveunted analiysis « Fuse: make the HOPS system [con like a regular Filepim, ond others on HDRS dod. so you CAD we b,r, a, coKeeper:- used tO manage Synchronizarfon sfor “the clusier- You won't be coorkt: much with zaokeeper, but te wOTKE fond 1 you. Tu think = yeu need tO write a pagan thot uiep zookeepey YOOX and cloud ve a Commitee ane etthey very, Very smart: rect , or you are for an Apache pre) vad doy: vey 8g b BGO date volume is growing exponentially . with time, fonter UF usa to talk about — MegePytcs oF app eyis. tetk abot data _ about to have @ Very \ 19 bas omvived when Of of terayts, pera kyks and alto volume 9A Qvourd §=1-82 ZB #9 2B I 305. Rat time Volure ™ fers zertaeytes | .Globat date tn aoil w& expecred f° be EDS enaon tet the global “ae _eveig, Heo years | trfermation ur doubles Oe ue\ | effective aralypis een uw of Brgotto Hee Oralysis oF Sigeta & useful fer organizarions? of Big Dare. provides a lot of business 10 qanontage a organizartoms will Leaxn ewhich areas focus On ord cwhich = areas are less important Big dara arobysis provides come candy Hey | indicarars thar con _ wevent the com pony from a huge loss or Yelp in groSpi7g a greor opportunity with open hands! 1g precise of Bigdota HPS tn decision rien, {| gor taavoree , a. dows people rely $0 much 09 Face bak ~ ane esi tber RényTecthvots: 3 Pq product z or Service: pect Meas ty Ear ‘Maneerpet, Hyderabad-500 016. ‘Ph: 040-6462 6789, 993 570 6789 noes oot RBESANS fps fo Heep Wet” ( Feorure: a 7 secure Resthe nti coxfern ges Se) we wk configure? Nomes gs deprecoted | -doprenid New anf gurerton Nomes No ye ys ‘ old wrapReduce apt ye a ge ys New MapReduce APT Gein / ues yes Taste Lakes) WapRaduce | vantine ges oS Wo Cetessie] MapReduce 2 wantine, NO No ys | Cyaan Fue HOFS federation NO NO ys fipps high availability NO No ye han — — L@® what 4% Madeop ? . Hadeop i arnlys for Bigdata. Hadeop is an open Source sfrarnework for Creating agpii cations that psiocess_ Huge amoun} dist ord of dara- gre dafinisfon of huge aQ5,cp0 machin ee yore vron 10 clusters 2 po of dart Ccompressed , unreplicated) Joo + eHeers “w ge lo‘coo + yobs / ween - / 3 sadoop th Q open source, Distributed , Batch processing ond faut - tolerasre spem which is Gpank of ~ th puge ascnant ‘of dora (TD ,PB Zettai kyles-de) - Hadeop See ae (=) ie . production envivonenenh Ged for stag! wt for soar & = Kroenen ve qos Tore?opiate Sepiem : = cchat & ouribud Fee SNe: gat perrnarentty Store data button , veplitation access to < siplen » Support File ord vernote Servers aivided tnto legical untts( fila , shard, chunks, bles) working ryjw based approach ‘ecaric g crplac than Regulax din file Systems, concuneny , dist pFs's are pps's are more CO ar file syste apierate node -failure eithour “ eerfferieg “1 9aFe eae eaccp pistibeted Els ured file spleen and it ues to Bb a dati + Hadeop Store eulk amounts F data line terabyler @n even pert Bytes » HORS Sopport nigh — threughpot mecranism for accessing this large amount qarformation- tm HORS fils are stored @ Sequential reclunctont eranner OEY the mutkple, machines and this. guarantes the forlcoi nf ones: © Durability +o -failure ® ugh nat lai iy +0 Very parailel appltaiions, cancel NES Crvetuork file syste) ryemole access to & single Single machine - y can -yisible a postion of it's loalfiles sna) clients and obo the client can tem direc! into “thely Poteract with It ag . NES give legicol volume stored O71 & NES sevveT gyre f° exter mount ths wemote file Sys nue Fila Sytem , and con trough te WT post Of the goantage NES uw srounsparency {thot is cleents do noted to be poxttulaxdy quware that they one cette eS lecou dbeve- Aa eA Wwcavorsages & BEES @ HOFS store lange amoant of iformation- robust — Coherency emccle) ® Hors is Simple ard @® ther je shoud Store reliability. @ Hors ‘uw Scalable ard fast access tO thy informertion , to Serve lange number of ana it abo possible clients by Simply aadiong, more machines to the cluster v rox eit eorth Hadesp mapReduce , Ost ee ees Ke ead and computed Opa Jocally wher posstole- By Mc. GOPAL'KRishaia, ForO.N ce + streasning read = per © Hors providing che HORE TRMFoISEion . itten to @ wre ewoill be . ee Flat No. 212, 2nd Floer, ead several Hes: Anonzare Stes Aly Ent neq 4 PARSER hoc a @ 1 overhead Of mg aie SA ghook sienply Pe veered from HORS Sorc: @ poutt-rolerarce my detecting faunts ard applying we cory - quion , auxomaric @ processig legic close to the dota, rather than “the gara close tO thE processing logic - commodity — hardware @ portabi lity across = he tesoge neous ark operating Systems’ @® Ecoromy distributing data ond processing across clustes of commodity persoral Competers ® Esficiencs py distyibuting dota ant jagie 10 process ve te paraltel 0" voles eohere dota is located. auto matically maintaining muitiple and aurtomar calles ‘recleploging m the event of failures. @ Revravileig by copica Of dota processing logicDa Adwantages Of HDFS . a ORAS Th distibuted file System , it vw Limited in its { in ite power. | she pu in an wes volume aul reside of a Single . poachine «This will “Credit Some probing a te dots rot gives any ella ty guarantees ef thar machine ges down: eer By replacing the files to other machine. @® Mat te velents must go to thy machine to re bi thety data. This Can overload the server if a — sno. of clients, ut, be handled one @® clients need ts copy the dota to thely lecot eoachines —- ke fore they Con operate on it. yea af HRS © very large dist buted file_Sysieen 1- 10K nadas_, 19 million fly, 10 PB . jrondware ; files are replicoteol +o 1 ® ogame commodity Detect failure, and recover harile hovtduvare failure - eon, - the: ® optimized for batch prom@ssing :. pata lecations expos eo thar computations Can move to cehere data mestdus: TE provides very high aggregate — kandevictth. yrs & bien Shoctured file gynem s- 310eK -- Blo aw the minimun unit oF dota that gold 19 HDFS, eohich typically GMB by ipfauit- revere qocrease the bketusize % un the multiples of =AmB- ewe can« Gach fik 4 broken into bi of ond thue blocks are stored aco fa Size Ver of ome er more machine With data Storage oe a Capacity . Trdividual machines im the cluster are called th, 1 Bata nodes - + A file Gn be made Of several biocKs ant rot mece ssorily Stored on the Same crachine- the target machine chte exch bck om on a been = by block boxy. SMe GOPAL Kr 4 so acess permission to a file meed . ‘the avion of «multiple machines and ft Supports cooper’ file size fox larges than a Single machine DES. a Trdividuel fil, aometima need losge Space “than a single hand deve eter’ coud hold. fF Severo musr be fAvolved in the Serving of a fk, eraicni 7S when &@ File Could Pe rendered — anavailable by the (05S Of ang ane af those = machind- HORS seplicox. €ath — block comers rhs probe ( ys a Took machine “(by default 2). 3% ee. Ee acrosse above figure the varaneacs — scpreacvin> ° to th muteiple fils with ication factor of 2 nd the, & fame pede Mops the filenames onto the bleem fadg- | a A P | © tn bln Shactured file Systems commonly cue | ; a bloen size OP the order of 4 oF BKB- ate default lot Size in HOFS & 64u KB. THY 6 to decrease the amount Of metedata po file. pieck shuctured file System , all the led by single rachine called . exerts HDF storage required HETDEBSTS 5 In HODES sfermarion's OF band au the metadata for the. une backs File nfo OHO, locaton Of each ! can te stored fa the main 4 ode caching Gh it peemits a client first conbacs Uist of lecatfons or files ‘ond yhuse lecotiony noth each blocK é aantiy WO ° eo cltents ren ead Silo dare directly $romn ‘the . pode | Server® vig posrattels 0 the Wo 4 a aivectly spvolved an the pulk data hamfer " yeod too parent mun + aecpr™y, ats ONEE « nomencde tn formoaHon %% Should be preserved even 7g the nomenode, erachine fail: © Nase Nod failure &% more severe for the cluster vow) pedo eda. foslure- eo when andividuol — Pote. nodes, crash the apttre CuBr et tt contour a op tone . NomeNnodse — wit yerdey te Cushy joss Of ” S accasetae ant) ft} mronuolly restored « ne) e c7 HOFS file System designed for stor seme wey craraccter sees «They are fe @ Sopport for very large files. ® Commodity Hartduoste. ® streaming dora access @ High - latencey dota actess joes Of Seent files © mouttiple writers , arbitiarg $ile modifi cantons @ moving computation % than moving dota. By © sepporr for very lage Hat Be GOPAL Risin “uy tho soppet Fi ore’, hundreds of erega bytes , Gigp Byly aera dgts 1 Size: tHadeop clu Ce yanning ecnnsl oat Store , of dom. VHX, lly oO 2 wy Fenast, pe ob NO Kel te sane oe 5) comnodty Hadar’ = segue ae ~ commodity Pandewane [ The Hiw oe OFS requires Cs unin Beet quailable for most of the vendtors] cop des “Tot reqyire high configuration ne, empenane Sho to. be part OF it's basic engtaniod ons . OFS Bw aways ceorting witht a notice ble interruption to the ealy.['Inm the face of -fatbever] For the Comecdt 0, ty Hardware chance of eke. failure — across the cluster 4 high, ot lest for longe clesters ©) Searing deme KES im mest effierent data processing 2 noes is the jy write oneajread erany time » porter: Thox = pattern e Hors will foil Streaming dora access [Gequential $ 5 ~ search for 6564 record > Search came trom 1 to 8564 Sepentig (wo vandam , tadex cee $$) ng @ Sequentfal Ftec meas hen on DES file , dara in the seqpential fasnion onte ut on Store a fik on top of MDFS $file we cpm “ot overs de trot file, aegtant oe “an Ke wee, file WhO local environment , TF oe enoify the sentent om oe on pur the Sore file on top F c By deleting whole HDFS file. (4 \ cco c| O sigh -tencey dare geass: — ese 1 a Generally applications thot - , : uy p regs re too -lareney acess 2 H we are wing 7 HDFS wed to : Ne ‘ qureunt of data Because 10 tate raone Se ie roy be ar the expemse of latency ad ceed nential aceoss) Roems (Rardemly Access) -- sar> ~ -)en. ain y = yen"? = \oo . var ge? Fe oat ci Oe 6 \ 2" vo x0 Ny aot goon: eo te osr08” nae eee a eo ux TO ater: ete “nore Se tn the RM OF milliseconds = vange , ewoill mOt work well coith HDFS ; Bye. GOPAL RRisaus @ lors of Seon fils y- Nore Oona 6 name wede & resporuible for maintaini the erera.dosa tofermarion of the rodacp file system - padeop file Sysen hove on the of memory no Of $l tn the amount Namevode -@ mules orliee , qeelnary ie metiyremens e fils in HDFS may be written to by Single . o ey tere 2 writers are Gherrys made at the end of -the File - | e thee &% 0 Support for multiple corfters Cory modifications abr arbitrary offset tn the file. [these might be opported tn the fexture, tout they are Ukely to be relatively) 9 seoying cempytatizn 2 say than moving alata: Bo Traditfonol cist bukd — Sysiern dota fren SAW Shared pccess NetwoorK) device will be copied to+ 1A computation requested Wy an rate ty much more ef firers | exeautal ean the En Operates em. Tha peg gre Of the. dara set | huge. a 4 ae eB exe WHEN the che auemption 4 that fr uy ofien berky to migrax ushere the dow & tocaied Faeries | TAM, . ; rot closer to ort To dota to ewhere the app ttaribn they “3Knfaces “or apptation +0 moe themselves the data % lecotad- wars provid closer to ashere Amitarioms Of HDFS .- ARR SOR 2 6 ceeHadoop eae will be classi fied. tanto 5 > podoop Arenitecture t's fanctionaléty. @ vatareode and it's functfonality @ porwecter ove te fanctforatity. @ yosnracker and it's -furctfonality. © secordany nomencde ord it's fantrronaltty. By Mr. GOPAL KRI: follows rmaster~ Slaves avenitectare - oe > tedeop architecture mane mo . oe ree Kelly Technolog!es Par Wo. 212, 2nd FIGOT, nagar, HyderabadS00 71 asvorpet gras oat SO 67E9Se \ N. = | (anal aob)pop)lpan Warne ode 2— «The easter note fy Had@p architecture Tt is tented as nome rode. » Wamenode % responsible for maintaining -the metadata, fofermattgn oF the adem Fle System ohare date. obey. prouss ‘ data. aote ee Nomenede maintains the file System. The file System vove metadaia for all the filu & directories. “Thy Yofermation % . Stored persivtantly on the lecal dis. manently Namencde nmintain the file system Namespace. The nomenode executes the cefile , The file operations Une opening , closing & weramivg Fils & directorita- The Namendde wit) opdore te Senpostant permanent fila cr Hadeop File System called the * feame space Ting © eit tag « NouneSpace access te files bg Clierrts « ayBy Combining which blocHs 0 sd File will genesaked By lr. GOPAL KRISHNA Cluster - © A Client access the file System 07 behalf of wer 4 Cornu NICOL sith the Namenod .. « The cient access File system toa portaple opens agen Jnterface (postx) + $0, the user code, das ant tO Kn abt =the Nameniode Blows to atancte lve nodes (through neaxt eeats) of data node los: OFS ° Assign «© Keeps track of e Umitares ve -replicaHen 79 cake S piock eretadata 1 eld «in reemany. sot} van ct & Poemorg eshen tea trong files exter. + a Steglepoint of failure “ty the’ System geome soluliory ttist-2 pare Mode :- ° Dam node Ba plate hold OF tre data ; data +p dotanody only in the form of Hops ‘peo 8 dee 648] ae he fast ‘pera.dota each ble size it patanedes are Sore ant Tetvieve. blocks , reporting to Nomeniede- A client actass the File Soptern 0 behalf of ae user Noles - by Communi tating entth “the Data UB, unde leg, regulon Fo for Storage (ay, xs) + TAKA Come of” dif buon of bieexs, AbIOs, diuky * Deolt we RATD ote ABS rreOMs more . To threughpur. Dots mot Kno abeuk the wet of the. Clas kev (Shave ‘rothing}) ) see. Teacher: = ob Teacker is one of the S}w daemons for’ Heckop Arent tecture: yoo Tracker % responsible for senedutling & Reschedlult the tBKS 7 the form of mapRecuce 008 + a qpomacker th alto getting the ack nowldgement Graponte] aK from the © qosn Tracker . Greneratly ‘qoomacner «will seid, on’ Pp OF the Namestee, jobtwacker manages the mapreduce Jobs | distibutes Irdividual Isr to machin winning the TasK Tradey, cece@ es Tracker +_ «TASK TIACKEY = FespOnsible for P tated teividual map § reluce coors. v Soto © Tsk Tracuer is abo Known of eee Sf daemon for © Task Tracker is primarily vetponsibl, for execesy. the 9 el | tosys, S879 by the yobTracker 3 - ; ae 2 the form ~ of ( © Gerevayt TOK tacKer aii! -yeckdex on top of the : Pata . — 5y Mtr. copay KRishing, @© yoracker & yasK Tracker Gre the eo tmportant an dozmory of Hadaop architecture, which are com, responsible for the pressing Of the data bey the mens oO mapReduce programming. les, © Sesetay Nocenede :- Kelly Tecnnclege . | Secondoay name tode 1 deprecated. Precieretrer iy . ‘Ph: 048-6462 6729, + TE perforns — periackic cheenpoints Of the warnespace. | ont helps Keep the Size of file Containing leg ob re) enodificaters within certain limits at the. Namentele + Te & veplaced by Checy paint note . Secondary namenote evi! att as Separaie physi cay . file . + chereves the primary made iy down in Hedesp ~ arenitecture , the Secendary .nomenate will Come into picture, * ax Con mever ever call a Secondary Ramenade ag 1! the veplica car primary Node. : + Secomdory — warmenode iy esperdiete yf the FST enage span & weeny , daily. the checxporntimadtss periodically — creoied cheer po748 of the eromes face: copay the wresuce Bear mechanism & speculative enaion Hadeop ? punt - ons : 9°" Pond dosmon « within 10 MIN time 4 fails to ‘response be. rwame node wor ox trae moda oither it wad , anv, slaw (or not -fumettooality oleto temediaely te gobtracker whl assign ee Sve ean to ane OFT with = hadeop ae “ps Keon 08 © SEE HiBeta Storage fo HOFS ese rooster - slave i G HDES architecture ce a ty HORS CUBIC coma apater node Oo pomenode: qoame node, the file Sgptero nose , Space regulates clients By Mr. GOPAL KRI data to acess 1 1% an Hors, file Sytem Name space allows oes fe & spit ito BleKs- pe stove f9 File Tooternally Blows OFF gtored = Ht St of potencies Narme node executes the file Sgpoten rae Space operations gy reramig fin & directories. i Une open 9 , closing " Senile te crapping Op blecrs 40 patanedes - parancdy Ove upon we ge ems we Re AO poo, abo perform blow creation, deletion and prstruction from the namenode.Teboduction apaut blocns » A any « MDFS & designed to Support very large datasets HORS Supports corte once] read many times a pn Fl. : Tn HOPS caddis Split into blacks and” diate buted eultiple des i the Chester. across: gach bieK 4 typraatly — ume (Od. 128 mB in Size. veptiaated muitipla time. gy defeult a time. Replicas are . Each bIcK replication factor of bien stored =O”) different data nodes - + toFs orilizes the lotal file Sytem to store ath HORS . blew os @ — Seporore file . { con net be Cempared with the traditions, HDFS — blecnsize fir Siem block Size + 203% Replication) ;- . fin application, san, gpectly the ro-of veplicar of a Bk. an : tation factor can be opecttied at file me and Con be changed later. she creation tf sthe tepli Cation factor are configurante per . haB-Sik-xot Fe ae |S pefawtt seplication factor of $e ~ 3 7 o-of COpid Licasion Oe Ble pete . 0 shor Fle “ t wsThe — placernent Of the replt 7 eplicas oa ty nk performance . % critical to HpES + optimizing ‘replica placement distinguishes Hy other aabi bud file systems. 3 DFS form Rac -omore replica placement Ss geal: fenprove weliani lity , ovail sity Sl Nie barcuoldth utilization: Rexearch OPC many wacKs , Com yelfapili muni Cation between yacKs OTe th bh Sypitthe + peruork — bancenidth vac, Be greater phan — those howrencds, determines the wacnid for Replicas: ore typically placed unique racege® OSES, puccen oeepening on me S07 an different acs: each = catandle . gtenple bar mon - cpeieral . ar are expensive tye see” PO . Replication factor & 3 ’ ePcegs, yeucen torre 7 a fn sre on a ode BH tecal 700K | acfferent noe in the local ay Replicas are placed in a differcnt rack. one oO on a node ome © V5 oF the vepica on arate, yo om Omen GT disp buted evenly across verraining TOCKs pepuce seution © By th GOPAL Ke Replica se uckion foe READ operotfon : PFS eens tne band width comumption Jarency Lig were Oo repo on He Reoster amen prefertd: oe that ple data antes + repli OFS cluster Oy spor malt in te lol dora conter prefered over the remote one:by interacting qwith the > HOFS They are pest appronene’ * toterfaces have 00 © Commandilire Interface - @ joobaid Approach © ) commonsie Toca Peer dling rarer face ayerorers 5 te MASE “or commoratire Eoeerface ee fone of the simplest art many enteractive Shell. Rd %, a i eae Ne we I e oeThe pevsinrone of fu = os fer fe) RE Spier metadata ;- Ta the HOFS, mamesvcde is responsible fer the file System mame space & ectit lay. : o> pnerodare howe. . wamenode wot) only updore nega gaafile © fismage tee yo aoe ah SA ot? ©- edie leg rea otha formoge :- or « the fierage fle wo pessistent cnecnpornt = of the File Syston eetadara: c powever Te Toe Oma for every file Syern vorite operation Since coritt cut the ftroge file which an gow © be giepdgta 1 size, ewoutd be vex Skeo Sma act net compromise resibena - mg ene By Er, COPAL KRisHEA conen, nosnenade F2hlh, = hen, we Tp the cramenaiz foils , then the latest sta of it's be veconatruckd = by leading the {Image petador con enio memory, then applying Ath of the fom ABH ° ppeaifons 1 the edt ag: joe py fact» *O% is precisely hat the Tame rede | ot sta OP C leon about Safe mode] ee | editleg :— 2 ent 9 fie , |g qehen a akan, Syke client performs a ewotite or moving & fle) uw | operation (such on creating | first record 30 edit log the mromencia oho hos in memory represenntatio? of teen mmere.daia , which FE — updates after | the tne Fle Se eattlog Os been moatified y Editleg to perenne tty wecord every Change that caus to file Sysgreen 2 Heat wegpeit a data: «the | ine memory cetadata “% usd «#0 Stne rapes: «othe edittog 4 cortte be fore @ flushed ant synced “after Cvery hyde a Success Code ik verened tO the ctfent © . » for Noumenodes thot wre to muliple Airectories, the covite muse Be ‘Pure and Synced TO OCI pefere rerenr nn Guccassfaatly: q 2 This ere nat no operation we lost cue to machine foilure: be. « crearing the mew Fite tn DES: a ‘record enone node f° prtlotly » charging the replication’ factor of a a neo yecor tO be Pnterte Into pruert into the Celttlag. a Si file Gwe athe Garttog: The pamencde sO file Gila gypien 40 store the editlog- gn tts localhost, OS a HDFS poame Space iw Stored PY the vaste: = uame node. we a sranzaction tog catleq the editleg: se pRs rarespae gmcbuding the mapping of ee we stord ima fila Giles 08 File Soplem proper called BS EmBGE’ + potenge Ue stored oS file i the mosme nodes loca file Speen teo- entire File Sys the nase node. Keeps on perage Of the Bleck map 7 MMENY: vromespace file» while doing tue Safe rode i- AAR ARRAS cohenever — clasier a hadeop Certain things esi done be the Some ening 4 Norreniccte Oo loading Al System related Configenation files. @® cheon fer the GeiSfactory replication Gr we doe. @ All Sysrern velotet deperdent files. atove all optrartons- the Namenode is tops part oP ead -only—rmode [ HORS Can rot be yeachey at that moment] this Stage i Kran as Gafemode oy After doing olf thee Stuff anttomrai Safe mate Comecut Of Safernode On >oFF ich meang be accessable meade. voll! tees coil But sometime" go-Ramode witl net be turned fmto oFE mode. AE that pone Of time Hadesp adminshot, to ren the buew Command. ®y Mtr, Gop PAL Ky podeop = dfsadmin — — srferrode leave of Dfhemede tf OFFlode fe Seen i gq the compiley to intrack_ radeep fs ’ pith naz local envi ‘ pvironment to HORS environment. podeop 8 B MOF Sopport the =v? Command » Because ors epriteorne - padeop & Support touche Commend . podaop fs loons only HORS directories & file ber not for \ocardirectors & files. tm woes file averteading — is not possible « tg ther o78 of the file moving one environment 40 sincr environment rere Fave only default previtlagel- . ewe con Ot “crear fie on top of HDFS ithe fil, on locau directory. cat, cat 1 Xe ZO eee ee . i oat e qrnUt oat sxcparh ocicop fe a rat Support revazons @v Sofelnns padeop & AH wot implement Ex0r infer marion ww Sent to ye gemk to SAC cccsaHonteesk mestontin of odep chute i i cod by i ope tee file: ft wish, tO 7 2 The cent the Fle System object » whith % on pastana oF DFS: ‘ snomendde , using RPC, to dekrmine of the blocks «= for the first few - file. For each blog , tre rome nedke,eturms the = add vesSes of the datanous thor mve a copy thar bien and the datanedes are stored according to thefr proximity +40 the clfent- a PS Patotnput Stream (an input Stream the «prs yours client for to euppents file Seers) 40 the weod coxa froTn- client — thea ONEay tees: col yeod() on the Stram- i yes crs tothe fist (closest) datanede for DFS Stream conn: tn the file: she first bloc gereared fram the ,datanode back to the client, Stream. eohich eas, read td reportedly on the \ > reckof the BICC we veached , DFSInputSheam Suepnnection tO the dotanade , then ashen the att) close Seed we west, -sieddnra. not for the nttr bIOCH - eohen the clfent ras finished reading , 7 calla closec) pepata Input Shem - on the mye Kelly Technologies nating DAI ta tha. 72, and Fl cient «et OK Anapurst Sion JaivaSncovs, Chen paarpat, Hydorabad-500 016. sa 6789 r 3 ~s n & roo* The client creates the file b . 4 Callicng Created) on OFS + DFS eaHes an RPC call to the mamenods to create a new file ww te Silaystem's — namespace , coith 0 piocas. assoiakd — eoith it. the rome node perfor vostous — CheCHS «tO AH, cur te file doesn't Already exist, and the a ’ nthe peg rerum Oo Fepatacuparsheam for the client to grant eri ting dota to: «as the cltent corites aor, DFS cutpub Sham pits ‘ tr ito pocrets, usbicn it writs fo an Pinter nod quae, cata the dato. queue. By lt. GOPAL KRi: ww consumed by the data nue the pata Steamer é ft UU to OBR ° the namenede to vyeaponst oi Lit bg pining list of Suitanle replicas. grea we pacrets to the im the prpelere , when gtora the packet as. tt to ehe second dotancte im tre similolly + the Second aotanodg Store me | forwards it te the third (and fast)data nce pipeline: prsourper Stearn ako rpinteiry 07 prucy nal Gpeue iti to re = acancnl of cress are eoatt' "Y Prenesis catlad we ACK: Gee Kelly 10S ares by ’ Fiat No Aa ag Ot Aenoerpee 09.88 510.6789) A pocnet is weeroved form the ac ra | ge hos been acrwmnnlaciged ky all the ony chen | be the pipette daranddes |what Tf 9 quranese pes on o SE: ) The pipeline a y Any packets IN the ACK GH RE closed are added to the 40 that dotanodey that front of the date qusee ore daonstearo frorn the foiled nade wit! ot miss any pacnets: foiled datanede apiil be deletey rege rent trighe! for nant of dora uploads 9) pe nologies Miss sane emmgure yeasts? 7-500 018 oe) Ammamenspiga e+be acassed fiom opplicatfons Pa enany . HRS On different warps . yors provid oO gave APL for applications tO Ue. _ arte Preset can alo be used * browse the files of an OFS postance » work & TD progress to expose = HDFS through the peppy pxorocel: Fg shell ' pe organfeed rn the form JOOS ures dara to and directoria- te provides a command line wh ed FS Shell: enterocr Fesrel, ofr doi Hors - worst of «file. anrerface * cou ~The ntax Of Hh Command set Similor to otha gneits (8 pap, oy) thet wes = are alseady familiar. wth. pe shell yargeta for opplicatfons need scaptteg enteract — ewrth the Stored daa \ 2 to ve jargungs | pes Admin i ect ts UNd tor odentorsivaticg ed a command shut Ore commands = rat Ee wed only wrcesser TQ Be Saat ane enroit COD wen CwebServer 40 eXpox | yy configurab TEP fore | aa to madigate the Mors RSPR, contents of ts sR fil using it Weds ter wo pnotogle® Kolly Te 2. a ¢ co| aD fon [Et F cgace Reclowenvton (Re Suse & cttteres J :- shen a file ke duleted by a CH OF ap application 4 wor generectately vemrmoved from HDFS- tt toa Fike. im the | Tiash Hors fixgr eras ivectory- the file can % yestored qpickly o4 long 04 Fr tn /Reah- By Mr. GOPAL KRISHMA a Configurable amos fra fe Pn hash, t rerrnl s lL. oa file verornt 79 ! moe. erfier the ep of : of # ae pume node delet the fu from the HORS TAME poe : q F | ele qhe deletfon He Come the btocygny SeseanEBEPeorin the file 10 te fread: Fiat Heck, Na a ecpet HMydorabad-500 018. i ets e189, 008 510 STE t vu (ee qthere Could be appreciable a fie B delet by o tre Hime recreate 1H free: Space 77 noes:] | of we conapondg Pg user con undelete Su afer duletfeg fe 04 long as Oa vemnni’ tn the Jtvosh rectory if a wea wants ul ao undelete file thar he] She ‘hoa deleted , he| She | con ravine the fash directory ond = rehieve the file. | = }ereah directory contain only the fort Copy | of the file thar wos deleted | tb Uke any ether rectory S| the Irosh dixcory 8 ” cen one speed fetntgs HFS applies Spectre police eng aaluet FU this altrectory. , te autorreit Cle ThE current = default policy %& to dele flu from J pros thor te = moe thon Chun old. this policy Will be Configurable an the Fete, syrough & evel defined — Tokerface «decrane Reptosfon fates + 3 when the veplitaon factor of fil i veduced are. Name nvode. Select excess veplicag thar aa te deluied - eo The "tor Hast bear tramfes thu tn formant +o. the Dara nade. fr on corresponding blocky wemoves the in. the comespording — free space appears «othe «patanode then arg the clung bees f07 eight be time delay between of the Set Replicaifon API Gail of free Space in the cluster, (ser, oF machine ) 2 been Tm DAA A D1 | Linus Commands riufotrctory Basics | @ gory > maxing tre cimctog ee mnrdir anttha ol _ | © aoe the directory. & rtra cd | cA ant By Mr. GOPAL KRISHBA | Jonttra 4 commands |@ Cleon —> clean the _5 present enorking directory: jo pe 1 @ dare > dipay Me © 1 ® uso amt > display the OT t , {oO = > pease Als 1® \s 5 Ust fils 1@ P a) copy fil a fer Cp SOI eter — Sample OF 3p ot atupee as on a ae ern orn * | Se mv = renaming the fils ope eee a” oe em eR |. we | @ In — Yor fils | ae ln carople: HE at canple jn vcreahng pord link | tet y <1 @& wwwatr 9 verove the Serory mst anita d Be uate he directory:Fue 29 > wor a view files & tnpues HO d ry YO nadep Note Bok 5 > Page whrcugh Flas ay (Uss trput teed hig FO podcop Tinga tte (END) “~ \ pead > view Lie oegi nin, ci 10 records dis ’ ee neod enput ter t payed) | NES. cs . ) Tail sSquied: Ale ending ar yo records a s rans Mboput bir: ) Nes note. Bok 3 17> eteatens Lines 3 Dz xxd > > wr vie y xa > view Tex ANE Lila + File creation & editing © emus > qexr edito¥ anvol 2 gi oa @ pice a Ter editor a i101 gnome 3 gait exal @ “ _y tex edt tor @ ee @ veer? ger dafoutt Fle Cs ! poectiors D sophie > aur] wore ex) oo oSOQ Stor > puplay file arti bute. -\ 717 utmp 8B orange aduonad fia attibute- Lise odventad file attri buies© comp _ 3055 iY Canty)Q zip Compress fis aaneaten soe? Ele goeentaee cms om EE oy df{2 Shoo free disnspace 3 rane a disk accessible iprm > gunove ® wmassem > Compute Check Sums. \ @ beip2 > Compress flu (Bzép2/) Qeindans up) | i tate Sows fale Kunze a operatfons OF SERRE © agg 3 compote fils Ure by Joon 9 leortop Spelling Ure ee diff tnpate Hr Sampeht J asp > eee ae to @® comm —» compare Sorted Filer spell > check Spellteg & comm — Popa te Sample: tet d Ben. Batch. ae © ounp > compare fils byte See by bate eye cmp inpur tar Sompa: ter J ae peckops al Reaaie Storage mt 3 contol a tape device dump > PacKUp din restore > Restore a dump 4a 3 Read] wile tape archives at vecord 3 Burn a CD, r -3 minors a Set OF “ee fils. fudio video gp > ploy cps & Tipps 5 xenms > Ploy Audio frlezprocessce @ ps > Ust all processes 3 lst wes processes eo” @ optieres View ane Eyer lead ® +op "> morito proesse @ xvod corivos spe 108 ® fee > puplay Hee eresrOry, @ wil > Tex mite process ; 6 mice > eek process priori ® venice > Change process prions ® @ a o ® © ® ® > © @ =e" BL ea aa Te ANNnetwor King ento remdoie hosts @ hs securely |g @® jelret > 1a into emote hosts scp > securely COPY filg between peeee ® ® #P > 8 e \ 6 > copy fila bla PG: © evolution 3 OOF email CUent e@ youtt > texrrboued eroil clfent 108 \o' | eat! 3 reinena) — éreb! cli no lOR ene catty oct es eect, wolla > eb WTOUses wo 2Nin, NAN 500 8 6 sooth Beara 85 i? @ Wr sexr only qeb - Browser ARCS nee @ wot Rewieve uebpog +o ausK @ erin > Read wert nets ~ : . gor = teurant messaging ERC ® } py bir, GOPAL KRISHMY ® 4alk 2 uur] ent chat i ® corte 3 sec op to terminal |® 9 3 prohibit talk] eortie BEES BleKE =~ . BlocHs Ore traditionally either Gum oF ems. , Pefoule wu Bums: the aie Seens 29 Compares tre motivation i to Mintmire jo tro.nufer Fore spransfer’ > The +0 SCH « —Time +0 Pox | ta Soy geextine = 10" erenager Ove = eo mel transfer vale. Lacntee seen time of Lh sBlcKs?ze onl need to be = lcoMB5 what % the difference ae FS shell elas to generic which On ony file Sypiers et: D FS 7 fi Spren poror yo ure local, HOFS ~~ a pe .srell MS jovored 4 pini.tiodep® fe au athe FS shell como, sqoke posh URE argucents: © The ‘ORT forrat serene: jJombority | @AP- the | Scheme gaurthority are onal + Bites, “BleIn sapecsnea, we defautt ecrere ified in te CO guration - 4 ured © parent Jenild pata rnst [ponent nid oe Cjpeneor ome) vemween pS Shell & OFS Sell pried fu Sgt ® \ pes Shell | 2e | | © ops % vay specific to HDFS - ors she & yovored Yn] hadep dfs. Al the commands tone argumens: re qutdrity] path sre scheme OM optfona - thority are tf vor specified , the défautt scheme Spe efied in te confige avon je ted. | | @ rr rhe HOFS aw rats the scene pa wore fe O eB jpavent |ont os raga: ae ventjonitd - jeorent cnet rears fy HADES corresponding pnast of the com) Ipchove LikeSel ———EE indo file System Crodemp &) :- « hadaop fadicating the compiler to fnteract with Lecal environ ment to DFS environment ttnux + yadap fs & not S the + | ee . upport ~vi Commard+ Because | pops writeonce: » hadeop — i Support fos tetichz, Comennd. » preset not? poun Kelly Technologies Flat No. 212, 2nd Poog, ‘Ameorpat, Hydra! 016. sre path: ‘Ph: 040-0462 6789, BGS 570 6759 , radeop & lwHs the only HES directory but Mot - Local divectory: ewe can not create a,fik on top of HOFS- fie on local. create the HDFS - Ge @n . ae Gon | ge con wet update fie of top g paortoms en \acol, after trot file put foto the H0FS- By Mr, GOPAL KRISHMA “Hadteop fe & wal doe ee Zoppoct tordinns (on Soft WOKS » hodeop & oe not &nplement User quota. + Exvor eafermarion Sent to Stdery & cenpur “w Gent 40. Stdcut puplay detailed help for a comman. hodasp & —hele 2 commnnl- name>HDF. =—_— 5S shell? — O cok — conten’ S$ Shell Gmeranay of ae . the it display Coo View. content aw er version cLAss NAME ® classpath @ balancer) @ doxmon ly i daranede | @ ainsi © ooeaner a) : (6) apps ode ( g secon ane praaK HORE! J @ safewote xtO |\mxdiy —s moKe the divector el radleop fo —enrdty fant reac te 4 yas Cea 29 | rodeop fs —mediy aniita of Sik wy -creakd on dtfauit holfs path [leser ee se e Oat, ewe es 8 en ® [help > display | belp for all Com radeop A =nedp J e 8 led TE path % nor Sectfied. & —-& Shoo the amare OF space, th eure, ened files tMmr eaten the Specttred frie ey & hostep A elk hacdeop & Adu < path > “Gil Coutirectory. @ ae a ve Capacity , fice amt cued space Of ge ee Fhvthe, Sila Sysiem has mney ut bie : Pe partitions ant wo path” to a particule faxtitten i Specified shen the Staras” of the paxtitfons coil! pe ‘Showon hodep fR ~olf @ avepesy dopga wit ys thar motth parry esre> fo } tration > | malt: erhen Copying mettiple fila The must be a dincier. - & hademp & ~cp anteha | @npors te Jon? thas . fF paths i be “Bors paths le es @ [tv > rove fils that mach the eptotfiat File patter | 2 <$re> to ABI! gehen rrovieg —muttfple filu , te desttnatton “ poust be a “directory: hadaop & -mv <,_eat poth> qnuar dosnt, 2h : deban 8112 PO wis OSE ey, 9 ae Snot rags ©€ces-drrectories Ken ym —> remove vecunss . Oo » Te sively. ith eo O1O ing Bic? 2g i we podeop fs samy i ~ a u & directory ® | pt > copy single sre (oy muttiple Sid From lecas File Sy to the destivation file Syilem- Also ready Pep stdin and cortter ta destination file System j & padeop f& —pat | fo par Pa ‘ poun, we | \oco wors party por:up, hadeop fs -gt Zarc > cops! a yoo) 5 Warn ean ypors +o Local OOS ne fortuna (aise a > yet path © ft Kel prenk D (expe > empty the tur Ny Tee’ dango Preetper Bi “as wp: hodeop fs -get divectory- Xe adap 6 From Local 5 ey sige ST @ roultiple src's from i) : Lo Syste yo destimivon Fit sie) 49, podeep Bi - ve frarmlacal zlecarsre> cde > Bd oo: yad@p B ekmege << local destination P 3 ae. poss pan yoco! par: 1. can Pe ser to erane adding a rewire ante addmb A ‘cnavacke” ar the end of each file: 7 _germerge eorepath > < EIEN” as adieop fe -9! 1g ‘ © opens Weors- a@ text S TaKes z oe a Sauce File eas ond cute tne hte = the allowed dora ave 2g P& Text Read Inet Stream J sy0r- hodeop =& -text By Le chek to see iF the file exists werarns HE bee _y checx +0 See te the fle ik ze0 Length vetarms 0 FF true wetern 1 tf path A directory ad chen ele ett OF hone 649: fadasp fs —test ~erd x A ‘ hedeop 3 — ear -& oe | podeop A — sr ~2 Returns the Stat Prformation on por. sod ¥ —stor < path ame HOES @ [yeti ouplys test Wlobyit Of the File to “sider on Con De AE A tn uni } J hed C | 83> Fadeop ___=f opti syne radcop A - tall < pathname s ah PY te COPaL key |@ [eceps cargo we repletion factor Fa Ale. for recursively intresting the -R aptfon aepicaion factor of fila wienin a | - > ranp 8 gene fuse [a —R v | x 2 yies posh oo. me>&} crgrp > change _ syn:- hadeoP % —chgre -R (Groep eR. g (Go BE el gy car? gut =R erOne fadeop fs —chayP group amodiaiton Gt fila. with Ff the cree recursively through the que be the cener Of fils (ou che a A _p? a ay RadooP Re cp paver ly threg cuner OF FUE or else ja ap wend BY COAT uo 3 v2 8 STOP Oey a reas OS O28 DoCS SES erie By ewner Cornedeel user Commands @ archive ;_ Hadaop Stores the Seal filly ineffiaenty euch a3 each file get stored) in a blot .& Namencele pos to Keep the metadata Prformation fn memory g0 with this yeoson most Of the = Namenode | memory win ger eat op bg aris coral Files only —wshich resus ar a Coatage , of memory: game problm ut wie badeop to avoid the exieion for all the oxcnives x) han fils (-PO* Cys archive $48) cohen Creat d archive directory whe inpur iy aa educe yoos , 90 GF SOP cot hadeop a tnpar fer our mapreduce programming, = By Mr. Gop; " 7 tas + GOPAL Kristian, gpeetal format archives vot gente maps toa He Seem directory . always foo ee hax extension 5 directory contains eetagaia (79 the poder are _moneinkr) and. Ee file. contol? a archive come axe gant Een tn : Sra \ocoHon eu ane Syl creare the ar cnive Sler- yadcop archive —axchtve Name wame TPesre> ay yadeop «= archive _archiveName — fear ht =p pAnitta loupe paaeop B ~US" Fegarenne ef « Fadcop fh 1Sr feegaventne| P08: hav et podeop fF - OF Jeryaxenive|-fba+hox] Pau—o© pusteg. dattbuted copy | The dutep command % tool wed for large inter and cluster coPgIOD: into oe rave qnuttiple haceop clistes are running we et pause geveral Heranyts of dara one cluster to another” fy hadeop clusters ave loaded with eerdoy Ks a dara: soe will one forever tO transfer terabytes oF dota ‘fom Oe Clutter to another: . putt cured no povaiel copying of data @n te god any & or & wmr distep dees. year dara from egiutior ~ patep uM mopreduce Job to | trantfer +o another* roast pa >} gorolfeo|ion \ He ayo. [padeop distep jecop onnep rage tmoriael EY gjimnat & t ie} Kol Tech: gly Technol” ge USES Aen jarani Of. Phe Geo wescvens: mpgitige ais ata 1m +8000) Feo! PO \ rasteop fe guapins o52e] F001? paSselnna, soro) 32% [feo el pagina 39020) srcltst “\ weipid nna: 8020] toa} S00 @ [Ba _Rum_& gene fru syprer cltear suo vant 2 ypdop fs (Generte -optiory } [coment oP - zor ORM pose? ce@yar> Rar gor file. Users Can Bundle ther eapreduce code in a jan file and erecuke tt WING = th Command ay streaming yous ave wun we Command. padeop sheaming ba utility tar Comes with tre hadeop abt bution» eyo,- hadeop jot : p jan edsce rogram corrtten fo comp prog ous Ruby , pytNon ant ct prog ramening dota excel for Shactu red dota and cut F & Torey { ovtomanre poviatlelézarton & dist bution Fautee tolerate » T]o scnedulinge © pront toring ond status uit eae weeserw™ MapRedace Queries ae 7g dora 09 padeopeety Bitten hi oa + appl wing yan” . _ yeHfons specify the tepals processi rapRedurce paradign: oe set yoo usually epifts the tmpot dora ~ chunks » ehicn are processed wy the qodependint a campetely ponaitel eranner* coos ene The pnpur ond PE the Framecworh taker core ord ve execuley the ce-pur —tecationg functfory Via Hadenp Panter face ant Reducer. These, ard other parameien b configuration » The Hadeqp yob client ob (jeslenccuranle , ete) ant Configerarion ousumes — the res porwi bility rhe JO! packer, canfch then te TON ahitbakng = the softecare | sgomuton 0. Oeslaves , ccheduling terns and on oa monitoring them , jprovidling ne diag™ Stic informaiton %0 te gob-clrent 5 . -client. + The rrop| Reduce -famennrk opel exclusively on < ey Vou a —trat & , the framewor vices the of | < Key, valu rt inpat ro the Job , ed pain and produces @ Seb yo! 4 oa et key values Pu the cutpur of the ob , Concetvably of different *YPH coho iy erapRedcs + coco S08 + gable eet -ng node! feorn LISP pregra.omt tex -femeronal language. » Gay to disibut acvoss necks Lice very |fotlure Semantiy posed «dig tetbuec comput. eqprem taHa core of we pouitiontng of ong the program "s execution + gore] merge ae The under laying the Tnpur dara, Sonedo Ut acwss sever ‘maching , warding machine failures , ard moanegi required enmity machine communicaiion- , . compuratforal proce sine occu on OH) in se ote 4 wnsracured dota * Sysitre « ON 7 ghesetured dasa + COH0NO% we “pee ae * Tried and tesied PD production ee . options - » oreny penglementaH oo relia, oF gepports thousands ok nady + Foutt _tolant , ord petarytes of data.mesiyarfon for mapReduce Ceond): - 1 jarge Stole para parce ssing. By Bir. COPAL KRISH A wel. want tO we 10'S Of CPEs. » Gur don't cant hawk t. 7 | ° a managing things | prempreduice archtHchuse providy . te putallelizatten yy distribution we epnttort NG, nagteuss 'Bereaniers + © nent wal the Order fm RIED Me maps ov. vecluctiony props and * Reclucty ie erport Guatlelism , gor eed . for © $ re on data genera 1 the Same mapRecee oo ChE gxoncles) eth a erdlex eptll abcoys be faster thon operators do rot tore place untill all maps ae ere (or rave fatied then been Sripped). ayumpion thar the output OF Reduce & Smaller enpat t0 rap + large dara Source Wed to final vabuu .ormeteduce Architecture: come file ye Fi le of ere atvided foro mute . oe chunk > wu be las spre hole process OF empl . ¥ @ eoracnt @ os TK scent oss ewes persone os 1 au the avatlable jpfedace TO wep ack fey to ack upon the Cr 2 Dake send tre proges youve ard they on Timely barr wannark ey chunks and each & arf ferent modu - wil be controlled by en the fom Of the eTracnes. assigned - tara, formation 40 thd« Incoue hoz ~ 7 D of fatlnee CLort t84KS) S Tob Tracker cuttl yeaseay) @. the tam tO Some other daly fe : fale avetlabe ay: TrACKers , 5 te conststs of tp phaser: map amd Reduce « By bn coapieduee © THe PARRY ee cohen the rap function — Stars pacing ouetpert ey Tot step written 0 dK. The proccss & more envolwed + ond advantage oF buffering eorPta Tn oun Ont dorog sore presorting for efficiency reasems. . ran rote Circular memory = buffer that FE . cath, MP to. The ‘buffer & Ico MB by default pod on the Site: when the Contents of certain thershoid Size BO @ a wut start to spill The Conients to. dash, the rap etl biccK IDET the dy” 4 wosh tes FO dix, dota & FER vue reducers Each partion, Sore ep there & a combiner funcaton , + of the Sor GO Mae ft weods dota ! CPR atpus zero oF wore Keyfvalue pata piay 2 The Mappt $ Sees PSsnopheduc the Rewer reppourpar Fe & Sitting on the fecal airs of weduce tak need, the wap cutpur pouitfen frorn Several Tasky The peachtne» ThE for ft's parson across the cluster’ the rap toaKs = MY finish ar diffrent times the veduce tory ateus Copying — their cotpuls a ean completes Ths % Kon Os thé copy econ prose of the py default 5 threods thu reduce tO, Gan wpy, we can change by property: te reap Oo Ko haw et e been Copied , the me Sort prose FOr exanngk, ef there were and the merge -facion unas 10° then theve eoould be 5 yond. Each rourd could rMHOTge to files anto one , SO OF he cmd there coculd be five enter ened Pate reannes filys FN youth 7 qrar merge rhue 5 Sls ant wove *0 » the prose ww puree aypically HDFS- cohen al yeduice tun roves BO map Epos » Super tr the Ove Bagra » oes ae gn ten ocd ate values for a give? poke rnediate Key are combined together soto a US a given Oo Reducer : ans “st °w othere ray bE & Single Reducer, muteiple Reducers + tho 4% gpectfied «of part OF the Job comfiguration AU values asociakd with @ pod? cular Panter mecliase quarantead +0 9° the Same Redueer. Rey ar enterenediare Key , and their value Usig , sed to the Reducer 7m Sorted Key order “suffle & Sort’. 2 The are “pas. . Thy step & Rneon as the tre Reducer outpulh = 2er0 or TMOrE feral Key fralue ey ur, g Eis * OPAL is crete ave weritten $0 ORS KRiStiaug In P fee gre Reuter wally ent a . OO / sagt eg | value pote for ene paper Key, 2 Reducers 0 Copuls Redurce FLO me of the porefom (Reducer, Rerordeortier , carport)» + €ach moaptedurs fess’ programming See? @ offferent proses of © rrap Reduce algorithm @ different gata types 7 conp Reduce 7 i rhe proces, 3, » mapReduce coors HY Dang $5 ato P En . @ crapper pre apie scat l abe, @ ork& ghusfle pre Ctogica! I ayy © Reduce phe é corn pra 0s Key value pais ag ° c efor « rage ge Og anette atuces fecttony.. (mapReduce) eo orp (5) eotil expect the Pnpat en the form of orpRecdusce Crey value) pats from HOS — fayer only» Once Te done with the — proce SSI jt will produte the ofp HDFS 7 the form of Chey, value) pa. agin of tOP OF i , Us cv) (K tise (Kn) | i}‘ sn tre erap pre Koy value iy ¢ ? fn the for Byte off Set volues + @ of » A ust oF data elements are provided to furcit Tapper NCHON called the apc) whicn bansforms — tnput nt ae cutpur dota tlement+ tntermediare oye, y Mr, GOPAL t Toput t- Go Ri — epapper SRES Or y Sidaaa, |. an mpi | | __heacop cx Bigdaia Avaliprit the be displayed tn the form of (Kw) pains. are pe cusipat value O leony 1 kv x, Weert) | got & SoetElas'- rapper output 08 tane the sort & Shute prat fopat- Sore t- Gore je ued to Uist the fnpubs in Sorted Oder. Shisha. crapReduce mares The guarantee thar the fnpur to every reducer we cored BY Keys Tr @ Known os the ently.sort § shasta. npur sou a steste er sort 4 shale Pope sort shuffle ousplr rely Se? eon eels an wteator imput value a Reducer Ae ieee» E compins these wale together Seo oa a siege Cuter ed : Reducer ourpari yes Code logic | > final cotpet Driver Code #> ' . | . configurette® level detaily eoith respect +0 “Job , | crearfon A / |. prapper, Reducet class level detail, | | Fira! ourput Key, value. data type detomy - . Tngt and ontpur DES paths - a me : and 9 One capper — Reduce’ i ee (apret eolurce F mapper peau one 2 Ore era.ppey 4k opp ee Sey ese [____ rnpReduce dor Stent rable " floor Proarcorttane | * clocible “Bien ole Cor FORTE long Long tori rouble shied Text boo leaf Bootaanroritanle + mrapheduste atgpyttns s- = configuiaro « In any ernpRedurce program with: irrespective Business logic wil divided anto 2 prod: 7 tex) detail of TARHodeop = mapledute OK SS rene sor dated farts ry gates ee ‘ore 0 ar on SUE aie fil qragen ai ' ! ‘ach omar A \ Record poder ¥ welass: ne prowtd ea ow? Record Render fenplanetaution ane from & FORE “| etfeco Race, + Renae oS i - ey Nolte Rene Roost = | | | | pagrtonemap tesce ea uur BOY PRUoc us, AARARA, g. velneve gob res Of Featve) Ox cae se! “jab qe rok Bear (reterns taux)client submits the soe ~ gnput file 40 the” 2 whenever Cponpreauce FO whe form of 70 . ac : ; gobconf — onjecr C-TaR qovrracey worl receive the file tO the Fe). tompacners fn HE fadeop cluster. awailadie By default every tosK Hacker supports ve will accepts ThE TANG PY pao rece FNS 80 ‘ yoo Tracke! and it worl! _ perform rhe vmraptorKs first ond eens thE progress informatton — back to the soo racer bg ‘the meomt Of eos Goat Hl jy Mr. GOPAL KRISHMA gnformation Seok by the i the progress gooTracner eof! nttate vs gored op > aosnvacner the TOD HOCK, wre ert prove tailed Reducer (pro v7ded Pf the. stomtelther a compoed crap prove 100") the —- Recluces cutpun from veCaive = al! yo her ye res assigned tne + one oor? wl we oe procera when te, catedit - Tolerant Fautt - TORRE OsHtrahey freeing UP terme chia tuk ye tah 1 koam en another cometnues to fat, 3°tracker failure WO RNS weceives nO neascHoend fiom pool of TaKtatKns +0 Schedule yoorraeker doc + Job TIACHcY Removes OAK TACK CT taynon- wo ffar au i aaialy wecieve Feaxkbar “report Fopreaccer made oe Gyene Of HE TOIKNALG ticular . oT IACKEY ® wonects sone. wt | 2 Singe point of foiluve i --7rit - 7 9 7 Foprzackes rede * . wa fap m fhe tog ic there : if arall we are geri ne ly i or all “even ° cot at where COU TAK OF geri con | Ash In this - Core gop tracker > eotll ry tO erecue the Same | ron 4 Em by default ~ | rn whe a areenpts the Save TOK is pen fotleng” | phen Jobracney ean) mar the ernteive yb ay farted tu complete = ashe aul, the tosscopedusting + ~ @ FIFO Scheduler Cootth prtorites) © faty scheduler ® Capacity Scnedutter: FIFO seme con xt P ssn a » Fotr qos are ass p wer gee pore o amber of slots assigned for FOuhs for war er Each pol ott ane same Ta-0f FONE glo by default, every Uset ges Oo gory srove og pre, Che eae overt. = Supp prea npr ses — ee > mony 3005 + scheduler, +~ pole gnc tO. pou Ga Cewtth prion) Graved | pucducrron cluster , go yous cenit thety tern. sdoxitigy for the ops in wlevey ChusteF pool per user by defauct) = the queue C5 qreu worth Kelly Tech nologies poeta bes: 212, rae saa Semen, Hee | stor for @ task ___JUiple uh ny LE “1S Can TUN JODS on the clugier OF th 1 came ve ecampee: By Mr. GOPAL KRI demand 80 780,00 peter Supmit yous rar rary, TOM spectively 420 rans ‘ves | Gay the Custer has Umi $0 allotal, 60 EONS OF most | pefauit benavteat t paribue tk fahly among 3 users that can be atomed in this clases 360 - con ve ‘Ser for a pool antrum = S¥aye satmum Share 0) gta the previous example , Say mary ros aM of 40 ahen tne Test & dittviberted reary nud be anetoted 4» evenly t0 corner OO?D cagaciy Seneduler in ‘ » stetlar goal to Fairy seheduler: pignt bere you fatsly among sens wons with queue tmuttad Of pool thinks he ai the cluster to himset with preg | & . A wer scheduling, but acterally shaving ereny with ones can erganizoten) ues: orien Secu quevsC sprpo scnedelieg 1 ae + supports patents: | ZAR eemurb - Tmprovieg eeformanee | : Technologies * Speculative tbecution « Kelty 242, 2nd Flam, i wen pan eG ot ) & Amesrpct, es 2 57d 6789 | Kecukon time Sensitive eG running terns ip darecs glee Tanning torKe, ant launches 1K A, backup. ayes ne i 2p iGnotmor, — eqycvalent wen yee? Rewacr ; ther gwn wros. for isolatton, + pouis Fein ty expensive & yobs op a Jum 4 xelaHvely starting are SOR. yo thee Costs when Jobs — rowe atiny Short Os aut we se. pofermance by conftgent ng 5 mproves 3 coo|| oes BT ’ | a i \ ia mrap- Se © map side orm © Reduce Side yor By Mr. GOPAL KRISHIA =) dore pa the map phase and done mop - gide join % fa memory worth map-side” soins are proplimns on Slave nodu cormenen* emory exceptions fornr becouse yoin operation . the most our of ™ zorn b one 1 EMTS cor wejartvely Smaller iper bw Source, +0 the cluskr. «Repl pur we vepiicord — dota gee Into a local Mh enn join relatively © rarger Pnput Source aeite each (etal posh tebe oe enepper do mappur~ sid ein | aagduqnteae an whe dor ghould be sorka order . side JOE: av ul I 2 3 gore EE gyal 02 © , 7 tooRedeice ~ Side goin ¢- Reduce ste going & ng dara fren ‘ sik join & weennique for mergi diffrent — Scarce a Sf i 7 reer bard en a Specrtic Hey. The : re are no enerory — veshicttons+ roo etd i OF sorting Lovervideden i not he ‘J ere 7 os ohn loos, yall, wera 28 Horie sorted, 2 Bat fae ME 00 ON ce 0s aaa penooehetagr 618" eno . © Aepipaled Aone += pubtbard cache ah? bubs application tft qeod ony fils ef erently- ee putibated cache is a faci 2p, ‘ity prowtded Reduce fameeork 80 Gaene files (ie ee 60 09) need by applasfons. ome ce opy the necessary: files to the SI e. the job are execentect a The framework wi tl mods before any reams for thor ode ae effiereney stems fam the } coptdals, Qize Fev JOD ard the ability to fila are only 4 conten, ware’ arvarchived on the Slaves Gore act vey . : - ik os a “rudimentary glow dishtburen ee ee be uted 2 HSA “sap andlor ‘rtduce FeINs + TE Can be we oe yor anh mattve Upraricy and my ® classpath of native Ubrarg a 1 . porn er @ c xO ~ Pp ga the Abinto «0h be pet on the she reapedsice rauns: the daihrbured cache % designed to dBpibute auererapes, vanging hom a fees veh fo ro-og medium sire yo few re Ob MBS c7 7 \ — © One drawback of Ye Gurren frplementaron of the dishtbuted cache i thar there & 1 Umy to cpecrty "Ap ov reduce Specifte averfacts. Counters Counters are the Useful Channel for gethe sing srorgics about he JOb- OR application level Starsticy i auiee to copings +- By Mir. GOPAL KRISHNA \ Hadeop maintains seme built tn Coumiers for every ¢ gob , which weport various —-mehtes for cur qob. expected amount Of INPOT ConScrened Be ~ expected amount Of CDT POT patedutced. ° ers» Some ~ putlt fn Counters | @-mrap topet RECO mummer of tmput records Conwumed cy by al whe mops 1 the JOR: Toccmented ewery time a recork & yeod «form Tnpubspltt (thrcugh Record Reader) i before passing to mopl) method “Of — mapper. @ wag cour par Recoxchs : - number of Cubpertl records jauced BY all the MOPS fm the JOB. Focremenies 21 P a catlecrtd method “% Called gn context every ovjeer, ‘Uke wise 5 fo Reduce Enpur Record @ Reduce ourpub record. viene* Counts are maintained by the tak cohth when rey are amotiakda , ard perfodicaily Sent to the tay fratne, ast then 40 ObTIACKer So Hhey all Can be globally aggregated: JOb counkis = ave actually maintained £0 they do net weed to be Set the all other Counters including o soThe puil-in by the yobnatKer , acrass the Mw unlike the urey dedtred NG + oy Copopression - MapReducd amy. 2: a pero, wItTD oe compression Zo MepReduce :- ‘ee a weduees nO1Oh byte) Written tof ra Compression vod fiom DFS: “L yCornpression effectively improves the ef ficlency Ge pew tarctwiath ond dian space qh) Gaver the amount of dora vding ponfoved map nodes to Repoce modes: pereneeo 2 tz bo compre sston|oecompressign brary: compression Fowerak rhLZ0 yadeop compre SsforsCadec? pressten +20 - Jeopcodic Caren Fadioop: Co ache fenplementatfon OF A compression — j | we: “4 da cowpression algostthen* Tp Hadenp , a “codec” ix represented | ant mentation Of THE “compre sston codec" Pier fae. | f gene using compression :~ | | @ Reduce Storage reqysirernent | @ sped up dato transfers (acres ty MYO from dixws) |Lzo Key crmracka Stig + ~ AY Sawn” | Leg a very fost da Compression fonal z bafky during the compre ssfon uk deperdt 69 thE Compre sston | (she A ano ow 4 tevel) i ees, mor requires the additt oral buffer during the than the Swrce an destination. decompresson 4 possible Reqpires On ode decomp ssien other e nafterses thas wby Fost cofth wep b20° piles the User ie adgust the be e bekocen ‘ Compression yatfon and — Compresston Speed, cwtthoert agfeotng ME speed: Of decompre ston “ nedcep exes Hae eer compression ' codacs sgh www o> com » hadanp « compre ggton + Default Codee «s oe “ ee oe com + hadeop « compression k20 Codec. yey cem.+ hadoop compression «Snappy se. wee ro Lz0 compre Sion — I roy 1 1 configuretens Raunt fer ropssd 2SRE ROL uy rot enabled 77 the | py default compre SSioN poapReduce Cire volue 4 false). TO achieve the Compression cue noe tO edit « the .. belenox property of 1 Zname> cenpred . cutpats compress cfprorers >yore 7? enable the Compresston, value should be , true" eehich Compression codec to be eed while Cormpressi ng Gy default "pefauttcodee " ewill be Used - Inordey qob cetput - other than default (Lz0 oF Sapp y) ,ue rave to replace | +o use whee corresponding — certecsy © ire Selo; . ' emame > mapred + CUuIpet + Compre Ssion «Codec EZ valued o1g- apache + hadeos fo. compress DefauttCoceo vole. 1: fn place & Ddelauttcodec , give LeD00dC | Srappal mete ” specify which compression codec to be the ap cEtputs: ‘tan abo while, compre sing «eee sea’ vughi ‘value compression] decompression -Gbeary. tr ° oim for mat rnw® = CompreSSion , or = Compataoiléty nor dots atner compression Ubyary- sien e Sroppy airy for vag hgh speeds and recuormble e sei preSHOD* ti Kolly a sehnologios Annan oon . Ameorpet, eerie | % ; oe uTnput Splits o~ yRedur tg orapRedusce Jobs Gan pracess the enttye tn phere will mor be an Pope tn y Concept ch povaielie, siegle Shot sth ‘whe sore veowgn = hadeop divider the input to enpRednce yous gnro number Of fixed Size ehu erwewn ad NPE SPIE @O SPH: ve | | ww | wte) | | | Kelly Technologies Flat No. 212, 290 ‘Floor, *¢ + Splild oa t~ size i cnapreds spt - cain~ Size “Annapurma Black, Asitya Enclava, * Ameaerpet, Sycorabad- 1-500 O16~ Phe o4e-sanz £7288, 89, 908 570 BTR tly ggpteal comiat of Mapper qypical Colpo Reclucer pot rane whe Fopur OF eu msitad , ft divider the chunts which eK o shad. always + Pe equal to of greaitr Won ewourd be pcnsfte Should be conept mapReduce AchievesLife splitstee less than blowsize oe wt : . 7 have Smailersize splits and thereafter, muitiple SO a mappes will Be create on each and every split which wn rauitanr = Oto A POOF performance . ropReduce yoo 4 a UTE Of work which elienr. is expective. TRS rrapfeduce OPS (ap be driven by Koo dormons Une 5 co-ordinate the tak & @ yootrackes [ewhicn, will the tox] pac Soneauilteg & Reschede ling ® rosnracter [ewhion B exactly reipomuible for execution of the task on Ene 4] rout oat quiet feat 22 TRAY Ret @ Tot, Forres © qutput foreas- oat? Lopat Forcast rs cee, reading dosa~ oe spect ficotioo for . pefing, naw 40 preok the fps . provides RecordReader objec ro wead the fily. Reod dato. ond convents to ckey, volue> gaia. x & passtoe yo write Gtom Tnpuk Foremls, pile dope Fo te tu base Implementation class for ail the mals + ’ pu geplementar?oo for at the classes - . eue ! y Texr TOper Forman i defoutt forerat Syn Byte OFF St each and every Key values x rine trated a value. volue > were“4 c Se mene! Soe te re dafosth Be formar of renprederce hich cot be A when WHE tacamtng dai 5 ro 8. & an the for of * Texr"'. | if cath and every une of the code ty the record | ord each & every mecord we Sept cod pee ; are eg | \ \ character ‘V+ © Tear Tnpat HU format, Genny Mr. G ney 9 Byte offset volves By Mr. GOPAL KRISHBA Value The entile texte Of He vecord K | hadoop 4g a bigdata tool) | \ pigdata u tevteg (ot ot) demand ) w Fee, poet ea eae, a { eget Tem pon’ see * prs bien (Gn te configured) Lune of texts Tne feed o% Corsfage ;uted +0 focare end of Une Key 2 bo lovitableé - osttton tn th 7 "3 P efile of HX. } | | | | wolue <9 Texr — UneKey~rvalue Text 1 p © Keyralue “Text Pt Srnats~ cach Une OS key-value Ceeb aulimited ) « Key value Tex Tpet Former with be ated ty erp Recluce programming —wheneveT une ate getting — the impur dora ta rhe town o CK, Vv): » By defouth, Chi) TexrInpar Format gi! Ar! a te . default datiovter ( Heuwever we Gn change the Game In cea) / eR ue hove the specie Key in Ins Imputfor he Byroffser valuer will not get be generoted. v nw fie hodacp _ \t_. -biglofa- Bigdaia. _\b_ _Aredlytis - Aradnota -- Nt -- moctleApps “igre. Bag emak.se 3 wakee Beet eats 3 [same 03 Fee Inpur formar pak sples equal +0 confeuel @ ny- erie - . wt OF 6 G go Foren ‘ + Recon Reader 4 a Class ehicn UY psieently retponstole O Jor re (Kiy) ps given 39 the Lapa Spits » rehe |. whenever we aye rove: 5 deta awh arto x fanctons a Dia Trutiple Sptis | tate mmappt 1 TF we howe — votTable ang. of recone, and each and every spt then we coh). mer awe prope: : e 7 | control on OFM exactly a particular seize ewtt | be | completa woth the Same reoton _ | veeordy tach & every Spier 2] _Ninelnpur Forma + *$ ce want to pura fixed -na-of then ue Gn go head with Split ~ nilness configured via mapred: line. input -formar or nine Topurformat. setNumlinePerSpiit (Job, 10d) gtegh Une of kext By Mr. GOPAL KRisHnas Recoed > { Sey 3 Lenghetratle = posttfon fo the Fite vont value 3 Texr—Une of texte wes wees ie : ets @ ase Somer fue ae nt probe 0. formar Mantumable +o re . converts dara ia Map Reduce + papper muse accept — proper wey Jwabuer . | oplit > Rows tn on Howe Region(provided scan may marrow dao in the esutk) yearned = Columns are Conholled ty a Record > Reno, iid Kans -prov Rey > Tmreutable yalue 3 Resute (1H Bove clos) Byks — wr?tablebivary represe otatfon + Special type of file to Store Key- value pairs. | store Rap ond vols 8 foyte | arrays we length entodid ytd at format often used os ?npur oF autput fowrnat -for mR Obs. Compression values + wadeop specific os wate 9 gpectfication for eoriting data - ewey value > pais are coritten 19-FO + Sequence File Durpat Forerat - binary sepresentatfon . Hodeop specific volt does curpur specification for that job: yo Oy your een annoying enessages wat cortpur atrecto"y arecady exists crtors genplermertatt on & — Record water. | Rapesoe FO actually wnting Bata Creates, Implementation of qurpat — Commnfiter- qosn's artifacr 1 Seep and Clan - Op qoo's ord (ex: atrectorie) * CormentE oF aicard = terns curpat[Ore cupet fewat ers plain tect « outp » caves Rey-value pS Seporaied by tab «configured vie rapredu ce + Cutpub- “etait Forme pavain | Oper | os = Perty Z | qecrourprforar - seraatparpain( yop , “reenpath(mypatn)) erat 1 “ | @ robe cen Sect: sient a] ento wTable tay | pur Key 4 ignore soe Boer yodue mul be AALS fear or delete ‘ irae . Reduce :~ ve “egaunaron {2 TOPOS cagBy Hit. GOPAL KRISHRCS wun be stored 19 yoy been complekea by TasmTracner, gtoved fy the local fe Siem op a aeason mIGh® PE eens (pat rot 0” Boke qosnnacnes por oF were a pes Reducer phose- neo Gro OF Hekucen to Some have to the mapper O/p , We whith Ua me oveshean + process due .-In i tf ue loss ob = mapper performance comm 09, per ofp eotll be Stored tn called pata Local carfon. tu only for the, mapper but | pole; pate locotkatfon weducer , The TeOATH mofgrk be the lp we gee HOM redetcer ts ushabeveySembiogy i~ TO optiettze the ew & varcleotath Umitarions comby. 0 Tey comept will be ued So mapRedueg Prograsming, mr Comtiney woh) ACF OF leat Yetercer (ory mind Redes comatever te don centumed by Redueey Copy wil) Fes¥de in the combi ney. Ker os PROSE | the Samy Hodaspt “dee not provide any guaranite execution...” Hoceop may COU csmbiney function zero . =, 2 ONE oF Honey far 2. pusatcuton map eesteere records ech noose ty TO 2 -3 se a. mee: et s while ttting reduce methad , Hadecp doo, Reems ang gurantte oo values in stor to a paxtculan Key, tO achfeve TRE we Cet Hap sorting: The combiner function deer not Feplate the function. 27 Combines ig Combiner 1 Specify. the. Combine, functon paxeitioners i= I ¢ nigpescttevoney allows you 4a Abiytbule baw CCU. Fron mien , storage aye Sent “tO the reducers, Brricaily gy | partiyoyy the — Reyspace- { we , poxtelgney combroly the Paxttioniny — the Keys Of dhe rmverensode wap-CPtS: The ey (eubser of the nay a , nya to devive the partition ¢ tre etal wb pein Sie OF he woof ene eg] l+ pai toner tun on the - Same machine after mapper fad complied it's execution , by consumt eappe entive emmppe cutpur(secord) 4 gent eee forers x (mo-0f ‘wedluce 705%) a ourpurs o ap: 3° ani pax Horer fron “ree MOPRY 1, py defautt radeop — frarmeworh Heth based pasutFoney » This poreiculos paxtitiors The Keyspace Py wing. the | hashcade + fotlaving y tegic Hoshpaxditioney exeCurey to a veducer for a paoetculos, rey” mum Reduce Tarns- jhe derev mine : (yey values Comet comity) threy aon Te-€ccaption, © 1) Wray To-Geception, _Talenaptel gecoption ae i hs ant Sum OF. Sree Segoe” for (ene voritane ols vobuey AS Deon ealrgeens 4200" sam+ 2 WO gel, 7 & ae oS SS, 5 3 oom %e ne gets seule ser (Sum) 2 Or ori £ a & a : OPAL KRiy h reasuit):es Kris in (ating £1 oxgh) Weer" Excaption f = OU Configyy fone): rato es | Com “yoo foo = me Joo (cont » Voom count"): as? jo cor TavbyCioss (WEaeres Neco sels): . est wort y+ ass) cok Sy ob. eee Mapperclass( Token lect 4 (on cee Combiner Class( tok Sue Reducer Ziass); 9555 , & Cn ig a OE eet "grea! oto clay (Text: clo) ca IR spo: Op Wey Texte Clem) —? pb Nabue Class (dotartsate «10897 Filer roe Peo path (angetod))s gslaourperforeret* get QuiparPatn( jo , NEw path (mpstD) (qeb-eense er Completion Crue) 2 CEDY Gagptern C2 « ai cuast class. & durived Fearn OE 3 4 we ue F coe 8 contiquratin > ve con Cro’ pre JOR Compuilsony conf “ will be Created +How to create nN ow ent program fp lnerecary (ou myecllipse Tbe Steph i= File > New —» youd projecr > cling of Bese step 2. word unr f seers Regt ctor > new > pacnage 7 Ga =| 3 class 4 eee Word Count gre cone aoqubk cick woe ——— Tre Ho Re Sysreen LIOMY _ au sepure Hof creare odd jor fly pore Count gre 5 ausid pa grr gure wow es congo exrtyoh ve ators vy oo aero orsC SPS +. ow to efor Sie yonfie to Unur. ord count gre Eg Gacanro0y” L@ TRE sypiem Leoreay . | SHEP Ue padcoP ‘| | | Class ram Zeyh neett ULE, wrth filer SHOPS utp path FS ort coat JO exordcoant — aviirha/Fopal TH artihafourpet Fl antiajowpur eh radeop 8 - OF awnttta faupstTreaties) Aoemy | MapRadvee | pata Size Ga - TBs TPs - PBs Access Interactive & Besteh Batch . updates Read fevrite mony time lovite onte , Yead many Structure Storic Ssenerna Dywamic pis . * Xhemna Foregri ty high Loew . “ Scaling non Uncor Linon MopReduce optimi tations :~ OP overage ph StShoFEL or) Sot ay . a rae az 7aleglee a Sapper locality ‘Ameetrsy ivat AF oe close #0 the dara. meen Tin ed STs ‘ a Schedule — MOpPers » Cornbi 7eY eras mas generic du ples np , can yun en Mapper node. «gran offer free Yeduicer , dove sire before trarsser ag ea reeeritd «~" / 4 Reducer, SH" a ue : a nat ‘ | G Speceslon ve exedision” fo netp with toad. balancing if ¥ a Some edt > rs toe", | Slowe “anothe other dlupttcale tess. gerting towad the end yy moe, tone fish ANsivey i | | }qransfore% Shauctured ObjeCK Mio A byk Sth cam. . angenissic s | por waryenission over the sneleaor- Hodleop USA QE for pevsisent storage on diss Serialfzation format, writable crucial CSnoffu and Sort prox); eshicn + Hadcop Cer ths wn © comparison of SPE u tau a custo Raw Comparatos , yedaop prov! By deserializahon i ovis fox raving fal contol of the | a GBtOM™. - cori Fone ntaron of dora ay ai — neprese c | Spe enernet Frareemorns are Alowedt enter Aveo. ce . th oF vortable -Lengin « cos cee xd 10S le tengin enconing ? peed engi | cahen we aunibutron of values Y uniform , yardqol engin: when phe Abshibuton of volug ror Uniformo uLe ppache Pig 4 ore. OF the. Components £ * 7 a oF =Had rc | & an obStack Jayer (high “vel procedural language) orm y top oF Hadeop and empreduce platoon } Pig wal Known as para few langage (012 Trankfireration | } larpage:' F ig yrivially — fatroduced axe youneo labovotavig at” t ny gsr ne jmovdey FO addyess thelr adn request: v . } eg vorguage 8 offic og opted by mpacne 79" Me 1 yer 008 trheve " Conan Kp, RR 10 pig j langnge called» Apache Pe rg language safer alton b the language tO Pe’ pe ance? os Pighotio + : wrth =mulliple rasuformation ' and through the tomforraHont. Wwe [ pera flow wot be . ' aa yO peanrformation language (oD Dataflow Lean cu ed 1 language: ig 3 a simple language ror corommnly exteCuty Staterenks. © pl iz by, A opegAtlior fae n tree TORE Popa - a grorerner no epee arn teboes on (sun ies cert endnote bog 8 e's utp Ma sr gpts. anorher corse oensitive - UDF nam OC ‘ ot ig torn me woot cpu Sl ~ cone, genstty ue ; ST * 2 CP = apig Vo oes wwe ef ws a procedural (now) @ SAL wy declarative. © ea ¥ © nese aeiarional data rae] © FI velarfonal date ; ® sone x3 optional © serene & required @ son annie One we @ ot oLAA workloads. wor Loos . . Lee query opumization © significant oppor tently . gov weed optimization odyartoas S ose BOs Syne, level languoge - Borah ance pregnant! produrcet Fisting oo A effort 5 AR CIES dupt cor prope ape 20 : ae rod0OP complexity. fo rare USC a Sequence oF crewed of creak : - od weno crakrg a prgmam in Pig 4 gimplr ro Ue tra the pee wu eosier tO Keep ok whee ger ore FN Khe« Debugging ervironrenr + Nested data mode} waar pig so + pig uo plarform for analy xing Jorge dota sets consists of a Nighalevel_— language fos expressing dota analipis © prageans. « pig generars ard compila a eoap)Reduce ~ programs ON, cm quer ally. | why Pig cose of “programing t= re wu. tyivial 4 achieve goriaitel execution. of Simple, “embrassingly’ pamatlel ” arolys tOaKA- . fy € dora. = compe cons Comprised of multiple intesye tteq wary-forations one expidty encoded of dota flow i oto anes aay 20 HTIE, urderstand and fh, Seapeee » ranteg oa rookerscain- “ 7 oO Pig taKes Gre Of ees c Pig aw 4 « ecrema ond . ture checing. ‘ 7 “pra vs OG rato efficent physical Dataficen. . cee sapere oF one of more MapRedaice oes) re : : 6 eeplore's dora = yeduaion epporter ores - G + (a conly poaiiat agargorion Via 0 cotter) uv L© bxecuhng Hc awe Cie ronning the mapRecuee yobs) racking Prog re8S 4» ertors , OC Pig Aachi tecture (srapReduice Bacher) Pig Rac awe oe eeegot fo be Suonrited Create a Job to “hadcop clusis parteg optotiaion of tepi@d Ree . Ea! jections 2 o vly — proje . @, Shearing Lead and Stove. weit © pushing Logica! plad *- . Directed ercyctic Graph + logical opeectors 4 a + gpicfien 3 629. a + Logica! operators: . ore per PHS Lape Geckieg with Scheer. ceecooe eo So ead ey Files bi Filler a by $095: pinto t nysudts'? Store ro physica! eva) corresponder® earth mos Cr0sS > Dariggt » Groep, @ CO-GroP, ey wee lei wo PIE nee fee grr operator 5 a-gic) iP tu cooverted {ute ads eo fr fo aE awh logical operators ard Orda: » except = sco Qearrarge C+ ore T Guotet Rearrorae(OD eden’ 7 geay Lusros ound} _ Rackage( PKA) Oe £085" @ OBDs.,0.23" — gp A ty Acoll. @ vy BCL 2.08" 4 : nen By Ke pt . AMPS, : _ ®) fs G9) 242 a 0 col 3 ia) pel -D _ Ameer, Ph gyPet+ prysitel to MapRduce (mR) plan conversion happens tmreaigh TE MAR Compiler: — comerts a prysicol PI openatons« Jan mto A DAG of miR © Boundary for jh frclude cogroup|arcup , divine j crossy ordy by, Umit Cin Somme (oie). subsequent operators beteween Co.group, co exe yto reduce - ented A = 2 MIR obs ~ a pun all cogroop odes ogy Temple : qoocento! compis chen) a the mR 91a to cornupuce a oncennel objeck + map, youeel (=) ey - en), ey Submittea to Hedeop Job Gonhe!l- weap, reduces + jeejar 8 created and CHL reCUre 4 ~ Back fA BERRLazy Executor i= Re eee useful wealy enecules undies reqpest — aelepue. » pothing 2 only wohen He grove, dump, explain, duscrioe ,. TMashaie commana 4 encountered dor te mlR ae piace atthe, oo Yor clusks or front end. advantages :- Ine emor"g pipelining priter ope ordering across multiple commands. pone tS? spit whe porentelisn on rmap- $7 operesows: By default | reducer / nd « prarrel Key ° cross FN? asin + order a group 1 COOP? Rung 2OL of execution Sap pg t- ey col node ar “ © Lo ‘iy, © repesuce ence “Ye . x a ed : oo. node t— oO ver pg. 1 Jocatmode , you Pe! Pe machine « Ay Fs OE qngtatled Bogue cess vo orale Mustog GOUT jecourest & 5} Ton we local mode gy Oe = Ce L Local ost od jowr file Syste? prototyping & debugging - forgor DEE ee iw the efawt (nap vedic) erode , ond HOFS gestallal mode * of mapreduce mode oot tn rmapreduce, pig WIN excpeck the pepur fe prs Ot procucee, the curpur from 29 top oF HDFS: gynrets- pig (Cv pig mapred ua cop clue and HOFS: ware a oonamiage OF OHO Ruretes £9 to © towractive mode (arnt shed) / \ (script most) qu enede ( emted 7 sratemenk to, yua) 8 ”m ane class porn qareractive boded sre /) D Grane Srv :- Grant Spell Ban enackly est get EE “output wonctber ar Success ©D fot : reguicr 058, «cop ip) | recat oF OF 18 0 prg- phere eontion + , eet “pet “Tt Se and hon Son 9Hfe Sexipe SORE I= Ge 2, ‘ Script made all the | Co [wit cranformattors] wil be tne tn ne : : tingle fle Called PD file» ti comple pig d £ peo tN excentive iM + PIG file , the le cot € pe executed “gn a Single ShOF Aa we ean ee ane focal garople PIF . sample pig a 5 © 16 embeded node .- - “1, ig certain grin oe divectly possible, gprougn the pronsformas : oho rovision OF Rvigig wer defined functions ceig ce [ue ON ee aces aes eoilt Cong, the Some © UDF Code, with om 7 ey cer agoert PE Stitag .i0 emp ered OF 9g ececutoms with i the oy pigeetPr coe | wT cou | hE jouamode pig UF Tn corporate eras oh igetioed Sewestor 2 — poles a ve mae Seo ho REarsTER > Regist* nlpacoa the Pg Tame = pert ve 3 crear an a coc upr , Streaming gener (™ specifica en . IMPORT > eenport — macros defined , ro, Sepoucue Se pago STP Reswo ty Pe: gic value. @ scolor type contain o SS orain Other type. e comptes type Cc © seat Wee S wee wre emallesr © Unjt in apaore p¥g- © asym arom? Y sth @, Single characler- yet ton denote with 6, $y a conics toe a - e every jenaratier & a oro + erame =, esa oor a @sr (000 \ooauple :- collecHon of all these fields @v ordered = Ser OF fields. Sipe © Complers type a . 7 sper con &% arfonal model. emprame SS oe exp : Of atory- i vested 744 Bo otning but collectfon esau : PONS awe eet = “Ata (oO ‘ (loxers), rBrceeit { (eo) | age! 22 cometen wtih MY leowop~~ stores relations Using Ficid ~“delimitd texr + Loads of foxenat 1 . gacn Une BrOReD into fields Uatn a Gonfyyratle Sierd — delient tev (defautts to @ tan Craracky) tO be Stored ty THE juple's fieida Er 8 the defauir Storage when one is = Specitfred. from os tO) bivnay file. leads @O Stores relarfons an fakernal pig formar 4 ued that Wer “hadeop cori toble objecs* xb bavily meskd dott. . Le StoTB8 jeading’ tnievinediole FAUUS tak were » wed for previously stored UIT tke Tex Loodey +— a loads relorfort” plain - kext fJoreax + Each to a tuple eohose SrBIe Held “8 tine corsespos ‘ . ; i ne OF HOE 2= “pig , No need *O tq ard Hadeop versions mut where Acs AN - # Pg Ssubm* Tom uW> discover of documeols Vanweaie * Mendalty ser boeta(ox es): % Frallad O71 qastall anything on the 84 art = execuks PY ant ier machine - tHodeop clusite. be Compatible. . joes 1 the Hacleop cluster. wnowldge fern lotge collectiony (actdemic papers) dis covertg * prop tages erouy Kea " SOn4, Pending Trending exploring unshuctured daia Sk from 1 ont dara feeds. as, doranse dumps Real wer: computa gientsticy on kane views , CllcHs' and user be howfour 0° tanger aeesthy after click. . atl, proesssing F » poke processing for cen Sco _ reas. - pane queries 4 eels dara Seg . a for processing large dota ses. + fapia protypt nay a en “866, re teasing - etl «machine © Tenr prcessi” Coan, Seon sracured) + para + processing eae muta sareeeREEEUSS expeck the language SsHills Business legic. D wapReduce programing for writog the. we (an do ang change mR program, we problems the process nw Mm we need wo certatn we Con je anti: © compra the progam © exaouting Wr pogrom © meets up we" ® sepleging che Cobre envit Sere on eh), ay pareductaivt ts woe: eventhough oF can ae grarerne 0 comet ood ~s @© pig “there: & m0 much of program SKiIwys BS we are wniting conote” rogic wii maxing He of pig, 7ANSforentioy (oo operators: . ue can with Simple avord they @® m we pia, = . 5 og te MR eeut’on Hie g ue can tat “YP @® Io px ef rR prOgrA™ > goods Jo ent of Cedex Iolernally» c Pig VT. ia seater ot peseviption Bas ® looa —5 rad data from the fle System @ store 5 wntte dora 10 the file Sysiem @ vempP 5 write quiper fe Stauth @ Foreach 5 poppy expression ro can ve an, * generate One or crore records: ‘lke . | @ fue => Apply predivate to each record art ver wecords ehere fale: @® Group }cogeoP _» colleer vedowts with the Same Key fom ore more inputs @ sans tao or crore. Top BRHad on ey 6 order 5 Sort records youd on @ key: @ puro > Remove, dusplicare yecordy. | uo 3 merge: dasases . is veente "Oe wecords s ay @ se ® cross if Diag nostic pia gece grovement @ peseribe @ - bump see a 3 9% achat o> 6 Explain cena . * 5798 Oy > Duplays a Step Myf Step exceurion ® Tiberete lok 0 Seqperze OF Staremen!s-piiter , Foren , Group , O92 4 soro(inn) 7pm (coter) quit -in teres 12 28 @ tape , 09, Map Specthie fuuntriony @ eval @ srg @ mars @ core Tree exalupsten( ROL) Teneo 20g 1 @ Ayo t ne the average .yalue Of enbieg in © coucar :- . COMA * concareraiy too OYte arrays or two chmacies arrays together . : colculorts rhe ap.of ennia i a vag N @® coowy t- @ count STB colculayes the ~ 07 0F Clempnis mm @ bag ® OIE t- compares 40 fields ino tuple. db fa bag ov crap & empty 3 lebepig 2- Pe @ marin ‘computa we pax furcon 10 ~ Tourer’ Vouer oF raraNTOG reoxtimum of | the too Stagle - clue 09° 3) magix oxy fanction to compe the minimum ga xt of mumertc voluts OF erar-ovvays iN A single column ) giet- coment eet elemen wared o> O79 Pig data > SQM i- Comput: he gum of A Sek of “numeric yous qn a Single - column 50g, JOKENEZE :~ TORENTZE furciion 40 SPU Oo ening GF words bag of anrda(@cr apd @ py ToReMast 2 Co words 19 O strgle +uple) into e 9as, A LA on ne ter the onset volue Of an expression. @ 9es:- jlong , float ox double. NOK y- expressiom Rerarrs the arc costne of an expression. whose, wvesuit w% type dable. mesule Y @® acos?- expression poos (express REND. @ asty:- Rerures the are Sint of O97 expression - Retarey the Te tangent of an expression @® arani- ® CERT: Reiurrs the - Cube rot of On expression: ® CeEbi- Returns te wolue of an expression rourded up to the revert Inkeger: © CeSi- Reaens the wigonorehst Cage of an expression. hyper bolic cosiee hag expression . \ ttm Eulers rember a raaksed AO pocues OF X- @ Cosp:- Returns the Rau of an expression vroundad oN: express folate ® Loditor- Reterrns the ume og san expression - : Se Flay ® Ranson :- Rear o pseud fe ease @ RQoump :- — Reeurnt the value of” essfoo wounded 10 an wm Rga- Rerarns the Sint of an expressionSQuare VOTH OF aN capo, b @ 2ei- Retarvu, the positive tangent of an angk @ raw: Qatarns the trignemearic tangent of an expression. TANS Repusry the hyper boc nq Commands += bg Comin, 7 ‘ 7 . A ‘ BD stove ‘1 eoitie sda t0 PE fie system. ( te, co 9) foreach :- display specttic column. : Be foreach A generate ids : ¢: ‘ cr ay whe recoss —Oscersirg crda I L order. -duph Y (\\ | \3 _& tod 2 . >® * wrecorcls decending order disgloy the > voese s-¢ D rye TYR pe cehtre folie. ce firey A by t= = Gopal’ eo none = =e 2) JO Yerte :- diploy ame yecorcu from begining . e “170 vuticcr:- Rewoving at dlupticae valuey © Grorp dg [ootrop ly -~ —Greupyhe dare mm stigle Relarfor ARS PARR we coud yh, Gocup By- G@map the dota tm tuo (vo more relatong & Court rhe CoGreup ey. es ig mame cz ‘Antena! ® yon 5- joins 400 or more xelasfony.© pigieouien BED em gore BaP ENE Right: Res yelartom: ® peseaibe points weloHon's Schema: x pesert Oe Ay @ Tyyseas pugiap Ste ~ PUT EP emcution Ff & CU miacnee “°F cecrement ats the logical and ‘prytical plans. ga & i © epg k @ unten im Combine HO ecore, vUaON = Into ort.UH) Ure ieee ARRAS TOLe Shee, Bx foreach oe va as A generte — TOKEMEECecord): Aggregate -funcktony ;- Canna soate the. aggregate ecae computers make He = gectp © coomr,. calcutare the corde A by jd; Cc 2gmp B by gE Dx foreach c generate TOOF entries 2a a bag. greep , Coonte gs): Cobcutote the sum of a Set of maumeric vethecey im @ Single -Column bag. @® max: cotcutar the roan Pe the merrmerfe valuer er ‘ cotemn pag F< foreach c generate grep _MAKCB-Sat);) 4 Compute te minimum of @ Set Of Mumeric voluiy of char -Orroyg en @ Single Colum [wach P » MEA (8: Sad)= 1 R22, ON " AMG ee Goleadates a a Bee tse OF entieg , in a bag. En ° Be Jereach C gerevote = group "AVG C@- age)Characiy «= arrays Together |x 2 foreach “]@ generate grep, cancer (8-727) _» user oefined Fancefons. +o define Gutom processing. © corttten pig tO Support © Gusto processi1"Q- : 3) Support for coiting and Ruby still evolving: provide on opportanity to exted pig foto appl cation uoFs fn python, yada Scrip ® v0rs goor “ domain - atte ua, UDF 3- pew 22 SEE Te ee ea.” the JWR > py thon 5 JAwacceppy | -¢eceosye Cc 7 ( ty nport, orgs opacne” pigs ten UHI cwrapped Tokcception ; pte cass opreR peters qalfurc

You might also like

• The Subtle Art of Not Giving a F*ck: A Counterintuitive Approach to Living a Good Life

  

  From Everand

  The Subtle Art of Not Giving a F*ck: A Counterintuitive Approach to Living a Good Life

  Mark Manson

  Rating: 4 out of 5 stars

  4/5 (5820)
• The Gifts of Imperfection: Let Go of Who You Think You're Supposed to Be and Embrace Who You Are

  

  From Everand

  The Gifts of Imperfection: Let Go of Who You Think You're Supposed to Be and Embrace Who You Are

  Brene Brown

  Rating: 4 out of 5 stars

  4/5 (1093)
• Never Split the Difference: Negotiating As If Your Life Depended On It

  

  From Everand

  Never Split the Difference: Negotiating As If Your Life Depended On It

  Chris Voss

  Rating: 4.5 out of 5 stars

  4.5/5 (845)
• • Magazines
  • Podcasts
  • Sheet music
• Principles: Life and Work

  

  From Everand

  Principles: Life and Work

  Ray Dalio

  Rating: 4 out of 5 stars

  4/5 (609)
• The Glass Castle: A Memoir

  

  From Everand

  The Glass Castle: A Memoir

  Jeannette Walls

  Rating: 4.5 out of 5 stars

  4.5/5 (1717)
• Grit: The Power of Passion and Perseverance

  

  From Everand

  Grit: The Power of Passion and Perseverance

  Angela Duckworth

  Rating: 4 out of 5 stars

  4/5 (590)
• Sing, Unburied, Sing: A Novel

  

  From Everand

  Sing, Unburied, Sing: A Novel

  Jesmyn Ward

  Rating: 4 out of 5 stars

  4/5 (1104)
• Hidden Figures: The American Dream and the Untold Story of the Black Women Mathematicians Who Helped Win the Space Race

  

  From Everand

  Hidden Figures: The American Dream and the Untold Story of the Black Women Mathematicians Who Helped Win the Space Race

  Margot Lee Shetterly

  Rating: 4 out of 5 stars

  4/5 (898)
• Shoe Dog: A Memoir by the Creator of Nike

  

  From Everand

  Shoe Dog: A Memoir by the Creator of Nike

  Phil Knight

  Rating: 4.5 out of 5 stars

  4.5/5 (540)
• The Perks of Being a Wallflower

  

  From Everand

  The Perks of Being a Wallflower

  Stephen Chbosky

  Rating: 4.5 out of 5 stars

  4.5/5 (2104)
• The Hard Thing About Hard Things: Building a Business When There Are No Easy Answers

  

  From Everand

  The Hard Thing About Hard Things: Building a Business When There Are No Easy Answers

  Ben Horowitz

  Rating: 4.5 out of 5 stars

  4.5/5 (349)
• Bad Feminist: Essays

  

  From Everand

  Bad Feminist: Essays

  Roxane Gay

  Rating: 4 out of 5 stars

  4/5 (1018)
• Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future

  

  From Everand

  Elon Musk: Tesla, SpaceX, and the Quest for a Fantastic Future

  Ashlee Vance

  Rating: 4.5 out of 5 stars

  4.5/5 (474)
• Her Body and Other Parties: Stories

  

  From Everand

  Her Body and Other Parties: Stories

  Carmen Maria Machado

  Rating: 4 out of 5 stars

  4/5 (822)
• The Outsider: A Novel

  

  From Everand

  The Outsider: A Novel

  Stephen King

  Rating: 4 out of 5 stars

  4/5 (1867)
• The Sympathizer: A Novel (Pulitzer Prize for Fiction)

  

  From Everand

  The Sympathizer: A Novel (Pulitzer Prize for Fiction)

  Viet Thanh Nguyen

  Rating: 4.5 out of 5 stars

  4.5/5 (122)
• The Emperor of All Maladies: A Biography of Cancer

  

  From Everand

  The Emperor of All Maladies: A Biography of Cancer

  Siddhartha Mukherjee

  Rating: 4.5 out of 5 stars

  4.5/5 (271)
• Angela's Ashes: A Memoir

  

  From Everand

  Angela's Ashes: A Memoir

  Frank McCourt

  Rating: 4.5 out of 5 stars

  4.5/5 (441)
• Brooklyn: A Novel

  

  From Everand

  Brooklyn: A Novel

  Colm Tóibín

  Rating: 3.5 out of 5 stars

  3.5/5 (1947)
• The Little Book of Hygge: Danish Secrets to Happy Living

  

  From Everand

  The Little Book of Hygge: Danish Secrets to Happy Living

  Meik Wiking

  Rating: 3.5 out of 5 stars

  3.5/5 (401)
• A Man Called Ove: A Novel

  

  From Everand

  A Man Called Ove: A Novel

  Fredrik Backman

  Rating: 4.5 out of 5 stars

  4.5/5 (4771)
• The World Is Flat 3.0: A Brief History of the Twenty-first Century

  

  From Everand

  The World Is Flat 3.0: A Brief History of the Twenty-first Century

  Thomas L. Friedman

  Rating: 3.5 out of 5 stars

  3.5/5 (2259)
• Steve Jobs

  

  From Everand

  Steve Jobs

  Walter Isaacson

  Rating: 4.5 out of 5 stars

  4.5/5 (808)
• The Yellow House: A Memoir (2019 National Book Award Winner)

  

  From Everand

  The Yellow House: A Memoir (2019 National Book Award Winner)

  Sarah M. Broom

  Rating: 4 out of 5 stars

  4/5 (98)
• Devil in the Grove: Thurgood Marshall, the Groveland Boys, and the Dawn of a New America

  

  From Everand

  Devil in the Grove: Thurgood Marshall, the Groveland Boys, and the Dawn of a New America

  Gilbert King

  Rating: 4.5 out of 5 stars

  4.5/5 (266)
• A Tree Grows in Brooklyn

  

  From Everand

  A Tree Grows in Brooklyn

  Betty Smith

  Rating: 4.5 out of 5 stars

  4.5/5 (1929)
• The Art of Racing in the Rain: A Novel

  

  From Everand

  The Art of Racing in the Rain: A Novel

  Garth Stein

  Rating: 4 out of 5 stars

  4/5 (4208)
• A Heartbreaking Work Of Staggering Genius: A Memoir Based on a True Story

  

  From Everand

  A Heartbreaking Work Of Staggering Genius: A Memoir Based on a True Story

  Dave Eggers

  Rating: 3.5 out of 5 stars

  3.5/5 (231)
• Yes Please

  

  From Everand

  Yes Please

  Amy Poehler

  Rating: 4 out of 5 stars

  4/5 (1903)
• Team of Rivals: The Political Genius of Abraham Lincoln

  

  From Everand

  Team of Rivals: The Political Genius of Abraham Lincoln

  Doris Kearns Goodwin

  Rating: 4.5 out of 5 stars

  4.5/5 (234)
• The Woman in Cabin 10

  

  From Everand

  The Woman in Cabin 10

  Ruth Ware

  Rating: 3.5 out of 5 stars

  3.5/5 (2522)
• Fear: Trump in the White House

  

  From Everand

  Fear: Trump in the White House

  Bob Woodward

  Rating: 3.5 out of 5 stars

  3.5/5 (738)
• Wolf Hall: A Novel

  

  From Everand

  Wolf Hall: A Novel

  Hilary Mantel

  Rating: 4 out of 5 stars

  4/5 (3811)
• John Adams

  

  From Everand

  John Adams

  David McCullough

  Rating: 4.5 out of 5 stars

  4.5/5 (2409)
• ITIL 4 Foundation

  

  Document2 pages

  ITIL 4 Foundation

  Sureshkumar Singampallik

  No ratings yet
• On Fire: The (Burning) Case for a Green New Deal

  

  From Everand

  On Fire: The (Burning) Case for a Green New Deal

  Naomi Klein

  Rating: 4 out of 5 stars

  4/5 (74)
• The Light Between Oceans: A Novel

  

  From Everand

  The Light Between Oceans: A Novel

  M.L. Stedman

  Rating: 4.5 out of 5 stars

  4.5/5 (789)
• Manhattan Beach: A Novel

  

  From Everand

  Manhattan Beach: A Novel

  Jennifer Egan

  Rating: 3.5 out of 5 stars

  3.5/5 (792)
• The Constant Gardener: A Novel

  

  From Everand

  The Constant Gardener: A Novel

  John le Carré

  Rating: 3.5 out of 5 stars

  3.5/5 (104)
• Splunk Presentation

  

  Document30 pages

  Splunk Presentation

  Sureshkumar Singampallik

  No ratings yet
• The Unwinding: An Inner History of the New America

  

  From Everand

  The Unwinding: An Inner History of the New America

  George Packer

  Rating: 4 out of 5 stars

  4/5 (45)
• Rise of ISIS: A Threat We Can't Ignore

  

  From Everand

  Rise of ISIS: A Threat We Can't Ignore

  Jay Sekulow

  Rating: 3.5 out of 5 stars

  3.5/5 (137)
• Little Women

  

  From Everand

  Little Women

  Louisa May Alcott

  Rating: 4 out of 5 stars

  4/5 (104)
• CEH Notes

  

  Document3 pages

  CEH Notes

  Sureshkumar Singampallik

  No ratings yet
• API Connect On Docker PDF

  

  Document19 pages

  API Connect On Docker PDF

  Sureshkumar Singampallik

  No ratings yet
• Yum Commands

  

  Document2 pages

  Yum Commands

  baraka08

  No ratings yet
• The Challenges Tomcat Faces in High Throughput Production Systems

  

  Document48 pages

  The Challenges Tomcat Faces in High Throughput Production Systems

  Sureshkumar Singampallik

  No ratings yet
• Microsoft Pass4sure AZ-103 v2020-04-24 by Liming 132q

  

  Document143 pages

  Microsoft Pass4sure AZ-103 v2020-04-24 by Liming 132q

  Sureshkumar Singampallik

  No ratings yet
• Nectar Monitoring

  

  Document35 pages

  Nectar Monitoring

  Sureshkumar Singampallik

  No ratings yet
• Skype For Business CloudBond 365

  

  Document3 pages

  Skype For Business CloudBond 365

  Sureshkumar Singampallik

  No ratings yet
• Microsoft Lync Server 2013 Step by Step For Anyone - REV014 PDF

  

  Document258 pages

  Microsoft Lync Server 2013 Step by Step For Anyone - REV014 PDF

  Sureshkumar Singampallik

  No ratings yet
• Failed Extensions

  

  Document1 page

  Failed Extensions

  Sureshkumar Singampallik

  No ratings yet
• Avaya Voice Portal: Configuration Note 3915

  

  Document20 pages

  Avaya Voice Portal: Configuration Note 3915

  Sureshkumar Singampallik

  No ratings yet
• Dateoptions - Aspx:: Here Is Some Sample Code..Try This... I Am Using This Type of Procedure

  

  Document5 pages

  Dateoptions - Aspx:: Here Is Some Sample Code..Try This... I Am Using This Type of Procedure

  Sureshkumar Singampallik

  No ratings yet
• HOWTO Compil Asterisk With H323

  

  Document3 pages

  HOWTO Compil Asterisk With H323

  Sureshkumar Singampallik

  No ratings yet