Uma introdução simples e intuitiva sobre o novo ramo de TI.Descrição completa
lồnDescripción completa
Data ScienceFull description
Descripción completa
big data
additional infoFull description
Descripción: Big Data
lồn
BIG DATADeskripsi lengkap
Big DataDescripción completa
Big Data is the new Buzz work connecting the new trends of data analytics. Data management has shifted its focus from an important competency to a critical differentiator
big data
articulo big data nuevo decarga descargaDescripción completa
Oracle Big data
Generación de talento big data
Descripción: Plan Big Data
Notes for Big Data
Full description
Mineria de Datos y Big DataDescripción completa
Big DataFull description
Big Data
BIG CPU, BIG DATA Solving the World’s Toughest Computational Problems ith Parallel Computing
Alan !amins"#
BIG CPU, BIG DATA Solving the World’s Toughest Computational Problems ith Parallel Computing
Alan !amins"# Department of Computer Science B. Thomas Golisano College of Computing and Information Sciences Rochester Institute of Technology
"he book BIG CPU, BIG DATA: Solving the World’s Toughest Computational Prolems !ith Parallel Computing is li#ensed $nder the Creative Commons Attrib$tion!onCommer#ial!o%erivs &.0 'nported (i#ense. "o vie) aaletter #opy to o* this li#ense+ visit http,--#reative#ommons.org-li#enses-by!n#!nd-&.0or send Creative Commons+ Castro Street+ S$ite /00+ o$ntain ie)+ Cali*ornia+ /01+ 'SA. "o red$#e #osts+ the hard#opy version is printed in bla#k and )hite. or a *$ll! #olor e!version+ see http,--))).#s.rit.ed$-3ark-b#bd-.
ront #over image, "he IB Bl$e ;ene-4 s$per#omp$ter installation at the Argonne (eadership Angela ?ang Comp$ting a#ility lo#ated in the Argonne ational (aboratory+ in (emont+ Illinois+ 'SA. Co$rtesy o* Argonne ational (aboratory. http,--#ommons.)ikimedia.org-)iki-ile,IBDBl$eD;eneD4Ds$per#omp$ter.pg
Alan Kaminsky 4ro*essor %epartment o* Comp$ter S#ien#e B. "homas ;olisano College o* Comp$ting and In*ormation S#ien#es @o#hester Instit$te o* "e#hnology arkE#s.rit.ed$ http,--))).#s.rit.ed$-3ark-
A$g$st 2015 edition
Pre$a%e
W
ith the book BIG CPU, BIG DATA, my goal is to tea#h yo$ ho) to )rite parallel programs that take *$ll advantage o* the vast pro#essing po)er o* modern m$lti#ore #omp$ters+ #omp$te #l$sters+ and graphi#s pro! #essing $nit 6;4'9 a##elerators. "he book is *ree+ Creative Commons li! #ensed+ and is available *rom my )eb site 6http,--))).#s.rit.ed$-3ark-b#bd-9. IFm not going to tea#h yo$ parallel programming $sing pop$lar parallel libraries like 4I+ >pen4+ and >penC(. 6I* yo$Fre interested in learning those+ plenty o* other books are available.9
•
I pre*er to program in ava. "he a*orementioned libraries do not+ and in my belie* never )ill+ s$pport ava. In my eHperien#e+ tea#hing and learning parallel programming )ith the a*orementioned libraries is more di**i#$lt than )ith ava.
Instead+ IFm going to $se my Parallel "ava # $irar% 6429 in this book. 42 is *ree+ ;' ;4( li#ensed so*t)are available *rom my )eb site 6http,-))).#s.rit.ed$-3ark-p2.shtml9. ?o$ #an do)nload the #omplete so$r#e *iles+ #ompiled #lass *iles+ and avado# do#$mentation. 42 re$ires ava %evelop! ment Kit 6%K9 1.J or higher. Installation instr$#tions are in#l$ded in the avado#. 42 is s$itable bothprogram *or tea#hing and learning mming and *or real!)orld parallel development. I $separallel 42 andprogra its prede#essor+ the 4arallel ava (ibrary 649+ in my #ryptography resear#h. >thers have $sed 4 to do page rank #al#$lations+ o#ean e#osystem modeling+ salmon pop$la! tion modeling and analysis+ medi#ation s#hed$ling *or patients in long term #are *a#ilities+ three!dimensional #ompleH!val$ed *ast o$rier trans*orms *or ele#troni# str$#t$re analysis and !ray #rystallography+ and onte Carlo sim$lation o* ele#tri#ity and gas markets. 4 )as also in#orporated into the IL open so$r#e ava image pro#essing appli#ation. I am happy to ans)er general $estions abo$t 42+ re#eive b$g reports+ and entertain re$ests *or additional *eat$res. 4lease #onta#t me by email at arkE#s.rit.ed$. I regret that I am $nable to provide te#hni#al s$pport+ spe#i*i# installation instr$#tions *or yo$r system+ or advi#e abo$t #on*ig$ring yo$r parallel #omp$ter hard)are. ore *$ndamental than the lang$age or library+ ho)ever+ are parallel pro!
iv
BIG CPU, BIG DATA
gramming #on#epts and patterns+ s$#h as )ork sharing parallel loops+ parallel red$#tion+ and #omm$ni#ation and #oordination. pen4Fs #ompiler dire#tives+ 4IFs message passing s$bro$tines+ or 42Fs ava #lasses+ the #on#epts and patterns are the same. >nly the syntaH di**ers. >n#e yo$Fve learned parallel programming in ava )ith 42+ yo$Fll be able to ap! ply the same #on#epts and patterns in C+ ortran+ or other lang$ages )ith >pen4+ 4I+ or other libraries. "o st$dy parallel programming )ith this book+ yo$Fll need the *ollo)ing prere$isite kno)ledge, ava programming C programming 6*or ;4' pro! grams9 #omp$ter organiMation #on#epts 6C4'+ memory+ #a#he+ and so on9 operating system #on#epts 6threads+ thread syn#hroniMation9. y pedagogi#al style is to tea#h by eHample. A##ordingly+ this book #on! sists o* a series o* #omplete parallel program eHamples that ill$strate vario$s aspe#ts o* parallel programming. "he eHample programsF so$r#e #ode is listed on the right!hand pages+ and eHplanatory narrative is on the le*t!hand pages. "he eHample so$r#e #ode is also in#l$ded in the 42 do)nload. "o )rite programs )ell+ yo$ m$st *irst learn to read programs so please avoid the temptation to gloss over the so$r#e #ode listings+ and #are*$lly st$dy both the so$r#e #ode and the eHplanations. Also st$dy the 42 avado# do#$mentation *or the vario$s #lasses $sed in the eHample programs. "he avado# in#l$des #omprehensive des#riptions o* ea#h #lass and method. Spa#e does not permit des#ribing all the #lasses in de ! tail in this book read the avado# *or *$rther in*ormation. "he book #onsists o* these parts, • • • • •
4art I #overs introd$#tory #on#epts. 4art II #overs parallel programming *or m$lti#ore #omp$ters. 4art III #overs parallel programming *or #omp$te #l$sters. 4art I #overs parallel programming on ;4's. 4art #overs big data parallel programming $sing map!red$#e.
Instr$#tors, "here are no 4o)er4oint slides to go )ith this book. Slide sho)s have their pla#e+ b$t the #lassroom is not it. othing is g$aranteed to p$t st$dents to sleep *aster than a 4o)er4oint le#t$re. An ar#hive #ontaining all the bookFs ill$strations in 4; *ormat is available *rom the bookFs )eb site please *eel *ree to $se these to develop yo$r o)n instr$#tional materials.
Alan Kaminsky A$g$st 2015
Table o$ Contents
4re*a#e ................................................................................................ iii Part I. Preliminaries Chapter 1. "he 4arallel (ands#ape .................................................. 1:1 Part II. Tightly Coupled Multicore Chapter 2. 4arallel (oops ................................................................ 2:1 Chapter &. 4arallel (oop S#hed$les ................................................. &:1 Chapter . 4arallel @ed$#tion .......................................................... :1 Chapter 5. @ed$#tion ariables ....................................................... 5:1 Chapter N. (oad Balan#ing .............................................................. N:1 Chapter J. >verlapping ................................................................... J:1 Chapter O. Se$ential %ependen#ies ............................................... O:1 Chapter /. Strong S#aling ................................................................ /:1 Chapter 10. $tp$t on a Cl$ster ............................................. Chapter 21. Intera#ting "asks ........................................................ Chapter 22. Cl$ster =e$risti# Sear#h ............................................. Chapter 2&. Cl$ster
1:1 15:1 1N:1 1J:1 1O:1 1/:1 20:1 21:1 22:1 2&:1
vi
BIG CPU, BIG DATA
Part IV. GPU Acceleration Chapter 2. ;4' assively 4arallel ............................................. Chapter 25. ;4' 4arallel @ed$#tion ............................................. Chapter 2N. $lti!;4' 4rogramming ........................................... Chapter 2J. ;4' Se$ential %ependen#ies ................................... Chapter 2O. >be#ts on the ;4' ....................................................
2:1 25:1 2N:1 2J:1 2O:1
Part V. Big Data Chapter 2/. Basi# ap!@ed$#e ..................................................... 2/:1 Chapter &0. Cl$ster ap!@ed$#e ................................................... &0:1 Chapter &1. Big %ata Analysis ....................................................... &1:1
PART I PRELIMINARIES
Titan, the U.S.A.’s fastest supercomputer Photograph courtesy Oak Ridge National a!oratory
Chapter 1 The Parallel Landscape ▼ Part I. Preliminaries Chapter 1. The Parallel Landscape ► Part II. Tightly Coupled Multicore ► Part III. Loosely Coupled Cluster ► Part IV. GPU Acceleration ► Part V. Map!educe
1–2
BIG CPU, BIG DATA
P
arallel computing is concerned with designing computer programs having two characteristics: they run on multiple processors, or cores, and all the cores cooperate with each other to solve a single problem. Both characteristics are necessary for a program to be considered a parallel program. A web server, lie the one that runs !oogle"s web site, typically runs on a multicore server machine, or even a group of several multicore server machines, so as to process hundreds or thousands of web page re#uests each second simultaneously with high throughput. $he web server thus displays the first characteristic of parallel computing. %owever, the web server"s cores are woring on different problems, namely the different user"s web page re#uests. $he web server"s cores are not cooperating with each other to solve the same problem. &o the web server does not display the second characteristic of parallel computing. ' would call a web server an e(ample of distributed computing, not parallel computing. 't"s a subtle distinction, but a crucial one. 'n this boo ' am going to be discussing parallel computing, not distributed computing. !oogle does, however, do other computations that are e(amples of parallel computing)namely, the big data analytics that convert the raw web pages swept up by !oogle"s web crawlers into the #uery inde(es that let !oogle speedily retrieve web pages that best match users" searches. %ere, !oogle uses multiple cores cooperating to solv e the single problem of converting enormous #uantities of web page te(t into enormous #uery inde(es. 'n fact, !oogle was the leader in populari*ing the map-reduce paradigm of parallel computing on massive data sets. !oogle, and many other ma+or 'nternet sites, also use map-reduce parallel computing to convert enormous #uantities of information gathered from us users)the #ueries we do, the web pages we visit, the contents of our !mail emails, aceboo posts, and $witter tweets) into advertisements targeted specifically at each of us. •
•
•
ther modern-day applications that use parallel computing include: omputational mathematics: numerical linear algebra, numerical solution of ordinary and partial differential e#uations, optimi*ation, combinatorics, graph theory, . . . &cientific computation: weather prediction, hurricane forecasting, climate modeling, astrophysics /gala(y and star cluster simulation, blac hole gravitational wave prediction0, chemistry /molecular dynamics, ab initio molecular structure prediction0, bioinformatics /A se#uence matching, protein se#uence matching, protein structure prediction, pharmaceutical drug design0, geology /seismic data analysis, oil and mineral prospecting0, . . . 3ngineering computation: computational fluid dynamics, simulated wind tunnels, finite element analysis, circuit verification, integrated circuit chip component and wiring placement, . . .
Chapter . The Parallel !andscape •
•
•
•
•
•
1–4
omputational finance: asset pricing, derivative pricing, maret modeling, algorithmic trading, . . . Big data analytics /as already mentioned0: data mining, web inde(ing, user characteri*ation, targeted advertising, . . . &ecurity and cryptography: password cracing, cipher attacs, bitcoin mining and transaction processing, . . . 3ntertainment: computer games, computer generated imagery /!'0 for visual effects, computer animated films, . . . ringe: 5ersenne prime searching, &earch for 3(traterrestrial 'ntelligence /&3$'0, computer chess, . . . And the list goes on.
't wasn6t always this way. 'n the beginning, up through the 1789s, parallel computers were esoteric and e(pensive. 3ach parallel computer vendor had its own proprietary hardware architectures, its own proprietary parallel programming languages /often nonstandard e(tensions to e(isting languages lie ortran and isp0, and its own proprietary parallel software libraries. or the most part, the only ones with budgets big enough to afford parallel computers were large industrial organi*ations and government funded research laboratories. onse#uently, parallel computing was used mostly for scientific and engineering applications. A paradigm shift in parallel computing too place near the end of the twentieth century. &etting the stage was the development during the 1789s and early 1779s of ine(pensive ; hardware, cheap 3thern et local area networ hardware, standard $;<'; networ protocols, and the free inu( operating system. A 177= paper> brought all these together and described how to build ?Beowulf,@ an e(ample of what is now called a cluster parallel computer, from standard, off-the-shelf hardware and software, for a mere fraction of cost computing of a proprietary publication ushered in an era of the parallel done supercomputer. with commodity$his chips and hardware rather than proprietary machines. 'n addition, parallel programming shifted to using standard languages lie ortran, , and later with standard parallel program ming libraries. 5essage ;assing 'nterface /5;'0, introduced in 177, became the de facto standard for parallel programming on cluster parallel computers. pen5;, introduced in 177C, became the de facto standard for parallel programming on multicore parallel computers. %owever, at that time multicore machines were still not commonD if you wanted lots of cores, you usually had to build a cluster of single-core ma> $. &terling, . Becer, . &avarese, E. orband, F. Ganawae, and . ;acer. Beowulf: A parallel worstation for scientific computation.Proceedings of the "#th International Conference on Parallel Processing $ICPP %%&',177=, volume 1, pages 11–1.
1–
BIG CPU, BIG DATA
chines. onse#uently, parallel computing was still concentrated in industrial and government research settings and was still dominated by scientific and engineering applications. A second paradigm shift in parallel computing too place in 299. Fntil then, processor chip manufacturers had e(ploited 5oore"s aw to steadily increase both the number of transistors on a chip and the chip speed, doubling the cloc fre#uency about every two years. But by 299, cloc fre#uencies had gotten fast enough)around 4 !%*)that any further increases would have caused the chips to melt from the heat they generated /unless outfitted with cooling systems lie 'ndianapolis =99 race cars0. &o while the manufacturers continued to increase the number of transistors per chip, they no longer increased the cloc fre#uencies. 'nstead, they started putting multiple processor cores on the chip. A chip with two cores operating at 4 !%* is, theoretically, e#uivalent to a chip with one core operating at H !%*. $he number of cores per chip continued to increase, until in 291= as ' am writing, e(er)one*s computer is a multicore parallel machine with two, four, eight, or more cores. At the same time, memory chip densities continued to increase, until now everyone"s computer has !B, 8 !B, 1H !B, or more of main memory. urthermore, with the rise in popularity of computer games, both on destop ;s and on gaming consoles, everyone"s computer has a graphics processing unit $GPU' with do*ens or hundreds of cores. Ihile srcinally intended for real-time 4- video rendering, !;Fs can do general-purpose calculations as well. $oday"s ; is a powerful parallel computer with capabilities undreamed of in the 1789s, with computing power e#uivalent to a 1779s-era cluster re#uiring a whole rac of e#uipment. $he modern era of ubi#uitous multicore machin es opened parallel com puting to a much broader range of applications than +ust scientific and engineering computation, as mentioned previously. 5any of these newer applications eschew ortran and and 5;'applications and pen5;, which arose during the era when scientific and engineering dominated parallel computing. 5odern applications also use newer languages lie Eava and newer programming paradigms lie map-reduce. A prime e(ample is Apache"s %adoop, a map-reduce library written in Eava, designed for parallel computation on massive data sets. $o use your modern computer)your modern parallel computer)to its full potential, it"s not enough to write plain old programs. Jou have to write parallel programs. $hese parallel programs have to e(ploit all the cores in the machine. 'f possible, these parallel programs also ought to utili*e all the cores in the !;F. $his boo, BIG CPU, BIG DATA, is intended to teach you how to write parallel programs for multiple-core, !;F-e#uipped parallel computers. espite the vast increase in the ;"s computing power, there still are computational problems too big for a single node to handle. $he problems
Chapter . The Parallel !andscape
1–=
re#uire more memory than can fit in a single node, or the problems re#uire so much computation that it would tae too long for them to finish when run on a single node /even a multicore node0, or both. &uch problems need to run on a cluster parallel computer, one with multiple nodes. An individual user could afford to set up a small-scale cluster with perhaps two or three or four ;-class nodes. A small company or academic department could afford to set up a medium-scale cluster with perhaps one or two do*en nodes. A large-scale cluster with hundreds or thousands of nodes re#uires the budgetary resources of a large company or a government. $he Fnited &tates government, for e(ample, funds a number of supercomputer centers at its national laboratories. $hese supercomputers are available to researchers who have government funding. $he most powerful supercomputer in the F.&., according to the Eune 291= $op=99 ist of the world"s fastest computers /www.top=99.org0, is the $itan computer at the a Gidge ational aboratory in $ennessee. $itan has 4=,99 nodes, each with 1H ;F cores and 2,H88 !;F cores, for a total of =H9,H9 ;F cores and 7,18C,=29 !;F cores. $itan is able to e(ecute the inpac linear algebra parallel processing benchmar at a rate of 1C.H petaflops /1C.HK19 1= floating point operations per second0. Jet even $itan is only number two on the Eune 291= $op=99 ist. irst place goes to the $ianhe-2 computer at the ational Fniversity of efense $echnology in hangsha, hina. $ianhe-2 has 1H,999 nodes, each with 2 ;F cores and 1C1 accelerator cores, for a total of 4,129,999 cores. $ianhe-2 e(ecutes inpac at 44.7 petaflops, nearly twice as fast as $itan. By the way, there"s nothing special about supercomputer hardware. &upercomputers nowadays use the same commodity chips as destop ;s. $ianhe-2 uses 'ntel Leon 3= ;F chips /2.2 !%* cloc0 and 'ntel Leon ;hi manycore accelerators. $itan uses A5 pteron ;F chips /2.2 !%* cloc0 along with vidia $eslapower M29(by !;F accelerators. &upercomputers getcores, their massive computational having large numbers of nodes and not from superior chips or enormous cloc fre#uencies. Ihat about the poor individual or department who needs large-scale computing power but doesn"t have large-scale fundingN Cloud computing is an interesting and viable alternative. A cloud computing service, such as Ama*on"s 32 /aws.ama*on.com
1–H
BIG CPU, BIG DATA
't"s important to now how to write parallel programs for cluster parallel computers, to solve problems that are +ust too large for a single node. BIG CPU, BIG DATA is intended to teach you how to write parallel programs for multiple-node cluster parallel computers, including clusters in the cloud. $o mae sense of the bewilderingly diverse landscape that is modern parallel computing, '"m going to characteri*e parallel computing hardware, software, and applications along several dimensions. igure 1.1 shows the eight dimensions. $he ne(t few sections discuss each dimension. Any particular parallel computer, parallel program, or parallel application can be pinpointed along each dimension, illuminating its place in the overall scheme of things.
Hardware Dimensions A node refers to an independent computer with its own ;F chip or chips, its own main memory, and its own networ interface. A parallel computer can consist of a single node, or of multiple nodes. A multinode parallel computer is called a cluster. A core refers to the hardware on a node that e(ecutes a stream of machine instructions. A node can have a single core, or multiple cores. 3ach core can also be hyperthreaded, meaning that the core can e(ecute more than one stream of machine instructions.
'n addition to the ;F, a node can have *ero or more accelerators. $hese are separate processors that can perform computations alongside the ;F. $here are several inds: A graphics processing unit $GPU' accelerator typically has a large number of cores that are mainly intende d for graphics rendering but that can do general calculations. A man)core accelerator is similar to a general purpose ;F, e(cept it typically has more cores than a ;F, and it omits all the peripheral interface circuitry /dis, networ, display, F&B, and so on0 not needed for doing pure calculation. A field programmable gate arra) $+PGA' accelerator is a digital chip whose logic circuitry can be reconfigured during operation, letting you create customi*ed high-speed processors. An application specific integrated circuit $AIC' accelerator uses speciali*ed chips hardwired to perform specific computations at very high speed. ne e(ample is bitcoin mining /bitcoin.org0. Jou can buy A&'s •
•
•
•
Chapter . The Parallel !andscape
1–C
Figure 1.1. imensions of the parallel landscape
that compute the &%A-2=H cryptographic hash function, part of the bitcoin protocol, at very high speed. %aving e(amined each hardware dimension by itself, let"s loo at e(amples of parallel computers and see where they fall along the dimensions.
Single-Core Computer
5ost computers bac in the twentieth century were single node single core unaccelerated /igure 1.20. 'n such a computer, the core has an instruction unit that reads the program"s machine instructions from memory, decodes them, and e(ecutes themD a number of functional units, such as adders, shifters, multipliers, and so on that carry out the machine instructionsD and a number of high-speed registers that hold intermediate results. Iith one instruction unit, the core can e(ecute only one stream of instructions)one thread)at a time. $o e(ecute more than one thread, the computer"s operating
1–8
BIG CPU, BIG DATA
system must do a contet sitch from one thread to another every so often. $he computer has a main memor) that stores program instructions and data. Because the main memory"s circuitry is much slower than the core"s circuitry, a le(el " $!"' cache sits between the core and the main memory. $he 2 cache is faster than the main memory, but smaller /typically a few megabytes0. Ihen the core reads a memory location into the instruction unit or into a register, an entire cache line, typically H or 128 bytes, is fetched from main memory and stored in the 2 cacheD from there, the re#uested data goes into the core. 'f the core reads an ad+acent memory location, the data is already in the 2 cache /a cache hit0 and can be read #uicly without waiting for the slow main memory. &till, the 2 cache"s circuitry is not as fast as the core"s circuitryD so another cache, the le(el $!' cache, sits between the core and the 2 cache. $he 1 cache is nearly as fast as the core but is smaller than the 2 cache. n its way to the core, data read from main memory is cached in 2 and in 1. 'f the instructions and data the program is accessing can fit entirely in 2, or better still 1, the core can run at nearly full speed. therwise, the core will have to pause while new data is fetched from the slow main memory, reducing the processing speed. &o as to eep the core running as fast as possible, the 1 and 2 caches are made as large as possible while still fitting on the ;F chip along with the rest of the core circuitry. ften, the ma+ority of a ;F chip"s area is devoted to the caches.
Multicore Computer
$o increase the processing power while not increasing the cloc fre#uency, computers switched from single core to multicore /igure 1.40. ow there are two or more cores, each with its own instruction unit, functional units, registers, and 1 cache. $he cores all share the 2 cache and the main memory. $he operating system can run multiple threads simultaneously, one in each core, without needing to conte(t switch. $he threads are running truly in parallel. $heoretically, a /-core computer ought to run / times faster than a single-core computer. /$his does not always happen in practice, though.0
Chapter . The Parallel !andscape
Figure 1.2. &ingle node single core computer
Figure 1.3. &ingle node multicore computer
1–7
1–19
BIG CPU, BIG DATA
Hyperthreaded Computer
Geplicating the instruction unit, functional units, registers, and 1 cache to get multiple cores re#uires a lot of transistors and chip area. $o run more threads without needing #uite so much area, multicore computers became h)perthreaded /igure 1.0. 'n a hyperthreaded core, the instruction units are replicated, but not the functional units, registers, or 1 cache. A dual-hyperthreaded core can run two threads simultaneously without needing to conte(t switch. $here"s a catch, though. As long as the two threads are using different functional units and registers, both threads can run at full speed. But if the threads both need to use the same functional unit or register at the same time, the hardware the threads taethread turns. stalls. Ihile$his one thread accessing the functional unitmaes or register, the other reducesisthe effective rate at which the threads can e(ecute instructions. 'n my e(perience, a dualhyperthreaded core typically does not run as fast as two regular coresD but it does run faster than one regular core.
Multicore Accelerated Computer
ately, parallel computers are including accelerators alongside the multicore hyperthreaded ;Fs. A GPU accelerator /igure 1.=0 repurposes a graphics card to do general calculations. A !;F card has numerous cores, anywhere from do*ens to thousands of them, as well as its own main memory. $he !;F"s main memory is lined with the ;F"s main memory over a high-speed bus, allowing data to be transferred between the ;F and !;F. &tandard !;F programming libraries, such as vidia orporation"s FA and the vendor-independent pen, mae it almost as easy to write !;F programs as it is to write ;F programs. Both the ;F and the !;F have to deal with their main memory"s large latency /access time0 relative to high speed of their cores. $he ;F deals with it by interposing 1 and 2 caches between the cores and the memory, devoting the bul of the chip area to cache, leaving room for only a few cores. $he !;F deals with it by reducing or even eliminating the cache, de -
voting the bul of the chip area to a large number of cores. $he !;F then runs large numbers of threads simultaneously on its cores. Ihen one thread stalls waiting to access main memory, another thread instantly taes its place. $here are enough threads so that whenever a core goes idle, a thread is available that has completed its previous memory access and is ready to run again. 'n this way, the cores stay busy all the time. $his techni#ue is called latenc) hiding. epending on the nature of the problem, a !;F can perform calculations do*ens or hundreds of times faster than a ;F. Iith !;Fs incorporated into +ust about every modern computer, it"s important to now how to write !;F parallel programs, to tae advantage of the !;F"s additional processing power. BIG CPU, BIG DATA is intended to teach you how to write parallel programs for !;F accelerated parallel computers. /'"m not going to cover the more esoteric manycore, ;!A, and A&' accelerators in this boo.0
Single-Core Cluster Computer
$urning away from single-node parallel computers, we come to multinode parallel computers, or clusters /igure 1.H0. $he cluster has some number of bac0end nodes that carry out computations. 3ach node has a single core plus its own main memory. Ie say the cluster has a distributed memor)1 the cluster"s memory is distributed across the nodes instead of concentrated in a single node. As we will see, the distributed memory has profound implications for the design of cluster parallel programs. $he cluster has a dedicated high-speed bac0end netor0 that allows the bacend nodes to communicate with each other. $he bacend networ may use commodity 3thernet hardware, or it may use speciali*ed faster technology such as 'nfiniband, 5yrinet, or &calable oherent 'nterface /&'0, or it may use a proprietary interconnect. $he cluster usually also has a frontend node, connected to the 'nternet to let users log in and run parallel programs on the cluster, and connected to the bacend networ to control and monitor the bacend nodes.
Multicore Cluster Computer
Chapter . The Parallel !andscape
1–14
Figure 1.6. 5ultinode single core cluster computer
%owever, nowadays it"s virtually impossible to buy a single-core node. &o a modern cluster parallel computer consists of multiple multicore bacend nodes /igure 1.C0. $he cores might or might not be hyperthreaded.
Multicore Accelerated Cluster Computer
'n addition, the nodes of a modern cluster parallel computer might include accelerators /igure 1.80. $he $itan supercomputer loos lie igure 1.8D it has 4=,99 nodes, each with 1H ;F cores and a !;F accelerator, and it uses a high-speed proprietary interconnect for its bacend networ.
Sotware Dimensions A parallel program consists of multiple threads performing computations simultaneously. 'n a single-node multicore parallel computer, the threads run on the cores of the node. 'n a cluster parallel computer, the threads run on the cores of all the nodes.
$he computations performed by the threads can be uncoupled, loosel) coupled, or tightl) coupled. 'n an uncoupled computation, the threads do not communicate or coordinate with each other at allD each thread runs and produces its results independently of all the other threads. 'n a loosely coupled computation, the threads communicate with each other, but only infre#uentlyD for e(ample, the threads compute results independently of each other, but at the end of the program the threads communicate their individual results to each other and combine them into one overall result. 'n a tightly coupled computation, the threads communicate with each other fre#uentlyD for e(ample, each thread e(ecutes a loop, and at the end of every loop iteration, the threads communicate the results of that iteration to the other threads before proceeding with the ne(t iteration. oupling also refers to the #uantity of data communicated between the threads. 'n an uncoupled computation, no data is e(changed between the threads. 'n a loosely coupled computation, a small amount of data is e(changed between the threads. 'n a tightly coupled computation, a large amount of data is e(changed between the threads. A particular parallel program might fall anywhere along the spectrum from uncoupled to tightly coupled.
Chapter . The Parallel !andscape
1–1=
'n a shared memor) parallel program, the data items the threads are accessing as they perform their computations)input parameters, intermediate values, output results)are stored in a single memory region that is shared by all the threads. $hus, any thread can get any other thread"s results simply by reading appropriate locations in thedoshared memory. memorythe parallel program, the threads not have access'ntoaanon-shared common shared memory.
'n a distributed memor) parallel program, the data items the threads are accessing are stored in multiple memory regions. 3ach thread can directly read and write locations in one of the memory regions, and the thread stores its own data in that memory region. 3ach thread can also access the contents of the other memory regions, but not by directly reading and writing them. Gather, one thread accesses another thread"s memory region via some form of e(plicit communication. $he communication can tae the form of message passing1 the thread that owns the data sends a message that is received by the thread that needs to use the data. /5;' is a library of message passing subroutines of this sort.0 ther possible communication mechanisms include remote procedure call /G;0, remote method in(oca tion /G5'0, and tuple space. 'n a non-distributed memory parallel program, the threads do not have any access to other threads" memory regions. %aving e(amined each software dimension by itself, let"s loo at e(amples of parallel computing software and see where they fall along all the di mensions.
Multicore Parallel Program
A parallel program intended to run on a single multicore node /including hyperthreaded cores0 typically uses a shared memory model /igure 1.70. $he program runs in a single process with multiple threads, each thread e(ecuting on a different core. $he program"s data is located in the computer"s memory.
1–1H
BIG CPU, BIG DATA
Figure 1.9. &hared memory parallel program running on a multicore node
Because all the threads are part of the same process, and the process consists of a single address space, each thread can access all the program"s dataD this is how the shared memory is achieved. $ypically, the data is partitioned into as many pieces as there are threads. 3ach thread computes and writes its own piece of the data. 3ach thread also reads the other pieces of the data as necessary. $hus, the threads communicate with each other by writing and reading the same shared memory locations. $he threads can also coordinate with each other using synchroni*ation primitives supported by most operating systems, such as semaphores, loc0s, and barriers. memory A parallel be tightly program coupled, still loosely coupled,&hared or uncoupled. looselyprograms coupled can or uncoupled loos lie igure 1.7D the only difference is the fre#uency with which one thread accesses another thread"s data, or the amount of data accessed. &hared memory parallel programs are +ust multithreaded programs, no more and no less. Jou can write a shared memory parallel program using the built-in threading constructs of a programming language or an operating system, such as Eava threads or Fni( pthreads. %owever, fols who need to do parallel computing often are not e(perts in writing threaded code. $hey want to write high-level application programs to solve their computational problemsD they don"t want to have to write low-level code to create threads, ac#uire and release semaphores and locs, destroy threads, and so on. onse#uently, most fols use a parallel programming librar), or application programming interface $API', to write shared memory parallel programs. $he A;' e(poses high-level application-oriented parallel programming constructs
Chapter . The Parallel !andscape
1–1C
to the programmers, and the A;' handles all the low-level threading details under the hood. 2pen3P /www.openmp.org0 is a widely used A;' for shared memory parallel programming. irst released in 177C, and now in its fourth revision /version .9 of the pen5; specification was released in Euly 29140, pen5; supports parallel programming in the ortran, , and languages. ' prefer to program in Eava. &o ' prefer not to use pen5;, which does not)and, in my belief, never will)support Eava. 'nstead, '"m going to use the Parallel 4a(a " !ibrar), which ' have developed, to teach you shared memory parallel programming in Eava.
Cluster Parallel Program
A parallel program intended to run on a cluster of single-core or multicore nodes typically uses a distributed memory model /igure 1.190. $he program runs in multiple processes, one process for each core of each bacend node. 3ach process has one thread running on the core plus data located in the node"s memory. $ypically, the data is partitioned into as many pieces as there are threads. But because the threads and data pieces are in different processes with different address spaces, the memory is not shared. 3ach thread can access its own data directly. But if one thread needs to use a data item lo cated in another thread"s memory region, message passing has to tae place. or e(ample, suppose thread C needs a data item located in thread 4"s memory region. $hread 4 has to retrieve the data item from its memory and load the data into a message of some ind. Because the threads are running on different nodes, thread 4 must send the message over the cluster"s bacend networ to thread C)an inter-node message. $hread C has to receive the message and e(tract the data. r suppose thread H needs a data item located in thread "s memory region. Although the threads are running on the same node, because the threads are running in different processes /different address spaces0, a message still has to go from thread to thread H)an intranode message. $he threads" programs need to be coded to invoe message passing operations e(plicitlyD this increases the comple(ity and programming effort for cluster parallel programs.
1–18
BIG CPU, BIG DATA
Figure 1.10. istributed memory parallel program running on a cluster of multicore nodes
5essage passing can, of course, be more complicated than these simple e(amples. ne owner thread might need to send data items to several recipient threads. ne recipient thread might need to gather data items from several owner threads. 3very thread might need to send data items to and receive data items from every other thread. luster parallel programs can be tightly coupled, loosely coupled, or uncoupled. An uncoupled program still loos lie igure 1.19, e(cept there is no message passing. A loosely coupled program loos lie igure 1.19, but does fewer or less fre#uent message passing operations, or sends less data, than a tightly coupled program. Jou can write a cluster parallel program using the interprocess communication $IPC' constructs of an operating system or using networing soft-
Chapter . The Parallel !andscape
1–17
ware lie $; socets. %owever, fols who need to do parallel computing often are not e(perts in writing '; or networing code. $hey want to write high-level application programs to solve their computational problemsD they don"t want to have to write low-level code to open and close socets, format data into and out of messages using some protocol, and so on. onse#uently, as with shared memory parallel programming, most fols use a parallel programming library or A;' to write cluster parallel programs. $he A;' e(poses high-level application-oriented parallel programming constructs to the programmers, and handles all the low-level networing details under the hood. $o achieve acceptable performance, a tightly coupled cluster parallel program needs to use a fast, low-overhead message passing library. 3essage Passing Interface $3PI' /mpi-forum.org0 is a widely used A;' for cluster parallel programming. irst released in 177, and updated to Oersion 4.9 in &eptember 2912, 5;' supports parallel programming in the ortran, , and languages. 5;' is a library of message passing subroutinesD programs written to call these subroutines can run on any machine that has an 5;' library installed. ften, a platform-specific 5;' implementation is used to wring the fastest possible performance out of the hardware. ;latform-independent 5;' implementations are also available but tend to be slower. Because ' prefer to program in Eava, ' prefer not to use 5;', which does not)and, in my belief, never will)support Eava. Also, '"m not fond of 5;'"s huge and complicated A;'. /'t fills an 8=2-page booP0 $he ;arallel Eava 2 ibrary includes message passing capabilities via a simple A;' called tuple space. %owever, the ;arallel Eava 2 ibrary is intended mainly for uncoupled and loosely coupled cluster parallel programs. Ihile tightly coupled cluster parallel programs can be written with ;arallel Eava 2, the tuple space"s platform independent implementation is not designed to achieve the highest possible speed. 'f you need e(tremely fast message passing, use 5;'.
Multicore Cluster Parallel Program
A tightly coupled parallel program running on a cluster of multicore nodes re#uires fre#uent e(change of copious data both between processes running on the same node and between processes running on different nodes,
1–29
BIG CPU, BIG DATA
Figure 1.11. %ybrid shared
as shown in igure 1.19. But it doesn"t mae sense to do message passing between processes running on the same node. &ending data between processes on a node is typically much slower than accessing the data directly. Ihat does mae sense is to use the shared memory model within each node and the distributed memory model between nodes)a h)brid shared5distributed memor) model /igure 1.110. n each node there is one process with multiple threads, one thread per core, with the threads directly accessing each other"s data in shared memory. Ihen data has to go from one process /node0 to another, message passing is used. By eliminating the bul of the messages needed in the pure distributed memory model /igure 1.190, the program"s performance is improved.
Chapter . The Parallel !andscape
1–21
&ome parallel programs have more than one level of parallelism. A program might perform many separate, uncoupled computations, which can therefore be done in parallel. 3ach of these might itself be a tightly coupled parallel computation. &uch a program is ideally suited to run on a multicore cluster parallel computer using the hybrid shared
!P" Accelerated Parallel Program
All the previous parallel software options involved unaccelerated nodes. igure 1.12 depicts a parallel program that uses a !;F accelerator. $he figure shows the simplest ind of !;F parallel program: running in a single ;F thread, on a single ;F core, with all the parallelism in the !;F. $he red arrows show what ' lie to call the GPU computational arc6 •
$he ;F sets up input data for the computation in the ;F memory. f-
•
ten the data is anthe array or data matri( consisting many, many $he ;F sends input from the ;Fofmemory to theelements. !;F memory. $he ;F launches a large number of GPU threads. 3ach thread will e(ecute a 0ernel function /denoted by ?M@ in igure 1.120. $he whole assemblage of !;F threads e(ecuting ernel functions is called the computational 0ernel, or +ust 0ernel. $he ;F waits for the ernel to finish. 3ach !;F thread e(ecutes the ernel function on a !;F core. ften, each individual data element is computed by its own separate !;F thread. $he computation"s output data is stored bac in the !;F memory. Ihen the ernel finishes, the ;F waes up and sucs the output data
•
from the !;F memory bac to the ;F memory. $he ;F outputs the computation"s results.
•
•
•
1–22
BIG CPU, BIG DATA
Figure 1.12. !;F accelerated parallel program
$he !;F cores achieve their best performance when they all e(ecute the e(act same stream of machine instructions in locstep, each on different data items)what is called single instruction stream multiple data stream $I3D' parallelism. $he !;F cores also achieve their best performance when the data they are accessing is stored in a regular pattern in memory, such as array or matri( elements in contiguous memory locations. A program that has a lot of data-dependent branching, with different instruction se#uences being e(ecuted depending on the data values, or a program that has irregular data access patterns, such as pointer chasing through lined data structures, will not perform well on a !;F. $hus, typically only a portion of a !;F parallel program runs on the actual !;F)namely, the &'5, regular-data-access, computational ernel. $he rest of the parallel program runs on the ;F. A !;F accelerated parallel program might run on more than one ;F core: the computational ernel runs in parallel on the !;F cores, and the non-ernel portion runs in parallel on the ;F cores. $he ;F threads might run in parallel with the !;F threads, rather than waiting for the ernel to finish. 5ultiple ernels might run on the !;F at the same time. And with a
Chapter . The Parallel !andscape
1–24
!;F accelerated cluster, all this could be happening in parallel repeatedly on multiple nodes. $he possibilities for parallelism are endless. vidia orporation pioneered general purpose computing on !;Fs with their proprietary Compute Unified De(ice Architecture $CUDA' and the programming A;' that goes with it. FA supports writing !;F ernel functions in ortran, , and . $he ;F main programs are written in the same languages. 2penC! /www.hronos.org
Application Dimensions ;arallel computing applications are characteri*ed along two orthogonal dimensions: little ;F)big ;F, and little data)big data.
Little CP" Little Data Application
A little ;F little data application wors with only a small amount of data, and does only a few calculations /;F cycles0 with each data item. &till, the calculations are or can be done in parallel. A spreadsheet is an e(ample. ompared to a supercomputer program, a spreadsheet wors with very little data /the cell values0 and does very little computation /the cell formulas0. %owever, the cells can be calculated in parallel, as long as any data dependencies between cells are obeyed.
#ig CP" Little Data Application
A big ;F little data application also wors with only a small amount of data, but it spends a large amount of ;F time doing calculations with that data. oing the calculations in parallel can speed up the application. rypto-
1–2
BIG CPU, BIG DATA
graphic applications are often of this ind. Bitcoin mining is one e(ample. A bitcoin ?bloc@ is a piece of digital cash. 't occupies +ust a few ilobytes of data. But determining the value of a certain field of the bloc)?mining@ the bitcoin, as it is called)re#uires calculating the &%A-2=H cryptographic hash function many, many times. $hese calculations can be, and usually are, performed in parallel. Also, bitcoin miners can mine multiple blocs in parallel.
Little CP" #ig Data Application
A little ;F big data application devotes only a little bit of ;F time to each data item, but wors with an enormous number of data items. onse#uently the application can tae a long time to run, and processing the data in parallel can speed it up. 5ap-reduce, pioneered by !oogle and implemented in the popular Apache %adoop, is a widely used paradigm for parallel bigdata applications. Apache"s ?;owered by %adoop@ web page > shows that many ma+or 'nternet players)Ama*on, eBay, aceboo, %ulu, ined'n, &potify, $witter, Jahoo, and others)use %adoop running on multicore clusters for their big data analytics. !oogle also does big data analytics with their own map-reduce software. $he ;arall el Eava 2 ibrary includes Parallel 4a(a 3ap 7educe $P437', a lightweight map-reduce framewor built on top of the ibrary"s cluster parallel programming capability.
#ig CP" #ig Data Application
inally, a big ;F big data application wors with lots and lots of data and does lots and lots of calculations with each data item. &cientific and engineering calculations on supercomputers are of this ind. As an e(ample of the e(treme scale of these applications, consider the A55;& molecular dynamics program, which simulates the motion of atoms from first principles of physics, running on the Meeneland supercomputer at the a Gidge ational aboratory. Meeneland is a medium-si*e cluster of 129 nodes, with two ;F cores and three !;F accelerators per node. A55;& running a one billion atom benchmar for 199 time steps re#uires about half a terabyte /=K19 11 bytes0 of data and would tae the better part of an hour /2,4=9 seconds0 on one core of Meeneland. Gunning on the entire cluster, the same benchmar taes +ust 1C.C seconds.Q > http:<
Chapter . The Parallel !andscape
1–2=
Points to $emem%er •
•
• •
•
•
•
•
A parallel program runs on multiple cores, with all the cores cooperating with each other to solve a single problem. owadays, improved computing performance comes from parallel programs, not increased ;F cloc speeds. Applications in every area of computing now run on parallel computers. ;arallel computing hardware can be characteri*ed along three dimensions: single node)multinode, single core)multicore)hyperthreaded, and unaccelerated)accelerated. ;arallel computing software can be characteri*ed along three dimensions: uncoupled)loosely coupled)tightly coupled, non-shared memory)shared memory, and non-distributed memory)distributed memory. ;arallel computing applications can be characteri*ed along two dimensions: little ;F)big ;F, and little data)big data. $he modern computer programmer must now how to write multicore, cluster, and !;F accelerated parallel programs. ;arallel programs are written using a library, such as pen5;, 5;', FA, pen, or the ;arallel Eava 2 ibrary.
1–2H
BIG CPU, BIG DATA
PART II TIGHTLY COUPLED MULTICORE
Parallel program running on a multicore node
Chapter 2 Parallel Loops ► Part I. Preliminaries ▼ Part II. Tightly Coupled Multicore Chapter 2. Parallel Loops Chapter 3. Parallel Loop Schedules Chapter 4. Parallel Reduction Chapter . Reduction !aria"les Chapter #. Load $alancing Chapter %. &'erlapping Chapter (. Se)uential *ependencies Chapter +. Strong Scaling Chapter ,-. ea/ Scaling Chapter ,,. 01hausti'e Search Chapter ,2. euristic Search Chapter ,3. Parallel or/ ueues ► Part III. Loosely Coupled Cluster ► Part I!. 5P6 7cceleration ► Part !. Map8Reduce
BIG CPU, BIG DATA
2–2
e begin our study of tightly coupled multicore parallel programming with a simple Parallel Java 2 program to test numbers for primality. Recall that a number is prime if it is divisible only by itself and 1. PrimeSeq !isting 2.1" is a sequential non#parallel" program to test the numbers speci# fied on the command line. $he program illustrates several features that %&ll include in all the parallel programs we study' •
•
•
•
•
•
$he program is implemented as a subclass of class $as( in pac(age edu.rit.p)2"* and the program&s code is in the body of the tas(&s main() method. +nli(e a typical Java program* the main() method is an instance method* not a static method. $he Parallel Java 2 middleware e,pects this. $he main() method is declared to throw any e,ception line -". %f an e,# ception is thrown anywhere in the program* this lets the e,ception propa# gate out of the main() method* which will terminate the program and print an error message with an e,ception stac( trace. % do this because %&m lay and % don&t want to write a handler for every e,ception. /t line 12* the program ma(es sure the proper command line arguments are present0 if not* the program prints an error message and e,its. %f % run the program with no arguments* this reminds me what the arguments should be. $he program e,its by throwing an %llegal/rgument,ception on line 1* which propagates out of the main() method* causing the program to ter# minate. The program must not call System.exit(); doing so interferes with the Parallel Java 2 middleware. $he static coresRequired() method lines 3–4" has been overridden to return the value 1* indicating that this program will use one core. $his is standard for a non#parallel program. %f the coresRequired() method is not overridden* the Parallel Java 2 middleware will assume that the program will use all the cores on the node.
$he loop on lines 13–15 is the heart of the program. $he loop iterates over the command line argument array* converts each number from a String to a long* calls the isPrime() method* and prints the number if isPrime() says it&s prime. $he isPrime() method uses trial division to test whether the number x is prime. $he method tries to divide x by 2 and by every odd num# ber p up through the square root of x. %f any remainder is 6* then p is a factor of x* so x is not pri me. %f none of the rema inders are 6* then x is prime. $here&s no point in trying factors greater than the square root of x* because if there were such a factor* x would have another factor less than the square root of x* and we would have found that other factor already." $rial division is not a very efficient algorithm for primality testing* but it suffices for this e,ample.
t%ro&s 'xception { alidate command line arguments. i (args.lengt% * +) usage(); Test numbers or primalit,. or (int i - ; i * args.lengt%; // i) i (isPrime (0ong.parse0ong (args#i$))) S,stem.out.print (1sn13 args#i$); 4 Test t%e gi"en number or primalit,. pri"ate static boolean isPrime (long x) { i (x 2 -- ) return alse; long p - 5; long psqr - p6p; &%ile (psqr *- x) { i (x p -- ) return alse; p /- 2; psqr - p6p; 4 return true; 4 Print a usage message and exit. pri"ate static "oid usage() { S,stem.err.println (17sage8 ja"a pj2 1 / 1edu.rit.pj2.example.PrimeSeq *number9 ...1); t%ro& ne& :llegalrgument'xception(); 4 Speci, t%at t%is task requires one core. protected static int coresRequired() { return +; 4 4
Listing 2.1. PrimeSeq.)ava
2–7
2–
BIG CPU, BIG DATA
% ran the PrimeSeq program on tardis* a cluster parallel compu ter with ten nodes* each node having four cores at a cloc( speed of 2.8 9:. ;or now %&ll confine myself to running multicore parallel programs on )ust one node of the cluster. !ater we will develop cluster parallel programs and run them on all the cluster nodes." = +2+ +5 ++5 ++> +2+ ?ob + makespan +>@22 msec
=ote that % ran the program by typing the command > ja"a pj2?. pj2 is the actual Java main program0 it is a launcher for Parallel Java 2 programs. pj2 creates an instance of the specified class class edu.rit.p)2.e,ample# .PrimeSeq in this case"* which must be a subclass of class $as(. pj2 then calls the tas(&s main() method* passing in an array of the command line ar# gument strings. % also included the option > debug-makespan? before the tas( class name. $his sets the pj2 program&s debug parameter to include the >ma(espan? de# bug printout. @a(espan is the elapsed wall cloc( time from when the tas( starts running to when the tas( finishes running. Aith that option* the pj2 program measures the ma(espan and prints it as the final line of output. Bou can turn on other debug printouts and set other pj2 parameters as well. Refer to the Javadoc documentation for the pj2 program." $he loop iterations e,ecuted sequentially one after another on a single core. $he running time measurement says that the whole program too( 1-. seconds. ;rom this % infer that each loop iterationCeach e,ecution of the isA Prime() methodCtoo( .- seconds.
:owever* for this program there&s no need to do the loop iterations in se# quence.
Chapter 2 Parallel !oops
2–3
PrimeSmp !isting 2.2" is a parallel version of the primality testing pro# gram. %t starts out the same as program PrimeSeq.
$he parallel for loop begins with parallelBor instead of )ust or. parA is actually a method of class $as(." $hen come the lower and upper bounds of the loop inde,. $he loop goes from lb to ub inclusive* so at line 18 % specified bounds of through args.lengt%A+ to loop over all the command line arguments. $he statement so far creates a parallel loop o&'ect $hen % called the parallel loop&s exec() method* passing in another ob)ect* namely the loop body. $he loop body is an instance of a subclass of class !oop in pac(age edu.rit.p)2"* which % created using Java&s anonymous inner class synta,. % put the code for one loop iteration in the !oop class&s run() allelBor()
method* whose argument is the loop inde, i0 this is the same code as the loop body in the sequential version. $he rest of the program is the same as the se# quential version. % did not override the coresRequired() method in the PrimeSmp pro# gram. $hus* by default* the Parallel Java 2 middleware will assume that the program will use all the cores on the node. Ahen the PrimeSmp program runs* the parallel for loop ob)ect that is created contains a hidden parallel thread team $here is one thread in the team for each core of the machine where the program is e,ecuting. $he loop iterations are partitioned among the team threads* and each team thread calls the loop body&s run() method repeatedly for a different subset of the loop in# de,es* concurrently with the other team threads. %n other words* the #or$ of the parallel for loop is shared among all the threads* rather than being e,e# cuted by a single thread as in the sequential version. Ahen the program runs on a multicore parallel computer* the Java Eirtual @achine and the operating
2–8
BIG CPU, BIG DATA
system schedule each team thread to run on a separate core* resulting in par# allel e,ecution of the loop iterations. /t the end of the parallel for loop* each team thread waits until all the team threads have finished e,ecuting their sub# sets of the loop iterations* then the program proceeds to e,ecute whatever comes after the parallel loop. $his implicit thread synchroniation is called a &arrier % ran the PrimeSmp program on tardis* with the same arguments as the previous e,ample. / tardis node has four cores* so the parallel for loop has four team threads. $he loop&s iterations are divided among the team threads0 thus* each team thread does one iteration. :ere is the result' < ja"a pj2 debug-makespan edu.rit.pj2.example.PrimeSmp = +5 ++5 ++> = +2+ ++5 ++> +5 +2+ ?ob 2 makespan @>C msec
=otice two things about the parallel program. ;irst* while the sequential program finished in 1-. seconds* the parallel program finished in only .seconds. ;rom this % infer that the four loop iterations were indeed performed simultaneously on four cores instead of one after another on a single core. $hat is* the parallel program yielded a speedup over the sequential program. $o be precise* the speedup factor was 1-22 F -65 G 7.-34. !ater we will study more about parallel program performance and the metrics with which we measure performance." Second* while the parallel program did determine correctly that every number on the command line was prime* the parallel program reported the prime numbers in a different order from the sequential program. $his hap# pened because each number was printed by a separate team thread* and there was nothing in the program to force the team threads to print the numbers in any particular order. %n this e,ample program* there&s no need to synchro# nie the threads so as to ma(e them do their printouts in a certain order."
package edu.rit.pj2.example; import edu.rit.pj2.0oop; import edu.rit.pj2.Task; public class PrimeSmp extends Task { !ain program. public "oid main (inal String#$ t%ro&s 'xceptionargs) { alidate command line arguments. i (args.lengt% * +) usage(); Test numbers or primalit,. parallelBor (3 args.lengt% A +) .exec (ne& 0oop() { public "oid run (int i) { i (isPrime (0ong.parse0ong (args#i$))) S,stem.out.print (1sn13 args#i$); 4 4); Test t%e gi"en number or primalit,. pri"ate static boolean isPrime (long x) { i (x 2 -- ) return alse; long p - 5; long psqr - p6p; &%ile (psqr *- x) { i (x p -- ) return alse; p /- 2; psqr - p6p; 4 return true; 4 Print a usage message and exit. pri"ate static "oid usage() { S,stem.err.println (17sage8 ja"a pj2 1 / 1edu.rit.pj2.example.PrimeSmp *number9 ...1); t%ro& ne& :llegalrgument'xception(); 4 4
Listing 2.2. PrimeSmp.)ava
2–5
BIG CPU, BIG DATA
2–4
Under the Hood ;igure 2.1 shows in more detail what happens when the PrimeSmp pro # gram e,ecutes the parallel for loop code* parallelBor (3 args.lengt% D +) .exec (ne& 0oop() { public { "oid run (int i) i (isPrime (0ong.parse0ong (args#i$))) S,stem.out.print (1sn13 args#i$); 4 4);
on a parallel computer with four cores. Heep in mind that all this happens au# tomatically. % only had to write the code above. Still* it&s helpful to under# stand what&s going on under the hood. %n the figure* the various ob)ects are arranged from top to bottom* and time flows from left to right. @ethods called on each ob)ect are depicted as gray bo,es. $he threads calling the methods are shown as thic( lines. / solid line means the thread is e,ecuting0 a dashed line means the thread is bloc(ed waiting for something to happen. ,ecution begins with the main program thread calling the main() method on the $as( ob)ect* an instance of the PrimeSmp subclass. $he main thread calls the tas(&s parallelBor() method with a lower bound of 6 and an upper bound of 7* which are the inde, bounds of the args array with four command line arguments. $he parallelBor() method creates a parallel for loop ob)ect* an instance of class %ntParallel;or!oop. :idden inside the paral# lel for loop is a team of threads* one thread for each core. ach team thread has a different ran$ in the range 6 through 7. %nitially* the team threads are bloc(ed until the program is ready to e,ecute the parallel for loop. $he main thread creates a loop ob)ect* which is an instance of an anony# mous inner subclass of class !oop. $he main thread calls the parallel for loop&s exec() method* passing in the loop ob)ect. $he exec() method cre# ates three additional copies of the loop ob)ect by calling the loop ob)ect&s clone() method. $he main thread unbloc(s the team threads* then bloc(s it# self inside the exec() method until the team threads finish. /t this point the parallel for loop begins e,ecuting. ach team thread calls the run() method on a different one of the loop ob)ects.
Chapter 2 Parallel !oops
2–-
Figure 2.1. Parallel for loop e,ecution flow
each thread prints on S,stem.out or S,stem.err" in separate per#thread in# ternal buffers. Ahen the program terminates* the buffers& contents are written to the program&s standard output stream or standard error stream"* one buffer at a time. $hus* characters printed by different threads are not commingled. $o emit a printout before the end of the program* after printing the charac# ters* call S,stem.out.lus%() or S,stem.err.lus%()". /fter returning from the loop ob)ect&s run() method* each team thread waits at a barrier. Ahen all the team threads have arrived at the barrier* the team threads bloc( themselves* and the main thread is unbloc(ed. $he main thread resumes e,ecuting and returns from the parallel for loop&s exec() main()
method.after $he the main threadfor continues method parallel loop. e,ecuting the code in the tas(&s $hus* the parallel program follows this pattern of e,ecution' • • •
Sequential section single main program thread" Parallel section multiple team threads" Sequential section single main program thread"
$his pattern* of one or more parallel sections interspersed within a sequential section* is found in almost every multicore parallel program. Inly a portion of the program is e,ecuted in parallel0 the rest of the program is e,ecuted se# quentially. Ae will return to this observation when we study parallel program performance. Ahat if we run the PrimeSmp program with four command line argu# ments on a parallel computer with more than four cores $he parallel team will have more threads than needed to handle all the loop inde,es. %n that
2–16
BIG CPU, BIG DATA
case* team threads ran( 6 through 7 call the loop ob)ects& run() methods as described above* and team threads ran( and higher merely proceed directly to the barrier without calling the run() method. Ahat if we do a parallel for loop with more loop inde,es than there are cores %n that case* each team thread will call its loop ob)ect&s run() method more than once Ae&ll study how that wor(s in the ne,t chapter.
Points to Remember • • •
•
• •
• •
•
•
• •
Arite a Parallel Java 2 program as a subclass of class $as(. Put the program code in the tas(&smain() method. $he tas(&s main() method must be an instance method* not a static method. Bou can parallelie a loop if there are no sequential dependencies among the loop iterations. +se the parallelBor() pattern to parallelie a for loop. $he parallel for loop inde, goes from the lower bound to the upper bound inclusive Put the loop body code in the run() method of the inner !oop subclass. $o terminate the program* don&t call S,stem.exit()0 instead* throw an e,ception. $o emit a printout before the end of the program* call S,stem.out .lus%() or S,stem.err.lus%(). %n a non#parallel program* override the static coresRequired() method to return the number of cores the program will use* namely 1. +se the >ja"a pj2? command to run your Parallel Java 2 program. +se the > debug-makespan? option to measure the program&s running time.
Chapter 3 Parallel Loop Schedules ► Part I. Preliminaries ▼ Part II. Tightly Coupled Multicore Chapter 2. Parallel Loops Chapter 3. Parallel Loop Schedules Chapter 4. Parallel Reduction Chapter 5. Reduction aria!les Chapter ". Load #alancing Chapter $. %&erlapping Chapter '. (e)uential *ependencies Chapter +. (trong (caling Chapter ,-. ea/ (caling Chapter ,,. 01hausti&e (earch Chapter ,2. euristic (earch Chapter ,3. Parallel or/ ueues ► Part III. Loosely Coupled Cluster ► Part I. P6 7cceleration ► Part . Map8Reduce
3–2
BIG CPU, BIG DATA
#
itcoin (bitcoin.org) is an open-source, peer-to-peer, digital currency system. Bitcoins are created by “mining,” reminiscent of ho the !"#$ %old &ush prospectors panned for gold. 'ur net program eample implements a simplified ersion of bitcoin mining. *ou specify a coin ID, hich is +ust a string of headecimal digits, along ith a number N. he program then finds a #-bit nonce such that the digest has N leading ero bits. he digest is computed by concatenating the coin /0 and the nonce, applying the 124 cryptographic hash function to that, and applying 1-24 to the result5 digest 6 1-24 (1-24 (coin /0 77 nonce)) he digest is a 24-bit number. n eample ill clarify hat the program does5 $ java pj2 edu.rit.pj2.example.MineCoinSeq fedcba98765!2"# "6 Coin % & fedcba98765!2"# 'once & ###########"9c8 %i(e)t & ####dec!25d"#67#f7abab#""a!7a#f2dcdaba#2a6b65a6a59e7 eec7ba778c
8or the coin /0 fedcba98765!2"#, the program found that the nonce "9c8 yielded a digest ith ! leading ero bits (# leading ero headecimal digits). o did it find this nonce9 By brute force search. he program simply started ith nonce :, computed the digest, incremented the nonce, and repeated until the digest had N leading ero bits. ;i2, independent of all the other bits. here fore, the probability that the N leading bits are all eroes is !>2N, and e epect to eamine 2N nonces on the aerage before e find the golden nonce. ?en for small N alues li
Chapter 3. Parallel Loop Schedules ! 2 3 # 4 D "
Main pro(ram. public void main Strin(34 ar() tro:) ;xception 1 ,terra, ar()3#4+ ' & nte(er.par)ent ar()3"4+ if " ? ' @@ ' ? 6! u)a(e+ Set up ma)* for leadin( eroe). ma)* & A"B 6 D ' D "B+ Set up for computin( a) di(e)t). coind0lu)'once & ne: b,te 3coind.len(t E 84+ S,)tem.arra,cop, coindF #F coind0lu)'onceF #F coind.len(t+ )a256 & ne: S-256+ di(e)t & ne: b,te 3)a256.di(e)tSie4+ /r, all nonce) until te di(e)t a) ' leadin( ero bit). for lon( nonce & #B+ nonce & #x7B+ EE nonce 1 /e)t nonce. 0ac*in(.unpac*Bon(>i(;ndian nonceF coind0lu)'onceF coind.len(t+ )a256.a) coind0lu)'once+ )a256.di(e)t di(e)t+ )a256.a) di(e)t+ )a256.di(e)t di(e)t+ if 0ac*in(.pac*Bon(>i(;ndian di(e)tF # G ma)* && #B 1 0rint re)ult). S,)tem.out.printf HCoin % & I)InHF -ex.toStrin( coind+ S,)tem.out.printf H'once -ex.toStrin( nonce+
& I)InHF
Listing 3.1. ineCoin1eA.+aa (part !)
3–#
BIG CPU, BIG DATA
Erogram ineCoin1eA (;isting 3.!) is a seAuential program that soles the preceding problem. /t begins by declaring seeral global ariables on lines $–!$. hese are actually fields of the as< subclass, so they are in scope for all of the tas<@s methods as ell as any inner classes. ;ines 2D–3: etract the coin /0 and N from the command line. ;ine 33 computes a mas< ith the N most significant bits set to ! and the other bits set to := the bits that are ! in the mas< correspond to the bit positions that must be : in the digest. ;ines 3–3" set up a byte array that ill be input to the hash function, ith the coin /0 at the beginning plus eight bytes at the end to hold the nonce. ;ine 3$ creates an ob+ect to compute the 1-24 hash function, using class edu.rit.crypto.124. ;ine #: sets up a byte array to hold the digest. ;ines #3–3 are the heart of the computation. he loop iterates oer all nonces from : to 2 3 – !. he loop body unpac
Ho let@s go parallel ith program ineCoin1mp (;isting 3.2). s / did ith the primality testing program in Chapter 2, /@ll change the for loop to a parallel for loop. But notice that the loop body code refers to the global ariables coind0lu)'once, )a256, an d di(e)t. /n program ineCoin1mp, there ill be multiple threads eecuting copies of the loop body concurrently. /f the threads all manipulate the same global ariables, the threads ill interfere ith each other= hen one thread changes the state of a shared global ariable, that ill ipe out another thread@s changes, and the program ill not find the correct anser. o aoid this, / ant each thread to hae its o#n separate cop of each of these ariables. hese are called thread$local %ari$ a&les. oeer, the other ariablesG coind, ', a nd ma)*Gcan remain shared global ariables, because the threads ill only read them, not change them. hese are called shared glo&al '()* %aria&les Gariables that are
S,)tem.out.printf H%i(e)t -ex.toStrin( di(e)t+ brea*+ L L
3–4 & I)InHF
L 0rint a u)a(e me))a(e and exit. private )tatic void u)a(e 1 S,)tem.err.println H)a(eN java pj2 H E Hedu.rit.pj2.example.MineCoinSeq coinid? '?H+ S,)tem.err.println Hcoinid? & Coin % exadecimalH+ S,)tem.err.println H'? & 'umber of leadin( ero bit) H E H" .. 6!H+ tro: ne: lle(alr(ument;xception+ L Specif, tat ti) ta)* require) one core. protected )tatic int core)Oequired 1 return "+ L L
pac*a(e edu.rit.pj2.example+ import edu.rit.cr,pto.S-256+ import edu.rit.pj2.Bon(Boop+ import edu.rit.pj2.Scedule+ import edu.rit.pj2./a)*+ import edu.rit.util.-ex+ import edu.rit.util.0ac*in(+ public cla)) MineCoinSmp extend) /a)* 1 Command line ar(ument). b,te34 coind+ int '+ Ma)* for leadin( eroe). lon( ma)*+ Main pro(ram. public void main Strin(34 ar() tro:) ;xception 1 ,terra, ar()3#4+ ' & nte(er.par)ent ar()3"4+ if " ? ' @@ ' ? 6! u)a(e+ Set up ma)* for leadin( eroe). ma)* & A"B 6 D ' D "B+
Listing 3.2. ineCoin1mp.+aa (part !)
3–
BIG CPU, BIG DATA
shared by all the threads, Fritten 'nce, and &ead any times. Erogram ineCoin1mp starts out the same as program ineCoin1eA, ith declarations for the shared global ariables coind, ', and ma)*. he main program code is the same, up to the loop at line 32, hich becomes a parallel for loop. here@s a call to the parallel for loop@s )cedule method, hich /@ll gloss oer for no. he loop body is in the inner ;ong;oop subclass. (he loop body uses class ;ong;oop, hich has a loop inde of type lon(, rather than class ;oop, hich has a loop inde of type int.) his time, / declare the coind0lu)'once, )a256, and di(e)t ariables as fields of the inner ;ong;oop subclass, hich ma
Chapter 3. Parallel Loop Schedules
public void run lon( nonce 1 /e)t nonce. 0ac*in(.unpac*Bon(>i(;ndian nonceF coind0lu)'onceF coind.len(t+
di(e)tF # G ma)*
& I)InHF & I)InHF & I)InHF
0rint a u)a(e me))a(e and exit. private )tatic void u)a(e 1 S,)tem.err.println H)a(eN java pj2 HE Hedu.rit.pj2.example.MineCoinSmp coinid? '?H+ S,)tem.err.println Hcoinid? & Coin % exadecimalH+ S,)tem.err.println H'? & 'umber of leadin( ero bit) H E H" .. 6!H+ tro: ne: lle(alr(ument;xception+ L L
Listing 3.2. ineCoin1mp.+aa (part 2)
3–"
BIG CPU, BIG DATA
mines ho the or< of the for loop is diided among the team threads. /f you don@t specify a schedule for a parallel for loop, the default is to use a i!ed schedule. fied schedule diides the loop inde range into / subranges, here / is the number of team threads. ?ach subrange is the same sie, ecept possibly the last. (/f the loop inde range is not eenly diisible by /, the last subrange ill include only the leftoer indees.) 8or eample, suppose the loop inde goes from ! to !::. ere@s ho a fied schedule ould diide the loop inde range among different numbers of team threads5 •
hus, ith a fied schedule, the or< of the parallel for loop is diided as eAually as possible among the team threads, and each team thread does a contiguous range of loop indees. Erogram ineCoin1mp@s parallel for loop goes through 2 3 indees (disregarding the early loop eit). /f / ere to use the default fied schedule, and if / ere to run the program on a tardi) node ith four cores, loo< at the indees each team thread ould do (in headecimal)5 • • • •
his is not hat / ant. /n effect, each team thread ould be doing a separate golden nonce search ithin a different inde range, so / ould epect each team thread to do an aerage of 2 N iterations before finding a golden nonce, so the parallel program ould ta
Chapter 3. Parallel Loop Schedules • • • •
3–$
&an< : does :, #, ", !2, . . . &an< ! does !, 4, $, !3, . . . &an< 2 does 2, , !:, !#, . . . &an< 3 does 3, D, !!, !4, . . .
Ho all the team threads are searching the same inde range starting at :, and as soon as any team thread finds the first golden nonce, the program terminates. he or< of searching this one inde range is diided eAually among the team threads, so ideally the parallel program should yield a speedup factor of #. his parallel for loop schedule is called a leaprog schedule. he loop indees are allocated in a round robin fashion among the / team threads. ?ach team thread does a series of noncontiguous loop indees, ith each inde eAualing the preious inde plus /. o get a leapfrog schedule in the ineCoin1mp program, / called the parallel for loop@s )cedule method on line 33, passing leapfro( as the argument, before actually eecuting the loop. he possible schedules, listed in enum edu.rit.p+2.1chedule, are fixed (the default), leapfro(, d,namic, proportional, and (uided. Erogram Erime1mp in Chapter 2 uses a fied schedule= program ineCoin1mp uses a leapfrog schedule= e ill see eamples of dynamic, proportional, and guided schedules in later chapters. ere@s a run of the parallel program on on a four-core tardi) node, ith the same arguments as the preious seAuential program run. $ java pj2 debu(&ma*e)pan edu.rit.pj2.example.MineCoinSmp J b!e5da6#""!57#6f 2 Coin % & b!e5da6#""!57#6f 'once & ##########f6fa72 %i(e)t & ######fa"86e57"af5e#9"#fc8a#ed87#78f7!!7#"6dd#6cf c!a#5b!28e8 Kob ma*e)pan 866 m)ec
he parallel program found the same golden nonce as the seAuential program, but it too< only ". seconds instead of 33.D secondsGa speedup factor of 33$" J "# 6 3."$".
Under the Hood 8igure 3.! shos in more detail hat happens hen the ineCoin1mp program eecutes the parallel for loop on a parallel computer ith four cores. (he main method is not shon.) Fhen the main thread calls the parallel for loop@s exec method, the main thread bloc
3–!:
BIG CPU, BIG DATA
Figure 3.1. Earallel for loop eecution flo, leapfrog schedule
threads doing one of the run inocationsGteam thread ran< ! calling run9 in this eampleGfinds the golden nonce, that team thread calls the loop ob+ect@s )top method and proceeds to the barrier. fter the loop has been stopped, as each team thread finishes its current inocation of the run method, each team thread proceeds to the barrier ithout calling the run method further. Fhen all the team threads hae arried at the barrier, the team threads bloc<, the main thread unbloc
Points to Remember •
•
•
•
•
•
•
0eclare ariables as shared global F'& ariables (fields of the as< subclass) if the parallel for loop team threads ill read them but not alter them. hread synchroniation is not needed hen reading shared global F'& ariables. 0eclare ariables as thread-local ariables (fields of the loop body subclass) if the parallel for loop team threads ill read and alter them. Do not initialie thread$local %aria&les as part o their declarations. /nitialie thread-local ariables in the )tart method. hread synchroniation is not needed hen reading and riting threadlocal ariables. 0ecide hich
Chapter 4 Parallel Reduction ► Part I. Preliminaries ▼ Part II. Tightly Coupled Multicore Chapter 2. Parallel Loops Chapter 3. Parallel Loop Schedules Chapter 4. Parallel Reduction Chapter 5. eduction !aria"les Chapter #. Load $alancing Chapter %. &'erlapping Chapter (. Se)uential *ependencies Chapter +. Strong Scaling Chapter ,-. ea/ Scaling Chapter ,,. 01hausti'e Search Chapter ,2. euristic Search Chapter ,3. Parallel or/ ueues ► Part III. Loosely Coupled Cluster ► Part I!. 4P 6cceleration ► Part !. Map7educe
4–2
BIG CPU, BIG DATA
I
magine a square dartboard (Figure 4.1) with sides of length 1 and with a quadrant of a circle drawn in it. Suppose we throw a large number N of darts at the dartboard, and suppose C of them land inside the circle quadrant. ssuming the darts land at random positions, the ratio C!N should be appro"# imatel$ the same as the ratio of the circle quadrant%s area & π!4 to the dart# board%s area & 1' C!N π!4, or π 4C!N. his suggests an algorithm for appro"imating π' *enerate N random ( x, y) points in the unit square+ count how man$ lie at a distance of 1 or less from the srcin ( x2 y2 - 1) $ielding C; report 4C!N as the estimate for π. ow# e/er, to get an accurate estimate, N needs to be /er$ large, so we want to do the computation in parallel. s shown in Figure 4.2, a sequential program simpl$ throws N darts, counts C, and calculates π. 0n the other hand, a parallel program running on, sa$, four cores partitions the computation among four threads. ach thread throws N!4 darts and counts how man$ of its own darts land within the circle quadrant. he program then has to add these per#thread counts, C through C3, together to get the total count C, from which the program calculates π. he process of combining multiple parallel threads% results into one o/er# all result is called reduction. ere the reduction operation is addition, or sum, and we refer to the reduction as a sum-reduce. (0ther programs would use other reduction operations as part of the same reduction pattern.) isting 4.1 is the sequential π program. 5t is mostl$ self#e"planator$. 0n line 26 5 created a pseudorandom numer !enerator "P#NG$, an instance of class edu.rit.util.7andom, to generate the random ( x, y) dart coordinates. he 879*%s seed (constructor argument) determines the sequence of random numbers the 879* will produce. wo 879*s initiali:ed with the same seed will generate generate different the same sequences. sequence+ two initiali:ed withand different seeds will he 879*s user specifies the seed the number of darts N on the command line. 5 used the 8arallel ;a/a 2 ibrar$ class edu.rit.util.7andom, rather than the standard ;a/a class bl (line 14)+ this class performs the reduction automaticall$. 5 initiali:ed the global counter
