Improved Statistical Test

Improved Statistical Test for  Multiple-Condition Multiple-Condition Microarray Data Integrating ANOVA wit Clustering

!ared "riedman

#

Tis researc was done at Te $oc%efeller  &niversity and supervised 'y (rian )ir% *

#+ #, .ast .nd Ave/ New 0or%/ N0/ N0/ ##*,+ .mail1 vicvic2att+net+ vicvic2att+ net+ 3one1 *#**45-#64 *+ .mail1 )ir%(2$oc%efeller+edu+ )ir%(2$oc%efeller+edu+ 3one1 *#*-6*7-784

Summary #

Microarrays are e9citing new 'iological 'iological instruments+ Microarrays promise to to ave important applications in many fields of 'iological researc/ 'ut tey are currently fairly inaccurate instruments and tis inaccuracy increases teir e9pense an d reduces teir usefulness+ Tis researc introduces a new tecni:ue for analy;ing data from some types of microarray e9periments tat promises to produce more accurate results+ .ssentially/ .ssentially/ te metod wor%s 'y identifying patterns in te genes+

A'stract Motivation1 Statistical significance significance analysis of microarray data is currently a pressing pressing  pro'lem+ A multitude of statistical statistical tecni:ues ave 'een proposed/ and in te case of simple two condition e9periments tese tecni:ues wor% well/ 'ut in te case of multiple condition e9periments tere is additional additional information tat none of tese tecni:ues ta%e into account+ Tis information is te sape of te e9pression vector for eac gene/ i+e+/ te ordered set of e9pression measurements/ and its usefulness lies in te fact tat genes tat are actually affected 'iologically  'y some e9perimental circumstance tend to fall into a relatively small num'er of clusters+ clusters+
Introduction Microarrays are part of an e9citing new class of 'iotecnologies tat promise to allow monitoring of genome-wide e9pression levels >#?+ Altoug te tecni:ue presented in tis tis  paper is :uite general and would/ in principle/ wor% wit oter types of data/ it as 'een developed for microarray data+ Microarray e9periments wor% 'y testing e9pression levels of some set of genes in organisms e9posed to at least two different conditions/ and usually ten trying to determine wic >if any? of tese genes are actually canging in response to te cange *

in e9perimental condition >*?+ &nfortunately/ &nfortunately/ te random variation in microarrays is often large large relative to te canges trying to 'e detected+ Tis ma%es it very ard to eliminate false positives/ positives/ genes wose random fluctuations ma%e tem erroneously e rroneously seem to respond to te e9perimental condition/ and false negatives/ genes wose random fluctuations mas% te fact tat tey are actually responding to te e9perimental condition+ condition+ It is standard statistical statistical practice to to andle tis  pro'lem 'y using replicates/ i+e+/ several microarrays run in identical settings settings >#-5?+ Altoug replicates are igly effective/ te e9pense of additional replicates ma%es it wortwile to minimi;e te num'er needed 'y using more effective statistical tests+

=en tere are only two conditions con ditions in an e9periment/ te conventional statistical test for differential e9pression is te t-test >*/8?+ (ut @ong/ et. al. >/ al. >/ also 8/7? reali;ed tat 'y running a separate t-test on eac gene/ valua'le information is lost/ namely te fact tat te actual  population standard deviation of all te genes is rater similar+ similar+ Due to te small num'er of replicates in microarray e9periments tere are usually a few gene s tat/ 'y cance/ app ear to ave e9tremely low standard deviations and tus register even a very small cange as igly significant 'y a conventional t-test+ In reality -- and as additional replicates would confirm -- te te standard deviations of tese genes are muc iger/ and te cange is not at all significant+ In response to tis issue/ @ong et. al. derived using (ayesian statistics a metod of transforming standard deviations tat/ in essence/ moves outlying standard deviations closer to te mean standard deviation+ &nfortunately/ &nfortunately/ te resulting algoritm/ wic tey call Cy'er-T/ Cy'er-T/ wor%s only wit two condition e9periments+ e9periments+ Since tis researc is concerned wit multiple condition e9periments/ it was necessary to e9pand te metod to wor% wit tis type of e9periment/ te standard statistical test for wic is called ANOVA ANOVA >ANalysis Of Variations? Variations? >,?+ Bere we report te successful e9tension of te Cy'er-T algoritm to multiple condition e9periments/ creating te algoritm we call Cy'er-ANOVA+

In microarray e9periments involving several conditions/ te e9pression vector of eac gene formed 'y consecutive measurements of its e9pression level as a distinctive sape/ wic contains useful information+ information+ Te more conditions in te e9periment e9periment >and tus num'ers in te e9pression vector?/ te more distinctive will 'e te te sape+ Often 'iologists are interested interested in finding genes wit e9pression vectors of similar sape/ and tis interest as gene rated a well6

developed process for grouping genes 'y sape sape %nown as clustering >5?+ Te 'asis of clustering clustering tecni:ues is te concept of correlation/ te similarity 'etween two vectors+ =e cose to use te 3earson Correlation/ 'ecause tis measure generally captures more accurately te actual  'iological idea of correlation >*?+ =it =it te 3earson correlation/ as elsewere/ distance is defined as one minus te correlation correlation and measures te dissimilarity dissimilarity 'etween two vectors+ &sing tis measure of distance/ it is possi'le to solve te pro'lem of ta%ing a large num'er of unorgani;ed vectors and placing tem into clusters of similar sapes1 tis is %nown as )-means clustering >#*?+ Once te )-means algoritm as 'een completed/ it is often useful useful to determine ow well eac gene as clustered+ Tis is done 'y computing a representative vector/ often called a center/ center/ for eac cluster and calculating te distance 'etween eac gene and te center of tat genes cluster >#4/#?+

Te fundamental tesis of tis researc is tat genes ac tually affected 'y e9perimental circumstances tend to fall into a relatively small num'er of clusters/ and tat tis information can  'e used to ma%e a more accurate statistical test+ =ile it is in principle possi'le/ it is is not often te case tat large num'ers of affected genes 'eave in seemingly random and entirely dissimilar   patterns+ #? and !iang et al+/ >##?+ (ot studies ran )-Means clustering clustering on multiple condition data and found several distinct cluster sapes tat accounted for all clearly significant genes+

=e used tis connection 'etween clustering and significance to create a more accu rate statistical test wose essential steps are te following+ "irst Cy'er-ANOVA Cy'er-ANOVA is run on te entire dataset and te igly significant significant genes are clustered+ Te oter genes are placed into te te cluster in wic tey fit 'est/ and te distance 'etween eac gene and te center of its cluster is computed+ Tere are now two values for eac gene/ te 3 value >pro'a'ility? >pro'a'ility? from te Cy'erANOVA ANOVA test test and te D value >distance? from te clustering+ Tese two values are com'ined and te resulting value is more accurate tan eiter one alone+ 4

Te potential advantage of tis algoritm was validated 'y te creation and successful testing of a large num'er of artificial artificial data sets+ Artificial data was cosen over real e9perimental data for two main reasons1 >#? in e9perimental data/ uncertainty always e9ists over wic genes are actually differentially e9pressedE tis presents maFor pro'lems in determining ow well some given algoritm wor%sE and >*? no single microarray e9periment is actually representative of all  possi'le microarray e9periments G to determine accurately te performance advantage of an algoritm/ a large num'er of very different e9periments must 'e used/ a proi'itively e9pensive and difficult process+ Artificial creation of data sets allows every parameter parameter of te data to 'e  perfectly controlled and te algoritms algoritms performance can 'e tested on every possi'le com'ination of parameters+

Te 'asic metod of analysis was to generate an artificial data set from a num'er of of  parameters/ run te clustering algoritm on te data and measure its performance/ run Cy'erCy'erANOVA ANOVA on tat data and on similar data 'ut 'u t wit additional replicates and measure its  performance/ and ten compare te results+ Te performance gain of te clustering algoritm is  pro'a'ly 'est e9pressed as e:uivalent replicate gain1 te num'er of additional replicates tat would 'e needed to produce te results of te clustering algoritm using a standard Cy'erANOVA+ ANOVA+ On reasona'le simulated data/ we found tat te 'enefit was appro9imately one additional replicate added to an original original two+ =it =it additional improvements to te algoritm algoritm design/ tis saving could pro'a'ly 'e increased still furter/ furter/ and te advantage is clearly significant enoug to warrant use in actual microarray e9periments+

Metods Algoritm Design Te algoritm is implemented as a large >H* lines? macro script in Microsoft Visual Visual (asic for Applications controlling Microsoft .9cel+ Te essential steps of te algoritm are te following1 #+ $un Cy'er-ANOVA Cy'er-ANOVA on all genesE *+ AdFust 3 values for multiple multiple testingE 

6+ &se te results of >*? to coose an appropriate num'er of most significant genesE 4+ Cluster tis selection of genes using )-MeansE + Ta%e te remaining genes and group eac into te cluster wit wic it correlates 'estE 8+ Compute te distance 'etween eac gene and te center of its clusterE 7+ Compute a com'ined measurement measurement of significance significance for eac gene+

#+ $un Cy'er-ANOV Cy'er-ANO VA on all genes+ Cy'er-ANOV Cy'er-ANO VA is Fust a simple e9tension of Cy'er-T Cy' er-T// 'ut its novelty causes it to warrant some discussion+ Cy'er-T ta%es as input a list list of genes/ means in eac condition/ standard deviations in eac condition/ condition/ and num'er of replicates+ It gives as output a set of 3-values 3-values muc li%e tose from a conventional t-test/ 'ut more accurate+ accurate+ Te improvement is 'ased on te idea of regulari;ing te standard deviations/ essentially moving outlying standard deviations closer to te mean+ At te same time/ it recogni;es tat standard deviation may cange significantly >and inversely? wit intensity level/ and tus genes are regulari;ed only wit genes of similar intensity+ intensity+ Te algoritm wor%s 'y first computing for eac gene a 'ac%ground standard deviation/ te average standard deviation of te undred genes wit closest intensity/ intensity/ and ten com'ining te 'ac%ground standard deviation d eviation wit te o'served standard deviation in a type of weigted average+ A standard standard t-test is ten run/ 'ut using te regulari;ed standard deviation instead of te o'served one+

Cy'er-ANOVA Cy'er-ANOVA wor%s 'y te same metod/ ta%ing as input means and standard deviations in any num'er of conditions+ conditions+ .ac 'ac%ground standard deviation is calculated using using te intensities of te genes in teir own con dition/ and te values are com'ined using te same formula+ (ut 'ecause tere are more tan two two conditions/ a t-test cannot 'e appliedE instead we use ANOVA/ inputting te regulari;ed standard deviation instead of te o'served one+ Te result gives a dramatic increase in accuracy/ an e:uivalent replicate gain of rougly one+

*+ AdFust 3 values for multiple testing+ To run te clustering algoritm/ we need to select a num'er of genes tat are igly significant to form te seed clusters around wic all oter genes will cluster+ cluster+ Te difficult step in te process is deciding deciding ow many genes to select1 it is critical to get at least a few seeds for all clusters 'ut also critical not to ave too many false 8

 positives+ Since te first re:uirement is impossi'le impossi'le to determine 'y calculation/ we use te second+ In order to coose an appropriate appropriate num'er of e9pected false positives/ positives/ we must ave an accurate estimate of te false positive rate+ Te simplest way of estimating te false positive rate is a (onferroni correction/ wic simply multiplies eac p value 'y te total num'er of tests+ Tis metod assumes tat all genes are independent/ and tends to 'e too conservative/  particularly wit small num'ers of replicates/ replicates/ and a num'er of more accurate tecni:ues ave  'een proposed+ Bowever/ for tis tis step/ only a roug estimate is necessary/ necessary/ and tus te (onferroni correction suffices+

6+ &se te results of >*? to coose an appropriate appropriate num'er of most significant significant genes+ Once te e9pected false positive rate for any num'er of te most significant genes as 'een estimated/ some num'er of genes must 'e selected suc tat te num'er of e9pected e9pe cted false positives is some  percentage of te num'er selected genes+ Tere is no particular correct way to define tis  percentageE it depends on ow significant te genes in te least significant cluster are/ wic cannot 'e determined directly+ directly+ Bowever/ we ave found tat te algoritm is tolerant tolerant to deviations from optimality >Ta'le >Ta'le 8?+

4+ Cluster tis selection selection of genes using )-Means+ Te cosen group of most significant significant genes is clustered using using te )-Means algoritm wit wit te 3earson correlation+ Te primary  pro'lem wit all )-means clustering algoritms algoritms is tat te num'er of clusters must 'e declared at te start >#6?+ >#6?+ (ut li%e te determination determination of te num'er of genes to include in te initial clustering/ tere is no pro'lem as long as ) is rater ig+ Tis can 'e e9plained conceptually 'y considering wat appens as ) increases to differentially e9pressed and non-differentially e9pressed genes+ If tere are only differentially differentially e9pressed genes in te original set/ ten as ) is raised/ genes tat were originally placed in one cluster will separate into multiple clusters+ "ortunately/ tis is guaranteed 'y te process of )-Means to lower te D values and create 'etter cluster sapes+ If tere are non-differentially non-differentially e9pressed genes/ ten as ) is increased tese genes will tend to separate out from te differentially e9pressed genes and form teir own clusters G a igly desira'le occurrence+ &nfortunately/ &nfortunately/ if ) 'ecomes very large/ te noise genes may start to cluster very well wit oter noise genes forming erroneously low distances/ and te oter noise genes added in te ne9t step will ave a 'etter cance of incidentally finding a cluster wit 7

wic tey correlate well+ In principle/ too large large a ) migt 'e partially partially monitored 'y cec%ing for very small clusters and for clusters wit very ig average distances/ 'ot of wic could 'e removed and teir genes forced to Foin a different cluster+ cluster+ Tese metods ave not yet 'een implemented+

+ Ta%e Ta%e te remaining genes and group eac into te cluster wit wic it correlates 'est+ Te clusters formed in in step 4 act as seeds for te remaining genes+ Te clusters formed optimally include all of te cluster sapes of actually differentially e9pressed genes and no oters+ It was possi'le to form tese tese clusters/ even if tey were less tan ideal/ 'y using only a su'set of te genes in te data set+ Now it is possi'le to loo% at te remaining genes and to see ow well eac 'elongs in one of te clusters+ Ideally/ Ideally/ eac differentially e9pressed gene would correlate perfectly wit e9actly one cluster and eac noise gene would correlate poorly wit all of tem+ In practice/ of course/ tis is far from from te case/ 'ut te principle principle remains+ Since eac gene is/ of course/ only e9pected e9pe cted to correlate well wit one cluster/ eac gene in te rest of te dataset is added to te cluster wit wic it correlates 'est+

8+ Compute te distance 'etween eac gene and te center of its cluster+ cluster+ Te measurement of ow well eac gene 'elongs in its cluster is now determined 'y co mputing te 3earson correlation 'etween eac gene and te center of its cluster+ cluster+ Te resulting correlations are converted to to distances >distance  #- correlation?+ Tese distance values values are te te %ey num'ers produced 'y te first alf of te algoritm+ algoritm+ Tey represent weter a gene vectors vectors sape is similar to te sape of gene vectors %nown to 'e significant+ Te 'asis of tis researc is tat low distances tend to imply differentially differentially e9pressed genes+ &nfortunately/ &nfortunately/ distance values alone are not an accurate measure of significance G ran%ing genes 'y distance alone produces results far worse tat te original Cy'er-ANOV Cy'er-ANOVA+ A+ Instead/ te distance >JDK? values must 'e com'ined wit te original 3 values to give new values 'etter tan eiter alone+

7+ Compute a com'ined measurement of significance significance for eac gene+ Te goal now is to ta%e te 3 value and te D value and com'ine tem in some way to ma%e a new value+ !ust li%e wit te calculation of te previous pro'a'ilities/ wat we are really interested in is finding te  pro'a'ility of a non-differentially non-differentially e9pressed gene wit a certain standard deviation attaining 'ot ,

a 3 value less tan or e:ual to te o'served 3 value and a D value less tan or e:ual to te o'served D value+ &nfortunately/ &nfortunately/ te teoretical 'asis for te connection 'etween 3 and D values is not well defined/ and it may 'e very complicated and e9periment-dependentE tere may well 'e no useful analytic solution to tis pro'lem+ Instead/ we generate te distri'ution of 3 and D values for eac e9periment+ "irst tousands of non-differentially non-differentially e9pressed genes are generated using te standard deviation tat would 'e e9pected at teir intensity level and teir 3 and D values are calculated+ Ten/ for eac gene in te data set/ set/ te num'er of non-differentially non-differentially e9pressed genes tat ave 'ot 3 and D values e:ual to or lower tan te 3 and D values of tat gene is counted+ Tat num'er is divided 'y te num'er of non-differentially non-differentially e9pressed genes generated/ and tat :uotient gives an appro9imate pro'a'ility+

It is important to note tat for igly significant genes/ tis metod does not give accurate  pro'a'ilities/ for if # non-differentially e9pressed genes are generated/ only genes wit an actual pro'a'ility of occurring of a'out #- are li%ely li%ely to 'e generated+ Tus/ genes in te te data set tat are actually significant to a pro'a'ility of 'elow #- will almost certainly not ave any generated genes more significant and will will all 'e assigned pro'a'ilities pro'a'ilities of ;ero+ (ut tis is not a maFor concern/ since tese genes are so significant tat tere is little :uestion tat tey are differentially differentially e9pressed/ and teir precise ran% is not li%ely to 'e important+ If it is/ ten te genes wit ;ero com'ined values can 'e ran%ed internally 'y teir 3 values+ In tis case/ witin witin te very most significant genes/ te algoritm will not ave ad any effect+


important parameters are te num'ers of differentially and non-differentially e9pressed genes/ num'er of groups/ cluster sapes/ standard deviations/ and fold canges+

"or eac data set/ a'out , genes were created/ wit te num'er of differentially e9pressed genes ranging from a'out # to to a'out + Te rest were genes wose mean value did not cange across conditions/ cond itions/ 'ut tat mean value did vary 'etween different nondifferentially differentially e9pressed genes+ Te num'er of groups used is a %ey determinant of te  performance of te algoritm/ more groups causing 'etter performance+ Most of te data sets generated used si9 groups/ 'ut a set wit twenty groups was tried+

Te cluster sapes of non-differentially e9pressed genes are/ of course/ flat lines/ for tat is te definition of 'eing non-differentially non-differentially e9pressed+ Te cluster sapes of te differentially differentially e9pressed genes were more complicated+ Tey are drawn loosely loosely from aoet. ao et. al. >#? al. >#? 'ut really te specific sapes cosen are not all tat important+ An important property of te 3earson correlation is tat/ given some e9pression vector/ te pro'a'ility of some oter e9pression vector aving a distance to te original vector 'elow some value is te same regardless of te sape of te original vectorE it depends only on te num'er of conditions conditions >,?+ On te oter and/ given two original vectors/ te pro'a'ility of a test vector aving a distance 'elow some value to eiter one of tem does depend on te sapes of te original vectors+ As intuition suggests/ vectors of opposite sape >li%e a linear increasing vector and linear decreasing vector? ma%e it more li%ely for a test vector to cluster well wit one of tem+ To te e9tent tat cluster sape does affect te algoritm/ te more similar te cluster sapes/ sapes/ te 'etter te algoritm will perform+ Cluster sapes can 'e graped and assigned names 'ased on teir sape+ Data was generated using up to twelve cluster sapes >"igure #a/ #'?+

#

Figure 1A. 1A. Six of the Twelv e Cluster Shapes 11

   l   e   v   e    L   n   o    i   s   s   e   r   p   x    E   g   o    L

10

Spike 9

Dip Cli Up

8

Cli Down !ill

7

"alle# 6 5 1

2

3

4

5

6

Condition

Figure 1B. The Other Othe r Six Cluster Shapes Shapes

Linear Gradient Up

11 Linear Gradient Down

10    l   e   v   e    L   n   o    i   s   s   e   r   p   x    E   g   o    L

9

Concave Up Gradient Up

8 Concave Down Gradient Up

7

Concave Up Gradient Down

6 5 1

2

3

4

5

6

Condition

Concave Down Gradient Down

Te metod of assigning standard deviations was co sen to 'e as realistic as possi'le+ Cy'er-ANOVA Cy'er-ANOVA ta%es into account te fact tat standard deviation often canges significantly wit a'solute intensity in microarray e9periments/ and te generated data model tis ##

 penomenon+ Te data of @ong/ et+ al/ >? wic is freely availa'le/ was graped/ mean intensity vs+ standard deviation/ and a :uadratic :uadratic regression calculated+ Tis regression e:uation is an e9plicit function for standard deviation in terms of mean/ and it was used wit a modification to calculate standard deviations for te generated data+ Te modification is re:uired re:uired 'ecause it is not te case tat standard deviation depends solely on intensity+ intensity+ Te mean intensity vs+ standard deviation grap does not ma%e a perfect lineE tere is considera'le widt to tat curve/ and tis too was simulated+ Te standard deviation itself was randomi;ed wit a small small meta-standard deviation+ Te assumption of normality normality in tis generation is is pro'a'ly false/ 'ut te cange in te algoritm performance caused 'y introducing te meta-randomi;ation at all is so small tat it seems igly unli%ely tat a 'etter model would produce a significant difference+

To ta%e full advantage of te standard deviation dependence on mean/ non-differentially e9pressed genes were generated wit means separated 'y undredts 'etween , and #8 >on a log* scale?+ Tis created a considera'le range of standard deviations for for eac data set+ set+ Te means of differentially e9pressed genes were generated starting from , different 'aselines/ te integers from , to #+

.ac differentially e9pressed gene was generated at five different fold canges1 #+/ #+7/ *+/ 6+/ and 8+ on an unlogged scale+ Te si9 and tree fold cange genes generally made up te seed clustering group/ and te #+ and #+7 fold cange genes were responsi'le for most of te difference in performance 'etween algoritms+

Scoring Algoritms Once data as 'een generated and assigned 3 values using Cy'er-ANOVA Cy'er-ANOVA and com'ined values using te clustering algoritm/ tere must 'e some way to compare te performance+ =e introduce a new metod for determining d etermining te performance of an algoritm/ wic we 'elieve to fi9 a sortcoming in te standard metod+

Te most common metod of scoring appears a ppears to 'e a consistency test >#-7? wic wor%s te following way+ Say tere is a data set wit N genes %nown to 'e differentially e9pressed+ #*

"irst/ some significance significance test is performed performed and eac gene assigned a significance level+ Ne9t/ te genes are ran%ed 'y significance level/ most significant first+ Out of te top N most significant genes/ te num'er D of genes g enes tat are actually differentially e9pressed is determined/ and te score is D L N+

Te pro'lem wit te consistency test is tat ignores te fact tat it often will matter weter te >N-D? genes are listed consecutively rigt after te Nt gene or at te very 'ottom of te list+ (iologists would prefer tat te differentially differentially e9pressed genes 'e listed as close to te top as possi'le/ for it is easier to find an interesting gene or group of genes if it is iger on te list+

As a solution/ we introduce introduce an algoritm called ran% sum+ $an% sum also ta%es a list of genes ran%ed 'y significance level/ 'ut computes te score differently+ differently+ $an% sum is e:ual to te sum of te ran%s of all te N differentially e9pressed genes minus >#*6N?/ te minimum score possi'le+ @ower ran% sums indicate indicate 'etter performance/ and a perfect significance significance test gives a ran% sum of ;ero+

(ut ran% sum as significant flaws/ too/ for it gives too muc weigt to te genes placed after te Nt ran%+ In te worst case/ one single misplaced gene could cange te score from  to te num'er of genes in te e9periment/ e9periment/ , in tis case+ Bere is a successful modification+ Instead of summing te actual ran%s/ we sum for all differentially e9pressed genes ">$an%?/ were " is some increasing concave-down function+ Tere is no particular rigt coice for "/ "/ for te preferred function will vary depending on te e9periment and e9perimenter/ 'ut 'ot logaritmic and polynomial >to a power less tan/ say/ say/ #L*? functions give almost identical results+

Te modified ran% sum successfully determines ow well a significance test as ran%ed te genes+ (ut te score from te ran% sum itself is not particularly particularly useful informationE it must  'e placed in a conte9t+ Since te goal of any significance test is ultimately to reduce te num'er of replicates necessary to attain accurate results/ we convert te ran% sum to a new measure tat we call e:uivalent replicate gain+ .:uivalent replicate gain is defined defined as te num'er of replicates replicates #6

needed to reac te same ran% sum using only a standard Cy'er-ANOVA+ Cy'er-ANOVA+ Specifically/ Specifically/ tis is done 'y running Cy'er-ANOVA Cy'er-ANOVA on several data sets wit parameters all identical e9cept for te num'er of replicates+ Te grap of num'er of replicates replicates vs+ ran% sum is drawn and a regression calculated+ Te clustering algoritm algoritm is ten run and a ran% sum o'tained+ Tat ran% sum is entered into te inverse of te regression function to find te appro9imate num'er of replicates re:uired to attain te same performance using only Cy'er-ANOVA+ Cy'er-ANOVA+ Te e:uivalent replicate gain is te difference 'etween tat num'er of replicates and te num'er of replicates actually used+ Anoter useful measure is te e:uivalent replicate gain percentage/ te e:uivalent replicate replicate gain divided 'y te e:uivalent replicate num'er+ num'er+ Tis gives an appro9imation appro9imation of te percent cost saving possi'le 'y using te te algoritm and fewer replicates+ replicates+ Te com'ination of te tecni:ues of ran% sum and e:uivalent replicate gain percentage as 'een very successful+

$esults =e ave generated a num'er of artificial data sets using a wide range of parameters and found te e:uivalent replicate gain percentage of te te algoritm in many situations+ situations+ Te most important parameters were found 'e te num'er of conditions/ te num'er of cluster sapes/ te num'er of replicates/ te num'er of non-differentiat n on-differentiated ed genes in te initial clustering/ and te num'er of clusters omitted from te initial clustering+

In a'solutely ideal situations/ situations/ te algoritm can give an enormous 'enefit+ An e9periment wit twenty conditions/ one cluster sape/ two replicates/ and perfect initial clustering approaces suc an ideal situation and gives e9cellent results >Ta'le >Ta'le #? +

 Num'er of Conditions 8 *

.:uivalent $eplicate  Num'er  6+46 8+*,

.:uivalent $eplicate
Ta'le #+ 3erformance of te clustering algoritm on data wit two num'ers of conditions/ conditions/ one cluster sape/ two replicates/ and perfect initial clustering+

A more realistic case would involve fewer conditions/ say/ si9/ and particularly wit tis smaller num'er of conditions/ te num'er of cluster sapes 'ecomes a factor in te performance of te algoritm+ Te performance 'enefit is still still :uite significant1 significant1 recall tat e:uivalent replicate gain percentage is an appro9imation appro9imation of te cost saving due to to fewer replicates+ As e9pected/ te performance 'enefit is reduced 'y additional cluster sapes >Ta'le *?+

 Num'er of Cluster

.:uivalent $eplicate

.:uivalent $eplicate

Sapes  Num'er 
.ven wen identical parameters are used to generate te data/ te ran% sum of any algoritm will fluctuate 'etween data sets differing only 'y random num'er ge neration+ "ortunately/ "ortunately/ tis variation appears to affect Cy'er-ANOVA Cy'er-ANOVA and te Clustering algoritm e:ually e:u ally// for te e:uivalent replicate gain percentage does not cange significantly >Ta'le >Ta'le 6?+ Tis invariance as te 'eneficial effect of ma%ing it unnece ssary to run multiple data sets for eac  parameter com'inationE more tan one adds little information+

Data Set

.:uivalent $eplicate

.:uivalent $eplicate

# * 6

 Num'er  6+64 6+47 6+*5


4

6+*

6,+ #

Ta'le 6+ 3erformance of te clustering algoritm on data generated randomly four times using identical parameters1 si9 conditions/ two replicates/ two cluster sapes/ and perfect initial clustering+

Cange in te standard deviation or in te fold canges of te genes will certainly affect  'ot te ran% sum from 'ot Cy'er-ANOVA Cy'er-ANOVA and clustering+ clustering+ Over small canges/ te e:uivalent replicate gain percentage is not seriously affected/ 'ut if te standard deviation 'ecomes so large as to corrupt te initial clustering/ te performance loss is significant >Ta'le >Ta'le 4?+

Standard Deviation

.:uivalent $eplicate

.:uivalent $eplicate

>in multiples of te

 Num'er 


normal SD used elsewere? 9# 6+64 4+#  9* 6+#4 68+6 94 *+86 *4+ Ta'le 4+ 3erformance of te clustering algoritm on data wit tree levels of standard standard deviation/ using si9 conditions/ two replicates/ two cluster sapes/ and perfect initial clustering+

@i%e Cy'er-ANOVA/ Cy'er-ANOVA/ and in fact all statistical tests/ te e:uivalent replicate gain  percentage of te algoritm decreases as te num'er of replicates 'ecomes larger and te test  'ecomes more accurate >Ta'le >Ta'le ?+ If te initial clustering is correct/ correct/ tere is no loss in  performance/ 'ut in e9periments wit parameters similar similar to tose of Ta'le Ta'le / te algoritm  pro'a'ly ceases to 'e wortwile after four replicates+ Algoritm Tested Clustering Clustering Clustering Cy'er-ANOVA Cy'er-ANOVA Cy'er-ANOVA

Num'er of $eplicates * 4 , * 4 ,

.:uivalent $e $eplicate  Num'er  *+,4 4+74 ,+#* *+56 +*6 ,+7

.:uivalent $eplicate
#8

Ta'le + 3erformance of te clustering algoritm on data wit tree num'ers of replicates/ replicates/ using si9 conditions/ twelve cluster cluster sapes/ and perfect initial initial clustering+ Te rate of decrease in  performance at iger num'er of replicates is compared wit tat rate in Cy'er-ANOV Cy'er-ANOVA+ A+ Note tat for te clustering algoritm/ te e:uivalent replicate gain percentage is e9pressed as a gain from Cy'er-ANOVA/ Cy'er-ANOVA/ 'ut tat for Cy'er-ANOVA/ Cy'er-ANOVA/ te e:uivalent replicate gain percentage is e9pressed e9press ed as a gain from a standard ANOVA ANOVA test+ Tus/ on te scale tat ta t Cy'er-ANOVA Cy'er-ANOVA is 'eing compared to/ te clustering algoritm result for a two replicate da ta set is e:uivalent to te plain ANOVA of four replicates+

Te most serious impact to te algoritms performance occurs if te initial clustering is done incorrectly+ incorrectly+ Te most difficult and important important step is coosing te num'er of genes to include in te initial groupE if eiter too many or too few are cosen/ te performance of te algoritm will 'e diminised significantly significantly++ Te ultimate goal is to coose a set of genes tat includes all te clusters c lusters of differentially e9pressed genes witout including any non-differentially e9pressed genes+ It is true tat tis migt migt 'e very difficult to do wit e9perimental data/ 'ut it is fortunately true tat te algoritm is ro'ust to to small departures from optimality >Ta'le >Ta'le 8?+ Te only time tat te algoritms performance drops 'elow te performance of Cy'er-ANOVA Cy'er-ANOVA is wen almost all of te clusters are left out+

 Num'er of

.9pected


 Num'er of

 Num'er of

.:uivalent

.:uivalent

Clusters

non-

$eplicate

$eplicate
Omitted

differentially

 Num'er 

3ercentage

e9pressed 6 # * 6 

+ +* + +5 +*4

# 5  6 

genes included     

#+7 #+56 *+## *+#* *+64

-*#+8 -6+6 +#, +,4 #4+7 #7

# # +6   *+86 *6+5 6 5#  # *+ *#+4 4 6  *6 *+67 #+,  458  5 *+5 4+66 Ta'le 8+ 3erformance of te clustering algoritm on data wit several e9pected false positive positive rates for te initial clustering/ using si9 conditions/ two replicates/ and twelve cluster sapes+

Discussion Te potential cost saving of te algoritm in cases wen te correlation 'etween distance and significance is strong is igly significant+ significant+ It is interesting to note tat for normal data wit wit two replicates suc as tat in Ta'le / te percent e:uivalent replicate gain 'etween Cy'erANOVA ANOVA and Clustering is rougly roug ly e:ual to te performance gain 'etween standard ANOVA ANOVA and Cy'er-ANOVA/ on te order of 6+

It is true tat tis performance degrades in some cases/ 'ut tis is also true of Cy'erANOVA/ ANOVA/ and also of oter algoritms >6/ 4?+ One situation tat causes te percentage e:uivalent replicate gain of all tese algoritms to decrease is iger num'er of replicates+ replicates+ &nfortunately/ &nfortunately/ te performance decreases more rapidly rapidly in te clustering algoritm+ algoritm+ Tis penomenon is  peraps 'est e9plained 'y a :uality of information argument1 additional replicates increase increase te :uality of information of te 3 values/ 'ut ave little effect on te :u ality of te information of te D values+ Additional replicates will cause differentially differentially e9pressed genes to cluster 'etter/ 'etter/ in a sense reducing te num'er of false negatives/ neg atives/ 'ut will not reduce te false positives/ 'ecause non-differentially e9pressed genes are Fust as li%ely to randomly ave low distances given any num'er of replicates+ Tus/ as te num'er of replicates replicates increases/ te D values gradually cause less improvement+

Te most maFor degradation deg radation of te clustering algoritms algoritms performance/ owever/ is in a situation tat does not affect affect te oter algoritms/ tat of 'ad initial clustering+ clustering+ $educing te severity of tis pro'lem will 'e te primary direction for furter researc on tis algoritm+ Some possi'le solutions include >#? clustering all te genes 'ut weiging teir significance in te clustering algoritm 'y 3 valueE and >*? setting te num'er of genes to use as seeds 'y trying #,

many values and seeing te point a'ove wic new clusters stop appearing to form+ $egardless of weter tis issue can 'e resolved in te general case/ if in some e9periment it is strongly suspected >peraps on 'iological grounds? tat no clusters ave 'een omitted from te initial set of genes/ tis clustering metod can 'e used and safely e9pected to produce a su'stantial  performance 'enefit+

$eferences #+ Dudoit/ Dudoit / S+ 0ang/ 0+/ 0+/ Callow/ M+!+/ and Speed/ T+3 T+3++ >*?+ Statistical Statistic al metods for identifying differentially differentially e9pressed genes in replicated cDNA microarray microarray e9periments+ Tecnical report P7,/ Stat Dept/ &C-(er%eley+ *. )nudsen/ S+ >**? A >**? A Biologist’s Guide to Analysis of DNA Microarray Data+ Data+ !on =iley Q Sons+ 6+ 3an/ =+/ >**? A comparative review of statistical metods for discovering differentially e9pressed genes in replicated microarray e9periments+  Bioinformatics #,>4?148-4  Bioinformatics #,>4?148-4 4. Tuser/ Tuser/ V+<+/ V+<+/ Ti'sirani/ $+/ and Cu/ <+ >*#?+ Significance analysis of microarrays applied to te ioni;ing radiation response+  Proc. Natl. Acad. Acad. Sci. &SA Sci. &SA 5,1##5-#*#+ + @ong/ D+/ Mangalam/ B+/ Can/ (+/ Tolleri/ Tolleri/ @+/ Batfield/ <+/ (aldi/ 3+/ 3+/ Improved statistical inference from DNA microarray microarray data using analysis of variance va riance and a (ayesian statistical "ramewor%+  Journal of Biological Chemistry. Chemistry. *78 >6?1 #5567-#5544+ 8. (aldi/ 3+/ 3+/ and (runa%/ S+/ >#55,? Bioinformatics: >#55,? Bioinformatics: he Machine !earning A""roach A""roach++ MIT 3ress/ Cam'ridge MA+ 7+ (aldi/ 3+ 3+ and @ong/ A+D+ A+D+ >*#?+ A (ayesian (ayesian framewor% for te analysis analysis of microarray e9pression data1 $egulari;ed t-test and statistical inferences of gene canges+  Bioinformatics #715-#5 ,+ )reys;ig/ .+ >#57? Mathematical >#57? Mathematical Statistics: Princi"les Princi"les and Methods+ Methods+ !on =iley =iley QSons/ Inc+ 5+ .wens/ =+/ =+/ and *#? Statistical Methods in Bioinformatics: An #ntroduction+ #ntroduction+ Spriger-Verlag/ N0+ N0+ #+ ao/ $+/ *?+ Analysis of p6-regulated gene e9pression patterns using oligonucleotide arrays+ Genes and De$elo"ment + #4 >,?1 5,#-556+ #5

##+ !iang M/ $yu !/ )iraly M/ Du%e )/ $ein%e V/ and )im S)+ #576? Pattern >#576? Pattern Classification and Scene Analysis+ Analysis+ !on =iley =iley and Sons+ #6+ Saran/ $+/ and Samir/ Samir/ $+ C@IC)1 a clustering clustering algoritm wit applications to to gene e9pression analysis+ In Proceedings In Proceedings of the &''' Conference Conference on #ntelligent Systems for  Molecular Biology ,#SMB''-0 !a Jolla0 CA0 67-6#8 CA0 67-6#8 #4+ Alon/ &+/ (ar%ai/ N/+ notterman/ D+A+/ #555? (road patterns of gene e9pression revalued 'y clustering analysis of tumor and normal colon tissues pro'ed 'y oligonucleotide arrays+  Proc. Natl. Acd. Sci. %SA/ %SA/ 581874-87 #+ Serloc%/ <+ Analysis of large-scale gene e9pression data+ Curr 1"in #mmunol  *  * AprE#*>*?1*#-

*