François François Pachet, Pachet, Active Active Listenin Listening: g: Whatis in theAir? the Air?Sony Sony CSLinternal Report, Report, 1999.
MusicListening:WhatisintheAir? FrançoisPachet SONYComputerScienceLaboratoryParis 6,rueAmyot 75005Paris-France
[email protected]
Abstract
The The XXth cent centur ury y is full full of tech techno nolo logi gica cal l inve invent ntio ions ns that that made made the the very very idea idea of a listeningdevicepossible,fromtheearlygramophonestothelatestportableminidisk playe players. rs. What What evolut evolutions ions can wepredictfor wepredictfor the listen listening ing device devices s ofthe future future, , and how how thes these e evol evolut utio ions ns will will chan change ge the the way way we acce access ss and and list listen en to musi music c ? In this this chapter,we chapter,wesugg suggestthat estthatlisten listeningdevicescanbe ingdevicescanbe greatly greatly enhancedbyprovidin enhancedbyprovidingnew gnew forms forms of user controls which provide provide userswith semanticall semantically yprese preserving rving variations. variations. Thesecontrolsareintendedtoallowlistenersdifferentmusicalperceptionsonapiece ofmusic,byoppositiontotraditio ofmusic,byoppositiontotraditionallisten nallistening,inwhichthemusica ing,inwhichthemusicalmediaisplayed lmediaisplayed pass passive ively ly by some some neut neutra ral l devi device ce. . The The obje object ctiv ive e is both both to incr increa ease se the the music musical al comfortoflisteners,and,whenpossible,toprovidelistenerswithsmootherpathsto newmusic(musictheydonotknow,ordonotlike).Thischapterillustratesthisidea onafewexamplesofactivelisteningprojectsconductedatSonyComputerScience Labor aborat ator ory y, Pari Paris, s, base based d on the the noti notion on of cons constr trai aine ned d explo explora rator tory y spac space. e. Thes These e constrainedspacessuggestthattheclassicalboundariesbetweencomposing,listening and and mixin mixing g may may be rede redefi fine ned, d, ther thereb eby y assig assigni ning ng new new role roles s to comp compos oser ers, s, sound sound engineersandlisteners. 1. FromButton FromButtonstoExp stoExplorat loration ion
Weproposetheideaofexploratorylisteningenvironments,asanaturalevolutionin thehistoryofmusicalcontrols.Wefirstsketchabriefhistoryofmusicalcontrols,and thenintroducethenotionofsemantic-preservingmusic thenintroducethenotionofse mantic-preservingmusicalexploratoryenvi alexploratoryenvironment. ronment. 1.1 Historyof HistoryofMus Musical icalCont Controls rols Eachtechnologicaladvancehasbroughtwithitnewformsofcontrols.Theoriginsof listeningmachineswithmass-producedmusicalmaterialsmaybetracedbacktothe Phonograph,inventedbyThomasEdisonin1878,whichusedtinfoilcylinders,and shortlyaftertheGramophone,inventedbyBerlinerin1888,whichusedflatdisks.In these these device devices, s, there there was no contro control l intent intention ionall allygiven ygiven to the user user (se, (se, e.g. e.g. Read Read & Welch, Welch, 1976). 1976). There There was, was, howeve however,an r,an uninte unintenti ntiona onal l contro control l in the Gramop Gramophon hone e in thatthehorncouldbeturned around,therebyinfluencingthedirectivityofthesound around,therebyinfluencingthedirectivityofthesound source source. . Electr Electrici icity ty soon soon beganto beganto beused for listen listening ing device devices, s, both both with with radio radio and
1
François François Pachet, Pachet, Active Active Listenin Listening: g: Whatis in theAir? the Air?Sony Sony CSLinternal Report, Report, 1999. with with new electr electrica ically lly recor recorded ded disk disk playe players rs in the 20s. 20s. The use of electr electrici icity ty also also introduce introduced d new controls: controls: the volume butt button on and and the the treble/bass button. button. Juke-boxes Juke-boxes were were introd introduce uced d in 1927, 1927, allowi allowing ng listen listeners ers to select explic explicitl itly y music music titles titles from from a given cat catalogue of disk isks, usin sing various sorts of push butto ttons. The next big techno technolog logica ical l advanc advance e was the invent invention ion of binaur binaural al (ster (stereo) eo) record recording ing method method in 1931. The correspond corresponding ing control control was the panoramicbuttonallowingtocontrolthe amount amount of signal signal in one loudsp loudspeak eaker er or the other. other. Finall Finally, y, digita digital l format format for audio audio intr introd oduc uced ed more more cont contro rols ls, , e.g. e.g. on the the equa equali liza zati tion on of soun sound. d. In all all thes these e case cases, s, technologicaladvanceswerefollowedbytheintroductionof“technical”controls,i.e. controlsoperatingdirectlyonthetechnol controlsoperatingdirectlyonthetechnology(seeFig ogy(seeFigure1). ure1).
Figur Figure e 1. A Phonog Phonograp raph h (Ediso (Edison, n, 1978, 1978, left); left); a Gr Gram amopho ophone ne (Be (Berli rliner ner, , 1988, 1988, middle),aRock-Ola120-selectionJuke-Box,andaMinidiskplayer(Sony,1997, righ right) t). . Adva Advanc nces es in tech techno nolo logy gy do not not nece necess ssar ary y impl imply y more more inte intell llig igen ent t user user control.
1.2 AMattero AMatterofSe fSemantic mantics s Theverynotionofmusicalcontrolraisestheissueof semantics.Theissueofmusical semant semantics ics - does does music music have have meanin meaning g ? - has been been long long debate debated d by musico musicolog logists ists, , leadin leading g to diffe differen rent t theori theories, es, which which usuall usually y parall parallele eled d the theori theories es of semant semantics ics for languages.Oneofthemaindistinctionmadebytheoristsistheoppositionbetween so-c so-cal alle led d “ref “refer eren enti tial alis ists ts” ” and and “abs “absol olut utis ists ts”. ”. Refe Refere rent ntia iali list sts s clai claim m that that musi musica cal l meaningcomesfromactualreferencesofmusicalformstooutsideobjects,i.e.music meanssomethingwhichisexternaltomusicitself.Forinstance,aparticularscalein India ndian n musi music c may may have have a refe refere renc nce e to a part partic icul ular ar huma human n mood mood. . Abso Absolu luti tist sts, s, e.g. e.g. Strawi Strawinsk nsky, y, claim claim on the contra contrary ry that that the meanin meaning g of music, music, if any, any, lies lies in music music itself,i.e.intherelationsentertainedbymusicalformstogether.Althoughthesetwo viewpointsarenotnecessarilyexclusive,asnotedbyMeyer(Meyer,1956),theyleave open open much much of the questi question on of meani meaning. ng. Eugen Eugene e Narmou Narmour r elabor elaborate ated d a much much more more precisetheoryofmusicalmeaningbasedonthepsychologicalnotionofexpectation (Narmour,1992).Inthistheory,meaningoccursonlywhenmusicalexpectationare deceiv deceived. ed. On the otherhand,Rosen otherhand,Rosen argue argues s (Rosen (Rosen, , 1994) 1994) that that the respon responsibi sibilit lityof yof preservingthemeaningofamusicalpieceliesonlyintheperformeritself,whohasto choosecarefullyamongainfinitesetofpossibleinterpretationswhichoneisclosestto theone“intended”bythecomposer. Withou Without t commit committin ting g to one partic particula ular r theory theory of musica musical l meanin meaning, g, we can note note that that meaning-whatev meaning-whateveritmeans-hastodowith eritmeans-hastodowithchoosingamongasetofinterpretations the “right “right one” one” or the “right “right ones”, ones”, i.e. those those intend intended ed by the compos composer er. . A second second remarkisthatthecontrolsgivenbythehistoryofsoundrecordingtechnologyhave
2
François François Pachet, Pachet, Active Active Listenin Listening: g: Whatis in theAir? the Air?Sony Sony CSLinternal Report, Report, 1999. neverhadanyconcernaboutmusicalsemantics:whatdoesitmeantoraisethesound level level of a record record ? to shiftthe shiftthe signal signal to the left left loudsp loudspeak eaker? er? to increa increase se the bass bass freq freque uenc ncy y ? Are Are the the inte intent ntio ions ns of the the comp compos oser ers, s, or even even of the the sound sound engi engine neer ers, s, preservedinanyway? From From this this remar remark, k, we sugge suggest st that that “inter “interest esting ing” ” musica musical l contro controls ls should should preser preserve ve some some sort sort of semant semantics ics of the musica musical l materi material, al, i.e. i.e. preser preserve ve intent intention ions, s, whenev whenever er possible. possible. We argue argue thatmore that more meaningful meaningful controls, controls, in theconte the context xt of modern modern digital digital multimedia multimedia technolog technology,amount y,amount to shiftingfrom shiftingfrom traditiona traditional l button-base button-based d technology technology tomusicalexplorationspaces. 1.3 MusicInte MusicInteract ractivity ivity Aswehaveseen,technologicalbuttonsbearnosemantics,becausetheyaredirectly groundedonthetechnolog groundedonthetechnology,witho y,withoutanymodelofthemusicbeingplay utanymodelofthemusicbeingplayed.Butwhat ed.Butwhat canbesuchamodel? Interes Interestingapproa tingapproachesin chesin musicalinteracti musicalinteractivity vity arethemusicnotatio arethe musicnotationsystems,inthe nsystems,inthe contextofannotationofmusicdocuments,asintheworksofLepain(1998),orinthe Acousmogr Acousmographsystem aphsystem (INA-GRM (INA-GRM).In ).Inthese these systems,theprimary systems,theprimary issueaddressedis issueaddressedis notmusiclisteningperse,butrathermusic notation,i.e.howtorepresentgraphically amusicaldocument(thedocumentitselfortheperceptionofthedocument),orhow toinferamodelofthemusicwhichcanbenotedorrepresentedg toinferamodelofthemusicwh ichcanbenotedorrepresentedgraphically. raphically. Anot Anothe her r answe answer r may may be foun found d in the the noti notion on of open initially lly develo developed ped in open form form, initia literature literature (Eco,1962),whichhashadmuchimpactonmusictheoryandcompos (Eco,1962),whichhashadmuchimpactonmusictheoryandcomposition ition (Stockhaus (Stockhausen,Boulez) en,Boulez).The .Theideaofmusicalopenform ideaofmusicalopenform isthatthe composerdoesnot composerdoesnot createaready-to-usescore,butratherasetofpotentialperformances,whichcanbe seenasa model ofscores,as ofscores,as explai explained ned by (Ecke (Eckel, l, 1997): 1997): “Music “Music isnot any longer longer conc concei eive ved d in form form of fini finite te unit units s but but in term terms s of mode models ls capa capable ble of prod produc ucin ing g a potentiallyinfinitenumberofvariantsofaparticularfamilyofmusicalideas”.The selectionorinstantiationoftheactualscoretobeplayedisdelegatedtotheperformer. In recent recent incarn incarnati ations ons of open open form, form, it is the listen listener er himsel himself f who instan instantia tiates tes the model, model, as for instan instance ce in the Cave (Cruz(Cruz-Nei Neira ra et al., al., 1993)or 1993)or CyberStage (Eckel, (Eckel, 1997).Inthesecases,theuserisimmersed 1997).Inthesecases,theuserisimmersedina ina realisticvirt realisticvirtualenvir ualenvironme onment,andhas nt,andhas the the cont contro rol l on his his posi positio tion n and and move moveme ment nt in a virt virtua ual l worl world. d. His His move moveme ment nts s are are translate translatedinto dinto variationsinthe variationsinthe musical musicalmate materialbeingheard rialbeingheard.Theseapproach .Theseapproachesmay esmay beconsideredasradical,inthesensethattheuserhasagreatdealofresponsibilityin makingthemusic.However,theissueofsemanticsisnotdirectlyaddressed,sincethe model modelinprincip inprincipleis leis under-de under-designe signed,i.e. d,i.e. allpossible exploration explorations sarealway arealways s“lic “licit”, it”, whatevertheymaybe.Inthisrespect,thereisastrongrelationbetweenopenform virtua virtual l enviro environme nments nts and progr programm amming ing langu language ages s for music music compos compositi ition, on, such such as OpenMusic(Assayagetal.,1997), CommonMusic(Taube,1991)or Elody(Orlareyet al.1997).Intheseapproachesindeed,thegoalistoproposetheusertoexplorespaces withasmuchfreedomaspossible,andnotconstraintheuserinspecificareas. 1.4 ActiveLi ActiveListeni stening ng ActiveListeni ActiveListeningreferstotheidea ngreferstotheidea thatlistenerscanbegivensomedegreeofcontrol thatlistenerscanbegivensomedegreeofcontrol onthemusictheylistento,thatgivesthepossibilityofproposingdifferentmusical
3
François François Pachet, Pachet, Active Active Listenin Listening: g: Whatis in theAir? the Air?Sony Sony CSLinternal Report, Report, 1999. perceptionsonapieceofmusic,byoppositiontotraditionallistening,inwhichthe musica musical l mediais mediais playe played d passiv passively ely by some some neutra neutral l device device. . The object objective ive isbothto increasethemusicalcomfortoflisteners,and,whenpossible,toprovidelistenerswith smoo smooth ther er path paths s to new new musi music c (mus (music ic they they do not not know know, , or do not not like like). ). Acti Active ve listeningisthusrelatedtothenotionofopenformoutlinedabovebutdiffersbytwo import important ant aspect aspects: s: 1) we seek to create create listeni listening ng enviro environme nments nts for existi existing ng music music repe repert rtoi oire res, s, rath rather er than than crea creati ting ng envi enviro ronm nmen ents ts for for comp compos osit itio ion n or free free musi musica cal l explor explorati ation on and 2) weaim atcreating atcreating envir environm onment ents s inwhich the variat variation ions s alway always s pres preser erve ve the the orig origin inal al sema semant ntic ics s of the the musi music, c, at leas least t when when this this sema semant ntic ics s can can be def defined ined pre precisel isely y. For us, us, the the issu issuee if ther thereefore ore not not to int introdu roducce yet anoth nother er tech techno nolo logi gica cal l butt button on in the inte interf rfac ace e of the the liste listeni ning ng devi device ce, , but but rath rather er to desi design gn butt button ons s that that “mak “make e sense sense”, ”, there thereby by brea breaki king ng the the long long trad tradit itio ion n of tech techno nolo logi gica cal l buttonsinitiatedbyEdison. What What “sense “sense”, ”, what what “meani “meaning” ng” are are we talkin talking g about about ? How can music music contro controls ls be desig designed ned to trigg trigger er semant semantic ic preser preservin ving g action actions s ? The answer answer stems stems from from the new landsc landscape ape of music music recor recordin ding g create created d by digita digital l multim multimedi edia, a, sketch sketched ed in the next section.Wewillthenillustr section.Wewillthenillustrateouridea ateourideasbytwoexample sbytwoexamplesofactiveliste sofactivelisteningpro ningprojects jects atSonyComputerScienceLaborato atSonyComputerScienceLaboratory-Paris. ry-Paris. 2. TheNewFact TheNewFactsofMu sofMultime ltimedia dia
Digitalizationofmultimediadatahasanumberoftechnicaladvantageswhicharewell knowntoday:bettersoundquality,bettercompression,losslesscopy,etc.Theaimof thischapteristoshowthatdigitalizationofmultimediadataalsoinduce-evenina stillpotentialform-anumberofrevolutionsinthewaymusicmaybeaccessedand listenedtobyendusers.Wewilloutlinethreeoftheserevolutions,whichformthe basis of ourargu our argumenta mentation, tion, focusing focusing on the paradigma paradigmatic tic shifts shifts they convey, convey, rather rather thanontechnicalaspects. 2.1 StructuredAudio:HomeasaRecons StructuredAudio:HomeasaReconstructionMachine tructionMachine Theideaofstructuredaudiohasinitiallybeendevisedtoallowbettercompressionof high qualityaudio.Standa qualityaudio.Standardiza rdizationeffo tioneffortsliketheMpeg-4projec rtsliketheMpeg-4projectembody tembodythis thisidea, idea, andtryto andtrytomakeitpracti makeitpracticalona calonalarg largescale(see,e.g.theMachin escale(see,e.g.theMachinelistening elisteningGroupof Groupof theMedialab,Sheireretal.,1998). Theideaissimple:insteadof transmittin Theideaissimple:insteadof transmittinga garead ready-t y-to-lis o-listensound,only tensound,onlya a descriptio description n ofhowtomakethesoundistransmitted.Theactualsoundisreconstructedathome, oratthelistener’slocation,providedofcoursehe/shehastherightsoftwaretoprocess this this reco recons nstr truc ucti tion on prop proper erly ly. . Stru Struct ctur ured ed audi audio o actu actual ally ly exte extend nds s this this basi basic c idea idea to include fully-fledged fully-fledged scene descriptions, that that is, not only only descr descript iption ions s ofindividu ofindividual al sounds sounds, , but descri descripti ptionof onof group groups s ofsounds ofsounds playi playingtoge ngtogethe ther r tomake upa pieceof pieceof music.Theactualtechnicaldetailsofscenedescriptionalsoincludeallwhatisneeded toreconstructasoundorpieceofmusicrightfully,e.g.effects,adaptationtothelocal soundreproductionsystem,andsoforth. Inourcontext,wearguethatthenotionofscenedescriptionopensupnewdoorsfor meani eaning ngfful con control trols. s. Indee ndeed, d, sinc since e the the musi music c is deli delive vere red d as a “ki “kit”, t”, lots lots of possibilitiescanbeimaginedtoinfluencethewaythekitisactuallybuilt,accordingto
4
François François Pachet, Pachet, Active Active Listenin Listening: g: Whatis in theAir? the Air?Sony Sony CSLinternal Report, Report, 1999. userprefere userpreferences.Ofcourse nces.Ofcourse,thesevariati ,thesevariationsaround onsaround howthekitshouldbeassembled howthekitshouldbeassembled havetobe“coherent”,whichareprecisely havetobe“coherent”,which arepreciselythematterofourwork. thematterofourwork. 2.2 Meta-data Meta-dataandA andAllT llThatJ hatJazz azz Thefactthatmusicaldataisnowproduced,codedandtransmittedinadigitalform hasnumerousandwell-knownadvantages:bettersoundquality,possibilityoflossless transm transmissi ission on and copyi copying ng (ther (thereby eby raisin raising g new copyr copyrigh ight t proble problems) ms). . An import important ant nontechnicalconsequenceisthepossibilitytoencodenotonlythemusicitself-the digi digita tali lize zed d soun sound d - but but also also any any sort sort of symb symbol olic ic info inform rmat atio ion. n. Such Such symb symbol olic ic info inform rmat atio ion n may may be used used to code code and and tran transm smit it data data on the the musi music c itse itself lf, , so-ca so-call lled ed informationoncontent,meta-dataoralso“bitsaboutbits”. Why wouldone wouldone want want totransmit totransmit such such meta-d meta-data ata ? The inter interest ests s are obviou obvious s inthe contex context t of docume document nt indexi indexing ng. . If musica musical l data data is accomp accompani anied ed with with corres correspon pondin ding g adequa adequate te descri descripti ptions ons, , digita digital l catal catalogu ogues es can can then then be accesse accessed d using using sophis sophistic ticate ated d quer query y syst system ems. s. Curr Curren ent t stan standa dard rdiz izat atio ion n effo effort rts s like like Mpeg Mpeg-7 -7 embo embody dy this this idea idea (MPEG7,1998),andtrytodefinestandardsfordescribingmeta-dataforallsortsof mult ultimed imedia ia doc docume uments. nts. MPE MPEG-7 G-7 aims ims for inst instan ancce at makin king the the web web more more sear search chab able le for for mult multim imed edia ia cont conten ent t than than it is toda today y, make make larg large e cont conten ent t arch archiv ives es accessibletothepublic. Here Here agai again, n, we woul would d like like to emph emphas asize ize the conc concep eptu tual al rath rather er than than the the tech techni nica cal l aspectsofthisparadigmshift:meta-dataopensalsodoorsforimaginingnewlistening systemsinwhichtheusermayaccessdatainadrasticallydifferentway.Insteadof bein being g a pass passiv ivee, neut neutrral supp suppo ort, mus music bec becomes omes an activ ctive, e, self self--docu docum mente nted knowledgebase.Again,whatkindoflisteningdevicescanbeimaginedthatexploit thisinformation? 2.3 SizeofD SizeofDigita igitalCa lCatalog talogues ues Digita Digitaliz lizati ation on of multim multimedi edia a data data has yet yet anothe another r conseq consequen uence: ce: the availa availabil bility ity of hugecataloguesofmultimediadatatousers.Inthecaseofmusic,thereis,herealso,a conceptua conceptualshiftwhichhasnothingtodowiththetechnolog lshiftwhichhasnothingtodowiththetechnologyoflarge yoflargedatabases. databases.The The mainissueraisedbythistechnologicaladvanceishowtoaccesshugecataloguesof music,notfromatechnicalviewpoint,butfromauser’sviewpoint.Recallthejuke box,inventedinthelate20s:atypicaljukeboxwouldcontainabout120titles,which is the the size size of an aver averag age e user user’s ’s disc discot othe hequ que. e. Brow Browsi sing ng thro throug ugh h all all the the titl titles es was was prob probab ably ly part part of the the plea pleasu sure re, , and and sele select ctio ion n coul could d be made made just just like like at home home: : by choosingoneitemoutofacollectionofitems,whichatleasttheuserhasseenonce. Now a typic typical al catalo catalogu gue e of a major major compan companyis yis about50.00 about50.000 0 items. items. What What happen happens s whenthecollectiontoselectfromissuchacatalogue?Evenmoreterrifying,what happensifalltherecordedtitlesbecomeavailablethroughnetworkstousersathome ? Esti Estima mati ting ng the the tota total l numb number er of all all reco record rded ed musi music c is diff diffic icul ult, t, but but it can can be appr approx oxim imat ated ed to abou about t 2 mill millio ion n titl titles es (see (see, , e.g. e.g. the the size size of Musi MusicB cBou oule leva vard rd or Amaz Amazon on data databa base ses) s). . The The figu figure re can can be prob probab ably ly doub double led d to incl includ ude e non non West Wester ern n musi music. c. Ever Every y mont month, h, abou about t 4000 4000 new new CDs CDs are are issu issued ed on the the mark market et. . It is clea clearl rly y imposs impossibl ible e to apply apply usual usual techni technique ques s of music music select selection ion in this this new contex context. t. What What doesitmeanto“lookfor”atitlewhenthemassoftitlesissohuge?
5
François François Pachet, Pachet, Active Active Listenin Listening: g: Whatis in theAir? the Air?Sony Sony CSLinternal Report, Report, 1999. 3. Spatializati Spatialization:The on:TheMusic MusicSpace SpaceProje Project ct
Thefirst parameter parameterwhichcomestomindwhenthinkin whichcomestomindwhenthinkingaboutusercontr gaboutusercontrolonmusic olonmusic is the the spat spatia iali lizat zatio ion n of sound sound sour source ces. s. We cond conduc uct t a proj projec ect t for for inve investi stiga gati ting ng the the tech techni niccal and and conc conceeptua ptuall issu issues es relat elated ed to mean eaning ingful user user--cont contro rol l of musi music c spatialization,called MusicSpace MusicSpace. 3.1 MotivationandDescriptionofMusicSpa MotivationandDescriptionofMusicSpace ce InMusicSpace,theusercanlistentopiecesofmusicusinganinterfaceinwhicheach instru instrumen ment t inthe pieceis pieceis repre represen sented ted by a graph graphica ical l object object (see (see Figure Figure 2). Moving Moving thes these e obje object cts s arou around nd modi modifi fies es the the mixi mixing ng of sound sound sour source ces s in the the glob global al soun sound. d. Moreover, Moreover, anobjectrepresenting anobjectrepresenting thelistenerhimself- thelistenerhimself- avatar- avatar- isalsorepresentedin isalsorepresentedin theinterface,sothatallthemixingparameters(volume,panoramicposition,etc.)are computedaccordingtotheavatar’sposition.Thebasicsystemprovidesthepossibility of 1) movi moving ng arou around nd the the avat avatar ar, , to indu induce ce a mixi mixing ng as if the the liste listene ner r was was movi moving ng around around the actual actual musical musical setup, and2) and 2) moving moving around around theinstruments theinstruments themselves themselves, , therebyinducingadifferentmixing therebyinducingadifferentmixingasifthelistenerwasasortofsoundproduce asifthelistenerwasasortofsoundproducer. r. Experimentationsofthisbasicsystemwereconductedonaveragelistenersandmusic composers.Itclearlyappearedthatalthoughthephysicalactionsofmovingavataror instr instrum umen ent t icon icons s arou around nd in a wind window ow are are very very simil similar ar, , the the poss possib ibili ility ty of movi moving ng aroundlistener’savatarsisquitedifferentconceptuallythanthepossibilityofmoving around around instrument instruments. s. Indeed, Indeed,movin moving g the avatar avatar correspon corresponds ds to theactio the action n of moving moving oneselfaroundamusicalsetting.Movinginstrumentscorrespondtoamoretechnical viewonthemusic-thesoundengineer’sview.Thissecondpossibilityappearedto someusersasheretic,sinceitpracticallygivesusersthepossibilityoftotallychanging theoverallmixingofthemusicalpiece! The secon econd d phas phase e of our our proje roject ct cons consis iste ted d in int introdu roduccing ing a way of some someho how w constr constrain aining ing user user action actions, s, to avoid avoid situat situation ions s where where the mixing mixing produc produced ed is totall totally y unrelated unrelated totheoriginalspiritofthe totheoriginalspiritofthe music(Pachet&Deleru music(Pachet&Delerue,1998).Weprocee e,1998).Weproceeded ded byintroducinga byintroducinga particula particular r technique, technique, called called constraint constraint perturbat perturbation,whichprecisely ion,whichprecisely allows allows instru instrume ments nts to be linked linked togeth together er by relati relations ons that that are alway always s enforc enforced: ed: the systemusestheseconstraintstopropagatechanges,sothatthesetupalwaysremain consistent.Forinstance,a“related”constraintmaybesetbetweenthedrumandthe bass, bass, so that that one of them them is moved moved closer closer to the listen listener’ er’s s avatar avatar, , the other other one is move moved d acco accord rdin ingl gly y (wit (with h the the same same dista distanc nce e rati ratio) o). . On the the cont contra rary ry, , a “bal “balan ance ce” ” constraintmaybesetbetweentwosoundsourcesthatshouldalwaysbemutuallyin oppo opposi siti tion on: : for for inst instan ance ce, , when when the the chor chorus usin ing g inst instru rume ment nt is brou brough ght t clos closer er, , the the accompanimentismovedaway.Theseconstraintscanfinallybecomposedtogetherto createrichenvironmentsinwhichusersmaychangetheinstrumentpositions,butthe constraint constraint system system ensures ensures thatthe overall overallmixing mixing always always remain remain consistentwith consistentwith the engineerorcomposerconstraints.
6
François François Pachet, Pachet, Active Active Listenin Listening: g: Whatis in theAir? the Air?Sony Sony CSLinternal Report, Report, 1999.
Figure Figure 2. The interface interfaceof of MusicSpace MusicSpace. . Instruments Instrumentsare arerelate related d by constraints constraints. . The Th e ava avatar tar as well well as inst instru rum ments ents can can be moved oved aroun round d by the the user. ser. Th The e constraintsembodyan“automatic”soundengineer.
3.2 Exploratio ExplorationSpa nSpace ce Ther There e are are two two way ways to inte interp rpre ret t Music MusicSp Spac ace. e. One One is to see see it as an embo embodi dime ment nt - simplisticbutoperational-ofasoundengineer:theusermaymovesoundsusinghigh leve level, l, simpl simple e acti action ons; s; the the syst system em “cor “corre rect cts” s” thes these e acti action ons s by movi moving ng othe other r sound sound source sources s accord according ing to his knowle knowledg dge e of sound sound mixing mixing. . This This knowle knowledge dge is explic explicitl itly y representedasconstraints. The The other other viewpo viewpoint int is to see mixing mixing constr constrain aints ts as anontology ontology of mixing mixing actions, actions, whichallowstomixintermsofpropertiesofsetups,ratherthanintermsofatomic actio tions on knobs and faders. This ontolog logy allows to spe specify properties of conf config igur urat atio ions ns, , whic which h are are guar guaran antie tied d to be alwa always ys enfo enforc rced ed, , rathe rather r than than spec specif ify y explic explicit it config configura uratio tions. ns. In this this respec respect, t, constr constrain aints ts repres represent ent a semanti semantics cs of sound sound source source config configur urati ation, on, and the result resulting ing - constr constrain ained ed - explor explorati ation on space space allow allows s to explorevariousconfigurationswithoutviolatingthespiritoftheoriginalmixing. MusicSpaceisalsotobeseenasanexampleofexploitationof“reconstructedmusic”. As outlin outlined ed in Sectio Section n 2.1, 2.1, future future standa standards rds will will delive deliver r music music by chunks chunks, , possib possibly ly tran transm smit itti ting ng soun sound d sour source ces s sepa separa rate tely ly, , toge togeth ther er with with spec specif ific icat atio ions ns on how how to reconstructthemusicwholefromtheparts.Constraintsareonewayofspecifyingthis recon reconstr struct uction ion, , which which nevert neverthel heless ess leaves leaves room room for new semant semanticic-pre preser servin ving g user user control.Assuch,itisaradicallynewformofGramophone,asdescribedin1.1:not only only does does MusicS MusicSpac pace e provid provide e more more refine refined d contro controls ls on sound sound spatia spatializ lizati ation on than than turningthehornaround,butthesecontrolspreservetheunderlyingintentionofsound sourceconfigurations. 4. MusicCatal MusicCatalogueA ogueAccess ccess
Theissueofmusicdeliveryconcernsthetransportationofmusicinadigitalformatto users.Musicdeliveryhasrecentlybenefitedfromtechnologicalprogressinnetwork transmissi transmission,compres on,compressionofaudio,andprotecti sionofaudio,andprotectionofdigitaldata(Memon onofdigitaldata(Memon &Wong, 7
François François Pachet, Pachet, Active Active Listenin Listening: g: Whatis in theAir? the Air?Sony Sony CSLinternal Report, Report, 1999. 1998).Theseadvancesallownoworinthenearfuturetodeliverquicklyandsafely musicto musicto users users in a digita digital l forma format t throug through h networ networks,eithe ks,either r intern internetor etor digita digital l audio audio broadcasting. Moreov Moreover, er, as seen seen in Sectio Section n 2.2, 2.2, digita digitaliz lizati ation on of data data makes makes it possib possible le today today to transport transport informatio informationon non content,and content,andnotonlydataitself.Toge notonlydataitself.Together,these ther,thesetechnique techniques s give give the user users, s, at home home, , acce access ss to huge huge cata catalo logu gues es of anno annota tate ted d mult multim imed edia ia data data, , musicinparticular.Thesetechniquesaimatsolvingthedistributionproblem,i.e.how to tran transpo sport rt data data quic quickl kly y and and safe safely ly to user users. s. Para Parado doxic xical ally ly, , thes these e tech techno nolo logi gica cal l adva advanc nces es also also raise raise a new new prob proble lem m for for the the user user: : how how to choo choose se amon among g such such huge huge catalogues? 4.1 Motivation Motivationand andIdeas Ideas Fromtheuserviewpoint,accessingalargequantityofmusicindeedisproblematic:it cannotbe cannotberedu reducedtoa cedtoa simpledatabaseprobl simpledatabaseproblem,because em,because, ,by bydefinitio definition,usersdonot n,usersdonot know know prec precise isely ly what what they they look look for. for. The The prob proble lem m of choo choosin sing g item items s is gene genera ral l in westernsocieties,inwhichthereisaneverincreasingnumberofproductsavailable. Forentertainmentandspeciallymusicthechoosingproblemisspecific,becausethe unde underl rly ying goal goals s - pers person onal al enjo enjoy yment ment and and exci excite teme ment nt - do not not fall fall in the the usua usual l categ categori ories es of ration rational al decisi decision on making making. . Althou Although gh unders understan tandin ding g a user’ user’s s goals goals in listeningtomusicisverycomplexinfullgenerality listeningtomusicisverycompl exinfullgenerality,wecansummarizetheproblemto ,wecansummarizetheproblemto twobasicandcontradictoryingredie twobasicandcontradictor yingredients:desireofrepetition,anddesireofsur nts:desireofrepetition,anddesireofsurprise. prise. Thedesireof repetitioniswellknowninmusictheoryandcognition.Experimental psychologyshowstheimportanceofrepetitionsinmusic.Atthemelodicorrhythmic levelsofmusic“repetitionbreedscontent”.Forinstance,sequencesofrepeatingnotes crea create te expe expect ctat atio ions ns of the the same same note note to occu occur. r. At a high higher er leve level, l, tona tonal l musi music, c, for for instan instance, ce, is based based on struct structure ures s that that create create strong strong expect expectati ations ons or the next musica musical l events events to come come (for (for instan instance, ce, a domina dominantseven ntseventhchord thchord create creates s anexpectat anexpectation ion ofa resol resolut utio ion) n). . Musi Music c theo theori rist sts s have have trie tried d to capt captur ure e this this phen phenom omen enon on by prop propos osin ing g variou various s theori theories es of musica musical l percep perceptio tion n based based on expecta expectatio tion n mechan mechanism isms s (see (see e.g. e.g. Meyer,1956),particularlyformodelingtheperceptionofmelodies(Narmour,1992). At the more more global global level level of music music select selection ion, , this this desire desire of repeti repetitio tion n tends tends to have have peop people le want wantin ing g to liste listen n musi music c that that they they know know alre alread ady y (and (and like like) ) or musi music c that that is similartomusictheyalreadyknow.Forinstance,aBeatlesfanwillmostprobablybe inte intere rest sted ed in list listen enin ing g the the late latest st Beat Beatle les s boot bootle leg g cont contai aini ning ng hith hither erto to unre unrele leas ased ed versionsofhisfavoritehits. Ontheotherhand,thedesirefor surprise isakeytounderstandingmusic,atalllevels ofperception ofperception. .Thevery Theverytheoriesthatemphas theoriesthatemphasizetheroleof izetheroleof expectatio expectationin nin music also show show that that liste listene ners rs do not not favo favor r expec expecta tatio tions ns that that are are alwa always ys fulf fulfil ille led, d, and and enjoy enjoy surprisesandunty surprisesanduntypicalmusic picalmusicalprogre alprogression ssions(seee.g. s(seee.g. SmithandMelara,1990).Ata SmithandMelara,1990).Ata largerlevel,listenerswantfromtimetotimetodiscovernewmusic,newtitles,new bands, bands, or new musica musical l genres genres. . This This desire desire is not necess necessari arily ly made made explici explicit, t, but is neverthelessasimportantasthedesireforrepetition. Of course course, , these these two desir desires es are contr contradi adicto ctory ry, , and the issue issue in music music select selection ion is preciselytofindtherightcompromisebetweenthesetwoforces:provideuserswith items they already already know, know, and provid provide e them them with with items items they they do not know, know, but will will probablylike. 8
François François Pachet, Pachet, Active Active Listenin Listening: g: Whatis in theAir? the Air?Sony Sony CSLinternal Report, Report, 1999. Fromtheviewpointofrecordcompanies,onegoalofmusicdeliveryistoachievea betterexploitationofthecatalogue.Indeed,recordcompanieshaveproblemswiththe exploi exploitat tation ion of their their catal catalog ogue ue using using standa standard rd distri distribut bution ion scheme schemes. s. For For techni technical cal reasons,only reasons,onlya a smallpartofthe smallpartofthe catalogue catalogueis is actually actually“active”, “active”,i.e.proposedtousers, i.e.proposedtousers, intheformofeasilyavailableproducts.Moreimportantly,theanalysisofmusicsales show shows s clea clearl rly y decr decrea ease ses s in the the sale sales s of albu albums ms, , and and shor shortt-te term rm poli polici cies es base based d on sell sellin ing g lots lots of copi copies es of a limi limite ted d numb number er of item items s (hit (hits) s) seem seem to be no long longer er profitabl profitable. e. Additional Additionally,the ly,the salesof generalgeneral-purpo purpose se“samp “samplers” lers” (e.g. (e.g. “Best ofLove Songs” Songs”) ) are no longer longer profit profitabl able, e, either either becaus because e usershavealrea usershavealready dy the hits hits intheir owndiscotheque,orbecausetheydonotwanttobuysamplersinwhichtheylikeonly afractionofthetitles.Exploitingmorefullythecatalogueshasbecomeanecessityfor recordcompanies.Insteadofproposingasmallnumberofhitstoalargeaudience,a natura natural l soluti solution on is to incre increase ase diver diversity sity, , by propos proposing ing more more custom customize ized d albums albums to users. 4.2 Approaches Approachesin inMusi MusicSel cSelectio ection n Curren Current t approa approache ches s in music music select selection ion can be split split up in two catego categorie ries: s: 1) query query syst system ems s for acc accessi essing ng musi usic cata catalo log gues, ues, and 2) rec recomme ommend ndat atio ion n syst system ems s for proposingnoveltitlestousers.Inbothcases,theseapproachesprovidesetsofitemsto theuser,whichhe/shehasstilltochoosefrom. Query Query systems systems address address mainly mainly database database issues for storing storing and representin representing g musical musical data. data. They They propos propose e means means of query querying ing musica musical l items items using using some some sort sort of semant semantic ic information.Variouskindsofqueriescanbeissuedbyusers,eitherveryspecific(e.g. the the titl title e of the the Beat Beatle les s song song whic which h cont contai ains ns the the word word “pep “peppe per” r”), ), or larg largel ely y unde under r specified(e.g.“Jazz”titles). Collab Collabora orativ tive e filter filtering ing approa approache ches s (Shard (Shardana anand, nd, and Maes, Maes, 1995) 1995) aim primar primarily ily at achievingthe“surprise”goal,i.e.issuerecommendationsofnoveltitlestousers,with thehopethattheserecommendationswillbeenjoyed.Collaborativefilteringisbased ontheideathatthereare patterns patternsintastes-tastesarenotdistributeduniformly.This ideacanbeimplementedverysimplybymanagingaso-called profileforeachuser conn connec ecte ted d to the serv servic ice. e. The The prof profil ile e is typic typical ally ly a set set of assoc associa iati tion ons s of item items s to grades.Forinstance,inthe MyLaunchsystem,gradesvaryfrom0(Ihateit)to5(this is my pref prefer erre red d item item). ). In the reco recomm mmen enda dati tion on phas phase, e, the the syste system m looks looks for for all all the the agentshavingasimilarprofiletheuser’s.Thissimilaritycanbecomputedeasilybya distan distance ce measu measure re on profil profiles, es, such such as a hammin hamming g distan distance. ce. Finall Finally, y, the syste system m will will lookforitemslikedbythesesimilaragents,whicharenotknownbytheuser,and recommendstheseitemstohim/her.Typicalcollaborativefilteringsystemsformusic aretheFireflysystem(Firefly,1998), MyLaunch MyLaunch(MyLaunch,1998),theAmazonweb site(Amazon,1998),orthesimilarityengine(Inf site(Amazon,1998),orthesimilarity engine(Infoglide,1998). oglide,1998). However,therearelimitationstothisapproach.Theselimitationsappearbystudying quan quanti tita tati tive ve simu simula lati tion ons s of coll collab abor orat ativ ive e filt filter erin ing g syst system ems, s, usin using g simu simula lati tion ons s techniquesinspiredfromworksonthedisseminationofculturaltastes(Epstein,1996; Cavalli-SforzaandFeldman,1981). Thefirstoneistheinclinationto“clusterformation”,whichisinducedbythevery dynam dynamics ics of the syste system. m. The exper experime imenta ntal l result results s achiev achieved ed so far show show that that such such systemsproduceinterestingrecommendationsfornaïveprofiles,butgetstuckassoon 9
François François Pachet, Pachet, Active Active Listenin Listening: g: Whatis in theAir? the Air?Sony Sony CSLinternal Report, Report, 1999. as the profiles get bigger (about 120 ite items): eclectic tic profiles les are som somehow disadvantaged. Another Another problem, problem, shown experimenta experimentally lly, , is that the dynamics dynamics inherently inherently favors favors the creat creation ion of hits, hits, i.e. i.e. items items which which are are likedby likedby a huge huge fract fractionof ionof the popula populatio tion. n. Of course,theexistenc course,theexistenceofhitsis eofhitsis notabadthinginitself notabadthinginitself,buthitsneverth ,buthitsnevertheless elesslimitthe limitthe probabilityofotheritemsto“survive”inaworlddom probabilityofotheritemsto“survive”inaworlddominatedbyweight inatedbyweightsums. sums. In short, short, collab collabora orativ tive e filte filterin ring g is a means means of buildi building ng similar similarity ity relat relation ions s betwee between n items, items, based based on statist statistica ical l proper propertie ties s of group groups s ofagents. ofagents. As such, such, it addre addresses sses the goal ofsurprise,ina ofsurprise,ina safeway,bypropos safeway,byproposingusersitem ingusersitemswhicharesimilar swhicharesimilarto toalrea already dy knownones.However, knownones.However, clusterforma clusterformationanduneve tionandunevendistribu ndistributionofchance tionofchancesto stoitems items (e.g.formati (e.g.formationofhits)arethemaindrawba onofhits)arethemaindrawbacksoftheapproach cksoftheapproach,bothfromtheuser’s ,bothfromtheuser’s viewpo viewpoint int (clust (cluster ers s from from which which it is diffic difficult ult to escape escape), ), and the conten content t provid provider’ er’s s viewpoint(nosystematicexploitationofthecatalogue). 4.3 On-the-flyMusicProgramGenerat On-the-flyMusicProgramGeneration ion The The Recita RecitalCo lCompo mposer ser Proje Project ct (Pache (Pachet t et al., 1999)is 1999)is basedon basedon a radic radicall allydiffe ydifferen rent t approachto approachto musicselection: musicselection: instead instead of proposing proposing users setsof sets of individual individual titles, we proposetobuildfully-fledgedmusicprograms,i.e.sequencesofmusictitles. There There are several several motivation motivations s forprodu for producing cing music programs, programs, rather rather than unordered unordered collectionsoftitles.Oneissimplybasedontherecognitionthatmusictitlesarerarely listen listened ed to inisolation inisolation: : CDs, CDs, radio radio progr program ams, s, concer concerts ts are all made made upof tempor temporal al sequencesofpieces,ina sequencesofpieces,ina certainorder.Thisorde certainorder.Thisorderismostof rismostof thetimesignifican thetimesignificant,i.e. t,i.e. differentordersdonotproducethesameimpressionsonlisteners.Inaway,thewhole craftof craftof music programselect programselectionisprecisely ionispreciselyto to build coherentsequen coherentsequences,rathe ces,ratherthan rthan simplyselectindividualtitles. Thesecon The second d motivation motivation is thatproperties thatproperties of sequencesplay sequencesplay animpor an importantrole tantrole in the perc percep epti tion on of music music: : for for insta instanc nce, e, seve severa ral l music music title titles s in a simi simila lar r style style conv convey ey a part partic icul ular ar atmo atmosp sphe here re, , and and crea create te expe expect ctat atio ions ns for for the the next next comi coming ng titl titles es. . As a cons conseq eque uenc nce, e, an indi indivi vidu dual al title title may may not not be part partic icul ular arly ly enjo enjoy yed by a list listen ener er in abstracto,butmaybetherightpieceattherighttimewithinasequence. Rather Rather than than focusi focusingon ngon simila similarit rity y ofindividua ofindividual l titles titles, , we can exploi exploit t proper propertie ties s of sequencestosatisfythethreegoalsofmusicselection.Theproposalisthereforethe following.Firstwebuildadatabaseoftitles,withcontentinformationforeachtitle. Then Then we specif specify y music music progr programs ams by giving giving the proper propertie ties s or patter patterns ns we want want the prog progra ram m to have have. . Thes These e prop proper erti ties es are are repr represe esent nted ed as const constra rain ints, ts, in the the sens sense e of constraint constraint satisfacti satisfactiontechnique ontechniques. s.Final Finally,a ly,a constraintsolvercompute constraintsolvercomputes s thesolutions thesolutions ofthecorrespondingcombinatorialpatternge ofthecorrespondingcombinatorialpatterngenerationproblem. nerationproblem. Theproblem,aswedefineit,isthereforetobuildmusicprograms,seenastemporal sequen sequences ces of titles titles, , in order order to satisf satisfy y the three three goals goals of music music selecti selection on proble problem: m: repetition,surprise,andfullexploitationofthecatalogue.Asanexample,wewilltake amusicprogramforwhichwespecifythedesiredproperties.Inthenextsections,we willfocusontheformatofthedatabaseandthenatureofconstraints. Hereisa“liner-note”likedescriptionofatypicalmusicprogram.Thepropertiesof the the sequ sequen ence ce may may be grou groupe ped d in thre three e cate catego gori ries es: : 1) user user pref prefer eren ence ces, s, 2) glob global al propertiesonthecoherenceofsequences,and3)constraintsontheexploitationofthe 10
François François Pachet, Pachet, Active Active Listenin Listening: g: Whatis in theAir? the Air?Sony Sony CSLinternal Report, Report, 1999. catalogue.Thefollowingexampledescribesamusicprogramcalled“DrivingaCar”, ideallysuitedforlisteningtomusicinacar: Userpreferences
Note Note that that thes these e cons constr trai aint nts s spec specif ify y glob global al prop proper erti ties es of the sequ sequen ence ce, , and and do not not specifythepositionofitemsinthesequence: • • • • • •
Noslow/veryslowtempos(CardinalityConstraint) Atleast30%female-typevoice Atleast30%purelyinstrumentalpieces Atleast40%brass Atmost20%“CountryPop”style Onesongby“HarryConnickJr”.
Constraintsonthecoherenceofthesequence •
•
Stylesoftitlesarecloseto Stylesoftitlesarecloseto theirneighbor theirneighbors(successo s(successorandpredecess randpredecessor).This or).Thisis is to ensuresomesortofcontinuityinthesequence,style-wise. Authorsarealldifferent.
Constraintsontheexploitationofthecatalogue • •
Containstwelvedifferentpieces.ThisistofitonatypicalCDorminidiskformat. Containsatleast5titlesfromthelabel“Epic/SonyMusic”.Thisisatypicalbiasto exploitthecatalogueinaparticularregion.
4.4 Databaseo DatabaseofMu fMusic sicTitle Titles s The The databa database se requir required ed for buildi building ng music music progr programs ams contai contains ns conten content t inform informati ation on neededforspecify neededforspecifyingtheconstra ingtheconstraints.Morepreci ints.Moreprecisely sely,eachitemis ,eachitemis describedby describedbya a set ofattributes,whichtaketheirvalueinapredefinedtaxonomy.Theattributesareof twosorts:technicalattributesandcontentattributes. Technicalattributesincludethenameofthetitle(e.g.“Learntoloveyou”),thename of the the auth author or (e.g (e.g. . “Con “Conni nick ck Harr Harry y Jr.” Jr.”), ), the the dura durati tion on (e.g (e.g. . “279 “279 sec” sec”), ), and and the the recordinglabel(e.g.“Epic/SonyMusic”).Contentattributearetypicalmeta-data:they describemusicalpropertiesofindividualtitles.Theattributesarethefollowing: style (e.g.“JazzCrooner”),typeofvoice(e.g.“muffled”),musicsetup(e.g.“instrumental”), (e.g. “brass “brass”), ”), tempo (e.g (e.g. . “slo “sloww-fa fast st”) ”), , and and othe other r opti option onal al type type of inst instru rume ment nts s (e.g. attributessuchasthetypeofmelody(e.g.“consonant”),orthemainthemeofthelyrics (e.g.“love”). Inthecurrentstateofourproject,thedatabaseiscreatedbyhand,bymusicexperts (includingthethirdauthor).However,itshouldbenotedthat1)someattributescould beextractedautomaticallyfromthesignal,suchasthetempo,seee.g.(Schei beextractedautomatically fromthesignal,suchasthetempo,seee.g.(Scheirer,1998) rer,1998) and2)alltheattributesaresimple, and2)alltheattributesaresimple,in in thesensethattheydonotrequiresoph thesensethattheydonotrequiresophistica isticated ted musicalanalysistobefilled. Animportantaspectofthedatabase Animportantaspectofthedatabaseis is thatthevaluesof thatthevaluesofcont contentattrib entattributesarelinked utesarelinked toeachotherbysimilarityrelations.Thesesimilarityrelationsareusedforspecifying constr constrain aints ts on the contin continuit uity y of the sequen sequence. ce. For For instan instance, ce, the prece precedin ding g exampl example e containsaconstraintonthecontinuityofstyles.Moregenerally,thetaxonomieson 11
François François Pachet, Pachet, Active Active Listenin Listening: g: Whatis in theAir? the Air?Sony Sony CSLinternal Report, Report, 1999. attributesvaluesallowtoestablishlinksofpartialsimilaritybetweenitems,according toaspecificdimensionofmusicalcontent. Someof theserelations aresimple ordering ordering relations. relations. Forinstancetempos cantake can take theirvalueintheorderedlist(fast,fast-slow,slow-fast,slow).Otherattributessuchas ake thei theirr val value in fullll-fledg ledgeed taxo taxono nom mies. ies. The The taxo taxono nom my of sty styles les is style, take partic particula ularly rly worth worth mentio mentionin ning, g, becaus because e it embodi embodies es a globa global l knowle knowledg dge e on music music whichisaclearaddedvalueforthesystem. Variou Various s taxono taxonomie mies s of musica musical l styles styles have have been been design designed, ed, partic particula ularly rly by inter internet net musicretailers,suchasAmazon(1998)orMusicBoulevard(1998).However,these classi classific ficati ations ons are are mainly mainly design designed ed with with a query query-ba -based sed approa approach. ch. For exampl example e the taxonomyofstylesproposedbyAmazonisatree-likeclassificationorientedtoward presentationofitemsinasearch-orientedway.Thistaxonomyembodiesarelationof “gener neraliz alizaatio tion/sp n/spec eciializ alizat atio ion” n” betw betweeen style tyles: s: “Bl “Blues” ues” is more ore genera neral l tha than “Memph “Memphis is Blues” Blues”. . As such, such, itis well well suited suited for naviga navigatin ting g in the catal catalogu ogue e to find find under-spe under-specifi cifieditems. editems. However,it However,it doesnotrepresentsimilarit doesnotrepresentsimilaritiesbetween iesbetween styles,for styles,for instance,similaritiesbetweenstylesthathavecommonorigins,like,say,“Soul-Blues” and“Jazz-Crooner”. Conver Conversel sely, y, we design designed ed a taxono taxonomy my of styles styles repre represen sentin ting g explic explicitl itly y relati relations ons of similaritybetweenstyles.Ourtaxonomyisanon-directedgraphinwhichverticesare stylesandedgesexpresssimilarity.Itcurrentlyincludes120differentstyles,covering mostofwesternmusic.ApartofthegraphisrepresentedinFigure3. Jazz-Swing
...
Jazz-Crooner
Latino-Jazz
Pop-Soul
Soul-Crooner
Soul-Blues
Soul-Jazz
Jazz-Crooner Soul-Funk
...
Country-Crooner Pop-Song
Pop-Rock
...
... CountryPop
WorldReggae Soul-Crooner
PopCalifornia
SoulFunk
Figu Figure re 3. A part part of a taxo taxono nomy my of music musical al style styles. s. Links Links indi indica cate te a simi simila lari rity ty relati relation on bet betwee ween n styles styles. . “Jazz“Jazz-Cro Croone oner” r” is repres represent ented ed as simila similar r with “Soul“SoulBlues”.
4.5 ServicesandInterfaceIssues ServicesandInterfaceIssues Computingmusi Computingmusicprogra cprogramsfromadatabase msfromadatabaseanda anda setofconstraintsisshown setofconstraintsisshowntobe tobea a complex complex combinator combinatorial ial problem. problem. Constraint Constraint satisfaction satisfaction techniques techniques may be used to solveit,asexplainedin(Pachet&al.,1999). The The resul resultin ting g techni technique que can beused tobuild a number number ofservices ofservices relat relatedto edto music music delivery delivery withlarg with large-sc e-scale ale music catalogue catalogues. s. We listhere examples examples of currently currently built applications:automaticCDassembly,aPathBuilderandaBaroquerecitalcomposer. Othe Other r appl applic icat atio ions ns are are envi envisa sage ged d for for setset-to topp-bo boxe xes s serv servic ices es and and dig digital ital audi audio o broadcastingwhichwedonotdetailhereforreasonsofspace. •
SamplerBuilder
Thesimplestapplicationofthistechnologyisasystemtargetedatmusicprofessionals for for buil buildi ding ng musi music c prog progra rams ms (so(so-ca call lled ed samp sample lers rs) ) from from a give given n data databa base se. . In the the 12
François François Pachet, Pachet, Active Active Listenin Listening: g: Whatis in theAir? the Air?Sony Sony CSLinternal Report, Report, 1999. applic applicati ation, on, the user user can specif specify y the constr constrain aints ts using using an interf interface ace, , and launch launch the syste system m on a databa database. se. This This syste system m is aimed aimed at profes professio sional nals s who want want to expres express s explicitly explicitly allthepropertiesofthe allthepropertiesofthedesir desiredprogra edprograms,andthus ms,andthus havefullcontrolon havefullcontrolonall all theconstraints. •
Progressiveprograms
Inthisscheme,theuseronlyspecifiesthestylisticstructureoftheprogram:thegenres of the beginn beginning ing, , middle middle and end. end. This This may be used used for instan instance ce for creati creating ng long long programsforparties,inwhichyouknowinadvancethestructure(e.g.beginwithPop, thenRock,thenSlows,etc.). •
Pathacrossdifferentstyles
Serv Servic ices es dedi dedica cate ted d to aver averag age e end end user users s shou should ld allo allow w them them to expr expres ess s only only thei their r preferences,possiblyusingautomaticprofilingsystems,andcontainpredefined,fixed cons constr trai aint nts s for for the the cohe cohere renc nce e prop proper erti ties es and and cata catalo logu gue e expl exploi oita tatio tion, n, acco accord rdin ing g to predeterminedambiencesorconfigurations.Atypicalconfigurationisapathbetween twotitles.Inthisscheme,theusercanspecifyastartingtitleandanendingtitle.The systemcontainshiddenconstraintsoncontinuityofgenres,andtemposarefixed.For instance,findacontinuouspathbetweenCélineDion’s“Allbymyself”,andMichael Jackson’s“Beatit”(seeFigure4).
Figure4.ThePathBuilderprogram.Theuserchoosesastartingandendingtitle, aswellasadegreeoftightnessbetweensuccessivetitles. •
Specificmusicdomains
The The approa approach ch can be used used to produc produce e music music progra programs ms in specif specific ic styles styles, , by adding adding domain domain specif specific ic constr constrain aints. ts. A proto prototy type pe applic applicati ation on dedic dedicate ated d to Baroqu Baroque e music music implementedinourlaballowstobuildvarious“recitals”inthedomainofBaroque harpsichordmusic.Baroquemusicisagoodexampleofaspecificdomain,because 13
François François Pachet, Pachet, Active Active Listenin Listening: g: Whatis in theAir? the Air?Sony Sony CSLinternal Report, Report, 1999. recit recitals als of Baroqu Baroque e music music (XVII (XVIIth century century) ) follow follow rules rules identified identified by musicolog musicologists ists (Bukofzer,1947),whileallowingagreatdealoffreedomtoperformers.Atypicalrule concerningthestructureofrecitalsisthe“continuityoftempos”betweenconsecutive pieces.Morespecificrulesarealsoused,suchasrulesonthetonality:atthisperiodof musicalhistory,recitalswhereallowedtomodulate-i.e.changetonality-onlyonce. Otherconstraintsconcernthestructureoftherecital(introductorypartwithnecessary piecetypes),aswellasnecessaryalterna piecetypes),aswellasnecessaryalternationofpiecetypes. tionofpiecetypes. The The syste system m allows allows the user user to create create and listen listen to diffe differen rent t music music progr programs ams, , while while ensuringtheconsiste ensuringtheconsistency ncy oftheseprograms,accord oftheseprograms,accordingtotherulesof ingtotherulesof thestructureof thestructureof recitals.Thedatabase recitals.Thedatabasecontainstitle containstitleswithcontentdescr swithcontentdescriptio iptionadaptedtothedomain nadaptedtothedomain. . For instance, instance, attribute attributes s suchthat“type”(e.g. such that“type”(e.g. “Gigue”,“Chaconne “Gigue”,“Chaconne”, ”, etc.), etc.), “tonality “tonality” ” and“density”areaddedtothedatabasefordescribingrelevantaspectsoftitles.The constraintsystemcontainstheconstraintscorrespondingtotherulesdescribedabove. Theresultingsystemallowstoproduceagreatnumberofdifferentrecitals,whichall havethedesiredpropertiesof“good”recitals,inthestyleofthecomposer’stime(see Figure5).
Figur Figure e 5. The interfa interface ce for BaroqueComposer. The user user can can sele select ct a Baro Baroqu que e comp compos oser er, , and and the then n a corr corres espo pond ndin ing g cata catalo logu gue e of piec pieces es of Baro Baroqu que e music music. . He/s He /she he can can then then bu buil ild d musi music c prog progra rams ms whic which h sati satisf sfy y the the cons constr trai aint nts s of the the Baroquestyle,andlistentothemintherightorder.
Thiskindofserviceliesbetweentwoextremebounds:fixedorderandrandomness. Ontheonehand,aCDplayedinastandardfashioncontainsafixedmusicprogram. Ontheotherhand,acommonfeatureofCDplayer Ontheotherhand,acommonfeatu reofCDplayers(orJukeboxes)isthe“random” s(orJukeboxes)isthe“random” select selection ion button button, , which which choose chooses s at random random betwee between n diffe differen rent t CDsand betwee between n the titles titles of the CDs. CDs. Constr Constrain aint t techni technique ques s provid provide e an interm intermedi ediar ary y degree degree of contro control l between between these twoextre two extremes,where mes,where the usercan stillexpress somepreferences,but somepreferences,but thesystemcomputesaprogramwhichy thesystemcomputesapr ogramwhichyieldspropertiesofcoherenc ieldspropertiesofcoherence. e.
14
François François Pachet, Pachet, Active Active Listenin Listening: g: Whatis in theAir? the Air?Sony Sony CSLinternal Report, Report, 1999. 4.6 Exploratio ExplorationSpa nSpace ce enabling ng techno technolog logy y for buildi building ng high-l high-leve evel l music music delive delivery ry RecitalComposer is an enabli servicesexploitinglarge-scalemusiccatalogues.Thesystemisbasedontheideaof creatingexplicitsequencesofitems,specifiedbytheirglobalproperties,ratherthan comput computing ing sets sets of items items satisf satisfyi ying ng querie queries. s. One ofits main main advant advantag ages es over over query query-basedorcollaborativefilteringapproachesisthatitproduces“readyforuse”ordered sequen sequences ces of items, items, which which satisf satisfy y the three three goals goals ofmusic select selection ion, , i.e. i.e. repeti repetitio tion, n, surprise, surprise, and exploitatio exploitation n of catalogue catalogues. s. It creates creates “coherent” “coherent” musicprog music programs rams from userspecifications,wherethecoherenceisspecifiedintermsofmeta-dataonmusic titles titles and assuch can beseenas anothe another r exampl example e of“semanti of“semantic”contr c”control, ol, where where the semanticsisthestructu semanticsisthestructureofmusicprog reofmusicprogram rams.Compare s.Comparedtothejukeboxofthe20s dtothejukeboxofthe20s,it ,it allow llows s to acce ccess much much lar larger musi usic cata catallogue ogues s with ith simp simple le con control trols s (e.. e.. user user preferences)which,onceagain,makesense,withoutrequiringan apriori knowledge oftheunderlyingmusiccatalogue. 5. Conclu Conclusion sion
The The new landsc landscape ape of digita digital l multim multimedi edia a opens opens new doors doors for intera interacti ctive ve listen listening ing environme environments nts whichprovi which provide de richer richer musical musical experience experiences. s. We have argued argued that such environmentsrequiresomesortofsemanticpreservingsystems.Wehaveillustrated this this idea idea with with two projec projects ts curren currently tly develo developed ped at SonyCSL,in SonyCSL,in the areas areas ofsound spatia spatializ lizati ation, on, and conten content-b t-base ased d music music select selection ion. . In both both cases, cases, the techno technolog logy y of cons constr trai aint nts s is prop propos osed ed for for repr repres esen enti ting ng thes these e “see “seeds ds of sema semant ntic ics” s”, , that that yield ield exploratio explorationspaceswithmeanin nspaceswithmeaningful gfulcontrols. controls.A Alotremai lotremainstobedone,inotherarea nstobedone,inotherareas s of musi music c list listen enin ing g and and perc percep epti tion on, , but but thes these e proj projec ects ts alre alread ady y sugg sugges est t that that the the tradit tradition ional al border borders s betwee between n compos compositi ition, on, produc productio tion n and listen listening ing may have have to be redefined.Inparticular,aquestionwhicharisesiswhatkindofmusiccomposerswill make,iftheyknowthatlistenershaveactivelisteningdevicesathome?Ifweknow whattechnologyisintheair,whatmusicwillbeinourears?
References AmazonMusicStorewebsite,http://www.amazon.com,1998. Assa ssayag G., Agon Agon C., C., Fineb ineber erg g, J., J., Hana Hanapp ppe e P., P., “An “An Obj Object Orie Orient nted ed Visua isual l Envi Enviro ronm nmen ent t For For Musi Musica cal l Comp Compos osit itio ion” n”, , Proc Procee eedi ding ngs s of the the Inte Intern rnat atio iona nal l ComputerMusicConference,pp.364-367,Thessaloniki,1997. Bukofzer,M. MusicintheBaroqueEra,fromMonteverditoBach MusicintheBaroqueEra,fromMonteverditoBach,W.W.Norton& Company,1947. Cava Cavall llii-Sf Sfor orza za, , L. and and Feldm eldman an, , M. Cult Cultur ural al Tran Transm smis isssion ion and and Evol Evolut utio ion: n: a QuantitativeApproach,PrincetonUniversityPress,1981. Cruz-Neira,C.,Leight,J.,Papka,M.,Barnes,C.,Cohen,S.M.,Das,S.,Engelmann, R., Hudson Hudson, , R., Roy, Roy, T., Siegel Siegel, , L., L., Vasila Vasilakis kis, , C., DeFant DeFanti, i, T.A., T.A., Sandin Sandin, , D.J., D.J., “Scientistsin “Scientistsin Wonderland: Wonderland: a Report Report on Visualizati Visualization on Applicati Applications ons in the CAVE Virtual Virtual Reality Reality Environme Environment”, nt”, Proc. IEEE IEEE Symp. Symp. on Research Research Frontier Frontiers s in V.R., pp.59-66,1993. Eco,U.OperaAperta.Bompiani(Milan),1962.
15
François François Pachet, Pachet, Active Active Listenin Listening: g: Whatis in theAir? the Air?Sony Sony CSLinternal Report, Report, 1999. EckelG.,“ExploringMusicalSpacebyMeansofVirtualArchitecture”, Proceedings th ,SchooloftheArtInstituteof ofthe8 InternationalSymposiumonElectronicArt ,SchooloftheArtInstituteof Chicago,1997. Epstein,JoshuaM.GrowingArtificialSocieties:SocialSciencefromtheBottomUp , MITPress,1996. Fireflywebsite,http://www.firefly.com,1998. Infoglidewebsite,http://www.infoglide.com,1998. Lepain,P. Lepain,P. Ecouteinteract Ecouteinteractivedes ivedesdocu documentsmusica mentsmusicauxnumériqu uxnumériques,in es,in Rechercheset Rechercheset ApplicationsenInformatiqueMusicale ,Chemillier&PachetEds,Hermes,Paris, 1998. Meyer,L. EmotionsandmeaninginMusic,UniversityofChicagoPress,1956. Memon,N,Wong,P.W.“ProtectingDigitalMediaContent”, Communicationsofthe ,July1998,pp.34-43,1998. ACM ,July1998,pp.34-43,1998. MPEG7RequirementsGroup,“MPEG-7RequirementsDocume MPEG7RequirementsGro up,“MPEG-7RequirementsDocument”,Doc.ISO/MPEG nt”,Doc.ISO/MPEG N2461,MPEGAtlanticCityMeeting,October1998. MyLaunchwebsite:www.mylaunch.com,1998. Narmour,E.Theanalysisandcognitionofmelodiccomplexity.UniversityofChicago Press,1992. OrlareyY.,FoberD.,LetzS.,“Elody:aJava+MidiShareBasedMusicComposition Environment”,ProceedingsoftheICMC ,1997. ,1997. Pach Pachet et, , F. Dele Deleru rue, e, O. “A Const Constra rain intt-ba base sed d Temp Tempor oral al Musi Music c Spat Spatia iali lize zer” r”, , ACM MultimediaConference,Brighton(UK),pp.351-360,1998. Pachet Pachet, , F. Roy, Roy, P. Cazaly Cazaly, , D. “ACombinat “ACombinatori orial al Approa Approachto chto Conten Content-b t-base ased d Music Music Selection”, IEEEInternationalConference IEEEInternationalConferenceonMultimediaCom onMultimediaComputingandSystems putingandSystems, Firenze(Italy),1999. Read, Oliver and and Walter Welch. ch. From Tin Foil to Stereo: Evolution of the the Phonograph.Indianapolis,1959,2ndedition1976. Rosen,C.TheFrontiersofMeanin TheFrontiersofMeaning:ThreeInformalLect g:ThreeInformalLecturesonMus uresonMusic ic ,Hill&Wang Pub,1994. Shar hardana danand nd, , U. and and Maes Maes, , P. “Soci Social al Infor nform matio ation n Filte ilterring ing: Algor lgorit ithm hms s for Automa Automatin ting g "Word "Word of Mouth' Mouth'”, ”, Proc Procee eedi ding ngs s of the 1995 1995 ACM ACM Conf Confer eren ence ce on HumanFactorsinComputingSystems,pp.210-217,1995. Sche Scheir irer er, , E. D. (199 (1998) 8). . “Tem “Tempo po and and beat beat anal analy ysis of acou acoust stic ic music musical al sign signal als” s”, , JournaloftheAcousticalSocietyofAmerica,103(1):588-601. Sheire Sheirer, r, E. Väänän Väänänen, en, R. Huopa Huopanie niemi, mi, J. “Audio “AudioBif Biffs: fs: The MPEG-4 MPEG-4 Standa Standard rd for Effects Effects Processing Processing”, ”, Proc Procee eedi ding ngs s of the the firs first t Digi Digita tal l Audi Audio o Effe Effect cts s Works orksho hop p , Barcelona,pp.159-167,November1998. Smith,D.Melara,R.“Aestheticpreferenceandsyntacticpro Smith,D.Melara,R.“Aestheticprefere nceandsyntacticprototypicality totypicalityinmusic:‘Tis inmusic:‘Tis thegifttobesimple”,Cognition,34,pp.279-298,1990. TaubeH.,“CommonMusic:AMusicCompositionLanguageinCommonLispand CLOS”,ComputerMusicJournal,vol.15,n°2,21-32,1991.
16
