Lecture 7 - CS 246h

Stanford CS 246H: Mining Massive Data Sets Hadoop Lab

Stanford CS 246H Winter ‘14

Machine Learning & Hadoop


Peanut BuCer and Chocolate? The Promise of Big Data™ •  Sounds great, but how? • 

•  • 

• 

Tools and toolkits starKng to appear • 

• 

Hadoop talent pool is small ML talent pool is Kny Mahout, Oryx, Alpine, Ayasdi, Skytree, etc.

Summary: Hadoop is hard, and ML is hard 1.  2. 

Lots of people/companies are trying to make it easy Don’t believe anyone who tells you they make it easy


Hadoop & ML: A Brief History 2005 – Taste project started on SourceForge •  2007 – Mahout project started at Apache •  2008 – Taste donated to Mahout •  … Kme passes … •  2012 – Myrrix is launched •  2013 – Cloudera ML project started on Github •  Late 2013 – Oryx project started on Github • 


Hadoop ML Family Tree

Andrew Ng

Taste

Lucene

Mahout Cloudera ML

Myrrix

Oryx


Apache Mahout


What is Mahout? • 

“Scalable machine learning” •  • 

• 

not just Hadoop-‐oriented machine learning not en%rely, that is. Just mostly.

Components •  •  •  •  • 

math library clustering classificaKon decomposiKons recommendaKons

©MapR Technologies 2013


Mahout Math • 

Goals are •  •  •  •  •  • 

• 

basic linear algebra, and staKsKcal sampling, and good clustering, decent speed, extensibility, especially for sparse data

But not •  •  • 

totally badass speed comprehensive set of algorithms opKmizaKon, root finders, quadrature



Caveat Emptor Mahout is a toolkit •  There is a command line interface • 

• 

You can’t always use it

Very oken end up wriKng code •  DocumentaKon is… ahem… scant • 

• 

Best reference is Mahout in AcKon

Varying levels of maturity •  Varying levels of Hadoop support • 


Matrices and Vectors • 

At the core: •  • 

• 

DenseVector, RandomAccessSparseVector DenseMatrix, SparseRowMatrix

Highly composable API m.viewDiagonal().assign(v)!

• 

Important ideas: •  • 

view*, assign and aggregate iteraKon



Assign? View? • 

Why assign? •  •  • 

• 

Why view? •  •  • 

• 

Copying is the major cost for naïve matrix packages In-‐place operaKons criKcal to reasonable performance Many kinds of updates required, so funcKonal style very helpful

In-‐place operaKons oken required for blocks, rows, columns or diagonals With views, we need #assign + #views methods Without views, we need #assign x #views methods

Synergies • 

With both views and assign, many loops become single line



Assign • 

Matrices Matrix Matrix Matrix Matrix Matrix

• 

assign(double value);! assign(double[][] values);! assign(Matrix other);! assign(DoubleFunction f);! assign(Matrix other, DoubleDoubleFunction f);!

Vectors Vector Vector Vector Vector Vector Vector


assign(double value);! assign(double[] values);! assign(Vector other);! assign(DoubleFunction f);! assign(Vector other, DoubleDoubleFunction f);! assign(DoubleDoubleFunction f, double y);!


Views • 

Matrices Matrix Matrix Vector Vector Vector

• 

viewPart(int[] offset, int[] size);! viewPart(int row, int rlen, int col, int clen);! viewRow(int row);! viewColumn(int column);! viewDiagonal();!

Vectors Vector viewPart(int offset, int length);!



Aggregates • 

Matrices double zSum();! Vector aggregateRows(VectorFunction f);! Vector aggregateColumns(VectorFunction f);! double aggregate(DoubleDoubleFunction combiner, DoubleFunction mapper);!

!

• 

Vectors double zSum();! double aggregate(! DoubleDoubleFunction reduce, DoubleFunction map);! double aggregate(Vector other, ! DoubleDoubleFunction aggregator, ! DoubleDoubleFunction combiner);!



Predefined FuncKons • 

Many handy funcKons ABS ACOS ASIN ATAN CEIL COS EXP FLOOR IDENTITY INV LOGARITHM!


LOG2 NEGATE RINT SIGN SIN SQRT SQUARE SIGMOID SIGMOIDGRADIENT TAN

! ! ! ! ! ! ! ! !


Examples A =α

double alpha; a.assign(alpha);

A = αB + β a.assign(b, FuncKons.chain( FuncKons.plus(beta), FuncKons.mult(alpha));



Sparse OpKmizaKons • 

DoubleDoubleFuncKon abstract properKes public public public public public public public public

• 

boolean boolean boolean boolean boolean boolean boolean boolean

isLikeRightPlus();! isLikeLeftMult();! isLikeRightMult();! isLikeMult();! isCommutative();! isAssociative();! isAssociativeAndCommutative();! isDensifying();!

And Vector properKes public public public public public


boolean isDense();! boolean isSequentialAccess();! double getLookupCost();! double getIteratorAdvanceCost();! boolean isAddConstantTime();! Stanford CS 246H Winter ‘14

Examples • 

The trace of a matrix m.viewDiagonal().zSum()!

• 

Set diagonal to zero m.viewDiagonal().assign(0)!

• 

Set diagonal to negaKve of row sums excluding the diagonal Vector diag = m.viewDiagonal().assign(0);! diag.assign(m.rowSums().assign(Functions.MINUS));!



IteraKon • 

Matrices are Iterable in Mahout // compute both row and columns sums in one pass! for (MatrixSlice row: m) {! rSums.set(row.index(), row.zSum());! cSums.assign(row, Functions.PLUS);! }!

• 

Vectors are densely or sparsely iterable double entropy = 0;! for (Vector.Element e: v.iterateNonZero()) {! entropy += e.get() * Math.log(e.get());! }!



Random Sampling • 

Samples from some type public interface Sampler {! T sample();! }! ! public abstract class AbstractSamplerFunction ! extends DoubleFunction ! implements Sampler!

• 

Lots of kinds ChineseRestaurant Empirical IndianBuffet


Missing Multinomial MultiNormal

Normal PoissonSampler Sampler !


Mahout Math Summary • 

Matrices, Vectors •  •  •  • 

• 

FuncKons •  • 

• 

lots built-‐in cooperate with sparse vector opKmizaKons

Sampling •  • 

• 

views in-‐place assignment aggregaKons iteraKons

abstract samplers samplers as funcKons

Other stuff … clustering, SVD



Other Stuff Matrix DecomposiKon •  ClassificaKon •  Clustering •  RecommendaKons • 


Focus: Machine Learning ApplicaKons

Examples

GeneKc

Freq. PaCern Mining

UKliKes Lucene/Vectorizer

ClassificaKon

Clustering

Math Vectors/Matrices/ SVD

Recommenders

CollecKons (primiKves)

Apache Hadoop

See hCp://cwiki.apache.org/confluence/display/MAHOUT/Algorithms ©Lucid ImaginaKon 2010


Prepare Data from Raw content • 

Data Sources: • 

Lucene integraKon • 

• 

Document Vectorizer •  • 

• 

• 

bin/mahout seqdirectory … bin/mahout seq2sparse …

ProgrammaKcally • 

• 

bin/mahout lucenevector …

See the UKls module in Mahout

Database File system

©Lucid ImaginaKon 2010


RecommendaKons Extensive framework for collaboraKve filtering •  Recommenders • 

• 

• 

Online and Offline support • 

• 

User based, Item based, ALS, SlopeOne, SVD, others Offline can uKlize Hadoop

Many different Similarity measures • 

Cosine, LLR, Tanimoto, Pearson, others



Clustering • 

Document level •  • 

• 

Group documents based on a noKon of similarity K-‐Means, Fuzzy K-‐Means, Dirichlet, Canopy, Mean-‐ •  Topic Modeling Shik •  Cluster words across Distance Measures documents to idenKfy •  ManhaCan, Euclidean, topics other •  Latent Dirichlet AllocaKon



CategorizaKon • 

Place new items into predefined categories: • 

• 

Sports, poliKcs, entertainment

Mahout has several implementaKons •  •  •  • 

Naïve Bayes Complementary Naïve Bayes Decision Forests LogisKc Regression (SGD)



Freq. PaCern Mining •  • 

IdenKfy frequently co-‐ occurrent items Useful for: • 

Query RecommendaKons • 

• 

Apple -‐> iPhone, orange, OS X

Related product placement • 

• 

hCp://www.amazon.com

“Beer and Diapers”

Spam DetecKon • 


Yahoo: hCp://www.slideshare.net/ hadoopusergroup/mail-‐ anKspam


EvoluKonary Map-‐Reduce ready fitness funcKons for geneKc programming •  IntegraKon with Watchmaker • 

• 

• 

hCp://watchmaker.uncommons.org/index.php

Problems solved: •  •  • 

Traveling salesman Class discovery Many others



Singular Value DecomposiKon Reduces a big matrix into a much smaller matrix by amplifying the important parts while removing/ reducing the less important parts •  Mahout has fully distributed Lanczos implementaKon • 

/bin/mahout svd -‐Dmapred.input.dir=path/ to/corpus -‐-‐tempDir path/for/svd-‐output -‐-‐rank 300 -‐-‐ numColumns -‐-‐numRows /bin/mahout cleansvd -‐-‐eigenInput path/ for/svd-‐output -‐-‐corpusInput path/to/corpus -‐-‐output path/for/cleanOutput -‐-‐maxError 0.1 -‐-‐minEigenvalue 10.0

• 

hCps://cwiki.apache.org/confluence/display/ MAHOUT/Dimensional+ReducKon



How to: Command Line • 

Most algorithms have a Driver program • 

• 

Prepare the Data • 

• 

Different algorithms require different setup

Run the algorithm •  • 

• 

Shell script in $MAHOUT_HOME/bin helps with most tasks

Single Node Hadoop

Print out the results • 

Several helper classes: • 


LDAPrintTopics, ClusterDumper, etc.


Ugly Demo II -‐ Prep • 

Data Set: Reuters •  • 

• 

hCp://www.daviddlewis.com/resources/testcollecKons/ reuters21578/ Convert to Text via hCp://www.lucenebootcamp.com/lucene-‐boot-‐camp-‐ preclass-‐training/

Convert to Sequence File: bin/mahout seqdirectory –input -‐-‐output -‐-‐ charset UTF-‐8

• 

Convert to Sparse Vector: bin/mahout seq2sparse -‐-‐input /content/reuters/ seqfiles/ -‐-‐norm 2 -‐-‐weight TF -‐-‐output /content/ reuters/seqfiles-‐TF/ -‐-‐minDF 5 -‐-‐maxDFPercent 90



Ugly Demo II: Topic Modeling • 

Latent Dirichlet AllocaKon ./mahout lda -‐-‐input /content/reuters/seqfiles-‐TF/ vectors/ -‐-‐output /content/reuters/seqfiles-‐TF/ lda-‐output -‐-‐numWords 34000 –numTopics 10 ./mahout org.apache.mahout.clustering.lda.LDAPrintTopics -‐-‐input /content/reuters/seqfiles-‐TF/lda-‐output/ state-‐19 -‐-‐dict /content/reuters/seqfiles-‐TF/ dictionary.file-‐0 -‐-‐words 10 -‐-‐output /content/ reuters/seqfiles-‐TF/lda-‐output/topics -‐-‐dictionaryType sequencefile

• 

Good feature reducKon (stopword removal) required



Ugly Demo III: Clustering • 

K-‐Means • 

Same Prep as UD II, except use TFIDF weight

./mahout kmeans -‐-‐input /content/reuters/seqfiles-‐ TFIDF/vectors/part-‐00000 -‐-‐k 15 -‐-‐output /content/ reuters/seqfiles-‐TFIDF/output-‐kmeans -‐-‐clusters / content/reuters/seqfiles-‐TFIDF/output-‐kmeans/clusters

• 

Print out the clusters:

./mahout clusterdump -‐-‐seqFileDir /content/reuters/ seqfiles-‐TFIDF/output-‐kmeans/clusters-‐15/ -‐-‐pointsDir /content/reuters/seqfiles-‐TFIDF/output-‐kmeans/points/ -‐-‐dictionary /content/reuters/seqfiles-‐TFIDF/ dictionary.file-‐0 -‐-‐dictionaryType sequencefile -‐-‐substring 20



Ugly Demo IV: Frequent PaCern Mining • 

Data: hCp://fimi.cs.helsinki.fi/data/

• 

./mahout fpg -‐i /content/freqitemset/ accidents.dat -‐o patterns -‐k 50 -‐method mapreduce -‐g 10 -‐regex [\ ] ./mahout seqdump -‐-‐seqFile patterns/fpgrowth/ part-‐r-‐00000

• 



Cloudera ML


Cloudera ML CollecKon of Java libraries and command-‐line tools •  Goal: make data scienKsts more producKve with CDH • 

•  •  •  • 

Exploratory data analysis Data preparaKon Model fi}ng Model evaluaKon

Apache 2.0 licensed •  Developed on GitHub • 

• 

hCp://github.com/cloudera/ml

37 Stanford CS 246H Winter ‘14

Cloudera ML: Building Blocks • 

Apache Hadoop • 

• 

Apache Hive • 

• 

Easy MapReduce pipelines

Apache Mahout • 

• 

Metadata for structured data in HDFS

Apache Crunch • 

• 

Scalable data storage (HDFS) and processing (MapReduce)

Vector interface

Apache Avro • 

SerializaKon format


Cloudera ML Workflow: Clustering


Cloudera ML: summary • 

client/bin/ml summary -‐-‐input-‐paths kddcup.data_10_percent (HDFS) -‐-‐format text -‐-‐header-‐file examples/kdd99/header.csv (local FS) -‐-‐summary-‐file examples/kdd99/s.json (local FS)


Cloudera ML: summary

HDFS

kddcup. data_10_percent

1. summary

Local FS

41

header.csv


Cloudera ML: summary

HDFS


1. summary

Local FS

42

header.csv

s.json


Cloudera ML: summary • 

s.json •  • 

Categorical features: histogram Numerical features: distribuKon summary


Cloudera ML: normalize • 

client/bin/ml normalize -‐-‐input-‐paths kddcup.data_10_percent (HDFS) -‐-‐format text -‐-‐summary-‐file examples/kdd99/s.json (local FS) -‐-‐transform Z -‐-‐output-‐path kdd99 (HDFS) -‐-‐output-‐type avro -‐-‐id-‐column category -‐-‐compress


Cloudera ML: normalize

HDFS


2. normalize

Local FS

45

header.csv

s.json


Cloudera ML: normalize

HDFS


kdd99/

2. normalize

Local FS

46

header.csv

s.json


Cloudera ML: normalize • 

kdd99/part-‐m-‐0000[0|1].avro • 

Examples (rows) •  •  • 

• 

Part 0: 442,454 vectors Part 1: 51,567 vectors Total: 494,021 vectors

Features (columns) •  • 

Before: 41 fields Aker: 143 fields


Cloudera ML: ksketch • 

client/bin/ml ksketch -‐-‐input-‐paths kdd99 (HDFS) -‐-‐format avro -‐-‐points-‐per-‐iteraKon 500 -‐-‐output-‐file wc.avro (local FS) -‐-‐seed 1729 -‐-‐iteraKons 5 -‐-‐cross-‐folds 2


Cloudera ML: ksketch

HDFS


kdd99/

3. ksketch

Local FS

49

header.csv

s.json


Cloudera ML: ksketch

HDFS


kdd99/

3. ksketch

Local FS

50

header.csv

s.json

wc.avro


Cloudera ML: ksketch • 

wc.avro • 

Examples (rows) •  •  •  • 

• 

2 “folds” of 2501 examples 1 iniKal example 500 examples from each iteraKon (5 iteraKons) Each example has an associated weight

Features (columns) • 

143 features (sKll)


Cloudera ML: kmeans • 

client/bin/ml kmeans -‐-‐input-‐file wc.avro (local FS) -‐-‐centers-‐file centers.avro (local FS) -‐-‐seed 19 -‐-‐clusters 1,10,25,35,45 -‐-‐best-‐of 2 -‐-‐num-‐threads 4 -‐-‐eval-‐stats-‐file kmeans_stats.csv (local FS)


Cloudera ML: kmeans

HDFS


kdd99/

4. kmeans

Local FS

53

header.csv

s.json

wc.avro


Cloudera ML: kmeans

HDFS


kdd99/

4. kmeans

Local FS

header.csv

s.json

wc.avro

centers.avro

kmeans_stats.csv

54


Cloudera ML: kmeans • 

centers.avro •  • 

• 

1 row for each run of k-‐means++ 9 total runs: 1 for k=1, 2 each for k=10, 25, 35, and 45

kmeans_stats.csv • 

Clustering quality scores


Cloudera ML: kassign • 

client/bin/ml kassign -‐-‐input-‐paths kdd99 (HDFS) -‐-‐format avro -‐-‐centers-‐file centers.avro (local FS) -‐-‐center-‐ids 4 -‐-‐output-‐path assigned (HDFS) -‐-‐output-‐type csv


Cloudera ML: kassign

HDFS


kdd99/

5. kassign

Local FS

57

header.csv

s.json

wc.avro

centers.avro


Cloudera ML: kassign

HDFS


kdd99/

assigned/

5. kassign

Local FS

58

header.csv

s.json

wc.avro

centers.avro


Cloudera ML: kassign • 

assigned/part-‐m-‐0000[0|1] • 

Rows •  •  • 

• 

Part 0: 442,454 Part 1: 51,567 Total: 494,021

Columns •  •  •  • 

Point ID (normal/aCack type, in this case) Index in centers.avro Assigned cluster ID Squared distance to nearest cluster


Cloudera ML: sample • 

client/bin/ml sample -‐-‐input-‐paths assigned (HDFS) -‐-‐format text -‐-‐header-‐file examples/kdd99/kassign_header.csv (local FS) -‐-‐weight-‐field squared_distance -‐-‐group-‐fields clustering_id,closest_center_id -‐-‐output-‐type csv -‐-‐size 20 -‐-‐output-‐path extremal (HDFS)


Cloudera ML: sample

HDFS


kdd99/

assigned/

6. sample

Local FS

61

header.csv

s.json

wc.avro

centers.avro

kassign_header.csv


Cloudera ML: sample

HDFS


kdd99/

assigned/

extremal/

6. sample

Local FS

62

header.csv

s.json

wc.avro

centers.avro

kassign_header.csv


Cloudera ML: sample • 

extremal/part-‐r-‐00000 • 

Rows •  • 

• 

Up to 20 examples from each cluster Examples that are furthest from the center of the cluster

Columns •  •  •  • 

Point ID (normal/aCack type, in this case) Index in centers.avro Assigned cluster ID Squared distance to nearest cluster


Oryx


2014: Lab to Factory

65


Data Science Will Be Opera-onal Analy-cs

66


I Built A Model. Now What?

Collect Input

Build Model

Query Model

Repeat

67


I Built A Model On Hadoop. Now What?

? ? ? Collect Input

Build Model

Query Model

Repeat

68


Example: Oryx


www.mwCl.com/wp-‐content/uploads/2013/11/IMG_5446_edited-‐2_mwCl.jpg

70


Gaps to fill, and Goals • 

Model Building •  •  •  • 

• 

Model Serving •  • 

71

Large-‐scale Con-nuous Apache Hadoop™-‐based Few, good algorithms Real-‐-me query Real-‐-me update

• 

Algorithms •  •  • 

• 

Parallelizable Updateable Works on diverse input

Interoperable •  •  • 

PMML model format Simple REST API Open source


Large-‐Scale or Real-‐Time? Large-‐Scale Offline Batch

vs

Real-‐Time Online Streaming

Why Don’t We Have Both?

λ!

72


Lambda Architecture Batch, Stream Processing are different •  Tackle separately in 2+ Layers •  Batch Layer: offline, asynchronous •  Serving / Speed Layer: real-‐Kme, incremental, approximate • 

… λ?

jameskinley.tumblr.com/post/37398560534/the-‐lambda-‐architecture-‐principles-‐for-‐architecKng

73


Serving/Speed

Batch

74


Two Layers • 

ComputaKon Layer •  •  • 

•  • 

Java-‐based server process Client of Hadoop 2.x Periodically builds “generaKon” from recent data and past model Baby-‐sits MapReduce* jobs (or, locally in-‐core) Publishes models

• 

Serving Layer •  •  •  •  •  • 

Apache Tomcat™-‐based server process Consumes models from HDFS (or local FS) Serves queries from model in memory Updates from new input Also writes input to HDFS Replicas for scale

* Apache Spark later

75


CollaboraKve Filtering : ALS AlternaKng Least Squares •  Latent-‐factor model •  Accepts implicit or explicit feedback •  Real-‐Kme update via fold-‐in of input •  No cold-‐start •  Parallelizable • 

76

YT

X


Clustering : k-‐means++ Well-‐known and understood •  Parallelizable •  Clusters updateable • 

cwiki.apache.org/confluence/display/MAHOUT/K-‐Means+Clustering

77


ClassificaKon / Regression : RDF Random Decision Forests •  Ensemble method •  Numeric, categorical features and target •  Very parallel •  Nodes updateable •  Works well on many problems • 

78

age$>$30

female?

income$>$20000

Yes

Yes

Yes

No


PMML PredicKve Modeling Markup Language •  XML-‐based format for predicKve models •  Standardized by Data Mining Group (www.dmg.org) •  Wide tool support • 

!
! ! ! …! ! ! ! ! … ! ! ! ! ! …! ! ! ! !

www.dmg.org/v4-‐1/TreeModel.html

79


HTTP REST API ConvenKon for RPC-‐like request / response •  HTTP verbs, transport •  GET : query •  POST : add input •  Easy from browser, CLI, Java, Python, Scala, etc. • 

GET /recommend/jwills!

HTTP/1.1 200 OK! Content-Type: text/plain! ! "Ray LaMontagne",0.951  "Fleet Foxes",0.7905! "The National",0.688! "Shearwater",0.3017!

80


Wish List • 

Revamp workflow • 

• 

De-‐emphasize model building •  • 

81

Spark / Crunch-‐like API, not raw M/R

Well-‐solved Bring your own

More component-‐ized •  Less black-‐box service •  Emphasize integraKon • 

• 

• 

PMML, etc.

“Pull” opKons •  • 

Ka‡a? Hive / Impala ?


Open Source

github.com/cloudera/oryx! 100% Apache License 2.0

82



Lecture 7 - CS 246h

Recommend Documents