COMP38120: Documents, Services and Data on the Web
Laboratory Exercise 1.3
Author: Cristiano Ruschel Marques Dias
Description The indexing algorithm implemented using the map-reduce architecture allows whomever has access to the output data to make queries taking on account the position of each word on the document, and the amount of occurrences of each word in each document. The features implemented were: ● Case Folding - A context based capitalization algorithm was used to decide when to leave a word capitalized. Essentially, whenever a word is the start of a sentence, it is presupposed that it normally would not be capitalized, and therefore is lower-cased. A lot of thought was given into this, specially if it would be worth to implement a casing algorithm, given that in general not using case-matching gives good enough - and arguably as good as. Since the usage of this does not have a great impact on performance, and according to the opinion viewed on [1], the algorithm uses it, but results are similar without it. ● Punctuation treating - instead of removing all punctuation, we instead trim the punctuation of the words, sinte punctuation in the middle of a word sometimes have meaning. For example, the number 8.5 or in-mapper . To do this we also had to separetely trim the tags for references generated by wikipedia, due to their peculiar form - [ number ] -, so mechanisms to modify the importance of words inside a reference in the document would be easily implemented from this point. ● Stop-Words and Stemming - After the aforementioned steps, stop words - words that do not add information to the text - are removed using the algorithm provided. After this, words are stemmed, also using a provided algorithm. ● In-mapper combining - The map-reduction pattern called “in mapper combining” was implemented. This means that instead of being directly written into the context to be treated by the reducer, for each mapper the key-value pairs are preprocessed, such as to lessen the amount of information that is sent to the reducers and increase overall speed of the run of the map-reduce. It was implemented in a way that the pre combined key-value pairs are written in the context as soon as the mapper finishes or the Map containing them has used too much memory - a constant value can be specified. It is similar to the last implementation found on [2] - though no code was copied.
● Positional indexing - The position of each of each occurrence of each token emitted by the mapper - which is a simplified version of a word resulted by the aforementioned operations - is kept, and propagated in the output, so that queries can take the position of the word in the document into account. ● Flagging of important items - The modifications needed to implement the propagations of the flagging of important items to the output were not made; therefore, even though the verification of this is being made in some points, this information is not sent to the output.
Performance All the operations made have a run-time complexity of O(n) in relation to the length of the input, which guarantees the speed and scalability of the algorithm implemented. The algorithm takes some time to run due to the overheads involved in the map-reduce architecture, though as the input grows, the overhead gets comparatively insignificant. The use of the in-mapper combining pattern helps us avoid bottlenecks, such as the algorithm running slowly due to the excessive - normally costly - memory operations that would be caused by the mapper sending an unnecessary large amount of data to the reducers, which makes the algorithm better scalable. Also the pattern does not overload the memory, and the usage of memory used by the in-map combiner may be changed enables the algorithm to be selectively tuned for different users or situations. The bottlenecks to the algorithm as implemented are the amount of memory on the machine - though it would need a really big input to cause real impact on the performance - and the amount of cores, since those limit the amount of map-reduce operations that can be run parallelly.
Sample Output
Man (Bart_the_Fink.txt.gz, [101, 1950]) Man (Bart_the_Mother.txt.gz, [178, 2268]) Manhattan (Bart_the_Murderer.txt.gz, [492]) Marg (Bart_the_Murderer.txt.gz, [134, 517, 2199]) Marg (Bart_the_Genius.txt.gz, [372, 402]) Marg (Bart_the_Fink.txt.gz, [130, 460, 639, 1978]) Marg (Bart_the_General.txt.gz, [257, 403]) Marg (Bart_the_Lover.txt.gz, [110, 625, 627, 2480]) Mark (Bart_the_Murderer.txt.gz, [1760]) Marri (Bart_the_Murderer.txt.gz, [133, 2198]) Marri (Bart_the_Lover.txt.gz, [109, 1573, 2479]) Marri (Bart_the_Fink.txt.gz, [1379]) Martin (Bart_the_Genius.txt.gz, [349, 466, 1034]) Martyn (Bart_the_Genius.txt.gz, [1257, 1686]) Martyn (Bart_the_Fink.txt.gz, [1461, 1619]) Martyn (Bart_the_Mother.txt.gz, [1681]) Martyn (Bart_the_Murderer.txt.gz, [1492, 1850]) Martyn (Bart_the_Lover.txt.gz, [1864, 2040]) Martyn (Bart_the_General.txt.gz, [860, 1350]) Mason (Bart_the_Lover.txt.gz, [1632]) Massachusett (Bart_the_Mother.txt.gz, [1433]) Masterpiec (Bart_the_Genius.txt.gz, [1986]) Masterpiec (Bart_the_Fink.txt.gz, [1855]) Matt (Bart_the_Fink.txt.gz, [73, 1704]) Matt (Bart_the_Genius.txt.gz, [27, 71, 874, 1722, 1789]) Matt (Bart_the_Lover.txt.gz, [53, 2102]) Matt (Bart_the_Mother.txt.gz, [78, 957, 1966]) Matt (Bart_the_General.txt.gz, [27, 39, 974, 1327]) Matt (Bart_the_Murderer.txt.gz, [78, 1926]) Max (Bart_the_Mother.txt.gz, [153, 2243]) Maximum (Bart_the_Mother.txt.gz, [167, 2257]) Mayor (Bart_the_Mother.txt.gz, [135, 858, 2225]) McClure (Bart_the_Mother.txt.gz, [75, 317, 1529, 1574]) McClure (Bart_the_Fink.txt.gz, [70, 1195]) McClure (Bart_the_Murderer.txt.gz, [59]) Me (Bart_the_Mother.txt.gz, [186, 2276]) Melissa (Bart_the_Genius.txt.gz, [1529]) Melros (Bart_the_Fink.txt.gz, [1371])
BasicInvertedIndex.java
/** * Basic Inverted Index * * This Map Reduce program should build an Inverted Index from a set of files. * Each token (the key) in a given file should reference the file it was found * in. * * The output of the program should look like this: * sometoken [file001, file002, ... ] * * @author Kristian Epps */ package uk.ac.man.cs.comp38120.exercise; import java.io.*; import java.util.*; import java.util.regex.Pattern; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configured; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.*; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.FileSplit; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.commons.cli.CommandLine; import org.apache.commons.cli.CommandLineParser; import org.apache.commons.cli.HelpFormatter; import org.apache.commons.cli.OptionBuilder;
import org.apache.commons.cli.Options; import org.apache.commons.cli.ParseException; import org.apache.hadoop.util.Tool; import org.apache.hadoop.util.ToolRunner; import org.apache.log4j.Logger; import uk.ac.man.cs.comp38120.io.array.ArrayListWritable; import uk.ac.man.cs.comp38120.io.pair.PairOfStringFloat; import uk.ac.man.cs.comp38120.io.pair.PairOfWritables; import uk.ac.man.cs.comp38120.util.XParser; import uk.ac.man.cs.comp38120.ir.StopAnalyser; import uk.ac.man.cs.comp38120.ir.Stemmer; import static java.lang.System.out; public class BasicInvertedIndex extends Configured implements Tool { private static final Logger LOG = Logger .getLogger(BasicInvertedIndex.class); public static class Map extends Mapper
>> { //In-map aggregator array java.util.Map> aggregator; final int MAX_AGGREGATOR_SIZE = 300000; //lazy initialization private java.util.Map> getAggregator() { if(aggregator == null) aggregator = new HashMap>(); return aggregator; } //function that writes into the context all the data on the aggregator array and cleans it private void dump(Context context) throws IOException, InterruptedException { Iterator>> iter; iter = getAggregator().entrySet().iterator(); while(iter.hasNext()) {
java.util.Map.Entry> aux = iter.next(); WORD.set(aux.getKey()); context.write(WORD, new PairOfWritables>(INPUTFILE, aux.getValue())); } aggregator = null; } //flushes the array should it use too much memory private void flush(Context context) throws IOException, InterruptedException { if(getAggregator().size() > MAX_AGGREGATOR_SIZE) { dump(context); } } //adds the given information to be written in the context to the aggregator array private void aggregate(String token, int position, Context context) throws IOException, InterruptedException { if(getAggregator().containsKey(token)) { ArrayListWritable l = getAggregator().get(token); l.add(new IntWritable(position)); getAggregator().put(token, l); } else { ArrayListWritable l = new ArrayListWritable(); l.add(new IntWritable(position)); getAggregator().put(token, l); } flush(context); } // INPUTFILE holds the name of the current file private final static Text INPUTFILE = new Text();
// TOKEN should be set to the current token rather than creating a // new Text object for each one @SuppressWarnings("unused") private final static Text TOKEN = new Text(); // The StopAnalyser class helps remove stop words @SuppressWarnings("unused") private StopAnalyser stopAnalyser = new StopAnalyser(); // The stem method wraps the functionality of the Stemmer // class, which trims extra characters from English words // Please refer to the Stemmer class for more comments @SuppressWarnings("unused") private String stem(String word) { Stemmer s = new Stemmer(); // A char[] word is added to the stemmer with its length, // then stemmed s.add(word.toCharArray(), word.length()); s.stem(); // return the stemmed char[] word as a string return s.toString(); } // This method gets the name of the file the current Mapper is working // on @Override public void setup(Context context) { String inputFilePath = ((FileSplit) context.getInputSplit()).getPath().toString(); String[] pathComponents = inputFilePath.split("/"); INPUTFILE.set(pathComponents[pathComponents.length - 1]); } //leaves upper-cased letters in beginning of sentences private String caseFolding(String text) { String result = new String(text); //for each sentence for(String sentence:text.split("\\. "))
{ for(String word:sentence.split(" ")) { //cleans the word of punctuation String aux = trimPunctuation(word); //gets the first word that was not only punctuation if(aux == null) continue; if(aux.length() <=0) continue; //makes it lower case if(Character.isUpperCase(aux.codePointAt(0))) { //TODO //IF NOT ACRONYM result = result.replace(word, word.toLowerCase()); break; } } } return result; } /// trims punctuation from start and end of string. returns null if string is only punctuation, else returns the trimmed string private String trimPunctuation(String str) { if(str.length() == 0) return null; String punct = new String("!\"#$%&\'*+,-./:;'\\'<=>?@[]^_`{|}~() \t\n\f\r"); //removes punctuation and other symbols from beginning and end of string int i = 0; //removes punctuation from beginning while(i < str.length() && punct.contains(str.substring(i, i+1))) i++;
str = str.substring(i); if(str.length() == 0) return null; int j = str.length() - 1; //removes punctuation from end while(j > 0 && punct.contains(str.substring(j, j+1))) j--; return str.substring(0,j+1); } //removes tags in the form [ number ] , which ocasionally remain after tokenization private String trimTags(String str) { if(str.length() == 0) return null; int i; for(i = 0; i < str.length() && str.codePointAt(i) != '['; i++); if(i==str.length()) return str; int j; for(j = i;j < str.length() && (Character.isDigit(str.codePointAt(j)) || str.codePointAt(j) == ']'); j++); if(i < str.length()) str = str.substring(0,i); if(j > i && j + 1 < str.length()) str+= str.substring(j+1,str.length()); return str; } private final static Text WORD = new Text();
// TODO // This Mapper should read in a line, convert it to a set of tokens // and output each modified token with the position of its occurrence in the document public void map(Object key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); //tokenizes the text after case folding StringTokenizer itr = new StringTokenizer(caseFolding(line));
for(int position=0; itr.hasMoreTokens(); position++) { String str = itr.nextToken(); //trims the tags on the form [number] str = trimTags(str); //trims punctuation str = trimPunctuation(str); //does not add words that became null after being stripped of punctuation if(str == null) continue; //disregards stop words if(StopAnalyser.isStopWord(str)) continue; //stem words str = stem(str);
//combines this output with the other output given by this mapper. //Implements the pattern of In-map combining or In-map agregation aggregate(str, position, context); } //guarantees that no information remains without being forwarded to the reducer dump(context); } }
public static class Reduce extends Reducer>> { private final static Text WORD = new Text(); // TODO // This Reduce Job should take in a key and an iterable of file names // It should convert this iterable to a writable array list and output // it along with the key public void reduce( Text key, Iterable>> values, Context context) throws IOException, InterruptedException { Iterator>> iter = values.iterator(); java.util.Map> combine = new HashMap>(); //for each value given by the mappers while(iter.hasNext()) { PairOfWritables> pair = iter.next(); //concatenates the position arrays for each document for all the tokens that appear on that document if(!combine.containsKey(pair.getLeftElement())) { combine.put(pair.getLeftElement(), pair.getRightElement()); } else { ArrayListWritable auxList = new ArrayListWritable(); auxList.addAll(pair.getRightElement()); auxList.addAll(combine.get(key)); combine.put(pair.getLeftElement(), auxList); } }
Iterator>> iter2 = combine.entrySet().iterator(); //writes the otput while(iter2.hasNext()) { java.util.Map.Entry> entry = iter2.next(); WORD.set(key); context.write(WORD, new PairOfWritables>(entry.getKey(),entry.getValue())); } } } // Lets create an object! :) public BasicInvertedIndex() { } // Variables to hold cmd line args private static final String INPUT = "input"; private static final String OUTPUT = "output"; private static final String NUM_REDUCERS = "numReducers"; @SuppressWarnings({ "static-access" }) public int run(String[] args) throws Exception { // Handle command line args Options options = new Options(); options.addOption(OptionBuilder.withArgName("path").hasArg() .withDescription("input path").create(INPUT)); options.addOption(OptionBuilder.withArgName("path").hasArg() .withDescription("output path").create(OUTPUT)); options.addOption(OptionBuilder.withArgName("num").hasArg() .withDescription("number of reducers").create(NUM_REDUCERS)); CommandLine cmdline = null; CommandLineParser parser = new XParser(true); try {
cmdline = parser.parse(options, args); } catch (ParseException exp) { System.err.println("Error parsing command line: " + exp.getMessage()); System.err.println(cmdline); return -1; } // If we are missing the input or output flag, let the user know if (!cmdline.hasOption(INPUT) || !cmdline.hasOption(OUTPUT)) { System.out.println("args: " + Arrays.toString(args)); HelpFormatter formatter = new HelpFormatter(); formatter.setWidth(120); formatter.printHelp(this.getClass().getName(), options); ToolRunner.printGenericCommandUsage(System.out); return -1; } // Create a new Map Reduce Job Configuration conf = new Configuration(); Job job = new Job(conf); String inputPath = cmdline.getOptionValue(INPUT); String outputPath = cmdline.getOptionValue(OUTPUT); int reduceTasks = cmdline.hasOption(NUM_REDUCERS) ? Integer .parseInt(cmdline.getOptionValue(NUM_REDUCERS)) : 1; // Set the name of the Job and the class it is in job.setJobName("Basic Inverted Index"); job.setJarByClass(BasicInvertedIndex.class); job.setNumReduceTasks(reduceTasks); // Set the Mapper and Reducer class (no need for combiner here) job.setMapperClass(Map.class); job.setReducerClass(Reduce.class); // Set the Output Classes job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(PairOfWritables.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(PairOfWritables.class);
// Set the input and output file paths FileInputFormat.setInputPaths(job, new Path(inputPath)); FileOutputFormat.setOutputPath(job, new Path(outputPath)); // Time the job whilst it is running long startTime = System.currentTimeMillis(); job.waitForCompletion(true); LOG.info("Job Finished in " + (System.currentTimeMillis() - startTime) / 1000.0 + " seconds"); // Returning 0 lets everyone know the job was successful return 0; } public static void main(String[] args) throws Exception { ToolRunner.run(new BasicInvertedIndex(), args); } }
