Just like MultipleInputs, Hadoop also supports MultipleOutputs, thanks to the equality, we can output different data\/format in the same MapReduce job.<\/p>\n
It’s very easy to use this useful feature, as before, I will mainly use Java code to demonstrate the usage, hope the code can explain itself \ud83d\ude42<\/p>\n
Note: I wrote and ran the following code using Hadoop 1.0.3, but it should be working in 0.20.205 as well<\/b><\/p>\n
<\/p>\n
First of all, import the MultipleOutputs, There are 5 parameters for this method:<\/p>\n What I tried to do here is to separate the columns for a given input, different columns go to different output.<\/p>\n Sample Data:<\/p>\n 1 APPLE RED Here I want to separate the fruit<\/b> column and the color<\/b> column.<\/p>\n The fruitOutputName<\/b> and colorOutputName<\/b> are string I defined, they are “fruit” and “color” respectively, so for fruit output, the file name will be fruit-r-000XX<\/b>.<\/p>\n The next important part is the reducer. For single output, we use Please pay attention to the setup<\/b> and cleanup<\/b> function, there will be error if you didn’t initialize or close the MultipleOutputs object.<\/p>\n As we expected, the output of the sample inputs will be:<\/p>\n
\nimport org.apache.hadoop.mapreduce.lib.output.MultipleOutputs;<\/code><\/p>\n2. introduce the
MultipleOutputs.addNamedOutput<\/code><\/h1>\nJob job<\/b>\r\n pass the haddop Job created\r\nString namedOutput<\/b>\r\n give a unique name for this output, the output for\r\n this one will be nameOutput-r-XXXXX\r\nClass<? extends OutputFormat> outputFormatClass<\/b> \r\n If you have a custom output format, pass the output \r\n format in, if you just output text format, \r\n use the hadoop TextOutputFormat.class\r\nClass<?> keyClass<\/b> \r\n the class type of the key, if you don't output key, \r\n use NullWritable.class<\/b>\r\nClass<?> valueClass<\/b>\r\n the class type of the value, if you have a custom \r\n value class, use it here, if the value is text \r\n format, use Text.class<\/pre>\n
3. Codes<\/h1>\n
\n2 ORANGE BLACK
\n3 BANANA GREEN<\/p>\n3.1 Setup the driver for this MapReduce job:<\/h2>\n
public static void main(String[] args) throws IOException, InterruptedException, ClassNotFoundException {\r\n Path inputDir = new Path(args[0]);\r\n Path outputDir = new Path(args[1]);\r\n\r\n Configuration conf = new Configuration();\r\n\r\n Job job = new Job(conf);\r\n job.setJarByClass(MultipleOutputsTest.class);\r\n job.setJobName(\"MultipleOutputs Test\");\r\n\r\n job.setMapOutputKeyClass(Text.class);\r\n job.setMapOutputValueClass(Text.class);\r\n\r\n job.setMapperClass(myMapper.class);\r\n job.setReducerClass(myReducer.class);\r\n\r\n FileInputFormat.setInputPaths(job, inputDir);\r\n FileOutputFormat.setOutputPath(job, outputDir);\r\n\r\n MultipleOutputs.addNamedOutput(job, fruitOutputName, TextOutputFormat.class, NullWritable.class, Text.class);<\/b>\r\n MultipleOutputs.addNamedOutput(job, colorOutputName, TextOutputFormat.class, NullWritable.class, Text.class);<\/b>\r\n\r\n job.waitForCompletion(true);\r\n}<\/pre>\n3.2 Reducer<\/h2>\n
context.write(KEY, VALUE)<\/code>, but here it’s different.<\/p>\n\r\npublic static class myReducer extends Reducer<Text, Text, Text, Writable> {\r\n MultipleOutputs<Text, Text> mos;\r\n\r\n @override\r\n public void setup(Context context) {\r\n mos = new MultipleOutputs(context);<\/b>\r\n }\r\n\r\n public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {\r\n for (Text value : values) {\r\n String str = value.toString();\r\n String[] items = str.split(\"\\t\");\r\n\r\n mos.write(fruitOutputName, NullWritable.get(), new Text(items[1]));<\/b>\r\n mos.write(colorOutputName, NullWritable.get(), new Text(items[2]));<\/b>\r\n }\r\n }\r\n\r\n @override\r\n protected void cleanup(Context context) throws IOException, InterruptedException {\r\n mos.close();<\/b>\r\n }\r\n}\r\n<\/pre>\n3.3 Output<\/h2>\n
\nfruit-r-00000:
\n APPLE
\n ORANGE
\n BANANA
\ncolor-r-00000:
\n RED
\n BLACK
\n GREEN<\/p>\n