{"id":69,"date":"2012-05-24T10:27:55","date_gmt":"2012-05-24T14:27:55","guid":{"rendered":"http:\/\/lichun.cc\/blog\/?p=69"},"modified":"2012-05-24T10:27:55","modified_gmt":"2012-05-24T14:27:55","slug":"hadoop-genericwritable-sample-usage","status":"publish","type":"post","link":"https:\/\/www.lichun.cc\/blog\/2012\/05\/hadoop-genericwritable-sample-usage\/","title":{"rendered":"Hadoop GenericWritable sample usage"},"content":{"rendered":"

GenericWritable<\/strong> is another Hadoop feature that let you pass values with different types to the reducer, it’s a wrapper for Writable instances.<\/p>\n

Suppose we have different input formats (see MultipleInput<\/a>), one is FirstClass<\/strong> and another one is SecondClass<\/strong>.(note: you can have multiple, not just 2). And you want to include both of them in your reducer based on the same key value<\/strong>, here is what you can do:<\/p>\n

We use the same code used in<\/span> MultipleInput<\/a>.<\/p>\n

<\/p>\n

1. Write a GenericWritable class MyGenericWritable<\/strong><\/p>\n

public class MyGenericWritable extends GenericWritable {\n\n\u00a0\u00a0\u00a0\u00a0private static Class<? extends Writable>[] CLASSES = null;\n\n\u00a0\u00a0 \u00a0static {\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0CLASSES = (Class<? extends Writable>[]) new Class[] {\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0FirstClass.class,\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0SecondClass.class\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \/\/add as many different class as you want\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0};\n\u00a0\u00a0 \u00a0}\n\u00a0\u00a0\u00a0\u00a0\/\/this empty initialize is required by Hadoop\n\u00a0\u00a0 \u00a0public MyGenericWritable() {\n\u00a0\u00a0 \u00a0}\n\n\u00a0\u00a0 \u00a0public MyGenericWritable(Writable instance) {\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0set(instance);\n\u00a0\u00a0 \u00a0}\n\n\u00a0\u00a0 \u00a0@Override\n\u00a0\u00a0 \u00a0protected Class<? extends Writable>[] getTypes() {\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0return CLASSES;\n\u00a0\u00a0 \u00a0}\n\n\u00a0\u00a0\u00a0\u00a0@Override\n\u00a0\u00a0\u00a0 public String toString() {\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 return \"MyGenericWritable [getTypes()=\" + Arrays.toString(getTypes()) + \"]\";\n\u00a0\u00a0\u00a0\u00a0}\n}<\/pre>\n
2. In your Mappers,<\/p>\n
public static class FirstMap extends Mapper<Text, FirstClass, Text, MyGenericWritable> {\n     public void map(Text key, FirstClass value, Context context) throws IOException, InterruptedException {\n         System.out.println(\"FirstMap:\" + key.toString() + \" \" + value.toString());\n         context.write(key, new MyGenericWritable(value));\n     }\n}<\/pre>\n
public static class SecondMap extends Mapper<Text, SecondClass, Text, MyGenericWritable> {\n     public void map(Text key, SecondClass value, Context context) throws IOException, InterruptedException {\n         System.out.println(\"FirstMap:\" + key.toString() + \" \" + value.toString());\n         context.write(key, new MyGenericWritable(value));\n     }\n}<\/pre>\n
3. In your Reducer, use it like the following:<\/p>\n
public class Reduce extends Reducer<Text, MyGenericWritable, Text, Text> {\n\u00a0\u00a0 \u00a0public void reduce(Text key, Iterable<MyGenericWritable> values, Context context) throws IOException, InterruptedException {\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0for (MyGenericWritable value : values) {\n            Writable rawValue = value.get();\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0if(rawValue instanceof FirstClass){\n                FirstClass firstClass = (FirstClass)rawValue;\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\/\/do something\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0}\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0if(rawValue instanceof SecondClass){\n                SecondClass firstClass = (SecondClass)rawValue;\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\/\/do something\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0}\n\u00a0\u00a0 \u00a0\u00a0\u00a0 \u00a0}\n\u00a0\u00a0 \u00a0}\n}<\/pre>\n
3. In your job configuration, change the map output value class to MyGenericWritable<\/strong><\/p>\n
job.setMapOutputValueClass(MyGenericWritable.class);<\/pre>\n
pretty simple right?<\/p>\n","protected":false},"excerpt":{"rendered":"
GenericWritable is another Hadoop feature that let you pass values with different types to the reducer, it’s a wrapper for Writable instances. Suppose we have different input formats (see MultipleInput), one is FirstClass and another one is SecondClass.(note: you can have multiple, not just 2). And you want to include both of them in your […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[19],"tags":[21,16,83],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p2s9sh-17","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/www.lichun.cc\/blog\/wp-json\/wp\/v2\/posts\/69"}],"collection":[{"href":"https:\/\/www.lichun.cc\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.lichun.cc\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.lichun.cc\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.lichun.cc\/blog\/wp-json\/wp\/v2\/comments?post=69"}],"version-history":[{"count":0,"href":"https:\/\/www.lichun.cc\/blog\/wp-json\/wp\/v2\/posts\/69\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.lichun.cc\/blog\/wp-json\/wp\/v2\/media?parent=69"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.lichun.cc\/blog\/wp-json\/wp\/v2\/categories?post=69"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.lichun.cc\/blog\/wp-json\/wp\/v2\/tags?post=69"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}