Hadoop实验-HDFS与Mapreduce操作
⼀、实验⽬的
1、利⽤虚拟机搭建集部署hadoop
2、HDFS⽂件操作以及⽂件接⼝编程；
3、MAPREDUCE并⾏程序开发、发布与调⽤。
⼆、实验内容
1、虚拟机集搭建部署hadoop
利⽤VMware、centOS-7、Xshell(secureCrt)等软件搭建集部署hadoop，具体操作参照
www.bilibili/video/BV1Kf4y1z7Nw?p=1
2、HDFS⽂件操作
（1）在分布式⽂件系统上验证HDFS⽂件命令
[-ls <path>]  //显⽰⽬标路径当前⽬录下的所有⽂件
[-du <path>]  //以字节为单位显⽰⽬录中所有⽂件的⼤⼩，或该⽂件的⼤⼩（如果path为⽂件）
（2）. HDFS⽂件操作
调⽤HDFS⽂件接⼝实现对分布式⽂件系统中⽂件的访问，如创建、修改、删除等。源代码：
package mapreduce;
import org.f.Configuration;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.log4j.BasicConfigurator;
public class test {
public static void main(String[] args) {
try {
Configuration conf = new Configuration();
FileSystem fs = (conf);
String filename = "hdfs://node01:8020/";
FSDataOutputStream os = fs.create(new Path(filename));
byte[] buff = "hello world!".getBytes();
os.write(buff, 0, buff.length);
System.out.println("Create" + filename);
} catch (Exception e) {
e.printStackTrace();
}
}
}
运⾏结果：
3.MAPREDUCE并⾏程序开发
1）求每年最⾼⽓温
源代码：
package mapreduce;
import java.io.IOException;
import org.f.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
public class Temperature {
static class TempMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
@Override
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { System.out.print("Before Mapper: " + key + ", " + value);
String line = String();
String year = line.substring(0, 4);
int temperature = Integer.parseInt(line.substring(8));
context.write(new Text(year), new IntWritable(temperature));
System.out.println("======" + "After Mapper:" + new Text(year) + ", " + new IntWritable(temperature));
}
}
static class TempReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
@Override
public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
int maxValue = Integer.MIN_VALUE;
StringBuffer sb = new StringBuffer();
for (IntWritable value : values) {
maxValue = Math.max(maxValue, ());
sb.append(value).append(",");
}
System.out.print("Before Reduce: " + key + ", " + sb.toString());
context.write(key, new IntWritable(maxValue));
System.out.println("======" + "After Reduce: " + key + ", " + maxValue);
}
}
hadoop分布式集搭建public static void main(String[] args) throws Exception {
String dst = "hdfs://node01:8020/user/";
String dstOut = "hdfs://node01:8020/user/zzy/output";
Configuration hadoopConfig = new Configuration();
hadoopConfig.set("fs.hdfs.impl", org.apache.hadoop.hdfs.Name()); hadoopConfig.set("fs.file.impl", org.apache.hadoop.fs.Name());
Job job = new Job(hadoopConfig);
// job.setJarByClass(NewMaxTemperature.class);
FileInputFormat.addInputPath(job, new Path(dst));
FileOutputFormat.setOutputPath(job, new Path(dstOut));
job.setMapperClass(TempMapper.class); job.setReducerClass(TempReducer.class); job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class); job.waitForCompletion(true);
System.out.println("Finished");
}
}
运⾏结果：

版权声明：本站内容均来自互联网，仅供演示用，请勿用于商业和其他非法用途。如果侵犯了您的权益请与我们联系QQ:729038198，我们将在24小时内删除。