JavaSpark读取ElasticSearch数据，并分析直接在ES服务器上获取的数据如下所⽰
使⽤的java代码如下所⽰（可以看到数据都被封装成了tuple）：
package mainPackage;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.api.java.function.VoidFunction;
import org.apache.spark.sql.SparkSession;
import org.elasticsearch.spark.rdd.api.java.JavaEsSpark;
import scala.Tuple2;
import java.util.Map;
public class MainClass {
public static void main(String[] args){
run1();
}
private static void run1(){
session如何设置和读取
SparkConf conf = new SparkConf();
conf.setMaster("local[1]");
conf.setAppName("SPARK ES");
conf.set("es.ate", "true");
conf.set("es.nodes", "roc-3");
conf.set("es.port", "8086");
JavaSparkContext sc = new JavaSparkContext(conf);
sc.setLogLevel("ERROR");
JavaPairRDD<String, Map<String, Object>> esRDD =
JavaEsSpark.esRDD(sc, "my-index4/person");
for(Tuple2 llect()){
System.out.print(tuple._1()+"----------");
System.out.println(tuple._2());
}
}
}
运⾏的语句如下：
spark2-submit --master local[*] --packages org.elasticsearch:elasticsearch-hadoop:6.4.0 --class mainPackage.MainClass --repositories maven.aliyun
如果读取的数据⼗分巨⼤，可以⽤下⾯的执⾏语句
spark2-submit --master local[*] --executor-memory 16G --driver-memory 32G --packages org.elasticsearch:elasticsearch-hadoop:6.4.0 --class mainPack
设置repository可以使⽤指定的资源库下载jar
最终结果数据
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
18/09/07 19:19:21 INFO SparkContext: Running Spark version 2.0.0
……
省略N⾏
……
18/09/07 19:19:23 INFO MemoryStore: MemoryStore started with capacity 893.1 MB
18/09/07 19:19:23 INFO SparkEnv: Registering OutputCommitCoordinator
18/09/07 19:19:23 INFO Utils: Successfully started service 'SparkUI' on port 4040.
18/09/07 19:19:23 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at 10.116.16.141:4040
18/09/07 19:19:23 INFO Executor: Starting executor ID driver on host localhost
18/09/07 19:19:23 INFO Utils: Successfully started service 'org.apache.sparkworkty.NettyBlockTransferService' on port 10681.
18/09/07 19:19:23 INFO NettyBlockTransferService: Server created on 10.116.16.141:10681
18/09/07 19:19:23 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 10.116.16.141, 10681)
18/09/07 19:19:23 INFO BlockManagerMasterEndpoint: Registering block manager 10.116.16.141:10681 with 893.1 MB RAM, BlockManagerId(driver, 10.116.16 18/09/07 19:19:23 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 10.116.16.141, 10681)
1----------{desc=数据库管理, user=张三, title=⼯程师}
7----------{desc=数据库管理, user=张5, title=⼯程师}
13----------{desc=数据库管理, user=张11, title=⼯程师}
16----------{desc=数据库管理, user=张14, title=⼯程师}
18----------{desc=数据库管理, user=张16, title=⼯程师}
2----------{desc=数据库管理, user=张0, title=⼯程师}
4----------{desc=数据库管理, user=张2, title=⼯程师}
6----------{desc=数据库管理, user=张4, title=⼯程师}
15----------{desc=数据库管理, user=张13, title=⼯程师}
20----------{desc=数据库管理, user=张18, title=⼯程师}
14----------{desc=数据库管理, user=张12, title=⼯程师}
19----------{desc=数据库管理, user=张17, title=⼯程师}
5----------{desc=数据库管理, user=张3, title=⼯程师}
8----------{desc=数据库管理, user=张6, title=⼯程师}
9----------{desc=数据库管理, user=张7, title=⼯程师}
10----------{desc=数据库管理, user=张8, title=⼯程师}
12----------{desc=数据库管理, user=张10, title=⼯程师}
21----------{desc=数据库管理, user=张19, title=⼯程师}
3----------{desc=数据库管理, user=张1, title=⼯程师}
11----------{desc=数据库管理, user=张9, title=⼯程师}
17----------{desc=数据库管理, user=张15, title=⼯程师}
Process finished with exit code 0
⼀些bug，配置版本和jackon时候需要注意，如果配置成如下就会出问题：
2.4.4。
⽽⽤这个版本就没事⼉
默认es6.4.0版本会使⽤2.6.5版本的jackson来解析json数据，这样就OK
实验证明6.4.0版本es连接6.3.1取数不会遇到版本不兼容问题
最终成功的maven⽂件配置：
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="/POM/4.0.0"
xmlns:xsi="/2001/XMLSchema-instance"
xsi:schemaLocation="/POM/4.0.0 /xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion>
<groupId>SparkReadES</groupId>
<artifactId>AID</artifactId>
<version>1.0-SNAPSHOT</version>
<repositories>
<repository>
<id>central</id>
<url>maven.aliyun/nexus/content/groups/public/</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
</repositories>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<mavenpiler.source>1.7</mavenpiler.source>
<mavenpiler.target>1.7</mavenpiler.target>
<spark.version>2.0.0</spark.version>
<elasticsearch.version>6.4.0</elasticsearch.version>
</properties>
<dependencies>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch-spark-20_2.11</artifactId>
<version>${elasticsearch.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-yarn_2.11</artifactId>
<version>${spark.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>${spark.version}</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.6.0</version>
<configuration>
<encoding>utf-8</encoding><!-- 指定项⽬源⽂件编码 -->
<source>${mavenpiler.source}</source><!-- 指定项⽬源⽂件jdk版本 -->
<target>${mavenpiler.target}</target><!-- 指定项⽬⽣成⽬标⽂件的jdk版本 --> </configuration>
</plugin>
</plugins>
</build>
</project>
Done
版权声明：本站内容均来自互联网，仅供演示用，请勿用于商业和其他非法用途。如果侵犯了您的权益请与我们联系QQ:729038198，我们将在24小时内删除。
发表评论