Sqoop2 1.99.4 安装
Sqoop 1.99.4 安装
1.安装准备工作：
已经装好的 hadoop 环境是 hadoop-2.5.1 64位
下载的sqoop安装包（注意是hadoop200）
www./dist/sqoop/1.99.4/sqoop-1.99.
2.解压文件到工作目录:
tar -xvf sqoop-1.99.
mv sqoop-1.99.4-bin-hadoop200 /usr/local/software/
3.修改环境变量：
vim /etc/profile
添加如下内容：
#sqoop
export SQOOP_HOME=/usr/local/software/sqoop-1.99.4-bin-hadoop200
export PATH=SQOOPHOME/bin:PATH
export CATALINA_HOME=SQOOPHOME/serverexport
LOGDIR=SQOOP_HOME/logs
保存退出即时生效：
source /etc/profile
4.修改sqoop配置：
vim /usr/local/sqoop-1.99.4-bin-hadoop200/server/conf/sqoop.properties
#修改指向我的hadoop安装目录
mysql下载之后是个文件夹
org.apache.figuration.directory=/usr/local/software/hadoop-2.5.1-64/
#把hadoop目录下的jar包都引进来
vim /usr/local/software/sqoop-1.99.4-bin-hadoop200/server/conf/catalina.properties
common.loader=/usr/local/software/hadoop-2.5.1-64/share/hadoop/common/*.jar,/usr/local/software/hadoop-2.5.1-
64/share/hadoop/common/lib/*.jar,/usr/local/software/hadoop-2.5.1-
64/share/hadoop/hdfs/*.jar,/usr/local/software/hadoop-2.5.1-64/share/hadoop/hdfs/lib/*.jar,/usr/local/software/hadoop-2.5.1-64/share/hadoop/mapreduce/*.jar,/usr/local/software/hadoop-2.5.1-
64/share/hadoop/mapreduce/lib/*.jar,/usr/local/software/hadoop-2.5.1-
64/share/hadoop/tools/*.jar,/usr/local/software/hadoop-2.5.1-64/share/hadoop/tools/lib/*.jar,/usr/local/software/hadoop-2.5.1-64/share/hadoop/yarn/*.jar,/usr/local/software/hadoop-2.5.1-
64/share/hadoop/yarn/lib/*.jar,/usr/local/software/hadoop-2.5.1-64/share/hadoop/httpfs/tomcat/lib/*.jar
或者
在$SQOOP_HOME中建个文件夹例如hadoop_lib，然后将这些jar包cp到此文件夹中，最后将此文件夹路径添加到common.loader属性中，这种方法更加直观些
5.下载mysql驱动包
mysql-connector-java-5.1.16-bin.jar
并放到 /usr/local/software/sqoop-1.99.4-bin-hadoop200/server/lib/ 目录下
[注意：下载的是 mysql-5.6. 只需要把里面的 mysql-connector-java.5.1.34-bin.jar 考出来即可，这是个坑啊]
6.启动/停止sqoop200
/usr/local/software/sqoop-1.99.4-bin-hadoop200/bin/sqoop.sh server start/stop
查看启动日志：
vim usr/local/software/sqoop-1.99.4-bin-hadoop200/server/logs/catalina.out
7.进入客户端交互目录
/usr/local/software/sqoop-1.99.4-bin-hadoop200/bin/sqoop.sh client
Sqoop home directory: /usr/local/software/sqoop-1.99.4-bin-hadoop200
Sqoop Shell: Type 'help' or '\h' for help.
sqoop:000>
为客户端配置服务器：set server --host hadoopMaster --port 12000 --webapp sqoop 【红部分为 本机hostname 主机名】Server is set successfully
sqoop:000> show version --all 查看版本信息
client version:
Sqoop 1.99.4 source revision 2475a76ef70a0660f381c75c3d47d0d24f00b57f
Compiled by gshapira on Sun Nov 16 02:50:00 PST 2014
server version:
Sqoop 1.99.4 source revision 2475a76ef70a0660f381c75c3d47d0d24f00b57f
Compiled by gshapira on Sun Nov 16 02:50:00 PST 2014
API versions:
[v1]
sqoop:000> show connector --all
【注意：以下是 1.99.4版本用的命令，要是 1.99.3以下版本请看最底部链接】
创建数据库连接：
1、创建hadoop连接
sqoop:000> create link --cid 1
Creating link for connector with id 1
Please fill following values to create new link object
Name: Second Link --设置连接名称
Link configuration
HDFS URI: hdfs://nameservice1:8020/ --HDFS访问地址
New link was successfully created with validation status OK and persistent id 2
2、创建mysql连接
sqoop:000> create link --cid 2
Creating link for connector with id 2
Please fill following values to create new link object
Name: First Link --设置连接名称
Link configuration
JDBC Driver Class: sql.jdbc.Driver --驱动
JDBC Connection String: jdbc:mysql://mysql.server/database --JDBC连接
Username: sqoop --数据库用户
Password: ***** --数据库密码
JDBC Connection Properties:
There are currently 0 values in the map:
entry#protocol=tcp --之后按回车
New link was successfully created with validation status OK and persistent id 1
建立job（MySql 到 HDFS）
sqoop:000>show link
+----+--------+-----------+---------+
| Id | Name | Connector | Enabled |
+----+--------+-----------+---------+
| 5 | mysql | 2 | true |
|6 | hadoop | 1 | true |
+----+--------+-----------+---------+
sqoop:000> create job -f 5 -t 6
Creating job for links with from id 1 and to id 6
Please fill following values to create new job object
Name: Sqoopy --设置任务名称
FromJob configuration
Schema name:(Required)sqoop --库名：必填
Table name:(Required)sqoop --表名：必填
Table SQL statement:(Optional) --选填
Table column names:(Optional) --选填
Partition column name:(Optional) id --选填
Null value allowed for the partition column:(Optional) --选填
Boundary query:(Optional) --选填
ToJob configuration
Output format:
0 : TEXT_FILE
1 : SEQUENCE_FILE
Output format:
0 : TEXT_FILE
1 : SEQUENCE_FILE
Choose: 0 --选择文件压缩格式
Compression format:
0 : NONE
1 : DEFAULT
2 : DEFLATE
3 : GZIP
4 : BZIP2
5 : LZO
6 : LZ4
7 : SNAPPY
8 : CUSTOM
Choose: 0 --选择压缩类型
Custom compression format:(Optional) --选填
Output directory:(Required)/root/projects/sqoop --HDFS存储目录（目的地）
Driver Config
Extractors: 2 --提取器
Loaders: 2 --加载器
New job was successfully created with validation status OK and persistent id 1
执行任务用
start job命令去执行这个任务，用--jid来传入任务id
sqoop:000> show job
+----+--------+----------------+--------------+---------+
| Id | Name | From Connector | To Connector | Enabled |
+----+--------+----------------+--------------+---------+
| 1 | tohdfs | 2 | 1 | true |
+----+--------+----------------+--------------+---------+
sqoop:000> start job --jid 1
Submission details
Job ID: 1
Server URL: localhost:12000/sqoop/
Created by: root
Creation date: 2014-11-04 19:43:29 PST
Lastly updated by: root
External ID: job_1412137947693_0001
loudera:8088/proxy/application_1412137947693_0001/
2014-11-04 19:43:29 PST: BOOTING - Progress is not available
建立job（HDFS 到 MySql）
sqoop:000>show link
+----+--------+-----------+---------+
| Id | Name | Connector | Enabled |
+----+--------+-----------+---------+
| 5 | mysql | 2 | true |
|6 | hadoop | 1 | true |
+----+--------+-----------+---------+
sqoop:000> create job -f 6 -t 5
Creating job for links with from id 1 and to id 5
Please fill following values to create new job object
Name: hdfs to mysql --设置 job 名称
From Job configuration
刚刚看到这个错误信息可能不容易往jar包冲突这方面想，出现这个错误的原因是刚刚$SQOOP_HOME/server/conf/catalina.properties 文件common.loader属性中的hadoop2.x lib中的jar包和sqoop server的web项目中的lib中的包冲突了，解决方法可以将$SQOOP_HOME/server/webapps/sqoop/WEB-INF/lib中的log4j-1.2.16.jar删掉就ok了。
问题四：
在start job(如：start job --jid 2)时常见错误：
Exception has occurred during processing command
Exception: org.apache.sqoopmon.SqoopException Message: CLIENT_0001:Server has returned exception
在sqoop客户端设置查看job详情：
set option --name verbose --value true
版权声明：本站内容均来自互联网，仅供演示用，请勿用于商业和其他非法用途。如果侵犯了您的权益请与我们联系QQ:729038198，我们将在24小时内删除。
发表评论