Nutch环境搭建1. 环境准备
HOST：Ubuntu12.04LTS
JDK： jdk-7u45-linux-i586.rpm
Nutch：apache-nutch-1.
Solr：solr-4.
⼯作⽬录： /home/zephyr/
1.1JDK安装
为了下载快点，⼀下脑残下了rpm.
可Ubuntu上没有rpm⼯具apt-get install rpm 后rpm -ivh jdk-7u45-linux-i586.rpm
提⽰要使⽤alien
apt-get install alien alien -i jdk-7u45-linux-i586.rpmlinux安装jdk rpm安装
alien -d jdk-7u45-linux-i586.rpm ⽣成deb⽂件不可⽤
⽆奈，重新使⽤apt-get install oracle-java7-installer* 直接使⽤失败
add-apt-repository ppa:webupd8team/java
apt-get update
apt-get install oracle-java7-installer
到/etc/profile ⽂件的最后增加上以下⼏⾏
export JAVA_HOME=/usr/java/jdk1.7.0_45
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
export NUTCH_HOME=/home/zephyr/apache-nutch-1.7
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$NUTCH_HOME/bin:$PATH
1.2Tomcat安装
apt-get install tomcat7 下载tomcat
修改tomcat的端⼝使⽤情况为
vi /etc/l
<Connector port="8088" protocol="HTTP/1.1"
connectionTimeout="20000"
URIEncoding="UTF-8"
redirectPort="8443" />
/etc/init.d/tomcat7 start
1.3Nutch 安装
⼯作⽬录下解压 tar xzvf apache-nutch-1.
测试Nutch 是否能正常⼯作
在urls⽬录touch⼀个 echo / >>
修改
# accept anything else
+.
为
+^([a-z0-9]*\.)*/
编辑l
<property>
<name>http.agent.name</name>
<value>My Nutch Spider</value>
</property>
.
nutch crawl ../urls -dir ../test -depth 1 -topN 10
报错
java/lang/NoClassDefFoundError: java/lang/Object
Exception in thread "main" java.lang.InternalError: internal error: SHA-1 not available
修正JDK
cd /usr/java/jdk1.7.0_45/lib 将pack结尾的⽂件都转成 jar 如：unpack200 tools.pack tools.jar
/usr/java/jdk1.7.0_45/jre/lib 也做相同操作
再操作成功。
1.4 Solr安装
tar xzvf solr-4.
ivy未安装，先安装ivy, apt-get install ivy
ant compile 失败，修正ln -s -T /usr/share/java/ivy.jar /usr/share/ant/lib
/ivy.jar 编译⽆响应
tar xzvf solr-4.
把Solr 预设的core 范例collection1 复制为core-nutch
把Nutch 提供的l 覆盖掉Solr core-nutch 的l
将Solr core-nutch l补上⼀⾏漏掉的栏位设定<field name=”_version_” type=”long” stored=”true” indexed=”true”multiValued=”false”/>
将core.properties中的name 改为 core-nutch
nutch crawl /home/zephyr/apache-nutch-1.7/urls -solr 192.168.153.128:8983/solr/core-nutch/ -threads 20 -depth 1 -topN 10也可以使⽤nutch solrindex 192.168.153.128:8983/solr/core-nutch/ /home/zephyr/apache-nutch-1.7/test/crawldb -linkdb
/home/zephyr/apache-nutch-1.7/test/linkdb/ /home/zephyr/apache-nutc
h-1.7/test/segments/*
1.5 solr通过tomcat7部署
apt-get install tomcat7-admin
/etc/tomcat7编辑l
<role rolename="manager-gui"/>
<role rolename="admin-gui"/>
<user username="tomcat" password="tomcat" roles="manager-gui,admin-gui"/>
cp /home/zephyr/solr-4.6.0/example/webapps/solr.war /var/lib/tomcat7/webapps/solr.war
cp –r /home/zephyr/solr-4.6.0/example/solr /var/lib/tomcat7/solr
cd /var/lib/tomcat7/conf/Catalina/localhost
l
<Context docBase="/var/lib/tomcat7/webapps/solr.war" debug="0" crossContext="true" >
<Environment name="solr/home" type="java.lang.String" value="/var/lib/tomcat7/solr" override="true" />
</Context>
cp /home/zephyr/solr-4.6.0/example/lib/ext/*.jar /usr/share/tomcat7/lib
cp /home/zephyr/solr-4.6.0/example/resources/log4j.properties /usr/share/tomcat7/lib/
vi log4j.properties 修改 solr.log=/var/logs/tomcat7/
否则solr⽆法在tomcat启动
“One of the things about running Solr in a container other than the embedded Jetty is that we need to setup the logging. This step is crucial, without doing this solr will never start and you will pull all of your hair out trying to figure out why”
启动运⾏提⽰出错：
SolrCore 'collection1' is not available due to init failure: Index locked for write for core collection1
cd /var/lib/tomcat7/solr/ collection1
chown tomcat7 data -R
cd /var/lib/tomcat7/solr/core-nutch
chown tomcat7 data –R
1.6 Nutch添加中⽂⽀持
编辑/home/zephyr/apache-nutch-1.7/l
<property>
<name>http.accept.language</name>
<value>ja-jp, en-us, zh-cn,en-gb,en;q=0.7,*;q=0.3</value>
<description>Value of the “Accept-Language” request header field.
This allows selecting non-English language as default one to retrieve.
It is a useful setting for search engines build for certain national group.
</description>
</property>

版权声明：本站内容均来自互联网，仅供演示用，请勿用于商业和其他非法用途。如果侵犯了您的权益请与我们联系QQ:729038198，我们将在24小时内删除。