airflow详细安装过程
现将安装过程及踩过的坑分享给⼤家。
安装airflow
（为了避免对其他程序造成影响，故不想替换掉原有的python2.6.6，此处希望2.6与2.7两个版本共存，⽽且安装的pip、virtualenv等软件，也只希望在python27中存在）
安装独⽴的python2.7，只需要在configure时指定prefix为不同的⽬录即可，这样make install时就会安装到prefix⽬录，⽽不
是/usr/local/bin
2、源码安装
su - root
cd /usr/local/
tar -zxvf Python-2.
mv Python-2.7.11 python27
cd python27
./configure --prefix=/usr/local/python27 #(修改为⾃⼰的路径)
make
make install
3、安装setuptools（需要将setuptools安装到python27下⾯, 服务器不能连接外⽹，故下载源码）
tar zvxf setuptools-23.1.
cd setuptools-23.1.0/
/usr/local/python27/python setup.py install
4、安装pip（需要将pip安装到python27下⾯, 服务器不能连接外⽹，故下载源码）（pypi可设置为⾖瓣的库）
tar zvxf pip-8.1.
cd pip-8.1.2/
/usr/local/python27/python setup.py install
tar zvxf virtualenv-15.0.
cd virtualenv-15.0.2/
/usr/local/python27/python setup.py install
还需在python2.6下安装⼀次，否则在python2.6下创建python2.7的virtualenv时⽆法执⾏
6、由于执⾏virtualenv命令时，需要联⽹，所以还是需要设置代理，这⾥使⽤ccproxy
需要在linux上设置环境变量
export https_x.xxx:808
export http_x.xxx:808
7、使⽤virtualenv⽣成临时环境
virtualenv --pythonp=/usr/local/python27/bin/pythonairflowenv
这样 source airflowenv/bin/activate之后，就是使⽤python2.7的shell了
8、安装mysql，不做赘述
9、使⽤root⽤户安装mysql-devel，yum install mysql-devel
10、安装mysql-python，python官⽹下载MySQL-python-1.2.5.zip，解压缩
source airflowenv/bin/activate
cd MySQL-python-1.2.5
python setup.py install
11、安装gevent
source airflowenv/bin/activate
pip install gevent
12、安装airflow
source airflowenv/bin/activate
export AIRFLOW_HOME=~/airflow （修改为⾃⼰的路径）
pip install airflow
# initialize the database
airflow initdb
13、vi $AIRFLOW_HOME/airflow.cfg⽂件
包括添加mysql的连接，设置executor等，其他参数请根据实际需要调整
executor = LocalExecutormysql下载的zip版本安装步骤
sql_alchemy_conn = mysql://username:password@ip:port/dbname
14、再次执⾏airflowinitdb，此时将在mysql中创建表
15、安装supervisor，使⽤supervisor启动airflow，⼀旦airflow挂掉，supervisor会⾃动重启airflow
source airflowenv/bin/activate
pip install supervisor
编辑f⽂件，指定要启动的程序和⽇志输出路径
[program:airflow_scheduler]
command=/xxx/airflowenv/bin/airflow scheduler
stdout_logfile=/tmp/airflow_scheduler.log
使⽤如下命令启动
supervisord -c /xxx/xxx/f
安装遇到的问题
1、airflowinitdb报错
(airflowenv)root@127.0.0.1:/xxx/xxx/airflowenv/bin$ airflow initdb
Traceback (most recent call last):
File "/xxx/xxx/airflowenv/bin/airflow", line 4, in <module>
from airflow import configuration
File "/xxx/xxx/airflowenv/lib/python2.7/site-packages/airflow/__init__.py",line 31, in <module>
dels import DAG
File "/xxx/xxx/airflowenv/lib/python2.7/site-packages/airflow/models.py",line 56, in <module>
from airflow import settings, utils
File "/xxx/xxx/airflowenv/lib/python2.7/site-packages/airflow/settings.py",line 76, in <module>
engine = create_engine(SQL_ALCHEMY_CONN, **engine_args)
File "/xxx/xxx/airflowenv/lib/python2.7/site-packages/sqlalchemy/engine/__init__.py",line 386, in create_engine    ate(*args, **kwargs)
File "/xxx/xxx/airflowenv/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py",line 75, in create
dbapi = dialect_cls.dbapi(**dbapi_args)
File "/xxx/xxx/airflowenv/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/mysqldb.py",line 92, in dbapi    return __import__('MySQLdb')
ImportError: No module named MySQLdb
缺少mysql-python模块，官⽹下载MySQL-python-1.2.5.zip，解压缩，
cd MySQL-python-1.2.5
python setup.py install
2、安装mysql-python后执⾏airflow initdb报错，
_mysql.c:36:23: error:my_config.h: No such file or directory
_mysql.c:38:19: error:mysql.h: No such file or directory
_mysql.c:39:26: error:mysqld_error.h: No such file or directory
_mysql.c:40:20: error:errmsg.h: No such file or directory
linux缺少mysql-devel包，使⽤yum install mysql-devel，或⼿⼯下载mysql-devel的rpm包，⾃⼰安装
3、执⾏airflow  webserver -p 8080启动webserver报错
Error: class uri 'gevent' invalid ornot found:
[Traceback (most recent call last):
File "/xxx/xxx/airflowenv/lib/python2.7/site-packages/gunicorn/util.py",line 140, in load_class
mod = import_module('.'.join(components))
File "/xxx/xxx/software/python27/lib/python2.7/importlib/__init__.py",line 37, in import_module
__import__(name)
File "/xxx/xxx/airflowenv/lib/python2.7/site-packages/gunicorn/workers/ggevent.py",line 22, in <module>  raise RuntimeError("You need gevent installed to use thisworker.")
RuntimeError: You need geventinstalled to use this worker.
]
使⽤pip命令安装gevent pip install gevent

版权声明：本站内容均来自互联网，仅供演示用，请勿用于商业和其他非法用途。如果侵犯了您的权益请与我们联系QQ:729038198，我们将在24小时内删除。