scrapy创建以及启动项⽬步骤！
1，创建项⽬：scrapy startproject biqukanproject
D:\pythonscrapy>scrapy startproject biqukanproject
New Scrapy project 'biqukanproject', using template directory 'd:\python_install\lib\site-packages\scrapy\templates\project', created in:
D:\pythonscrapy\biqukanproject
You can start your first spider with:
cd biqukanproject
scrapy genspider example example
2，进⼊被创建项⽬，再创建爬⾍！
D:\pythonscrapy>cd biqukanproject
D:\pythonscrapy\biqukanproject>
3，创建爬⾍：scrapy genspider biqukanspider biqukan
D:\pythonscrapy\biqukanproject>scrapy genspider biqukanspider biqukan
Created spider 'biqukanspider' using template 'basic' in module:
biqukanproject.spiders.biqukanspider
4，启动：
启动名字是上⾯图⾥⾯圈起来的！
命令： scrapy crawl biqukanspider
结果：
D:\pythonscrapy\biqukanproject>scrapy crawl biqukanspider
2020-02-28 14:36:39 [scrapy.utils.log] INFO: Scrapy 1.8.0 started (bot: biqukanproject)
2020-02-28 14:36:39 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.10.0, Python 3.8.1 (tags/v
2020-02-28 14:36:39 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.5, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 19.10.0, Python 3.8.1
2020-02-28 14:36:39 [awler] INFO: Overridden settings: {'BOT_NAME': 'biqukanproject', 'NEWSPIDER_MODULE': 'biqukanproject.spiders', 'ROBOTSTX 2020-02-28 14:36:39 [lnet] INFO: Telnet Password: 1de4a9895047eee8
2020-02-28 14:36:40 [scrapy.middleware] INFO: Enabled extensions:
['stats.CoreStats',
'lnet.TelnetConsole',
'sions.logstats.LogStats']
2020-02-28 14:36:41 [scrapy.middleware] INFO: Enabled downloader middlewares:
['botstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'RetryMiddleware',
'direct.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'direct.RedirectMiddleware',
'kies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-02-28 14:36:41 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'ferer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-02-28 14:36:41 [scrapy.middleware] INFO: Enabled item pipelines:
['biqukanproject.pipelines.BiqukanprojectPipeline']
2020-02-28 14:36:41 [ine] INFO: Spider opened
2020-02-28 14:36:41 [sions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-02-28 14:36:41 [lnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-02-28 14:36:42 [] DEBUG: Retrying <GET > (failed 1 times): [<twisted.python.failure.Failur 2020-02-28 14:36:42 [] DEBUG: Retrying <GET > (failed 2 times): [<twisted.python.failure.Failur 2020-02-28 14:36:42 [] DEBUG: Gave up retrying <GET > (failed 3 times): [<twisted.python.failu 2020-02-28 14:36:42 [botstxt] ERROR: Error downloading <GET >: [<twisted.python.failure.Failure Traceback (most recent call last):
File "d:\python_install\lib\site-packages\scrapy\core\downloader\middleware.py", line 44, in process_
request
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure ConnectionLost: Connection to the other side was lost in a 2020-02-28 14:36:43 [] DEBUG: Retrying <GET biqukan/0_790/> (failed 1 times): [<twisted.python.failure.Failure t 2020-02-28 14:36:46 [] DEBUG: Retrying <GET biqukan/0_790/> (failed 2 times): [<twisted.python.failure.Failure t 2020-02-28 14:36:52 [] DEBUG: Gave up retrying <GET biqukan/0_790/> (failed 3 times): [<twisted.python.failure. 2020-02-28 14:36:52 [scraper] ERROR: Error downloading <GET biqukan/0_790/>
Traceback (most recent call last):
File "d:\python_install\lib\site-packages\scrapy\core\downloader\middleware.py", line 44, in process_request
twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure ConnectionLost: Connection to the other side was lost in a 2020-02-28 14:36:52 [ine] INFO: Closing spider (finished)
2020-02-28 14:36:52 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 6,
'downloader/exception_type_count/twisted.web._newclient.ResponseNeverReceived': 6,
'downloader/request_bytes': 1758,python新建项目教程
'downloader/request_count': 6,
'downloader/request_method_count/GET': 6,
'elapsed_time_seconds': 10.784783,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2020, 2, 28, 6, 36, 52, 625867),
'log_count/DEBUG': 6,
'log_count/ERROR': 2,
'log_count/INFO': 10,
'retry/count': 4,
'retry/max_reached': 2,
'retry/reason_count/twisted.web._newclient.ResponseNeverReceived': 4,
"robotstxt/exception_count/<class 'twisted.web._newclient.ResponseNeverReceived'>": 1,
'robotstxt/request_count': 1,
'scheduler/dequeued': 3,
'scheduler/dequeued': 3,
'scheduler/dequeued/memory': 3,
'scheduler/enqueued': 3,
'scheduler/enqueued/memory': 3,
'start_time': datetime.datetime(2020, 2, 28, 6, 36, 41, 841084)}
2020-02-28 14:36:52 [ine] INFO: Spider closed (finished) D:\pythonscrapy\biqukanproject>

版权声明：本站内容均来自互联网，仅供演示用，请勿用于商业和其他非法用途。如果侵犯了您的权益请与我们联系QQ:729038198，我们将在24小时内删除。