python之初学爬⾍并且将爬回来的数据存为csv⽂件
⼀、开发⼯具：
1. 运⾏环境： python3.7  win10
2. python 第三⽅库： requests (⾃⾏安装）  >>> cmd --->pip install requests, 具体不做介绍)
⼆、检测是否安装成功
在命令⾏中输⼊python，敲击回车，进⼊python环境。
再输⼊以下指令并回车：
  import requests  如果不报错，那⼀般是已经安装好了。
三、request库简介：
四、response属性
五、我们⽤requeses库的个体（）函数访问必应主页20次，打印返回状态，text内容，并且计算text（）属性和content属性返回⽹页内容的长度
代码如下：
import requests
def getHTMLText(url):
try:
for i in range(0,20):                  #访问20次
r = (url, timeout=30)
r.raise_for_status() #如果状态不是200，引发异常
return r.status_,r.content,),t)  ##返回状态，text和content内容，text（）和content（）⽹页的长度
except:
return""
url = "cn.bing/?toHttps=1&redig=731C98468AFA474D85AECB7DB98B95D9"
print(getHTMLText(url))
运⾏结果如下：显⽰结果太多了所以只截⼀部分
六、最后分享⼀下我爬取的2019年中国最好⼤学的排名（这⾥只显⽰排名前⼗的学校了）并且把它保存为csv⽂件
代码如下：
import requests
import csv
import os
import codecs
from bs4 import BeautifulSoup
allUniv = []
def getHTMLText(url):
try:
r = (url, timeout=30)
r.raise_for_status()
except:
return""
def fillUnivList(soup):
data = soup.find_all('tr')
for tr in data:
ltd = tr.find_all('td')
if len(ltd)==0:
continue
singleUniv = []
for td in ltd:
singleUniv.append(td.string)
allUniv.append(singleUniv)
def printUnivList(num):
print("{:^4}{:^10}{:^5}{:^8}{:^10}".format("排名","学校名称","省市","总分","培养规模"))
for i in range(num):
u=allUniv[i]
print("{:^4}{:^10}{:^5}{:^8}{:^10}".format(u[0],u[1],u[2],u[3],u[6]))
'''def write_csv_file(path, head, data):
try:
with open(path, 'w', newline='') as csv_file:
writer = csv.writer(csv_file, dialect='excel')
if head is not None:
python怎么读csv数据writer.writerow(head)
for row in data:
writer.writerow(row)
print("Write a CSV file to path %s Successful." % path)
except Exception as e:
print("Write an CSV file to path: %s, Case: %s" % (path, e))'''
def writercsv(save_road,num,title):
if os.path.isfile(save_road):
with open(save_road,'a',newline='')as f:
csv_write=csv.writer(f,dialect='excel')
for i in range(num):
u=allUniv[i]
csv_write.writerow(u)
else:
with open(save_road,'w',newline='')as f:
csv_write=csv.writer(f,dialect='excel')
csv_write.writerow(title)
for i in range(num):
u=allUniv[i]
csv_write.writerow(u)
title=["排名","学校名称","省市","总分","⽣源质量","培养结果","科研规模","科研质量","顶尖成果","顶尖⼈才","科技服务","产学研究合作","成果转化"] save_road="F:\\python\csvData.csv"
def main():
url = 'www.zuihaodaxue/zuihaodaxuepaiming2019.html'
html = getHTMLText(url)
soup = BeautifulSoup(html, "html.parser")
fillUnivList(soup)
printUnivList(10)
writercsv('F:\\python\csvData.csv',10,title)
main()
代码显⽰结果如下：
打开⽂件:
好了，今天的分享就到这⾥了~~~~~~

版权声明：本站内容均来自互联网，仅供演示用，请勿用于商业和其他非法用途。如果侵犯了您的权益请与我们联系QQ:729038198，我们将在24小时内删除。