使⽤python抓取App数据
使⽤python抓取App数据
本⽂链接：
App中的数据可以⽤⽹络爬⾍抓取么
答案是完全肯定的：凡是可以看到的APP数据都可以抓取。
下⾯我就介绍下⾃⼰的学习经验和⼀些⽅法吧本篇适合有过web爬⾍基础的程序猿看
没有的的话学的可能会吃⼒⼀些
App接⼝爬取数据过程
使⽤抓包⼯具
⼿机使⽤代理，app所有请求通过抓包⼯具
获得接⼝，分析接⼝
反编译apk获取key
突破反爬限制
需要的⼯具：
Pycharm
实现过程
⾸先下载夜神模拟器模拟⼿机也可以⽤真机，然后下载Fiddler抓取⼿机APP数据包，分析接⼝完成以后使⽤Python实现爬⾍程序
Fiddler安装配置过程
第⼀步：下载神器Fiddler
下载完成之后，傻⽠式的安装⼀下！
第⼆步：设置Fiddler
打开Fiddler, Tools-> Fiddler Options (配置完后记得要重启Fiddler)
选中"Decrpt HTTPS traffic", Fiddler就可以截获HTTPS请求
选中"Allow remote computers to connect". 是允许别的机器把HTTP/HTTPS请求发送到Fiddler上来
记住这个端⼝号是:8888
夜神模拟器安装配置过程
>#第⼀步：下载安装
下载完成之后，傻⽠式的安装⼀下！
>#第⼆步：配置桥接实现互通
⾸先将当前⼿机⽹络桥接到本电脑⽹络实现互通
安装完成桥接驱动后配置IP地址，要配成和本机互通的⽹段，配置完成后打开主机cmd终端ping通ok
第三步：配置代理
1. 打开主机cmd
2. 输⼊ipconfig查看本机IP
3. 配置代理
进⼊夜神模拟器–打开设置–打开WLAN
点击修改⽹络–配置代理如下图：
配置完后保存
到这⾥我们就设置好所有的值，下⾯就来测试⼀下，打开⼿机的超级课程表APP
4. 在夜神模拟器上下载你想爬取得App使⽤Fiddler抓包分析api后使⽤python进⾏爬取就可以了
####爬取充电⽹APP实例
爬取部分内容截图：
部分python代码分享：
import requests
import city
import json
import jsonpath
import re
city_list = city.jsons
tags_list = city.Tag
def city_func(city_id):
try:
city = jsonpath.jsonpath(city_list, '$..sub[?(@.code=={})]'.format(int(city_id)))[0]["name"]
except:
city = jsonpath.jsonpath(city_list, '$[?(@.code=={})]'.format(int(city_id)))[0]["name"]
return city
def tags_func(tags_id):
tags_join = []
if tags_id:
for tags in tags_id:
t = jsonpath.jsonpath(tags_list,'$..spotFilterTags[?(@.id=={})]'.format(int(tags)))
tags_join.append(t[0]["title"])
return ('-'.join(tags_join))
def split_n(ags):
return re.sub('\n',' ',ags)
def request(page):
print('开始下载第%d页'%page)
url = 'app-api.chargerlink/spot/searchSpot'
two_url = "app-api.chargerlink/spot/getSpotDetail?spotId={d}"
head = {
"device": "client=android&cityName=%E5%8C%97%E4%BA%AC%E5%B8%82&cityCode=110106&lng=116.32154281224254&device_id=8A261C9D60ACEBDED7CD3706C92DD68E&ver=3.7.7&lat=39.8950241 "appId": "20171010",
"timestamp": "1532342711477",
"signature": "36daaa33e7b0d5d29ac9c64a2ce6c4cf",
"forcecheck": "1",
"Content-Type": "application/x-www-form-urlencoded",
"Content-Length": "68",
"Host": "app-api.chargerlink",
"Connection": "Keep-Alive",
"User-Agent": "okhttp/3.2.0"
}
data = {
"userFilter[operateType]": 2,
"cityCode": 110000,
"sort": 1,
"page": page,
"limit": 10,
}
response = requests.post(url,data=data,headers=head)
#获取数据
data = response.json()
for i in data['data']:
c = []
id = i['id']
name = i["name"] #充电桩名
phone = i["phone"] #⼿机号
num = i['quantity'] #有⼏个充电桩
city = city_func(i["provinceCode"]) #城市
tags =tags_func(i["tags"].split(','))#标签
message = c + [id,name,phone,num,city,tags]
parse_info(two_url.format(d=id),message)
def parse_info(url,message):
#打开⽂件
with open('car.csv','a',encoding='utf-8')as c:
head = {
"device": "client=android&cityName=&cityCode=&lng=116.32154281224254&device_id=8A261C9D60ACEBDED7CD3706C92DD68E&ver=3.7.7&lat=39.895024107858724&network=WIFI&os_version=19"
"TOKEN": "036c8e24266c9089db50899287a99e65dc3bf95f",
"appId": "20171010",
"timestamp": "1532357165598",
"signature": "734ecec249f86193d6e54449ec5e8ff6",
"forcecheck": "1",
"Host": "app-api.chargerlink",
"Connection": "Keep-Alive",
"User-Agent": "okhttp/3.2.0",
}
#发起详情请求
res = (url,headers=head)
price = split_n(jsonpath.jsonpath(json.),'$..chargingFeeDesc')[0]) #价钱
payType = jsonpath.jsonpath(json.),'$..payTypeDesc')[0] #⽀付⽅式
businessTime =split_n(jsonpath.jsonpath(json.),'$..businessTime')[0]) #营业时间
result = (message + [price,payType,businessTime])
r = ','.join([str(i) for i in result])+',\n'
c.write(r)
def get_page():
url = 'app-api.chargerlink/spot/searchSpot'
head = {
"device": "client=android&cityName=%E5%8C%97%E4%BA%AC%E5%B8%82&cityCode=110106&lng=116.32154281224254&device_id=8A261C9D60ACEBDED7CD3706C92DD68E&ver=3.7.7&lat=39.8950241 "appId": "20171010",
"timestamp": "1532342711477",
"signature": "36daaa33e7b0d5d29ac9c64a2ce6c4cf",
"forcecheck": "1",
python在线模拟器"Content-Type": "application/x-www-form-urlencoded",
"Content-Length": "68",
"Host": "app-api.chargerlink",
"Connection": "Keep-Alive",
"User-Agent": "okhttp/3.2.0"
}
data = {
"userFilter[operateType]": 2,
"cityCode": 110000,
"sort": 1,
"page": 1,
"limit": 10,
}
response = requests.post(url, data=data, headers=head)
# 获取数据
data = response.json()
total = (data["pager"]["total"])
page_Size = (data["pager"]["pageSize"])
totalPage = (data['pager']["totalPage"])
print('当前共有{total}个充电桩，每页展⽰{page_Size}个，共{totalPage}页'.format(total=total,page_Size=page_Size,totalPage=totalPage)) if __name__ == '__main__':
get_page()
start = int(input("亲，请输⼊您要获取的开始页:"))
end = int(input("亲，请输⼊您要获取的结束页:"))
for i in range(start,end+1):
request(i)
总结：
app⾥的数据⽐web端更容易抓取，反爬⾍也没拿么强，⼤部分也都是http/https协议，返回的数据类型⼤多数为json
版权声明：本站内容均来自互联网，仅供演示用，请勿用于商业和其他非法用途。如果侵犯了您的权益请与我们联系QQ:729038198，我们将在24小时内删除。
发表评论