Crossin的编程教室

标题: scrapy post请求使用mata参数传递表单中的参数，报错 [打印本页]

作者: kkw1201 时间: 2018-5-1 14:58
标题: scrapy post请求使用mata参数传递表单中的参数，报错
class QingHaiSpider(scrapy.Spider):
name = "qinghai"
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.146 Safari/537.36', 'Accept': 'application/json, text/javascript, */*; q=0.01'}
results = {"001002":"01", "001001":"02", "001005":"03"}
def start_requests(self):
      url = 'http://111.44.251.34/inteligentsearch/rest/inteligentSearch/getFullTextData'
      for k in self.results:
         formdata1 = {"token":"","pn":0,"rn":10,"sdt":"","edt":"","wd":"","inc_wd":"","exc_wd":"","fields":"title","cnum":"001;002;003;004;005;006;007;008;009;010","sort":"{\"showdate\":\"0\"}","ssort":"title","cl":200,"terminal":"","condition":[{"fieldName":"categorynum","isLike":True,"likeType":2,"equal":k}],"time":None,"highlights":"title","statistics":None,"unionCondition":None,"accuracy":"100","noParticiple":"0","searchRange":None,"isBusiness":1}
         formdata = json.dumps(formdata1)
         request = scrapy.Request(url=url, method='POST', body=formdata ,headers=self.headers, callback=self.parse, dont_filter=False, meta={'key':k})

         yield request

def parse(self, response):
      w = response.meta['key']
      max_counts = json.loads(response.text)["result"]["totalcount"]
      datas = json.loads(response.text)["result"]["records"]
      for data in datas:
         item = QhItem()
         item["title"] = data["title"]
         item["date"] = data["showdate"]
         item["detail_url"] = "http://111.44.251.34" + data["linkurl"]
         item["area_code"] = "QINGHAI"
         item["publish_id"] = "181818"
         item["thing_id"] = "42"
         if "001002" in item["detail_url"]:
            item["content_type"] = "01"

         elif "001001" in item["detail_url"]:
            item["content_type"] = "02"
         else:
            item["content_type"] = "03"

         yield item

      for i in range(10, max_counts + 10, 10):

         formdata2 = {"token":"","pn":i,"rn":10,"sdt":"","edt":"","wd":"","inc_wd":"","exc_wd":"","fields":"title","cnum":"001;002;003;004;005;006;007;008;009;010","sort":"{\"showdate\":\"0\"}","ssort":"title","cl":200,"terminal":"","condition":[{"fieldName":"categorynum","isLike":True,"likeType":2,"equal":w}],"time":None,"highlights":"title","statistics":None,"unionCondition":None,"accuracy":"100","noParticiple":"0","searchRange":None,"isBusiness":1}
         formdata = json.dumps(formdata2)
         url = 'http://111.44.251.34/inteligentsearch/rest/inteligentSearch/getFullTextData'
         yield scrapy.Request(url=url, method='POST', body=formdata ,headers=self.headers, callback=self.parse, dont_filter=False)

作者: crossin先生 时间: 2018-5-1 15:57
报错信息呢？

作者: kkw1201 时间: 2018-5-1 16:25

crossin先生发表于 2018-5-1 15:57
报错信息呢？

KeyError: 'key',这个报错很容易看懂，字典中实际是有的。

作者: kkw1201 时间: 2018-5-1 16:29

crossin先生发表于 2018-5-1 15:57
报错信息呢？

crossin老师，其实我想问的就是post请求，因为url地址不变，如果增量爬取，那就是post表单中数据变化，比如我这个的Post请求表单中"pn"表示页数，0是第一页，10是第二页，20是第三页.....那我遍历递增到最大页数，这样写是没问题的呀？

作者: crossin先生 时间: 2018-5-1 23:50

kkw1201 发表于 2018-5-1 16:29
crossin老师，其实我想问的就是post请求，因为url地址不变，如果增量爬取，那就是post表单中数据变化，比 ...

你确认对方服务器接受的就是这种参数形式那就没问题

作者: kkw1201 时间: 2018-5-3 09:55

crossin先生发表于 2018-5-1 23:50
你确认对方服务器接受的就是这种参数形式那就没问题

恩，但是现在的问题是爬取重复，而且爬虫好像停不下来，一直在爬。不知道代码哪里不对

作者: crossin先生 时间: 2018-5-3 12:30

kkw1201 发表于 2018-5-3 09:55
恩，但是现在的问题是爬取重复，而且爬虫好像停不下来，一直在爬。不知道代码哪里不对 ...

那这是你自己代码逻辑，你要自己调试
scrapy本身不会帮你自动去重

作者: kkw1201 时间: 2018-5-3 14:19

crossin先生发表于 2018-5-3 12:30
那这是你自己代码逻辑，你要自己调试
scrapy本身不会帮你自动去重

emmmm.我自己debug好像没啥问题。。

作者: crossin先生 时间: 2018-5-3 23:19

kkw1201 发表于 2018-5-3 14:19
emmmm.我自己debug好像没啥问题。。

遇到重复的就跳过

欢迎光临 Crossin的编程教室 (https://bbs.crossincode.com/)