遇到了淘票票但反爬，希望能有高手解决

fa2003

参展咱们编程实验室的影票抓取项目(https://github.com/zx576/film_tickets)做了练习。但在实际过程中遇到了问题。早先按照咱们的程序的逻辑是可以抓取淘票票数据的。但最近抓取页面：

https://dianying.taobao.com/cinemaDetailSchedule.htm?showId=&cinemaId=12870&n_s=new

失败了，返回信息为：

{"rgv587_flag":"sm","url":"https://login.taobao.com/member/login.jhtml?style=mini&from=sm&full_redirect=false&redirectURL=https%3a%2f%2fdianying.taobao.com:443/cinemaDetailSchedule.htm/_____tmd_____/punish%3fx5secdata=5e0c8e1365474455070961b803bd560607b52cabf5960afff39b64ce58073f78d9a2b4abf56353114a114a2786bd71ffbd61c9c7de1498153ee27d034dd29417e1aa383c78c023ac77cac15e4e0554d6a92fb4abc87ae99131c8d47b82a43d36ae810c33b03e9e301a8b7313941c3ba83cd2956a5d237d86525648da8f9fed127ce4089340650c86d53a52b948174f9c892c2047f12a7329443a3ce389a6805531d18ce548cb7abb31f72075b03941692dd01723fe318ee0a0c0f83a3a32365cdb0596d471ef4ea741afb4dc1e71b21f9040dc69738eb838d0061923008f5355a27e92c104d0e0362e265bf99befabb0054167327ceb04e0da3c5e48345554c07addcdfbcb8f55ae76a8aa9416aa0c9c5800a16c924e11b7539f43bcc3be5dc7f9b86c27692bf843654469bc1ede3d7d27d9078c80590f25410a9d5e17d2b6f30b88264a7cdd701087faa1ba705b03f94c064a42424fa07de3912d37fe56189e1ef6064a3134649e8032b2082dd627fc0a3a390a97a5ec84fed7c369cae29223773e7c76d7114c9bf98a93ff242a4091e9e825cd94b0de354b3509d7552729d0769c110d480ed312bd5496dbc85534f4557849e7e7aff98381e5bb219996b1ebd32c28776febbda2f29fcfe855ba0745d0d9ac90b61c55a8f0de9e2bda31eb757fd20d5cf9d9649577502bb0ce2c5e04561f9832b286df69b9f4596b5bd932ee7e35e29e895dd5559881ed86fa34fa7b%26x5step=100"}

显然是被他们识别出来了。现在希望能读取到这个页面的内容，同时不被反爬机制发现。看有没有高手能指点迷津。只要能正常读取这个页面的内容即可，不需要解析什么的。

可测试使用的目标URL列表为：
https://dianying.taobao.com/cine ... d=12870&n_s=new
https://dianying.taobao.com/cine ... d=39223&n_s=new
https://dianying.taobao.com/cine ... d=38335&n_s=new
https://dianying.taobao.com/cine ... d=43845&n_s=new
https://dianying.taobao.com/cine ... Id=9823&n_s=new
https://dianying.taobao.com/cine ... Id=5274&n_s=new
https://dianying.taobao.com/cine ... Id=4585&n_s=new
https://dianying.taobao.com/cine ... Id=6030&n_s=new
https://dianying.taobao.com/cine ... d=41618&n_s=new
https://dianying.taobao.com/cine ... d=49053&n_s=new
https://dianying.taobao.com/cine ... d=37559&n_s=new
https://dianying.taobao.com/cine ... d=29654&n_s=new
https://dianying.taobao.com/cine ... d=52825&n_s=new
https://dianying.taobao.com/cine ... d=39279&n_s=new
https://dianying.taobao.com/cine ... d=38135&n_s=new
https://dianying.taobao.com/cine ... d=23800&n_s=new
https://dianying.taobao.com/cine ... d=52857&n_s=new
https://dianying.taobao.com/cine ... d=42750&n_s=new
https://dianying.taobao.com/cine ... d=13349&n_s=new
https://dianying.taobao.com/cine ... d=10195&n_s=new
https://dianying.taobao.com/cine ... d=55065&n_s=new
https://dianying.taobao.com/cine ... d=16927&n_s=new
https://dianying.taobao.com/cine ... d=48854&n_s=new
https://dianying.taobao.com/cine ... d=23389&n_s=new
https://dianying.taobao.com/cine ... d=38750&n_s=new
https://dianying.taobao.com/cine ... d=38242&n_s=new
https://dianying.taobao.com/cine ... d=19720&n_s=new
https://dianying.taobao.com/cine ... d=49694&n_s=new
https://dianying.taobao.com/cine ... d=39037&n_s=new
https://dianying.taobao.com/cine ... d=14744&n_s=new
https://dianying.taobao.com/cine ... d=15517&n_s=new
https://dianying.taobao.com/cine ... d=25389&n_s=new
https://dianying.taobao.com/cine ... d=30797&n_s=new
https://dianying.taobao.com/cine ... d=29426&n_s=new
https://dianying.taobao.com/cine ... d=43008&n_s=new
https://dianying.taobao.com/cine ... d=33813&n_s=new
https://dianying.taobao.com/cine ... d=14870&n_s=new
https://dianying.taobao.com/cine ... Id=6297&n_s=new
https://dianying.taobao.com/cine ... Id=5667&n_s=new
https://dianying.taobao.com/cine ... Id=5295&n_s=new
https://dianying.taobao.com/cine ... d=48705&n_s=new
https://dianying.taobao.com/cine ... Id=4292&n_s=new
https://dianying.taobao.com/cine ... Id=4322&n_s=new
https://dianying.taobao.com/cine ... d=55034&n_s=new
https://dianying.taobao.com/cine ... Id=9495&n_s=new
https://dianying.taobao.com/cine ... Id=5150&n_s=new
https://dianying.taobao.com/cine ... Id=6096&n_s=new
https://dianying.taobao.com/cine ... Id=8018&n_s=new
https://dianying.taobao.com/cine ... Id=9845&n_s=new
https://dianying.taobao.com/cine ... Id=5245&n_s=new
https://dianying.taobao.com/cine ... d=18591&n_s=new
https://dianying.taobao.com/cine ... Id=4839&n_s=new
https://dianying.taobao.com/cine ... d=56302&n_s=new
https://dianying.taobao.com/cine ... d=31630&n_s=new
https://dianying.taobao.com/cine ... d=31167&n_s=new
https://dianying.taobao.com/cine ... d=33235&n_s=new
https://dianying.taobao.com/cine ... d=29086&n_s=new
https://dianying.taobao.com/cine ... d=38064&n_s=new
https://dianying.taobao.com/cine ... Id=6227&n_s=new
https://dianying.taobao.com/cine ... d=31206&n_s=new
https://dianying.taobao.com/cine ... d=41735&n_s=new
https://dianying.taobao.com/cine ... d=46221&n_s=new
https://dianying.taobao.com/cine ... d=46349&n_s=new
https://dianying.taobao.com/cine ... d=45615&n_s=new
https://dianying.taobao.com/cine ... d=36200&n_s=new
https://dianying.taobao.com/cine ... d=33157&n_s=new
https://dianying.taobao.com/cine ... d=29857&n_s=new
https://dianying.taobao.com/cine ... Id=5284&n_s=new
https://dianying.taobao.com/cine ... d=14759&n_s=new
https://dianying.taobao.com/cine ... d=49455&n_s=new
https://dianying.taobao.com/cine ... Id=9778&n_s=new
https://dianying.taobao.com/cine ... Id=4662&n_s=new
https://dianying.taobao.com/cine ... Id=5195&n_s=new
https://dianying.taobao.com/cine ... Id=9675&n_s=new
https://dianying.taobao.com/cine ... Id=6257&n_s=new
https://dianying.taobao.com/cine ... Id=9977&n_s=new
https://dianying.taobao.com/cine ... d=42365&n_s=new
https://dianying.taobao.com/cine ... d=24391&n_s=new
https://dianying.taobao.com/cine ... d=29790&n_s=new
https://dianying.taobao.com/cine ... Id=5354&n_s=new
https://dianying.taobao.com/cine ... Id=5483&n_s=new
https://dianying.taobao.com/cine ... d=48513&n_s=new
https://dianying.taobao.com/cine ... d=22161&n_s=new
https://dianying.taobao.com/cine ... d=43177&n_s=new
https://dianying.taobao.com/cine ... d=37957&n_s=new
https://dianying.taobao.com/cine ... d=38742&n_s=new
https://dianying.taobao.com/cine ... d=31205&n_s=new
https://dianying.taobao.com/cine ... d=29812&n_s=new
https://dianying.taobao.com/cine ... d=43432&n_s=new
https://dianying.taobao.com/cine ... d=49211&n_s=new
https://dianying.taobao.com/cine ... d=16970&n_s=new
https://dianying.taobao.com/cine ... d=10072&n_s=new
https://dianying.taobao.com/cine ... d=19829&n_s=new
https://dianying.taobao.com/cine ... d=13886&n_s=new

crossin先生

我刚检查了下，这个地址请求多了会跳机器人验证，而且请求的时候得加 cookie 才行，user-agent、referer 可能也要带上。
直接请求是不行了

fa2003

crossin先生 发表于 2018-10-13 16:06
我刚检查了下，这个地址请求多了会跳机器人验证，而且请求的时候得加 cookie 才行，user-agent、referer 可 ...

多谢，老师， 我先修改下。有不行再问。