python使用scrapy发送post请求的坑


Posted in Python onSeptember 04, 2018

使用requests发送post请求

先来看看使用requests来发送post请求是多少好用,发送请求

Requests 简便的 API 意味着所有 HTTP 请求类型都是显而易见的。例如,你可以这样发送一个 HTTP POST 请求:

>>>r = requests.post('http://httpbin.org/post', data = {'key':'value'})

使用data可以传递字典作为参数,同时也可以传递元祖

>>>payload = (('key1', 'value1'), ('key1', 'value2'))
>>>r = requests.post('http://httpbin.org/post', data=payload)
>>>print(r.text)
{
 ...
 "form": {
  "key1": [
   "value1",
   "value2"
  ]
 },
 ...
}

传递json是这样

>>>import json

>>>url = 'https://api.github.com/some/endpoint'
>>>payload = {'some': 'data'}

>>>r = requests.post(url, data=json.dumps(payload))

2.4.2 版的新加功能:

>>>url = 'https://api.github.com/some/endpoint'
>>>payload = {'some': 'data'}

>>>r = requests.post(url, json=payload)

也就是说,你不需要对参数做什么变化,只需要关注使用data=还是json=,其余的requests都已经帮你做好了。

使用scrapy发送post请求

通过源码可知scrapy默认发送的get请求,当我们需要发送携带参数的请求或登录时,是需要post、请求的,以下面为例

from scrapy.spider import CrawlSpider
from scrapy.selector import Selector
import scrapy
import json
class LaGou(CrawlSpider):
  name = 'myspider'
  def start_requests(self):
    yield scrapy.FormRequest(
      url='https://www.******.com/jobs/positionAjax.json?city=%E5%B9%BF%E5%B7%9E&needAddtionalResult=false',
      formdata={
        'first': 'true',#这里不能给bool类型的True,requests模块中可以
        'pn': '1',#这里不能给int类型的1,requests模块中可以
        'kd': 'python'
      },这里的formdata相当于requ模块中的data,key和value只能是键值对形式
      callback=self.parse
    )
  def parse(self, response):
    datas=json.loads(response.body.decode())['content']['positionResult']['result']
    for data in datas:
      print(data['companyFullName'] + str(data['positionId']))

官方推荐的 Using FormRequest to send data via HTTP POST

return [FormRequest(url="http://www.example.com/post/action",
          formdata={'name': 'John Doe', 'age': '27'},
          callback=self.after_post)]

这里使用的是FormRequest,并使用formdata传递参数,看到这里也是一个字典。

但是,超级坑的一点来了,今天折腾了一下午,使用这种方法发送请求,怎么发都会出问题,返回的数据一直都不是我想要的

return scrapy.FormRequest(url, formdata=(payload))

在网上找了很久,最终找到一种方法,使用scrapy.Request发送请求,就可以正常的获取数据。

return scrapy.Request(url, body=json.dumps(payload), method='POST', headers={'Content-Type': 'application/json'},)

参考:Send Post Request in Scrapy

my_data = {'field1': 'value1', 'field2': 'value2'}
request = scrapy.Request( url, method='POST', 
             body=json.dumps(my_data), 
             headers={'Content-Type':'application/json'} )

FormRequest 与 Request 区别

在文档中,几乎看不到差别,

The FormRequest class adds a new argument to the constructor. The remaining arguments are the same as for the Request class and are not documented here.
Parameters: formdata (dict or iterable of tuples) ? is a dictionary (or iterable of (key, value) tuples) containing HTML Form data which will be url-encoded and assigned to the body of the request.

说FormRequest新增加了一个参数formdata,接受包含表单数据的字典或者可迭代的元组,并将其转化为请求的body。并且FormRequest是继承Request的

class FormRequest(Request):

  def __init__(self, *args, **kwargs):
    formdata = kwargs.pop('formdata', None)
    if formdata and kwargs.get('method') is None:
      kwargs['method'] = 'POST'

    super(FormRequest, self).__init__(*args, **kwargs)

    if formdata:
      items = formdata.items() if isinstance(formdata, dict) else formdata
      querystr = _urlencode(items, self.encoding)
      if self.method == 'POST':
        self.headers.setdefault(b'Content-Type', b'application/x-www-form-urlencoded')
        self._set_body(querystr)
      else:
        self._set_url(self.url + ('&' if '?' in self.url else '?') + querystr)
      ###


def _urlencode(seq, enc):
  values = [(to_bytes(k, enc), to_bytes(v, enc))
       for k, vs in seq
       for v in (vs if is_listlike(vs) else [vs])]
  return urlencode(values, doseq=1)

最终我们传递的{‘key': ‘value', ‘k': ‘v'}会被转化为'key=value&k=v' 并且默认的method是POST,再来看看Request

class Request(object_ref):

  def __init__(self, url, callback=None, method='GET', headers=None, body=None,
         cookies=None, meta=None, encoding='utf-8', priority=0,
         dont_filter=False, errback=None, flags=None):

    self._encoding = encoding # this one has to be set first
    self.method = str(method).upper()

默认的方法是GET,其实并不影响。仍然可以发送post请求。这让我想起来requests中的request用法,这是定义请求的基础方法。

def request(method, url, **kwargs):
  """Constructs and sends a :class:`Request <Request>`.

  :param method: method for the new :class:`Request` object.
  :param url: URL for the new :class:`Request` object.
  :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
  :param data: (optional) Dictionary or list of tuples ``[(key, value)]`` (will be form-encoded), bytes, or file-like object to send in the body of the :class:`Request`.
  :param json: (optional) json data to send in the body of the :class:`Request`.
  :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
  :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
  :param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': file-tuple}``) for multipart encoding upload.
    ``file-tuple`` can be a 2-tuple ``('filename', fileobj)``, 3-tuple ``('filename', fileobj, 'content_type')``
    or a 4-tuple ``('filename', fileobj, 'content_type', custom_headers)``, where ``'content-type'`` is a string
    defining the content type of the given file and ``custom_headers`` a dict-like object containing additional headers
    to add for the file.
  :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
  :param timeout: (optional) How many seconds to wait for the server to send data
    before giving up, as a float, or a :ref:`(connect timeout, read
    timeout) <timeouts>` tuple.
  :type timeout: float or tuple
  :param allow_redirects: (optional) Boolean. Enable/disable GET/OPTIONS/POST/PUT/PATCH/DELETE/HEAD redirection. Defaults to ``True``.
  :type allow_redirects: bool
  :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
  :param verify: (optional) Either a boolean, in which case it controls whether we verify
      the server's TLS certificate, or a string, in which case it must be a path
      to a CA bundle to use. Defaults to ``True``.
  :param stream: (optional) if ``False``, the response content will be immediately downloaded.
  :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.
  :return: :class:`Response <Response>` object
  :rtype: requests.Response

  Usage::

   >>> import requests
   >>> req = requests.request('GET', 'http://httpbin.org/get')
   <Response [200]>
  """

  # By using the 'with' statement we are sure the session is closed, thus we
  # avoid leaving sockets open which can trigger a ResourceWarning in some
  # cases, and look like a memory leak in others.
  with sessions.Session() as session:
    return session.request(method=method, url=url, **kwargs)

以上就是本文的全部内容,希望对大家的学习有所帮助,也希望大家多多支持三水点靠木。

Python 相关文章推荐
python制作爬虫爬取京东商品评论教程
Dec 16 Python
python计算日期之间的放假日期
Jun 05 Python
Python高级特性切片(Slice)操作详解
Sep 27 Python
Python推导式简单示例【列表推导式、字典推导式与集合推导式】
Dec 04 Python
python引用(import)某个模块提示没找到对应模块的解决方法
Jan 19 Python
python ChainMap的使用和说明详解
Jun 11 Python
python函数修饰符@的使用方法解析
Sep 02 Python
Django xadmin开启搜索功能的实现
Nov 15 Python
Python爬虫库BeautifulSoup的介绍与简单使用实例
Jan 25 Python
浅谈Tensorflow加载Vgg预训练模型的几个注意事项
May 26 Python
python 下划线的不同用法
Oct 24 Python
python中mongodb包操作数据库
Apr 19 Python
解决win64 Python下安装PIL出错问题(图解)
Sep 03 #Python
Python全局变量与局部变量区别及用法分析
Sep 03 #Python
Python wxPython库Core组件BoxSizer用法示例
Sep 03 #Python
深入浅析Python中list的复制及深拷贝与浅拷贝
Sep 03 #Python
Python wxPython库使用wx.ListBox创建列表框示例
Sep 03 #Python
Python wxPython库消息对话框MessageDialog用法示例
Sep 03 #Python
Python中关键字global和nonlocal的区别详解
Sep 03 #Python
You might like
PHP 得到根目录的 __FILE__ 常量
2008/07/23 PHP
php命令行(cli)下执行PHP脚本文件的相对路径的问题解决方法
2015/05/25 PHP
php类自动加载器实现方法
2015/07/28 PHP
总结PHP删除字符串最后一个字符的三种方法
2016/08/30 PHP
PHP+Mysql无刷新问答评论系统(源码)
2016/12/20 PHP
vmware linux系统安装最新的php7图解
2019/04/14 PHP
PHP 实现 JSON 数据的编码和解码操作详解
2020/04/22 PHP
JavaScript中的apply()方法和call()方法使用介绍
2012/07/25 Javascript
控制input输入框中提示信息的显示和隐藏的方法
2014/02/12 Javascript
jQuery图片特效插件Revealing实现拉伸放大
2015/04/22 Javascript
利用纯Vue.js构建Bootstrap组件
2016/11/03 Javascript
浅谈js中的变量名和函数名重名
2017/02/13 Javascript
React组件生命周期详解
2017/07/03 Javascript
JS中定位 position 的使用实例代码
2017/08/06 Javascript
javascript深拷贝、浅拷贝和循环引用深入理解
2018/05/27 Javascript
vue之将echart封装为组件
2018/06/02 Javascript
JavaScript检测是否开启了控制台(F12调试工具)
2020/10/02 Javascript
[10:54]Team Spirit vs Navi
2018/06/07 DOTA
python实现巡检系统(solaris)示例
2014/04/02 Python
Python兔子毒药问题实例分析
2015/03/05 Python
在GitHub Pages上使用Pelican搭建博客的教程
2015/04/25 Python
Python中单、双下划线的区别总结
2017/12/01 Python
Python 绘制酷炫的三维图步骤详解
2019/07/12 Python
关于Python 中的时间处理包datetime和arrow的方法详解
2020/03/19 Python
Python响应对象text属性乱码解决方案
2020/03/31 Python
Python 实现打印单词的菱形字符图案
2020/04/12 Python
Django 如何使用日期时间选择器规范用户的时间输入示例代码详解
2020/05/22 Python
Keras—embedding嵌入层的用法详解
2020/06/10 Python
CSS中几个与换行有关的属性简明总结
2014/04/15 HTML / CSS
十佳大学生村官事迹
2014/01/09 职场文书
大学生军训自我鉴定
2014/02/12 职场文书
干部鉴定材料
2014/05/18 职场文书
学校领导班子群众路线整改措施
2014/09/16 职场文书
三年级上册科学教学计划
2015/01/21 职场文书
详解Laravel服务容器的优势
2021/05/29 PHP
mysql的Buffer Pool存储及原理
2022/04/02 MySQL