Python开发的HTTP库requests详解


Posted in Python onAugust 29, 2017

Requests 是使用 Apache2 Licensed 许可证的 基于Python开发的HTTP 库,其在Python内置模块的基础上进行了高度的封装,从而使得Pythoner进行网络请求时,变得美好了许多,使用Requests可以轻而易举的完成浏览器可有的任何操作。

1. GET请求

# 1、无参数实例
  
 import requests
  
 ret = requests.get('https://github.com/timeline.json')
  
 print(ret.url)
 print(ret.text)

 # 2、有参数实例
  
 import requests
  
 payload = {'key1': 'value1', 'key2': 'value2'}
 ret = requests.get("http://httpbin.org/get", params=payload)
  
 print(ret.url)
 print(ret.text)

2. POST请求

# 1、基本POST实例
  
 import requests
  
 payload = {'key1': 'value1', 'key2': 'value2'}
 ret = requests.post("http://httpbin.org/post", data=payload)
  
 print(ret.text)
  
  
 # 2、发送请求头和数据实例
  
 import requests
 import json
  
 url = 'https://api.github.com/some/endpoint'
 payload = {'some': 'data'}
 headers = {'content-type': 'application/json'}
  
 ret = requests.post(url, data=json.dumps(payload), headers=headers)
  
 print(ret.text)
 print(ret.cookies)

3. 其它请求

requests.get(url, params=None, **kwargs)
 requests.post(url, data=None, json=None, **kwargs)
 requests.put(url, data=None, **kwargs)
 requests.head(url, **kwargs)
 requests.delete(url, **kwargs)
 requests.patch(url, data=None, **kwargs)
 requests.options(url, **kwargs)
  
 # 以上方法均是在此方法的基础上构建
 requests.request(method, url, **kwargs)

4. 请求参数

def request(method, url, **kwargs):
  """Constructs and sends a :class:`Request <Request>`.

  :param method: method for the new :class:`Request` object.
  :param url: URL for the new :class:`Request` object.
  :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
  :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
  :param json: (optional) json data to send in the body of the :class:`Request`.
  :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
  :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
  :param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': file-tuple}``) for multipart encoding upload.
    ``file-tuple`` can be a 2-tuple ``('filename', fileobj)``, 3-tuple ``('filename', fileobj, 'content_type')``
    or a 4-tuple ``('filename', fileobj, 'content_type', custom_headers)``, where ``'content-type'`` is a string
    defining the content type of the given file and ``custom_headers`` a dict-like object containing additional headers
    to add for the file.
  :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
  :param timeout: (optional) How long to wait for the server to send data
    before giving up, as a float, or a :ref:`(connect timeout, read
    timeout) <timeouts>` tuple.
  :type timeout: float or tuple
  :param allow_redirects: (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed.
  :type allow_redirects: bool
  :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
  :param verify: (optional) whether the SSL cert will be verified. A CA_BUNDLE path can also be provided. Defaults to ``True``.
  :param stream: (optional) if ``False``, the response content will be immediately downloaded.
  :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.
  :return: :class:`Response <Response>` object
  :rtype: requests.Response

  Usage::

   >>> import requests
   >>> req = requests.request('GET', 'http://httpbin.org/get')
   <Response [200]>
  """

5. 参数示例

def param_method_url():
  # requests.request(method='get', url='http://127.0.0.1:8000/test/')
  # requests.request(method='post', url='http://127.0.0.1:8000/test/')
  pass


def param_param():
  # - 可以是字典
  # - 可以是字符串
  # - 可以是字节(ascii编码以内)

  # requests.request(method='get',
  # url='http://127.0.0.1:8000/test/',
  # params={'k1': 'v1', 'k2': '水电费'})

  # requests.request(method='get',
  # url='http://127.0.0.1:8000/test/',
  # params="k1=v1&k2=水电费&k3=v3&k3=vv3")

  # requests.request(method='get',
  # url='http://127.0.0.1:8000/test/',
  # params=bytes("k1=v1&k2=k2&k3=v3&k3=vv3", encoding='utf8'))

  # 错误
  # requests.request(method='get',
  # url='http://127.0.0.1:8000/test/',
  # params=bytes("k1=v1&k2=水电费&k3=v3&k3=vv3", encoding='utf8'))
  pass


def param_data():
  # 可以是字典
  # 可以是字符串
  # 可以是字节
  # 可以是文件对象

  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # data={'k1': 'v1', 'k2': '水电费'})

  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # data="k1=v1; k2=v2; k3=v3; k3=v4"
  # )

  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # data="k1=v1;k2=v2;k3=v3;k3=v4",
  # headers={'Content-Type': 'application/x-www-form-urlencoded'}
  # )

  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # data=open('data_file.py', mode='r', encoding='utf-8'), # 文件内容是:k1=v1;k2=v2;k3=v3;k3=v4
  # headers={'Content-Type': 'application/x-www-form-urlencoded'}
  # )
  pass


def param_json():
  # 将json中对应的数据进行序列化成一个字符串,json.dumps(...)
  # 然后发送到服务器端的body中,并且Content-Type是 {'Content-Type': 'application/json'}
  requests.request(method='POST',
           url='http://127.0.0.1:8000/test/',
           json={'k1': 'v1', 'k2': '水电费'})


def param_headers():
  # 发送请求头到服务器端
  requests.request(method='POST',
           url='http://127.0.0.1:8000/test/',
           json={'k1': 'v1', 'k2': '水电费'},
           headers={'Content-Type': 'application/x-www-form-urlencoded'}
           )


def param_cookies():
  # 发送Cookie到服务器端
  requests.request(method='POST',
           url='http://127.0.0.1:8000/test/',
           data={'k1': 'v1', 'k2': 'v2'},
           cookies={'cook1': 'value1'},
           )
  # 也可以使用CookieJar(字典形式就是在此基础上封装)
  from http.cookiejar import CookieJar
  from http.cookiejar import Cookie

  obj = CookieJar()
  obj.set_cookie(Cookie(version=0, name='c1', value='v1', port=None, domain='', path='/', secure=False, expires=None,
             discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False,
             port_specified=False, domain_specified=False, domain_initial_dot=False, path_specified=False)
          )
  requests.request(method='POST',
           url='http://127.0.0.1:8000/test/',
           data={'k1': 'v1', 'k2': 'v2'},
           cookies=obj)


def param_files():
  # 发送文件
  # file_dict = {
  # 'f1': open('readme', 'rb')
  # }
  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # files=file_dict)

  # 发送文件,定制文件名
  # file_dict = {
  # 'f1': ('test.txt', open('readme', 'rb'))
  # }
  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # files=file_dict)

  # 发送文件,定制文件名
  # file_dict = {
  # 'f1': ('test.txt', "hahsfaksfa9kasdjflaksdjf")
  # }
  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # files=file_dict)

  # 发送文件,定制文件名
  # file_dict = {
  #   'f1': ('test.txt', "hahsfaksfa9kasdjflaksdjf", 'application/text', {'k1': '0'})
  # }
  # requests.request(method='POST',
  #         url='http://127.0.0.1:8000/test/',
  #         files=file_dict)

  pass


def param_auth():
  from requests.auth import HTTPBasicAuth, HTTPDigestAuth

  ret = requests.get('https://api.github.com/user', auth=HTTPBasicAuth('wupeiqi', 'sdfasdfasdf'))
  print(ret.text)

  # ret = requests.get('http://192.168.1.1',
  # auth=HTTPBasicAuth('admin', 'admin'))
  # ret.encoding = 'gbk'
  # print(ret.text)

  # ret = requests.get('http://httpbin.org/digest-auth/auth/user/pass', auth=HTTPDigestAuth('user', 'pass'))
  # print(ret)
  #


def param_timeout():
  # ret = requests.get('http://google.com/', timeout=1)
  # print(ret)

  # ret = requests.get('http://google.com/', timeout=(5, 1))
  # print(ret)
  pass


def param_allow_redirects():
  ret = requests.get('http://127.0.0.1:8000/test/', allow_redirects=False)
  print(ret.text)


def param_proxies():
  # proxies = {
  # "http": "61.172.249.96:80",
  # "https": "http://61.185.219.126:3128",
  # }

  # proxies = {'http://10.20.1.128': 'http://10.10.1.10:5323'}

  # ret = requests.get("http://www.proxy360.cn/Proxy", proxies=proxies)
  # print(ret.headers)


  # from requests.auth import HTTPProxyAuth
  #
  # proxyDict = {
  # 'http': '77.75.105.165',
  # 'https': '77.75.105.165'
  # }
  # auth = HTTPProxyAuth('username', 'mypassword')
  #
  # r = requests.get("http://www.google.com", proxies=proxyDict, auth=auth)
  # print(r.text)

  pass


def param_stream():
  ret = requests.get('http://127.0.0.1:8000/test/', stream=True)
  print(ret.content)
  ret.close()

  # from contextlib import closing
  # with closing(requests.get('http://httpbin.org/get', stream=True)) as r:
  # # 在此处理响应。
  # for i in r.iter_content():
  # print(i)


def requests_session():
  import requests

  session = requests.Session()

  ### 1、首先登陆任何页面,获取cookie

  i1 = session.get(url="http://dig.chouti.com/help/service")

  ### 2、用户登陆,携带上一次的cookie,后台对cookie中的 gpsd 进行授权
  i2 = session.post(
    url="http://dig.chouti.com/login",
    data={
      'phone': "8615131255089",
      'password': "xxxxxx",
      'oneMonth': ""
    }
  )

  i3 = session.post(
    url="http://dig.chouti.com/link/vote?linksId=8589623",
  )
  print(i3.text)

6. requests模拟登陆GitHub

import requests
 from bs4 import BeautifulSoup
 
 
 def login_github():
   """
   通过requests模块模拟浏览器登陆GitHub
   :return: 
   """
   # 获取csrf_token
   r1 = requests.get('https://github.com/login')  # 获得get请求的对象
   s1 = BeautifulSoup(r1.text, 'html.parser')   # 使用bs4解析HTML对象
   token = s1.find('input', attrs={'name': 'authenticity_token'}).get('value')   # 获取登陆授权码,即csrf_token
   get_cookies = r1.cookies.get_dict()   # 获取get请求的cookies,post请求时必须携带
   
   # 发送post登陆请求
   '''
   post登陆参数
   commit  Sign+in
   utf8  ✓
   authenticity_token  E961jQMIyC9NPwL54YPj70gv2hbXWJ…fTUd+e4lT5RAizKbfzQo4eRHsfg==
   login  JackUpDown(用户名)
   password  **********(密码)
   '''
   r2 = requests.post(
     'https://github.com/session',
     data={
       'commit': 'Sign+in',
       'utf8': '✓',
       'authenticity_token': token,
       'login': 'JackUpDown',
       'password': '**********'
     },
     cookies=get_cookies   # 携带get请求的cookies
            )
   login_cookies = r2.cookies.get_dict()  # 获得登陆成功的cookies,携带此cookies就可以访问任意GitHub页面
 
   # 携带post cookies跳转任意页面
   r3 = requests.get('https://github.com/settings/emails', cookies=login_cookies)
   print(r3.text)
Python 相关文章推荐
简单的Python2.7编程初学经验总结
Apr 01 Python
Python基于smtplib实现异步发送邮件服务
May 28 Python
python并发编程之多进程、多线程、异步和协程详解
Oct 28 Python
Python 3中的yield from语法详解
Jan 18 Python
Python实现对象转换为xml的方法示例
Jun 08 Python
Python中defaultdict与lambda表达式用法实例小结
Apr 09 Python
对Python 文件夹遍历和文件查找的实例讲解
Apr 26 Python
python实现音乐下载的统计
Jun 20 Python
eclipse创建python项目步骤详解
May 10 Python
django多个APP的urls设置方法(views重复问题解决)
Jul 19 Python
python中类的输出或类的实例输出为这种形式的原因
Aug 12 Python
PO模式在selenium自动化测试框架的优势
Mar 20 Python
Python实现按学生年龄排序的实际问题详解
Aug 29 #Python
详解Python进程间通信之命名管道
Aug 28 #Python
基于Python闭包及其作用域详解
Aug 28 #Python
利用Python查看目录中的文件示例详解
Aug 28 #Python
Python如何通过subprocess调用adb命令详解
Aug 27 #Python
Python中序列的修改、散列与切片详解
Aug 27 #Python
Python正确重载运算符的方法示例详解
Aug 27 #Python
You might like
php代码把全角数字转为半角数字
2007/12/10 PHP
PHP 字符串编码截取函数(兼容utf-8和gb2312)
2009/05/02 PHP
用PHP解决的一个栈的面试题
2014/07/02 PHP
在SAE上搭建最新wordpress的方法
2014/12/21 PHP
PHP中strcmp()和strcasecmp()函数字符串比较用法分析
2016/01/07 PHP
Yii中的relations数据关联查询及统计功能用法详解
2016/07/14 PHP
PHP实现浏览器格式化显示XML的方法示例
2019/01/22 PHP
网上应用的一个不错common.js脚本
2007/08/08 Javascript
两个select之间option的互相添加操作(jquery实现)
2009/11/12 Javascript
javascript的函数、创建对象、封装、属性和方法、继承
2011/03/10 Javascript
js限制textarea每行输入字符串长度的代码
2012/10/31 Javascript
js动态拼接正则表达式的两种方法
2014/03/04 Javascript
Javascript学习笔记之数组的构造函数
2014/11/23 Javascript
window.location.hash知识汇总
2015/11/09 Javascript
iframe中使用jquery进行查找的方法【案例分析】
2016/06/17 Javascript
js传值后台中文出现乱码的解决方法
2016/06/30 Javascript
浅谈JQ中mouseover和mouseenter的区别
2016/09/13 Javascript
微信小程序使用第三方库Immutable.js实例详解
2016/09/27 Javascript
JS中type=&quot;button&quot;和type=&quot;submit&quot;的区别
2017/07/04 Javascript
Three.JS实现三维场景
2018/12/30 Javascript
Vue js with语句原理及用法解析
2020/09/03 Javascript
[55:16]Mski vs VGJ.S Supermajor小组赛C组 BO3 第二场 6.3
2018/06/04 DOTA
python自然语言编码转换模块codecs介绍
2015/04/08 Python
Python实现字典的key和values的交换
2015/08/04 Python
python实现多层感知器
2019/01/18 Python
python使用time、datetime返回工作日列表实例代码
2019/05/09 Python
如何使用Python 打印各种三角形
2019/06/28 Python
windows下Python安装、使用教程和Notepad++的使用教程
2019/10/06 Python
win10从零安装配置pytorch全过程图文详解
2020/05/08 Python
深入浅析HTML5中的article和section的区别
2018/05/15 HTML / CSS
Java提供了哪些企业应用编程接口
2015/02/13 面试题
酒店服务实习自我鉴定
2013/09/22 职场文书
投资公司董事长岗位职责
2015/04/16 职场文书
小学生安全教育心得体会
2016/01/15 职场文书
Mysql MVCC机制原理详解
2021/04/20 MySQL
python​格式化字符串
2022/04/20 Python