Python开发的HTTP库requests详解


Posted in Python onAugust 29, 2017

Requests 是使用 Apache2 Licensed 许可证的 基于Python开发的HTTP 库,其在Python内置模块的基础上进行了高度的封装,从而使得Pythoner进行网络请求时,变得美好了许多,使用Requests可以轻而易举的完成浏览器可有的任何操作。

1. GET请求

# 1、无参数实例
  
 import requests
  
 ret = requests.get('https://github.com/timeline.json')
  
 print(ret.url)
 print(ret.text)

 # 2、有参数实例
  
 import requests
  
 payload = {'key1': 'value1', 'key2': 'value2'}
 ret = requests.get("http://httpbin.org/get", params=payload)
  
 print(ret.url)
 print(ret.text)

2. POST请求

# 1、基本POST实例
  
 import requests
  
 payload = {'key1': 'value1', 'key2': 'value2'}
 ret = requests.post("http://httpbin.org/post", data=payload)
  
 print(ret.text)
  
  
 # 2、发送请求头和数据实例
  
 import requests
 import json
  
 url = 'https://api.github.com/some/endpoint'
 payload = {'some': 'data'}
 headers = {'content-type': 'application/json'}
  
 ret = requests.post(url, data=json.dumps(payload), headers=headers)
  
 print(ret.text)
 print(ret.cookies)

3. 其它请求

requests.get(url, params=None, **kwargs)
 requests.post(url, data=None, json=None, **kwargs)
 requests.put(url, data=None, **kwargs)
 requests.head(url, **kwargs)
 requests.delete(url, **kwargs)
 requests.patch(url, data=None, **kwargs)
 requests.options(url, **kwargs)
  
 # 以上方法均是在此方法的基础上构建
 requests.request(method, url, **kwargs)

4. 请求参数

def request(method, url, **kwargs):
  """Constructs and sends a :class:`Request <Request>`.

  :param method: method for the new :class:`Request` object.
  :param url: URL for the new :class:`Request` object.
  :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
  :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
  :param json: (optional) json data to send in the body of the :class:`Request`.
  :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
  :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
  :param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': file-tuple}``) for multipart encoding upload.
    ``file-tuple`` can be a 2-tuple ``('filename', fileobj)``, 3-tuple ``('filename', fileobj, 'content_type')``
    or a 4-tuple ``('filename', fileobj, 'content_type', custom_headers)``, where ``'content-type'`` is a string
    defining the content type of the given file and ``custom_headers`` a dict-like object containing additional headers
    to add for the file.
  :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
  :param timeout: (optional) How long to wait for the server to send data
    before giving up, as a float, or a :ref:`(connect timeout, read
    timeout) <timeouts>` tuple.
  :type timeout: float or tuple
  :param allow_redirects: (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed.
  :type allow_redirects: bool
  :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
  :param verify: (optional) whether the SSL cert will be verified. A CA_BUNDLE path can also be provided. Defaults to ``True``.
  :param stream: (optional) if ``False``, the response content will be immediately downloaded.
  :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.
  :return: :class:`Response <Response>` object
  :rtype: requests.Response

  Usage::

   >>> import requests
   >>> req = requests.request('GET', 'http://httpbin.org/get')
   <Response [200]>
  """

5. 参数示例

def param_method_url():
  # requests.request(method='get', url='http://127.0.0.1:8000/test/')
  # requests.request(method='post', url='http://127.0.0.1:8000/test/')
  pass


def param_param():
  # - 可以是字典
  # - 可以是字符串
  # - 可以是字节(ascii编码以内)

  # requests.request(method='get',
  # url='http://127.0.0.1:8000/test/',
  # params={'k1': 'v1', 'k2': '水电费'})

  # requests.request(method='get',
  # url='http://127.0.0.1:8000/test/',
  # params="k1=v1&k2=水电费&k3=v3&k3=vv3")

  # requests.request(method='get',
  # url='http://127.0.0.1:8000/test/',
  # params=bytes("k1=v1&k2=k2&k3=v3&k3=vv3", encoding='utf8'))

  # 错误
  # requests.request(method='get',
  # url='http://127.0.0.1:8000/test/',
  # params=bytes("k1=v1&k2=水电费&k3=v3&k3=vv3", encoding='utf8'))
  pass


def param_data():
  # 可以是字典
  # 可以是字符串
  # 可以是字节
  # 可以是文件对象

  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # data={'k1': 'v1', 'k2': '水电费'})

  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # data="k1=v1; k2=v2; k3=v3; k3=v4"
  # )

  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # data="k1=v1;k2=v2;k3=v3;k3=v4",
  # headers={'Content-Type': 'application/x-www-form-urlencoded'}
  # )

  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # data=open('data_file.py', mode='r', encoding='utf-8'), # 文件内容是:k1=v1;k2=v2;k3=v3;k3=v4
  # headers={'Content-Type': 'application/x-www-form-urlencoded'}
  # )
  pass


def param_json():
  # 将json中对应的数据进行序列化成一个字符串,json.dumps(...)
  # 然后发送到服务器端的body中,并且Content-Type是 {'Content-Type': 'application/json'}
  requests.request(method='POST',
           url='http://127.0.0.1:8000/test/',
           json={'k1': 'v1', 'k2': '水电费'})


def param_headers():
  # 发送请求头到服务器端
  requests.request(method='POST',
           url='http://127.0.0.1:8000/test/',
           json={'k1': 'v1', 'k2': '水电费'},
           headers={'Content-Type': 'application/x-www-form-urlencoded'}
           )


def param_cookies():
  # 发送Cookie到服务器端
  requests.request(method='POST',
           url='http://127.0.0.1:8000/test/',
           data={'k1': 'v1', 'k2': 'v2'},
           cookies={'cook1': 'value1'},
           )
  # 也可以使用CookieJar(字典形式就是在此基础上封装)
  from http.cookiejar import CookieJar
  from http.cookiejar import Cookie

  obj = CookieJar()
  obj.set_cookie(Cookie(version=0, name='c1', value='v1', port=None, domain='', path='/', secure=False, expires=None,
             discard=True, comment=None, comment_url=None, rest={'HttpOnly': None}, rfc2109=False,
             port_specified=False, domain_specified=False, domain_initial_dot=False, path_specified=False)
          )
  requests.request(method='POST',
           url='http://127.0.0.1:8000/test/',
           data={'k1': 'v1', 'k2': 'v2'},
           cookies=obj)


def param_files():
  # 发送文件
  # file_dict = {
  # 'f1': open('readme', 'rb')
  # }
  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # files=file_dict)

  # 发送文件,定制文件名
  # file_dict = {
  # 'f1': ('test.txt', open('readme', 'rb'))
  # }
  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # files=file_dict)

  # 发送文件,定制文件名
  # file_dict = {
  # 'f1': ('test.txt', "hahsfaksfa9kasdjflaksdjf")
  # }
  # requests.request(method='POST',
  # url='http://127.0.0.1:8000/test/',
  # files=file_dict)

  # 发送文件,定制文件名
  # file_dict = {
  #   'f1': ('test.txt', "hahsfaksfa9kasdjflaksdjf", 'application/text', {'k1': '0'})
  # }
  # requests.request(method='POST',
  #         url='http://127.0.0.1:8000/test/',
  #         files=file_dict)

  pass


def param_auth():
  from requests.auth import HTTPBasicAuth, HTTPDigestAuth

  ret = requests.get('https://api.github.com/user', auth=HTTPBasicAuth('wupeiqi', 'sdfasdfasdf'))
  print(ret.text)

  # ret = requests.get('http://192.168.1.1',
  # auth=HTTPBasicAuth('admin', 'admin'))
  # ret.encoding = 'gbk'
  # print(ret.text)

  # ret = requests.get('http://httpbin.org/digest-auth/auth/user/pass', auth=HTTPDigestAuth('user', 'pass'))
  # print(ret)
  #


def param_timeout():
  # ret = requests.get('http://google.com/', timeout=1)
  # print(ret)

  # ret = requests.get('http://google.com/', timeout=(5, 1))
  # print(ret)
  pass


def param_allow_redirects():
  ret = requests.get('http://127.0.0.1:8000/test/', allow_redirects=False)
  print(ret.text)


def param_proxies():
  # proxies = {
  # "http": "61.172.249.96:80",
  # "https": "http://61.185.219.126:3128",
  # }

  # proxies = {'http://10.20.1.128': 'http://10.10.1.10:5323'}

  # ret = requests.get("http://www.proxy360.cn/Proxy", proxies=proxies)
  # print(ret.headers)


  # from requests.auth import HTTPProxyAuth
  #
  # proxyDict = {
  # 'http': '77.75.105.165',
  # 'https': '77.75.105.165'
  # }
  # auth = HTTPProxyAuth('username', 'mypassword')
  #
  # r = requests.get("http://www.google.com", proxies=proxyDict, auth=auth)
  # print(r.text)

  pass


def param_stream():
  ret = requests.get('http://127.0.0.1:8000/test/', stream=True)
  print(ret.content)
  ret.close()

  # from contextlib import closing
  # with closing(requests.get('http://httpbin.org/get', stream=True)) as r:
  # # 在此处理响应。
  # for i in r.iter_content():
  # print(i)


def requests_session():
  import requests

  session = requests.Session()

  ### 1、首先登陆任何页面,获取cookie

  i1 = session.get(url="http://dig.chouti.com/help/service")

  ### 2、用户登陆,携带上一次的cookie,后台对cookie中的 gpsd 进行授权
  i2 = session.post(
    url="http://dig.chouti.com/login",
    data={
      'phone': "8615131255089",
      'password': "xxxxxx",
      'oneMonth': ""
    }
  )

  i3 = session.post(
    url="http://dig.chouti.com/link/vote?linksId=8589623",
  )
  print(i3.text)

6. requests模拟登陆GitHub

import requests
 from bs4 import BeautifulSoup
 
 
 def login_github():
   """
   通过requests模块模拟浏览器登陆GitHub
   :return: 
   """
   # 获取csrf_token
   r1 = requests.get('https://github.com/login')  # 获得get请求的对象
   s1 = BeautifulSoup(r1.text, 'html.parser')   # 使用bs4解析HTML对象
   token = s1.find('input', attrs={'name': 'authenticity_token'}).get('value')   # 获取登陆授权码,即csrf_token
   get_cookies = r1.cookies.get_dict()   # 获取get请求的cookies,post请求时必须携带
   
   # 发送post登陆请求
   '''
   post登陆参数
   commit  Sign+in
   utf8  ✓
   authenticity_token  E961jQMIyC9NPwL54YPj70gv2hbXWJ…fTUd+e4lT5RAizKbfzQo4eRHsfg==
   login  JackUpDown(用户名)
   password  **********(密码)
   '''
   r2 = requests.post(
     'https://github.com/session',
     data={
       'commit': 'Sign+in',
       'utf8': '✓',
       'authenticity_token': token,
       'login': 'JackUpDown',
       'password': '**********'
     },
     cookies=get_cookies   # 携带get请求的cookies
            )
   login_cookies = r2.cookies.get_dict()  # 获得登陆成功的cookies,携带此cookies就可以访问任意GitHub页面
 
   # 携带post cookies跳转任意页面
   r3 = requests.get('https://github.com/settings/emails', cookies=login_cookies)
   print(r3.text)
Python 相关文章推荐
Windows下Python的Django框架环境部署及应用编写入门
Mar 10 Python
python+mongodb数据抓取详细介绍
Oct 25 Python
Python之pandas读写文件乱码的解决方法
Apr 20 Python
django自带serializers序列化返回指定字段的方法
Aug 21 Python
python2和python3应该学哪个(python3.6与python3.7的选择)
Oct 01 Python
三个python爬虫项目实例代码
Dec 28 Python
基于pygame实现童年掌机打砖块游戏
Feb 25 Python
Python装饰器用法与知识点小结
Mar 09 Python
python实现遍历文件夹图片并重命名
Mar 23 Python
python如何停止递归
Sep 09 Python
Python脚本调试工具安装过程
Jan 11 Python
pandas时间序列之pd.to_datetime()的实现
Jun 16 Python
Python实现按学生年龄排序的实际问题详解
Aug 29 #Python
详解Python进程间通信之命名管道
Aug 28 #Python
基于Python闭包及其作用域详解
Aug 28 #Python
利用Python查看目录中的文件示例详解
Aug 28 #Python
Python如何通过subprocess调用adb命令详解
Aug 27 #Python
Python中序列的修改、散列与切片详解
Aug 27 #Python
Python正确重载运算符的方法示例详解
Aug 27 #Python
You might like
做一个有下拉功能的留言版
2006/10/09 PHP
腾讯微博提示missing parameter errorcode 102 错误的解决方法
2014/12/22 PHP
浅析PHP中的 inet_pton 网络函数
2019/12/16 PHP
javascript 类型判断代码分析
2010/03/28 Javascript
下载网站打开页面后间隔多少时间才显示下载链接地址的代码
2010/04/25 Javascript
Javascript中匿名函数的调用与写法实例详解(多种)
2016/01/26 Javascript
node.js学习之base64编码解码
2016/10/21 Javascript
BOM之navigator对象和用户代理检测
2017/02/10 Javascript
ReactJs设置css样式的方法
2017/06/08 Javascript
解决使用bootstrap的dropdown部件时报错:error:Bootstrap dropdown require Popper.js问题
2018/08/30 Javascript
Vue项目打包编译优化方案
2020/09/16 Javascript
[00:35]DOTA2上海特级锦标赛 MVP.Phx战队宣传片
2016/03/04 DOTA
[01:12:44]VG vs Mineski Supermajor 败者组 BO3 第二场 6.6
2018/06/07 DOTA
python编程-将Python程序转化为可执行程序[整理]
2007/04/09 Python
Python标准库之多进程(multiprocessing包)介绍
2014/11/25 Python
浅谈python中截取字符函数strip,lstrip,rstrip
2015/07/17 Python
使用Django的模版来配合字符串翻译工作
2015/07/27 Python
Python绘制七段数码管实例代码
2017/12/20 Python
urllib和BeautifulSoup爬取维基百科的词条简单实例
2018/01/17 Python
利用Python+Java调用Shell脚本时的死锁陷阱详解
2018/01/24 Python
python分布式编程实现过程解析
2019/11/08 Python
pycharm运行scrapy过程图解
2019/11/22 Python
Tensorflow的常用矩阵生成方式
2020/01/04 Python
Python更新所有已安装包的操作
2020/02/13 Python
Python3标准库之threading进程中管理并发操作方法
2020/03/30 Python
parser.add_argument中的action使用
2020/04/20 Python
thinkphp5 路由分发原理
2021/03/18 PHP
Senreve官网:美国旧金山的奢侈手袋品牌
2019/03/21 全球购物
工厂保安员岗位职责
2014/01/31 职场文书
触摸春天教学反思
2014/02/03 职场文书
建筑工地质量标语
2014/06/12 职场文书
2014红色之旅心得体会
2014/10/07 职场文书
秦始皇兵马俑导游词
2015/02/02 职场文书
python opencv常用图形绘制方法(线段、矩形、圆形、椭圆、文本)
2021/04/12 Python
Vue监视数据的原理详解
2022/02/24 Vue.js
使用Apache Camel表达REST服务的方法
2022/06/10 Servers