python网页请求urllib2模块简单封装代码


Posted in Python onFebruary 07, 2014

对python网页请求模块urllib2进行简单的封装。

例子:

#!/usr/bin/python
#coding: utf-8
import base64
import urllib
import urllib2
import time
class SendRequest:
  '''
  This class use to set and request the http, and get the info of response.
  e.g. set Authorization Type, request tyep..
  e.g. get html content, state code, cookie..
  SendRequest('http://10.75.0.103:8850/2/photos/square/type.json', 
              data='source=216274069', type='POST', auth='base',
     user='zl2010', password='111111')
  '''
  def __init__(self, url, data=None, type='GET', auth=None, user=None, password=None, cookie = None, **header):
    '''
    url:request, raise error if none
    date: data for post or get, must be dict type
    type: GET, POST
    auth: option, if has the value must be 'base' or 'cookie'
    user: user for auth
    password: password for auth
    cookie: if request with cookie 
    other header info: 
    e.g. referer='www.sina.com.cn'    
    '''
    self.url = url
    self.data = data
    self.type = type
    self.auth = auth
    self.user = user
    self.password = password 
    self.cookie = cookie 
    if 'referer' in header:
      self.referer = header[referer]
    else:
      self.referer = None
    if 'user-agent' in header:
      self.user_agent = header[user-agent]
    else:
      self.user_agent = None
    self.setup_request()
    self.send_request() 
  def setup_request(self):
    '''
    setup a request 
    '''
    if self.url == None or self.url == '':
      raise 'The url should not empty!'
    # set request type 
    #print self.url
    #print self.type
    #print self.data
    #print self.auth
    #print self.user
    #print self.password  
    if self.type == 'POST': 
      self.Req = urllib2.Request(self.url, self.data)
    elif self.type == 'GET':
      if self.data == None:
          self.Req = urllib2.Request(self.url)
      else:
        self.Req = urllib2.Request(self.url + '?' + self.data)
    else:
      print 'The http request type NOT support now!'
    ##set auth type 
    if self.auth == 'base':
      if self.user == None or self.password == None:
        raise 'The user or password was not given!'
      else:
        auth_info = base64.encodestring(self.user + ':' + self.password).replace('\n','')
        auth_info = 'Basic ' + auth_info 
        #print auth_info  
        self.Req.add_header("Authorization", auth_info)
    elif self.auth == 'cookie':
      if self.cookie == None:
        raise 'The cookie was not given!'
      else:
        self.Req.add_header("Cookie", self.cookie) 
    else:
      pass    ##add other auth type here 
    ##set other header info 
    if self.referer:
      self.Req.add_header('referer', self.referer)
    if self.user_agent:
      self.Req.add_header('user-agent', self.user_agent)
      
  def send_request(self):  
    '''
    send a request 
    '''
    # get a response object 
    try:
      self.Res = urllib2.urlopen(self.Req)
      self.source = self.Res.read()
      self.goal_url = self.Res.geturl()
      self.code = self.Res.getcode()
      self.head_dict = self.Res.info().dict
      self.Res.close()
    except urllib2.HTTPError, e:
      self.code = e.code
      print e
        
  def get_code(self):
    return self.code
  def get_url(self):
    return self.goal_url
  def get_source(self):        
    return self.source
  def get_header_info(self):
    return self.head_dict
  def get_cookie(self):
    if 'set-cookie' in self.head_dict:
      return self.head_dict['set-cookie']
    else:
      return None    
  def get_content_type(self):
    if 'content-type' in self.head_dict:
      return self.head_dict['content-type']
    else:
      return None
  def get_expires_time(self):
    if 'expires' in self.head_dict:
      return self.head_dict['expires']
    else:
      return None    
  def get_server_name(self):
    if 'server' in self.head_dict:
      return self.head_dict['server']
    else:
      return None   
  def __del__(self):
    pass   
__all__ = [SendRequest,]
if __name__ == '__main__':
  '''
  The example for using the SendRequest class 
  '''
  value = {'source':'216274069'}
  data = urllib.urlencode(value)
  url = 'http://10.75.0.103:8850/2/photos/square/type.json'
  user = 'wz_0001'
  password = '111111'
  auth = 'base'
  type = 'POST'
  t2 = time.time()
  rs = SendRequest('http://www.google.com')
  #rs = SendRequest(url, data=data, type=type, auth=auth, user=user, password=password)
  print 't2: ' + str(time.time() - t2)
  print '---------------get_code()---------------'
  print rs.get_code()
  print '---------------get_url()---------------'
  print rs.get_url()
  print '---------------get_source()---------------'
  print rs.get_source()
  print '---------------get_cookie()---------------'
  print rs.get_cookie()
  rs = None
Python 相关文章推荐
python网络编程之读取网站根目录实例
Sep 30 Python
python使用threading获取线程函数返回值的实现方法
Nov 15 Python
Python实现将MySQL数据库表中的数据导出生成csv格式文件的方法
Jan 11 Python
python读取和保存视频文件
Apr 16 Python
python 读取txt,json和hdf5文件的实例
Jun 05 Python
python中退出多层循环的方法
Nov 27 Python
在pycharm上mongodb配置及可视化设置方法
Nov 30 Python
PIL对上传到Django的图片进行处理并保存的实例
Aug 07 Python
python实现爬虫抓取小说功能示例【抓取金庸小说】
Aug 09 Python
Python Django 简单分页的实现代码解析
Aug 21 Python
Python模块常用四种安装方式
Oct 20 Python
Python+腾讯云服务器实现每日自动健康打卡
Dec 06 Python
python解析xml模块封装代码
Feb 07 #Python
python 解析XML python模块xml.dom解析xml实例代码
Feb 07 #Python
python合并文本文件示例
Feb 07 #Python
python实现哈希表
Feb 07 #Python
python处理cookie详解
Feb 07 #Python
urllib2自定义opener详解
Feb 07 #Python
python解析html开发库pyquery使用方法
Feb 07 #Python
You might like
基于mysql的bbs设计(三)
2006/10/09 PHP
在yii中新增一个用户验证的方法详解
2013/06/20 PHP
浅谈php扩展imagick
2014/06/02 PHP
php中json_encode UTF-8中文乱码的更好解决方法
2014/09/28 PHP
支付宝支付开发――当面付条码支付和扫码支付实例
2016/11/04 PHP
thinkPHP5.1框架路由::get、post请求简单用法示例
2019/05/06 PHP
幻宇的层模拟窗口效果-提供演示和下载
2007/01/20 Javascript
JavaScript DOM 添加事件
2009/02/14 Javascript
jquery.bgiframe.js在IE9下提示INVALID_CHARACTER_ERR错误
2013/01/11 Javascript
JavaScript调用ajax获取文本文件内容实现代码
2014/03/28 Javascript
jQuery EasyUI Dialog拖不下来如何解决
2015/09/28 Javascript
jQuery实现简单隔行变色的方法
2016/02/20 Javascript
NodeJS配置HTTPS服务实例分享
2017/02/19 NodeJs
详解vue事件对象、冒泡、阻止默认行为
2017/03/20 Javascript
详解React开发中使用require.ensure()按需加载ES6组件
2017/05/12 Javascript
Ionic3 UI组件之autocomplete详解
2017/06/08 Javascript
详解在Vue中如何使用axios跨域访问数据
2017/07/07 Javascript
vee-validate vue 2.0自定义表单验证的实例
2018/08/28 Javascript
JS/HTML5游戏常用算法之碰撞检测 地图格子算法实例详解
2018/12/12 Javascript
Node.js+Vue脚手架环境搭建的方法步骤
2020/03/08 Javascript
JS中准确判断变量类型的方法
2020/06/01 Javascript
vue 实现tab切换保持数据状态
2020/07/21 Javascript
全网小程序接口请求封装实例代码
2020/11/06 Javascript
[46:25]DOTA2上海特级锦标赛主赛事日 - 4 败者组第五轮 MVP.Phx VS EG第二局
2016/03/05 DOTA
Python语言实现百度语音识别API的使用实例
2017/12/13 Python
python http接口自动化脚本详解
2018/01/02 Python
python里dict变成list实例方法
2019/06/26 Python
python之拟合的实现
2019/07/19 Python
Django实现web端tailf日志文件功能及实例详解
2019/07/28 Python
在spyder IPython console中,运行代码加入参数的实例
2020/04/20 Python
详解CSS3中@media的实际使用
2015/08/04 HTML / CSS
新春寄语大全
2014/04/09 职场文书
2014年“世界无车日”活动方案
2014/09/21 职场文书
无保留意见审计报告
2015/06/05 职场文书
八年级语文教学反思
2016/03/03 职场文书
详细了解MVC+proxy
2021/07/09 Java/Android