python网页请求urllib2模块简单封装代码


Posted in Python onFebruary 07, 2014

对python网页请求模块urllib2进行简单的封装。

例子:

#!/usr/bin/python
#coding: utf-8
import base64
import urllib
import urllib2
import time
class SendRequest:
  '''
  This class use to set and request the http, and get the info of response.
  e.g. set Authorization Type, request tyep..
  e.g. get html content, state code, cookie..
  SendRequest('http://10.75.0.103:8850/2/photos/square/type.json', 
              data='source=216274069', type='POST', auth='base',
     user='zl2010', password='111111')
  '''
  def __init__(self, url, data=None, type='GET', auth=None, user=None, password=None, cookie = None, **header):
    '''
    url:request, raise error if none
    date: data for post or get, must be dict type
    type: GET, POST
    auth: option, if has the value must be 'base' or 'cookie'
    user: user for auth
    password: password for auth
    cookie: if request with cookie 
    other header info: 
    e.g. referer='www.sina.com.cn'    
    '''
    self.url = url
    self.data = data
    self.type = type
    self.auth = auth
    self.user = user
    self.password = password 
    self.cookie = cookie 
    if 'referer' in header:
      self.referer = header[referer]
    else:
      self.referer = None
    if 'user-agent' in header:
      self.user_agent = header[user-agent]
    else:
      self.user_agent = None
    self.setup_request()
    self.send_request() 
  def setup_request(self):
    '''
    setup a request 
    '''
    if self.url == None or self.url == '':
      raise 'The url should not empty!'
    # set request type 
    #print self.url
    #print self.type
    #print self.data
    #print self.auth
    #print self.user
    #print self.password  
    if self.type == 'POST': 
      self.Req = urllib2.Request(self.url, self.data)
    elif self.type == 'GET':
      if self.data == None:
          self.Req = urllib2.Request(self.url)
      else:
        self.Req = urllib2.Request(self.url + '?' + self.data)
    else:
      print 'The http request type NOT support now!'
    ##set auth type 
    if self.auth == 'base':
      if self.user == None or self.password == None:
        raise 'The user or password was not given!'
      else:
        auth_info = base64.encodestring(self.user + ':' + self.password).replace('\n','')
        auth_info = 'Basic ' + auth_info 
        #print auth_info  
        self.Req.add_header("Authorization", auth_info)
    elif self.auth == 'cookie':
      if self.cookie == None:
        raise 'The cookie was not given!'
      else:
        self.Req.add_header("Cookie", self.cookie) 
    else:
      pass    ##add other auth type here 
    ##set other header info 
    if self.referer:
      self.Req.add_header('referer', self.referer)
    if self.user_agent:
      self.Req.add_header('user-agent', self.user_agent)
      
  def send_request(self):  
    '''
    send a request 
    '''
    # get a response object 
    try:
      self.Res = urllib2.urlopen(self.Req)
      self.source = self.Res.read()
      self.goal_url = self.Res.geturl()
      self.code = self.Res.getcode()
      self.head_dict = self.Res.info().dict
      self.Res.close()
    except urllib2.HTTPError, e:
      self.code = e.code
      print e
        
  def get_code(self):
    return self.code
  def get_url(self):
    return self.goal_url
  def get_source(self):        
    return self.source
  def get_header_info(self):
    return self.head_dict
  def get_cookie(self):
    if 'set-cookie' in self.head_dict:
      return self.head_dict['set-cookie']
    else:
      return None    
  def get_content_type(self):
    if 'content-type' in self.head_dict:
      return self.head_dict['content-type']
    else:
      return None
  def get_expires_time(self):
    if 'expires' in self.head_dict:
      return self.head_dict['expires']
    else:
      return None    
  def get_server_name(self):
    if 'server' in self.head_dict:
      return self.head_dict['server']
    else:
      return None   
  def __del__(self):
    pass   
__all__ = [SendRequest,]
if __name__ == '__main__':
  '''
  The example for using the SendRequest class 
  '''
  value = {'source':'216274069'}
  data = urllib.urlencode(value)
  url = 'http://10.75.0.103:8850/2/photos/square/type.json'
  user = 'wz_0001'
  password = '111111'
  auth = 'base'
  type = 'POST'
  t2 = time.time()
  rs = SendRequest('http://www.google.com')
  #rs = SendRequest(url, data=data, type=type, auth=auth, user=user, password=password)
  print 't2: ' + str(time.time() - t2)
  print '---------------get_code()---------------'
  print rs.get_code()
  print '---------------get_url()---------------'
  print rs.get_url()
  print '---------------get_source()---------------'
  print rs.get_source()
  print '---------------get_cookie()---------------'
  print rs.get_cookie()
  rs = None
Python 相关文章推荐
django数据库migrate失败的解决方法解析
Feb 08 Python
分析python动态规划的递归、非递归实现
Mar 04 Python
Python列表生成式与生成器操作示例
Aug 01 Python
python opencv摄像头的简单应用
Jun 06 Python
python替换字符串中的子串图文步骤
Jun 19 Python
python 梯度法求解函数极值的实例
Jul 10 Python
python 爬取马蜂窝景点翻页文字评论的实现
Jan 20 Python
python多进程使用函数封装实例
May 02 Python
Python调用.net动态库实现过程解析
Jun 05 Python
python如何编写win程序
Jun 08 Python
Python卷积神经网络图片分类框架详解分析
Nov 07 Python
详解Python flask的前后端交互
Mar 31 Python
python解析xml模块封装代码
Feb 07 #Python
python 解析XML python模块xml.dom解析xml实例代码
Feb 07 #Python
python合并文本文件示例
Feb 07 #Python
python实现哈希表
Feb 07 #Python
python处理cookie详解
Feb 07 #Python
urllib2自定义opener详解
Feb 07 #Python
python解析html开发库pyquery使用方法
Feb 07 #Python
You might like
PHP如何使用cURL实现Get和Post请求
2020/07/11 PHP
jQuery判断iframe中元素是否存在的方法
2013/05/11 Javascript
动态获取复选框checkbox选中个数的jquery代码
2013/06/25 Javascript
JavaScript截取字符串的Slice、Substring、Substr函数详解和比较
2014/03/20 Javascript
javascript跑马灯抽奖实例讲解
2020/04/17 Javascript
基于canvas实现的绚丽圆圈效果完整实例
2016/01/26 Javascript
js表单元素checked、radio被选中的几种方法(详解)
2016/08/22 Javascript
Angularjs 动态改变title标题(兼容ios)
2016/12/29 Javascript
jQuery滚动插件scrollable.js用法分析
2017/05/25 jQuery
JS switch判断 三目运算 while 及 属性操作代码
2017/09/03 Javascript
聊聊那些使用前端Javascript实现的机器学习类库
2017/09/18 Javascript
微信小程序下拉框组件使用方法详解
2018/12/28 Javascript
微信小程序实现页面浮动导航
2019/01/28 Javascript
原生JavaScript实现五子棋游戏
2020/11/09 Javascript
Python实现带参数与不带参数的多重继承示例
2018/01/30 Python
Python3.x爬虫下载网页图片的实例讲解
2018/05/22 Python
基于django channel实现websocket的聊天室的方法示例
2019/04/11 Python
Python+opencv+pyaudio实现带声音屏幕录制
2019/12/23 Python
Python logging模块handlers用法详解
2020/08/14 Python
python中pyplot基础图标函数整理
2020/11/10 Python
英国口碑最好的的维他命胶囊品牌:Myvitamins(有中文站)
2016/12/03 全球购物
Richards网上商店:当代时尚,遍布巴西
2019/11/03 全球购物
如何写一个Java类既可以用作applet也可以用作java应用
2016/01/18 面试题
毕业生的自我鉴定该怎么写
2013/12/02 职场文书
优秀员工评语
2014/02/10 职场文书
房产代理公证处委托书
2014/04/04 职场文书
开学寄语大全
2014/04/08 职场文书
幼儿园家长评语大全
2014/04/16 职场文书
庆元旦演讲稿
2014/09/15 职场文书
关于运动会的广播稿50字
2014/10/17 职场文书
会计人员岗位职责
2015/02/03 职场文书
幼儿园六一儿童节活动总结
2015/02/10 职场文书
2015年社区纪检工作总结
2015/04/21 职场文书
写给纪委的违纪检讨书
2015/05/05 职场文书
寻找最美乡村教师观后感
2015/06/18 职场文书
《鲁班学艺》读后感3篇
2019/11/27 职场文书