Python爬虫包BeautifulSoup实例(三)


Posted in Python onJune 17, 2018

一步一步构建一个爬虫实例,抓取糗事百科的段子

先不用beautifulsoup包来进行解析

第一步,访问网址并抓取源码

# -*- coding: utf-8 -*-
# @Author: HaonanWu
# @Date:  2016-12-22 16:16:08
# @Last Modified by:  HaonanWu
# @Last Modified time: 2016-12-22 20:17:13

import urllib
import urllib2
import re
import os

if __name__ == '__main__':
  # 访问网址并抓取源码
  url = 'http://www.qiushibaike.com/textnew/page/1/?s=4941357'
  user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36'
  headers = {'User-Agent':user_agent}
  try:
    request = urllib2.Request(url = url, headers = headers)
    response = urllib2.urlopen(request)
    content = response.read()
  except urllib2.HTTPError as e:
    print e
    exit()
  except urllib2.URLError as e:
    print e
    exit()
  print content.decode('utf-8')

第二步,利用正则表达式提取信息

首先先观察源码中,你需要的内容的位置以及如何识别
然后用正则表达式去识别读取
注意正则表达式中的 . 是不能匹配\n的,所以需要设置一下匹配模式。

# -*- coding: utf-8 -*-
# @Author: HaonanWu
# @Date:  2016-12-22 16:16:08
# @Last Modified by:  HaonanWu
# @Last Modified time: 2016-12-22 20:17:13

import urllib
import urllib2
import re
import os

if __name__ == '__main__':
  # 访问网址并抓取源码
  url = 'http://www.qiushibaike.com/textnew/page/1/?s=4941357'
  user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36'
  headers = {'User-Agent':user_agent}
  try:
    request = urllib2.Request(url = url, headers = headers)
    response = urllib2.urlopen(request)
    content = response.read()
  except urllib2.HTTPError as e:
    print e
    exit()
  except urllib2.URLError as e:
    print e
    exit()

  regex = re.compile('<div class="content">.*?<span>(.*?)</span>.*?</div>', re.S)
  items = re.findall(regex, content)

  # 提取数据
  # 注意换行符,设置 . 能够匹配换行符
  for item in items:
    print item

第三步,修正数据并保存到文件中

# -*- coding: utf-8 -*-
# @Author: HaonanWu
# @Date:  2016-12-22 16:16:08
# @Last Modified by:  HaonanWu
# @Last Modified time: 2016-12-22 21:41:32

import urllib
import urllib2
import re
import os

if __name__ == '__main__':
  # 访问网址并抓取源码
  url = 'http://www.qiushibaike.com/textnew/page/1/?s=4941357'
  user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36'
  headers = {'User-Agent':user_agent}
  try:
    request = urllib2.Request(url = url, headers = headers)
    response = urllib2.urlopen(request)
    content = response.read()
  except urllib2.HTTPError as e:
    print e
    exit()
  except urllib2.URLError as e:
    print e
    exit()

  regex = re.compile('<div class="content">.*?<span>(.*?)</span>.*?</div>', re.S)
  items = re.findall(regex, content)

  # 提取数据
  # 注意换行符,设置 . 能够匹配换行符
  path = './qiubai'
  if not os.path.exists(path):
    os.makedirs(path)
  count = 1
  for item in items:
    #整理数据,去掉\n,将<br/>换成\n
    item = item.replace('\n', '').replace('<br/>', '\n')
    filepath = path + '/' + str(count) + '.txt'
    f = open(filepath, 'w')
    f.write(item)
    f.close()
    count += 1

第四步,将多个页面下的内容都抓取下来

# -*- coding: utf-8 -*-
# @Author: HaonanWu
# @Date:  2016-12-22 16:16:08
# @Last Modified by:  HaonanWu
# @Last Modified time: 2016-12-22 20:17:13

import urllib
import urllib2
import re
import os

if __name__ == '__main__':
  # 访问网址并抓取源码
  path = './qiubai'
  if not os.path.exists(path):
    os.makedirs(path)
  user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36'
  headers = {'User-Agent':user_agent}
  regex = re.compile('<div class="content">.*?<span>(.*?)</span>.*?</div>', re.S)
  count = 1
  for cnt in range(1, 35):
    print '第' + str(cnt) + '轮'
    url = 'http://www.qiushibaike.com/textnew/page/' + str(cnt) + '/?s=4941357'
    try:
      request = urllib2.Request(url = url, headers = headers)
      response = urllib2.urlopen(request)
      content = response.read()
    except urllib2.HTTPError as e:
      print e
      exit()
    except urllib2.URLError as e:
      print e
      exit()
    # print content

    # 提取数据
    # 注意换行符,设置 . 能够匹配换行符
    items = re.findall(regex, content)

    # 保存信息
    for item in items:
      #  print item
      #整理数据,去掉\n,将<br/>换成\n
      item = item.replace('\n', '').replace('<br/>', '\n')
      filepath = path + '/' + str(count) + '.txt'
      f = open(filepath, 'w')
      f.write(item)
      f.close()
      count += 1

  print '完成'

使用BeautifulSoup对源码进行解析

# -*- coding: utf-8 -*-
# @Author: HaonanWu
# @Date:  2016-12-22 16:16:08
# @Last Modified by:  HaonanWu
# @Last Modified time: 2016-12-22 21:34:02

import urllib
import urllib2
import re
import os
from bs4 import BeautifulSoup

if __name__ == '__main__':
  url = 'http://www.qiushibaike.com/textnew/page/1/?s=4941357'
  user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36'
  headers = {'User-Agent':user_agent}
  request = urllib2.Request(url = url, headers = headers)
  response = urllib2.urlopen(request)
  # print response.read()
  soup_packetpage = BeautifulSoup(response, 'lxml')
  items = soup_packetpage.find_all("div", class_="content")

  for item in items:
    try:
      content = item.span.string
    except AttributeError as e:
      print e
      exit()

    if content:
      print content + "\n"

这是用BeautifulSoup去抓取书本以及其价格的代码
可以通过对比得出到bs4对标签的读取以及标签内容的读取
(因为我自己也没有学到这一部分,目前只能依葫芦画瓢地写)

# -*- coding: utf-8 -*-
# @Author: HaonanWu
# @Date:  2016-12-22 20:37:38
# @Last Modified by:  HaonanWu
# @Last Modified time: 2016-12-22 21:27:30
import urllib2
import urllib
import re 

from bs4 import BeautifulSoup 


url = "https://www.packtpub.com/all"
try:
  html = urllib2.urlopen(url) 
except urllib2.HTTPError as e:
  print e
  exit()

soup_packtpage = BeautifulSoup(html, 'lxml') 
all_book_title = soup_packtpage.find_all("div", class_="book-block-title") 

price_regexp = re.compile(u"\s+\$\s\d+\.\d+") 

for book_title in all_book_title: 
  try:
    print "Book's name is " + book_title.string.strip()
  except AttributeError as e:
    print e
    exit()
  book_price = book_title.find_next(text=price_regexp) 
  try:
    print "Book's price is "+ book_price.strip()
  except AttributeError as e:
    print e
    exit()
  print ""

以上全部为本篇文章的全部内容,希望对大家的学习有所帮助,也希望大家多多支持三水点靠木。

Python 相关文章推荐
在Python中使用__slots__方法的详细教程
Apr 28 Python
安装Python和pygame及相应的环境变量配置(图文教程)
Jun 04 Python
python编写暴力破解zip文档程序的实例讲解
Apr 24 Python
python斐波那契数列的计算方法
Sep 27 Python
Python爬虫实战之12306抢票开源
Jan 24 Python
如何利用Python分析出微信朋友男女统计图
Jan 25 Python
使用Python-OpenCV向图片添加噪声的实现(高斯噪声、椒盐噪声)
May 28 Python
Django中提供的6种缓存方式详解
Aug 05 Python
python zip,lambda,map函数代码实例
Apr 04 Python
python 数据分析实现长宽格式的转换
May 18 Python
利用keras使用神经网络预测销量操作
Jul 07 Python
浅析python实现动态规划背包问题
Dec 31 Python
Python爬虫包BeautifulSoup异常处理(二)
Jun 17 #Python
Python爬虫包BeautifulSoup简介与安装(一)
Jun 17 #Python
python主线程捕获子线程的方法
Jun 17 #Python
Python实现获取邮箱内容并解析的方法示例
Jun 16 #Python
Python实现自定义函数的5种常见形式分析
Jun 16 #Python
Python基于jieba库进行简单分词及词云功能实现方法
Jun 16 #Python
Python实现简单的文本相似度分析操作详解
Jun 16 #Python
You might like
php 生成随机验证码图片代码
2010/02/08 PHP
PHP环境中Memcache的安装和使用
2015/11/05 PHP
PHP 数组遍历foreach语法结构及实例
2016/06/13 PHP
php实现压缩合并js的方法【附demo源码下载】
2016/09/22 PHP
php实现XML和数组的相互转化功能示例
2017/02/08 PHP
PHP的mysqli_select_db()函数讲解
2019/01/23 PHP
thinkPHP5.1框架路由::get、post请求简单用法示例
2019/05/06 PHP
28个JS验证函数收集
2010/03/02 Javascript
jquery入门—选择器实现隔行变色实例代码
2013/01/04 Javascript
document.getElementBy(&quot;id&quot;)与$(&quot;#id&quot;)有什么区别
2013/09/22 Javascript
JS Date函数整理方便使用
2013/10/23 Javascript
js实现动态改变字体大小代码
2014/01/02 Javascript
详解webpack-dev-server 设置反向代理解决跨域问题
2018/04/18 Javascript
vue完成项目后,打包成静态文件的方法
2018/09/03 Javascript
一步快速解决微信小程序中textarea层级太高遮挡其他组件
2019/03/04 Javascript
详解Vue 匿名、具名和作用域插槽的使用方法
2019/04/22 Javascript
详解JS判断页面是在手机端还是在PC端打开的方法
2019/04/26 Javascript
layui问题之渲染数据表格时,仅出现10条数据的解决方法
2019/09/12 Javascript
JS实现图片懒加载(lazyload)过程详解
2020/04/02 Javascript
vue3+typeScript穿梭框的实现示例
2020/12/29 Vue.js
一篇文章入门Python生态系统(Python新手入门指导)
2015/12/11 Python
Python之os操作方法(详解)
2017/06/15 Python
Python 微信爬虫完整实例【单线程与多线程】
2019/07/06 Python
Python实现简单的列表冒泡排序和反转列表操作示例
2019/07/10 Python
wxPython多个窗口的基本结构
2019/11/19 Python
Python正则表达式如何匹配中文
2020/05/27 Python
canvas基础之图形验证码的示例
2018/01/02 HTML / CSS
美国领先的个性化礼品商城:Personalization Mall
2019/07/27 全球购物
亚马逊意大利站点:Amazon.it
2020/12/31 全球购物
联想C++笔试题
2012/06/13 面试题
初中三年学生的学习自我评价
2013/11/13 职场文书
党课培训心得体会
2014/09/02 职场文书
依法行政工作汇报材料
2014/10/28 职场文书
公务员政审材料
2014/12/23 职场文书
MySQL 语句执行顺序举例解析
2022/06/05 MySQL
V Rising 服务器搭建图文教程
2022/06/16 Servers