Python爬虫包BeautifulSoup实例(三)


Posted in Python onJune 17, 2018

一步一步构建一个爬虫实例,抓取糗事百科的段子

先不用beautifulsoup包来进行解析

第一步,访问网址并抓取源码

# -*- coding: utf-8 -*-
# @Author: HaonanWu
# @Date:  2016-12-22 16:16:08
# @Last Modified by:  HaonanWu
# @Last Modified time: 2016-12-22 20:17:13

import urllib
import urllib2
import re
import os

if __name__ == '__main__':
  # 访问网址并抓取源码
  url = 'http://www.qiushibaike.com/textnew/page/1/?s=4941357'
  user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36'
  headers = {'User-Agent':user_agent}
  try:
    request = urllib2.Request(url = url, headers = headers)
    response = urllib2.urlopen(request)
    content = response.read()
  except urllib2.HTTPError as e:
    print e
    exit()
  except urllib2.URLError as e:
    print e
    exit()
  print content.decode('utf-8')

第二步,利用正则表达式提取信息

首先先观察源码中,你需要的内容的位置以及如何识别
然后用正则表达式去识别读取
注意正则表达式中的 . 是不能匹配\n的,所以需要设置一下匹配模式。

# -*- coding: utf-8 -*-
# @Author: HaonanWu
# @Date:  2016-12-22 16:16:08
# @Last Modified by:  HaonanWu
# @Last Modified time: 2016-12-22 20:17:13

import urllib
import urllib2
import re
import os

if __name__ == '__main__':
  # 访问网址并抓取源码
  url = 'http://www.qiushibaike.com/textnew/page/1/?s=4941357'
  user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36'
  headers = {'User-Agent':user_agent}
  try:
    request = urllib2.Request(url = url, headers = headers)
    response = urllib2.urlopen(request)
    content = response.read()
  except urllib2.HTTPError as e:
    print e
    exit()
  except urllib2.URLError as e:
    print e
    exit()

  regex = re.compile('<div class="content">.*?<span>(.*?)</span>.*?</div>', re.S)
  items = re.findall(regex, content)

  # 提取数据
  # 注意换行符,设置 . 能够匹配换行符
  for item in items:
    print item

第三步,修正数据并保存到文件中

# -*- coding: utf-8 -*-
# @Author: HaonanWu
# @Date:  2016-12-22 16:16:08
# @Last Modified by:  HaonanWu
# @Last Modified time: 2016-12-22 21:41:32

import urllib
import urllib2
import re
import os

if __name__ == '__main__':
  # 访问网址并抓取源码
  url = 'http://www.qiushibaike.com/textnew/page/1/?s=4941357'
  user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36'
  headers = {'User-Agent':user_agent}
  try:
    request = urllib2.Request(url = url, headers = headers)
    response = urllib2.urlopen(request)
    content = response.read()
  except urllib2.HTTPError as e:
    print e
    exit()
  except urllib2.URLError as e:
    print e
    exit()

  regex = re.compile('<div class="content">.*?<span>(.*?)</span>.*?</div>', re.S)
  items = re.findall(regex, content)

  # 提取数据
  # 注意换行符,设置 . 能够匹配换行符
  path = './qiubai'
  if not os.path.exists(path):
    os.makedirs(path)
  count = 1
  for item in items:
    #整理数据,去掉\n,将<br/>换成\n
    item = item.replace('\n', '').replace('<br/>', '\n')
    filepath = path + '/' + str(count) + '.txt'
    f = open(filepath, 'w')
    f.write(item)
    f.close()
    count += 1

第四步,将多个页面下的内容都抓取下来

# -*- coding: utf-8 -*-
# @Author: HaonanWu
# @Date:  2016-12-22 16:16:08
# @Last Modified by:  HaonanWu
# @Last Modified time: 2016-12-22 20:17:13

import urllib
import urllib2
import re
import os

if __name__ == '__main__':
  # 访问网址并抓取源码
  path = './qiubai'
  if not os.path.exists(path):
    os.makedirs(path)
  user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36'
  headers = {'User-Agent':user_agent}
  regex = re.compile('<div class="content">.*?<span>(.*?)</span>.*?</div>', re.S)
  count = 1
  for cnt in range(1, 35):
    print '第' + str(cnt) + '轮'
    url = 'http://www.qiushibaike.com/textnew/page/' + str(cnt) + '/?s=4941357'
    try:
      request = urllib2.Request(url = url, headers = headers)
      response = urllib2.urlopen(request)
      content = response.read()
    except urllib2.HTTPError as e:
      print e
      exit()
    except urllib2.URLError as e:
      print e
      exit()
    # print content

    # 提取数据
    # 注意换行符,设置 . 能够匹配换行符
    items = re.findall(regex, content)

    # 保存信息
    for item in items:
      #  print item
      #整理数据,去掉\n,将<br/>换成\n
      item = item.replace('\n', '').replace('<br/>', '\n')
      filepath = path + '/' + str(count) + '.txt'
      f = open(filepath, 'w')
      f.write(item)
      f.close()
      count += 1

  print '完成'

使用BeautifulSoup对源码进行解析

# -*- coding: utf-8 -*-
# @Author: HaonanWu
# @Date:  2016-12-22 16:16:08
# @Last Modified by:  HaonanWu
# @Last Modified time: 2016-12-22 21:34:02

import urllib
import urllib2
import re
import os
from bs4 import BeautifulSoup

if __name__ == '__main__':
  url = 'http://www.qiushibaike.com/textnew/page/1/?s=4941357'
  user_agent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36'
  headers = {'User-Agent':user_agent}
  request = urllib2.Request(url = url, headers = headers)
  response = urllib2.urlopen(request)
  # print response.read()
  soup_packetpage = BeautifulSoup(response, 'lxml')
  items = soup_packetpage.find_all("div", class_="content")

  for item in items:
    try:
      content = item.span.string
    except AttributeError as e:
      print e
      exit()

    if content:
      print content + "\n"

这是用BeautifulSoup去抓取书本以及其价格的代码
可以通过对比得出到bs4对标签的读取以及标签内容的读取
(因为我自己也没有学到这一部分,目前只能依葫芦画瓢地写)

# -*- coding: utf-8 -*-
# @Author: HaonanWu
# @Date:  2016-12-22 20:37:38
# @Last Modified by:  HaonanWu
# @Last Modified time: 2016-12-22 21:27:30
import urllib2
import urllib
import re 

from bs4 import BeautifulSoup 


url = "https://www.packtpub.com/all"
try:
  html = urllib2.urlopen(url) 
except urllib2.HTTPError as e:
  print e
  exit()

soup_packtpage = BeautifulSoup(html, 'lxml') 
all_book_title = soup_packtpage.find_all("div", class_="book-block-title") 

price_regexp = re.compile(u"\s+\$\s\d+\.\d+") 

for book_title in all_book_title: 
  try:
    print "Book's name is " + book_title.string.strip()
  except AttributeError as e:
    print e
    exit()
  book_price = book_title.find_next(text=price_regexp) 
  try:
    print "Book's price is "+ book_price.strip()
  except AttributeError as e:
    print e
    exit()
  print ""

以上全部为本篇文章的全部内容,希望对大家的学习有所帮助,也希望大家多多支持三水点靠木。

Python 相关文章推荐
python读取TXT到数组及列表去重后按原来顺序排序的方法
Jun 26 Python
从源码解析Python的Flask框架中request对象的用法
Jun 02 Python
Python实现曲线点抽稀算法的示例
Oct 12 Python
Linux下python与C++使用dlib实现人脸检测
Jun 29 Python
在Python中pandas.DataFrame重置索引名称的实例
Nov 06 Python
python对于requests的封装方法详解
Jan 03 Python
通过 Python 和 OpenCV 实现目标数量监控
Jan 05 Python
关于Tensorflow分布式并行策略
Feb 03 Python
python如何提取英语pdf内容并翻译
Mar 03 Python
Django框架models使用group by详解
Mar 11 Python
Python图像处理之膨胀与腐蚀的操作
Feb 07 Python
Python道路车道线检测的实现
Jun 27 Python
Python爬虫包BeautifulSoup异常处理(二)
Jun 17 #Python
Python爬虫包BeautifulSoup简介与安装(一)
Jun 17 #Python
python主线程捕获子线程的方法
Jun 17 #Python
Python实现获取邮箱内容并解析的方法示例
Jun 16 #Python
Python实现自定义函数的5种常见形式分析
Jun 16 #Python
Python基于jieba库进行简单分词及词云功能实现方法
Jun 16 #Python
Python实现简单的文本相似度分析操作详解
Jun 16 #Python
You might like
又一个php 分页类实现代码
2009/12/03 PHP
PHP中创建图像并绘制文字的例子
2014/11/19 PHP
php判断linux下程序问题实例
2015/07/09 PHP
PHP实现简单爬虫的方法
2015/07/29 PHP
Yii+MYSQL锁表防止并发情况下重复数据的方法
2016/07/14 PHP
前端开发的开始---基于面向对象的Ajax类
2010/09/17 Javascript
初窥JQuery(二) 事件机制(1)
2010/11/25 Javascript
jquery显示和隐藏div特效实例
2013/02/27 Javascript
JS测试显示屏分辨率以及屏幕尺寸的方法
2013/11/22 Javascript
jquery背景跟随鼠标滑动导航
2015/11/20 Javascript
JavaScript中循环遍历Array与Map的方法小结
2016/03/12 Javascript
JS获取元素多层嵌套思路详解
2016/05/16 Javascript
分享JavaScript监听全部Ajax请求事件的方法
2016/08/28 Javascript
详解vue中this.$emit()的返回值是什么
2019/04/07 Javascript
使用JQuery自动完成插件Auto Complete详解
2019/06/18 jQuery
JavaScript DOM常用操作代码汇总
2020/07/03 Javascript
解决vue中axios设置超时(超过5分钟)没反应的问题
2020/09/04 Javascript
windows如何把已安装的nodejs高版本降级为低版本(图文教程)
2020/12/14 NodeJs
python中循环语句while用法实例
2015/05/16 Python
python求最大连续子数组的和
2018/07/07 Python
解决Pandas的DataFrame输出截断和省略的问题
2019/02/08 Python
python的range和linspace使用详解
2019/11/27 Python
浅析Python模块之间的相互引用问题
2021/02/26 Python
html5实现canvas阴影效果示例
2014/05/07 HTML / CSS
电影T恤、80年代T恤和80年代服装:TV Store Online
2020/01/05 全球购物
数据员岗位职责
2013/11/19 职场文书
端午节活动策划方案
2014/03/09 职场文书
小学生节约用水倡议书
2014/05/15 职场文书
药剂专业毕业生求职信
2014/06/24 职场文书
学校感恩教育活动总结
2014/07/07 职场文书
平安工地汇报材料
2014/08/19 职场文书
教师党员自我剖析材料
2014/09/29 职场文书
优秀的商业计划书,让融资一步到位
2019/05/07 职场文书
【海涛解说】暗牧也疯狂,牛蛙成配角
2022/04/01 DOTA
Redis基本数据类型String常用操作命令
2022/06/01 Redis
python读取mat文件生成h5文件的实现
2022/07/15 Python