首页
所有分类
TAGS
文字工具 EMOJI BIBLE
编程 Python

Python爬虫库BeautifulSoup的介绍与简单使用实例

Posted in Python onJanuary 25, 2020

一、介绍

BeautifulSoup库是灵活又方便的网页解析库，处理高效，支持多种解析器。利用它不用编写正则表达式即可方便地实现网页信息的提取。

Python常用解析库

解析器 使用方法 优势 劣势
Python标准库 BeautifulSoup(markup, “html.parser”) Python的内置标准库、执行速度适中 、文档容错能力强 Python 2.7.3 or 3.2.2)前的版本中文容错能力差
lxml HTML 解析器 BeautifulSoup(markup, “lxml”) 速度快、文档容错能力强 需要安装C语言库
lxml XML 解析器 BeautifulSoup(markup, “xml”) 速度快、唯一支持XML的解析器 需要安装C语言库
html5lib BeautifulSoup(markup, “html5lib”) 最好的容错性、以浏览器的方式解析文档、生成HTML5格式的文档 速度慢、不依赖外部扩展

二、快速开始

给定html文档，产生BeautifulSoup对象

from bs4 import BeautifulSoup
html_doc = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>

<p class="story">...</p>
"""
soup = BeautifulSoup(html_doc,'lxml')

输出完整文本

print(soup.prettify())
<html>
 <head>
 <title>
  The Dormouse's story
 </title>
 </head>
 <body>
 <p class="title">
  <b>
  The Dormouse's story
  </b>
 </p>
 <p class="story">
  Once upon a time there were three little sisters; and their names were
  <a class="sister" href="http://example.com/elsie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">
  Elsie
  </a>
  ,
  <a class="sister" href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">
  Lacie
  </a>
  and
  <a class="sister" href="http://example.com/tillie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">
  Tillie
  </a>
  ;
and they lived at the bottom of a well.
 </p>
 <p class="story">
  ...
 </p>
 </body>
</html>

浏览结构化数据

print(soup.title) #<title>标签及内容
print(soup.title.name) #<title>name属性
print(soup.title.string) #<title>内的字符串
print(soup.title.parent.name) #<title>的父标签name属性(head)
print(soup.p) # 第一个<p></p>
print(soup.p['class']) #第一个<p></p>的class
print(soup.a) # 第一个<a></a>
print(soup.find_all('a')) # 所有<a></a>
print(soup.find(id="link3")) # 所有id='link3'的标签
<title>The Dormouse's story</title>
title
The Dormouse's story
head
<p class="title"><b>The Dormouse's story</b></p>
['title']
<a class="sister" href="http://example.com/elsie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">Elsie</a>
[<a class="sister" href="http://example.com/elsie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">Elsie</a>, <a class="sister" href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>, <a class="sister" href="http://example.com/tillie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>]
<a class="sister" href="http://example.com/tillie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>

找出所有标签内的链接

for link in soup.find_all('a'):
  print(link.get('href'))
http://example.com/elsie
http://example.com/lacie
http://example.com/tillie

获得所有文字内容

print(soup.get_text())
The Dormouse's story

The Dormouse's story
Once upon a time there were three little sisters; and their names were
Elsie,
Lacie and
Tillie;
and they lived at the bottom of a well.
...

自动补全标签并进行格式化

html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')#传入解析器：lxml
print(soup.prettify())#格式化代码，自动补全
print(soup.title.string)#得到title标签里的内容

标签选择器

选择元素

html = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title" name="dromouse"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister" id="link1"><!-- Elsie --></a>,
<a href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>
<p class="story">...</p>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')#传入解析器：lxml
print(soup.title)#选择了title标签
print(type(soup.title))#查看类型
print(soup.head)

获取标签名称

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')#传入解析器：lxml
print(soup.title.name)

获取标签属性

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')#传入解析器：lxml
print(soup.p.attrs['name'])#获取p标签中，name这个属性的值
print(soup.p['name'])#另一种写法，比较直接

获取标签内容

print(soup.p.string)

标签嵌套选择

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')#传入解析器：lxml
print(soup.head.title.string)

子节点和子孙节点

html = """
<html>
  <head>
    <title>The Dormouse's story</title>
  </head>
  <body>
    <p class="story">
      Once upon a time there were three little sisters; and their names were
      <a href="http://example.com/elsie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister" id="link1">
        <span>Elsie</span>
      </a>
      <a href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister" id="link2">Lacie</a> 
      and
      <a href="http://example.com/tillie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister" id="link3">Tillie</a>
      and they lived at the bottom of a well.
    </p>
    <p class="story">...</p>
"""


from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')#传入解析器：lxml
print(soup.p.contents)#获取指定标签的子节点，类型是list

另一个方法，child：

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')#传入解析器：lxml
print(soup.p.children)#获取指定标签的子节点的迭代器对象
for i,children in enumerate(soup.p.children):#i接受索引，children接受内容
	print(i,children)

输出结果与上面的一样，多了一个索引。注意，只能用循环来迭代出子节点的信息。因为直接返回的只是一个迭代器对象。

获取子孙节点：

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')#传入解析器：lxml
print(soup.p.descendants)#获取指定标签的子孙节点的迭代器对象
for i,child in enumerate(soup.p.descendants):#i接受索引，child接受内容
	print(i,child)

父节点和祖先节点

parent

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')#传入解析器：lxml
print(soup.a.parent)#获取指定标签的父节点

parents

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')#传入解析器：lxml
print(list(enumerate(soup.a.parents)))#获取指定标签的祖先节点

兄弟节点

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')#传入解析器：lxml
print(list(enumerate(soup.a.next_siblings)))#获取指定标签的后面的兄弟节点
print(list(enumerate(soup.a.previous_siblings)))#获取指定标签的前面的兄弟节点

标准选择器

find_all( name , attrs , recursive , text , **kwargs )

可根据标签名、属性、内容查找文档。

name

html='''
<div class="panel">
  <div class="panel-heading">
    <h4>Hello</h4>
  </div>
  <div class="panel-body">
    <ul class="list" id="list-1">
      <li class="element">Foo</li>
      <li class="element">Bar</li>
      <li class="element">Jay</li>
    </ul>
    <ul class="list list-small" id="list-2">
      <li class="element">Foo</li>
      <li class="element">Bar</li>
    </ul>
  </div>
</div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.find_all('ul'))#查找所有ul标签下的内容
print(type(soup.find_all('ul')[0]))#查看其类型

下面的例子就是查找所有ul标签下的li标签：

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
for ul in soup.find_all('ul'):
  print(ul.find_all('li'))

attrs（属性）

通过属性进行元素的查找

html='''
<div class="panel">
  <div class="panel-heading">
    <h4>Hello</h4>
  </div>
  <div class="panel-body">
    <ul class="list" id="list-1" name="elements">
      <li class="element">Foo</li>
      <li class="element">Bar</li>
      <li class="element">Jay</li>
    </ul>
    <ul class="list list-small" id="list-2">
      <li class="element">Foo</li>
      <li class="element">Bar</li>
    </ul>
  </div>
</div>
'''


from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.find_all(attrs={'id': 'list-1'}))#传入的是一个字典类型，也就是想要查找的属性
print(soup.find_all(attrs={'name': 'elements'}))

查找到的是同样的内容，因为这两个属性是在同一个标签里面的。

特殊类型的参数查找：

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.find_all(id='list-1'))#id是个特殊的属性，可以直接使用
print(soup.find_all(class_='element')) #class是关键字所以要用class_

text

根据文本内容来进行选择：

html='''
<div class="panel">
  <div class="panel-heading">
    <h4>Hello</h4>
  </div>
  <div class="panel-body">
    <ul class="list" id="list-1">
      <li class="element">Foo</li>
      <li class="element">Bar</li>
      <li class="element">Jay</li>
    </ul>
    <ul class="list list-small" id="list-2">
      <li class="element">Foo</li>
      <li class="element">Bar</li>
    </ul>
  </div>
</div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.find_all(text='Foo'))#查找文本为Foo的内容，但是返回的不是标签

所以说这个text在做内容匹配的时候比较方便，但是在做内容查找的时候并不是太方便。

方法

find

find用法和findall一模一样，但是返回的是找到的第一个符合条件的内容输出。

ind_parents()， find_parent()

find_parents()返回所有祖先节点，find_parent()返回直接父节点。

find_next_siblings() ,find_next_sibling()

find_next_siblings()返回后面的所有兄弟节点，find_next_sibling()返回后面的第一个兄弟节点

find_previous_siblings(),find_previous_sibling()

find_previous_siblings()返回前面所有兄弟节点,find_previous_sibling()返回前面第一个兄弟节点

find_all_next(),find_next()

find_all_next()返回节点后所有符合条件的节点，find_next()返回后面第一个符合条件的节点

find_all_previous(),find_previous()

find_all_previous()返回节点前所有符合条件的节点，find_previous()返回前面第一个符合条件的节点

CSS选择器 通过select()直接传入CSS选择器即可完成选择

html='''
<div class="panel">
  <div class="panel-heading">
    <h4>Hello</h4>
  </div>
  <div class="panel-body">
    <ul class="list" id="list-1">
      <li class="element">Foo</li>
      <li class="element">Bar</li>
      <li class="element">Jay</li>
    </ul>
    <ul class="list list-small" id="list-2">
      <li class="element">Foo</li>
      <li class="element">Bar</li>
    </ul>
  </div>
</div>
'''
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
print(soup.select('.panel .panel-heading'))#.代表class，中间需要空格来分隔
print(soup.select('ul li')) #选择ul标签下面的li标签
print(soup.select('#list-2 .element')) #'#'代表id。这句的意思是查找id为"list-2"的标签下的，class=element的元素
print(type(soup.select('ul')[0]))#打印节点类型

再看看层层嵌套的选择：

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
for ul in soup.select('ul'):
	print(ul.select('li'))

获取属性

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
for ul in soup.select('ul'):
  print(ul['id'])# 用[ ]即可获取属性
  print(ul.attrs['id'])#另一种写法

获取内容

from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'lxml')
for li in soup.select('li'):
  print(li.get_text())

用get_text（）方法就能获取内容了。

总结

推荐使用lxml解析库，必要时使用html.parser

标签选择筛选功能弱但是速度快 建议使用find()、find_all() 查询匹配单个结果或者多个结果

如果对CSS选择器熟悉建议使用select()

记住常用的获取属性和文本值的方法

更多关于Python爬虫库BeautifulSoup的介绍与简单使用实例请点击下面的相关链接

Python爬虫库BeautifulSoup的介绍与简单使用实例

- Author -

BQW_

声明：登载此文出于传递更多信息之目的，并不意味着赞同其观点或证实其描述。

Python 相关文章推荐
Cython 三分钟入门教程
Sep 17 Python
Python下线程之间的共享和释放示例
May 04 Python
Python实现的简单模板引擎功能示例
Sep 02 Python
Python+OpenCV实现车牌字符分割和识别
Mar 31 Python
python中cPickle类使用方法详解
Aug 27 Python
python绘制地震散点图
Jun 18 Python
python多线程http压力测试脚本
Jun 25 Python
python中的decimal类型转换实例详解
Jun 26 Python
django将数组传递给前台模板的方法
Aug 06 Python
Numpy的简单用法小结
Aug 28 Python
Python Opencv轮廓常用操作代码实例解析
Sep 01 Python
Python三维绘图之Matplotlib库的使用方法
Sep 20 Python
使用Python爬虫库requests发送表单数据和JSON数据
Jan 25 #Python
Python爬虫库requests获取响应内容、响应状态码、响应头
Jan 25 #Python
使用Python爬虫库requests发送请求、传递URL参数、定制headers
Jan 25 #Python
flask框架自定义url转换器操作详解
Jan 25 #Python
常用python爬虫库介绍与简要说明
Jan 25 #Python
flask框架url与重定向操作实例详解
Jan 25 #Python
flask框架蓝图和子域名配置详解
Jan 25 #Python
沙特阿拉伯购物网站(9) 匈牙利购物网站(11) 秘鲁购物网站(1) 捷克购物网站(26) 卡塔尔购物网站(3) 斯洛伐克购物网站(7) 智利购物网站(5) 菲律宾购物网站(13) 波兰购物网站(52) 新西兰购物网站(42)
You might like
ThinkPHP Mobile使用方法简明教程
2014/06/18 PHP
php生成数字字母的验证码图片
2015/07/14 PHP
解决Laravel5.5下的toArray问题
2019/10/15 PHP
javascript中的对象和数组的应用技巧
2007/01/07 Javascript
jquery 事件冒泡的介绍以及如何阻止事件冒泡
2012/12/25 Javascript
Javascript跨域请求的4种解决方式
2013/03/17 Javascript
javascript(js)的小数点乘法除法问题详解
2014/03/07 Javascript
javascript 小数取整简单实现方式
2014/05/30 Javascript
动态加载js的方法汇总
2015/02/13 Javascript
jQuery焦点图切换特效代码分享
2015/09/15 Javascript
angular2使用简单介绍
2016/03/01 Javascript
深入浅析jQuery对象$.html
2016/08/22 Javascript
three.js中文文档学习之如何本地运行详解
2017/11/20 Javascript
微信小程序基于slider组件动态修改标签透明度的方法示例
2017/12/04 Javascript
vue2.0实现移动端的输入框实时检索更新列表功能
2018/05/08 Javascript
JS插件clipboard.js实现一键复制粘贴功能
2020/12/04 Javascript
Node.js EventEmmitter事件监听器用法实例分析
2019/01/07 Javascript
vue2.0项目集成Cesium的实现方法
2019/07/30 Javascript
NodeJS http模块用法示例【创建web服务器/客户端】
2019/11/05 NodeJs
JS实现多功能计算器
2020/10/28 Javascript
python实现两个文件合并功能
2018/04/01 Python
Python 中 function(#) (X)格式 和 (#)在Python3.*中的注意事项
2018/11/30 Python
python实现五子棋游戏
2019/06/18 Python
如何用Python制作微信好友个性签名词云图
2019/06/28 Python
解决Keras的自定义lambda层去reshape张量时model保存出错问题
2020/07/01 Python
使用CSS3制作版头动画效果
2020/12/24 HTML / CSS
荷兰在线钓鱼商店：Raven
2019/06/26 全球购物
计算机专业毕业生自我鉴定
2014/01/16 职场文书
人事主管岗位职责
2014/01/30 职场文书
商铺消防安全责任书
2014/07/29 职场文书
2014院党委领导班子对照检查材料思想汇报
2014/09/24 职场文书
合作协议书模板2014
2014/09/26 职场文书
2014年幼儿园后勤工作总结
2014/11/10 职场文书
2015年销售助理工作总结
2015/05/11 职场文书
在Windows Server 2012上安装 .NET Framework 3.5 所遇到的问题
2022/04/29 Servers
HttpClient实现文件上传功能
2022/08/14 Java/Android