首页
所有分类
TAGS
文字工具 EMOJI BIBLE
编程 Python

使用Python爬虫库BeautifulSoup遍历文档树并对标签进行操作详解

Posted in Python onJanuary 25, 2020

下面就是使用Python爬虫库BeautifulSoup对文档树进行遍历并对标签进行操作的实例，都是最基础的内容

html_doc = """
<html><head><title>The Dormouse's story</title></head>

<p class="title"><b>The Dormouse's story</b></p>

<p class="story">Once upon a time there were three little sisters; and their names were
<a href="http://example.com/elsie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister" id="link1">Elsie</a>,
<a href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister" id="link2">Lacie</a> and
<a href="http://example.com/tillie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" class="sister" id="link3">Tillie</a>;
and they lived at the bottom of a well.</p>

<p class="story">...</p>
"""

from bs4 import BeautifulSoup
soup = BeautifulSoup(html_doc,'lxml')

一、子节点

一个Tag可能包含多个字符串或者其他Tag，这些都是这个Tag的子节点.BeautifulSoup提供了许多操作和遍历子结点的属性。

1.通过Tag的名字来获得Tag

print(soup.head)
print(soup.title)
<head><title>The Dormouse's story</title></head>
<title>The Dormouse's story</title>

通过名字的方法只能获得第一个Tag，如果要获得所有的某种Tag可以使用find_all方法

soup.find_all('a')
[<a class="sister" href="http://example.com/elsie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">Elsie</a>,
 <a class="sister" href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>,
 <a class="sister" href="http://example.com/tillie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>]

2.contents属性：将Tag的子节点通过列表的方式返回

head_tag = soup.head
head_tag.contents
[<title>The Dormouse's story</title>]
title_tag = head_tag.contents[0]
title_tag
<title>The Dormouse's story</title>
title_tag.contents
["The Dormouse's story"]

3.children：通过该属性对子节点进行循环

for child in title_tag.children:
  print(child)
The Dormouse's story

4.descendants： 不论是contents还是children都是返回直接子节点，而descendants对所有tag的子孙节点进行递归循环

for child in head_tag.children:
  print(child)
<title>The Dormouse's story</title>
for child in head_tag.descendants:
  print(child)
<title>The Dormouse's story</title>
The Dormouse's story

5.string 如果tag只有一个NavigableString类型的子节点，那么tag可以使用.string得到该子节点

title_tag.string
"The Dormouse's story"

如果一个tag只有一个子节点，那么使用.string可以获得其唯一子结点的NavigableString.

head_tag.string
"The Dormouse's story"

如果tag有多个子节点，tag无法确定.string对应的是那个子结点的内容，故返回None

print(soup.html.string)
None

6.strings和stripped_strings

如果tag包含多个字符串，可以使用.strings循环获取

for string in soup.strings:
  print(string)
The Dormouse's story


The Dormouse's story


Once upon a time there were three little sisters; and their names were

Elsie
,

Lacie
 and

Tillie
;
and they lived at the bottom of a well.


...

.string输出的内容包含了许多空格和空行，使用strpped_strings去除这些空白内容

for string in soup.stripped_strings:
  print(string)
The Dormouse's story
The Dormouse's story
Once upon a time there were three little sisters; and their names were
Elsie
,
Lacie
and
Tillie
;
and they lived at the bottom of a well.
...

二、父节点

1.parent：获得某个元素的父节点

title_tag = soup.title
title_tag.parent
<head><title>The Dormouse's story</title></head>

字符串也有父节点

title_tag.string.parent
<title>The Dormouse's story</title>

2.parents：递归的获得所有父辈节点

link = soup.a
for parent in link.parents:
  if parent is None:
    print(parent)
  else:
    print(parent.name)
p
body
html
[document]

三、兄弟结点

sibling_soup = BeautifulSoup("<a><b>text1</b><c>text2</c></b></a>",'lxml')
print(sibling_soup.prettify())
<html>
 <body>
 <a>
  <b>
  text1
  </b>
  <c>
  text2
  </c>
 </a>
 </body>
</html>

1.next_sibling和previous_sibling

sibling_soup.b.next_sibling
<c>text2</c>
sibling_soup.c.previous_sibling
<b>text1</b>

在实际文档中.next_sibling和previous_sibling通常是字符串或者空白符

soup.find_all('a')
[<a class="sister" href="http://example.com/elsie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">Elsie</a>,
 <a class="sister" href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>,
 <a class="sister" href="http://example.com/tillie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>]
soup.a.next_sibling # 第一个<a></a>的next_sibling是,\n
',\n'
soup.a.next_sibling.next_sibling
<a class="sister" href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>

2.next_siblings和previous_siblings

for sibling in soup.a.next_siblings:
  print(repr(sibling))
',\n'
<a class="sister" href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>
' and\n'
<a class="sister" href="http://example.com/tillie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link3">Tillie</a>
';\nand they lived at the bottom of a well.'
for sibling in soup.find(id="link3").previous_siblings:
  print(repr(sibling))
' and\n'
<a class="sister" href="http://example.com/lacie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link2">Lacie</a>
',\n'
<a class="sister" href="http://example.com/elsie" rel="external nofollow" rel="external nofollow" rel="external nofollow" rel="external nofollow" id="link1">Elsie</a>
'Once upon a time there were three little sisters; and their names were\n'

四、回退与前进

1.next_element和previous_element

指向下一个或者前一个被解析的对象(字符串或tag)，即深度优先遍历的后序节点和前序节点

last_a_tag = soup.find("a", id="link3")
print(last_a_tag.next_sibling)
print(last_a_tag.next_element)
;
and they lived at the bottom of a well.
Tillie
last_a_tag.previous_element
' and\n'

2.next_elements和previous_elements

通过.next_elements和previous_elements可以向前或向后访问文档的解析内容，就好像文档正在被解析一样

for element in last_a_tag.next_elements:
  print(repr(element))
'Tillie'
';\nand they lived at the bottom of a well.'
'\n'
<p class="story">...</p>
'...'
'\n'

更多关于使用Python爬虫库BeautifulSoup遍历文档树并对标签进行操作的方法与文章大家可以点击下面的相关文章

使用Python爬虫库BeautifulSoup遍历文档树并对标签进行操作详解

- Author -

BQW_

声明：登载此文出于传递更多信息之目的，并不意味着赞同其观点或证实其描述。

Python 相关文章推荐
Python编程使用*解包和itertools.product()求笛卡尔积的方法
Dec 18 Python
unittest+coverage单元测试代码覆盖操作实例详解
Apr 04 Python
Python实现中一次读取多个值的方法
Apr 22 Python
django将图片上传数据库后在前端显式的方法
May 25 Python
python3 实现验证码图片切割的方法
Dec 07 Python
在pycharm中使用git版本管理以及同步github的方法
Jan 16 Python
Python3环境安装Scrapy爬虫框架过程及常见错误
Jul 12 Python
使用apiDoc实现python接口文档编写
Nov 19 Python
Python模块/包/库安装的六种方法及区别
Feb 24 Python
Python openpyxl 插入折线图实例
Apr 17 Python
pycharm 使用tab跳出正在编辑的括号(){}{}等问题
Feb 26 Python
python微信智能AI机器人实现多种支付方式
Apr 12 Python
Python爬虫库BeautifulSoup获取对象(标签)名,属性,内容,注释
Jan 25 #Python
Python爬虫库BeautifulSoup的介绍与简单使用实例
Jan 25 #Python
使用Python爬虫库requests发送表单数据和JSON数据
Jan 25 #Python
Python爬虫库requests获取响应内容、响应状态码、响应头
Jan 25 #Python
使用Python爬虫库requests发送请求、传递URL参数、定制headers
Jan 25 #Python
flask框架自定义url转换器操作详解
Jan 25 #Python
常用python爬虫库介绍与简要说明
Jan 25 #Python
方案(1080) 答谢词(97) 计划书(118) 感言(394) 辞职信(98) 教学反思(909) 接收函(23) 广播稿(323) 事迹材料(636) 证明书(326)
You might like
php基础知识:类与对象(1)
2006/12/13 PHP
Win7下手动安装apache2.2、php5.4笔记
2015/04/03 PHP
PHP+Mysql无刷新问答评论系统(源码)
2016/12/20 PHP
PHP用函数嵌入网站访问量计数器
2017/10/27 PHP
浅析PHP开发规范
2018/02/05 PHP
php读取本地json文件的实例
2018/03/07 PHP
IE Firefox 使用自定义标签的区别
2009/10/15 Javascript
js遍历、动态的添加数据的小例子
2013/06/22 Javascript
jQuery 获取、设置HTML或TEXT内容的两种方法
2014/05/23 Javascript
js中this的用法实例分析
2015/01/10 Javascript
javascript中substring()、substr()、slice()的区别
2015/08/30 Javascript
基于angularjs实现图片放大镜效果
2016/08/31 Javascript
angularjs中使用ng-bind-html和ng-include的实例
2017/04/28 Javascript
vue之nextTick全面解析
2017/05/17 Javascript
vue-resource调用promise取数据方式详解
2017/07/21 Javascript
Grunt针对静态文件的压缩,版本控制打包的实例讲解
2017/09/29 Javascript
vue中实现在外部调用methods的方法(推荐)
2018/02/08 Javascript
JS实现的tab页切换效果完整示例
2018/12/18 Javascript
Node.js一行代码实现静态文件服务器的方法步骤
2019/05/07 Javascript
微信小程序实现滚动Tab选项卡
2020/11/16 Javascript
python基础教程之Hello World!
2014/08/29 Python
python统计字符串中指定字符出现次数的方法
2015/04/04 Python
Python模块文件结构代码详解
2018/02/03 Python
python 实现对文件夹中的图像连续重命名方法
2018/10/25 Python
numpy数组之存取文件的实现示例
2019/05/24 Python
Python的numpy库下的几个小函数的用法(小结)
2019/07/12 Python
python关于矩阵重复赋值覆盖问题的解决方法
2019/07/19 Python
Python3 无重复字符的最长子串的实现
2019/10/08 Python
python二维图制作的实例代码
2020/12/03 Python
python爬取豆瓣电影排行榜(requests)的示例代码
2021/02/18 Python
Cult Gaia官网：美国生活方式品牌
2019/08/16 全球购物
美国基督教约会网站：ChristianCafe.com
2020/02/04 全球购物
2014法院干警廉洁警示教育思想汇报
2014/09/13 职场文书
论群众路线学习心得体会
2014/10/31 职场文书
法人身份证明书
2015/06/18 职场文书
pygame面向对象的飞行小鸟实现(Flappy bird)
2021/04/01 Python