Python实现简单HTML表格解析的方法


Posted in Python onJune 15, 2015

本文实例讲述了Python实现简单HTML表格解析的方法。分享给大家供大家参考。具体分析如下:

这里依赖libxml2dom,确保首先安装!导入到你的脚步并调用parse_tables() 函数。

1. source = a string containing the source code you can pass in just the table or the entire page code

2. headers = a list of ints OR a list of strings
If the headers are ints this is for tables with no header, just list the 0 based index of the rows in which you want to extract data.
If the headers are strings this is for tables with header columns (with the tags) it will pull the information from the specified columns

3. The 0 based index of the table in the source code. If there are multiple tables and the table you want to parse is the third table in the code then pass in the number 2 here

It will return a list of lists. each inner list will contain the parsed information.

具体代码如下:

#The goal of table parser is to get specific information from specific
#columns in a table.
#Input: source code from a typical website
#Arguments: a list of headers the user wants to return
#Output: A list of lists of the data in each row
import libxml2dom
def parse_tables(source, headers, table_index):
  """parse_tables(string source, list headers, table_index)
    headers may be a list of strings if the table has headers defined or
    headers may be a list of ints if no headers defined this will get data
    from the rows index.
    This method returns a list of lists
    """
  #Determine if the headers list is strings or ints and make sure they
  #are all the same type
  j = 0
  print 'Printing headers: ',headers
  #route to the correct function
  #if the header type is int
  if type(headers[0]) == type(1):
    #run no_header function
    return no_header(source, headers, table_index)
  #if the header type is string
  elif type(headers[0]) == type('a'):
    #run the header_given function
    return header_given(source, headers, table_index)
  else:
    #return none if the headers aren't correct
    return None
#This function takes in the source code of the whole page a string list of
#headers and the index number of the table on the page. It returns a list of
#lists with the scraped information
def header_given(source, headers, table_index):
  #initiate a list to hole the return list
  return_list = []
  #initiate a list to hold the index numbers of the data in the rows
  header_index = []
  #get a document object out of the source code
  doc = libxml2dom.parseString(source,html=1)
  #get the tables from the document
  tables = doc.getElementsByTagName('table')
  try:
    #try to get focue on the desired table
    main_table = tables[table_index]
  except:
    #if the table doesn't exits then return an error
    return ['The table index was not found']
  #get a list of headers in the table
  table_headers = main_table.getElementsByTagName('th')
  #need a sentry value for the header loop
  loop_sentry = 0
  #loop through each header looking for matches
  for header in table_headers:
    #if the header is in the desired headers list 
    if header.textContent in headers:
      #add it to the header_index
      header_index.append(loop_sentry)
    #add one to the loop_sentry
    loop_sentry+=1
  #get the rows from the table
  rows = main_table.getElementsByTagName('tr')
  #sentry value detecting if the first row is being viewed
  row_sentry = 0
  #loop through the rows in the table, skipping the first row
  for row in rows:
    #if row_sentry is 0 this is our first row
    if row_sentry == 0:
      #make the row_sentry not 0
      row_sentry = 1337
      continue
    #get all cells from the current row
    cells = row.getElementsByTagName('td')
    #initiate a list to append into the return_list
    cell_list = []
    #iterate through all of the header index's
    for i in header_index:
      #append the cells text content to the cell_list
      cell_list.append(cells[i].textContent)
    #append the cell_list to the return_list
    return_list.append(cell_list)
  #return the return_list
  return return_list
#This function takes in the source code of the whole page an int list of
#headers indicating the index number of the needed item and the index number
#of the table on the page. It returns a list of lists with the scraped info
def no_header(source, headers, table_index):
  #initiate a list to hold the return list
  return_list = []
  #get a document object out of the source code
  doc = libxml2dom.parseString(source, html=1)
  #get the tables from document
  tables = doc.getElementsByTagName('table')
  try:
    #Try to get focus on the desired table
    main_table = tables[table_index]
  except:
    #if the table doesn't exits then return an error
    return ['The table index was not found']
  #get all of the rows out of the main_table
  rows = main_table.getElementsByTagName('tr')
  #loop through each row
  for row in rows:
    #get all cells from the current row
    cells = row.getElementsByTagName('td')
    #initiate a list to append into the return_list
    cell_list = []
    #loop through the list of desired headers
    for i in headers:
      try:
        #try to add text from the cell into the cell_list
        cell_list.append(cells[i].textContent)
      except:
        #if there is an error usually an index error just continue
        continue
    #append the data scraped into the return_list    
    return_list.append(cell_list)
  #return the return list
  return return_list

希望本文所述对大家的Python程序设计有所帮助。

Python 相关文章推荐
wxpython 学习笔记 第一天
Mar 16 Python
python和pyqt实现360的CLable控件
Feb 21 Python
Python ORM框架SQLAlchemy学习笔记之数据添加和事务回滚介绍
Jun 10 Python
python操作mongodb根据_id查询数据的实现方法
May 20 Python
Python 26进制计算实现方法
May 28 Python
Python如何读取MySQL数据库表数据
Mar 11 Python
Python中return self的用法详解
Jul 27 Python
在Django model中设置多个字段联合唯一约束的实例
Jul 17 Python
VS2019+python3.7+opencv4.1+tensorflow1.13配置详解
Apr 16 Python
ITK 实现多张图像转成单个nii.gz或mha文件案例
Jul 01 Python
python help函数实例用法
Dec 06 Python
OpenCV 图像梯度的实现方法
Jul 25 Python
Python判断Abundant Number的方法
Jun 15 #Python
Python计算一个文件里字数的方法
Jun 15 #Python
Python素数检测实例分析
Jun 15 #Python
Python计算三维矢量幅度的方法
Jun 15 #Python
Python栈类实例分析
Jun 15 #Python
Python实现股市信息下载的方法
Jun 15 #Python
给Python入门者的一些编程建议
Jun 15 #Python
You might like
基于PHP编程注意事项的小结
2013/04/27 PHP
PHP屏蔽过滤指定关键字的方法
2014/11/03 PHP
php array_walk_recursive 使用自定的函数处理数组中的每一个元素
2016/11/16 PHP
PHP水印类,支持添加图片、文字、填充颜色区域的实现
2017/02/04 PHP
PHP添加PNG图片背景透明水印操作类定义与用法示例
2019/03/12 PHP
PHP简单验证码功能机制实例详解
2019/03/27 PHP
JQuery 获取和设置Select选项的代码
2010/02/07 Javascript
Javascript window对象详解
2014/11/12 Javascript
JS实现文字链接感应鼠标淡入淡出改变颜色的方法
2015/02/26 Javascript
详解JavaScript中的构造器Constructor模式
2016/01/14 Javascript
基于node.js依赖express解析post请求四种数据格式
2017/02/13 Javascript
JS实现队列的先进先出功能示例
2017/05/10 Javascript
JS仿淘宝搜索框用户输入事件的实现
2017/06/19 Javascript
jQuery实现的模仿雨滴下落动画效果
2018/12/11 jQuery
[02:49]DOTA2完美大师赛首日观众采访
2017/11/23 DOTA
wxPython窗口中文乱码解决方法
2014/10/11 Python
Cpy和Python的效率对比
2015/03/20 Python
Python中threading模块join函数用法实例分析
2015/06/04 Python
Python简单获取自身外网IP的方法
2016/09/18 Python
Python探索之静态方法和类方法的区别详解
2017/10/27 Python
Python多图片合并PDF的方法
2019/01/03 Python
解决django服务器重启端口被占用的问题
2019/07/26 Python
Python列表元素常见操作简单示例
2019/10/25 Python
python实现图片插入文字
2019/11/26 Python
Python paramiko 模块浅谈与SSH主要功能模拟解析
2020/02/29 Python
python的json包位置及用法总结
2020/06/21 Python
Python的Tqdm模块实现进度条配置
2021/02/24 Python
基于Html5实现的语音搜索功能
2019/05/13 HTML / CSS
lookfantastic荷兰:在线购买奢华护肤、护发和化妆品
2018/11/27 全球购物
高中生自我评价个人范文
2013/11/09 职场文书
中医专业职业生涯规划书范文
2014/01/04 职场文书
领导班子整改方案和个人整改措施
2014/10/25 职场文书
2015年销售员工作总结范文
2015/04/07 职场文书
新娘婚礼致辞
2015/07/27 职场文书
[有人@你]你有一封绿色倡议书,请查收!
2019/07/18 职场文书
CSS实现渐变色边框(Gradient borders)的5种方法
2022/03/25 HTML / CSS