详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库


Posted in Python onJanuary 24, 2021

获取要爬取的URL

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

爬虫前期工作

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

用Pycharm打开项目开始写爬虫文件

字段文件items

# Define here the models for your scraped items
#
# See documentation in:
# https://docs.scrapy.org/en/latest/topics/items.html

import scrapy


class NbaprojectItem(scrapy.Item):
  # define the fields for your item here like:
  # name = scrapy.Field()
  # pass
  # 创建字段的固定格式-->scrapy.Field()
  # 英文名
  engName = scrapy.Field()
  # 中文名
  chName = scrapy.Field()
  # 身高
  height = scrapy.Field()
  # 体重
  weight = scrapy.Field()
  # 国家英文名
  contryEn = scrapy.Field()
  # 国家中文名
  contryCh = scrapy.Field()
  # NBA球龄
  experience = scrapy.Field()
  # 球衣号码
  jerseyNo = scrapy.Field()
  # 入选年
  draftYear = scrapy.Field()
  # 队伍英文名
  engTeam = scrapy.Field()
  # 队伍中文名
  chTeam = scrapy.Field()
  # 位置
  position = scrapy.Field()
  # 东南部
  displayConference = scrapy.Field()
  # 分区
  division = scrapy.Field()

爬虫文件

import scrapy
import json
from nbaProject.items import NbaprojectItem

class NbaspiderSpider(scrapy.Spider):
  name = 'nbaSpider'
  allowed_domains = ['nba.com']
  # 第一次爬取的网址,可以写多个网址
  # start_urls = ['http://nba.com/']
  start_urls = ['https://china.nba.com/static/data/league/playerlist.json']
  # 处理网址的response
  def parse(self, response):
    # 因为访问的网站返回的是json格式,首先用第三方包处理json数据
    data = json.loads(response.text)['payload']['players']
    # 以下列表用来存放不同的字段
    # 英文名
    engName = []
    # 中文名
    chName = []
    # 身高
    height = []
    # 体重
    weight = []
    # 国家英文名
    contryEn = []
    # 国家中文名
    contryCh = []
    # NBA球龄
    experience = []
    # 球衣号码
    jerseyNo = []
    # 入选年
    draftYear = []
    # 队伍英文名
    engTeam = []
    # 队伍中文名
    chTeam = []
    # 位置
    position = []
    # 东南部
    displayConference = []
    # 分区
    division = []
    # 计数
    count = 1
    for i in data:
      # 英文名
      engName.append(str(i['playerProfile']['firstNameEn'] + i['playerProfile']['lastNameEn']))
      # 中文名
      chName.append(str(i['playerProfile']['firstName'] + i['playerProfile']['lastName']))
      # 国家英文名
      contryEn.append(str(i['playerProfile']['countryEn']))
      # 国家中文
      contryCh.append(str(i['playerProfile']['country']))
      # 身高
      height.append(str(i['playerProfile']['height']))
      # 体重
      weight.append(str(i['playerProfile']['weight']))
      # NBA球龄
      experience.append(str(i['playerProfile']['experience']))
      # 球衣号码
      jerseyNo.append(str(i['playerProfile']['jerseyNo']))
      # 入选年
      draftYear.append(str(i['playerProfile']['draftYear']))
      # 队伍英文名
      engTeam.append(str(i['teamProfile']['code']))
      # 队伍中文名
      chTeam.append(str(i['teamProfile']['displayAbbr']))
      # 位置
      position.append(str(i['playerProfile']['position']))
      # 东南部
      displayConference.append(str(i['teamProfile']['displayConference']))
      # 分区
      division.append(str(i['teamProfile']['division']))

      # 创建item字段对象,用来存储信息 这里的item就是对应上面导的NbaprojectItem
      item = NbaprojectItem()
      item['engName'] = str(i['playerProfile']['firstNameEn'] + i['playerProfile']['lastNameEn'])
      item['chName'] = str(i['playerProfile']['firstName'] + i['playerProfile']['lastName'])
      item['contryEn'] = str(i['playerProfile']['countryEn'])
      item['contryCh'] = str(i['playerProfile']['country'])
      item['height'] = str(i['playerProfile']['height'])
      item['weight'] = str(i['playerProfile']['weight'])
      item['experience'] = str(i['playerProfile']['experience'])
      item['jerseyNo'] = str(i['playerProfile']['jerseyNo'])
      item['draftYear'] = str(i['playerProfile']['draftYear'])
      item['engTeam'] = str(i['teamProfile']['code'])
      item['chTeam'] = str(i['teamProfile']['displayAbbr'])
      item['position'] = str(i['playerProfile']['position'])
      item['displayConference'] = str(i['teamProfile']['displayConference'])
      item['division'] = str(i['teamProfile']['division'])
      # 打印爬取信息
      print("传输了",count,"条字段")
      count += 1
      # 将字段交回给引擎 -> 管道文件
      yield item

配置文件->开启管道文件

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

# Scrapy settings for nbaProject project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#   https://docs.scrapy.org/en/latest/topics/settings.html
#   https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#   https://docs.scrapy.org/en/latest/topics/spider-middleware.html
# ----------不做修改部分---------
BOT_NAME = 'nbaProject'

SPIDER_MODULES = ['nbaProject.spiders']
NEWSPIDER_MODULE = 'nbaProject.spiders'
# ----------不做修改部分---------

# Crawl responsibly by identifying yourself (and your website) on the user-agent
#USER_AGENT = 'nbaProject (+http://www.yourdomain.com)'

# Obey robots.txt rules
# ----------修改部分(可以自行查这是啥东西)---------
# ROBOTSTXT_OBEY = True
# ----------修改部分---------

# Configure maximum concurrent requests performed by Scrapy (default: 16)
#CONCURRENT_REQUESTS = 32

# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
#DOWNLOAD_DELAY = 3
# The download delay setting will honor only one of:
#CONCURRENT_REQUESTS_PER_DOMAIN = 16
#CONCURRENT_REQUESTS_PER_IP = 16

# Disable cookies (enabled by default)
#COOKIES_ENABLED = False

# Disable Telnet Console (enabled by default)
#TELNETCONSOLE_ENABLED = False

# Override the default request headers:
#DEFAULT_REQUEST_HEADERS = {
#  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
#  'Accept-Language': 'en',
#}

# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
#SPIDER_MIDDLEWARES = {
#  'nbaProject.middlewares.NbaprojectSpiderMiddleware': 543,
#}

# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#DOWNLOADER_MIDDLEWARES = {
#  'nbaProject.middlewares.NbaprojectDownloaderMiddleware': 543,
#}

# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
#EXTENSIONS = {
#  'scrapy.extensions.telnet.TelnetConsole': None,
#}

# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
# 开启管道文件
# ----------修改部分---------
ITEM_PIPELINES = {
  'nbaProject.pipelines.NbaprojectPipeline': 300,
}
# ----------修改部分---------
# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
#AUTOTHROTTLE_ENABLED = True
# The initial download delay
#AUTOTHROTTLE_START_DELAY = 5
# The maximum download delay to be set in case of high latencies
#AUTOTHROTTLE_MAX_DELAY = 60
# The average number of requests Scrapy should be sending in parallel to
# each remote server
#AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
# Enable showing throttling stats for every response received:
#AUTOTHROTTLE_DEBUG = False

# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
#HTTPCACHE_ENABLED = True
#HTTPCACHE_EXPIRATION_SECS = 0
#HTTPCACHE_DIR = 'httpcache'
#HTTPCACHE_IGNORE_HTTP_CODES = []
#HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'

管道文件 -> 将字段写进mysql

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html


# useful for handling different item types with a single interface
from itemadapter import ItemAdapter

import pymysql
class NbaprojectPipeline:
	# 初始化函数
  def __init__(self):
    # 连接数据库 注意修改数据库信息
    self.connect = pymysql.connect(host='域名', user='用户名', passwd='密码',
                    db='数据库', port=端口号) 
    # 获取游标
    self.cursor = self.connect.cursor()
    # 创建一个表用于存放item字段的数据
    createTableSql = """
              create table if not exists `nbaPlayer`(
              playerId INT UNSIGNED AUTO_INCREMENT,
              engName varchar(80),
              chName varchar(20),
              height varchar(20),
              weight varchar(20),
              contryEn varchar(50),
              contryCh varchar(20),
              experience int,
              jerseyNo int,
              draftYear int,
              engTeam varchar(50),
              chTeam varchar(50),
              position varchar(50),
              displayConference varchar(50),
              division varchar(50),
              primary key(playerId)
              )charset=utf8;
              """
    # 执行sql语句
    self.cursor.execute(createTableSql)
    self.connect.commit()
    print("完成了创建表的工作")
	#每次yield回来的字段会在这里做处理
  def process_item(self, item, spider):
  	# 打印item增加观赏性
  	print(item)
    # sql语句
    insert_sql = """
    insert into nbaPlayer(
    playerId, engName, 
    chName,height,
    weight,contryEn,
    contryCh,experience,
    jerseyNo,draftYear
    ,engTeam,chTeam,
    position,displayConference,
    division
    ) VALUES (null,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)
    """
    # 执行插入数据到数据库操作
    # 参数(sql语句,用item字段里的内容替换sql语句的占位符)
    self.cursor.execute(insert_sql, (item['engName'], item['chName'], item['height'], item['weight']
                     , item['contryEn'], item['contryCh'], item['experience'], item['jerseyNo'],
                     item['draftYear'], item['engTeam'], item['chTeam'], item['position'],
                     item['displayConference'], item['division']))
    # 提交,不进行提交无法保存到数据库
    self.connect.commit()
    print("数据提交成功!")

启动爬虫

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

屏幕上滚动的数据

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

去数据库查看数据

详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库

简简单单就把球员数据爬回来啦~

到此这篇关于详解Python之Scrapy爬虫教程NBA球员数据存放到Mysql数据库的文章就介绍到这了,更多相关Scrapy爬虫员数据存放到Mysql内容请搜索三水点靠木以前的文章或继续浏览下面的相关文章希望大家以后多多支持三水点靠木!

Python 相关文章推荐
python通过pil模块将raw图片转换成png图片的方法
Mar 16 Python
python爬取51job中hr的邮箱
May 14 Python
Python实现按学生年龄排序的实际问题详解
Aug 29 Python
pandas apply 函数 实现多进程的示例讲解
Apr 20 Python
python实现电脑自动关机
Jun 20 Python
Python传递参数的多种方式(小结)
Sep 18 Python
python实现修改固定模式的字符串内容操作示例
Dec 30 Python
Tensorflow 1.0之后模型文件、权重数值的读取方式
Feb 12 Python
PYQT5 vscode联合操作qtdesigner的方法
Mar 24 Python
使用pandas库对csv文件进行筛选保存
May 25 Python
10个python爬虫入门基础代码实例 + 1个简单的python爬虫完整实例
Dec 16 Python
python中numpy数组与list相互转换实例方法
Jan 29 Python
Ubuntu20下的Django安装的方法步骤
Jan 24 #Python
selenium+超级鹰实现模拟登录12306
Jan 24 #Python
使用numpngw和matplotlib生成png动画的示例代码
Jan 24 #Python
详解如何修改jupyter notebook的默认目录和默认浏览器
Jan 24 #Python
详解修改Anaconda中的Jupyter Notebook默认工作路径的三种方式
Jan 24 #Python
浅析python字符串前加r、f、u、l 的区别
Jan 24 #Python
python 图像增强算法实现详解
Jan 24 #Python
You might like
谈谈PHP语法(4)
2006/10/09 PHP
php抓取页面的几种方法详解
2013/06/17 PHP
使用PHP破解防盗链图片的一个简单方法
2014/06/07 PHP
PHP中使用gettext解决国际化问题的例子(i18n)
2014/06/13 PHP
dedecms中使用php语句指南
2014/11/13 PHP
PHP使用星号隐藏用户名,手机和邮箱的实现方法
2016/09/22 PHP
基于jQuery的投票系统显示结果插件
2011/08/12 Javascript
js实现支持手机滑动切换的轮播图片效果实例
2015/04/29 Javascript
全面总结Javascript对数组对象的各种操作
2017/01/22 Javascript
基于Angular中ng-controller父子级嵌套的相关属性详解
2018/10/08 Javascript
JavaScript实现五子棋游戏的方法详解
2019/07/08 Javascript
Node.js开发之套接字(socket)编程入门示例
2019/11/05 Javascript
layui table表格数据的新增,修改,删除,查询,双击获取行数据方式
2019/11/14 Javascript
15分钟学会vue项目改造成SSR(小白教程)
2019/12/17 Javascript
如何管理Vue中的缓存页面
2021/02/06 Vue.js
js实现Element中input组件的部分功能并封装成组件(实例代码)
2021/03/02 Javascript
[47:52]DOTA2-DPC中国联赛正赛 iG vs LBZS BO3 第二场 3月4日
2021/03/11 DOTA
python实现根据月份和日期得到星座的方法
2015/03/27 Python
Python实现计算两个时间之间相差天数的方法
2017/05/10 Python
Python 数据处理库 pandas进阶教程
2018/04/21 Python
python实现彩票系统
2020/06/28 Python
python利用appium实现手机APP自动化的示例
2021/01/26 Python
基于HTML5 Canvas 实现商场监控实例详解
2017/11/20 HTML / CSS
html特殊符号示例 html特殊字符编码对照表
2014/01/14 HTML / CSS
深入浅析HTML5中的SVG
2015/11/27 HTML / CSS
Farfetch台湾官网:奢侈品牌时尚购物平台
2019/06/17 全球购物
C#面试问题
2016/07/29 面试题
七年级音乐教学反思
2014/01/26 职场文书
城建学院毕业生自荐信
2014/01/31 职场文书
《猫》教学反思
2014/02/26 职场文书
大课间体育活动方案
2014/03/12 职场文书
文明美德伴我成长演讲稿
2014/05/12 职场文书
公司建议书怎么写
2014/05/15 职场文书
流动人口婚育证明范本
2014/09/26 职场文书
监考失职检讨书
2015/01/26 职场文书
springboot+rabbitmq实现智能家居实例详解
2022/07/23 Java/Android