Python爬虫代理池搭建的方法步骤


Posted in Python onSeptember 28, 2020

一、为什么要搭建爬虫代理池

在众多的网站防爬措施中,有一种是根据ip的访问频率进行限制,即在某一时间段内,当某个ip的访问次数达到一定的阀值时,该ip就会被拉黑、在一段时间内禁止访问。

应对的方法有两种:

1. 降低爬虫的爬取频率,避免IP被限制访问,缺点显而易见:会大大降低爬取的效率。

2. 搭建一个IP代理池,使用不同的IP轮流进行爬取。

二、搭建思路

1、从代理网站(如:西刺代理、快代理、云代理、无忧代理)爬取代理IP;

2、验证代理IP的可用性(使用代理IP去请求指定URL,根据响应验证代理IP是否生效);

3、将可用的代理IP保存到数据库;

常用代理网站:西刺代理 、云代理 、IP海 、无忧代理 、飞蚁代理 、快代理

三、代码实现

工程结构如下:

Python爬虫代理池搭建的方法步骤

ipproxy.py

IPProxy代理类定义了要爬取的IP代理的字段信息和一些基础方法。

# -*- coding: utf-8 -*-
import re
import time
from settings import PROXY_URL_FORMATTER

schema_pattern = re.compile(r'http|https$', re.I)
ip_pattern = re.compile(r'^([0-9]{1,3}.){3}[0-9]{1,3}$', re.I)
port_pattern = re.compile(r'^[0-9]{2,5}$', re.I)

class IPProxy:
  '''
  {
    "schema": "http", # 代理的类型
    "ip": "127.0.0.1", # 代理的IP地址
    "port": "8050", # 代理的端口号
    "used_total": 11, # 代理的使用次数
    "success_times": 5, # 代理请求成功的次数
    "continuous_failed": 3, # 使用代理发送请求,连续失败的次数
    "created_time": "2018-05-02" # 代理的爬取时间
  }
  '''

  def __init__(self, schema, ip, port, used_total=0, success_times=0, continuous_failed=0,
         created_time=None):
    """Initialize the proxy instance"""
    if schema == "" or schema is None:
      schema = "http"
    self.schema = schema.lower()
    self.ip = ip
    self.port = port
    self.used_total = used_total
    self.success_times = success_times
    self.continuous_failed = continuous_failed
    if created_time is None:
      created_time = time.strftime('%Y-%m-%d', time.localtime(time.time()))
    self.created_time = created_time

  def _get_url(self):
    ''' Return the proxy url'''
    return PROXY_URL_FORMATTER % {'schema': self.schema, 'ip': self.ip, 'port': self.port}

  def _check_format(self):
    ''' Return True if the proxy fields are well-formed,otherwise return False'''
    if self.schema is not None and self.ip is not None and self.port is not None:
      if schema_pattern.match(self.schema) and ip_pattern.match(self.ip) and port_pattern.match(self.port):
        return True
    return False

  def _is_https(self):
    ''' Return True if the proxy is https,otherwise return False'''
    return self.schema == 'https'

  def _update(self, successed=False):
    ''' Update proxy based on the result of the request's response'''
    self.used_total = self.used_total + 1
    if successed:
      self.continuous_failed = 0
      self.success_times = self.success_times + 1
    else:
      print(self.continuous_failed)
      self.continuous_failed = self.continuous_failed + 1

if __name__ == '__main__':
  proxy = IPProxy('HTTPS', '192.168.2.25', "8080")
  print(proxy._get_url())
  print(proxy._check_format())
  print(proxy._is_https())

settings.py

settings.py中汇聚了工程所需要的配置信息。

# 指定Redis的主机名和端口
REDIS_HOST = 'localhost'
REDIS_PORT = 6379

# 代理保存到Redis key 格式化字符串
PROXIES_REDIS_FORMATTER = 'proxies::{}'
# 已经存在的HTTP代理和HTTPS代理集合
PROXIES_REDIS_EXISTED = 'proxies::existed'

# 最多连续失败几次
MAX_CONTINUOUS_TIMES = 3
# 代理地址的格式化字符串
PROXY_URL_FORMATTER = '%(schema)s://%(ip)s:%(port)s'

USER_AGENT_LIST = [
  "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",
  "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",
  "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",
  "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",
  "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",
  "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",
  "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",
  "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
  "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",
  "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
  "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",
  "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
  "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
  "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",
  "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",
  "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",
  "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"
]
# 爬取到的代理保存前先检验是否可用,默认True
PROXY_CHECK_BEFOREADD = True
# 检验代理可用性的请求地址,支持多个
PROXY_CHECK_URLS = {'https':['https://icanhazip.com'],'http':['http://icanhazip.com']}

proxy_util.py

proxy_util.py 中主要定义了一些实用方法,例如:proxy_to_dict(proxy)用来将IPProxy代理实例转换成字典;proxy_from_dict(d)用来将字典转换为IPProxy实例;request_page()用来发送请求;_is_proxy_available()用来校验代理IP是否可用。

# -*- coding: utf-8 -*-
import random
import logging
import requests
from ipproxy import IPProxy
from settings import USER_AGENT_LIST, PROXY_CHECK_URLS

# Setting logger output format
logging.basicConfig(level=logging.INFO,
          format='[%(asctime)-15s] [%(levelname)8s] [%(name)10s ] - %(message)s (%(filename)s:%(lineno)s)',
          datefmt='%Y-%m-%d %T'
          )
logger = logging.getLogger(__name__)


def proxy_to_dict(proxy):
  d = {
    "schema": proxy.schema,
    "ip": proxy.ip,
    "port": proxy.port,
    "used_total": proxy.used_total,
    "success_times": proxy.success_times,
    "continuous_failed": proxy.continuous_failed,
    "created_time": proxy.created_time
  }
  return d


def proxy_from_dict(d):
  return IPProxy(schema=d['schema'], ip=d['ip'], port=d['port'], used_total=d['used_total'],
          success_times=d['success_times'], continuous_failed=d['continuous_failed'],
          created_time=d['created_time'])


# Truncate header and tailer blanks
def strip(data):
  if data is not None:
    return data.strip()
  return data


base_headers = {
  'Accept-Encoding': 'gzip, deflate, br',
  'Accept-Language': 'en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7'
}


def request_page(url, options={}, encoding='utf-8'):
  """send a request,get response"""
  headers = dict(base_headers, **options)
  if 'User-Agent' not in headers.keys():
    headers['User-Agent'] = random.choice(USER_AGENT_LIST)

  logger.info('正在抓取: ' + url)
  try:
    response = requests.get(url, headers=headers)
    if response.status_code == 200:
      logger.info('抓取成功: ' + url)
      return response.content.decode(encoding=encoding)
  except ConnectionError:
    logger.error('抓取失败' + url)
    return None

def _is_proxy_available(proxy, options={}):
  """Check whether the Proxy is available or not"""
  headers = dict(base_headers, **options)
  if 'User-Agent' not in headers.keys():
    headers['User-Agent'] = random.choice(USER_AGENT_LIST)
  proxies = {proxy.schema: proxy._get_url()}
  check_urls = PROXY_CHECK_URLS[proxy.schema]
  for url in check_urls:
    try:
      response = requests.get(url=url, proxies=proxies, headers=headers, timeout=5)
    except BaseException:
      logger.info("< " + url + " > 验证代理 < " + proxy._get_url() + " > 结果: 不可用 ")
    else:
      if response.status_code == 200:
        logger.info("< " + url + " > 验证代理 < " + proxy._get_url() + " > 结果: 可用 ")
        return True
      else:
        logger.info("< " + url + " > 验证代理 < " + proxy._get_url() + " > 结果: 不可用 ")
  return False


if __name__ == '__main__':
  headers = dict(base_headers)
  if 'User-Agent' not in headers.keys():
    headers['User-Agent'] = random.choice(USER_AGENT_LIST)
  proxies = {"https": "https://163.125.255.154:9797"}
  response = requests.get("https://www.baidu.com", headers=headers, proxies=proxies, timeout=3)
  print(response.content)

proxy_queue.py

代理队列用来保存并对外提供 IP代理,不同的代理队列内代理IP的保存和提取策略可以不同。在这里, BaseQueue 是所有代理队列的基类,其中声明了所有代理队列都需要实现的保存代理IP、提取代理IP、查看代理IP数量等接口。示例的 FifoQueue 是一个先进先出队列,底层使用 Redis 列表实现,为了确保同一个代理IP只能被放入队列一次,这里使用了一个Redis proxies::existed 集合进行入队前重复校验。

# -*- coding: utf-8 -*-
from proxy_util import logger
import json
import redis
from ipproxy import IPProxy
from proxy_util import proxy_to_dict, proxy_from_dict, _is_proxy_available
from settings import PROXIES_REDIS_EXISTED, PROXIES_REDIS_FORMATTER, MAX_CONTINUOUS_TIMES, PROXY_CHECK_BEFOREADD

"""
Proxy Queue Base Class
"""
class BaseQueue(object):

  def __init__(self, server):
    """Initialize the proxy queue instance

    Parameters
    ----------
    server : StrictRedis
      Redis client instance
    """
    self.server = server

  def _serialize_proxy(self, proxy):
    """Serialize proxy instance"""
    return proxy_to_dict(proxy)

  def _deserialize_proxy(self, serialized_proxy):
    """deserialize proxy instance"""
    return proxy_from_dict(eval(serialized_proxy))

  def __len__(self, schema='http'):
    """Return the length of the queue"""
    raise NotImplementedError

  def push(self, proxy, need_check):
    """Push a proxy"""
    raise NotImplementedError

  def pop(self, schema='http', timeout=0):
    """Pop a proxy"""
    raise NotImplementedError


class FifoQueue(BaseQueue):
  """First in first out queue"""

  def __len__(self, schema='http'):
    """Return the length of the queue"""
    return self.server.llen(PROXIES_REDIS_FORMATTER.format(schema))

  def push(self, proxy, need_check=PROXY_CHECK_BEFOREADD):
    """Push a proxy"""
    if need_check and not _is_proxy_available(proxy):
      return
    elif proxy.continuous_failed < MAX_CONTINUOUS_TIMES and not self._is_existed(proxy):
      key = PROXIES_REDIS_FORMATTER.format(proxy.schema)
      self.server.rpush(key, json.dumps(self._serialize_proxy(proxy),ensure_ascii=False))

  def pop(self, schema='http', timeout=0):
    """Pop a proxy"""
    if timeout > 0:
      p = self.server.blpop(PROXIES_REDIS_FORMATTER.format(schema.lower()), timeout)
      if isinstance(p, tuple):
        p = p[1]
    else:
      p = self.server.lpop(PROXIES_REDIS_FORMATTER.format(schema.lower()))
    if p:
      p = self._deserialize_proxy(p)
      self.server.srem(PROXIES_REDIS_EXISTED, p._get_url())
      return p

  def _is_existed(self, proxy):
    added = self.server.sadd(PROXIES_REDIS_EXISTED, proxy._get_url())
    return added == 0


if __name__ == '__main__':
  r = redis.StrictRedis(host='localhost', port=6379)
  queue = FifoQueue(r)
  proxy = IPProxy('http', '218.66.253.144', '80')
  queue.push(proxy)
  proxy = queue.pop(schema='http')
  print(proxy._get_url())

proxy_crawlers.py

ProxyBaseCrawler 是所有代理爬虫的基类,其中只定义了一个 _start_crawl() 方法用来从搜集到的代理网站爬取代理IP。

# -*- coding: utf-8 -*-
from lxml import etree
from ipproxy import IPProxy
from proxy_util import strip, request_page, logger


class ProxyBaseCrawler(object):

  def __init__(self, queue=None, website=None, urls=[]):
    self.queue = queue
    self.website = website
    self.urls = urls

  def _start_crawl(self):
    raise NotImplementedError


class KuaiDailiCrawler(ProxyBaseCrawler): # 快代理
  def _start_crawl(self):
    for url_dict in self.urls:
      logger.info("开始爬取 [ " + self.website + " ] :::> [ " + url_dict['type'] + " ]")
      has_more = True
      url = None
      while has_more:
        if 'page' in url_dict.keys() and str.find(url_dict['url'], '{}') != -1:
          url = url_dict['url'].format(str(url_dict['page']))
          url_dict['page'] = url_dict['page'] + 1
        else:
          url = url_dict['url']
          has_more = False
        html = etree.HTML(request_page(url))
        tr_list = html.xpath("//table[@class='table table-bordered table-striped']/tbody/tr")
        for tr in tr_list:
          ip = tr.xpath("./td[@data-title='IP']/text()")[0] if len(
            tr.xpath("./td[@data-title='IP']/text()")) else None
          port = tr.xpath("./td[@data-title='PORT']/text()")[0] if len(
            tr.xpath("./td[@data-title='PORT']/text()")) else None
          schema = tr.xpath("./td[@data-title='类型']/text()")[0] if len(
            tr.xpath("./td[@data-title='类型']/text()")) else None
          proxy = IPProxy(schema=strip(schema), ip=strip(ip), port=strip(port))
          if proxy._check_format():
            self.queue.push(proxy)
        if tr_list is None:
          has_more = False


class FeiyiDailiCrawler(ProxyBaseCrawler): # 飞蚁代理
  def _start_crawl(self):
    for url_dict in self.urls:
      logger.info("开始爬取 [ " + self.website + " ] :::> [ " + url_dict['type'] + " ]")
      has_more = True
      url = None
      while has_more:
        if 'page' in url_dict.keys() and str.find(url_dict['url'], '{}') != -1:
          url = url_dict['url'].format(str(url_dict['page']))
          url_dict['page'] = url_dict['page'] + 1
        else:
          url = url_dict['url']
          has_more = False
        html = etree.HTML(request_page(url))
        tr_list = html.xpath("//div[@id='main-content']//table/tr[position()>1]")
        for tr in tr_list:
          ip = tr.xpath("./td[1]/text()")[0] if len(tr.xpath("./td[1]/text()")) else None
          port = tr.xpath("./td[2]/text()")[0] if len(tr.xpath("./td[2]/text()")) else None
          schema = tr.xpath("./td[4]/text()")[0] if len(tr.xpath("./td[4]/text()")) else None
          proxy = IPProxy(schema=strip(schema), ip=strip(ip), port=strip(port))
          if proxy._check_format():
            self.queue.push(proxy)
        if tr_list is None:
          has_more = False


class WuyouDailiCrawler(ProxyBaseCrawler): # 无忧代理
  def _start_crawl(self):
    for url_dict in self.urls:
      logger.info("开始爬取 [ " + self.website + " ] :::> [ " + url_dict['type'] + " ]")
      has_more = True
      url = None
      while has_more:
        if 'page' in url_dict.keys() and str.find(url_dict['url'], '{}') != -1:
          url = url_dict['url'].format(str(url_dict['page']))
          url_dict['page'] = url_dict['page'] + 1
        else:
          url = url_dict['url']
          has_more = False
        html = etree.HTML(request_page(url))
        ul_list = html.xpath("//div[@class='wlist'][2]//ul[@class='l2']")
        for ul in ul_list:
          ip = ul.xpath("./span[1]/li/text()")[0] if len(ul.xpath("./span[1]/li/text()")) else None
          port = ul.xpath("./span[2]/li/text()")[0] if len(ul.xpath("./span[2]/li/text()")) else None
          schema = ul.xpath("./span[4]/li/text()")[0] if len(ul.xpath("./span[4]/li/text()")) else None
          proxy = IPProxy(schema=strip(schema), ip=strip(ip), port=strip(port))
          if proxy._check_format():
            self.queue.push(proxy)
        if ul_list is None:
          has_more = False


class IPhaiDailiCrawler(ProxyBaseCrawler): # IP海代理
  def _start_crawl(self):
    for url_dict in self.urls:
      logger.info("开始爬取 [ " + self.website + " ] :::> [ " + url_dict['type'] + " ]")
      has_more = True
      url = None
      while has_more:
        if 'page' in url_dict.keys() and str.find(url_dict['url'], '{}') != -1:
          url = url_dict['url'].format(str(url_dict['page']))
          url_dict['page'] = url_dict['page'] + 1
        else:
          url = url_dict['url']
          has_more = False
        html = etree.HTML(request_page(url))
        tr_list = html.xpath("//table//tr[position()>1]")
        for tr in tr_list:
          ip = tr.xpath("./td[1]/text()")[0] if len(tr.xpath("./td[1]/text()")) else None
          port = tr.xpath("./td[2]/text()")[0] if len(tr.xpath("./td[2]/text()")) else None
          schema = tr.xpath("./td[4]/text()")[0] if len(tr.xpath("./td[4]/text()")) else None
          proxy = IPProxy(schema=strip(schema), ip=strip(ip), port=strip(port))
          if proxy._check_format():
            self.queue.push(proxy)
        if tr_list is None:
          has_more = False


class YunDailiCrawler(ProxyBaseCrawler): # 云代理
  def _start_crawl(self):
    for url_dict in self.urls:
      logger.info("开始爬取 [ " + self.website + " ] :::> [ " + url_dict['type'] + " ]")
      has_more = True
      url = None
      while has_more:
        if 'page' in url_dict.keys() and str.find(url_dict['url'], '{}') != -1:
          url = url_dict['url'].format(str(url_dict['page']))
          url_dict['page'] = url_dict['page'] + 1
        else:
          url = url_dict['url']
          has_more = False
        html = etree.HTML(request_page(url, encoding='gbk'))
        tr_list = html.xpath("//table/tbody/tr")
        for tr in tr_list:
          ip = tr.xpath("./td[1]/text()")[0] if len(tr.xpath("./td[1]/text()")) else None
          port = tr.xpath("./td[2]/text()")[0] if len(tr.xpath("./td[2]/text()")) else None
          schema = tr.xpath("./td[4]/text()")[0] if len(tr.xpath("./td[4]/text()")) else None
          proxy = IPProxy(schema=strip(schema), ip=strip(ip), port=strip(port))
          if proxy._check_format():
            self.queue.push(proxy)
        if tr_list is None:
          has_more = False


class XiCiDailiCrawler(ProxyBaseCrawler): # 西刺代理
  def _start_crawl(self):
    for url_dict in self.urls:
      logger.info("开始爬取 [ " + self.website + " ] :::> [ " + url_dict['type'] + " ]")
      has_more = True
      url = None
      while has_more:
        if 'page' in url_dict.keys() and str.find(url_dict['url'], '{}') != -1:
          url = url_dict['url'].format(str(url_dict['page']))
          url_dict['page'] = url_dict['page'] + 1
        else:
          url = url_dict['url']
          has_more = False
        html = etree.HTML(request_page(url))
        tr_list = html.xpath("//table[@id='ip_list']//tr[@class!='subtitle']")
        for tr in tr_list:
          ip = tr.xpath("./td[2]/text()")[0] if len(tr.xpath("./td[2]/text()")) else None
          port = tr.xpath("./td[3]/text()")[0] if len(tr.xpath("./td[3]/text()")) else None
          schema = tr.xpath("./td[6]/text()")[0] if len(tr.xpath("./td[6]/text()")) else None
          if schema.lower() == "http" or schema.lower() == "https":
            proxy = IPProxy(schema=strip(schema), ip=strip(ip), port=strip(port))
            if proxy._check_format():
              self.queue.push(proxy)
        if tr_list is None:
          has_more = False

run.py

通过run.py启动各个代理网站爬虫。

# -*- coding: utf-8 -*-
import redis
from proxy_queue import FifoQueue
from settings import REDIS_HOST, REDIS_PORT
from proxy_crawlers import WuyouDailiCrawler, FeiyiDailiCrawler, KuaiDailiCrawler, IPhaiDailiCrawler, YunDailiCrawler, \
  XiCiDailiCrawler

r = redis.StrictRedis(host=REDIS_HOST, port=REDIS_PORT)
fifo_queue = FifoQueue(r)


def run_kuai():
  kuaidailiCrawler = KuaiDailiCrawler(queue=fifo_queue, website='快代理[国内高匿]',
                    urls=[{'url': 'https://www.kuaidaili.com/free/inha/{}/', 'type': '国内高匿',
                        'page': 1},
                       {'url': 'https://www.kuaidaili.com/free/intr/{}/', 'type': '国内普通',
                        'page': 1}])
  kuaidailiCrawler._start_crawl()


def run_feiyi():
  feiyidailiCrawler = FeiyiDailiCrawler(queue=fifo_queue, website='飞蚁代理',
                     urls=[{'url': 'http://www.feiyiproxy.com/?page_id=1457', 'type': '首页推荐'}])
  feiyidailiCrawler._start_crawl()


def run_wuyou():
  wuyoudailiCrawler = WuyouDailiCrawler(queue=fifo_queue, website='无忧代理',
                     urls=[{'url': 'http://www.data5u.com/free/index.html', 'type': '首页推荐'},
                        {'url': 'http://www.data5u.com/free/gngn/index.shtml', 'type': '国内高匿'},
                        {'url': 'http://www.data5u.com/free/gnpt/index.shtml', 'type': '国内普通'}])
  wuyoudailiCrawler._start_crawl()


def run_iphai():
  crawler = IPhaiDailiCrawler(queue=fifo_queue, website='IP海代理',
                urls=[{'url': 'http://www.iphai.com/free/ng', 'type': '国内高匿'},
                   {'url': 'http://www.iphai.com/free/np', 'type': '国内普通'},
                   {'url': 'http://www.iphai.com/free/wg', 'type': '国外高匿'},
                   {'url': 'http://www.iphai.com/free/wp', 'type': '国外普通'}])
  crawler._start_crawl()


def run_yun():
  crawler = YunDailiCrawler(queue=fifo_queue, website='云代理',
               urls=[{'url': 'http://www.ip3366.net/free/?stype=1&page={}', 'type': '国内高匿', 'page': 1},
                  {'url': 'http://www.ip3366.net/free/?stype=2&page={}', 'type': '国内普通', 'page': 1},
                  {'url': 'http://www.ip3366.net/free/?stype=3&page={}', 'type': '国外高匿', 'page': 1},
                  {'url': 'http://www.ip3366.net/free/?stype=4&page={}', 'type': '国外普通', 'page': 1}])
  crawler._start_crawl()


def run_xici():
  crawler = XiCiDailiCrawler(queue=fifo_queue, website='西刺代理',
                urls=[{'url': 'https://www.xicidaili.com/', 'type': '首页推荐'},
                   {'url': 'https://www.xicidaili.com/nn/{}', 'type': '国内高匿', 'page': 1},
                   {'url': 'https://www.xicidaili.com/nt/{}', 'type': '国内普通', 'page': 1},
                   {'url': 'https://www.xicidaili.com/wn/{}', 'type': '国外高匿', 'page': 1},
                   {'url': 'https://www.xicidaili.com/wt/{}', 'type': '国外普通', 'page': 1}])
  crawler._start_crawl()


if __name__ == '__main__':
  run_xici()
  run_iphai()
  run_kuai()
  run_feiyi()
  run_yun()
  run_wuyou()

爬取西刺代理时,后台日志示例如下:

Python爬虫代理池搭建的方法步骤

Redis数据库中爬取到的代理IP的数据结构如下:

Python爬虫代理池搭建的方法步骤

四、代理测试

 接下来,使用爬取好的代理来请求 http://icanhazip.com 进行测试,代码如下:

# -*- coding: utf-8 -*-
import random
import requests
from proxy_util import logger
from run import fifo_queue
from settings import USER_AGENT_LIST
from proxy_util import base_headers

# 测试地址
url = 'http://icanhazip.com'

# 获取代理
proxy = fifo_queue.pop(schema='http')
proxies = {proxy.schema:proxy._get_url()}

# 构造请求头
headers = dict(base_headers)
if 'User-Agent' not in headers.keys():
  headers['User-Agent'] = random.choice(USER_AGENT_LIST)

response = None
successed = False
try:
  response = requests.get(url,headers=headers,proxies = proxies,timeout=5)
except BaseException:
  logger.error("使用代理< "+proxy._get_url()+" > 请求 < "+url+" > 结果: 失败 ")
else:
  if (response.status_code == 200):
    logger.info(response.content.decode())
    successed = True
    logger.info("使用代理< " + proxy._get_url() + " > 请求 < " + url + " > 结果: 成功 ")
  else:
    logger.info(response.content.decode())
    logger.info("使用代理< " + proxy._get_url() + " > 请求 < " + url + " > 结果: 失败 ")

# 根据请求的响应结果更新代理
proxy._update(successed)
# 将代理返还给队列,返还时不校验可用性
fifo_queue.push(proxy,need_check=False)

使用 http://218.66.253.144:80 代理请求成功后将代理重新放回队列,并将 Redis 中该代理的 used_total 、success_times 、continuous_failed三个字段信息进行了相应的更新。

Python爬虫代理池搭建的方法步骤

项目地址:https://github.com/pengjunlee/ipproxy_pool.git

到此这篇关于Python爬虫代理池搭建的方法步骤的文章就介绍到这了,更多相关Python爬虫代理池搭建内容请搜索三水点靠木以前的文章或继续浏览下面的相关文章希望大家以后多多支持三水点靠木!

Python 相关文章推荐
使用Python3编写抓取网页和只抓网页图片的脚本
Aug 20 Python
Python读取MRI并显示为灰度图像实例代码
Jan 03 Python
对numpy数据写入文件的方法讲解
Jul 09 Python
使用Python实现微信提醒备忘录功能
Dec 04 Python
Python 实现文件打包、上传与校验的方法
Feb 13 Python
详解python做UI界面的方法
Feb 27 Python
django富文本编辑器的实现示例
Apr 10 Python
实例详解Python装饰器与闭包
Jul 29 Python
pandas DataFrame的修改方法(值、列、索引)
Aug 02 Python
python 中值滤波,椒盐去噪,图片增强实例
Dec 18 Python
详解Django3中直接添加Websockets方式
Feb 12 Python
Python 图片处理库exifread详解
Feb 25 Python
浅析python 通⽤爬⾍和聚焦爬⾍
Sep 28 #Python
Scrapy 配置动态代理IP的实现
Sep 28 #Python
Scrapy中如何向Spider传入参数的方法实现
Sep 28 #Python
详解向scrapy中的spider传递参数的几种方法(2种)
Sep 28 #Python
小结Python的反射机制
Sep 28 #Python
scrapy与selenium结合爬取数据(爬取动态网站)的示例代码
Sep 28 #Python
scrapy结合selenium解析动态页面的实现
Sep 28 #Python
You might like
PHP 文件上传进度条的两种实现方法的代码
2007/11/25 PHP
ThinkPHP3.1新特性之查询条件预处理简介
2014/06/19 PHP
Yii中render和renderPartial的区别
2014/09/03 PHP
PHP计算近1年的所有月份
2017/03/13 PHP
PHP回调函数概念与用法实例分析
2017/11/03 PHP
Laravel项目中timeAgo字段语言转换的改善方法示例
2019/09/16 PHP
PHP Pipeline 实现中间件的示例代码
2020/04/26 PHP
arguments对象
2006/11/20 Javascript
jQuery1.3.2 升级到jQuery1.4.4需要修改的地方
2011/01/06 Javascript
js字符编码函数区别分析
2011/12/28 Javascript
设置iframe的document.designMode后仅Firefox中其body.innerHTML为br
2012/02/27 Javascript
关于jQuery对象数据缓存Cache原理以及jQuery.data详解
2013/04/07 Javascript
Js获取数组最大和最小值示例代码
2013/10/29 Javascript
json属性名为什么要双引号(个人猜测)
2014/07/31 Javascript
jquery实现漫天雪花飞舞的圣诞祝福雪花效果代码分享
2015/08/20 Javascript
Vue登录注册并保持登录状态的方法
2018/08/17 Javascript
node全局变量__dirname与__filename的区别
2019/01/14 Javascript
Vue 后台管理类项目兼容IE9+的方法示例
2019/02/20 Javascript
用Vue.js方法创建模板并使用多个模板合成
2019/06/28 Javascript
vue中get请求如何传递数组参数的方法示例
2019/11/08 Javascript
python实现批量转换文件编码(批转换编码示例)
2014/01/23 Python
python实现判断一个字符串是否是合法IP地址的示例
2018/06/04 Python
Python爬虫之正则表达式基本用法实例分析
2018/08/08 Python
Django REST framework 单元测试实例解析
2019/11/07 Python
python编写一个会算账的脚本的示例代码
2020/06/02 Python
带你学习Python如何实现回归树模型
2020/07/16 Python
Python colormap库的安装和使用详情
2020/10/06 Python
Fox Racing英国官网:越野摩托车和山地自行车服装
2020/02/26 全球购物
Shell如何接收变量输入
2012/09/24 面试题
大四自我鉴定范文
2013/10/06 职场文书
英语专业推荐信
2013/11/16 职场文书
公务员职业生涯规划书范文  
2014/01/19 职场文书
超市创业计划书
2014/04/24 职场文书
群众路线个人自我剖析材料
2014/10/07 职场文书
2015年世界环境日活动总结
2015/02/11 职场文书
班主任高考寄语
2015/02/26 职场文书