python中threading和queue库实现多线程编程


Posted in Python onFebruary 06, 2021

摘要

本文主要介绍了利用python的 threading和queue库实现多线程编程,并封装为一个类,方便读者嵌入自己的业务逻辑。最后以机器学习的一个超参数选择为例进行演示。

多线程实现逻辑封装

实例化该类后,在.object_func函数中加入自己的业务逻辑,再调用.run方法即可。

# -*- coding: utf-8 -*-
# @Time : 2021/2/4 14:36
# @Author : CyrusMay WJ
# @FileName: run.py
# @Software: PyCharm
# @Blog :https://blog.csdn.net/Cyrus_May
import queue
import threading

class CyrusThread(object):
  def __init__(self,num_thread = 10,logger=None):
    """
    
    :param num_thread: 线程数
    :param logger: 日志对象
    """
    self.num_thread = num_thread
    self.logger = logger

  def object_func(self,args_queue,max_q):
    while 1:
      try:
        arg = args_queue.get_nowait()
        step = args_queue.qsize()
        self.logger.info("progress:{}\{}".format(max_q,step))
      except:
        self.logger.info("no more arg for args_queue!")
        break
        
        
        """
        此处加入自己的业务逻辑代码
        """
        
        
  def run(self,args):
    args_queue = queue.Queue()
    for value in args:
      args_queue.put(value)
    threads = []
    for i in range(self.num_thread):
      threads.append(threading.Thread(target=self.object_func,args = args_queue))
    for t in threads:
      t.start()
    for t in threads:
      t.join()

模型参数选择实例

# -*- coding: utf-8 -*-
# @Time : 2021/2/4 14:36
# @Author : CyrusMay WJ
# @FileName: run.py
# @Software: PyCharm
# @Blog :https://blog.csdn.net/Cyrus_May
import queue
import threading
import numpy as np
from sklearn.datasets import load_boston
from sklearn.svm import SVR
import logging
import sys


class CyrusThread(object):
  def __init__(self,num_thread = 10,logger=None):
    """

    :param num_thread: 线程数
    :param logger: 日志对象
    """
    self.num_thread = num_thread
    self.logger = logger

  def object_func(self,args_queue,max_q):
    while 1:
      try:
        arg = args_queue.get_nowait()
        step = args_queue.qsize()
        self.logger.info("progress:{}\{}".format(max_q,max_q-step))
      except:
        self.logger.info("no more arg for args_queue!")
        break
      # 业务代码
      C, epsilon, gamma = arg[0], arg[1], arg[2]
      svr_model = SVR(C=C, epsilon=epsilon, gamma=gamma)
      x, y = load_boston()["data"], load_boston()["target"]
      svr_model.fit(x, y)
      self.logger.info("score:{}".format(svr_model.score(x,y)))


  def run(self,args):
    args_queue = queue.Queue()
    max_q = 0
    for value in args:
      args_queue.put(value)
      max_q += 1
    threads = []
    for i in range(self.num_thread):
      threads.append(threading.Thread(target=self.object_func,args = (args_queue,max_q)))
    for t in threads:
      t.start()
    for t in threads:
      t.join()

# 创建日志对象
logger = logging.getLogger()
logger.setLevel(logging.INFO)
screen_handler = logging.StreamHandler(sys.stdout)
screen_handler.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(module)s.%(funcName)s:%(lineno)d - %(levelname)s - %(message)s')
screen_handler.setFormatter(formatter)
logger.addHandler(screen_handler)

# 创建需要调整参数的集合
args = []
for C in [i for i in np.arange(0.01,1,0.01)]:
  for epsilon in [i for i in np.arange(0.001,1,0.01)] + [i for i in range(1,10,1)]:
    for gamma in [i for i in np.arange(0.001,1,0.01)] + [i for i in range(1,10,1)]:
      args.append([C,epsilon,gamma])

# 创建多线程工具
threading_tool = CyrusThread(num_thread=20,logger=logger)
threading_tool.run(args)

运行结果

2021-02-04 20:52:22,824 - run.object_func:31 - INFO - progress:1176219\1
2021-02-04 20:52:22,824 - run.object_func:31 - INFO - progress:1176219\2
2021-02-04 20:52:22,826 - run.object_func:31 - INFO - progress:1176219\3
2021-02-04 20:52:22,833 - run.object_func:31 - INFO - progress:1176219\4
2021-02-04 20:52:22,837 - run.object_func:31 - INFO - progress:1176219\5
2021-02-04 20:52:22,838 - run.object_func:31 - INFO - progress:1176219\6
2021-02-04 20:52:22,841 - run.object_func:31 - INFO - progress:1176219\7
2021-02-04 20:52:22,862 - run.object_func:31 - INFO - progress:1176219\8
2021-02-04 20:52:22,873 - run.object_func:31 - INFO - progress:1176219\9
2021-02-04 20:52:22,884 - run.object_func:31 - INFO - progress:1176219\10
2021-02-04 20:52:22,885 - run.object_func:31 - INFO - progress:1176219\11
2021-02-04 20:52:22,897 - run.object_func:31 - INFO - progress:1176219\12
2021-02-04 20:52:22,900 - run.object_func:31 - INFO - progress:1176219\13
2021-02-04 20:52:22,904 - run.object_func:31 - INFO - progress:1176219\14
2021-02-04 20:52:22,912 - run.object_func:31 - INFO - progress:1176219\15
2021-02-04 20:52:22,920 - run.object_func:31 - INFO - progress:1176219\16
2021-02-04 20:52:22,920 - run.object_func:39 - INFO - score:-0.01674283914287855
2021-02-04 20:52:22,929 - run.object_func:31 - INFO - progress:1176219\17
2021-02-04 20:52:22,932 - run.object_func:39 - INFO - score:-0.007992354170952565
2021-02-04 20:52:22,932 - run.object_func:31 - INFO - progress:1176219\18
2021-02-04 20:52:22,945 - run.object_func:31 - INFO - progress:1176219\19
2021-02-04 20:52:22,954 - run.object_func:31 - INFO - progress:1176219\20
2021-02-04 20:52:22,978 - run.object_func:31 - INFO - progress:1176219\21
2021-02-04 20:52:22,984 - run.object_func:39 - INFO - score:-0.018769934807246536
2021-02-04 20:52:22,985 - run.object_func:31 - INFO - progress:1176219\22

到此这篇关于python中threading和queue库实现多线程编程的文章就介绍到这了,更多相关python 多线程编程内容请搜索三水点靠木以前的文章或继续浏览下面的相关文章希望大家以后多多支持三水点靠木!

Python 相关文章推荐
python 输出一个两行字符的变量
Feb 05 Python
python连接MySQL数据库实例分析
May 12 Python
python+selenium实现登录账户后自动点击的示例
Dec 22 Python
python实现内存监控系统
Mar 07 Python
详解python3 + Scrapy爬虫学习之创建项目
Apr 12 Python
Python将文字转成语音并读出来的实例详解
Jul 15 Python
python实现多进程通信实例分析
Sep 01 Python
Python3 把一个列表按指定数目分成多个列表的方式
Dec 25 Python
Python实现鼠标自动在屏幕上随机移动功能
Mar 14 Python
Python无头爬虫下载文件的实现
Apr 02 Python
Python自动化之UnitTest框架实战记录
Sep 08 Python
一文搞懂如何实现Go 超时控制
Mar 30 Python
Python3爬虫ChromeDriver的安装实例
Feb 06 #Python
解决pycharm修改代码后第一次运行不生效的问题
Feb 06 #Python
Python tkinter之ComboBox(下拉框)的使用简介
Feb 05 #Python
python批量提取图片信息并保存的实现
Feb 05 #Python
Python的轻量级ORM框架peewee使用教程
Feb 05 #Python
pycharm 实现光标快速移动到括号外或行尾的操作
Feb 05 #Python
pycharm进入时每次都是insert模式的解决方式
Feb 05 #Python
You might like
用php+mysql一个名片库程序
2006/10/09 PHP
php获得文件扩展名三法
2006/11/25 PHP
php生成xml简单实例代码
2009/12/16 PHP
PHP中HTML标签过滤技巧
2014/01/07 PHP
smarty高级特性之过滤器的使用方法
2015/12/25 PHP
php+html5实现无刷新图片上传教程
2016/01/22 PHP
javascript语句中的CDATA标签的意义
2007/05/09 Javascript
JS继承--原型链继承和类式继承
2013/04/08 Javascript
JavaScript的jQuery库中function的存在和参数问题
2015/08/13 Javascript
js实现文本框支持加减运算的方法
2015/08/19 Javascript
JavaScript电子时钟倒计时第二款
2016/01/10 Javascript
ionic进入多级目录后隐藏底部导航栏(tabs)的完美解决方案
2016/11/23 Javascript
详解Angular如何正确的操作DOM
2018/07/06 Javascript
微信小程序收藏功能的实现代码
2020/06/19 Javascript
pycharm 使用心得(九)解决No Python interpreter selected的问题
2014/06/06 Python
Django与遗留的数据库整合的方法指南
2015/07/24 Python
python抓取网页中链接的静态图片
2018/01/29 Python
python 图片去噪的方法示例
2019/07/09 Python
Django集成celery发送异步邮件实例
2019/12/17 Python
python logging设置level失败的解决方法
2020/02/19 Python
python批量修改交换机密码的示例
2020/09/22 Python
python实现粒子群算法
2020/10/15 Python
K近邻法(KNN)相关知识总结以及如何用python实现
2021/01/28 Python
中国电子产品外贸网站:MiniIntheBox
2017/02/06 全球购物
Stio官网:男女、儿童户外服装
2019/12/13 全球购物
社区包粽子活动方案
2014/01/21 职场文书
村官工作鉴定评语
2014/01/27 职场文书
文秘人员工作职责
2014/01/31 职场文书
会计专业大学生职业生涯规划书
2014/02/11 职场文书
教研处工作方案
2014/05/26 职场文书
人事代理委托书
2014/09/27 职场文书
电子银行业务授权委托书
2014/10/10 职场文书
2014年中职班主任工作总结
2014/12/16 职场文书
2015年护士工作总结范文
2015/03/31 职场文书
机械生产实习心得体会
2016/01/22 职场文书
缓存替换策略及应用(以Redis、InnoDB为例)
2021/07/25 Redis