python中threading和queue库实现多线程编程


Posted in Python onFebruary 06, 2021

摘要

本文主要介绍了利用python的 threading和queue库实现多线程编程,并封装为一个类,方便读者嵌入自己的业务逻辑。最后以机器学习的一个超参数选择为例进行演示。

多线程实现逻辑封装

实例化该类后,在.object_func函数中加入自己的业务逻辑,再调用.run方法即可。

# -*- coding: utf-8 -*-
# @Time : 2021/2/4 14:36
# @Author : CyrusMay WJ
# @FileName: run.py
# @Software: PyCharm
# @Blog :https://blog.csdn.net/Cyrus_May
import queue
import threading

class CyrusThread(object):
  def __init__(self,num_thread = 10,logger=None):
    """
    
    :param num_thread: 线程数
    :param logger: 日志对象
    """
    self.num_thread = num_thread
    self.logger = logger

  def object_func(self,args_queue,max_q):
    while 1:
      try:
        arg = args_queue.get_nowait()
        step = args_queue.qsize()
        self.logger.info("progress:{}\{}".format(max_q,step))
      except:
        self.logger.info("no more arg for args_queue!")
        break
        
        
        """
        此处加入自己的业务逻辑代码
        """
        
        
  def run(self,args):
    args_queue = queue.Queue()
    for value in args:
      args_queue.put(value)
    threads = []
    for i in range(self.num_thread):
      threads.append(threading.Thread(target=self.object_func,args = args_queue))
    for t in threads:
      t.start()
    for t in threads:
      t.join()

模型参数选择实例

# -*- coding: utf-8 -*-
# @Time : 2021/2/4 14:36
# @Author : CyrusMay WJ
# @FileName: run.py
# @Software: PyCharm
# @Blog :https://blog.csdn.net/Cyrus_May
import queue
import threading
import numpy as np
from sklearn.datasets import load_boston
from sklearn.svm import SVR
import logging
import sys


class CyrusThread(object):
  def __init__(self,num_thread = 10,logger=None):
    """

    :param num_thread: 线程数
    :param logger: 日志对象
    """
    self.num_thread = num_thread
    self.logger = logger

  def object_func(self,args_queue,max_q):
    while 1:
      try:
        arg = args_queue.get_nowait()
        step = args_queue.qsize()
        self.logger.info("progress:{}\{}".format(max_q,max_q-step))
      except:
        self.logger.info("no more arg for args_queue!")
        break
      # 业务代码
      C, epsilon, gamma = arg[0], arg[1], arg[2]
      svr_model = SVR(C=C, epsilon=epsilon, gamma=gamma)
      x, y = load_boston()["data"], load_boston()["target"]
      svr_model.fit(x, y)
      self.logger.info("score:{}".format(svr_model.score(x,y)))


  def run(self,args):
    args_queue = queue.Queue()
    max_q = 0
    for value in args:
      args_queue.put(value)
      max_q += 1
    threads = []
    for i in range(self.num_thread):
      threads.append(threading.Thread(target=self.object_func,args = (args_queue,max_q)))
    for t in threads:
      t.start()
    for t in threads:
      t.join()

# 创建日志对象
logger = logging.getLogger()
logger.setLevel(logging.INFO)
screen_handler = logging.StreamHandler(sys.stdout)
screen_handler.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(module)s.%(funcName)s:%(lineno)d - %(levelname)s - %(message)s')
screen_handler.setFormatter(formatter)
logger.addHandler(screen_handler)

# 创建需要调整参数的集合
args = []
for C in [i for i in np.arange(0.01,1,0.01)]:
  for epsilon in [i for i in np.arange(0.001,1,0.01)] + [i for i in range(1,10,1)]:
    for gamma in [i for i in np.arange(0.001,1,0.01)] + [i for i in range(1,10,1)]:
      args.append([C,epsilon,gamma])

# 创建多线程工具
threading_tool = CyrusThread(num_thread=20,logger=logger)
threading_tool.run(args)

运行结果

2021-02-04 20:52:22,824 - run.object_func:31 - INFO - progress:1176219\1
2021-02-04 20:52:22,824 - run.object_func:31 - INFO - progress:1176219\2
2021-02-04 20:52:22,826 - run.object_func:31 - INFO - progress:1176219\3
2021-02-04 20:52:22,833 - run.object_func:31 - INFO - progress:1176219\4
2021-02-04 20:52:22,837 - run.object_func:31 - INFO - progress:1176219\5
2021-02-04 20:52:22,838 - run.object_func:31 - INFO - progress:1176219\6
2021-02-04 20:52:22,841 - run.object_func:31 - INFO - progress:1176219\7
2021-02-04 20:52:22,862 - run.object_func:31 - INFO - progress:1176219\8
2021-02-04 20:52:22,873 - run.object_func:31 - INFO - progress:1176219\9
2021-02-04 20:52:22,884 - run.object_func:31 - INFO - progress:1176219\10
2021-02-04 20:52:22,885 - run.object_func:31 - INFO - progress:1176219\11
2021-02-04 20:52:22,897 - run.object_func:31 - INFO - progress:1176219\12
2021-02-04 20:52:22,900 - run.object_func:31 - INFO - progress:1176219\13
2021-02-04 20:52:22,904 - run.object_func:31 - INFO - progress:1176219\14
2021-02-04 20:52:22,912 - run.object_func:31 - INFO - progress:1176219\15
2021-02-04 20:52:22,920 - run.object_func:31 - INFO - progress:1176219\16
2021-02-04 20:52:22,920 - run.object_func:39 - INFO - score:-0.01674283914287855
2021-02-04 20:52:22,929 - run.object_func:31 - INFO - progress:1176219\17
2021-02-04 20:52:22,932 - run.object_func:39 - INFO - score:-0.007992354170952565
2021-02-04 20:52:22,932 - run.object_func:31 - INFO - progress:1176219\18
2021-02-04 20:52:22,945 - run.object_func:31 - INFO - progress:1176219\19
2021-02-04 20:52:22,954 - run.object_func:31 - INFO - progress:1176219\20
2021-02-04 20:52:22,978 - run.object_func:31 - INFO - progress:1176219\21
2021-02-04 20:52:22,984 - run.object_func:39 - INFO - score:-0.018769934807246536
2021-02-04 20:52:22,985 - run.object_func:31 - INFO - progress:1176219\22

到此这篇关于python中threading和queue库实现多线程编程的文章就介绍到这了,更多相关python 多线程编程内容请搜索三水点靠木以前的文章或继续浏览下面的相关文章希望大家以后多多支持三水点靠木!

Python 相关文章推荐
一个简单的python程序实例(通讯录)
Nov 29 Python
Python数组定义方法
Apr 13 Python
Python 2与Python 3版本和编码的对比
Feb 14 Python
Python3中使用PyMongo的方法详解
Jul 28 Python
Python编程django实现同一个ip十分钟内只能注册一次
Nov 03 Python
解决python3中解压zip文件是文件名乱码的问题
Mar 22 Python
Numpy数组的保存与读取方法
Apr 04 Python
使用python爬取B站千万级数据
Jun 08 Python
Tensorflow 合并通道及加载子模型的方法
Jul 26 Python
pyqt5 lineEdit设置密码隐藏,删除lineEdit已输入的内容等属性方法
Jun 24 Python
Python图像处理库PIL的ImageFilter模块使用介绍
Feb 26 Python
利用python进行文件操作
Dec 04 Python
Python3爬虫ChromeDriver的安装实例
Feb 06 #Python
解决pycharm修改代码后第一次运行不生效的问题
Feb 06 #Python
Python tkinter之ComboBox(下拉框)的使用简介
Feb 05 #Python
python批量提取图片信息并保存的实现
Feb 05 #Python
Python的轻量级ORM框架peewee使用教程
Feb 05 #Python
pycharm 实现光标快速移动到括号外或行尾的操作
Feb 05 #Python
pycharm进入时每次都是insert模式的解决方式
Feb 05 #Python
You might like
CakePHP去除默认显示的标题及图标的方法
2008/10/22 PHP
windows7下安装php的php-ssh2扩展教程
2014/07/04 PHP
10个超级有用的PHP代码片段果断收藏
2015/09/23 PHP
YUI 读码日记之 YAHOO.util.Dom - Part.1
2008/03/22 Javascript
javascript的onchange事件与jQuery的change()方法比较
2009/09/28 Javascript
ExtJS 设置级联菜单的默认值
2010/06/13 Javascript
jquery点击页面任何区域实现鼠标焦点十字效果
2013/06/21 Javascript
JS自动适应的图片弹窗实例
2013/06/29 Javascript
javascript控制Div层透明属性由浅变深由深变浅逐渐显示
2013/11/12 Javascript
详细解读JavaScript的跨浏览器事件处理
2015/08/12 Javascript
js动态生成form 并用ajax方式提交的实现方法
2016/09/09 Javascript
React中的render何时执行过程
2018/04/13 Javascript
JavaScript 正则命名分组【推荐】
2018/06/07 Javascript
快速解决vue-cli在ie9+中无效的问题
2018/09/04 Javascript
微信小程序学习笔记之跳转页面、传递参数获得数据操作图文详解
2019/03/28 Javascript
Electron 调用命令行(cmd)
2019/09/23 Javascript
javascript实现鼠标点击生成文字特效
2019/12/24 Javascript
node事件循环和process模块实例分析
2020/02/14 Javascript
[43:32]2014 DOTA2华西杯精英邀请赛 5 25 LGD VS NewBee第一场
2014/05/26 DOTA
Ubuntu下安装PyV8
2016/03/13 Python
python实现基于SVM手写数字识别功能
2020/05/27 Python
获取python文件扩展名和文件名方法
2018/02/02 Python
python买卖股票的最佳时机(基于贪心/蛮力算法)
2019/07/05 Python
python线程安全及多进程多线程实现方法详解
2019/09/27 Python
python飞机大战 pygame游戏创建快速入门详解
2019/12/17 Python
使用Python来做一个屏幕录制工具的操作代码
2020/01/18 Python
Python实现链表反转的方法分析【迭代法与递归法】
2020/02/22 Python
python matplotlib 绘图 和 dpi对应关系详解
2020/03/14 Python
python使用PIL剪切和拼接图片
2020/03/23 Python
python之pygame模块实现飞机大战完整代码
2020/11/29 Python
继承时候类的执行顺序问题,一般都是选择题,问你将会打印出什么?
2015/11/18 面试题
毕业设计计划书
2014/01/09 职场文书
幼儿园小班家长评语
2014/12/30 职场文书
2016年12月份红领巾广播稿
2015/12/21 职场文书
Python四款GUI图形界面库介绍
2022/06/05 Python
使用python生成大量数据写入es数据库并查询操作(2)
2022/09/23 Python