Python多进程multiprocessing用法实例分析


Posted in Python onAugust 18, 2017

本文实例讲述了Python多进程multiprocessing用法。分享给大家供大家参考,具体如下:

mutilprocess简介

像线程一样管理进程,这个是mutilprocess的核心,他与threading很是相像,对多核CPU的利用率会比threading好的多。

简单的创建进程:

import multiprocessing
def worker(num):
  """thread worker function"""
  print 'Worker:', num
  return
if __name__ == '__main__':
  jobs = []
  for i in range(5):
    p = multiprocessing.Process(target=worker, args=(i,))
    jobs.append(p)
    p.start()

确定当前的进程,即是给进程命名,方便标识区分,跟踪

import multiprocessing
import time
def worker():
  name = multiprocessing.current_process().name
  print name, 'Starting'
  time.sleep(2)
  print name, 'Exiting'
def my_service():
  name = multiprocessing.current_process().name
  print name, 'Starting'
  time.sleep(3)
  print name, 'Exiting'
if __name__ == '__main__':
  service = multiprocessing.Process(name='my_service',
                   target=my_service)
  worker_1 = multiprocessing.Process(name='worker 1',
                    target=worker)
  worker_2 = multiprocessing.Process(target=worker) # default name
  worker_1.start()
  worker_2.start()
  service.start()

守护进程就是不阻挡主程序退出,自己干自己的 mutilprocess.setDaemon(True)就这句等待守护进程退出,要加上join,join可以传入浮点数值,等待n久就不等了

守护进程:

import multiprocessing
import time
import sys
def daemon():
  name = multiprocessing.current_process().name
  print 'Starting:', name
  time.sleep(2)
  print 'Exiting :', name
def non_daemon():
  name = multiprocessing.current_process().name
  print 'Starting:', name
  print 'Exiting :', name
if __name__ == '__main__':
  d = multiprocessing.Process(name='daemon',
                target=daemon)
  d.daemon = True
  n = multiprocessing.Process(name='non-daemon',
                target=non_daemon)
  n.daemon = False
  d.start()
  n.start()
  d.join(1)
  print 'd.is_alive()', d.is_alive()
  n.join()

最好使用 poison pill,强制的使用terminate()注意 terminate之后要join,使其可以更新状态

终止进程:

import multiprocessing
import time
def slow_worker():
  print 'Starting worker'
  time.sleep(0.1)
  print 'Finished worker'
if __name__ == '__main__':
  p = multiprocessing.Process(target=slow_worker)
  print 'BEFORE:', p, p.is_alive()
  p.start()
  print 'DURING:', p, p.is_alive()
  p.terminate()
  print 'TERMINATED:', p, p.is_alive()
  p.join()
  print 'JOINED:', p, p.is_alive()

①. == 0 未生成任何错误 
②. 0 进程有一个错误,并以该错误码退出
③. < 0 进程由一个-1 * exitcode信号结束

进程的退出状态:

import multiprocessing
import sys
import time
def exit_error():
  sys.exit(1)
def exit_ok():
  return
def return_value():
  return 1
def raises():
  raise RuntimeError('There was an error!')
def terminated():
  time.sleep(3)
if __name__ == '__main__':
  jobs = []
  for f in [exit_error, exit_ok, return_value, raises, terminated]:
    print 'Starting process for', f.func_name
    j = multiprocessing.Process(target=f, name=f.func_name)
    jobs.append(j)
    j.start()
  jobs[-1].terminate()
  for j in jobs:
    j.join()
    print '%15s.exitcode = %s' % (j.name, j.exitcode)

方便的调试,可以用logging

日志:

import multiprocessing
import logging
import sys
def worker():
  print 'Doing some work'
  sys.stdout.flush()
if __name__ == '__main__':
  multiprocessing.log_to_stderr()
  logger = multiprocessing.get_logger()
  logger.setLevel(logging.INFO)
  p = multiprocessing.Process(target=worker)
  p.start()
  p.join()

利用class来创建进程,定制子类

派生进程:

import multiprocessing
class Worker(multiprocessing.Process):
  def run(self):
    print 'In %s' % self.name
    return
if __name__ == '__main__':
  jobs = []
  for i in range(5):
    p = Worker()
    jobs.append(p)
    p.start()
  for j in jobs:
    j.join()

python进程间传递消息:

import multiprocessing
class MyFancyClass(object):
  def __init__(self, name):
    self.name = name
  def do_something(self):
    proc_name = multiprocessing.current_process().name
    print 'Doing something fancy in %s for %s!' % \
      (proc_name, self.name)
def worker(q):
  obj = q.get()
  obj.do_something()
if __name__ == '__main__':
  queue = multiprocessing.Queue()
  p = multiprocessing.Process(target=worker, args=(queue,))
  p.start()
  queue.put(MyFancyClass('Fancy Dan'))
  # Wait for the worker to finish
  queue.close()
  queue.join_thread()
  p.join()
import multiprocessing
import time
class Consumer(multiprocessing.Process):
  def __init__(self, task_queue, result_queue):
    multiprocessing.Process.__init__(self)
    self.task_queue = task_queue
    self.result_queue = result_queue
  def run(self):
    proc_name = self.name
    while True:
      next_task = self.task_queue.get()
      if next_task is None:
        # Poison pill means shutdown
        print '%s: Exiting' % proc_name
        self.task_queue.task_done()
        break
      print '%s: %s' % (proc_name, next_task)
      answer = next_task()
      self.task_queue.task_done()
      self.result_queue.put(answer)
    return
class Task(object):
  def __init__(self, a, b):
    self.a = a
    self.b = b
  def __call__(self):
    time.sleep(0.1) # pretend to take some time to do the work
    return '%s * %s = %s' % (self.a, self.b, self.a * self.b)
  def __str__(self):
    return '%s * %s' % (self.a, self.b)
if __name__ == '__main__':
  # Establish communication queues
  tasks = multiprocessing.JoinableQueue()
  results = multiprocessing.Queue()
  # Start consumers
  num_consumers = multiprocessing.cpu_count() * 2
  print 'Creating %d consumers' % num_consumers
  consumers = [ Consumer(tasks, results)
         for i in xrange(num_consumers) ]
  for w in consumers:
    w.start()
  # Enqueue jobs
  num_jobs = 10
  for i in xrange(num_jobs):
    tasks.put(Task(i, i))
  # Add a poison pill for each consumer
  for i in xrange(num_consumers):
    tasks.put(None)
  # Wait for all of the tasks to finish
  tasks.join()
  # Start printing results
  while num_jobs:
    result = results.get()
    print 'Result:', result
    num_jobs -= 1

Event提供一种简单的方法,可以在进程间传递状态信息。事件可以切换设置和未设置状态。通过使用一个可选的超时值,时间对象的用户可以等待其状态从未设置变为设置。

进程间信号传递:

import multiprocessing
import time
def wait_for_event(e):
  """Wait for the event to be set before doing anything"""
  print 'wait_for_event: starting'
  e.wait()
  print 'wait_for_event: e.is_set()->', e.is_set()
def wait_for_event_timeout(e, t):
  """Wait t seconds and then timeout"""
  print 'wait_for_event_timeout: starting'
  e.wait(t)
  print 'wait_for_event_timeout: e.is_set()->', e.is_set()
if __name__ == '__main__':
  e = multiprocessing.Event()
  w1 = multiprocessing.Process(name='block', 
                 target=wait_for_event,
                 args=(e,))
  w1.start()
  w2 = multiprocessing.Process(name='nonblock', 
                 target=wait_for_event_timeout, 
                 args=(e, 2))
  w2.start()
  print 'main: waiting before calling Event.set()'
  time.sleep(3)
  e.set()
  print 'main: event is set'

Python多进程,一般的情况是Queue来传递。

Queue:

from multiprocessing import Process, Queue
def f(q):
  q.put([42, None, 'hello'])
if __name__ == '__main__':
  q = Queue()
  p = Process(target=f, args=(q,))
  p.start()
  print q.get()  # prints "[42, None, 'hello']"
  p.join()

多线程优先队列Queue:

import Queue
import threading
import time
exitFlag = 0
class myThread (threading.Thread):
  def __init__(self, threadID, name, q):
    threading.Thread.__init__(self)
    self.threadID = threadID
    self.name = name
    self.q = q
  def run(self):
    print "Starting " + self.name
    process_data(self.name, self.q)
    print "Exiting " + self.name
def process_data(threadName, q):
  while not exitFlag:
    queueLock.acquire()
    if not workQueue.empty():
      data = q.get()
      queueLock.release()
      print "%s processing %s" % (threadName, data)
    else:
      queueLock.release()
    time.sleep(1)
threadList = ["Thread-1", "Thread-2", "Thread-3"]
nameList = ["One", "Two", "Three", "Four", "Five"]
queueLock = threading.Lock()
workQueue = Queue.Queue(10)
threads = []
threadID = 1
# Create new threads
for tName in threadList:
  thread = myThread(threadID, tName, workQueue)
  thread.start()
  threads.append(thread)
  threadID += 1
# Fill the queue
queueLock.acquire()
for word in nameList:
  workQueue.put(word)
queueLock.release()
# Wait for queue to empty
while not workQueue.empty():
  pass
# Notify threads it's time to exit
exitFlag = 1
# Wait for all threads to complete
for t in threads:
  t.join()
print "Exiting Main Thread"

多进程使用Queue通信的例子

import time
from multiprocessing import Process,Queue
MSG_QUEUE = Queue(5)
def startA(msgQueue):
  while True:
    if msgQueue.empty() > 0:
      print ('queue is empty %d' % (msgQueue.qsize()))
    else:
      msg = msgQueue.get()
      print( 'get msg %s' % (msg,))
    time.sleep(1)
def startB(msgQueue):
  while True:
    msgQueue.put('hello world')
    print( 'put hello world queue size is %d' % (msgQueue.qsize(),))
    time.sleep(3)
if __name__ == '__main__':
  processA = Process(target=startA,args=(MSG_QUEUE,))
  processB = Process(target=startB,args=(MSG_QUEUE,))
  processA.start()
  print( 'processA start..')

主进程定义了一个Queue类型的变量,并作为Process的args参数传给子进程processA和processB,两个进程一个向队列中写数据,一个读数据。

希望本文所述对大家Python程序设计有所帮助。

Python 相关文章推荐
Python日期操作学习笔记
Oct 07 Python
python连接mysql并提交mysql事务示例
Mar 05 Python
Python getopt模块处理命令行选项实例
May 13 Python
python中getaddrinfo()基本用法实例分析
Jun 28 Python
Python基于TCP实现会聊天的小机器人功能示例
Apr 09 Python
python celery分布式任务队列的使用详解
Jul 08 Python
python批量处理txt文件的实例代码
Jan 13 Python
django创建超级用户时指定添加其它字段方式
May 14 Python
python爬取代理IP并进行有效的IP测试实现
Oct 09 Python
python 如何使用find和find_all爬虫、找文本的实现
Oct 16 Python
python 实现一个简单的线性回归案例
Dec 17 Python
python双向链表实例详解
May 25 Python
利用python求相邻数的方法示例
Aug 18 #Python
Python3.5编程实现修改IIS WEB.CONFIG的方法示例
Aug 18 #Python
Python 中 Virtualenv 和 pip 的简单用法详解
Aug 18 #Python
Python3编程实现获取阿里云ECS实例及监控的方法
Aug 18 #Python
浅谈django开发者模式中的autoreload是如何实现的
Aug 18 #Python
Python绑定方法与非绑定方法详解
Aug 18 #Python
python字典DICT类型合并详解
Aug 17 #Python
You might like
《星际争霸》各版本雷兽特点图文解析 雷兽不同形态一览
2020/03/02 星际争霸
php设计模式 Composite (组合模式)
2011/06/26 PHP
php实现读取超大文件的方法
2014/07/28 PHP
PHP中引用类型和值类型功能与用法示例
2019/02/26 PHP
微信公众平台开发教程③ PHP实现微信公众号支付功能图文详解
2019/04/10 PHP
jQuery的链式调用浅析
2010/12/03 Javascript
Jquery通过Ajax访问XML数据的小例子
2013/11/18 Javascript
AngularJS 实现按需异步加载实例代码
2015/10/18 Javascript
快速学习jQuery插件 jquery.validate.js表单验证插件使用方法
2015/12/01 Javascript
Bootstrap媒体对象的实现
2016/05/01 Javascript
41个Web开发者必须收藏的JavaScript实用技巧
2016/07/22 Javascript
AngularJS 实现弹性盒子布局的方法
2016/08/30 Javascript
Angularjs 实现分页功能及示例代码
2016/09/14 Javascript
同步异步动态引入js文件的几种方法总结
2016/09/23 Javascript
jQuery实现的手风琴侧边菜单效果
2017/03/29 jQuery
BootStrap Table 后台数据绑定、特殊列处理、排序功能
2017/05/27 Javascript
Vue2.0 事件的广播与接收(观察者模式)
2018/03/14 Javascript
Vue页面跳转动画效果的实现方法
2018/09/23 Javascript
基于Three.js实现360度全景图片
2018/12/30 Javascript
Vue 开发必须知道的36个技巧(小结)
2019/10/09 Javascript
[08:29]DOTA2每周TOP10 精彩击杀集锦vol.7
2014/06/25 DOTA
python实现去除下载电影和电视剧文件名中的多余字符的方法
2014/09/23 Python
JSON Web Tokens的实现原理
2017/04/02 Python
python 重定向获取真实url的方法
2018/05/11 Python
Python如何使用argparse模块处理命令行参数
2019/12/11 Python
解决python replace函数替换无效问题
2020/01/18 Python
什么是python的id函数
2020/06/11 Python
纯html5+css3下拉导航菜单实现代码
2013/03/18 HTML / CSS
英国领先的维生素和补充剂品牌:Higher Nature
2019/08/26 全球购物
举例说明类变量和实例变量的区别
2016/06/30 面试题
勾股定理课后反思
2014/04/26 职场文书
公司员工辞职信范文
2015/05/12 职场文书
2015年社区教育工作总结
2015/05/13 职场文书
MATLAB 如何求取离散点的曲率最大值
2021/04/16 Python
PyQt5爬取12306车票信息程序的实现
2021/05/14 Python
python_tkinter事件类型详情
2022/03/20 Python