探究Python多进程编程下线程之间变量的共享问题


Posted in Python onMay 05, 2015

 1、问题:

群中有同学贴了如下一段代码,问为何 list 最后打印的是空值?
 

from multiprocessing import Process, Manager
import os
 
manager = Manager()
vip_list = []
#vip_list = manager.list()
 
def testFunc(cc):
  vip_list.append(cc)
  print 'process id:', os.getpid()
 
if __name__ == '__main__':
  threads = []
 
  for ll in range(10):
    t = Process(target=testFunc, args=(ll,))
    t.daemon = True
    threads.append(t)
 
  for i in range(len(threads)):
    threads[i].start()
 
  for j in range(len(threads)):
    threads[j].join()
 
  print "------------------------"
  print 'process id:', os.getpid()
  print vip_list

其实如果你了解 python 的多线程模型,GIL 问题,然后了解多线程、多进程原理,上述问题不难回答,不过如果你不知道也没关系,跑一下上面的代码你就知道是什么问题了。
 

python aa.py
process id: 632
process id: 635
process id: 637
process id: 633
process id: 636
process id: 634
process id: 639
process id: 638
process id: 641
process id: 640
------------------------
process id: 619
[]

将第 6 行注释开启,你会看到如下结果:
 

process id: 32074
process id: 32073
process id: 32072
process id: 32078
process id: 32076
process id: 32071
process id: 32077
process id: 32079
process id: 32075
process id: 32080
------------------------
process id: 32066
[3, 2, 1, 7, 5, 0, 6, 8, 4, 9]

2、python 多进程共享变量的几种方式:
(1)Shared memory:
Data can be stored in a shared memory map using Value or Array. For example, the following code

http://docs.python.org/2/library/multiprocessing.html#sharing-state-between-processes
 

from multiprocessing import Process, Value, Array
 
def f(n, a):
  n.value = 3.1415927
  for i in range(len(a)):
    a[i] = -a[i]
 
if __name__ == '__main__':
  num = Value('d', 0.0)
  arr = Array('i', range(10))
 
  p = Process(target=f, args=(num, arr))
  p.start()
  p.join()
 
  print num.value
  print arr[:]

结果:
 

3.1415927
[0, -1, -2, -3, -4, -5, -6, -7, -8, -9]

(2)Server process:

A manager object returned by Manager() controls a server process which holds Python objects and allows other processes to manipulate them using proxies.
A manager returned by Manager() will support types list, dict, Namespace, Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Queue, Value and Array.
代码见开头的例子。

http://docs.python.org/2/library/multiprocessing.html#managers
3、多进程的问题远不止这么多:数据的同步

看段简单的代码:一个简单的计数器:
 

from multiprocessing import Process, Manager
import os
 
manager = Manager()
sum = manager.Value('tmp', 0)
 
def testFunc(cc):
  sum.value += cc
 
if __name__ == '__main__':
  threads = []
 
  for ll in range(100):
    t = Process(target=testFunc, args=(1,))
    t.daemon = True
    threads.append(t)
 
  for i in range(len(threads)):
    threads[i].start()
 
  for j in range(len(threads)):
    threads[j].join()
 
  print "------------------------"
  print 'process id:', os.getpid()
  print sum.value

结果:
 

------------------------
process id: 17378
97

也许你会问:WTF?其实这个问题在多线程时代就存在了,只是在多进程时代又杯具重演了而已:Lock!
 

from multiprocessing import Process, Manager, Lock
import os
 
lock = Lock()
manager = Manager()
sum = manager.Value('tmp', 0)
 
 
def testFunc(cc, lock):
  with lock:
    sum.value += cc
 
 
if __name__ == '__main__':
  threads = []
 
  for ll in range(100):
    t = Process(target=testFunc, args=(1, lock))
    t.daemon = True
    threads.append(t)
 
  for i in range(len(threads)):
    threads[i].start()
 
  for j in range(len(threads)):
    threads[j].join()
 
  print "------------------------"
  print 'process id:', os.getpid()
  print sum.value

这段代码性能如何呢?跑跑看,或者加大循环次数试一下。。。
4、最后的建议:

    Note that usually sharing data between processes may not be the best choice, because of all the synchronization issues; an approach involving actors exchanging messages is usually seen as a better choice. See also Python documentation: As mentioned above, when doing concurrent programming it is usually best to avoid using shared state as far as possible. This is particularly true when using multiple processes. However, if you really do need to use some shared data then multiprocessing provides a couple of ways of doing so.

5、Refer:

http://stackoverflow.com/questions/14124588/python-multiprocessing-shared-memory

http://eli.thegreenplace.net/2012/01/04/shared-counter-with-pythons-multiprocessing/

http://docs.python.org/2/library/multiprocessing.html#multiprocessing.sharedctypes.synchronized

Python 相关文章推荐
Python微信库:itchat的用法详解
Aug 14 Python
利用selenium 3.7和python3添加cookie模拟登陆的实现
Nov 20 Python
pandas 小数位数 精度的处理方法
Jun 09 Python
Django中数据库的数据关系:一对一,一对多,多对多
Oct 21 Python
windows下 兼容Python2和Python3的解决方法
Dec 05 Python
pandas分别写入excel的不同sheet方法
Dec 11 Python
pthon贪吃蛇游戏详细代码
Jan 27 Python
用Python实现最速下降法求极值的方法
Jul 10 Python
Python使用百度api做人脸对比的方法
Aug 28 Python
python GUI库图形界面开发之PyQt5中QMainWindow, QWidget以及QDialog的区别和选择
Feb 26 Python
python爬虫利用selenium实现自动翻页爬取某鱼数据的思路详解
Dec 22 Python
python数字图像处理之图像自动阈值分割示例
Jun 28 Python
浅谈Python中的数据类型
May 05 #Python
用Python实现一个简单的能够上传下载的HTTP服务器
May 05 #Python
使用Python程序抓取新浪在国内的所有IP的教程
May 04 #Python
Python版微信红包分配算法
May 04 #Python
用Python编写一个每天都在系统下新建一个文件夹的脚本
May 04 #Python
用Python编写生成树状结构的文件目录的脚本的教程
May 04 #Python
使用Python脚本将Bing的每日图片作为桌面的教程
May 04 #Python
You might like
《逃离塔科夫》——“萌新劝退,老手自嗨”的硬核FPS游戏
2020/04/03 其他游戏
php header()函数使用说明
2008/07/10 PHP
mysql limit查询优化分析
2008/11/12 PHP
php生成rss类用法实例
2015/04/14 PHP
教你在header中隐藏php的版本信息
2016/08/10 PHP
浅析PHP echo 和 print 语句
2020/06/30 PHP
JavaScipt基本教程之前言
2008/01/16 Javascript
多选列表框动态添加,移动,删除,全选等操作的简单实例
2014/01/13 Javascript
基于HTML+CSS,jQuery编写的简易计算器后续(添加了键盘监听)
2016/01/05 Javascript
Angular.js 实现数字转换汉字实例代码
2016/07/14 Javascript
javascript与jquery动态创建html元素示例
2016/07/25 Javascript
Javascript 获取鼠标当前的位置实现方法
2016/10/27 Javascript
jQuery时间验证和转换为标准格式的时间格式
2017/03/06 Javascript
JavaScript hasOwnProperty() 函数实例详解
2017/08/04 Javascript
js 倒计时(高效率服务器时间同步)
2017/09/12 Javascript
JavaScript实现shuffle数组洗牌操作示例
2019/01/03 Javascript
微信小程序实现同一页面取值的方法分析
2019/04/30 Javascript
vue+mock.js实现前后端分离
2019/07/24 Javascript
详解element上传组件before-remove钩子问题解决
2020/04/08 Javascript
浅谈Vuex的this.$store.commit和在Vue项目中引用公共方法
2020/07/24 Javascript
Vue和React有哪些区别
2020/09/12 Javascript
[06:07]刀塔密之二:攻之吾命受之吾幸
2014/07/03 DOTA
[02:02:38]VG vs Mineski Supermajor 败者组 BO3 第一场 6.6
2018/06/07 DOTA
Python库urllib与urllib2主要区别分析
2014/07/13 Python
Python爬虫实现爬取京东手机页面的图片(实例代码)
2017/11/30 Python
利用Python如何批量更新服务器文件
2018/07/29 Python
对python制作自己的数据集实例讲解
2018/12/12 Python
python 用下标截取字符串的实例
2018/12/25 Python
Python for i in range ()用法详解
2020/09/18 Python
python 截取XML中bndbox的坐标中的图像,另存为jpg的实例
2020/03/10 Python
Jupyter notebook 远程配置及SSL加密教程
2020/04/14 Python
Nike英国官网:Nike.com (UK)
2017/02/13 全球购物
Conforama西班牙:您的家具、装饰和电器商店
2020/02/21 全球购物
环保建议书400字
2014/05/14 职场文书
堂吉诃德读书笔记
2015/06/30 职场文书
Vue如何清空对象
2022/03/03 Vue.js