python队列queue模块详解


Posted in Python onApril 27, 2018

队列queue 多应用在多线程应用中,多线程访问共享变量。对于多线程而言,访问共享变量时,队列queue是线程安全的。从queue队列的具体实现中,可以看出queue使用了1个线程互斥锁(pthread.Lock()),以及3个条件标量(pthread.condition()),来保证了线程安全。

queue队列的互斥锁和条件变量,可以参考另一篇文章:python线程中同步锁

queue的用法如下:

import Queque 
a=[1,2,3] 
device_que=Queque.queue() 
device_que.put(a) 
device=device_que.get()

先看看它的初始化函数__init__(self,maxsize=0):

def __init__(self, maxsize=0): 
 self.maxsize = maxsize 
 self._init(maxsize) 
 # mutex must be held whenever the queue is mutating. All methods 
 # that acquire mutex must release it before returning. mutex 
 # is shared between the three conditions, so acquiring and 
 # releasing the conditions also acquires and releases mutex. 
 self.mutex = _threading.Lock() 
 # Notify not_empty whenever an item is added to the queue; a 
 # thread waiting to get is notified then. 
 self.not_empty = _threading.Condition(self.mutex) 
 # Notify not_full whenever an item is removed from the queue; 
 # a thread waiting to put is notified then. 
 self.not_full = _threading.Condition(self.mutex) 
 # Notify all_tasks_done whenever the number of unfinished tasks 
 # drops to zero; thread waiting to join() is notified to resume 
 self.all_tasks_done = _threading.Condition(self.mutex) 
 self.unfinished_tasks = 0

定义队列时有一个默认的参数maxsize, 如果不指定队列的长度,即manxsize=0,那么队列的长度为无限长,如果定义了大于0的值,那么队列的长度就是maxsize

self._init(maxsize):使用了python自带的双端队列deque,来存储元素。

self.mutex互斥锁:任何获取队列的状态(empty(),qsize()等),或者修改队列的内容的操作(get,put等)都必须持有该互斥锁。共有两种操作require获取锁,release释放锁。同时该互斥锁被三个共享变量同时享有,即操作conditiond时的require和release操作也就是操作了该互斥锁。

self.not_full条件变量:当队列中有元素添加后,会通知notify其他等待添加元素的线程,唤醒等待require互斥锁,或者有线程从队列中取出一个元素后,通知其它线程唤醒以等待require互斥锁。

self.not empty条件变量:线程添加数据到队列中后,会调用self.not_empty.notify()通知其它线程,唤醒等待require互斥锁后,读取队列。

self.all_tasks_done条件变量:消费者线程从队列中get到任务后,任务处理完成,当所有的队列中的任务处理完成后,会使调用queue.join()的线程返回,表示队列中任务以处理完毕。

queue.put(self, item, block=True, timeout=None)函数:

申请获得互斥锁,获得后,如果队列未满,则向队列中添加数据,并通知notify其它阻塞的某个线程,唤醒等待获取require互斥锁。如果队列已满,则会wait等待。最后处理完成后释放互斥锁。其中还有阻塞block以及非阻塞,超时等逻辑,可以自己看一下:

def put(self, item, block=True, timeout=None): 
 """Put an item into the queue. 
 
 If optional args 'block' is true and 'timeout' is None (the default), 
 block if necessary until a free slot is available. If 'timeout' is 
 a non-negative number, it blocks at most 'timeout' seconds and raises 
 the Full exception if no free slot was available within that time. 
 Otherwise ('block' is false), put an item on the queue if a free slot 
 is immediately available, else raise the Full exception ('timeout' 
 is ignored in that case). 
 """ 
 self.not_full.acquire() 
 try: 
  if self.maxsize > 0: 
   if not block: 
    if self._qsize() == self.maxsize: 
     raise Full 
   elif timeout is None: 
    while self._qsize() == self.maxsize: 
     self.not_full.wait() 
   elif timeout < 0: 
    raise ValueError("'timeout' must be a non-negative number") 
   else: 
    endtime = _time() + timeout 
    while self._qsize() == self.maxsize: 
     remaining = endtime - _time() 
     if remaining <= 0.0: 
      raise Full 
     self.not_full.wait(remaining) 
  self._put(item) 
  self.unfinished_tasks += 1 
  self.not_empty.notify() 
 finally: 
  self.not_full.release()

queue.get(self, block=True, timeout=None)函数:

从队列中获取任务,并且从队列中移除此任务。首先尝试获取互斥锁,获取成功则队列中get任务,如果此时队列为空,则wait等待生产者线程添加数据。get到任务后,会调用self.not_full.notify()通知生产者线程,队列可以添加元素了。最后释放互斥锁。

def get(self, block=True, timeout=None): 
 """Remove and return an item from the queue. 
 
 If optional args 'block' is true and 'timeout' is None (the default), 
 block if necessary until an item is available. If 'timeout' is 
 a non-negative number, it blocks at most 'timeout' seconds and raises 
 the Empty exception if no item was available within that time. 
 Otherwise ('block' is false), return an item if one is immediately 
 available, else raise the Empty exception ('timeout' is ignored 
 in that case). 
 """ 
 self.not_empty.acquire() 
 try: 
  if not block: 
   if not self._qsize(): 
    raise Empty 
  elif timeout is None: 
   while not self._qsize(): 
    self.not_empty.wait() 
  elif timeout < 0: 
   raise ValueError("'timeout' must be a non-negative number") 
  else: 
   endtime = _time() + timeout 
   while not self._qsize(): 
    remaining = endtime - _time() 
    if remaining <= 0.0: 
     raise Empty 
    self.not_empty.wait(remaining) 
  item = self._get() 
  self.not_full.notify() 
  return item 
 finally: 
  self.not_empty.release()

queue.put_nowait():无阻塞的向队列中添加任务,当队列为满时,不等待,而是直接抛出full异常,重点是理解block=False:

def put_nowait(self, item): 
 """Put an item into the queue without blocking. 
 
 Only enqueue the item if a free slot is immediately available. 
 Otherwise raise the Full exception. 
 """ 
 return self.put(item, False)

queue.get_nowait():无阻塞的向队列中get任务,当队列为空时,不等待,而是直接抛出empty异常,重点是理解block=False:

def get_nowait(self): 
  """Remove and return an item from the queue without blocking. 
 
  Only get an item if one is immediately available. Otherwise 
  raise the Empty exception. 
  """ 
  return self.get(False)

queue.qsize empty full 分别获取队列的长度,是否为空,是否已满等:

def qsize(self): 
 """Return the approximate size of the queue (not reliable!).""" 
 self.mutex.acquire() 
 n = self._qsize() 
 self.mutex.release() 
 return n 
 
def empty(self): 
 """Return True if the queue is empty, False otherwise (not reliable!).""" 
 self.mutex.acquire() 
 n = not self._qsize() 
 self.mutex.release() 
 return n 
 
def full(self): 
 """Return True if the queue is full, False otherwise (not reliable!).""" 
 self.mutex.acquire() 
 n = 0 < self.maxsize == self._qsize() 
 self.mutex.release() 
 return n

queue.join()阻塞等待队列中任务全部处理完毕,需要配合queue.task_done使用:

def task_done(self): 
 """Indicate that a formerly enqueued task is complete. 
 
 Used by Queue consumer threads. For each get() used to fetch a task, 
 a subsequent call to task_done() tells the queue that the processing 
 on the task is complete. 
 
 If a join() is currently blocking, it will resume when all items 
 have been processed (meaning that a task_done() call was received 
 for every item that had been put() into the queue). 
 
 Raises a ValueError if called more times than there were items 
 placed in the queue. 
 """ 
 self.all_tasks_done.acquire() 
 try: 
  unfinished = self.unfinished_tasks - 1 
  if unfinished <= 0: 
   if unfinished < 0: 
    raise ValueError('task_done() called too many times') 
   self.all_tasks_done.notify_all() 
  self.unfinished_tasks = unfinished 
 finally: 
  self.all_tasks_done.release() 
 
def join(self): 
 """Blocks until all items in the Queue have been gotten and processed. 
 
 The count of unfinished tasks goes up whenever an item is added to the 
 queue. The count goes down whenever a consumer thread calls task_done() 
 to indicate the item was retrieved and all work on it is complete. 
 
 When the count of unfinished tasks drops to zero, join() unblocks. 
 """ 
 self.all_tasks_done.acquire() 
 try: 
  while self.unfinished_tasks: 
   self.all_tasks_done.wait() 
 finally: 
  self.all_tasks_done.release()

Queue模块除了queue线性安全队列(先进先出),还有优先级队列LifoQueue(后进先出),也就是新添加的先被get到。PriorityQueue具有优先级的队列,即队列中的元素是一个元祖类型,(优先级级别,数据)。

class PriorityQueue(Queue): 
 '''''Variant of Queue that retrieves open entries in priority order (lowest first). 
 
 Entries are typically tuples of the form: (priority number, data). 
 ''' 
 
 def _init(self, maxsize): 
  self.queue = [] 
 
 def _qsize(self, len=len): 
  return len(self.queue) 
 
 def _put(self, item, heappush=heapq.heappush): 
  heappush(self.queue, item) 
 
 def _get(self, heappop=heapq.heappop): 
  return heappop(self.queue) 
 
 
class LifoQueue(Queue): 
 '''''Variant of Queue that retrieves most recently added entries first.''' 
 
 def _init(self, maxsize): 
  self.queue = [] 
 
 def _qsize(self, len=len): 
  return len(self.queue) 
 
 def _put(self, item): 
  self.queue.append(item) 
 
 def _get(self): 
  return self.queue.pop()

至此queue模块介绍完毕,重点是理解互斥锁,条件变量如果协同工作,保证队列的线程安全。

下面是queue的完全代码:

class Queue: 
 """Create a queue object with a given maximum size. 
 
 If maxsize is <= 0, the queue size is infinite. 
 """ 
 def __init__(self, maxsize=0): 
  self.maxsize = maxsize 
  self._init(maxsize) 
  # mutex must be held whenever the queue is mutating. All methods 
  # that acquire mutex must release it before returning. mutex 
  # is shared between the three conditions, so acquiring and 
  # releasing the conditions also acquires and releases mutex. 
  self.mutex = _threading.Lock() 
  # Notify not_empty whenever an item is added to the queue; a 
  # thread waiting to get is notified then. 
  self.not_empty = _threading.Condition(self.mutex) 
  # Notify not_full whenever an item is removed from the queue; 
  # a thread waiting to put is notified then. 
  self.not_full = _threading.Condition(self.mutex) 
  # Notify all_tasks_done whenever the number of unfinished tasks 
  # drops to zero; thread waiting to join() is notified to resume 
  self.all_tasks_done = _threading.Condition(self.mutex) 
  self.unfinished_tasks = 0 
 
 def task_done(self): 
  """Indicate that a formerly enqueued task is complete. 
 
  Used by Queue consumer threads. For each get() used to fetch a task, 
  a subsequent call to task_done() tells the queue that the processing 
  on the task is complete. 
 
  If a join() is currently blocking, it will resume when all items 
  have been processed (meaning that a task_done() call was received 
  for every item that had been put() into the queue). 
 
  Raises a ValueError if called more times than there were items 
  placed in the queue. 
  """ 
  self.all_tasks_done.acquire() 
  try: 
   unfinished = self.unfinished_tasks - 1 
   if unfinished <= 0: 
    if unfinished < 0: 
     raise ValueError('task_done() called too many times') 
    self.all_tasks_done.notify_all() 
   self.unfinished_tasks = unfinished 
  finally: 
   self.all_tasks_done.release() 
 
 def join(self): 
  """Blocks until all items in the Queue have been gotten and processed. 
 
  The count of unfinished tasks goes up whenever an item is added to the 
  queue. The count goes down whenever a consumer thread calls task_done() 
  to indicate the item was retrieved and all work on it is complete. 
 
  When the count of unfinished tasks drops to zero, join() unblocks. 
  """ 
  self.all_tasks_done.acquire() 
  try: 
   while self.unfinished_tasks: 
    self.all_tasks_done.wait() 
  finally: 
   self.all_tasks_done.release() 
 
 def qsize(self): 
  """Return the approximate size of the queue (not reliable!).""" 
  self.mutex.acquire() 
  n = self._qsize() 
  self.mutex.release() 
  return n 
 
 def empty(self): 
  """Return True if the queue is empty, False otherwise (not reliable!).""" 
  self.mutex.acquire() 
  n = not self._qsize() 
  self.mutex.release() 
  return n 
 
 def full(self): 
  """Return True if the queue is full, False otherwise (not reliable!).""" 
  self.mutex.acquire() 
  n = 0 < self.maxsize == self._qsize() 
  self.mutex.release() 
  return n 
 
 def put(self, item, block=True, timeout=None): 
  """Put an item into the queue. 
 
  If optional args 'block' is true and 'timeout' is None (the default), 
  block if necessary until a free slot is available. If 'timeout' is 
  a non-negative number, it blocks at most 'timeout' seconds and raises 
  the Full exception if no free slot was available within that time. 
  Otherwise ('block' is false), put an item on the queue if a free slot 
  is immediately available, else raise the Full exception ('timeout' 
  is ignored in that case). 
  """ 
  self.not_full.acquire() 
  try: 
   if self.maxsize > 0: 
    if not block: 
     if self._qsize() == self.maxsize: 
      raise Full 
    elif timeout is None: 
     while self._qsize() == self.maxsize: 
      self.not_full.wait() 
    elif timeout < 0: 
     raise ValueError("'timeout' must be a non-negative number") 
    else: 
     endtime = _time() + timeout 
     while self._qsize() == self.maxsize: 
      remaining = endtime - _time() 
      if remaining <= 0.0: 
       raise Full 
      self.not_full.wait(remaining) 
   self._put(item) 
   self.unfinished_tasks += 1 
   self.not_empty.notify() 
  finally: 
   self.not_full.release() 
 
 def put_nowait(self, item): 
  """Put an item into the queue without blocking. 
 
  Only enqueue the item if a free slot is immediately available. 
  Otherwise raise the Full exception. 
  """ 
  return self.put(item, False) 
 
 def get(self, block=True, timeout=None): 
  """Remove and return an item from the queue. 
 
  If optional args 'block' is true and 'timeout' is None (the default), 
  block if necessary until an item is available. If 'timeout' is 
  a non-negative number, it blocks at most 'timeout' seconds and raises 
  the Empty exception if no item was available within that time. 
  Otherwise ('block' is false), return an item if one is immediately 
  available, else raise the Empty exception ('timeout' is ignored 
  in that case). 
  """ 
  self.not_empty.acquire() 
  try: 
   if not block: 
    if not self._qsize(): 
     raise Empty 
   elif timeout is None: 
    while not self._qsize(): 
     self.not_empty.wait() 
   elif timeout < 0: 
    raise ValueError("'timeout' must be a non-negative number") 
   else: 
    endtime = _time() + timeout 
    while not self._qsize(): 
     remaining = endtime - _time() 
     if remaining <= 0.0: 
      raise Empty 
     self.not_empty.wait(remaining) 
   item = self._get() 
   self.not_full.notify() 
   return item 
  finally: 
   self.not_empty.release() 
 
 def get_nowait(self): 
  """Remove and return an item from the queue without blocking. 
 
  Only get an item if one is immediately available. Otherwise 
  raise the Empty exception. 
  """ 
  return self.get(False) 
 
 # Override these methods to implement other queue organizations 
 # (e.g. stack or priority queue). 
 # These will only be called with appropriate locks held 
 
 # Initialize the queue representation 
 def _init(self, maxsize): 
  self.queue = deque() 
 
 def _qsize(self, len=len): 
  return len(self.queue) 
 
 # Put a new item in the queue 
 def _put(self, item): 
  self.queue.append(item) 
 
 # Get an item from the queue 
 def _get(self): 
  return self.queue.popleft()

以上就是本文的全部内容,希望对大家的学习有所帮助,也希望大家多多支持三水点靠木。

Python 相关文章推荐
使用BeautifulSoup爬虫程序获取百度搜索结果的标题和url示例
Jan 19 Python
Python实现截屏的函数
Jul 25 Python
python实现在IDLE中输入多行的方法
Apr 19 Python
Python操作MySQL数据库的方法
Jun 20 Python
Python协程操作之gevent(yield阻塞,greenlet),协程实现多任务(有规律的交替协作执行)用法详解
Oct 14 Python
详解Django CAS 解决方案
Oct 30 Python
在python中创建指定大小的多维数组方式
Nov 28 Python
在Python IDLE 下调用anaconda中的库教程
Mar 09 Python
jupyternotebook 撤销删除的操作方式
Apr 17 Python
在django admin中配置搜索域是一个外键时的处理方法
May 20 Python
python 实现德洛内三角剖分的操作
Apr 22 Python
详解pytorch创建tensor函数
Mar 22 Python
浅谈tensorflow1.0 池化层(pooling)和全连接层(dense)
Apr 27 #Python
python线程中同步锁详解
Apr 27 #Python
python数字图像处理之高级形态学处理
Apr 27 #Python
python线程池threadpool实现篇
Apr 27 #Python
python数字图像处理之骨架提取与分水岭算法
Apr 27 #Python
python多线程之事件Event的使用详解
Apr 27 #Python
python线程池threadpool使用篇
Apr 27 #Python
You might like
农民和部队如何穿矿
2020/03/04 星际争霸
判断php数组是否为索引数组的实现方法
2013/06/13 PHP
两个php日期控制类实例
2014/12/09 PHP
PHP结合jquery ajax实现上传多张图片,并限制图片大小操作示例
2019/03/01 PHP
javascript实现控制浏览器全屏
2015/03/30 Javascript
JavaScript数据库TaffyDB用法实例分析
2015/07/27 Javascript
jQuery实现Email邮箱地址自动补全功能代码
2015/11/03 Javascript
学习JavaScript设计模式(策略模式)
2015/11/26 Javascript
微信小程序 教程之WXSS
2016/10/18 Javascript
浅析JavaScript中break、continue和return的区别
2016/11/30 Javascript
js 轮播效果实例分享
2016/12/28 Javascript
前端框架学习总结之Angular、React与Vue的比较详解
2017/03/14 Javascript
js异步编程小技巧详解
2017/08/14 Javascript
js判断传入时间和当前时间大小实例(超简单)
2018/01/11 Javascript
Javasript设计模式之链式调用详解
2018/04/26 Javascript
Element-ui tree组件自定义节点使用方法代码详解
2018/09/17 Javascript
Python struct.unpack
2008/09/06 Python
python迭代器实例简析
2014/09/25 Python
python实现决策树分类(2)
2018/08/30 Python
Python 普通最小二乘法(OLS)进行多项式拟合的方法
2018/12/29 Python
python通过SSH登陆linux并操作的实现
2019/10/10 Python
Python是怎样处理json模块的
2020/07/16 Python
html5记忆翻牌游戏实现思路及代码
2013/07/25 HTML / CSS
浅谈利用缓存来优化HTML5 Canvas程序的性能
2015/05/12 HTML / CSS
图片上传插件ImgUploadJS:用HTML5 File API 实现截图粘贴上传、拖拽上传
2016/01/20 HTML / CSS
台湾SHOPRO购物行家:亚洲首创影视.3C.家电.优质购物平台
2018/05/07 全球购物
什么是规则表达式
2012/05/03 面试题
事业单位分类改革实施方案
2014/03/21 职场文书
驾驶员培训方案
2014/05/01 职场文书
舞蹈专业求职信
2014/06/13 职场文书
理财学专业自荐书
2014/06/28 职场文书
三好学生事迹材料
2014/12/24 职场文书
2016春季田径运动会广播稿
2015/12/21 职场文书
2016党校学习心得体会
2016/01/07 职场文书
mysql函数全面总结
2021/11/11 MySQL
口袋妖怪冰系十大最强精灵,几何雪花排第七,第六类似北极熊
2022/03/18 日漫