python开发之基于thread线程搜索本地文件的方法


Posted in Python onNovember 11, 2015

本文实例讲述了python开发之基于thread线程搜索本地文件的方法。分享给大家供大家参考,具体如下:

先来看看运行效果图:

python开发之基于thread线程搜索本地文件的方法

利用多个线程处理搜索的问题,我们可以发现他很快....

下面是代码部分:

# A parallelized "find(1)" using the thread module.
# This demonstrates the use of a work queue and worker threads.
# It really does do more stats/sec when using multiple threads,
# although the improvement is only about 20-30 percent.
# (That was 8 years ago. In 2002, on Linux, I can't measure
# a speedup. :-( )
# I'm too lazy to write a command line parser for the full find(1)
# command line syntax, so the predicate it searches for is wired-in,
# see function selector() below. (It currently searches for files with
# world write permission.)
# Usage: parfind.py [-w nworkers] [directory] ...
# Default nworkers is 4
import sys
import getopt
import time
import os
from stat import *
import _thread as thread
# Work queue class. Usage:
#  wq = WorkQ()
#  wq.addwork(func, (arg1, arg2, ...)) # one or more calls
#  wq.run(nworkers)
# The work is done when wq.run() completes.
# The function calls executed by the workers may add more work.
# Don't use keyboard interrupts!
class WorkQ:
  # Invariants:
  # - busy and work are only modified when mutex is locked
  # - len(work) is the number of jobs ready to be taken
  # - busy is the number of jobs being done
  # - todo is locked iff there is no work and somebody is busy
  def __init__(self):
    self.mutex = thread.allocate()
    self.todo = thread.allocate()
    self.todo.acquire()
    self.work = []
    self.busy = 0
  def addwork(self, func, args):
    job = (func, args)
    self.mutex.acquire()
    self.work.append(job)
    self.mutex.release()
    if len(self.work) == 1:
      self.todo.release()
  def _getwork(self):
    self.todo.acquire()
    self.mutex.acquire()
    if self.busy == 0 and len(self.work) == 0:
      self.mutex.release()
      self.todo.release()
      return None
    job = self.work[0]
    del self.work[0]
    self.busy = self.busy + 1
    self.mutex.release()
    if len(self.work) > 0:
      self.todo.release()
    return job
  def _donework(self):
    self.mutex.acquire()
    self.busy = self.busy - 1
    if self.busy == 0 and len(self.work) == 0:
      self.todo.release()
    self.mutex.release()
  def _worker(self):
    time.sleep(0.00001)   # Let other threads run
    while 1:
      job = self._getwork()
      if not job:
        break
      func, args = job
      func(*args)
      self._donework()
  def run(self, nworkers):
    if not self.work:
      return # Nothing to do
    for i in range(nworkers-1):
      thread.start_new(self._worker, ())
    self._worker()
    self.todo.acquire()
# Main program
def main():
  nworkers = 4
  #print(getopt.getopt(sys.argv[1:], '-w:'))
  opts, args = getopt.getopt(sys.argv[1:], '-w:')
  for opt, arg in opts:
    if opt == '-w':
      nworkers = int(arg)
  if not args:
    #print(os.curdir)
    args = [os.curdir]
  wq = WorkQ()
  for dir in args:
    wq.addwork(find, (dir, selector, wq))
  t1 = time.time()
  wq.run(nworkers)
  t2 = time.time()
  sys.stderr.write('Total time %r sec.\n' % (t2-t1))
# The predicate -- defines what files we look for.
# Feel free to change this to suit your purpose
def selector(dir, name, fullname, stat):
  # Look for world writable files that are not symlinks
  return (stat[ST_MODE] & 0o002) != 0 and not S_ISLNK(stat[ST_MODE])
# The find procedure -- calls wq.addwork() for subdirectories
def find(dir, pred, wq):
  try:
    names = os.listdir(dir)
  except os.error as msg:
    print(repr(dir), ':', msg)
    return
  for name in names:
    if name not in (os.curdir, os.pardir):
      fullname = os.path.join(dir, name)
      try:
        stat = os.lstat(fullname)
      except os.error as msg:
        print(repr(fullname), ':', msg)
        continue
      if pred(dir, name, fullname, stat):
        print(fullname)
      if S_ISDIR(stat[ST_MODE]):
        if not os.path.ismount(fullname):
          wq.addwork(find, (fullname, pred, wq))
# Call the main program
main()

希望本文所述对大家Python程序设计有所帮助。

Python 相关文章推荐
基于python的汉字转GBK码实现代码
Feb 19 Python
Python urllib模块urlopen()与urlretrieve()详解
Nov 01 Python
详解详解Python中writelines()方法的使用
May 25 Python
python爬取51job中hr的邮箱
May 14 Python
Python教程之全局变量用法
Jun 27 Python
老生常谈Python序列化和反序列化
Jun 28 Python
Python利用ElementTree模块处理XML的方法详解
Aug 31 Python
详解python 拆包可迭代数据如tuple, list
Dec 29 Python
Python 实现选择排序的算法步骤
Apr 22 Python
Python基于pyCUDA实现GPU加速并行计算功能入门教程
Jun 19 Python
对django layer弹窗组件的使用详解
Aug 31 Python
Python实现曲线拟合的最小二乘法
Feb 19 Python
python开发之tkinter实现图形随鼠标移动的方法
Nov 11 #Python
Python复制文件操作实例详解
Nov 10 #Python
Python中对元组和列表按条件进行排序的方法示例
Nov 10 #Python
Python 文件管理实例详解
Nov 10 #Python
Python list操作用法总结
Nov 10 #Python
python控制台中实现进度条功能
Nov 10 #Python
使用Python发送各种形式的邮件的方法汇总
Nov 09 #Python
You might like
APACHE的AcceptPathInfo指令使用介绍
2013/01/18 PHP
Windows下的PHP安装文件线程安全和非线程安全的区别
2014/04/23 PHP
PHP中开启gzip压缩的2种方法
2015/01/31 PHP
Yii2框架自定义验证规则操作示例
2019/02/08 PHP
jQuery EasyUI API 中文文档 DateTimeBox日期时间框
2011/10/16 Javascript
为什么要在引入的css或者js文件后面加参数的详细讲解
2013/05/03 Javascript
ExtJs设置GridPanel表格文本垂直居中示例
2013/07/15 Javascript
JavaScript中常见的字符串操作函数及用法汇总
2015/05/04 Javascript
js判断手机号运营商的方法
2015/10/23 Javascript
基于jQuery实现动态搜索显示功能
2016/05/05 Javascript
DOM中事件处理概览与原理的全面解析
2016/08/16 Javascript
详解vue2.0组件通信各种情况总结与实例分析
2017/03/22 Javascript
selenium 与 chrome 进行qq登录并发邮件操作实例详解
2017/04/06 Javascript
js实现随机点名小功能
2017/08/17 Javascript
jQuery实现使用sort方法对json数据排序的方法
2018/04/17 jQuery
vue中npm包全局安装和局部安装过程
2019/09/03 Javascript
vue实现图片上传到后台
2020/06/29 Javascript
[04:45]上海特级锦标赛主赛事第三日TOP10
2016/03/05 DOTA
Python内置函数的用法实例教程
2014/09/08 Python
简单了解OpenCV是个什么东西
2017/11/10 Python
对Python生成汉字字库文字,以及转换为文字图片的实例详解
2019/01/29 Python
Python之时间和日期使用小结
2019/02/14 Python
pandas dataframe的合并实现(append, merge, concat)
2019/06/24 Python
浅谈对pytroch中torch.autograd.backward的思考
2019/12/27 Python
Python pyautogui模块实现鼠标键盘自动化方法详解
2020/02/17 Python
Python Django view 两种return的实现方式
2020/03/16 Python
Python实现动态循环输出文字功能
2020/05/07 Python
keras 获取某层输出 获取复用层的多次输出实例
2020/05/23 Python
为什么相对PHP黑python的更少
2020/06/21 Python
PyTorch安装与基本使用详解
2020/08/31 Python
Farnell德国:电子元器件供应商
2018/07/10 全球购物
ECCO英国官网:丹麦鞋履品牌
2019/09/03 全球购物
zooplus德国:便宜地订购动物用品、动物饲料、动物食品
2020/05/06 全球购物
平面设计专业求职信
2014/08/09 职场文书
HTML怎么设置下划线?html文字加下划线方法
2021/12/06 HTML / CSS
Python实现信息管理系统
2022/06/05 Python