python开发之基于thread线程搜索本地文件的方法


Posted in Python onNovember 11, 2015

本文实例讲述了python开发之基于thread线程搜索本地文件的方法。分享给大家供大家参考,具体如下:

先来看看运行效果图:

python开发之基于thread线程搜索本地文件的方法

利用多个线程处理搜索的问题,我们可以发现他很快....

下面是代码部分:

# A parallelized "find(1)" using the thread module.
# This demonstrates the use of a work queue and worker threads.
# It really does do more stats/sec when using multiple threads,
# although the improvement is only about 20-30 percent.
# (That was 8 years ago. In 2002, on Linux, I can't measure
# a speedup. :-( )
# I'm too lazy to write a command line parser for the full find(1)
# command line syntax, so the predicate it searches for is wired-in,
# see function selector() below. (It currently searches for files with
# world write permission.)
# Usage: parfind.py [-w nworkers] [directory] ...
# Default nworkers is 4
import sys
import getopt
import time
import os
from stat import *
import _thread as thread
# Work queue class. Usage:
#  wq = WorkQ()
#  wq.addwork(func, (arg1, arg2, ...)) # one or more calls
#  wq.run(nworkers)
# The work is done when wq.run() completes.
# The function calls executed by the workers may add more work.
# Don't use keyboard interrupts!
class WorkQ:
  # Invariants:
  # - busy and work are only modified when mutex is locked
  # - len(work) is the number of jobs ready to be taken
  # - busy is the number of jobs being done
  # - todo is locked iff there is no work and somebody is busy
  def __init__(self):
    self.mutex = thread.allocate()
    self.todo = thread.allocate()
    self.todo.acquire()
    self.work = []
    self.busy = 0
  def addwork(self, func, args):
    job = (func, args)
    self.mutex.acquire()
    self.work.append(job)
    self.mutex.release()
    if len(self.work) == 1:
      self.todo.release()
  def _getwork(self):
    self.todo.acquire()
    self.mutex.acquire()
    if self.busy == 0 and len(self.work) == 0:
      self.mutex.release()
      self.todo.release()
      return None
    job = self.work[0]
    del self.work[0]
    self.busy = self.busy + 1
    self.mutex.release()
    if len(self.work) > 0:
      self.todo.release()
    return job
  def _donework(self):
    self.mutex.acquire()
    self.busy = self.busy - 1
    if self.busy == 0 and len(self.work) == 0:
      self.todo.release()
    self.mutex.release()
  def _worker(self):
    time.sleep(0.00001)   # Let other threads run
    while 1:
      job = self._getwork()
      if not job:
        break
      func, args = job
      func(*args)
      self._donework()
  def run(self, nworkers):
    if not self.work:
      return # Nothing to do
    for i in range(nworkers-1):
      thread.start_new(self._worker, ())
    self._worker()
    self.todo.acquire()
# Main program
def main():
  nworkers = 4
  #print(getopt.getopt(sys.argv[1:], '-w:'))
  opts, args = getopt.getopt(sys.argv[1:], '-w:')
  for opt, arg in opts:
    if opt == '-w':
      nworkers = int(arg)
  if not args:
    #print(os.curdir)
    args = [os.curdir]
  wq = WorkQ()
  for dir in args:
    wq.addwork(find, (dir, selector, wq))
  t1 = time.time()
  wq.run(nworkers)
  t2 = time.time()
  sys.stderr.write('Total time %r sec.\n' % (t2-t1))
# The predicate -- defines what files we look for.
# Feel free to change this to suit your purpose
def selector(dir, name, fullname, stat):
  # Look for world writable files that are not symlinks
  return (stat[ST_MODE] & 0o002) != 0 and not S_ISLNK(stat[ST_MODE])
# The find procedure -- calls wq.addwork() for subdirectories
def find(dir, pred, wq):
  try:
    names = os.listdir(dir)
  except os.error as msg:
    print(repr(dir), ':', msg)
    return
  for name in names:
    if name not in (os.curdir, os.pardir):
      fullname = os.path.join(dir, name)
      try:
        stat = os.lstat(fullname)
      except os.error as msg:
        print(repr(fullname), ':', msg)
        continue
      if pred(dir, name, fullname, stat):
        print(fullname)
      if S_ISDIR(stat[ST_MODE]):
        if not os.path.ismount(fullname):
          wq.addwork(find, (fullname, pred, wq))
# Call the main program
main()

希望本文所述对大家Python程序设计有所帮助。

Python 相关文章推荐
python判断、获取一张图片主色调的2个实例
Apr 10 Python
Python实现字符串反转的常用方法分析【4种方法】
Sep 30 Python
python如何实现int函数的方法示例
Feb 19 Python
关于python2 csv写入空白行的问题
Jun 22 Python
django连接mysql配置方法总结(推荐)
Aug 18 Python
pandas实现将dataframe满足某一条件的值选出
Jun 12 Python
python字符串查找函数的用法详解
Jul 08 Python
Python 使用指定的网卡发送HTTP请求的实例
Aug 21 Python
python爬虫 猫眼电影和电影天堂数据csv和mysql存储过程解析
Sep 05 Python
flask框架配置mysql数据库操作详解
Nov 29 Python
基于FME使用Python过程图解
May 13 Python
10个顶级Python实用库推荐
Mar 04 Python
python开发之tkinter实现图形随鼠标移动的方法
Nov 11 #Python
Python复制文件操作实例详解
Nov 10 #Python
Python中对元组和列表按条件进行排序的方法示例
Nov 10 #Python
Python 文件管理实例详解
Nov 10 #Python
Python list操作用法总结
Nov 10 #Python
python控制台中实现进度条功能
Nov 10 #Python
使用Python发送各种形式的邮件的方法汇总
Nov 09 #Python
You might like
深入PHP autoload机制的详解
2013/06/09 PHP
php curl post 时出现的问题解决
2014/01/30 PHP
php中array_column函数简单实现方法
2016/07/11 PHP
php中引用&的用法分析【变量引用,函数引用,对象引用】
2016/12/12 PHP
网页源代码保护(禁止右键、复制、另存为、查看源文件)
2012/05/23 Javascript
jQuery动态添加 input type=file的实现代码
2012/06/14 Javascript
JS 加入收藏夹的代码(主流浏览器通用)
2013/05/13 Javascript
js过滤HTML标签以及空格的思路及代码
2013/05/24 Javascript
js图片滚动效果时间可随意设定当鼠标移上去时停止
2014/06/26 Javascript
基于jQuery实现最基本的淡入淡出效果实例
2015/02/02 Javascript
JS获取iframe中marginHeight和marginWidth属性的方法
2015/04/01 Javascript
AngularJS使用ng-repeat指令实现下拉框
2016/08/23 Javascript
JS常用算法实现代码
2016/11/14 Javascript
基于Javascript实现的不重复ID的生成器
2016/12/25 Javascript
Vue侧滑菜单组件——DrawerLayout
2017/12/18 Javascript
vue2过滤器模糊查询方法
2018/09/16 Javascript
JS执行控制之节流模式实例分析
2018/12/21 Javascript
python数据结构之列表和元组的详解
2017/09/23 Python
Python实现按特定格式对文件进行读写的方法示例
2017/11/30 Python
Python中列表与元组的乘法操作示例
2018/02/10 Python
python实现远程通过网络邮件控制计算机重启或关机
2018/02/22 Python
Python破解BiliBili滑块验证码的思路详解(完美避开人机识别)
2020/02/17 Python
Python开发之pip安装及使用方法详解
2020/02/21 Python
Python数据可视化实现漏斗图过程图解
2020/07/20 Python
分享PyCharm最新激活码(真永久激活方法)不用每月找安装参数或最新激活码了
2020/12/27 Python
Foot Locker德国官方网站:美国运动服和鞋类零售商
2018/11/01 全球购物
5个HTML5的常用本地存储方式详解与介绍
2021/03/27 HTML / CSS
计算机专业毕业生推荐信
2013/11/25 职场文书
试用期转正鉴定评语
2014/01/27 职场文书
新文化运动的基本口号
2014/06/21 职场文书
现实表现证明材料
2015/06/19 职场文书
pytorch常用数据类型所占字节数对照表一览
2021/05/17 Python
Python数据可视化之绘制柱状图和条形图
2021/05/25 Python
浅谈MySQL函数
2021/10/05 MySQL
MySQL创建管理RANGE分区
2022/04/13 MySQL
JavaScript实现九宫格拖拽效果
2022/06/28 Javascript