python开发之基于thread线程搜索本地文件的方法


Posted in Python onNovember 11, 2015

本文实例讲述了python开发之基于thread线程搜索本地文件的方法。分享给大家供大家参考,具体如下:

先来看看运行效果图:

python开发之基于thread线程搜索本地文件的方法

利用多个线程处理搜索的问题,我们可以发现他很快....

下面是代码部分:

# A parallelized "find(1)" using the thread module.
# This demonstrates the use of a work queue and worker threads.
# It really does do more stats/sec when using multiple threads,
# although the improvement is only about 20-30 percent.
# (That was 8 years ago. In 2002, on Linux, I can't measure
# a speedup. :-( )
# I'm too lazy to write a command line parser for the full find(1)
# command line syntax, so the predicate it searches for is wired-in,
# see function selector() below. (It currently searches for files with
# world write permission.)
# Usage: parfind.py [-w nworkers] [directory] ...
# Default nworkers is 4
import sys
import getopt
import time
import os
from stat import *
import _thread as thread
# Work queue class. Usage:
#  wq = WorkQ()
#  wq.addwork(func, (arg1, arg2, ...)) # one or more calls
#  wq.run(nworkers)
# The work is done when wq.run() completes.
# The function calls executed by the workers may add more work.
# Don't use keyboard interrupts!
class WorkQ:
  # Invariants:
  # - busy and work are only modified when mutex is locked
  # - len(work) is the number of jobs ready to be taken
  # - busy is the number of jobs being done
  # - todo is locked iff there is no work and somebody is busy
  def __init__(self):
    self.mutex = thread.allocate()
    self.todo = thread.allocate()
    self.todo.acquire()
    self.work = []
    self.busy = 0
  def addwork(self, func, args):
    job = (func, args)
    self.mutex.acquire()
    self.work.append(job)
    self.mutex.release()
    if len(self.work) == 1:
      self.todo.release()
  def _getwork(self):
    self.todo.acquire()
    self.mutex.acquire()
    if self.busy == 0 and len(self.work) == 0:
      self.mutex.release()
      self.todo.release()
      return None
    job = self.work[0]
    del self.work[0]
    self.busy = self.busy + 1
    self.mutex.release()
    if len(self.work) > 0:
      self.todo.release()
    return job
  def _donework(self):
    self.mutex.acquire()
    self.busy = self.busy - 1
    if self.busy == 0 and len(self.work) == 0:
      self.todo.release()
    self.mutex.release()
  def _worker(self):
    time.sleep(0.00001)   # Let other threads run
    while 1:
      job = self._getwork()
      if not job:
        break
      func, args = job
      func(*args)
      self._donework()
  def run(self, nworkers):
    if not self.work:
      return # Nothing to do
    for i in range(nworkers-1):
      thread.start_new(self._worker, ())
    self._worker()
    self.todo.acquire()
# Main program
def main():
  nworkers = 4
  #print(getopt.getopt(sys.argv[1:], '-w:'))
  opts, args = getopt.getopt(sys.argv[1:], '-w:')
  for opt, arg in opts:
    if opt == '-w':
      nworkers = int(arg)
  if not args:
    #print(os.curdir)
    args = [os.curdir]
  wq = WorkQ()
  for dir in args:
    wq.addwork(find, (dir, selector, wq))
  t1 = time.time()
  wq.run(nworkers)
  t2 = time.time()
  sys.stderr.write('Total time %r sec.\n' % (t2-t1))
# The predicate -- defines what files we look for.
# Feel free to change this to suit your purpose
def selector(dir, name, fullname, stat):
  # Look for world writable files that are not symlinks
  return (stat[ST_MODE] & 0o002) != 0 and not S_ISLNK(stat[ST_MODE])
# The find procedure -- calls wq.addwork() for subdirectories
def find(dir, pred, wq):
  try:
    names = os.listdir(dir)
  except os.error as msg:
    print(repr(dir), ':', msg)
    return
  for name in names:
    if name not in (os.curdir, os.pardir):
      fullname = os.path.join(dir, name)
      try:
        stat = os.lstat(fullname)
      except os.error as msg:
        print(repr(fullname), ':', msg)
        continue
      if pred(dir, name, fullname, stat):
        print(fullname)
      if S_ISDIR(stat[ST_MODE]):
        if not os.path.ismount(fullname):
          wq.addwork(find, (fullname, pred, wq))
# Call the main program
main()

希望本文所述对大家Python程序设计有所帮助。

Python 相关文章推荐
Python日期操作学习笔记
Oct 07 Python
python 安装virtualenv和virtualenvwrapper的方法
Jan 13 Python
Python字符串处理实例详解
May 18 Python
放弃 Python 转向 Go语言有人给出了 9 大理由
Oct 20 Python
python基础教程项目五之虚拟茶话会
Apr 02 Python
python爬取足球直播吧五大联赛积分榜
Jun 13 Python
Python设计模式之享元模式原理与用法实例分析
Jan 11 Python
关于PyTorch 自动求导机制详解
Aug 18 Python
python opencv根据颜色进行目标检测的方法示例
Jan 15 Python
pycharm中import呈现灰色原因的解决方法
Mar 04 Python
Pytorch损失函数nn.NLLLoss2d()用法说明
Jul 07 Python
七个Python必备的GUI库
Apr 27 Python
python开发之tkinter实现图形随鼠标移动的方法
Nov 11 #Python
Python复制文件操作实例详解
Nov 10 #Python
Python中对元组和列表按条件进行排序的方法示例
Nov 10 #Python
Python 文件管理实例详解
Nov 10 #Python
Python list操作用法总结
Nov 10 #Python
python控制台中实现进度条功能
Nov 10 #Python
使用Python发送各种形式的邮件的方法汇总
Nov 09 #Python
You might like
Apache+php+mysql在windows下的安装与配置图解(最新版)
2008/11/30 PHP
php如何调用webservice应用介绍
2012/11/24 PHP
zf框架db类的分页示例分享
2014/03/14 PHP
VB中的RasEnumConnections函数返回632错误解决方法
2014/07/29 PHP
php接口实现拖拽排序功能
2018/04/23 PHP
一端时间轮换的广告
2006/06/26 Javascript
jquery中对表单的基本操作代码
2010/07/29 Javascript
jQuery 源码分析笔记(3) Deferred机制
2011/06/19 Javascript
jQuery侧边栏随窗口滚动实现方法
2013/03/04 Javascript
轻松创建nodejs服务器(4):路由
2014/12/18 NodeJs
Node.js node-schedule定时任务隔多少分钟执行一次的方法
2015/02/10 Javascript
jQuery+css实现的蓝色水平二级导航菜单效果代码
2015/09/11 Javascript
基于 Vue 实现一个酷炫的 menu插件
2017/11/14 Javascript
Js中使用正则表达式验证输入是否有特殊字符
2018/09/07 Javascript
分享5个好用的javascript文件上传插件
2018/09/16 Javascript
详解关于element级联选择器数据回显问题
2019/02/20 Javascript
微信打开网址添加在浏览器中打开提示的办法
2019/05/20 Javascript
layui 弹出层回调获取弹出层数据的例子
2019/09/02 Javascript
vue中使用element ui的弹窗与echarts之间的问题详解
2019/10/25 Javascript
VUE.CLI4.0配置多页面入口的实现
2019/11/25 Javascript
Vue 图片压缩并上传至服务器功能
2020/01/15 Javascript
node脚手架搭建服务器实现token验证的方法
2021/01/20 Javascript
[05:09]2016国际邀请赛中国区预选赛淘汰赛首日精彩回顾
2016/06/29 DOTA
Python搭建HTTP服务器和FTP服务器
2017/03/09 Python
python回调函数中使用多线程的方法
2017/12/25 Python
python获取微信小程序手机号并绑定遇到的坑
2018/11/19 Python
Python学习笔记之文件的读写操作实例分析
2019/08/07 Python
利用Python产生加密表和解密表的实现方法
2019/10/15 Python
python的time模块和datetime模块实例解析
2019/11/29 Python
pytorch 求网络模型参数实例
2019/12/30 Python
Python基于stuck实现scoket文件传输
2020/04/02 Python
Python中lru_cache的使用和实现详解
2021/01/25 Python
元旦促销方案
2014/03/15 职场文书
民政局标准版离婚协议书
2014/12/01 职场文书
幼儿园教研工作总结2015
2015/05/12 职场文书
浏览器常用基本操作之python3+selenium4自动化测试(基础篇3)
2021/05/21 Python