python开发之基于thread线程搜索本地文件的方法


Posted in Python onNovember 11, 2015

本文实例讲述了python开发之基于thread线程搜索本地文件的方法。分享给大家供大家参考,具体如下:

先来看看运行效果图:

python开发之基于thread线程搜索本地文件的方法

利用多个线程处理搜索的问题,我们可以发现他很快....

下面是代码部分:

# A parallelized "find(1)" using the thread module.
# This demonstrates the use of a work queue and worker threads.
# It really does do more stats/sec when using multiple threads,
# although the improvement is only about 20-30 percent.
# (That was 8 years ago. In 2002, on Linux, I can't measure
# a speedup. :-( )
# I'm too lazy to write a command line parser for the full find(1)
# command line syntax, so the predicate it searches for is wired-in,
# see function selector() below. (It currently searches for files with
# world write permission.)
# Usage: parfind.py [-w nworkers] [directory] ...
# Default nworkers is 4
import sys
import getopt
import time
import os
from stat import *
import _thread as thread
# Work queue class. Usage:
#  wq = WorkQ()
#  wq.addwork(func, (arg1, arg2, ...)) # one or more calls
#  wq.run(nworkers)
# The work is done when wq.run() completes.
# The function calls executed by the workers may add more work.
# Don't use keyboard interrupts!
class WorkQ:
  # Invariants:
  # - busy and work are only modified when mutex is locked
  # - len(work) is the number of jobs ready to be taken
  # - busy is the number of jobs being done
  # - todo is locked iff there is no work and somebody is busy
  def __init__(self):
    self.mutex = thread.allocate()
    self.todo = thread.allocate()
    self.todo.acquire()
    self.work = []
    self.busy = 0
  def addwork(self, func, args):
    job = (func, args)
    self.mutex.acquire()
    self.work.append(job)
    self.mutex.release()
    if len(self.work) == 1:
      self.todo.release()
  def _getwork(self):
    self.todo.acquire()
    self.mutex.acquire()
    if self.busy == 0 and len(self.work) == 0:
      self.mutex.release()
      self.todo.release()
      return None
    job = self.work[0]
    del self.work[0]
    self.busy = self.busy + 1
    self.mutex.release()
    if len(self.work) > 0:
      self.todo.release()
    return job
  def _donework(self):
    self.mutex.acquire()
    self.busy = self.busy - 1
    if self.busy == 0 and len(self.work) == 0:
      self.todo.release()
    self.mutex.release()
  def _worker(self):
    time.sleep(0.00001)   # Let other threads run
    while 1:
      job = self._getwork()
      if not job:
        break
      func, args = job
      func(*args)
      self._donework()
  def run(self, nworkers):
    if not self.work:
      return # Nothing to do
    for i in range(nworkers-1):
      thread.start_new(self._worker, ())
    self._worker()
    self.todo.acquire()
# Main program
def main():
  nworkers = 4
  #print(getopt.getopt(sys.argv[1:], '-w:'))
  opts, args = getopt.getopt(sys.argv[1:], '-w:')
  for opt, arg in opts:
    if opt == '-w':
      nworkers = int(arg)
  if not args:
    #print(os.curdir)
    args = [os.curdir]
  wq = WorkQ()
  for dir in args:
    wq.addwork(find, (dir, selector, wq))
  t1 = time.time()
  wq.run(nworkers)
  t2 = time.time()
  sys.stderr.write('Total time %r sec.\n' % (t2-t1))
# The predicate -- defines what files we look for.
# Feel free to change this to suit your purpose
def selector(dir, name, fullname, stat):
  # Look for world writable files that are not symlinks
  return (stat[ST_MODE] & 0o002) != 0 and not S_ISLNK(stat[ST_MODE])
# The find procedure -- calls wq.addwork() for subdirectories
def find(dir, pred, wq):
  try:
    names = os.listdir(dir)
  except os.error as msg:
    print(repr(dir), ':', msg)
    return
  for name in names:
    if name not in (os.curdir, os.pardir):
      fullname = os.path.join(dir, name)
      try:
        stat = os.lstat(fullname)
      except os.error as msg:
        print(repr(fullname), ':', msg)
        continue
      if pred(dir, name, fullname, stat):
        print(fullname)
      if S_ISDIR(stat[ST_MODE]):
        if not os.path.ismount(fullname):
          wq.addwork(find, (fullname, pred, wq))
# Call the main program
main()

希望本文所述对大家Python程序设计有所帮助。

Python 相关文章推荐
Python笔记(叁)继续学习
Oct 24 Python
Python序列操作之进阶篇
Dec 08 Python
Python的语言类型(详解)
Jun 24 Python
python自动裁剪图像代码分享
Nov 25 Python
Python使用pickle模块储存对象操作示例
Aug 15 Python
Python中collections模块的基本使用教程
Dec 07 Python
selenium+python自动化测试之环境搭建
Jan 23 Python
python爬取内容存入Excel实例
Feb 20 Python
Python pip替换为阿里源的方法步骤
Jul 02 Python
Python爬取破解无线网络wifi密码过程解析
Sep 17 Python
python怎么判断素数
Jul 01 Python
python 输入字符串生成所有有效的IP地址(LeetCode 93号题)
Oct 15 Python
python开发之tkinter实现图形随鼠标移动的方法
Nov 11 #Python
Python复制文件操作实例详解
Nov 10 #Python
Python中对元组和列表按条件进行排序的方法示例
Nov 10 #Python
Python 文件管理实例详解
Nov 10 #Python
Python list操作用法总结
Nov 10 #Python
python控制台中实现进度条功能
Nov 10 #Python
使用Python发送各种形式的邮件的方法汇总
Nov 09 #Python
You might like
PHP的FTP学习(一)[转自奥索]
2006/10/09 PHP
php 常用字符串函数总结
2008/03/15 PHP
用PHP ob_start()控制浏览器cache、生成html实现代码
2010/02/16 PHP
php中对2个数组相加的函数
2011/06/24 PHP
Laravel学习教程之View模块详解
2017/09/18 PHP
Smarty模板变量与调节器实例详解
2019/07/20 PHP
我的javascript 函数链之演变
2011/04/07 Javascript
将nodejs打包工具整合到鼠标右键的方法
2013/05/11 NodeJs
从数组中随机取x条不重复数据的JS代码
2013/12/24 Javascript
调整小数的格式保留小数点后两位
2014/05/14 Javascript
javascript继承的六大模式小结
2015/04/13 Javascript
jQuery验证插件validation使用指南
2015/04/21 Javascript
写给小白的JavaScript引擎指南
2015/12/04 Javascript
jquery获取table指定行和列的数据方法(当前选中行、列)
2016/11/07 Javascript
Vue事件修饰符native、self示例详解
2019/07/09 Javascript
Vue中computed和watch有哪些区别
2020/12/19 Vue.js
[01:33:25]DOTA2-DPC中国联赛 正赛 Elephant vs IG BO3 第一场 1月24日
2021/03/11 DOTA
Windows中安装使用Virtualenv来创建独立Python环境
2016/05/31 Python
python中如何使用正则表达式的非贪婪模式示例
2017/10/09 Python
解决nohup重定向python输出到文件不成功的问题
2018/05/11 Python
解决Python print输出不换行没空格的问题
2018/11/14 Python
3分钟学会一个Python小技巧
2018/11/23 Python
python使用xlrd模块读取xlsx文件中的ip方法
2019/01/11 Python
如何使用Flask-Migrate拓展数据库表结构
2019/07/24 Python
Django Python 获取请求头信息Content-Range的方法
2019/08/06 Python
Python缓存技术实现过程详解
2019/09/25 Python
Python Pickle 实现在同一个文件中序列化多个对象
2019/12/30 Python
用canvas显示验证码的实现
2020/04/10 HTML / CSS
个人自我评价分享
2013/12/20 职场文书
结婚邀请函范文
2014/01/14 职场文书
《月光启蒙》教学反思
2014/03/01 职场文书
应届生找工作求职信
2014/06/24 职场文书
海底两万里读书笔记
2015/06/26 职场文书
Python Pandas 删除列操作
2022/03/16 Python
Golang流模式之grpc的四种数据流
2022/04/13 Golang
面试官问我Mysql的存储引擎了解多少
2022/08/05 MySQL