8g内存用python读取10文件_面试题-python 如何读取一个大于 10G 的txt文件?


Posted in Python onMay 28, 2021

前言

用python 读取一个大于10G 的文件,自己电脑只有8G内存,一运行就报内存溢出:MemoryError

python 如何用open函数读取大文件呢?

读取大文件

首先可以自己先制作一个大于10G的txt文件

a = '''
2021-02-02 21:33:31,678 [django.request:93] [base:get_response] [WARNING]- Not Found: /http:/123.125.114.144/
2021-02-02 21:33:31,679 [django.server:124] [basehttp:log_message] [WARNING]- "HEAD http://123.125.114.144/ HTTP/1.1" 404 1678
2021-02-02 22:14:04,121 [django.server:124] [basehttp:log_message] [INFO]- code 400, message Bad request version ('HTTP')
2021-02-02 22:14:04,122 [django.server:124] [basehttp:log_message] [WARNING]- "GET ../../mnt/custom/ProductDefinition HTTP" 400 -
2021-02-02 22:16:21,052 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/login HTTP/1.1" 301 0
2021-02-02 22:16:21,123 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/login/ HTTP/1.1" 200 3876
2021-02-02 22:16:21,192 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/img/main_bg.png HTTP/1.1" 200 2801
2021-02-02 22:16:21,196 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/iconfont/style.css HTTP/1.1" 200 1638
2021-02-02 22:16:21,229 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/img/bg.jpg HTTP/1.1" 200 135990
2021-02-02 22:16:21,307 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/iconfont/fonts/icomoon.ttf?u4m6fy HTTP/1.1" 200 6900
2021-02-02 22:16:23,525 [django.server:124] [basehttp:log_message] [INFO]- "POST /api/login/ HTTP/1.1" 302 0
2021-02-02 22:16:23,618 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/index/ HTTP/1.1" 200 18447
2021-02-02 22:16:23,709 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/js/commons.js HTTP/1.1" 200 13209
2021-02-02 22:16:23,712 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/css/admin.css HTTP/1.1" 200 19660
2021-02-02 22:16:23,712 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/css/common.css HTTP/1.1" 200 1004
2021-02-02 22:16:23,714 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/js/app.js HTTP/1.1" 200 20844
2021-02-02 22:16:26,509 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/report_list/1/ HTTP/1.1" 200 14649
2021-02-02 22:16:51,496 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/test_list/1/ HTTP/1.1" 200 24874
2021-02-02 22:16:51,721 [django.server:124] [basehttp:log_message] [INFO]- "POST /api/add_case/ HTTP/1.1" 200 0
2021-02-02 22:16:59,707 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/test_list/1/ HTTP/1.1" 200 24874
2021-02-03 22:16:59,909 [django.server:124] [basehttp:log_message] [INFO]- "POST /api/add_case/ HTTP/1.1" 200 0
2021-02-03 22:17:01,306 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/edit_case/1/ HTTP/1.1" 200 36504
2021-02-03 22:17:06,265 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/add_project/ HTTP/1.1" 200 17737
2021-02-03 22:17:07,825 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/project_list/1/ HTTP/1.1" 200 29789
2021-02-03 22:17:13,116 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/add_config/ HTTP/1.1" 200 24816
2021-02-03 22:17:19,671 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/config_list/1/ HTTP/1.1" 200 19532
'''

while True:
with open("xxx.log", "a", encoding="utf-8") as fp:
fp.write(a)

循环写入到 xxx.log 文件,运行 3-5 分钟,pycharm 打开查看文件大小大于 10G

8g内存用python读取10文件_面试题-python 如何读取一个大于 10G 的txt文件?

于是我用open函数 直接读取

f = open("xxx.log", 'r')
print(f.read())
f.close()

抛出内存溢出异常:MemoryError

Traceback (most recent call last):
File "D:/2021kecheng06/demo/txt.py", line 35, in
print(f.read())
MemoryError

运行的时候可以看下自己电脑的内存已经占了100%, cpu高达91% ,不挂掉才怪了!

8g内存用python读取10文件_面试题-python 如何读取一个大于 10G 的txt文件?

这种错误的原因在于,read()方法执行操作是一次性的都读入内存中,显然文件大于内存就会报错。

read() 的几种方法

1.read() 方法可以带参数 n, n 是每次读取的大小长度,也就是可以每次读一部分,这样就不会导致内存溢出

f = open("xxx.log", 'r')
print(f.read(2048))
f.close()

运行结果

2019-10-24 21:33:31,678 [django.request:93] [base:get_response] [WARNING]- Not Found: /http:/123.125.114.144/
2019-10-24 21:33:31,679 [django.server:124] [basehttp:log_message] [WARNING]- "HEAD http://123.125.114.144/ HTTP/1.1" 404 1678
2019-10-24 22:14:04,121 [django.server:124] [basehttp:log_message] [INFO]- code 400, message Bad request version ('HTTP')
2019-10-24 22:14:04,122 [django.server:124] [basehttp:log_message] [WARNING]- "GET ../../mnt/custom/ProductDefinition HTTP" 400 -
2019-10-24 22:16:21,052 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/login HTTP/1.1" 301 0
2019-10-24 22:16:21,123 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/login/ HTTP/1.1" 200 3876
2019-10-24 22:16:21,192 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/img/main_bg.png HTTP/1.1" 200 2801
2019-10-24 22:16:21,196 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/iconfont/style.css HTTP/1.1" 200 1638
2019-10-24 22:16:21,229 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/img/bg.jpg HTTP/1.1" 200 135990
2019-10-24 22:16:21,307 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/iconfont/fonts/icomoon.ttf?u4m6fy HTTP/1.1" 200 6900
2019-10-24 22:16:23,525 [django.server:124] [basehttp:log_message] [INFO]- "POST /api/login/ HTTP/1.1" 302 0
2019-10-24 22:16:23,618 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/index/ HTTP/1.1" 200 18447
2019-10-24 22:16:23,709 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/js/commons.js HTTP/1.1" 200 13209
2019-10-24 22:16:23,712 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/css/admin.css HTTP/1.1" 200 19660
2019-10-24 22:16:23,712 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/css/common.css HTTP/1.1" 200 1004
2019-10-24 22:16:23,714 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/js/app.js HTTP/1.1" 200 20844

这样就只读取了2048个字符,全部读取的话,循环读就行

f = open("xxx.log", 'r')
while True:
block = f.read(2048)
print(block)
if not block:
break
f.close()
2.readline():每次读取一行,这个方法也不会报错
f = open("xxx.log", 'r')
while True:
line = f.readline()
print(line, end="")
if not line:
break
f.close()

3.readlines():读取全部的行,生成一个list,通过list来对文件进行处理,显然这种方式依然会造成:MemoyError

真正 Pythonic 的方法

真正 Pythonci 的方法,使用 with 结构打开文件,fp 是一个可迭代对象,可以用 for 遍历读取每行的文件内容

with open("xxx.log", 'r') as fp:

for line in fp:

print(line, end="")

yield 生成器读取大文件

前面一篇讲yield 生成器的时候提到读取大文件,函数返回一个可迭代对象,用next()方法读取文件内容

def read_file(fpath):

BLOCK_SIZE = 1024

with open(fpath, 'rb') as f:

while True:

block = f.read(BLOCK_SIZE)

if block:

yield block

else:

return

if __name__ == '__main__':

a = read_file("xxx.log")

print(a) # generator objec

print(next(a)) # bytes类型

print(next(a).decode("utf-8")) # str

运行结果

b'\r\n2019-10-24 21:33:31,678 [django.request:93] [base:get_response] [WARNING]- Not Found: /http:/123.125.114.144/\r\n2019-10-24 21:33:31,679 [django.server:124] [basehttp:log_message] [WARNING]- "HEAD http://123.125.114.144/ HTTP/1.1" 404 1678\r\n2019-10-24 22:14:04,121 [django.server:124] [basehttp:log_message] [INFO]- code 400, message Bad request version (\'HTTP\')\r\n2019-10-24 22:14:04,122 [django.server:124] [basehttp:log_message] [WARNING]- "GET ../../mnt/custom/ProductDefinition HTTP" 400 -\r\n2019-10-24 22:16:21,052 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/login HTTP/1.1" 301 0\r\n2019-10-24 22:16:21,123 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/login/ HTTP/1.1" 200 3876\r\n2019-10-24 22:16:21,192 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/img/main_bg.png HTTP/1.1" 200 2801\r\n2019-10-24 22:16:21,196 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/iconfont/style.css HTTP/1.1" 200 1638\r\n2019-10-24 22:16:21,229 [django.server:124] '

Python 相关文章推荐
Scrapy-redis爬虫分布式爬取的分析和实现
Feb 07 Python
python生成词云的实现方法(推荐)
Jun 13 Python
Python图像处理之颜色的定义与使用分析
Jan 03 Python
浅谈python新式类和旧式类区别
Apr 26 Python
python绘图模块matplotlib示例详解
Jul 26 Python
python中的列表与元组的使用
Aug 08 Python
Django实现文件上传和下载功能
Oct 06 Python
Python实现GIF图倒放
Jul 16 Python
基于python实现图片转字符画代码实例
Sep 04 Python
浅谈Python 中的复数问题
May 19 Python
Python字符串常规操作小结
Apr 03 Python
Django框架中模型的用法
Jun 10 Python
用python画城市轮播地图
用Python实现一个打字速度测试工具来测试你的手速
解决Pytorch dataloader时报错每个tensor维度不一样的问题
May 28 #Python
pytorch锁死在dataloader(训练时卡死)
Python趣味爬虫之用Python实现智慧校园一键评教
Pytorch 如何加速Dataloader提升数据读取速度
在前女友婚礼上,用Python破解了现场的WIFI还把名称改成了
You might like
火影忍者:三大瞳力之一的白眼,为什么没有写轮眼那么出色?
2020/03/02 日漫
php session安全问题分析
2011/06/24 PHP
php常见的魔术方法详解
2014/12/25 PHP
Gambit vs CL BO3 第一场 2.13
2021/03/10 DOTA
jquery高效反选具体实现
2013/05/05 Javascript
JS版元素周期表实现方法
2015/08/05 Javascript
javascript同步服务器时间和同步倒计时小技巧
2015/09/24 Javascript
jquery如何获取元素的滚动条高度等实现代码
2015/10/19 Javascript
全面解析Bootstrap表单使用方法(表单按钮)
2015/11/24 Javascript
jQuery实现移动端滑块拖动选择数字效果
2015/12/24 Javascript
有关jquery与DOM节点操作方法和属性记录
2016/04/15 Javascript
将Sublime Text 3 添加到右键中的简单方法
2017/12/12 Javascript
checkbox在vue中的用法小结
2018/11/13 Javascript
微信小程序位置授权处理方法
2019/06/13 Javascript
js实现鼠标滑动到某个div禁止滚动
2020/09/17 Javascript
微信小程序基于高德地图API实现天气组件(动态效果)
2020/10/22 Javascript
[02:53]2018年度DOTA2最佳战队-完美盛典
2018/12/17 DOTA
python正常时间和unix时间戳相互转换的方法
2015/04/23 Python
在Python中操作日期和时间之gmtime()方法的使用
2015/05/22 Python
linecache模块加载和缓存文件内容详解
2018/01/11 Python
通过python顺序修改文件名字的方法
2018/07/11 Python
Python使用Selenium模块模拟浏览器抓取斗鱼直播间信息示例
2018/07/18 Python
Python面向对象之接口、抽象类与多态详解
2018/08/27 Python
python实现C4.5决策树算法
2018/08/29 Python
详解Python with/as使用说明
2018/12/13 Python
python3中eval函数用法使用简介
2019/08/02 Python
Python循环实现n的全排列功能
2019/09/16 Python
Python缓存技术实现过程详解
2019/09/25 Python
Python跑循环时内存泄露的解决方法
2020/01/13 Python
用什么库写 Python 命令行程序(示例代码详解)
2020/02/20 Python
Python yield的用法实例分析
2020/03/06 Python
基于python实现可视化生成二维码工具
2020/07/08 Python
css背景图片的背景裁切、背景透明度、背景变换等效果运用
2012/12/24 HTML / CSS
搬家公司的创业计划书
2014/01/01 职场文书
《我不是最弱小的》教学反思
2014/02/23 职场文书
Go语言中的UTF-8实现
2021/04/26 Golang