8g内存用python读取10文件_面试题-python 如何读取一个大于 10G 的txt文件?


Posted in Python onMay 28, 2021

前言

用python 读取一个大于10G 的文件,自己电脑只有8G内存,一运行就报内存溢出:MemoryError

python 如何用open函数读取大文件呢?

读取大文件

首先可以自己先制作一个大于10G的txt文件

a = '''
2021-02-02 21:33:31,678 [django.request:93] [base:get_response] [WARNING]- Not Found: /http:/123.125.114.144/
2021-02-02 21:33:31,679 [django.server:124] [basehttp:log_message] [WARNING]- "HEAD http://123.125.114.144/ HTTP/1.1" 404 1678
2021-02-02 22:14:04,121 [django.server:124] [basehttp:log_message] [INFO]- code 400, message Bad request version ('HTTP')
2021-02-02 22:14:04,122 [django.server:124] [basehttp:log_message] [WARNING]- "GET ../../mnt/custom/ProductDefinition HTTP" 400 -
2021-02-02 22:16:21,052 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/login HTTP/1.1" 301 0
2021-02-02 22:16:21,123 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/login/ HTTP/1.1" 200 3876
2021-02-02 22:16:21,192 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/img/main_bg.png HTTP/1.1" 200 2801
2021-02-02 22:16:21,196 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/iconfont/style.css HTTP/1.1" 200 1638
2021-02-02 22:16:21,229 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/img/bg.jpg HTTP/1.1" 200 135990
2021-02-02 22:16:21,307 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/iconfont/fonts/icomoon.ttf?u4m6fy HTTP/1.1" 200 6900
2021-02-02 22:16:23,525 [django.server:124] [basehttp:log_message] [INFO]- "POST /api/login/ HTTP/1.1" 302 0
2021-02-02 22:16:23,618 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/index/ HTTP/1.1" 200 18447
2021-02-02 22:16:23,709 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/js/commons.js HTTP/1.1" 200 13209
2021-02-02 22:16:23,712 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/css/admin.css HTTP/1.1" 200 19660
2021-02-02 22:16:23,712 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/css/common.css HTTP/1.1" 200 1004
2021-02-02 22:16:23,714 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/js/app.js HTTP/1.1" 200 20844
2021-02-02 22:16:26,509 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/report_list/1/ HTTP/1.1" 200 14649
2021-02-02 22:16:51,496 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/test_list/1/ HTTP/1.1" 200 24874
2021-02-02 22:16:51,721 [django.server:124] [basehttp:log_message] [INFO]- "POST /api/add_case/ HTTP/1.1" 200 0
2021-02-02 22:16:59,707 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/test_list/1/ HTTP/1.1" 200 24874
2021-02-03 22:16:59,909 [django.server:124] [basehttp:log_message] [INFO]- "POST /api/add_case/ HTTP/1.1" 200 0
2021-02-03 22:17:01,306 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/edit_case/1/ HTTP/1.1" 200 36504
2021-02-03 22:17:06,265 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/add_project/ HTTP/1.1" 200 17737
2021-02-03 22:17:07,825 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/project_list/1/ HTTP/1.1" 200 29789
2021-02-03 22:17:13,116 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/add_config/ HTTP/1.1" 200 24816
2021-02-03 22:17:19,671 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/config_list/1/ HTTP/1.1" 200 19532
'''

while True:
with open("xxx.log", "a", encoding="utf-8") as fp:
fp.write(a)

循环写入到 xxx.log 文件,运行 3-5 分钟,pycharm 打开查看文件大小大于 10G

8g内存用python读取10文件_面试题-python 如何读取一个大于 10G 的txt文件?

于是我用open函数 直接读取

f = open("xxx.log", 'r')
print(f.read())
f.close()

抛出内存溢出异常:MemoryError

Traceback (most recent call last):
File "D:/2021kecheng06/demo/txt.py", line 35, in
print(f.read())
MemoryError

运行的时候可以看下自己电脑的内存已经占了100%, cpu高达91% ,不挂掉才怪了!

8g内存用python读取10文件_面试题-python 如何读取一个大于 10G 的txt文件?

这种错误的原因在于,read()方法执行操作是一次性的都读入内存中,显然文件大于内存就会报错。

read() 的几种方法

1.read() 方法可以带参数 n, n 是每次读取的大小长度,也就是可以每次读一部分,这样就不会导致内存溢出

f = open("xxx.log", 'r')
print(f.read(2048))
f.close()

运行结果

2019-10-24 21:33:31,678 [django.request:93] [base:get_response] [WARNING]- Not Found: /http:/123.125.114.144/
2019-10-24 21:33:31,679 [django.server:124] [basehttp:log_message] [WARNING]- "HEAD http://123.125.114.144/ HTTP/1.1" 404 1678
2019-10-24 22:14:04,121 [django.server:124] [basehttp:log_message] [INFO]- code 400, message Bad request version ('HTTP')
2019-10-24 22:14:04,122 [django.server:124] [basehttp:log_message] [WARNING]- "GET ../../mnt/custom/ProductDefinition HTTP" 400 -
2019-10-24 22:16:21,052 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/login HTTP/1.1" 301 0
2019-10-24 22:16:21,123 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/login/ HTTP/1.1" 200 3876
2019-10-24 22:16:21,192 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/img/main_bg.png HTTP/1.1" 200 2801
2019-10-24 22:16:21,196 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/iconfont/style.css HTTP/1.1" 200 1638
2019-10-24 22:16:21,229 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/img/bg.jpg HTTP/1.1" 200 135990
2019-10-24 22:16:21,307 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/iconfont/fonts/icomoon.ttf?u4m6fy HTTP/1.1" 200 6900
2019-10-24 22:16:23,525 [django.server:124] [basehttp:log_message] [INFO]- "POST /api/login/ HTTP/1.1" 302 0
2019-10-24 22:16:23,618 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/index/ HTTP/1.1" 200 18447
2019-10-24 22:16:23,709 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/js/commons.js HTTP/1.1" 200 13209
2019-10-24 22:16:23,712 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/css/admin.css HTTP/1.1" 200 19660
2019-10-24 22:16:23,712 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/css/common.css HTTP/1.1" 200 1004
2019-10-24 22:16:23,714 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/js/app.js HTTP/1.1" 200 20844

这样就只读取了2048个字符,全部读取的话,循环读就行

f = open("xxx.log", 'r')
while True:
block = f.read(2048)
print(block)
if not block:
break
f.close()
2.readline():每次读取一行,这个方法也不会报错
f = open("xxx.log", 'r')
while True:
line = f.readline()
print(line, end="")
if not line:
break
f.close()

3.readlines():读取全部的行,生成一个list,通过list来对文件进行处理,显然这种方式依然会造成:MemoyError

真正 Pythonic 的方法

真正 Pythonci 的方法,使用 with 结构打开文件,fp 是一个可迭代对象,可以用 for 遍历读取每行的文件内容

with open("xxx.log", 'r') as fp:

for line in fp:

print(line, end="")

yield 生成器读取大文件

前面一篇讲yield 生成器的时候提到读取大文件,函数返回一个可迭代对象,用next()方法读取文件内容

def read_file(fpath):

BLOCK_SIZE = 1024

with open(fpath, 'rb') as f:

while True:

block = f.read(BLOCK_SIZE)

if block:

yield block

else:

return

if __name__ == '__main__':

a = read_file("xxx.log")

print(a) # generator objec

print(next(a)) # bytes类型

print(next(a).decode("utf-8")) # str

运行结果

b'\r\n2019-10-24 21:33:31,678 [django.request:93] [base:get_response] [WARNING]- Not Found: /http:/123.125.114.144/\r\n2019-10-24 21:33:31,679 [django.server:124] [basehttp:log_message] [WARNING]- "HEAD http://123.125.114.144/ HTTP/1.1" 404 1678\r\n2019-10-24 22:14:04,121 [django.server:124] [basehttp:log_message] [INFO]- code 400, message Bad request version (\'HTTP\')\r\n2019-10-24 22:14:04,122 [django.server:124] [basehttp:log_message] [WARNING]- "GET ../../mnt/custom/ProductDefinition HTTP" 400 -\r\n2019-10-24 22:16:21,052 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/login HTTP/1.1" 301 0\r\n2019-10-24 22:16:21,123 [django.server:124] [basehttp:log_message] [INFO]- "GET /api/login/ HTTP/1.1" 200 3876\r\n2019-10-24 22:16:21,192 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/img/main_bg.png HTTP/1.1" 200 2801\r\n2019-10-24 22:16:21,196 [django.server:124] [basehttp:log_message] [INFO]- "GET /static/assets/iconfont/style.css HTTP/1.1" 200 1638\r\n2019-10-24 22:16:21,229 [django.server:124] '

Python 相关文章推荐
Python中使用 Selenium 实现网页截图实例
Jul 18 Python
python中MethodType方法介绍与使用示例
Aug 03 Python
python list元素为tuple时的排序方法
Apr 18 Python
python版本单链表实现代码
Sep 28 Python
Python数据可视化教程之Matplotlib实现各种图表实例
Jan 13 Python
Python使用线程来接收串口数据的示例
Jul 02 Python
python应用文件读取与登录注册功能
Sep 23 Python
解决django后台管理界面添加中文内容乱码问题
Nov 15 Python
从pandas一个单元格的字符串中提取字符串方式
Dec 17 Python
python代码实现TSNE降维数据可视化教程
Feb 28 Python
python名片管理系统开发
Jun 18 Python
python办公自动化之excel的操作
May 23 Python
用python画城市轮播地图
用Python实现一个打字速度测试工具来测试你的手速
解决Pytorch dataloader时报错每个tensor维度不一样的问题
May 28 #Python
pytorch锁死在dataloader(训练时卡死)
Python趣味爬虫之用Python实现智慧校园一键评教
Pytorch 如何加速Dataloader提升数据读取速度
在前女友婚礼上,用Python破解了现场的WIFI还把名称改成了
You might like
php模拟socket一次连接,多次发送数据的实现代码
2011/07/26 PHP
PHP简单实现欧拉函数Euler功能示例
2017/11/06 PHP
用PHP做了一个领取优惠券活动的示例代码
2019/07/05 PHP
jQuery实现带动画效果的二级下拉导航方法
2015/03/11 Javascript
最棒的Angular2表格控件
2016/08/10 Javascript
bootstrap table小案例
2016/10/21 Javascript
微信小程序商城项目之商品属性分类(4)
2017/04/17 Javascript
Vue动态实现评分效果
2017/05/24 Javascript
JS实现弹出下载对话框及常见文件类型的下载
2017/07/13 Javascript
JavaScript实现正则去除a标签并保留内容的方法【测试可用】
2018/07/18 Javascript
vue elementUI使用tabs与导航栏联动
2019/06/21 Javascript
浅析Angular 实现一个repeat指令的方法
2019/07/21 Javascript
浅谈Webpack4 Tree Shaking 终极优化指南
2019/11/18 Javascript
详解javascript中var与ES6规范中let、const区别与用法
2020/01/11 Javascript
[01:06]欢迎来到上海,TI9
2018/08/26 DOTA
Python函数式编程指南(一):函数式编程概述
2015/06/24 Python
详解Python的Flask框架中的signals信号机制
2016/06/13 Python
pandas数值计算与排序方法
2018/04/12 Python
Python实现随机取一个矩阵数组的某几行
2019/11/26 Python
Python grequests模块使用场景及代码实例
2020/08/10 Python
Python实现哲学家就餐问题实例代码
2020/11/09 Python
用HTML5制作数字时钟的教程
2015/05/11 HTML / CSS
固特异美国在线轮胎店:Goodyear Tire
2019/02/23 全球购物
测绘工程个人的自我评价
2013/11/10 职场文书
营业员演讲稿
2013/12/30 职场文书
安全生产月演讲稿
2014/05/09 职场文书
机械专业毕业生自我鉴定2014
2014/10/04 职场文书
2014年助理政工师工作总结
2014/12/19 职场文书
装修公司工程部经理岗位职责
2015/04/09 职场文书
2015年卫生监督工作总结
2015/05/21 职场文书
同学聚会感言一句话
2015/07/30 职场文书
环保主题班会教案
2015/08/13 职场文书
《猴王出世》教学反思
2016/02/23 职场文书
MySQL数字类型自增的坑
2021/05/07 MySQL
HTML中的表格元素介绍
2022/02/28 HTML / CSS
Go语言grpc和protobuf
2022/04/13 Golang