解决TensorFlow程序无限制占用GPU的方法


Posted in Python onJune 30, 2020

今天遇到一个奇怪的现象,使用tensorflow-gpu的时候,出现内存超额~~如果我训练什么大型数据也就算了,关键我就写了一个y=W*x…显示如下图所示:

程序如下:

import tensorflow as tf

w = tf.Variable([[1.0,2.0]])
b = tf.Variable([[2.],[3.]])

y = tf.multiply(w,b)

init_op = tf.global_variables_initializer()

with tf.Session() as sess:
 sess.run(init_op)
 print(sess.run(y))

出错提示:

占用的内存越来越多,程序崩溃之后,整个电脑都奔溃了,因为整个显卡全被吃了

2018-06-10 18:28:00.263424: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2018-06-10 18:28:00.598075: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 0 with properties: 
name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:01:00.0
totalMemory: 6.00GiB freeMemory: 4.97GiB
2018-06-10 18:28:00.598453: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-10 18:28:01.265600: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-10 18:28:01.265826: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:929]  0 
2018-06-10 18:28:01.265971: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 0: N 
2018-06-10 18:28:01.266220: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4740 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-06-10 18:28:01.331056: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 4.63G (4970853120 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:01.399111: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 4.17G (4473767936 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:01.468293: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 3.75G (4026391040 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:01.533138: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 3.37G (3623751936 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:01.602452: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 3.04G (3261376768 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:01.670225: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 2.73G (2935238912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:01.733120: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 2.46G (2641714944 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:01.800101: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 2.21G (2377543424 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:01.862064: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 1.99G (2139789056 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:01.925434: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 1.79G (1925810176 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:01.986180: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 1.61G (1733229056 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:02.043456: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 1.45G (1559906048 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:02.103531: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 1.31G (1403915520 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:02.168973: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 1.18G (1263524096 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:02.229387: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 1.06G (1137171712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:02.292997: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 976.04M (1023454720 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:02.356714: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 878.44M (921109248 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:02.418167: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 790.59M (828998400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:02.482394: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 711.54M (746098688 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY

分析原因:

显卡驱动不是最新版本,用__驱动软件__更新一下驱动,或者自己去下载更新。

TF运行太多,注销全部程序冲洗打开。

由于TF内核编写的原因,默认占用全部的GPU去训练自己的东西,也就是像meiguo一样优先政策吧

这个时候我们得设置两个方面:

  • 选择什么样的占用方式?优先占用__还是__按需占用
  • 选择最大占用多少GPU,因为占用过大GPU会导致其它程序奔溃。最好在0.7以下

先更新驱动:

解决TensorFlow程序无限制占用GPU的方法

再设置TF程序:

注意:单独设置一个不行!按照网上大神博客试了,结果效果还是很差(占用很多GPU)

设置TF:

  • 按需占用
  • 最大占用70%GPU

修改代码如下:

import tensorflow as tf

w = tf.Variable([[1.0,2.0]])
b = tf.Variable([[2.],[3.]])

y = tf.multiply(w,b)

init_op = tf.global_variables_initializer()

config = tf.ConfigProto(allow_soft_placement=True)
gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.7)
config.gpu_options.allow_growth = True
with tf.Session(config=config) as sess:
 sess.run(init_op)
 print(sess.run(y))

成功解决:

2018-06-10 18:21:17.532630: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2018-06-10 18:21:17.852442: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 0 with properties: 
name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:01:00.0
totalMemory: 6.00GiB freeMemory: 4.97GiB
2018-06-10 18:21:17.852817: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-10 18:21:18.511176: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-10 18:21:18.511397: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:929]  0 
2018-06-10 18:21:18.511544: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 0: N 
2018-06-10 18:21:18.511815: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4740 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
[[2. 4.]
 [3. 6.]]

参考资料:

主要参考博客

错误实例

到此这篇关于解决TensorFlow程序无限制占用GPU的方法 的文章就介绍到这了,更多相关TensorFlow 占用GPU内容请搜索三水点靠木以前的文章或继续浏览下面的相关文章希望大家以后多多支持三水点靠木!

Python 相关文章推荐
python绘图方法实例入门
May 19 Python
python学习之面向对象【入门初级篇】
Jan 21 Python
对python-3-print重定向输出的几种方法总结
May 11 Python
Python实现的绘制三维双螺旋线图形功能示例
Jun 23 Python
Python设置在shell脚本中自动补全功能的方法
Jun 25 Python
Python实现朴素贝叶斯分类器的方法详解
Jul 04 Python
Python中函数的基本定义与调用及内置函数详解
May 13 Python
python的pytest框架之命令行参数详解(上)
Jun 27 Python
使用pandas实现连续数据的离散化处理方式(分箱操作)
Nov 22 Python
Python爬取新型冠状病毒“谣言”新闻进行数据分析
Feb 16 Python
python selenium 获取接口数据的实现
Dec 07 Python
Python 实现二叉查找树的示例代码
Dec 21 Python
tensorflow 大于某个值为1,小于为0的实例
Jun 30 #Python
基于tf.shape(tensor)和tensor.shape()的区别说明
Jun 30 #Python
Tensorflow全局设置可见GPU编号操作
Jun 30 #Python
Python logging模块异步线程写日志实现过程解析
Jun 30 #Python
浅谈多卡服务器下隐藏部分 GPU 和 TensorFlow 的显存使用设置
Jun 30 #Python
Tensorflow中批量读取数据的案列分析及TFRecord文件的打包与读取
Jun 30 #Python
使用Tensorflow-GPU禁用GPU设置(CPU与GPU速度对比)
Jun 30 #Python
You might like
windows xp下安装pear
2006/12/02 PHP
将PHP程序中返回的JSON格式数据用gzip压缩输出的方法
2016/03/03 PHP
Javascript & DHTML 实例编程(教程)(三)初级实例篇1—上传文件控件实例
2007/06/02 Javascript
JS和Jquery获取和修改label的值的示例代码
2014/01/15 Javascript
JavaScript获取flash对象与网上的有所不同
2014/04/21 Javascript
JQuery实现的购物车功能(可以减少或者添加商品并自动计算价格)
2015/01/13 Javascript
JavaScript原生对象之Number对象的属性和方法详解
2015/03/13 Javascript
jQuery超精致图片轮播幻灯片特效代码分享
2015/09/10 Javascript
使用getBoundingClientRect方法实现简洁的sticky组件的方法
2016/03/22 Javascript
jQuery表单事件实例代码分享
2016/08/18 Javascript
Node.js中常规的文件操作总结
2016/10/13 Javascript
JS中append字符串包含onclick无效传递参数失败的解决方案
2016/12/26 Javascript
手机端js和html5刮刮卡效果
2020/09/29 Javascript
Vue2.0实现将页面中表格数据导出excel的实例
2017/08/09 Javascript
JavaScript创建对象的四种常用模式实例分析
2019/01/11 Javascript
使用ESLint禁止项目导入特定模块的方法步骤
2019/03/04 Javascript
Vue实现购物车的全选、单选、显示商品价格代码实例
2019/05/06 Javascript
javascript数组元素删除方法delete和splice解析
2019/12/09 Javascript
vue 组件间的通信之子组件向父组件传值的方式
2020/07/29 Javascript
天翼开放平台免费短信验证码接口使用实例
2013/12/18 Python
Python中的Classes和Metaclasses详解
2015/04/02 Python
Python判断一个三位数是否为水仙花数的示例
2018/11/13 Python
Python面向对象之类的封装操作示例
2019/06/08 Python
django框架基于模板 生成 excel(xls) 文件操作示例
2019/06/19 Python
Python 获取ftp服务器文件时间的方法
2019/07/02 Python
Python3从零开始搭建一个语音对话机器人的实现
2019/08/23 Python
python 利用turtle库绘制笑脸和哭脸的例子
2019/11/23 Python
django xadmin action兼容自定义model权限教程
2020/03/30 Python
机电专业体育教师求职信
2013/09/21 职场文书
会计与审计毕业生自荐信范文
2013/12/30 职场文书
网上开商店的创业计划书
2014/01/19 职场文书
《燕子专列》教学反思
2014/02/21 职场文书
业务内勤岗位职责
2014/04/30 职场文书
毕业生个人自荐书
2015/03/05 职场文书
2015年组织部工作总结
2015/04/03 职场文书
css3带你实现3D转换效果
2022/02/24 HTML / CSS