解决TensorFlow程序无限制占用GPU的方法


Posted in Python onJune 30, 2020

今天遇到一个奇怪的现象,使用tensorflow-gpu的时候,出现内存超额~~如果我训练什么大型数据也就算了,关键我就写了一个y=W*x…显示如下图所示:

程序如下:

import tensorflow as tf

w = tf.Variable([[1.0,2.0]])
b = tf.Variable([[2.],[3.]])

y = tf.multiply(w,b)

init_op = tf.global_variables_initializer()

with tf.Session() as sess:
 sess.run(init_op)
 print(sess.run(y))

出错提示:

占用的内存越来越多,程序崩溃之后,整个电脑都奔溃了,因为整个显卡全被吃了

2018-06-10 18:28:00.263424: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2018-06-10 18:28:00.598075: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 0 with properties: 
name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:01:00.0
totalMemory: 6.00GiB freeMemory: 4.97GiB
2018-06-10 18:28:00.598453: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-10 18:28:01.265600: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-10 18:28:01.265826: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:929]  0 
2018-06-10 18:28:01.265971: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 0: N 
2018-06-10 18:28:01.266220: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4740 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-06-10 18:28:01.331056: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 4.63G (4970853120 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:01.399111: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 4.17G (4473767936 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:01.468293: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 3.75G (4026391040 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:01.533138: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 3.37G (3623751936 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:01.602452: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 3.04G (3261376768 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:01.670225: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 2.73G (2935238912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:01.733120: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 2.46G (2641714944 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:01.800101: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 2.21G (2377543424 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:01.862064: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 1.99G (2139789056 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:01.925434: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 1.79G (1925810176 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:01.986180: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 1.61G (1733229056 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:02.043456: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 1.45G (1559906048 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:02.103531: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 1.31G (1403915520 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:02.168973: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 1.18G (1263524096 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:02.229387: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 1.06G (1137171712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:02.292997: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 976.04M (1023454720 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:02.356714: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 878.44M (921109248 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:02.418167: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 790.59M (828998400 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
2018-06-10 18:28:02.482394: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 711.54M (746098688 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY

分析原因:

显卡驱动不是最新版本,用__驱动软件__更新一下驱动,或者自己去下载更新。

TF运行太多,注销全部程序冲洗打开。

由于TF内核编写的原因,默认占用全部的GPU去训练自己的东西,也就是像meiguo一样优先政策吧

这个时候我们得设置两个方面:

  • 选择什么样的占用方式?优先占用__还是__按需占用
  • 选择最大占用多少GPU,因为占用过大GPU会导致其它程序奔溃。最好在0.7以下

先更新驱动:

解决TensorFlow程序无限制占用GPU的方法

再设置TF程序:

注意:单独设置一个不行!按照网上大神博客试了,结果效果还是很差(占用很多GPU)

设置TF:

  • 按需占用
  • 最大占用70%GPU

修改代码如下:

import tensorflow as tf

w = tf.Variable([[1.0,2.0]])
b = tf.Variable([[2.],[3.]])

y = tf.multiply(w,b)

init_op = tf.global_variables_initializer()

config = tf.ConfigProto(allow_soft_placement=True)
gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.7)
config.gpu_options.allow_growth = True
with tf.Session(config=config) as sess:
 sess.run(init_op)
 print(sess.run(y))

成功解决:

2018-06-10 18:21:17.532630: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2018-06-10 18:21:17.852442: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 0 with properties: 
name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:01:00.0
totalMemory: 6.00GiB freeMemory: 4.97GiB
2018-06-10 18:21:17.852817: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-10 18:21:18.511176: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-10 18:21:18.511397: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:929]  0 
2018-06-10 18:21:18.511544: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 0: N 
2018-06-10 18:21:18.511815: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4740 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
[[2. 4.]
 [3. 6.]]

参考资料:

主要参考博客

错误实例

到此这篇关于解决TensorFlow程序无限制占用GPU的方法 的文章就介绍到这了,更多相关TensorFlow 占用GPU内容请搜索三水点靠木以前的文章或继续浏览下面的相关文章希望大家以后多多支持三水点靠木!

Python 相关文章推荐
Python实现短网址ShortUrl的Hash运算实例讲解
Aug 10 Python
Python编码爬坑指南(必看)
Jun 10 Python
python使用response.read()接收json数据的实例
Dec 19 Python
Python将文字转成语音并读出来的实例详解
Jul 15 Python
pyinstaller打包opencv和numpy程序运行错误解决
Aug 16 Python
利用Python实现kNN算法的代码
Aug 16 Python
python bluetooth蓝牙信息获取蓝牙设备类型的方法
Nov 29 Python
python字典setdefault方法和get方法使用实例
Dec 25 Python
Python使用requests xpath 并开启多线程爬取西刺代理ip实例
Mar 06 Python
Python3.7 读取音频根据文件名生成脚本的代码
Apr 07 Python
解决python中0x80072ee2错误的方法
Jul 19 Python
python图像处理基本操作总结(PIL库、Matplotlib及Numpy)
Jun 08 Python
tensorflow 大于某个值为1,小于为0的实例
Jun 30 #Python
基于tf.shape(tensor)和tensor.shape()的区别说明
Jun 30 #Python
Tensorflow全局设置可见GPU编号操作
Jun 30 #Python
Python logging模块异步线程写日志实现过程解析
Jun 30 #Python
浅谈多卡服务器下隐藏部分 GPU 和 TensorFlow 的显存使用设置
Jun 30 #Python
Tensorflow中批量读取数据的案列分析及TFRecord文件的打包与读取
Jun 30 #Python
使用Tensorflow-GPU禁用GPU设置(CPU与GPU速度对比)
Jun 30 #Python
You might like
基于ThinkPHP5.0实现图片上传插件
2017/09/25 PHP
JavaScript isPrototypeOf和hasOwnProperty使用区别
2010/03/04 Javascript
javascript 实现键盘上下左右功能的小例子
2013/09/15 Javascript
jquery获得option的值和对option进行操作
2013/12/13 Javascript
JS+DIV实现鼠标划过切换层效果的方法
2015/05/25 Javascript
jquery+html5烂漫爱心表白动画代码分享
2015/08/24 Javascript
JS中生成随机数的用法及相关函数
2016/01/09 Javascript
JS中改变this指向的方法(call和apply、bind)
2016/03/26 Javascript
关于Jquery中的bind(),on()绑定事件方式总结
2016/10/26 Javascript
详解webpack异步加载业务模块
2017/06/23 Javascript
vue.js组件之间传递数据的方法
2017/07/10 Javascript
vue 设置路由的登录权限的方法
2018/07/03 Javascript
微信小程序与后台PHP交互的方法实例分析
2018/12/10 Javascript
angular6根据environments配置文件更改开发所需要的环境的方法
2019/03/06 Javascript
js实现坦克移动小游戏
2019/10/28 Javascript
highcharts.js数据绑定方式代码实例
2019/11/13 Javascript
webpack+vue-cil 中proxyTable配置接口地址代理操作
2020/07/18 Javascript
微信小程序实现日历签到
2020/09/21 Javascript
[02:12]2015国际邀请赛 SHOWOPEN
2015/08/05 DOTA
[47:20]DAC2018 4.4 淘汰赛 Optic vs Mineski 第一场
2018/04/05 DOTA
详解python之配置日志的几种方式
2017/05/22 Python
Python_LDA实现方法详解
2017/10/25 Python
pandas计算最大连续间隔的方法
2019/07/04 Python
python实现全排列代码(回溯、深度优先搜索)
2020/02/26 Python
HTML5 textarea高度自适应的两种方案
2020/04/08 HTML / CSS
html5实现输入框fixed定位在屏幕最底部兼容性
2020/07/03 HTML / CSS
阿根廷票务网站:StubHub阿根廷
2018/04/13 全球购物
Hobbs官方网站:英国奢华女性时尚服装
2020/02/22 全球购物
上班睡觉检讨书
2014/01/09 职场文书
远程研修随笔感言
2014/02/10 职场文书
优秀团员事迹材料2000字
2014/08/20 职场文书
暑期培训班策划方案
2014/08/26 职场文书
如何利用map实现Nginx允许多个域名跨域
2021/03/31 Servers
详解Vue router路由
2021/11/20 Vue.js
MySQL创建表操作命令分享
2022/03/25 MySQL
PC版《死亡搁浅导剪版》现已发售 展开全新的探险
2022/04/03 其他游戏