TensorFlow GPU Memory Error

Traceback (most recent call last):

  File "tr_agent.py", line 190, in <module>

    tf.app.run()

  File "C:\Users\Heeseok\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run

    _sys.exit(main(_sys.argv[:1] + flags_passthrough))

  File "tr_agent.py", line 184, in main

    train()

  File "tr_agent.py", line 145, in train

    saver.save(sess, 'model/dqn.ckpt', global_step=time_step)

  File "C:\Users\Heeseok\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\training\saver.py", line 1601, in save

    self.export_meta_graph(meta_graph_filename)

  File "C:\Users\Heeseok\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\training\saver.py", line 1639, in export_meta_graph

    clear_extraneous_savers=clear_extraneous_savers)

  File "C:\Users\Heeseok\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\training\saver.py", line 1883, in export_meta_graph

    **kwargs)

  File "C:\Users\Heeseok\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\meta_graph.py", line 836, in export_scoped_meta_graph

    **kwargs)

  File "C:\Users\Heeseok\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\meta_graph.py", line 505, in create_meta_graph_def

    meta_graph_def.graph_def.MergeFrom(graph_def)

  File "C:\Users\Heeseok\AppData\Local\Programs\Python\Python36\lib\site-packages\google\protobuf\internal\python_message.py", line 1244, in MergeFrom

    field_value.MergeFrom(value)

  File "C:\Users\Heeseok\AppData\Local\Programs\Python\Python36\lib\site-packages\google\protobuf\internal\containers.py", line 397, in MergeFrom

    self.extend(other._values)

  File "C:\Users\Heeseok\AppData\Local\Programs\Python\Python36\lib\site-packages\google\protobuf\internal\containers.py", line 390, in extend

    values.append(new_element)

MemoryError

D:\python\Trading>python tr_agent.py --train

train..

2018-01-13 09:43:32.662167: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2

2018-01-13 09:43:33.716437: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Found device 0 with properties:

name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.695

pciBusID: 0000:01:00.0

totalMemory: 8.00GiB freeMemory: 6.65GiB

2018-01-13 09:43:33.717723: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)

2018-01-13 09:43:34.820224: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 6.32G (6782146560 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY

2018-01-13 09:43:35.167710: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 5.68G (6103931904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY

D:\python\Trading>python tr_agent.py --train

train..

2018-01-13 09:55:36.666583: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2

2018-01-13 09:55:37.703689: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Found device 0 with properties:

name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.695

pciBusID: 0000:01:00.0

totalMemory: 8.00GiB freeMemory: 6.65GiB

2018-01-13 09:55:37.703872: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)

global_episode: 1099

global_time_step: 15487

global_epsilon: 0.010000

2018-01-13 09:55:46.151217: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR

2018-01-13 09:55:46.151569: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:352] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM

2018-01-13 09:55:46.152184: F C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\kernels\conv_ops.cc:667] Check failed: stream->parent()->GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo<T>(), &algorithms)

< 메모리 에러 해결책 >

https://stackoverflow.com/questions/34199233/how-to-prevent-tensorflow-from-allocating-the-totality-of-a-gpu-memory

config = tf.ConfigProto()
config.gpu_options.allow_growth=True
sess = tf.Session(config=config)

https://github.com/tensorflow/tensorflow/issues/1578

https://stackoverflow.com/questions/39465503/cuda-error-out-of-memory-in-tensorflow

https://github.com/tensorflow/tensorflow/issues/6698