머신러닝

TensorFlow GPU Memory Error

브라이언7 2018. 1. 13. 13:50

Traceback (most recent call last):

  File "tr_agent.py", line 190, in <module>

    tf.app.run()

  File "C:\Users\Heeseok\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run

    _sys.exit(main(_sys.argv[:1] + flags_passthrough))

  File "tr_agent.py", line 184, in main

    train()

  File "tr_agent.py", line 145, in train

    saver.save(sess, 'model/dqn.ckpt', global_step=time_step)

  File "C:\Users\Heeseok\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\training\saver.py", line 1601, in save

    self.export_meta_graph(meta_graph_filename)

  File "C:\Users\Heeseok\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\training\saver.py", line 1639, in export_meta_graph

    clear_extraneous_savers=clear_extraneous_savers)

  File "C:\Users\Heeseok\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\training\saver.py", line 1883, in export_meta_graph

    **kwargs)

  File "C:\Users\Heeseok\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\meta_graph.py", line 836, in export_scoped_meta_graph

    **kwargs)

  File "C:\Users\Heeseok\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\framework\meta_graph.py", line 505, in create_meta_graph_def

    meta_graph_def.graph_def.MergeFrom(graph_def)

  File "C:\Users\Heeseok\AppData\Local\Programs\Python\Python36\lib\site-packages\google\protobuf\internal\python_message.py", line 1244, in MergeFrom

    field_value.MergeFrom(value)

  File "C:\Users\Heeseok\AppData\Local\Programs\Python\Python36\lib\site-packages\google\protobuf\internal\containers.py", line 397, in MergeFrom

    self.extend(other._values)

  File "C:\Users\Heeseok\AppData\Local\Programs\Python\Python36\lib\site-packages\google\protobuf\internal\containers.py", line 390, in extend

    values.append(new_element)

MemoryError


D:\python\Trading>python tr_agent.py --train

train..

2018-01-13 09:43:32.662167: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2

2018-01-13 09:43:33.716437: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Found device 0 with properties:

name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.695

pciBusID: 0000:01:00.0

totalMemory: 8.00GiB freeMemory: 6.65GiB

2018-01-13 09:43:33.717723: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)

2018-01-13 09:43:34.820224: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 6.32G (6782146560 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY

2018-01-13 09:43:35.167710: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_driver.cc:936] failed to allocate 5.68G (6103931904 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY


D:\python\Trading>python tr_agent.py --train
train..
2018-01-13 09:55:36.666583: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2018-01-13 09:55:37.703689: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1030] Found device 0 with properties:
name: GeForce GTX 1070 major: 6 minor: 1 memoryClockRate(GHz): 1.695
pciBusID: 0000:01:00.0
totalMemory: 8.00GiB freeMemory: 6.65GiB
2018-01-13 09:55:37.703872: I C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1070, pci bus id: 0000:01:00.0, compute capability: 6.1)
global_episode: 1099
global_time_step: 15487
global_epsilon: 0.010000
2018-01-13 09:55:46.151217: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2018-01-13 09:55:46.151569: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:352] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM
2018-01-13 09:55:46.152184: F C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\36\tensorflow\core\kernels\conv_ops.cc:667] Check failed: stream->parent()->GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo<T>(), &algorithms)



저작자표시