text:
Keras to IR: ResourceExhaustedError: OOM when allocating tensor with shape[4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

I want convert this model into pytorch, I downloaded the converted  weight file Keras_FCN8s_face_seg_YuvalNirkin.h5. Then I add
jsonstr=model.to_json()
with open("Keras_FCN8s_Model.json","w+") as fp:
    fp.write(jsonstr)

at the end of FCN8s_keras_demo.ipynb, it works well and I got a keras json model. Then I try to convert keras into IR, got an error as title. I don't know how to do. Here's my environment:
Platform: Red Hat 4.8.5-11 with 4 Nvidia Tesla P40 GPUs
Python version: Python 3.6.5: Anaconda, Inc.
MMdnn: installed from  master (0bf2f4d) (pip install -U git+https://github.com/Microsoft/MMdnn.git@master), version 0.2.3
Source framework with version : Keras 2.2.4 GPU from pip, with backend tensorflow-gpu 1.12.0
Destination framework with version : IR
Pre-trained model path : local disk
Running scripts: mmtoir -f keras -d fcn8s_ir -w Keras_FCN8s_face_seg_YuvalNirkin.h5 -n Keras_FCN8s_Model.json
Output Messages:
[liuqixuan_i@ml-gpu-ser544 face-segmentation-keras]$ mmtoir -f keras -d fcn8s_ir -w Keras_FCN8s_face_seg_YuvalNirkin.h5 -n Keras_FCN8s_Model.json
/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
  from ._conv import register_converters as _register_converters
Using TensorFlow backend.
2019-01-29 16:21:48.091209: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-01-29 16:21:49.432212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties:
name: Tesla P40 major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:02:00.0
totalMemory: 22.38GiB freeMemory: 839.12MiB
2019-01-29 16:21:49.625955: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 1 with properties:
name: Tesla P40 major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:03:00.0
totalMemory: 22.38GiB freeMemory: 21.98GiB
2019-01-29 16:21:49.822598: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 2 with properties:
name: Tesla P40 major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:83:00.0
totalMemory: 22.38GiB freeMemory: 21.98GiB
2019-01-29 16:21:50.031561: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 3 with properties:
name: Tesla P40 major: 6 minor: 1 memoryClockRate(GHz): 1.531
pciBusID: 0000:84:00.0
totalMemory: 22.38GiB freeMemory: 21.98GiB
2019-01-29 16:21:50.031966: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0, 1, 2, 3
2019-01-29 16:21:51.567269: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-01-29 16:21:51.567336: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 1 2 3
2019-01-29 16:21:51.567352: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N Y N N
2019-01-29 16:21:51.567364: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 1:   Y N N N
2019-01-29 16:21:51.567376: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 2:   N N N Y
2019-01-29 16:21:51.567386: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 3:   N N Y N
2019-01-29 16:21:51.569215: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 550 MB memory) -> physical GPU (device: 0, name: Tesla P40, pci bus id: 0000:02:00.0, compute capability: 6.1)
2019-01-29 16:21:51.570124: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 21319 MB memory) -> physical GPU (device: 1, name: Tesla P40, pci bus id: 0000:03:00.0, compute capability: 6.1)
2019-01-29 16:21:51.570908: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 21319 MB memory) -> physical GPU (device: 2, name: Tesla P40, pci bus id: 0000:83:00.0, compute capability: 6.1)
2019-01-29 16:21:51.571207: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 21319 MB memory) -> physical GPU (device: 3, name: Tesla P40, pci bus id: 0000:84:00.0, compute capability: 6.1)
2019-01-29 16:22:01.794290: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 16.0KiB.  Current allocation summary follows.
2019-01-29 16:22:01.794355: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:22:01.794370: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:22:01.794381: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:22:01.794393: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:22:01.794404: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:22:01.794414: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:01.794425: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:22:01.794435: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:01.794444: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:01.794455: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:22:01.794467: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:22:01.794478: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:22:01.794488: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:22:01.794499: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:01.794518: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:01.794529: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:22:01.794539: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:01.794549: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:01.794558: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:01.794568: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:01.794580: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:22:01.794590: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 16.0KiB was 16.0KiB, Chunk State:
2019-01-29 16:22:01.794607: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:22:01.794614: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:22:01.794622: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:22:01.794628: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:22:01.794635: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:22:01.794641: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:22:01.794647: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:22:01.794654: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:22:01.794661: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:22:01.794668: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:22:01.794674: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:22:01.794680: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:22:01.794687: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:22:01.794693: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:22:01.794700: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:22:01.794706: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:22:01.794732: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:22:01.794740: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:22:01.794746: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:22:01.794753: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:22:01.794759: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:22:01.794765: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:22:01.794771: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:22:01.794778: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:22:01.794784: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:22:01.794790: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:22:01.794796: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:22:01.794802: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:22:01.794809: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:22:01.794815: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:22:01.794821: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:22:01.794827: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:22:01.794833: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:22:01.794839: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:22:01.794845: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:22:01.794852: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:22:01.794858: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:22:01.794864: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:22:01.794871: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:22:01.794877: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:22:01.794884: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:22:01.794890: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:22:01.794897: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:22:01.794903: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:22:01.794910: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:22:01.794916: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:22:01.794922: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:22:01.794929: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:22:01.794935: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:22:01.794941: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:22:01.794948: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:22:01.794954: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:22:01.794961: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:22:01.794967: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:22:01.795052: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:22:01.795060: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:22:01.795067: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:22:01.795073: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:22:01.795080: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:22:01.795086: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:22:01.795092: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:22:01.795099: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:22:01.795105: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:22:01.795113: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:22:01.795121: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:22:01.795131: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:22:01.795139: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:22:01.795147: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:22:01.795154: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:22:01.795162: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:22:01.795169: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:22:01.795177: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:22:01.795184: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:22:01.795192: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:22:01.795200: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:22:01.795207: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:22:01.795214: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:22:01.795222: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:22:01.795229: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:22:01.795237: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:22:01.795245: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:22:01.795256: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:22:01.795273: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:22:01.795316: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at assign_op.h:117 : Resource exhausted: OOM when allocating tensor with shape[4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:22:11.795511: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 16.0KiB.  Current allocation summary follows.
2019-01-29 16:22:11.795574: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:22:11.795589: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:22:11.795599: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:22:11.795610: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:22:11.795621: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:22:11.795631: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:11.795642: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:22:11.795652: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:11.795661: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:11.795672: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:22:11.795684: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:22:11.795695: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:22:11.795705: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:22:11.795715: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:11.795725: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:11.795736: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:22:11.795746: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:11.795755: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:11.795764: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:11.795773: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:11.795784: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:22:11.795794: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 16.0KiB was 16.0KiB, Chunk State:
2019-01-29 16:22:11.795804: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:22:11.795811: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:22:11.795818: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:22:11.795824: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:22:11.795831: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:22:11.795837: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:22:11.795844: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:22:11.795850: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:22:11.795857: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:22:11.795864: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:22:11.795870: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:22:11.795876: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:22:11.795883: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:22:11.795889: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:22:11.795896: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:22:11.795902: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:22:11.795966: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:22:11.795974: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:22:11.795981: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:22:11.795987: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:22:11.795994: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:22:11.796000: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:22:11.796006: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:22:11.796013: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:22:11.796019: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:22:11.796025: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:22:11.796031: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:22:11.796038: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:22:11.796044: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:22:11.796050: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:22:11.796056: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:22:11.796062: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:22:11.796068: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:22:11.796075: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:22:11.796081: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:22:11.796087: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:22:11.796093: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:22:11.796100: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:22:11.796106: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:22:11.796112: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:22:11.796119: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:22:11.796125: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:22:11.796131: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:22:11.796138: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:22:11.796144: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:22:11.796151: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:22:11.796157: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:22:11.796163: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:22:11.796170: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:22:11.796176: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:22:11.796182: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:22:11.796189: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:22:11.796195: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:22:11.796202: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:22:11.796287: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:22:11.796296: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:22:11.796302: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:22:11.796309: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:22:11.796315: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:22:11.796322: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:22:11.796328: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:22:11.796334: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:22:11.796341: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:22:11.796349: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:22:11.796356: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:22:11.796366: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:22:11.796374: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:22:11.796381: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:22:11.796389: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:22:11.796396: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:22:11.796404: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:22:11.796411: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:22:11.796419: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:22:11.796427: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:22:11.796435: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:22:11.796442: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:22:11.796450: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:22:11.796457: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:22:11.796465: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:22:11.796473: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:22:11.796480: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:22:11.796492: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:22:11.796515: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:22:11.796547: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at assign_op.h:117 : Resource exhausted: OOM when allocating tensor with shape[4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:22:21.796736: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 64.00MiB.  Current allocation summary follows.
2019-01-29 16:22:21.796796: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:22:21.796810: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:22:21.796822: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:22:21.796833: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:22:21.796844: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:22:21.796854: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:21.796866: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:22:21.796875: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:21.796885: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:21.796897: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:22:21.796908: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:22:21.796919: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:22:21.796929: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:22:21.796939: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:21.796949: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:21.796960: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:22:21.796970: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:21.796979: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:21.796988: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:21.796997: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:21.797008: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:22:21.797018: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 64.00MiB was 64.00MiB, Chunk State:
2019-01-29 16:22:21.797028: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:22:21.797035: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:22:21.797042: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:22:21.797048: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:22:21.797055: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:22:21.797061: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:22:21.797068: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:22:21.797074: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:22:21.797081: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:22:21.797087: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:22:21.797094: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:22:21.797100: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:22:21.797106: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:22:21.797113: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:22:21.797120: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:22:21.797126: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:22:21.797150: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:22:21.797157: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:22:21.797164: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:22:21.797170: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:22:21.797176: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:22:21.797182: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:22:21.797188: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:22:21.797195: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:22:21.797201: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:22:21.797207: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:22:21.797213: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:22:21.797219: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:22:21.797225: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:22:21.797232: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:22:21.797238: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:22:21.797244: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:22:21.797250: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:22:21.797256: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:22:21.797262: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:22:21.797269: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:22:21.797275: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:22:21.797281: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:22:21.797287: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:22:21.797293: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:22:21.797300: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:22:21.797306: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:22:21.797313: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:22:21.797319: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:22:21.797325: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:22:21.797332: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:22:21.797338: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:22:21.797344: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:22:21.797350: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:22:21.797357: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:22:21.797363: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:22:21.797369: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:22:21.797376: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:22:21.797382: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:22:21.797469: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:22:21.797478: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:22:21.797484: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:22:21.797490: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:22:21.797497: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:22:21.797509: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:22:21.797515: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:22:21.797522: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:22:21.797529: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:22:21.797537: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:22:21.797543: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:22:21.797553: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:22:21.797561: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:22:21.797569: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:22:21.797576: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:22:21.797584: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:22:21.797592: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:22:21.797599: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:22:21.797607: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:22:21.797615: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:22:21.797622: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:22:21.797630: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:22:21.797637: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:22:21.797644: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:22:21.797652: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:22:21.797660: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:22:21.797668: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:22:21.797679: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:22:21.797696: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:22:21.797728: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at random_op.cc:202 : Resource exhausted: OOM when allocating tensor with shape[1,1,4096,4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:22:31.797915: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 336.0KiB.  Current allocation summary follows.
2019-01-29 16:22:31.797981: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:22:31.797995: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:22:31.798006: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:22:31.798017: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:22:31.798028: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:22:31.798038: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:31.798048: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:22:31.798058: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:31.798067: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:31.798078: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:22:31.798089: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:22:31.798100: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:22:31.798111: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:22:31.798122: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:31.798132: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:31.798143: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:22:31.798152: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:31.798162: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:31.798171: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:31.798180: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:31.798191: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:22:31.798201: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 336.0KiB was 256.0KiB, Chunk State:
2019-01-29 16:22:31.798212: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:22:31.798219: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:22:31.798225: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:22:31.798232: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:22:31.798238: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:22:31.798244: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:22:31.798251: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:22:31.798257: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:22:31.798264: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:22:31.798271: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:22:31.798277: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:22:31.798283: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:22:31.798289: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:22:31.798296: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:22:31.798302: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:22:31.798308: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:22:31.798339: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:22:31.798346: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:22:31.798353: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:22:31.798359: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:22:31.798365: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:22:31.798372: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:22:31.798378: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:22:31.798384: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:22:31.798390: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:22:31.798397: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:22:31.798403: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:22:31.798409: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:22:31.798415: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:22:31.798421: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:22:31.798427: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:22:31.798433: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:22:31.798439: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:22:31.798445: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:22:31.798451: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:22:31.798458: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:22:31.798464: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:22:31.798470: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:22:31.798476: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:22:31.798482: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:22:31.798489: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:22:31.798495: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:22:31.798513: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:22:31.798520: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:22:31.798527: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:22:31.798533: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:22:31.798540: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:22:31.798546: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:22:31.798552: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:22:31.798559: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:22:31.798565: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:22:31.798572: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:22:31.798578: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:22:31.798584: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:22:31.798671: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:22:31.798680: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:22:31.798686: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:22:31.798692: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:22:31.798699: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:22:31.798705: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:22:31.798711: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:22:31.798718: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:22:31.798724: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:22:31.798731: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:22:31.798739: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:22:31.798748: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:22:31.798756: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:22:31.798764: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:22:31.798771: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:22:31.798779: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:22:31.798786: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:22:31.798793: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:22:31.798801: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:22:31.798809: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:22:31.798817: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:22:31.798824: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:22:31.798832: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:22:31.798839: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:22:31.798847: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:22:31.798854: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:22:31.798862: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:22:31.798873: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:22:31.798890: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:22:31.798922: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at random_op.cc:202 : Resource exhausted: OOM when allocating tensor with shape[1,1,4096,21] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:22:41.799103: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 21.0KiB.  Current allocation summary follows.
2019-01-29 16:22:41.799165: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:22:41.799179: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:22:41.799190: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:22:41.799202: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:22:41.799212: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:22:41.799222: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:41.799234: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:22:41.799243: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:41.799253: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:41.799264: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:22:41.799275: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:22:41.799286: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:22:41.799297: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:22:41.799307: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:41.799317: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:41.799328: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:22:41.799338: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:41.799347: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:41.799356: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:41.799365: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:41.799377: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:22:41.799386: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 21.0KiB was 16.0KiB, Chunk State:
2019-01-29 16:22:41.799397: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:22:41.799404: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:22:41.799411: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:22:41.799417: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:22:41.799424: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:22:41.799430: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:22:41.799436: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:22:41.799442: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:22:41.799449: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:22:41.799456: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:22:41.799462: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:22:41.799468: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:22:41.799475: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:22:41.799481: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:22:41.799488: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:22:41.799494: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:22:41.799529: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:22:41.799537: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:22:41.799543: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:22:41.799550: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:22:41.799556: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:22:41.799562: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:22:41.799568: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:22:41.799575: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:22:41.799581: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:22:41.799587: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:22:41.799593: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:22:41.799600: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:22:41.799606: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:22:41.799612: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:22:41.799618: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:22:41.799624: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:22:41.799631: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:22:41.799637: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:22:41.799643: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:22:41.799649: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:22:41.799655: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:22:41.799662: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:22:41.799668: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:22:41.799674: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:22:41.799681: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:22:41.799687: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:22:41.799694: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:22:41.799700: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:22:41.799706: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:22:41.799713: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:22:41.799719: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:22:41.799725: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:22:41.799731: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:22:41.799738: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:22:41.799744: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:22:41.799751: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:22:41.799757: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:22:41.799763: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:22:41.799847: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:22:41.799856: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:22:41.799862: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:22:41.799868: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:22:41.799875: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:22:41.799881: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:22:41.799888: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:22:41.799894: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:22:41.799902: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:22:41.799909: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:22:41.799916: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:22:41.799925: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:22:41.799934: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:22:41.799941: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:22:41.799949: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:22:41.799956: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:22:41.799964: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:22:41.799971: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:22:41.799979: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:22:41.799987: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:22:41.799995: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:22:41.800003: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:22:41.800010: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:22:41.800018: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:22:41.800025: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:22:41.800033: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:22:41.800041: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:22:41.800052: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:22:41.800070: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:22:41.800103: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at random_op.cc:202 : Resource exhausted: OOM when allocating tensor with shape[1,1,256,21] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:22:51.800275: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 84B.  Current allocation summary follows.
2019-01-29 16:22:51.800344: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:22:51.800358: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:22:51.800369: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:22:51.800381: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:22:51.800392: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:22:51.800402: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:51.800413: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:22:51.800423: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:51.800433: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:51.800444: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:22:51.800456: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:22:51.800466: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:22:51.800477: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:22:51.800487: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:51.800497: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:22:51.800518: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:22:51.800529: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:51.800539: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:51.800548: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:51.800558: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:22:51.800569: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:22:51.800578: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 256B was 256B, Chunk State:
2019-01-29 16:22:51.800589: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:22:51.800596: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:22:51.800603: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:22:51.800609: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:22:51.800615: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:22:51.800622: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:22:51.800628: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:22:51.800634: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:22:51.800641: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:22:51.800648: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:22:51.800654: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:22:51.800660: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:22:51.800666: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:22:51.800673: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:22:51.800679: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:22:51.800686: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:22:51.800717: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:22:51.800725: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:22:51.800731: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:22:51.800738: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:22:51.800744: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:22:51.800750: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:22:51.800756: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:22:51.800762: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:22:51.800769: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:22:51.800775: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:22:51.800781: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:22:51.800787: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:22:51.800793: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:22:51.800799: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:22:51.800805: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:22:51.800812: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:22:51.800818: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:22:51.800824: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:22:51.800830: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:22:51.800836: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:22:51.800842: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:22:51.800849: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:22:51.800855: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:22:51.800861: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:22:51.800868: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:22:51.800874: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:22:51.800880: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:22:51.800886: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:22:51.800893: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:22:51.800899: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:22:51.800906: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:22:51.800912: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:22:51.800918: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:22:51.800924: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:22:51.800931: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:22:51.800937: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:22:51.800943: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:22:51.800950: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:22:51.801032: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:22:51.801040: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:22:51.801046: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:22:51.801052: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:22:51.801059: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:22:51.801065: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:22:51.801072: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:22:51.801078: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:22:51.801084: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:22:51.801092: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:22:51.801099: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:22:51.801109: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:22:51.801117: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:22:51.801124: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:22:51.801131: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:22:51.801139: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:22:51.801146: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:22:51.801154: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:22:51.801161: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:22:51.801169: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:22:51.801176: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:22:51.801184: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:22:51.801191: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:22:51.801198: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:22:51.801206: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:22:51.801214: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:22:51.801222: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:22:51.801233: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:22:51.801250: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:22:51.801282: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at assign_op.h:117 : Resource exhausted: OOM when allocating tensor with shape[21] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:23:01.801458: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 84B.  Current allocation summary follows.
2019-01-29 16:23:01.801533: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:23:01.801548: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:23:01.801559: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:23:01.801572: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:23:01.801582: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:23:01.801592: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:01.801603: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:23:01.801613: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:01.801622: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:01.801634: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:23:01.801645: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:23:01.801656: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:23:01.801667: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:23:01.801677: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:01.801688: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:01.801699: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:23:01.801709: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:01.801718: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:01.801728: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:01.801737: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:01.801748: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:23:01.801757: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 256B was 256B, Chunk State:
2019-01-29 16:23:01.801768: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:23:01.801775: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:23:01.801782: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:23:01.801788: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:23:01.801795: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:23:01.801801: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:23:01.801807: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:23:01.801814: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:23:01.801821: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:23:01.801827: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:23:01.801833: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:23:01.801840: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:23:01.801846: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:23:01.801853: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:23:01.801859: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:23:01.801866: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:23:01.801899: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:23:01.801907: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:23:01.801913: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:23:01.801920: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:23:01.801926: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:23:01.801932: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:23:01.801938: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:23:01.801945: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:23:01.801951: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:23:01.801957: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:23:01.801964: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:23:01.801970: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:23:01.801976: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:23:01.801982: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:23:01.801989: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:23:01.801995: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:23:01.802001: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:23:01.802007: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:23:01.802013: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:23:01.802020: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:23:01.802026: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:23:01.802032: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:23:01.802038: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:23:01.802045: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:23:01.802051: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:23:01.802058: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:23:01.802064: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:23:01.802070: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:23:01.802077: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:23:01.802083: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:23:01.802090: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:23:01.802096: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:23:01.802102: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:23:01.802109: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:23:01.802115: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:23:01.802122: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:23:01.802128: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:23:01.802135: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:23:01.802222: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:23:01.802231: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:23:01.802237: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:23:01.802243: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:23:01.802250: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:23:01.802257: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:23:01.802263: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:23:01.802269: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:23:01.802277: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:23:01.802285: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:23:01.802291: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:23:01.802301: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:23:01.802310: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:23:01.802317: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:23:01.802324: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:23:01.802332: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:23:01.802340: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:23:01.802347: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:23:01.802355: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:23:01.802363: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:23:01.802371: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:23:01.802378: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:23:01.802386: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:23:01.802393: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:23:01.802401: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:23:01.802409: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:23:01.802417: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:23:01.802428: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:23:01.802445: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:23:01.802477: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at assign_op.h:117 : Resource exhausted: OOM when allocating tensor with shape[21] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:23:11.802660: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 84B.  Current allocation summary follows.
2019-01-29 16:23:11.802728: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:23:11.802743: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:23:11.802754: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:23:11.802766: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:23:11.802776: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:23:11.802787: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:11.802798: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:23:11.802808: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:11.802817: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:11.802828: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:23:11.802840: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:23:11.802851: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:23:11.802861: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:23:11.802872: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:11.802882: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:11.802893: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:23:11.802903: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:11.802913: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:11.802922: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:11.802931: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:11.802943: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:23:11.802952: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 256B was 256B, Chunk State:
2019-01-29 16:23:11.802963: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:23:11.802969: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:23:11.802976: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:23:11.802982: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:23:11.802989: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:23:11.802995: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:23:11.803001: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:23:11.803008: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:23:11.803015: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:23:11.803021: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:23:11.803028: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:23:11.803034: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:23:11.803040: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:23:11.803046: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:23:11.803053: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:23:11.803059: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:23:11.803088: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:23:11.803095: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:23:11.803102: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:23:11.803108: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:23:11.803115: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:23:11.803121: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:23:11.803127: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:23:11.803133: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:23:11.803140: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:23:11.803146: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:23:11.803152: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:23:11.803158: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:23:11.803164: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:23:11.803170: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:23:11.803177: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:23:11.803183: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:23:11.803189: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:23:11.803195: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:23:11.803201: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:23:11.803207: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:23:11.803214: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:23:11.803220: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:23:11.803226: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:23:11.803233: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:23:11.803239: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:23:11.803246: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:23:11.803252: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:23:11.803258: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:23:11.803264: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:23:11.803271: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:23:11.803277: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:23:11.803283: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:23:11.803290: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:23:11.803296: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:23:11.803302: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:23:11.803309: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:23:11.803315: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:23:11.803322: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:23:11.803403: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:23:11.803411: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:23:11.803417: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:23:11.803423: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:23:11.803430: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:23:11.803436: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:23:11.803443: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:23:11.803449: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:23:11.803456: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:23:11.803463: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:23:11.803471: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:23:11.803480: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:23:11.803488: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:23:11.803496: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:23:11.803511: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:23:11.803519: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:23:11.803527: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:23:11.803534: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:23:11.803542: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:23:11.803550: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:23:11.803558: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:23:11.803565: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:23:11.803573: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:23:11.803580: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:23:11.803588: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:23:11.803596: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:23:11.803604: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:23:11.803615: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:23:11.803632: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:23:11.803664: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at assign_op.h:117 : Resource exhausted: OOM when allocating tensor with shape[21] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:23:21.803849: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 42.0KiB.  Current allocation summary follows.
2019-01-29 16:23:21.803906: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:23:21.803920: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:23:21.803932: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:23:21.803943: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:23:21.803954: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:23:21.803964: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:21.803975: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:23:21.803985: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:21.803994: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:21.804005: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:23:21.804017: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:23:21.804029: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:23:21.804039: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:23:21.804050: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:21.804060: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:21.804071: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:23:21.804081: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:21.804090: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:21.804099: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:21.804108: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:21.804120: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:23:21.804129: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 42.0KiB was 32.0KiB, Chunk State:
2019-01-29 16:23:21.804140: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:23:21.804147: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:23:21.804154: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:23:21.804160: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:23:21.804167: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:23:21.804173: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:23:21.804180: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:23:21.804186: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:23:21.804193: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:23:21.804199: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:23:21.804205: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:23:21.804212: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:23:21.804218: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:23:21.804225: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:23:21.804231: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:23:21.804238: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:23:21.804299: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:23:21.804307: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:23:21.804313: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:23:21.804319: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:23:21.804326: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:23:21.804332: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:23:21.804338: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:23:21.804344: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:23:21.804351: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:23:21.804357: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:23:21.804363: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:23:21.804369: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:23:21.804376: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:23:21.804382: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:23:21.804388: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:23:21.804394: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:23:21.804400: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:23:21.804406: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:23:21.804413: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:23:21.804419: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:23:21.804425: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:23:21.804431: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:23:21.804437: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:23:21.804444: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:23:21.804450: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:23:21.804457: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:23:21.804463: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:23:21.804469: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:23:21.804475: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:23:21.804482: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:23:21.804488: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:23:21.804495: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:23:21.804507: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:23:21.804515: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:23:21.804521: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:23:21.804528: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:23:21.804534: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:23:21.804541: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:23:21.804621: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:23:21.804629: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:23:21.804636: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:23:21.804642: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:23:21.804649: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:23:21.804655: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:23:21.804662: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:23:21.804668: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:23:21.804676: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:23:21.804683: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:23:21.804690: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:23:21.804699: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:23:21.804707: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:23:21.804715: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:23:21.804722: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:23:21.804730: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:23:21.804737: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:23:21.804745: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:23:21.804753: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:23:21.804760: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:23:21.804768: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:23:21.804775: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:23:21.804783: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:23:21.804790: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:23:21.804798: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:23:21.804806: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:23:21.804814: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:23:21.804825: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:23:21.804841: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:23:21.804873: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at random_op.cc:202 : Resource exhausted: OOM when allocating tensor with shape[1,1,512,21] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:23:31.805056: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 441.0KiB.  Current allocation summary follows.
2019-01-29 16:23:31.805120: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:23:31.805134: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:23:31.805145: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:23:31.805157: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:23:31.805168: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:23:31.805179: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:31.805190: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:23:31.805200: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:31.805210: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:31.805222: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:23:31.805233: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:23:31.805244: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:23:31.805256: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:23:31.805267: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:31.805277: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:31.805288: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:23:31.805298: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:31.805307: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:31.805316: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:31.805325: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:31.805337: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:23:31.805347: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 441.0KiB was 256.0KiB, Chunk State:
2019-01-29 16:23:31.805357: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:23:31.805364: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:23:31.805371: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:23:31.805377: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:23:31.805384: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:23:31.805390: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:23:31.805396: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:23:31.805403: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:23:31.805410: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:23:31.805416: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:23:31.805423: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:23:31.805429: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:23:31.805435: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:23:31.805441: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:23:31.805448: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:23:31.805454: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:23:31.805477: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:23:31.805485: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:23:31.805491: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:23:31.805498: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:23:31.805514: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:23:31.805521: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:23:31.805527: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:23:31.805534: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:23:31.805540: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:23:31.805546: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:23:31.805553: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:23:31.805559: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:23:31.805565: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:23:31.805572: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:23:31.805578: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:23:31.805584: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:23:31.805590: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:23:31.805596: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:23:31.805602: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:23:31.805608: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:23:31.805615: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:23:31.805621: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:23:31.805627: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:23:31.805634: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:23:31.805640: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:23:31.805647: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:23:31.805653: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:23:31.805659: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:23:31.805665: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:23:31.805672: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:23:31.805678: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:23:31.805685: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:23:31.805691: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:23:31.805697: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:23:31.805703: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:23:31.805710: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:23:31.805716: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:23:31.805722: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:23:31.805806: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:23:31.805814: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:23:31.805820: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:23:31.805826: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:23:31.805833: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:23:31.805839: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:23:31.805846: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:23:31.805852: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:23:31.805858: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:23:31.805865: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:23:31.805872: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:23:31.805883: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:23:31.805892: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:23:31.805899: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:23:31.805907: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:23:31.805915: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:23:31.805922: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:23:31.805930: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:23:31.805938: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:23:31.805946: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:23:31.805953: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:23:31.805961: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:23:31.805968: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:23:31.805975: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:23:31.805983: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:23:31.805991: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:23:31.805999: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:23:31.806011: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:23:31.806028: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:23:31.806060: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at random_op.cc:202 : Resource exhausted: OOM when allocating tensor with shape[16,16,21,21] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:23:41.806240: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 27.6KiB.  Current allocation summary follows.
2019-01-29 16:23:41.806303: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:23:41.806317: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:23:41.806328: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:23:41.806340: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:23:41.806351: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:23:41.806361: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:41.806372: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:23:41.806382: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:41.806392: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:41.806404: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:23:41.806415: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:23:41.806426: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:23:41.806437: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:23:41.806447: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:41.806458: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:41.806469: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:23:41.806478: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:41.806488: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:41.806497: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:41.806517: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:41.806530: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:23:41.806540: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 27.8KiB was 16.0KiB, Chunk State:
2019-01-29 16:23:41.806551: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:23:41.806558: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:23:41.806565: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:23:41.806571: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:23:41.806578: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:23:41.806584: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:23:41.806591: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:23:41.806597: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:23:41.806604: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:23:41.806610: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:23:41.806617: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:23:41.806623: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:23:41.806629: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:23:41.806636: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:23:41.806642: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:23:41.806649: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:23:41.806678: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:23:41.806686: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:23:41.806692: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:23:41.806699: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:23:41.806705: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:23:41.806711: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:23:41.806717: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:23:41.806724: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:23:41.806730: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:23:41.806736: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:23:41.806742: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:23:41.806748: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:23:41.806754: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:23:41.806761: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:23:41.806767: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:23:41.806773: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:23:41.806779: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:23:41.806785: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:23:41.806791: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:23:41.806797: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:23:41.806803: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:23:41.806810: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:23:41.806816: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:23:41.806822: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:23:41.806829: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:23:41.806835: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:23:41.806842: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:23:41.806848: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:23:41.806854: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:23:41.806860: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:23:41.806867: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:23:41.806873: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:23:41.806879: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:23:41.806886: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:23:41.806892: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:23:41.806898: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:23:41.806905: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:23:41.806911: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:23:41.806994: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:23:41.807003: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:23:41.807010: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:23:41.807016: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:23:41.807022: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:23:41.807029: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:23:41.807035: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:23:41.807041: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:23:41.807048: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:23:41.807056: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:23:41.807063: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:23:41.807073: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:23:41.807081: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:23:41.807088: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:23:41.807096: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:23:41.807103: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:23:41.807111: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:23:41.807118: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:23:41.807126: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:23:41.807134: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:23:41.807142: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:23:41.807149: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:23:41.807157: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:23:41.807164: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:23:41.807172: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:23:41.807180: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:23:41.807188: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:23:41.807199: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:23:41.807216: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:23:41.807248: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at random_op.cc:202 : Resource exhausted: OOM when allocating tensor with shape[4,4,21,21] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2019-01-29 16:23:51.807434: W tensorflow/core/common_runtime/bfc_allocator.cc:267] Allocator (GPU_0_bfc) ran out of memory trying to allocate 27.6KiB.  Current allocation summary follows.
2019-01-29 16:23:51.807496: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (256):   Total Chunks: 34, Chunks in use: 34. 8.5KiB allocated for chunks. 8.5KiB in use in bin. 972B client-requested in use in bin.
2019-01-29 16:23:51.807518: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (512):   Total Chunks: 3, Chunks in use: 3. 1.5KiB allocated for chunks. 1.5KiB in use in bin. 1.5KiB client-requested in use in bin.
2019-01-29 16:23:51.807530: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1024):  Total Chunks: 5, Chunks in use: 5. 5.2KiB allocated for chunks. 5.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2019-01-29 16:23:51.807543: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2048):  Total Chunks: 7, Chunks in use: 7. 14.0KiB allocated for chunks. 14.0KiB in use in bin. 14.0KiB client-requested in use in bin.
2019-01-29 16:23:51.807554: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4096):  Total Chunks: 1, Chunks in use: 1. 6.8KiB allocated for chunks. 6.8KiB in use in bin. 6.8KiB client-requested in use in bin.
2019-01-29 16:23:51.807564: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8192):  Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:51.807575: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16384):         Total Chunks: 1, Chunks in use: 1. 16.0KiB allocated for chunks. 16.0KiB in use in bin. 16.0KiB client-requested in use in bin.
2019-01-29 16:23:51.807585: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (32768):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:51.807595: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (65536):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:51.807607: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (131072):        Total Chunks: 1, Chunks in use: 1. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2019-01-29 16:23:51.807618: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (262144):        Total Chunks: 1, Chunks in use: 1. 288.0KiB allocated for chunks. 288.0KiB in use in bin. 288.0KiB client-requested in use in bin.
2019-01-29 16:23:51.807629: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (524288):        Total Chunks: 1, Chunks in use: 1. 576.0KiB allocated for chunks. 576.0KiB in use in bin. 576.0KiB client-requested in use in bin.
2019-01-29 16:23:51.807640: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (1048576):       Total Chunks: 1, Chunks in use: 1. 1.12MiB allocated for chunks. 1.12MiB in use in bin. 1.12MiB client-requested in use in bin.
2019-01-29 16:23:51.807650: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (2097152):       Total Chunks: 2, Chunks in use: 2. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:51.807660: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (4194304):       Total Chunks: 1, Chunks in use: 1. 4.50MiB allocated for chunks. 4.50MiB in use in bin. 4.50MiB client-requested in use in bin.
2019-01-29 16:23:51.807671: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (8388608):       Total Chunks: 5, Chunks in use: 5. 45.00MiB allocated for chunks. 45.00MiB in use in bin. 45.00MiB client-requested in use in bin.
2019-01-29 16:23:51.807681: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (16777216):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:51.807691: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (33554432):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:51.807700: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (67108864):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:51.807709: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (134217728):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-01-29 16:23:51.807720: I tensorflow/core/common_runtime/bfc_allocator.cc:597] Bin (268435456):     Total Chunks: 1, Chunks in use: 1. 493.96MiB allocated for chunks. 493.96MiB in use in bin. 392.00MiB client-requested in use in bin.
2019-01-29 16:23:51.807730: I tensorflow/core/common_runtime/bfc_allocator.cc:613] Bin for 27.8KiB was 16.0KiB, Chunk State:
2019-01-29 16:23:51.807741: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800000 of size 1280
2019-01-29 16:23:51.807748: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800500 of size 256
2019-01-29 16:23:51.807755: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800600 of size 256
2019-01-29 16:23:51.807761: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800700 of size 256
2019-01-29 16:23:51.807767: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800800 of size 256
2019-01-29 16:23:51.807774: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800900 of size 256
2019-01-29 16:23:51.807780: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800a00 of size 256
2019-01-29 16:23:51.807787: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800b00 of size 256
2019-01-29 16:23:51.807794: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800c00 of size 512
2019-01-29 16:23:51.807801: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800e00 of size 256
2019-01-29 16:23:51.807807: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d800f00 of size 256
2019-01-29 16:23:51.807813: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801000 of size 256
2019-01-29 16:23:51.807819: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801100 of size 256
2019-01-29 16:23:51.807826: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801200 of size 1024
2019-01-29 16:23:51.807833: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801600 of size 256
2019-01-29 16:23:51.807839: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801700 of size 256
2019-01-29 16:23:51.807862: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801800 of size 256
2019-01-29 16:23:51.807869: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801900 of size 256
2019-01-29 16:23:51.807875: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d801a00 of size 2048
2019-01-29 16:23:51.807882: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802200 of size 256
2019-01-29 16:23:51.807888: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802300 of size 256
2019-01-29 16:23:51.807894: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802400 of size 256
2019-01-29 16:23:51.807900: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802500 of size 256
2019-01-29 16:23:51.807907: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d802600 of size 16384
2019-01-29 16:23:51.807913: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806600 of size 256
2019-01-29 16:23:51.807919: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806700 of size 256
2019-01-29 16:23:51.807926: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806800 of size 256
2019-01-29 16:23:51.807932: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806900 of size 256
2019-01-29 16:23:51.807938: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806a00 of size 256
2019-01-29 16:23:51.807944: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806b00 of size 256
2019-01-29 16:23:51.807950: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806c00 of size 256
2019-01-29 16:23:51.807956: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806d00 of size 256
2019-01-29 16:23:51.807963: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806e00 of size 256
2019-01-29 16:23:51.807969: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d806f00 of size 256
2019-01-29 16:23:51.807975: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807000 of size 256
2019-01-29 16:23:51.807981: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807100 of size 256
2019-01-29 16:23:51.807987: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807200 of size 256
2019-01-29 16:23:51.807994: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d807300 of size 6912
2019-01-29 16:23:51.808000: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808e00 of size 256
2019-01-29 16:23:51.808006: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d808f00 of size 256
2019-01-29 16:23:51.808013: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d809000 of size 147456
2019-01-29 16:23:51.808019: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d82d000 of size 294912
2019-01-29 16:23:51.808026: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875000 of size 512
2019-01-29 16:23:51.808032: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875200 of size 512
2019-01-29 16:23:51.808038: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d875400 of size 589824
2019-01-29 16:23:51.808045: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090d905400 of size 1179648
2019-01-29 16:23:51.808051: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25400 of size 1024
2019-01-29 16:23:51.808057: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25800 of size 1024
2019-01-29 16:23:51.808064: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da25c00 of size 1024
2019-01-29 16:23:51.808070: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090da26000 of size 2359296
2019-01-29 16:23:51.808076: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dc66000 of size 2359296
2019-01-29 16:23:51.808083: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090dea6000 of size 4718592
2019-01-29 16:23:51.808089: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326000 of size 2048
2019-01-29 16:23:51.808096: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e326800 of size 2048
2019-01-29 16:23:51.808181: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327000 of size 2048
2019-01-29 16:23:51.808189: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e327800 of size 2048
2019-01-29 16:23:51.808195: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328000 of size 2048
2019-01-29 16:23:51.808201: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e328800 of size 2048
2019-01-29 16:23:51.808208: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090e329000 of size 9437184
2019-01-29 16:23:51.808214: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090ec29000 of size 9437184
2019-01-29 16:23:51.808221: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090f529000 of size 9437184
2019-01-29 16:23:51.808227: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f090fe29000 of size 9437184
2019-01-29 16:23:51.808234: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0910729000 of size 9437184
2019-01-29 16:23:51.808241: I tensorflow/core/common_runtime/bfc_allocator.cc:632] Chunk at 0x7f0911029000 of size 517959680
2019-01-29 16:23:51.808248: I tensorflow/core/common_runtime/bfc_allocator.cc:638]      Summary of in-use Chunks by size:
2019-01-29 16:23:51.808258: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 34 Chunks of size 256 totalling 8.5KiB
2019-01-29 16:23:51.808266: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 3 Chunks of size 512 totalling 1.5KiB
2019-01-29 16:23:51.808274: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 4 Chunks of size 1024 totalling 4.0KiB
2019-01-29 16:23:51.808281: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1280 totalling 1.2KiB
2019-01-29 16:23:51.808289: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 7 Chunks of size 2048 totalling 14.0KiB
2019-01-29 16:23:51.808296: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 6912 totalling 6.8KiB
2019-01-29 16:23:51.808304: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 16384 totalling 16.0KiB
2019-01-29 16:23:51.808311: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 147456 totalling 144.0KiB
2019-01-29 16:23:51.808319: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 294912 totalling 288.0KiB
2019-01-29 16:23:51.808327: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 589824 totalling 576.0KiB
2019-01-29 16:23:51.808334: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 1179648 totalling 1.12MiB
2019-01-29 16:23:51.808342: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 2 Chunks of size 2359296 totalling 4.50MiB
2019-01-29 16:23:51.808349: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 4718592 totalling 4.50MiB
2019-01-29 16:23:51.808357: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 5 Chunks of size 9437184 totalling 45.00MiB
2019-01-29 16:23:51.808365: I tensorflow/core/common_runtime/bfc_allocator.cc:641] 1 Chunks of size 517959680 totalling 493.96MiB
2019-01-29 16:23:51.808373: I tensorflow/core/common_runtime/bfc_allocator.cc:645] Sum Total of in-use chunks: 550.12MiB
2019-01-29 16:23:51.808384: I tensorflow/core/common_runtime/bfc_allocator.cc:647] Stats:
Limit:                   576847872
InUse:                   576847872
MaxInUse:                576847872
NumAllocs:                      64
MaxAllocSize:            517959680

2019-01-29 16:23:51.808401: W tensorflow/core/common_runtime/bfc_allocator.cc:271] **********************************************************************************xxxxxxxxxxxxxxxxxx
2019-01-29 16:23:51.808433: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at random_op.cc:202 : Resource exhausted: OOM when allocating tensor with shape[4,4,21,21] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[{{node fc6/bias/Assign}} = Assign[T=DT_FLOAT, _grappler_relax_allocator_constraints=true, use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](fc6/bias, fc6/Const)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/liuqixuan_i/anaconda3/bin/mmtoir", line 11, in <module>
    sys.exit(_main())
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/mmdnn/conversion/_script/convertToIR.py", line 192, in _main
    ret = _convert(args)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/mmdnn/conversion/_script/convertToIR.py", line 46, in _convert
    parser = Keras2Parser(model)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/mmdnn/conversion/keras/keras2_parser.py", line 126, in __init__
    model = self._load_model(model[0], model[1])
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/mmdnn/conversion/keras/keras2_parser.py", line 83, in _load_model
    loaded_model.load_weights(model_weight_path)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/engine/network.py", line 1166, in load_weights
    f, self.layers, reshape=reshape)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 1058, in load_weights_from_hdf5_group
    K.batch_set_value(weight_value_tuples)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2470, in batch_set_value
    get_session().run(assign_ops, feed_dict=feed_dict)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 206, in get_session
    session.run(tf.variables_initializer(uninitialized_vars))
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[node fc6/bias/Assign (defined at /home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:402)  = Assign[T=DT_FLOAT, _grappler_relax_allocator_constraints=true, use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](fc6/bias, fc6/Const)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Caused by op 'fc6/bias/Assign', defined at:
  File "/home/liuqixuan_i/anaconda3/bin/mmtoir", line 11, in <module>
    sys.exit(_main())
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/mmdnn/conversion/_script/convertToIR.py", line 192, in _main
    ret = _convert(args)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/mmdnn/conversion/_script/convertToIR.py", line 46, in _convert
    parser = Keras2Parser(model)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/mmdnn/conversion/keras/keras2_parser.py", line 126, in __init__
    model = self._load_model(model[0], model[1])
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/mmdnn/conversion/keras/keras2_parser.py", line 78, in _load_model
    'DepthwiseConv2D': layers.DepthwiseConv2D})
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/engine/saving.py", line 492, in model_from_json
    return deserialize(config, custom_objects=custom_objects)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/layers/__init__.py", line 55, in deserialize
    printable_module_name='layer')
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/utils/generic_utils.py", line 145, in deserialize_keras_object
    list(custom_objects.items())))
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/engine/network.py", line 1032, in from_config
    process_node(layer, node_data)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/engine/network.py", line 991, in process_node
    layer(unpack_singleton(input_tensors), **kwargs)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/engine/base_layer.py", line 431, in __call__
    self.build(unpack_singleton(input_shapes))
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/layers/convolutional.py", line 147, in build
    constraint=self.bias_constraint)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/engine/base_layer.py", line 252, in add_weight
    constraint=constraint)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 402, in variable
    v = tf.Variable(value, dtype=tf.as_dtype(dtype), name=name)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 183, in __call__
    return cls._variable_v1_call(*args, **kwargs)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 146, in _variable_v1_call
    aggregation=aggregation)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 125, in <lambda>
    previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py", line 2444, in default_variable_creator
    expected_shape=expected_shape, import_scope=import_scope)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 187, in __call__
    return super(VariableMetaclass, cls).__call__(*args, **kwargs)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 1329, in __init__
    constraint=constraint)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/variables.py", line 1481, in _init_from_args
    validate_shape=validate_shape).op
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 221, in assign
    validate_shape=validate_shape)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 61, in assign
    use_locking=use_locking, name=name)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[4096] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
         [[node fc6/bias/Assign (defined at /home/liuqixuan_i/anaconda3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:402)  = Assign[T=DT_FLOAT, _grappler_relax_allocator_constraints=true, use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](fc6/bias, fc6/Const)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

Hi @missingdaysqxy , seems that your graphics memory is not enough. Thanks!

authors: missingdaysqxy rainLiuplus

repo: Microsoft/MMdnn