text:
Problem with gpu memory allocation

Click to expand!
Issue Type
Others
Source
source
Tensorflow Version
2.8.0
Custom Code
Yes
OS Platform and Distribution
Windows 10
Mobile device
No response
Python version
3.9.8
Bazel version
No response
GCC/Compiler version
No response
CUDA/cuDNN version
11.2
GPU model and memory
notebook version of nvidia gtx 1060 (6GB)
Current Behaviour?
I use tensorflow for image classification (20 classes) with convolutions. My dataset contains about 20000 train images and 5000 test images. Images (RGB) have 200x256 pixels. When I run script to train model using cpu everything seems fine. However when I try run script using gpu, after loading my training and test data I get error on model_fit function.

Num GPUs Available:  1
2022-05-04 17:58:58.482057: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-05-04 17:59:03.655618: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 4634 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1060, pci bus id: xxxx:xx:xx.x, compute capability: 6.1
1 Physical GPUs, 1 Logical GPUs
Path: D:/Dataset/seg_train
Loading seg_train
Path: D:/Dataset/seg_test
Loading seg_test
2022-05-04 18:02:48.971100: W tensorflow/core/common_runtime/bfc_allocator.cc:462] Allocator (GPU_0_bfc) ran out of memory trying to allocate 10.44GiB (rounded to 11206656000)requested by op _EagerConst
If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation.
Current allocation summary follows.
Current allocation summary follows.
2022-05-04 18:02:48.996013: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] BFCAllocator dump for GPU_0_bfc
2022-05-04 18:02:48.996173: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (256):  Total Chunks: 16, Chunks in use: 16. 4.0KiB allocated for chunks. 4.0KiB in use in bin. 392B client-requested in use in bin.
2022-05-04 18:02:48.996308: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (512):  Total Chunks: 1, Chunks in use: 1. 512B allocated for chunks. 512B in use in bin. 512B client-requested in use in bin.
2022-05-04 18:02:48.996473: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (1024):         Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2022-05-04 18:02:48.996629: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (2048):         Total Chunks: 2, Chunks in use: 1. 7.0KiB allocated for chunks. 3.5KiB in use in bin. 3.4KiB client-requested in use in bin.
2022-05-04 18:02:48.996889: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (4096):         Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-04 18:02:48.997493: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (8192):         Total Chunks: 1, Chunks in use: 1. 9.5KiB allocated for chunks. 9.5KiB in use in bin. 9.5KiB client-requested in use in bin.
2022-05-04 18:02:48.997960: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (16384):        Total Chunks: 1, Chunks in use: 0. 19.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-04 18:02:48.998482: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (32768):        Total Chunks: 2, Chunks in use: 1. 79.5KiB allocated for chunks. 36.0KiB in use in bin. 36.0KiB client-requested in use in bin.
2022-05-04 18:02:48.999113: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (65536):        Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-04 18:02:48.999710: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (131072):       Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-04 18:02:49.000273: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (262144):       Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-04 18:02:49.000742: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (524288):       Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-04 18:02:49.001208: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (1048576):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-04 18:02:49.001671: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (2097152):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-04 18:02:49.002131: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (4194304):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-04 18:02:49.002700: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (8388608):      Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-04 18:02:49.004034: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (16777216):     Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-04 18:02:49.004682: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (33554432):     Total Chunks: 1, Chunks in use: 1. 44.56MiB allocated for chunks. 44.56MiB in use in bin. 44.56MiB client-requested in use in bin.
2022-05-04 18:02:49.005383: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (67108864):     Total Chunks: 1, Chunks in use: 0. 89.12MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-04 18:02:49.007520: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (134217728):    Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-04 18:02:49.008016: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (268435456):    Total Chunks: 1, Chunks in use: 0. 4.39GiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-04 18:02:49.008477: I tensorflow/core/common_runtime/bfc_allocator.cc:1033] Bin for 10.44GiB was 256.00MiB, Chunk State:
2022-05-04 18:02:49.008888: I tensorflow/core/common_runtime/bfc_allocator.cc:1039]   Size: 4.39GiB | Requested Size: 0B | in_use: 0 | bin_num: 20, prev:   Size: 44.56MiB | Requested Size: 44.56MiB | in_use: 1 | bin_num: -1
2022-05-04 18:02:49.009335: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Next region of size 4859428864
2022-05-04 18:02:49.025604: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00000 of size 256 next 1
2022-05-04 18:02:49.025772: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00100 of size 1280 next 2
2022-05-04 18:02:49.026373: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00600 of size 256 next 3
2022-05-04 18:02:49.026991: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00700 of size 256 next 4
2022-05-04 18:02:49.028407: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00800 of size 256 next 5
2022-05-04 18:02:49.028560: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00900 of size 256 next 6
2022-05-04 18:02:49.029196: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00a00 of size 256 next 9
2022-05-04 18:02:49.029937: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00b00 of size 256 next 10
2022-05-04 18:02:49.030556: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00c00 of size 256 next 11
2022-05-04 18:02:49.031054: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00d00 of size 256 next 14
2022-05-04 18:02:49.031553: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00e00 of size 256 next 15
2022-05-04 18:02:49.031906: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00f00 of size 512 next 16
2022-05-04 18:02:49.032334: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a01100 of size 256 next 19
2022-05-04 18:02:49.032719: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a01200 of size 256 next 20
2022-05-04 18:02:49.033158: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a01300 of size 256 next 21
2022-05-04 18:02:49.033523: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a01400 of size 256 next 24
2022-05-04 18:02:49.033892: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a01500 of size 256 next 25
2022-05-04 18:02:49.034323: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a01600 of size 256 next 26
2022-05-04 18:02:49.034824: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at b03a01700 of size 3584 next 7
2022-05-04 18:02:49.035472: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a02500 of size 3584 next 8
2022-05-04 18:02:49.035923: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at b03a03300 of size 19456 next 23
2022-05-04 18:02:49.036957: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a07f00 of size 9728 next 22
2022-05-04 18:02:49.039251: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at b03a0a500 of size 44544 next 13
2022-05-04 18:02:49.039789: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a15300 of size 36864 next 12
2022-05-04 18:02:49.040234: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at b03a1e300 of size 93454336 next 18
2022-05-04 18:02:49.040779: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0933e300 of size 46727168 next 17
2022-05-04 18:02:49.041233: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at b0bfce300 of size 4719123712 next 18446744073709551615
2022-05-04 18:02:49.041719: I tensorflow/core/common_runtime/bfc_allocator.cc:1071]      Summary of in-use Chunks by size:
2022-05-04 18:02:49.042440: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 16 Chunks of size 256 totalling 4.0KiB
2022-05-04 18:02:49.042831: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 512 totalling 512B
2022-05-04 18:02:49.043889: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 1280 totalling 1.2KiB
2022-05-04 18:02:49.044474: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 3584 totalling 3.5KiB
2022-05-04 18:02:49.044901: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 9728 totalling 9.5KiB
2022-05-04 18:02:49.045330: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 36864 totalling 36.0KiB
2022-05-04 18:02:49.045784: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 46727168 totalling 44.56MiB
2022-05-04 18:02:49.046196: I tensorflow/core/common_runtime/bfc_allocator.cc:1078] Sum Total of in-use chunks: 44.62MiB
2022-05-04 18:02:49.046552: I tensorflow/core/common_runtime/bfc_allocator.cc:1080] total_region_allocated_bytes_: 4859428864 memory_limit_: 4859428864 available bytes: 0 curr_region_allocation_bytes_: 9718857728
2022-05-04 18:02:49.046902: I tensorflow/core/common_runtime/bfc_allocator.cc:1086] Stats:
Limit:                      4859428864
InUse:                        46783232
MaxInUse:                    140225792
NumAllocs:                          34
MaxAllocSize:                 46727168
Reserved:                            0
PeakReserved:                        0
LargestFreeBlock:                    0

2022-05-04 18:02:49.047317: W tensorflow/core/common_runtime/bfc_allocator.cc:474] ***_________________________________________________________________________________________________
Traceback (most recent call last):
  File "D:\DatasetProcessing\ImageClassification.py", line 394, in <module>
    main()
  File "D:\DatasetProcessing\ImageClassification.py", line 387, in main
    first_model()
  File "D:\DatasetProcessing\ImageClassification.py", line 162, in first_model
    history = model.fit(train_images, train_labels, batch_size=2, epochs=4)
  File "D:\WinPython\WPy64-3980\python-3.9.8.amd64\lib\site-packages\keras\utils\traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "D:\WinPython\WPy64-3980\python-3.9.8.amd64\lib\site-packages\tensorflow\python\framework\constant_op.py", line 102, in convert_to_eager_tensor
    return ops.EagerTensor(value, ctx.device_name, dtype)
tensorflow.python.framework.errors_impl.InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run _EagerConst: Dst tensor is not initialized.

I have a notebook with nvidia gtx 1060 (6GB) gpu. I've installed the newest driver available for this gpu and cuda version 11.2. I checked gpu values in task manager when script was running but it was 1% up to 5%. It looks like tensorflow is not using gpu at all.

I tried to use:

TF_GPU_ALLOCATOR=cuda_malloc_async

and

memory_limit=4096

and

allow_growth=True

I've also decreased batch_size from 128 to 2. But neither of these options worked.

Standalone code to reproduce the issue
model = tf.keras.Sequential([
            tf.keras.layers.Conv2D(32, (3,3), activation = 'relu', input_shape = (200, 256, 3)),
            tf.keras.layers.MaxPooling2D(2,2),
            tf.keras.layers.Conv2D(32, (3,3), activation = 'relu'),
            tf.keras.layers.MaxPooling2D(2,2),
            tf.keras.layers.Flatten(),
            tf.keras.layers.Dense(128, activation = tf.nn.relu),
            tf.keras.layers.Dense(20, activation = tf.nn.softmax)
        ])
model.compile(optimizer = 'adam', loss = 'sparse_categorical_crossentropy', metrics=['accuracy'])
history = model.fit(train_images, train_labels, batch_size=2, epochs=4)

Relevant log output
No response

@JamesJacquesDiego ,
Usually, this error raises when GPU memory is full.Please clear the memory and test your code.Also please take a look at this issue1 and issue2 with the similar error.Thanks!

Thank you for your suggestions. But I've checked gpu usage in task manager when script was running. And the usage was from 1% to 5%. It looks like tensorflow is not using gpu memory at all.
In the info there was information that tensorflow tried to allocate about 10GB of memory (but I have only 6GB gpu). I've also tried memory_limit=4096 and allow_growth=True.
PS
When I use the same code but with cpu there is no problem.

This looks like a GPU related OOM. Here is a more reproducible example based on what you provided earlier. You could try varying the input size and see if reducing the input size starts the network training.
import tensorflow as tf
import numpy as np
rand = np.random.default_rng()

x = np.random.rand(1000, 200, 256, 3)
y = np.array([rand.integers(20) for i in range(1000)])

model = tf.keras.Sequential([
            tf.keras.layers.Conv2D(32, (3,3), activation = 'relu', input_shape = (200, 256, 3)),
            tf.keras.layers.MaxPooling2D(2,2),
            tf.keras.layers.Conv2D(32, (3,3), activation = 'relu'),
            tf.keras.layers.MaxPooling2D(2,2),
            tf.keras.layers.Flatten(),
            tf.keras.layers.Dense(128, activation = tf.nn.relu),
            tf.keras.layers.Dense(20, activation = tf.nn.softmax)
        ])
model.compile(optimizer = 'adam', loss = 'sparse_categorical_crossentropy', metrics=['accuracy'])
history = model.fit(x, y, batch_size=2, epochs=4)

I've added three lines of code after y variable:
config = tf.compat.v1.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction=0.5
sess = tf.compat.v1.Session(config=config)
I've run your code and first 5 times it worked, but after that I got error. Here is what i get when it works:
2022-05-11 02:29:31.044577: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-05-11 02:29:31.044577: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-05-11 02:29:37.276618: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3071 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1060, pci bus id: xxxx:xx:xx:x, compute capability: 6.1
2022-05-11 02:29:37.415072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3071 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1060, pci bus id: xxxx:xx:xx:x, compute capability: 6.1
Epoch 1/4
2022-05-11 02:29:31.044577: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-05-11 02:29:37.276618: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3071 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1060, pci bus id: xxxx:xx:xx:x, compute capability: 6.1
2022-05-11 02:29:37.415072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3071 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1060, pci bus id: xxxx:xx:xx:x, compute capability: 6.1
2022-05-11 02:29:42.112058: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8101
500/500 [==============================] - 13s 12ms/step - loss: 3.0854 - accuracy: 0.0520
Epoch 2/4
500/500 [==============================] - 6s 12ms/step - loss: 2.9949 - accuracy: 0.0510
Epoch 3/4
500/500 [==============================] - 6s 12ms/step - loss: 2.9938 - accuracy: 0.0550
Epoch 4/4
500/500 [==============================] - 6s 11ms/step - loss: 2.9932 - accuracy: 0.0580

And here is what I get when it doesn't work:

2022-05-11 02:29:31.044577: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-05-11 02:29:37.276618: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3071 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1060, pci bus id: xxxx:xx:xx.x, compute capability: 6.1
2022-05-11 02:29:37.415072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3071 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1060, pci bus id: xxxx:xx:xx:x, compute capability: 6.1
2022-05-11 02:29:42.112058: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8101
2022-05-11 02:31:19.739728: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3071 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1060, pci bus id: xxxx:xx:xx:x, compute capability: 6.1
2022-05-11 02:32:20.290753: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3071 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1060, pci bus id: xxxx:xx:xx:x, compute capability: 6.1
2022-05-11 02:32:50.716189: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3071 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1060, pci bus id: xxxx:xx:xx:x, compute capability: 6.1
2022-05-11 02:33:19.014200: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3071 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1060, pci bus id: xxxx:xx:xx:x, compute capability: 6.1
Traceback (most recent call last):

  File "D:scripts\gpu_test.py", line 33, in <module>
    history = model.fit(x, y, batch_size=2, epochs=4)

  File "D:WinPython\WPy64-3980\python-3.9.8.amd64\lib\site-packages\keras\utils\traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None

  File "D:WinPython\WPy64-3980\python-3.9.8.amd64\lib\site-packages\tensorflow\python\framework\constant_op.py", line 102, in convert_to_eager_tensor
    return ops.EagerTensor(value, ctx.device_name, dtype)

InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run _EagerConst: Dst tensor is not initialized.

2022-05-11 02:29:31.044577: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-05-11 02:29:37.276618: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3071 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1060, pci bus id: xxxx:xx:xx:x, compute capability: 6.1
2022-05-11 02:29:37.415072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3071 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1060, pci bus id: xxxx:xx:xx:x, compute capability: 6.1
2022-05-11 02:29:42.112058: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8101
2022-05-11 02:31:19.739728: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3071 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1060, pci bus id: xxxx:xx:xx:x, compute capability: 6.1
2022-05-11 02:32:20.290753: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3071 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1060, pci bus id: xxxx:xx:xx:x, compute capability: 6.1
2022-05-11 02:32:50.716189: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3071 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1060, pci bus id: xxxx:xx:xx:x, compute capability: 6.1
2022-05-11 02:33:19.014200: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3071 MB memory:  -> device: 0, name: NVIDIA GeForce GTX 1060, pci bus id: xxxx:xx:xx:x, compute capability: 6.1
2022-05-11 02:33:29.241618: W tensorflow/core/common_runtime/bfc_allocator.cc:462] Allocator (GPU_0_bfc) ran out of memory trying to allocate 585.94MiB (rounded to 614400000)requested by op _EagerConst
If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation.
Current allocation summary follows.
Current allocation summary follows.
2022-05-11 02:33:29.241729: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] BFCAllocator dump for GPU_0_bfc
2022-05-11 02:33:29.241792: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (256): 	Total Chunks: 107, Chunks in use: 107. 26.8KiB allocated for chunks. 26.8KiB in use in bin. 4.6KiB client-requested in use in bin.
2022-05-11 02:33:29.241846: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (512): 	Total Chunks: 13, Chunks in use: 13. 6.5KiB allocated for chunks. 6.5KiB in use in bin. 6.5KiB client-requested in use in bin.
2022-05-11 02:33:29.241897: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (1024): 	Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2022-05-11 02:33:29.241948: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (2048): 	Total Chunks: 10, Chunks in use: 10. 35.0KiB allocated for chunks. 35.0KiB in use in bin. 33.8KiB client-requested in use in bin.
2022-05-11 02:33:29.241998: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (4096): 	Total Chunks: 4, Chunks in use: 3. 23.8KiB allocated for chunks. 17.5KiB in use in bin. 10.1KiB client-requested in use in bin.
2022-05-11 02:33:29.242049: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (8192): 	Total Chunks: 16, Chunks in use: 16. 158.0KiB allocated for chunks. 158.0KiB in use in bin. 153.4KiB client-requested in use in bin.
2022-05-11 02:33:29.242098: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (16384): 	Total Chunks: 1, Chunks in use: 0. 20.0KiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-11 02:33:29.242343: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (32768): 	Total Chunks: 14, Chunks in use: 13. 534.0KiB allocated for chunks. 492.0KiB in use in bin. 468.0KiB client-requested in use in bin.
2022-05-11 02:33:29.242411: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (65536): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-11 02:33:29.242460: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (131072): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-11 02:33:29.242504: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (262144): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-11 02:33:29.242548: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (524288): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-11 02:33:29.242591: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (1048576): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-11 02:33:29.242631: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (2097152): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-11 02:33:29.242671: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (4194304): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-11 02:33:29.242711: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (8388608): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-11 02:33:29.242752: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (16777216): 	Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-11 02:33:29.242800: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (33554432): 	Total Chunks: 14, Chunks in use: 13. 650.33MiB allocated for chunks. 604.50MiB in use in bin. 604.50MiB client-requested in use in bin.
2022-05-11 02:33:29.242845: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (67108864): 	Total Chunks: 1, Chunks in use: 0. 93.00MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-11 02:33:29.242890: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (134217728): 	Total Chunks: 1, Chunks in use: 0. 167.44MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2022-05-11 02:33:29.242937: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (268435456): 	Total Chunks: 4, Chunks in use: 3. 2.11GiB allocated for chunks. 1.72GiB in use in bin. 1.72GiB client-requested in use in bin.
2022-05-11 02:33:29.242978: I tensorflow/core/common_runtime/bfc_allocator.cc:1033] Bin for 585.94MiB was 256.00MiB, Chunk State:
2022-05-11 02:33:29.243026: I tensorflow/core/common_runtime/bfc_allocator.cc:1039]   Size: 402.54MiB | Requested Size: 2.96MiB | in_use: 0 | bin_num: 20, prev:   Size: 585.94MiB | Requested Size: 585.94MiB | in_use: 1 | bin_num: -1
2022-05-11 02:33:29.243062: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Next region of size 3221127168
2022-05-11 02:33:29.243140: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00000 of size 256 next 1
2022-05-11 02:33:29.243175: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00100 of size 1280 next 2
2022-05-11 02:33:29.243205: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00600 of size 256 next 3
2022-05-11 02:33:29.243232: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00700 of size 256 next 4
2022-05-11 02:33:29.243267: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00800 of size 256 next 5
2022-05-11 02:33:29.243389: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00900 of size 256 next 6
2022-05-11 02:33:29.243430: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00a00 of size 256 next 9
2022-05-11 02:33:29.243460: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00b00 of size 256 next 10
2022-05-11 02:33:29.243489: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00c00 of size 256 next 11
2022-05-11 02:33:29.243517: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00d00 of size 256 next 14
2022-05-11 02:33:29.243544: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00e00 of size 256 next 15
2022-05-11 02:33:29.243573: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a00f00 of size 512 next 16
2022-05-11 02:33:29.243601: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a01100 of size 256 next 19
2022-05-11 02:33:29.243629: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a01200 of size 256 next 20
2022-05-11 02:33:29.243656: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a01300 of size 256 next 21
2022-05-11 02:33:29.243684: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a01400 of size 256 next 24
2022-05-11 02:33:29.243712: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a01500 of size 256 next 25
2022-05-11 02:33:29.243740: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a01600 of size 256 next 26
2022-05-11 02:33:29.243767: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a01700 of size 256 next 29
2022-05-11 02:33:29.243795: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a01800 of size 256 next 30
2022-05-11 02:33:29.243823: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a01900 of size 256 next 31
2022-05-11 02:33:29.243850: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a01a00 of size 256 next 32
2022-05-11 02:33:29.243878: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a01b00 of size 256 next 33
2022-05-11 02:33:29.243906: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a01c00 of size 256 next 34
2022-05-11 02:33:29.243933: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a01d00 of size 256 next 35
2022-05-11 02:33:29.243961: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a01e00 of size 256 next 36
2022-05-11 02:33:29.243989: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a01f00 of size 256 next 38
2022-05-11 02:33:29.244017: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a02000 of size 256 next 39
2022-05-11 02:33:29.244045: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a02100 of size 512 next 41
2022-05-11 02:33:29.244072: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a02300 of size 256 next 43
2022-05-11 02:33:29.244100: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a02400 of size 256 next 7
2022-05-11 02:33:29.244128: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a02500 of size 3584 next 8
2022-05-11 02:33:29.244156: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a03300 of size 256 next 90
2022-05-11 02:33:29.244184: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a03400 of size 256 next 91
2022-05-11 02:33:29.244211: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a03500 of size 256 next 92
2022-05-11 02:33:29.244239: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a03600 of size 256 next 93
2022-05-11 02:33:29.244276: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a03700 of size 256 next 94
2022-05-11 02:33:29.244305: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a03800 of size 256 next 95
2022-05-11 02:33:29.244337: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a03900 of size 256 next 106
2022-05-11 02:33:29.244368: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a03a00 of size 256 next 112
2022-05-11 02:33:29.244396: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a03b00 of size 256 next 113
2022-05-11 02:33:29.244424: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a03c00 of size 256 next 104
2022-05-11 02:33:29.244452: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a03d00 of size 256 next 108
2022-05-11 02:33:29.244481: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a03e00 of size 5376 next 28
2022-05-11 02:33:29.244508: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a05300 of size 3584 next 37
2022-05-11 02:33:29.244537: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a06100 of size 3584 next 44
2022-05-11 02:33:29.244564: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a06f00 of size 256 next 46
2022-05-11 02:33:29.244593: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a07000 of size 512 next 48
2022-05-11 02:33:29.244620: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a07200 of size 256 next 50
2022-05-11 02:33:29.244648: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a07300 of size 256 next 51
2022-05-11 02:33:29.244676: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a07400 of size 256 next 52
2022-05-11 02:33:29.244703: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a07500 of size 256 next 53
2022-05-11 02:33:29.244731: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a07600 of size 256 next 54
2022-05-11 02:33:29.244758: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a07700 of size 256 next 55
2022-05-11 02:33:29.244786: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a07800 of size 256 next 56
2022-05-11 02:33:29.244813: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a07900 of size 256 next 57
2022-05-11 02:33:29.244841: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a07a00 of size 256 next 58
2022-05-11 02:33:29.244869: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a07b00 of size 256 next 76
2022-05-11 02:33:29.244896: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a07c00 of size 256 next 73
2022-05-11 02:33:29.244924: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a07d00 of size 512 next 67
2022-05-11 02:33:29.244952: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a07f00 of size 256 next 70
2022-05-11 02:33:29.244980: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a08000 of size 256 next 66
2022-05-11 02:33:29.245008: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a08100 of size 256 next 115
2022-05-11 02:33:29.245035: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a08200 of size 256 next 23
2022-05-11 02:33:29.245063: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a08300 of size 10240 next 22
2022-05-11 02:33:29.245092: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a0ab00 of size 43008 next 13
2022-05-11 02:33:29.245121: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a15300 of size 36864 next 12
2022-05-11 02:33:29.245149: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03a1e300 of size 48758784 next 40
2022-05-11 02:33:29.245176: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0689e300 of size 10240 next 42
2022-05-11 02:33:29.245204: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068a0b00 of size 36864 next 45
2022-05-11 02:33:29.245231: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068a9b00 of size 10240 next 49
2022-05-11 02:33:29.245363: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068ac300 of size 256 next 64
2022-05-11 02:33:29.245404: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068ac400 of size 6912 next 60
2022-05-11 02:33:29.245433: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068adf00 of size 3584 next 68
2022-05-11 02:33:29.245462: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068aed00 of size 8192 next 74
2022-05-11 02:33:29.245490: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068b0d00 of size 256 next 72
2022-05-11 02:33:29.245517: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068b0e00 of size 256 next 77
2022-05-11 02:33:29.245545: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068b0f00 of size 512 next 79
2022-05-11 02:33:29.245573: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068b1100 of size 11264 next 75
2022-05-11 02:33:29.245600: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068b3d00 of size 10240 next 69
2022-05-11 02:33:29.245628: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068b6500 of size 43008 next 65
2022-05-11 02:33:29.245655: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068c0d00 of size 36864 next 63
2022-05-11 02:33:29.245683: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068c9d00 of size 256 next 80
2022-05-11 02:33:29.245710: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068c9e00 of size 3584 next 81
2022-05-11 02:33:29.245738: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068cac00 of size 256 next 82
2022-05-11 02:33:29.245766: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068cad00 of size 36864 next 83
2022-05-11 02:33:29.245794: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068d3d00 of size 256 next 84
2022-05-11 02:33:29.245822: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068d3e00 of size 512 next 85
2022-05-11 02:33:29.245850: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068d4000 of size 10240 next 86
2022-05-11 02:33:29.245877: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068d6800 of size 256 next 87
2022-05-11 02:33:29.245905: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068d6900 of size 512 next 59
2022-05-11 02:33:29.245933: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068d6b00 of size 256 next 88
2022-05-11 02:33:29.245961: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068d6c00 of size 256 next 111
2022-05-11 02:33:29.245989: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068d6d00 of size 256 next 105
2022-05-11 02:33:29.246017: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068d6e00 of size 256 next 97
2022-05-11 02:33:29.246044: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068d6f00 of size 256 next 100
2022-05-11 02:33:29.246072: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068d7000 of size 256 next 102
2022-05-11 02:33:29.246100: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068d7100 of size 512 next 117
2022-05-11 02:33:29.246127: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068d7300 of size 256 next 118
2022-05-11 02:33:29.246155: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068d7400 of size 256 next 120
2022-05-11 02:33:29.246183: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068d7500 of size 256 next 122
2022-05-11 02:33:29.246210: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068d7600 of size 256 next 103
2022-05-11 02:33:29.246238: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068d7700 of size 3584 next 109
2022-05-11 02:33:29.246276: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068d8500 of size 8192 next 99
2022-05-11 02:33:29.246414: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068da500 of size 12288 next 98
2022-05-11 02:33:29.246455: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068dd500 of size 10240 next 89
2022-05-11 02:33:29.246484: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068dfd00 of size 43008 next 101
2022-05-11 02:33:29.246512: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068ea500 of size 36864 next 96
2022-05-11 02:33:29.246540: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068f3500 of size 3584 next 119
2022-05-11 02:33:29.246568: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068f4300 of size 36864 next 121
2022-05-11 02:33:29.246597: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068fd300 of size 512 next 123
2022-05-11 02:33:29.246624: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068fd500 of size 10240 next 124
2022-05-11 02:33:29.246652: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068ffd00 of size 256 next 125
2022-05-11 02:33:29.246680: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068ffe00 of size 256 next 126
2022-05-11 02:33:29.246708: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b068fff00 of size 256 next 127
2022-05-11 02:33:29.246735: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06900000 of size 256 next 128
2022-05-11 02:33:29.246763: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06900100 of size 256 next 129
2022-05-11 02:33:29.246791: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06900200 of size 256 next 130
2022-05-11 02:33:29.246819: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06900300 of size 256 next 131
2022-05-11 02:33:29.246846: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06900400 of size 256 next 132
2022-05-11 02:33:29.246874: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06900500 of size 256 next 144
2022-05-11 02:33:29.246902: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06900600 of size 256 next 134
2022-05-11 02:33:29.246930: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06900700 of size 256 next 141
2022-05-11 02:33:29.246958: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06900800 of size 512 next 149
2022-05-11 02:33:29.246986: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06900a00 of size 256 next 139
2022-05-11 02:33:29.247014: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06900b00 of size 256 next 151
2022-05-11 02:33:29.247042: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06900c00 of size 256 next 145
2022-05-11 02:33:29.247070: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06900d00 of size 256 next 138
2022-05-11 02:33:29.247098: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06900e00 of size 5632 next 148
2022-05-11 02:33:29.247126: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06902400 of size 3584 next 153
2022-05-11 02:33:29.247154: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06903200 of size 8192 next 152
2022-05-11 02:33:29.247182: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06905200 of size 256 next 135
2022-05-11 02:33:29.247209: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06905300 of size 256 next 142
2022-05-11 02:33:29.247237: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06905400 of size 512 next 154
2022-05-11 02:33:29.247410: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06905600 of size 11264 next 146
2022-05-11 02:33:29.247503: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06908200 of size 10240 next 133
2022-05-11 02:33:29.247584: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0690aa00 of size 43008 next 150
2022-05-11 02:33:29.247687: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06915200 of size 36864 next 140
2022-05-11 02:33:29.247751: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0691e200 of size 256 next 155
2022-05-11 02:33:29.247782: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0691e300 of size 3584 next 156
2022-05-11 02:33:29.247811: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0691f100 of size 256 next 157
2022-05-11 02:33:29.247840: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0691f200 of size 36864 next 158
2022-05-11 02:33:29.247868: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06928200 of size 256 next 159
2022-05-11 02:33:29.247896: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06928300 of size 512 next 160
2022-05-11 02:33:29.247924: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06928500 of size 10240 next 161
2022-05-11 02:33:29.247952: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0692ad00 of size 256 next 162
2022-05-11 02:33:29.247981: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0692ae00 of size 256 next 163
2022-05-11 02:33:29.248009: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0692af00 of size 256 next 164
2022-05-11 02:33:29.248037: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0692b000 of size 256 next 165
2022-05-11 02:33:29.248065: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0692b100 of size 256 next 166
2022-05-11 02:33:29.248093: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0692b200 of size 256 next 167
2022-05-11 02:33:29.248121: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0692b300 of size 256 next 168
2022-05-11 02:33:29.248149: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0692b400 of size 256 next 169
2022-05-11 02:33:29.248177: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0692b500 of size 256 next 170
2022-05-11 02:33:29.248206: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0692b600 of size 256 next 173
2022-05-11 02:33:29.248233: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0692b700 of size 256 next 184
2022-05-11 02:33:29.248272: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0692b800 of size 512 next 179
2022-05-11 02:33:29.248300: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0692ba00 of size 256 next 174
2022-05-11 02:33:29.248328: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0692bb00 of size 256 next 171
2022-05-11 02:33:29.248356: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0692bc00 of size 256 next 188
2022-05-11 02:33:29.248384: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0692bd00 of size 256 next 182
2022-05-11 02:33:29.248412: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at b0692be00 of size 6400 next 190
2022-05-11 02:33:29.248440: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0692d700 of size 3584 next 176
2022-05-11 02:33:29.248468: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at b0692e500 of size 20480 next 185
2022-05-11 02:33:29.248495: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06933500 of size 10240 next 181
2022-05-11 02:33:29.248523: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at b06935d00 of size 43008 next 175
2022-05-11 02:33:29.248551: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b06940500 of size 36864 next 187
2022-05-11 02:33:29.248579: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at b06949500 of size 48057856 next 18
2022-05-11 02:33:29.248607: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0971e300 of size 48758784 next 17
2022-05-11 02:33:29.248636: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0c59e300 of size 48758784 next 116
2022-05-11 02:33:29.248670: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0f41e300 of size 48758784 next 114
2022-05-11 02:33:29.248701: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b1229e300 of size 48758784 next 107
2022-05-11 02:33:29.248730: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b1511e300 of size 48758784 next 137
2022-05-11 02:33:29.248757: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b17f9e300 of size 48758784 next 143
2022-05-11 02:33:29.248785: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b1ae1e300 of size 48758784 next 147
2022-05-11 02:33:29.248812: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at b1dc9e300 of size 97517568 next 178
2022-05-11 02:33:29.248840: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b2399e300 of size 48758784 next 186
2022-05-11 02:33:29.248867: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at b2681e300 of size 175570944 next 27
2022-05-11 02:33:29.248895: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b30f8e300 of size 48758784 next 47
2022-05-11 02:33:29.248923: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b33e0e300 of size 48758784 next 78
2022-05-11 02:33:29.248950: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b36c8e300 of size 48758784 next 61
2022-05-11 02:33:29.248978: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b39b0e300 of size 48758784 next 71
2022-05-11 02:33:29.249006: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b3c98e300 of size 614400000 next 62
2022-05-11 02:33:29.249034: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b6137e300 of size 614400000 next 110
2022-05-11 02:33:29.249061: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b85d6e300 of size 614400000 next 136
2022-05-11 02:33:29.249090: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free  at baa75e300 of size 422092032 next 18446744073709551615
2022-05-11 02:33:29.249119: I tensorflow/core/common_runtime/bfc_allocator.cc:1071]      Summary of in-use Chunks by size:
2022-05-11 02:33:29.249164: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 107 Chunks of size 256 totalling 26.8KiB
2022-05-11 02:33:29.249202: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 13 Chunks of size 512 totalling 6.5KiB
2022-05-11 02:33:29.249252: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 1280 totalling 1.2KiB
2022-05-11 02:33:29.249291: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 10 Chunks of size 3584 totalling 35.0KiB
2022-05-11 02:33:29.249326: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 5376 totalling 5.2KiB
2022-05-11 02:33:29.249361: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 5632 totalling 5.5KiB
2022-05-11 02:33:29.249396: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 6912 totalling 6.8KiB
2022-05-11 02:33:29.249431: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 3 Chunks of size 8192 totalling 24.0KiB
2022-05-11 02:33:29.249467: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 10 Chunks of size 10240 totalling 100.0KiB
2022-05-11 02:33:29.249503: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 11264 totalling 22.0KiB
2022-05-11 02:33:29.249538: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 12288 totalling 12.0KiB
2022-05-11 02:33:29.249572: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 9 Chunks of size 36864 totalling 324.0KiB
2022-05-11 02:33:29.249607: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 4 Chunks of size 43008 totalling 168.0KiB
![image1-before running script](https://user-images.githubusercontent.com/44622349/167747019-e74d87d2-5612-409c-a945-d3d65631782f.PNG)
![image2-running script](https://user-images.githubusercontent.com/44622349/167747021-1682e710-809e-4e3f-b2d2-dac0e3ab5576.PNG)
![image3-error](https://user-images.githubusercontent.com/44622349/167747022-8cd3dbd1-1541-4b77-9681-a9e12a4d5ab6.PNG)

2022-05-11 02:33:29.249643: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 13 Chunks of size 48758784 totalling 604.50MiB
2022-05-11 02:33:29.249679: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 3 Chunks of size 614400000 totalling 1.72GiB
2022-05-11 02:33:29.249716: I tensorflow/core/common_runtime/bfc_allocator.cc:1078] Sum Total of in-use chunks: 2.31GiB
2022-05-11 02:33:29.249757: I tensorflow/core/common_runtime/bfc_allocator.cc:1080] total_region_allocated_bytes_: 3221127168 memory_limit_: 3221127168 available bytes: 0 curr_region_allocation_bytes_: 6442254336
2022-05-11 02:33:29.249807: I tensorflow/core/common_runtime/bfc_allocator.cc:1086] Stats:
Limit:                      3221127168
InUse:                      2477818880
MaxInUse:                   2575324928
NumAllocs:                      432324
MaxAllocSize:                614400000
Reserved:                            0
PeakReserved:                        0
LargestFreeBlock:                    0

2022-05-11 02:33:29.249874: W tensorflow/core/common_runtime/bfc_allocator.cc:474] **_***********__***____****************************************************************_____________

I used OHM to check gpu memory values. I've added images.

The lines you added have no effect, I assume you read this issue, you need to use the instructions related to TensorFlow 2. However, it seems that you don't have enough memory on the GPU.
To test this assertion you could change the input size( e.g.  x = np.random.rand(1000, 200, 256, 3) to  x = np.random.rand(1000, 100, 100, 3) ) and see if the error persis

So those lines don't work?
config = tf.compat.v1.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction=0.5
sess = tf.compat.v1.Session(config=config)

The code you've pasted  (with input size you set) works first few times and after 4th or 5th time I get error.
It works after I've change input size to x = np.random.rand(1000, 100, 100, 3) as you advised but reducing resolution of my images in my network is not a solution. What if I will need more images in the future to train my network or if smaller resolution would be bad for classification results?
When I tried to run the script with my network then I get info that tensorflow tries to allocate about 10GB. I have gtx 1060 so I have only 6GB.
@Kubasinska Don't get me wrong. Thanks for your suggestions and help. I am grateful but I'm trying to find the best solution.
I have only about 25000 images in my dataset. Some people train networks with few millions images. I don't know how they do it. Do they divide their data somehow into chunks?

@JamesJacquesDiego ,
In order to expedite the trouble-shooting process, could you please provide the complete code to reproduce the issue reported here.Thanks!

Well, it's almost a complete code. Above I load my train and test images with labels. Below I generate plots to display data. I've decreased resolution of my images and now it works. But it should be some way to load only part of dataset to gpu memory or create additional memory like with ram we can do (virtual ram)?

@JamesJacquesDiego ,
I was facing different error while executing the mentioned code.Please find the gist of it here.

train_images and train_labels are images and classes which i load. Here is used
x = np.random.rand(1000, 200, 256, 3) y = np.array([rand.integers(20) for i in range(1000)])
to simulate those data

authors: JamesJacquesDiego Kubasinska tilakrayal


text:
Vulnerability CVE-2022-32207、CVE-2022-32205、CVE-2022-32206、CVE-2022-32208、CVE-2022-35252

Click to expand!
Issue Type
Bug
Source
binary
Tensorflow Version
tf 2.6.5
Custom Code
No
OS Platform and Distribution
EulerOS 2.5\EulerOS 2.10
Mobile device

Python version
3.9
Bazel version

GCC/Compiler version

CUDA/cuDNN version

GPU model and memory

Current Behaviour?
Hi there, it seems that curl 7.83.1 has a vulnerability CVE-2022-32207 CVE-2022-32205 CVE-2022-32206 CVE-2022-32208 CVE-2022-35252. Please fix it or confirm tf 2.6.5 is not affected by it.
Here is the detail:
https://nvd.nist.gov/vuln/detail/CVE-2022-32207
https://nvd.nist.gov/vuln/detail/CVE-2022-32205
https://nvd.nist.gov/vuln/detail/CVE-2022-32206
https://nvd.nist.gov/vuln/detail/CVE-2022-32208
https://vulners.com/redhatcve/RH:CVE-2022-35252

Standalone code to reproduce the issue
-

Relevant log output
-

TF 2.6 is no longer in the window of supported versions

Can you help me analyze whether the TF 2.8.3 version is affected by these vulnerabilities?

Main branch: https://github.com/tensorflow/tensorflow/blob/92d57e61d41f8e6fca50d582ac7a56046a391591/tensorflow/workspace2.bzl#L493-L500
r2.10: https://github.com/tensorflow/tensorflow/blob/359c3cdfc5fabac82b3c70b3b6de2b0a8c16874f/tensorflow/workspace2.bzl#L491-L498
r2.9: https://github.com/tensorflow/tensorflow/blob/18960c44ad3f5219c22dca55f842912dbce78a07/tensorflow/workspace2.bzl#L483-L490
r2.8: https://github.com/tensorflow/tensorflow/blob/92a51d52ad199319e4f9de83fcbe970151dfed7e/tensorflow/workspace2.bzl#L490-L497
So, if you take the last release on each of these branches you are not affected

https://curl.se/docs/CVE-2022-35252.html
CVE-2022-35252 is fixed in CURL versions 7.85. 0 and later.

Then you might be affected on all supported versions. TF does not generate VEX statements at this point, so we cannot be certain.

@crbean,
Could you please take a look at this comment from the developer. Thank you!

source: gharchive/issue

authors: crbean mihaimaruseac tilakrayal

repo: tensorflow/tensorflow

license_type: permissive

license_source: github-api


text:
TensorBoard ImportError: No module named werkzeug

using the current HEAD of tensorflow, I have bumped into an issue when I execute tensorboard.
Version is reported as: 0.12.head
git rev-parse HEAD: a12c7dc3d83049e10c1dca8903d73cc71d3cb7b2
Linux: Linux tensor 4.4.0-53-generic #74-Ubuntu SMP Fri Dec 2 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

2017-01-25 17:02:36: I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcudnn.so.5 locally
2017-01-25 17:02:36: I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcufft.so.8.0 locally
2017-01-25 17:02:36: I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcuda.so.1 locally
2017-01-25 17:02:36: I tensorflow/stream_executor/dso_loader.cc:125] successfully opened CUDA library libcurand.so.8.0 locally
Traceback (most recent call last):
File "/usr/local/bin/tensorboard", line 9, in
load_entry_point('tensorflow', 'console_scripts', 'tensorboard')()
File "/home/greg/.local/lib/python2.7/site-packages/pkg_resources/init.py", line 542, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/home/greg/.local/lib/python2.7/site-packages/pkg_resources/init.py", line 2575, in load_entry_point
return ep.load()
File "/home/greg/.local/lib/python2.7/site-packages/pkg_resources/init.py", line 2235, in load
return self.resolve()
File "/home/greg/.local/lib/python2.7/site-packages/pkg_resources/init.py", line 2241, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "/home/greg/tensorflow/_python_build/tensorflow/tensorboard/tensorboard.py", line 26, in
from werkzeug import serving
ImportError: No module named werkzeug

I see the werkzeug.BUILD file on my system so not sure why it cannot be found.

Just to be clear, you ran configure, built the whl file, installed, and running from there?
/cc @dandelionmane

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/pip_package/setup.py#L38 also lists werkzeug, so it's not clear why pip installing the package would not install werkzeug too.

Sorry,
I ran configure and use TensorFlow from source, but don't build the whl, I am setup for "Setting up TensorFlow for Development".  So I use the following:
# To build with GPU support:
bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package

mkdir _python_build
cd _python_build
ln -s ../bazel-bin/tensorflow/tools/pip_package/build_pip_package.runfiles/org_tensorflow/* .
ln -s ../tensorflow/tools/pip_package/* .
python setup.py develop

I am not super familiar with that style of development: I don't know why calling the setup.py with 'develop' mode doesn't install the required packages.
You should be able to get around this by pip install werkzeug manually.

I can try the PIP but a little concerned it might cause problems later as the PIP should be bundled with the development install.
Here is the link to the "setting up with TensorFlow for Development":
https://www.tensorflow.org/get_started/os_setup#setting_up_tensorflow_for_development

If you are concerned about polluting your environment, you should run virtualenv. We need werkzeug to run.

I understand, but wanted to report the break in the current HEAD development build.  This is the first include I have needed to PIP install that appears it should have been installed with bazel.  My only concern is I will not be able to detect when it is corrected in the future.

Right, we are thinking that setup.py should have installed it, but for some reason didn't. Is it working for you now?

Yes, I will remove the PIP package once you guys determine how to move forward as you could add the package as a prerequisite.

Yes, it looks like it's a documentation issue. According to stackoverflow, the develop mode doesn't install the requirements. You have to install them separately. Closing a looping internally to update the doc.

I also encountered the same problem, but I found the werkzeug package. But still did not solve the problem

I have the same problem with TwistedW ,and still did not solve,too.

I have the same problem and pip install does not solve it.

This is strictly a TB dependency.
You should reach out to tensorboard to look into fixing this dependency in tensorboard pip package.

authors: Mazecreator TwistedW XiaoxueZang drpngx gunan phipps553 vrv


text:
Fix mnist_softmax tutorial: W should be randomly initialized, should …

Fix mnist_softmax tutorial: W should be randomly initialized, should not be initialized as zeros. Otherwise, you will get a symmetric NN.
# This is wrong and misleading. W should not initialized as zeros.
# If you do so, you will get a symmetric network.
W = tf.Variable(tf.zeros([784, 10]))

Instead, W should be randomly initialized
W = tf.Variable(tf.random_normal([784, 10]))

Actually, there is another mistake:
# This is not how we use mini-batch in practical environment
for _ in range(1000):
    batch_xs, batch_ys = mnist.train.next_batch(100)
    sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})

Instead, mini-batch should be used as follows:
for _ in range(1000):
   for batch in range(550):
      batch_xs, batch_ys = mnist.train.next_batch(100)
      sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})

But maybe for simplicity, we can ignore this mistake

Can one of the admins verify this patch?

I signed it!

@eeandrew this no-hidden-layer network, multinomial or softmax logistic regression, is convex in the parameters, so initialization to zeros is fine. Symmetry-breaking is a problem in networks with hidden layers.

In terms of not breaking the docs this looks fine to me?  (Though I don't think I'm technically supposed to approve code sample changes.)

Hi thanks for the PR.
This PR would only make sense on master, otherwise it would only be fixed in 1.4.
But I'm not sure this one is worth pursuing:

@JesseLivezey if has a good (if subtle), point.
We'll be moving and modernizing these examples shortly.

source: gharchive/pull-request

authors: JesseLivezey MarkDaoust dr4b eeandrew tensorflow-jenkins


text:
Patch absl after download to work around an nvcc bug.

The nvcc bug will be fixed in the CUDA 10.2 release.

@bas-aarts Can you please resolve conflicts? Thanks!

@mihaimaruseac , @r4nt, in the hope to avoid another merge conflict for this PR, any change to have someone review this? Thanks.

authors: bas-aarts gbaned


text:
Finalize porting S3

@mihaimaruseac
This PR adds all the missing parts from core/platform/s3. We now finish porting s3 to c_api.
Actually, We are missing a logging mechanism LOG(INFO), etc. Could we use std::cout ?

Question: Does S3 guarantee that only / can be used as a path separator and never \\?

Yes. S3 Uri will have a form of s3://<bucket>/<object>

Apologies, I missed the std::cout question. It's preferable to use logging so that users would be able to hide the logs / integrate with custom logging solutions. But since C API does not offer logging support at the moment, I am unsure what we can do.
Adding a C API wrapper around the logging macros will also be a nice addition


text:
Xla exception changes

To add new error codes to be used in Status objects arising from XLA specific code. I would like to create new custom Python exceptions according to the new error codes being used. What additional changes do I need to do to achieve this?

Can you please make this against master branch?

Closing since PRs should be against master branch

authors: mihaimaruseac vinayburugu


text:
[Linaro:ARM_CI] Revert adding install of tensorflow-io

Do not install tensorflow-io as an extra package, it is not needed and can cause additional problems and failed to achieve the original intent as it attempted to fix a problem that was actually in the building of the tensorflow-io-gcs-filesystem package for AARCH64

Probably best to wait for a new release of tensorflow-io before merging this PR

No reason to wait for new release of tensorflow-io. Failures seen in ARM_CI run were not related.


text:
watcher error with test_moreh_groupnorm

Happens on GS or WH
TT_METAL_WATCHER=60 pytest tests/tt_eager/python_api_testing/unit_testing/misc/test_moreh_groupnorm.py::test_moreh_groupnorm
TT_METAL_WATCHER=60 pytest  tests/tt_eager/python_api_testing/unit_testing/misc/test_moreh_groupnorm.py::test_moreh_groupnorm_backward

@razorback3 , do you mind help getting someone assigned to work on this?

@seunghwan100 Would you take a look?

@jliangTT

@jliangTT
One of our team's engineer says this problem might be related to #5868 and guesses that same reason with #5868.
Why don't we wait for #5868 to be resolved and then look this up again later?

being debugged in #5868

authors: TT-billteng jliangTT razorback3

repo: tenstorrent-metal/tt-metal


text:
Tile not supported in unsqueeze

The current implementation of ttnn unsqueeze does not support tiled layout. This should be generally supported.

Tile is now supported in main

repo: tenstorrent/tt-metal


text:
Llama3 Chunked Prefill

Llama3-7-B on T3K supports up to 128k context length with batch=1. However, it is limited such that the maximum prompt length is 32k. That means that the full 128k context can be used by prefill len + decode len only if prefill len <= 32k.
This constraint is caused by the fact that prefill activations for long sequences become large, leading to OOM on device DRAM. For example, the activations are 128k * 8k * 2B = 2 GB for input len 128k.
The solution to this problem is to implement chunked prefill. Given some prompt_len and a chunk_size  (<= 32k), the prompt is split into prompt_len / chunk_size chunks and iteratively prefilled. Chunked prefill will reduce the size of the activations which should solve the OOM error.
Steps
#15876

SDPA must know how to read K, V from paged cache
SDPA must know that its causal mask is offset by the number of previously prefilled chunks
New testing coverage

Add support to the old Llama3 codebase

Set chunk_size in model config
Pass chunk_id and num_chunks around
In attention, use different SDPA if doing chunking
New testing coverage

Nice!

This work uncovered a small bug in paged SDPA decode. The fix is in PR https://github.com/tenstorrent/tt-metal/pull/16059

All relevant PRs are merged. Doing final testing now on main

authors: cglagovichTT davorchap


text:
Allow TTRT Perf Mode to be invoked with a Capsule

Motivations

Unfortunately this issue acts more as a placeholder. TTRT Perf requires a subprocess of ttrt run to execute, this creates a gap where a capsule can't be used to generate a performance trace.

Planned Deltas

To support this, Tracy will probably have to be configured such that it doesn't require this subprocess.

No longer relevant due to issues with running TTRT Perf through the API (re: #1168)

repo: tenstorrent/tt-mlir


text:
Prepare repository for 2021.04.29-rc1

:notebook_with_decorative_cover: Description
This makes the necessary changes for the first release candidate for the release planned for 2012-04-29.
:memo: Checklist

[x] All user-facing changes have changelog entries.
[x] The changes are reflected on docs.tenzir.com/vast, if necessary.
[x] The PR description contains instructions for the reviewer, if necessary.

:dart: Review Instructions
Commit-by-commit.

Is there actually an automated script that you can invoke that does this work for you?
You guessed it, I'm asking for the "Prepare Release" button. 🙂

You guessed it, I'm asking for the "Prepare Release" button. 🙂

Not yet, but if you write a story for it I can always spend the hour to create that workflow.

authors: dominiklohmann mavam

repo: tenzir/vast


text:
add simple logging about HttpResponseStatus.TOO_MANY_REQUESTS and Htt…

…pResponseStatus.INTERNAL_SERVER_ERROR

partial fix for #55


text:
Bash integration

Current llvmenv zsh uses two zsh features

typeset -U: keep unique $PATH
precmd hook

Although Bash itself does not have these features. It is not so hard to realized these.

https://github.com/rcaloras/bash-preexec may help precmd hook

https://unix.stackexchange.com/questions/14895/duplicate-entries-in-path-a-problem
Duplicate $PATH can be avoided like this

repo: termoshtt/llvmenv

license: MIT


text:
🛑 Login LR is down

In 50145b2, Login LR (https://www.la-razon.com/login) was down:

HTTP code: 503
Response time: 39 ms

Resolved: Login LR is back up in 19f9cf4.

repo: terorero/monitor


text:
do the module support aws_dms_replication_config?

Is your request related to a new offering from AWS?
Is this functionality available in the AWS provider for Terraform? See CHANGELOG.md, too.

No 🛑: please wait to file a request until the functionality is avaialble in the AWS provider
Yes ✅: please list the AWS provider version which introduced this functionality

Is your request related to a problem? Please describe.
i need to enable aws_dms_replication_config to setup serverless feature
Describe the solution you'd like.
i would like to have a way to enable aws_dms_replication_config
Describe alternatives you've considered.
fork the repo and enable it
Additional context

It would be great if this is supported.

This issue has been resolved in version 2.2.0 :tada:

authors: antonbabenko roelywoely sergiovr123

repo: terraform-aws-modules/terraform-aws-dms


text:
Apply fails to replace a node group (for adding a launch template) with "NodeGroup already exists with name internal-t2_medium and cluster name development-pod01"

I have issues
I'm submitting a...

[x] bug report
[ ] feature request
[ ] support request - read the FAQ first!
[x] kudos, thank you, warm fuzzy

What is the current behavior?
Node group replacement fails with "Error: error creating EKS Node Group (clustername:nodegroupname): ResourceInUseException: NodeGroup already exists with name nodegroupname and cluster name clustername"
Terraform plan indicates that the node groups need to be replaced:
Resource actions are indicated with the following symbols:
+ create
+/- create replacement and then destroy

...

  # module.eks.module.eks.module.node_groups.aws_eks_node_group.workers["1"] must be replaced
+/- resource "aws_eks_node_group" "workers" {
...

  # module.eks.module.eks.module.node_groups.random_pet.node_groups["1"] must be replaced
+/- resource "random_pet" "node_groups" {

Application of the plan fails
module.eks.module.eks.module.node_groups.random_pet.node_groups["1"]: Creating...
module.eks.module.eks.module.node_groups.random_pet.node_groups["1"]: Creation complete after 0s [id=immortal-parrot]
module.eks.module.eks.module.node_groups.aws_eks_node_group.workers["1"]: Creating...

Error: error creating EKS Node Group (clustername:nodegroupname): ResourceInUseException: NodeGroup already exists with name nodegroupname and cluster name clustername
{
  RespMetadata: {
    StatusCode: 409,
    RequestID: "abc"
  },
  ClusterName: "clustername",
  Message_: "NodeGroup already exists with name nodegroupname and cluster name clustername",
  NodegroupName: "nodegroupname"
}

If this is a bug, how to reproduce? Please include a code sample if relevant.
I added a launch template to the definition of an existing node group by adding this:
      launch_template_id      = aws_launch_template.default.id
      launch_template_version = aws_launch_template.default.default_version

What's the expected behavior?
The node group is replaced and the new node group has a launch template.
Are you able to fix this problem and submit a PR? Link here if you have already.
Environment details

Affected module version:
terraform-aws-modules/eks/aws: 13.2.1
kubernetes/1.13.3
local/2.0.0
null/3.0.0
template/2.2.0
random/3.0.1
aws/3.24.1

OS: amazon linux 2

Terraform version: 0.14.2

Any other relevant info

This might be related to #1152

This might be related to #1152

repo: terraform-aws-modules/terraform-aws-eks


text:
Not able to run example/basic

I have issues
shailendras-mbp:basic shaikuma$ terraform init
Initializing modules...

module.vpc
Error downloading modules: Error loading modules: no versions for "terraform-aws-modules/vpc/aws" found matching "5.0.0"
shailendras-mbp:basic shaikuma$ pwd
/Users/shaikuma/Desktop/EKS/terraform-aws-eks_new/terraform-aws-eks/examples/basic

I'm submitting a...

[ ] bug report
[ ] feature request
[ ] support request
[ ] kudos, thank you, warm fuzzy

What is the current behavior?
When I am initializing examples/baisc, it's not able to download terraform-aws-modules/vpc/aws I noticed there is no directory inside terraform-aws-modules with vpc/aws name
If this is a bug, how to reproduce? Please include a code sample if relevant.
What's the expected behavior?
Are you able to fix this problem and submit a PR? Link here if you have already.
Environment details

Affected module version:
OS:
Terraform version:

Any other relevant info

I thought the purpose of these examples are just to show the developer how to use the module and not to really work...

Did you modify the example in any way? It looks like you've modified the VPC module import to use version = 5.0.0 which does not exist. It should be 2.something. The example currently has 2.6.0.
terraform init works for the example/basic in v5.0.0 and master:
$ terraform init
Initializing modules...
- eks in ../..
Downloading terraform-aws-modules/vpc/aws 2.6.0 for vpc...
- vpc in .terraform/modules/vpc/terraform-aws-modules-terraform-aws-vpc-4b28d3d

Initializing the backend...

with 2.6.0 I am getting different type of error:
shailendras-mbp:basic shaikuma$ pwd
/Users/shaikuma/Desktop/EKS/terraform-aws-eks_new/terraform-aws-eks/examples/basic
shailendras-mbp:basic shaikuma$

updated terraform from 0.11.8 to  0.12.3 version. Now it's working.

Cool. Glad you got it working.

authors: confiq dpiddockcmp kshailen max-rocket-internet


text:
worker nodes unable to connect to the cluster

I'm submitting a...

[ ] bug report
[ ] feature request
[ * ] support request
[ ] kudos, thank you, warm fuzzy

What is the current behavior?
I am having issues getting the worker nodes to connect to my eks cluster. when i logged in to the worker node i am getting the error below. This is also using a public subnet from my vpc. aws_auth gets created with no issues and ec2 instances are also using the right instance profile. the version im using is Version":"5.0.0"
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.149994    4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.250192    4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.350403    4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.450605    4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.550837    4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.651064    4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.751273    4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.770595    4420 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Unauthorized
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.851962    4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.900817    4420 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:453: Failed to list *v1.Node: Unauthorized
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.900859    4420 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Unauthorized
Jun 30 03:08:19 ip-10-100-5-7 kubelet: E0630 03:08:19.952483    4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.052714    4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.152924    4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.253136    4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.353369    4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.453600    4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.553818    4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.654032    4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.754252    4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.854445    4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.897784    4420 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:444: Failed to list *v1.Service: Unauthorized
Jun 30 03:08:20 ip-10-100-5-7 kubelet: E0630 03:08:20.954949    4420 kubelet.go:2266] node "ip-10-100-5-7.us-west-2.compute.internal" not found
If this is a bug, how to reproduce? Please include a code sample if relevant.
Here is my main.tf
variable "vpc_id" {
default = "vpc-001670fb"
}
variable "subnets" {
default = ["subnet-0a4c92b3", "subnet-0e414456"]
type    = list(string)
}
variable "vpc_env_name" {
default = "production"
}
variable "cluster_version" {
default = "1.13"
}
variable "cluster_name" {
default = "eks-cluster-us-west-2-01"
}
variable "office_sg" {
default = ["sg-0a92c8b"]
type    = list(string)
}
variable "map_roles" {
description = "Additional IAM roles to add to the aws-auth configmap."
type        = list(map(string))
default = [
{
role_arn = "arn:aws:iam:::role/devops"
username = "devops"
group    = "system:masters"
},
]
}
provider "aws" {
region  = "us-west-2"
profile = "sa-terraform"
}
terraform {
backend "s3" {
bucket  = "eks-cluster-operations-terraform"
key     = "eks-clusters-terraform-module/eks-cluster-us-west-2-01"
region  = "us-west-2"
profile = "sa-terraform"
}
}
module "eks-clusters-terraform-module" {
source                               = "terraform-aws-modules/eks/aws"
cluster_name                         = var.cluster_name
cluster_version                      = "1.13"
subnets                              = var.subnets
vpc_id                               = var.vpc_id
worker_additional_security_group_ids = var.office_sg
manage_aws_auth                      = "true"
map_roles                            = var.map_roles
kubeconfig_aws_authenticator_env_variables = {
AWS_PROFILE = "devops"
}
tags = {
Environment = "shared"
}
worker_groups = [
{
name = "generalPurpose_2a"
instance_type = "t2.medium"
asg_min_size = 1
asg_max_size  = 2
autoscaling_enabled = true
protect_scale_in = true
asg_desired_capacity = 1
key_name = ""
},
{
name = "generalPurpose_2b"
instance_type = "t2.medium"
asg_min_size = 1
asg_max_size  = 2
autoscaling_enabled = true
protect_scale_in = true
asg_desired_capacity = 1
key_name = ""
}
]
}
What's the expected behavior?
Are you able to fix this problem and submit a PR? Link here if you have already.
Environment details

Affected module version:
OS: OSX
Terraform version:
Terraform v0.12.3

provider.aws v2.17.0
provider.local v1.3.0
provider.null v2.1.2
provider.template v2.1.2

Any other relevant info

Updating with the AMI i am using: amazon-eks-node-1.13-v20190614

I think this is your problem:  Failed to list *v1.Pod: Unauthorized
Check that your aws-auth configmap was applied and has the correct rolearn

@max-rocket-internet config map looks good to me
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
data:
mapRoles: |
- rolearn: arn:aws:iam:::role/eks-cluster-us-west-2-0120190701201308460600000006
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
- rolearn: arn:aws:iam:::role/devops
  username: devops
  groups:
    - system:masters

mapUsers: |
mapAccounts: |

i also checked the kubeconfig inside the worker nodes seems right.
apiVersion: v1
clusters:

cluster:
certificate-authority: /etc/kubernetes/pki/ca.crt
server: https://140922E6BD597ABCDEC9E72A3CE9CE.yl4.us-west-2.eks.amazonaws.com
name: kubernetes
contexts:
context:
cluster: kubernetes
user: kubelet
name: kubelet
current-context: kubelet
kind: Config
preferences: {}
users:
name: kubelet
user:
exec:
apiVersion: client.authentication.k8s.io/v1alpha1
args:
- token
- -i
- eks-cluster-us-west-2-01
command: /usr/bin/aws-iam-authenticator
env: null

I tried running this on public subnets now but same issue. seems like
Jul  1 21:03:15 ip-10-72-171-192 kubelet: I0701 21:03:15.275924    4431 kubelet_node_status.go:279] Setting node annotation to enable volume controller attach/detach
Jul  1 21:03:15 ip-10-72-171-192 kubelet: I0701 21:03:15.276524    4431 kubelet_node_status.go:327] Adding node label from cloud provider: beta.kubernetes.io/instance-type=t2.medium
Jul  1 21:03:15 ip-10-72-171-192 kubelet: I0701 21:03:15.276873    4431 kubelet_node_status.go:338] Adding node label from cloud provider: failure-domain.beta.kubernetes.io/zone=us-west-2b
Jul  1 21:03:15 ip-10-72-171-192 kubelet: I0701 21:03:15.277221    4431 kubelet_node_status.go:342] Adding node label from cloud provider: failure-domain.beta.kubernetes.io/region=us-west-2

i got this figured out. looks like the aws_auth configmap is not applying automatically, I had to apply it manually in order for the workers to join the cluster. is this a bug?

Same here, but still haven't figured out how to re-apply configmap/aws-auth so the EC2 instance can join...
The command below did not work:
kubectl get -n kube-system configmap/aws-auth -o yaml | kubectl replace -f -

@davi5e are you connected to the cluster? what is the output of your kubectl cluster-info

$ kubectl cluster-info
Kubernetes master is running at https://FD682B64C213C073209FDA092F6CCBCC.gr7.us-east-1.eks.amazonaws.com
CoreDNS is running at https://FD682B64C213C073209FDA092F6CCBCC.gr7.us-east-1.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

So, I figured out what was causing my problem, but I don't know why...
Once I compared the EKS basic example with the VPC module I used as a template I noticed that EC2 endpoints were a really bad idea and now the nodes join just fine.
In the VPC (using terraform-aws-modules/vpc/aws v2.7.0) I just removed the following:
  # VPC Endpoint for EC2
  enable_ec2_endpoint              = false
  ec2_endpoint_private_dns_enabled = false
  ec2_endpoint_security_group_ids  = []

So this may be circumstantial but I found that when following the basic example, the created VPC lacks enable_dns_hostnames = true, which, to my understanding makes it impossible to resolve hostnames like ip-10-100-5-7.us-west-2.compute.internal. AWS docs on it are somewhat muddy (as usual) but I think this setting enables VPC-internal resolution of assigned private and public IPs. Enabling it seems to work with some delay - I was able to dig/ssh to a worker node from a bastion host in the same VPC but a public subnet, using its hostname rather than the IP.
While I was able to resolve a worker's hostname from the bastion, kubernetes continued to fail to do so - same error in logs, which went like:
Jul 24 11:18:46 ip-10-0-1-36.eu-west-1.compute.internal kubelet[4355]: E0724 11:18:46.602539    4355 kubelet.go:2236] node "ip-10-0-1-36.eu-west-1.compute.internal" not found
Jul 24 11:18:46 ip-10-0-1-36.eu-west-1.compute.internal kubelet[4355]: E0724 11:18:46.682540    4355 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:442: Failed to list *v1.Service: Unauthorized
Jul 24 11:18:46 ip-10-0-1-36.eu-west-1.compute.internal kubelet[4355]: E0724 11:18:46.703201    4355 kubelet.go:2236] node "ip-10-0-1-36.eu-west-1.compute.internal" not found
Jul 24 11:18:46 ip-10-0-1-36.eu-west-1.compute.internal kubelet[4355]: E0724 11:18:46.803935    4355 kubelet.go:2236] node "ip-10-0-1-36.eu-west-1.compute.internal" not found
Jul 24 11:18:46 ip-10-0-1-36.eu-west-1.compute.internal kubelet[4355]: E0724 11:18:46.900234    4355 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Unauthorized
Jul 24 11:18:46 ip-10-0-1-36.eu-west-1.compute.internal kubelet[4355]: E0724 11:18:46.900771    4355 reflector.go:125] k8s.io/kubernetes/pkg/kubelet/kubelet.go:451: Failed to list *v1.Node: Unauthorized

however after correcting and applying terraform (i previously downscaled the workers and enabled DNS resolution in the VPC manually), it updated the workers launch template and the cluster, after which nodes started being discovered.
There are 2 possibilities then:

Not enabling DNS resolution may break node registration (yesterday out of my 2 attempts it worked once and didn't work once, today it failed once, then i turned DNS resolution on)
DNS resolution may have nothing to do with it and I just got lucky when doing update-in-place with TF today. However, when I destroyed and recreated the cluster, it found the nodes immediately.

I cannot run many experiments like that because the full create-destroy cycle takes around 25 minutes.
Seriously, if you can, just use K8s on GCP.

So to keep it short, one actionable item from my comment above is the question:
Why is DNS resolution not enabled by default in the example? Could this interfere with node discovery or cluster configuration?

Why is DNS resolution not enabled by default in the example?

It is now: https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/examples/basic/main.tf#L96
Will close this as there's been no updates. Also issues just like this have been covered in detail so I think there's already enough info available.

authors: davi5e karolinepauls max-rocket-internet opencatersf renperez


text:
chore(deps): update common-dev-assets digest to d9a2d01

This PR contains the following updates:

Package
Update
Change

common-dev-assets
digest
b1e90d1 -> d9a2d01

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline


text:
chore(deps): update ci dependencies

This PR contains the following updates:

Package
Type
Update
Change

common-dev-assets

digest
c00572c -> b4db5fd

github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper
require
minor
v1.33.2 -> v1.34.1

Release Notes

terraform-ibm-modules/ibmcloud-terratest-wrapper (github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper)
v1.34.1
Compare Source
Bug Fixes

Fixed bug where Projects deploy fail reason was not being output (#819) (ede430e)

v1.34.0
Compare Source
Features

Projects parallel deploy and cleanup (#818) (cc9aa18)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

/run pipeline

:tada: This PR is included in version 1.1.3 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


text:
chore(deps): update ci dependencies

This PR contains the following updates:

Package
Type
Update
Change

github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper
require
minor
v1.10.19 -> v1.11.3

terraform-ibm-modules/common-pipeline-assets
action
patch
v1.17.0 -> v1.17.4

Release Notes

terraform-ibm-modules/ibmcloud-terratest-wrapper (github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper)
v1.11.3
Compare Source
Bug Fixes

bug setting the pr dir (#617) (00131bb)

v1.11.2
Compare Source
Bug Fixes

lock into vpc-go-sdk v0.41.0 (#616) (ef9cacf)

v1.11.1
Compare Source
Bug Fixes

add logging (#615) (beb86f4)

v1.11.0
Compare Source
Features

addition of configurable base repo or branch for upgrade test (4fd1e34)

terraform-ibm-modules/common-pipeline-assets (terraform-ibm-modules/common-pipeline-assets)
v1.17.4
Compare Source
Bug Fixes

remove collab check - no longer required (#523) (be1c5de)

v1.17.3
Compare Source
Bug Fixes

Debug logging (#522) (0fe31b8)

v1.17.2
Compare Source
Bug Fixes

bug with trim (60c8a95)

v1.17.1
Compare Source
Bug Fixes

add debug and trim comment (#520) (e0bb379)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

/run pipeline

:tada: This PR is included in version 1.2.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


text:
chore(deps): update module github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper to v1.4.1

This PR contains the following updates:

Package
Type
Update
Change

github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper
require
patch
v1.4.0 -> v1.4.1

Release Notes

terraform-ibm-modules/ibmcloud-terratest-wrapper
v1.4.1
Compare Source
Bug Fixes

deps: update gomod (#324) (f7e090e)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, click this checkbox.

This PR has been generated by Renovate Bot.

:tada: This PR is included in version 2.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


text:
chore(deps): update common-dev-assets digest to ec50ee2

This PR contains the following updates:

Package
Update
Change

common-dev-assets
digest
31297ec -> ec50ee2

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, click this checkbox.

This PR has been generated by Renovate Bot.

:tada: This PR is included in version 3.1.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


text:
chore(deps): update common-dev-assets digest to 1d3b343

This PR contains the following updates:

Package
Update
Change

common-dev-assets
digest
23a510d -> 1d3b343

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

:tada: This PR is included in version 4.13.4 :tada:
The release is available on:

GitHub release
v4.13.4

Your semantic-release bot :package::rocket:


text:
chore(deps): update module github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper to v1.10.7

This PR contains the following updates:

Package
Type
Update
Change

github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper
require
patch
v1.10.6 -> v1.10.7

Release Notes

terraform-ibm-modules/ibmcloud-terratest-wrapper
v1.10.7
Compare Source
Bug Fixes

deps: update gomod (#572) (93b5090)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

:tada: This PR is included in version 7.3.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


text:
chore(deps): update common-dev-assets digest to 3d1a2f8

This PR contains the following updates:

Package
Update
Change

common-dev-assets
digest
247369d -> 3d1a2f8

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

:tada: This PR is included in version 1.0.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket: