text:
tf.keras SparseCategoricalCrossentropy with sample_weight on TPU: Error DenseToDenseSetOperation (No registered 'DenseToDenseSetOperation)

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04(Colab)
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: No
TensorFlow installed from (source or binary):
TensorFlow version (use command below): TF-2.3.0
Colab TPU: Yes

Objective: I am trying to using SparseCategoricalCrossentropy with sample_weight argument on TPU to train Mask Language Model.
Note: Connecting TPU with tf.distribute.cluster_resolver.TPUClusterResolver()
Loss function Looks like this:
def masked_sparse_categorical_crossentropy(y_true, y_pred, sample_weight):

  loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(
    reduction=tf.keras.losses.Reduction.NONE,
      from_logits=True
  )
  return loss_fn(y_true, y_pred, sample_weight=sample_weight)

When I'm running on CPU and GPU the code runs fine. But when I am running on TPU, I am getting an error on XLA_TPU_JIT: DenseToDenseSetOperation (No registered 'DenseToDenseSetOperation'
(0) Invalid argument: {{function_node __inference_train_function_196061}} Compilation failure: Detected unsupported operations when trying to compile graph broadcast_weights_assert_broadcastable_is_valid_shape_has_valid_nonscalar_shape_true_195924_const_0[] on XLA_TPU_JIT: DenseToDenseSetOperation (No registered 'DenseToDenseSetOperation' OpKernel for XLA_TPU_JIT devices compatible with node {{node broadcast_weights/assert_broadcastable/is_valid_shape/has_valid_nonscalar_shape/has_invalid_dims/DenseToDenseSetOperation}}){{node broadcast_weights/assert_broadcastable/is_valid_shape/has_valid_nonscalar_shape/has_invalid_dims/DenseToDenseSetOperation}}
	 [[broadcast_weights/assert_broadcastable/is_valid_shape/has_valid_nonscalar_shape]]
	 [[broadcast_weights/assert_broadcastable/is_valid_shape]]
	TPU compilation failed
	 [[tpu_compile_succeeded_assert/_18335038187613310273/_6]]
	 [[tpu_compile_succeeded_assert/_18335038187613310273/_6/_265]]
  (1) Invalid argument: {{function_node __inference_train_function_196061}} Compilation failure: Detected unsupported operations when trying to compile graph broadcast_weights_assert_broadcastable_is_valid_shape_has_valid_nonscalar_shape_true_195924_const_0[] on XLA_TPU_JIT: DenseToDenseSetOperation (No registered 'DenseToDenseSetOperation' OpKernel for XLA_TPU_JIT devices compatible with node {{node broadcast_weights/assert_broadcastable/is_valid_shape/has_valid_nonscalar_shape/has_invalid_dims/DenseToDenseSetOperation}}){{node broadcast_weights/assert_broadcastable/is_valid_shape/has_valid_nonscalar_shape/has_invalid_dims/DenseToDenseSetOperation}}
	 [[broadcast_weights/assert_broadcastable/is_valid_shape/has_valid_nonscalar_shape]]
	 [[broadcast_weights/assert_broadcastable/is_valid_shape]]
	TPU compilation failed
	 [[tpu_compile_succeeded_assert/_18335038187613310273/_6]]
	 [[tpu_compile_succeeded_assert/_18335038187613310273/_6/_237]]
  (2) Invalid argument: {{function_node __inference_train_function_196061}} Compilation failure: Detected unsupported operations when trying to compile graph broadcast_weights_assert_broadcastable_is_valid_shape_has_valid_nonscalar_shape_true_195924_const_0[] on XLA_TPU_JIT: DenseToDenseSetOperation (No registered 'DenseToDenseSetOperation' OpKernel for XLA_TPU_JIT devices compatible with node {{node broadcast_weights/assert_broadcastable/is_valid_shape/has_valid_nonscalar_shape/has_invalid_dims/DenseToDenseSetOperation}}){{node broadcast_weights/assert_broadcastable/is_valid_shape/has_valid_nonscalar_shape/has_invalid_dims/DenseToDenseSetOperation}}
	 [[broadcast_weights/assert_broadcastable/is_valid_shape/has_valid_nonscalar_shape]]
	 [[broadcast_weights/assert_broadcastable/is_valid_shape]]
	TPU compilation failed
	 [[tpu_compile_succeeded_assert/_18335038187613310273/_6]]
	 [[tpu_compile_succeeded_assert/_18335038187613310273/_6/_251]]
  (3) Invalid argument: {{function_node __inference_train_function_196061}} Compilation failure: Detected unsupported operations when trying to compile graph broadcast_weights_assert_broa ... [truncated]

Please help me to resolve this.
Regards,
Ankur

@Ankur3107
I ran the code on colab tf 2.3 and nighly TPU and do not face any error, please find the gist here.
Is the code provided complete to replicate issue reported or please share complete code or colab link with error reported.

@Saduf2019
Thank you for your immediate reply. My complete code with an error is on https://colab.research.google.com/drive/1HYASbm7hqrL-bYZxXE7FPiWNNeOon48z?usp=sharing .
Please find the link.

I am able to replicate this issue, please find the gist here

I also hit the same issue. My model has 2 outputs and each has its own sample_weights. I am using the sample_weights as masks (i.e. they are 0 or 1). The setup ran fine on GPU.
Does it look like there's a workaround?

Seems that class weights are not supported with TPUs this way. I am linking to two Kaggle threads that could help with finding a workaround.
Flower Classification with TPUs
ALASKA2 Image Steganalysis

It may be related to an issue @tf-marissaw is working on.
Meanwhile, can you check if the problem goes away if you set drop_remainder=True when calling dataset.batch?

I have merged a fix into TensorFlow nightly and it should appear in the TensorFlow 2.5 release.
Can you try running your model with TF nightly to see if the fix resolved this issue for you?

@tf-marissaw,
I ran the above model with Tensorflow Nightly (2.5.0-dev20201211) and it is resulting in the error,
NotFoundError: 'ComputeBatchSize' is neither a type of a primitive operation nor a name of a function registered in binary running on n-dda18766-w-0. Make sure the operation or function is registered in the binary running in this process.
Please find the Gist. Thanks!

I'm able to run your code in Tensorflow 2.7 and the latest transformers version without any error.
Here is the gist for reference.
Feel free to reopen the issue if you still face an error. Thanks!

authors: Ankur3107 Saduf2019 kechan nikitamaia rmothukuru rxsang sachinprasadhs tf-marissaw


text:
model doesn't learn when using shuffle='batch'

Please make sure that this is a bug. As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 20.04
TensorFlow installed from (source or binary): pip install tf-nightly-gpu==2.5.0.dev20201108
TensorFlow version (use command below): 2.5.0
Python version:3.8
GCC/Compiler version (if compiling from source):9.3.0
CUDA/cuDNN version:11.1.1/8.0.5
GPU model and memory:RTX 3090 (24GMiB) driver 455.38

Describe the current behavior
Becasue I train my model with data from HDF5 files, so I used model.fit(, , , shuffle='batch').
https://www.tensorflow.org/api_docs/python/tf/keras/Sequential:
"shuffle | Boolean (whether to shuffle the training data before each epoch) or str (for 'batch'). This argument is ignored when x is a generator. 'batch' is a special option for dealing with the limitations of HDF5 data; it shuffles in batch-sized chunks. Has no effect when steps_per_epoch is not None."
Before I upgrade my hardware, I used tensorflow 1.x as Keras 2.x's backend. My model used shuffle='batch' without any problems. Now, I have a new machine, so I need to transfer my codes. However, the new code doesn't work anymore.
Describe the expected behavior
I used MNIST dataset to show what happended: Code from (https://www.machinecurve.com/index.php/2020/04/13/how-to-use-h5py-and-keras-to-train-with-data-from-hdf5-files/)
Standalone code to reproduce the issue
import h5py
from tensorflow.keras.datasets import cifar10
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Flatten, Conv2D
from tensorflow.keras.losses import sparse_categorical_crossentropy
from tensorflow.keras.optimizers import Adam

# Model configuration
batch_size = 50
img_width, img_height, img_num_channels = 28, 28, 1
loss_function = sparse_categorical_crossentropy
no_classes = 10
no_epochs = 25
optimizer = Adam()
validation_split = 0.2
verbosity = 1

# Load MNIST data
f = h5py.File('train.hdf5', 'r')
input_train = f['image'][...]
label_train = f['label'][...]
f.close()
f = h5py.File('test.hdf5', 'r')
input_test = f['image'][...]
label_test = f['label'][...]
f.close()

# Reshape data
input_train = input_train.reshape((len(input_train), img_width, img_height, img_num_channels))
input_test  = input_test.reshape((len(input_test), img_width, img_height, img_num_channels))

# Determine shape of the data
input_shape = (img_width, img_height, img_num_channels)

# Create the model
model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=input_shape))
model.add(Conv2D(64, kernel_size=(3, 3), activation='relu'))
model.add(Conv2D(128, kernel_size=(3, 3), activation='relu'))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dense(no_classes, activation='softmax'))

# Display a model summary
model.summary()

# Compile the model
model.compile(loss=loss_function,
              optimizer=optimizer,
              metrics=['accuracy'])

# Fit data to model
history = model.fit(input_train, label_train,
            batch_size=batch_size,
            epochs=no_epochs,
            verbose=verbosity,shuffle='batch',
            validation_split=validation_split)

# Generate generalization metrics
score = model.evaluate(input_test, label_test, verbose=0)
print(f'Test loss: {score[0]} / Test accuracy: {score[1]}')

**Other info / logs** Include any logs or source code that would be helpful to
The output is like this
Model: "sequential_3"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_9 (Conv2D)            (None, 26, 26, 32)        320
_________________________________________________________________
conv2d_10 (Conv2D)           (None, 24, 24, 64)        18496
_________________________________________________________________
conv2d_11 (Conv2D)           (None, 22, 22, 128)       73856
_________________________________________________________________
flatten_3 (Flatten)          (None, 61952)             0
_________________________________________________________________
dense_6 (Dense)              (None, 128)               7929984
_________________________________________________________________
dense_7 (Dense)              (None, 10)                1290
=================================================================
Total params: 8,023,946
Trainable params: 8,023,946
Non-trainable params: 0
_________________________________________________________________
Epoch 1/25
960/960 [==============================] - 3s 3ms/step - loss: 6.4279 - accuracy: 0.1099 - val_loss: 2.3022 - val_accuracy: 0.1060
Epoch 2/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3012 - accuracy: 0.1141 - val_loss: 2.3012 - val_accuracy: 0.1060
Epoch 3/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3011 - accuracy: 0.1149 - val_loss: 2.3020 - val_accuracy: 0.1060
Epoch 4/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3010 - accuracy: 0.1142 - val_loss: 2.3021 - val_accuracy: 0.1060
Epoch 5/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3010 - accuracy: 0.1141 - val_loss: 2.3019 - val_accuracy: 0.1060
Epoch 6/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3010 - accuracy: 0.1162 - val_loss: 2.3019 - val_accuracy: 0.1060
Epoch 7/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3012 - accuracy: 0.1139 - val_loss: 2.3020 - val_accuracy: 0.1060
Epoch 8/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3013 - accuracy: 0.1128 - val_loss: 2.3025 - val_accuracy: 0.1060
Epoch 9/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3013 - accuracy: 0.1131 - val_loss: 2.3020 - val_accuracy: 0.1060
Epoch 10/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3011 - accuracy: 0.1156 - val_loss: 2.3021 - val_accuracy: 0.1060
Epoch 11/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3013 - accuracy: 0.1127 - val_loss: 2.3022 - val_accuracy: 0.1060
Epoch 12/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3010 - accuracy: 0.1143 - val_loss: 2.3024 - val_accuracy: 0.1060
Epoch 13/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3012 - accuracy: 0.1131 - val_loss: 2.3025 - val_accuracy: 0.1060
Epoch 14/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3009 - accuracy: 0.1148 - val_loss: 2.3019 - val_accuracy: 0.1060
Epoch 15/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3012 - accuracy: 0.1152 - val_loss: 2.3019 - val_accuracy: 0.1060
Epoch 16/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3009 - accuracy: 0.1149 - val_loss: 2.3020 - val_accuracy: 0.1060
Epoch 17/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3009 - accuracy: 0.1143 - val_loss: 2.3020 - val_accuracy: 0.1060
Epoch 18/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3014 - accuracy: 0.1125 - val_loss: 2.3022 - val_accuracy: 0.1060
Epoch 19/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3011 - accuracy: 0.1147 - val_loss: 2.3019 - val_accuracy: 0.1060
Epoch 20/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3011 - accuracy: 0.1144 - val_loss: 2.3020 - val_accuracy: 0.1060
Epoch 21/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3013 - accuracy: 0.1128 - val_loss: 2.3022 - val_accuracy: 0.1060
Epoch 22/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3012 - accuracy: 0.1122 - val_loss: 2.3024 - val_accuracy: 0.1060
Epoch 23/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3003 - accuracy: 0.1163 - val_loss: 2.3021 - val_accuracy: 0.1060
Epoch 24/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3011 - accuracy: 0.1151 - val_loss: 2.3021 - val_accuracy: 0.1060
Epoch 25/25
960/960 [==============================] - 3s 3ms/step - loss: 2.3012 - accuracy: 0.1131 - val_loss: 2.3021 - val_accuracy: 0.1060
Test loss: 2.3010358810424805 / Test accuracy: 0.11349999904632568

If I changed shuffle='batch' to shuffle=True or shuffle=False
I got convergent results like this

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d (Conv2D)              (None, 26, 26, 32)        320
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 24, 24, 64)        18496
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 22, 22, 128)       73856
_________________________________________________________________
flatten (Flatten)            (None, 61952)             0
_________________________________________________________________
dense (Dense)                (None, 128)               7929984
_________________________________________________________________
dense_1 (Dense)              (None, 10)                1290
=================================================================
Total params: 8,023,946
Trainable params: 8,023,946
Non-trainable params: 0
_________________________________________________________________
Epoch 1/25
960/960 [==============================] - 5s 3ms/step - loss: 2.3020 - accuracy: 0.9032 - val_loss: 0.0738 - val_accuracy: 0.9786
Epoch 2/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0502 - accuracy: 0.9853 - val_loss: 0.0621 - val_accuracy: 0.9824
Epoch 3/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0333 - accuracy: 0.9896 - val_loss: 0.0811 - val_accuracy: 0.9792
Epoch 4/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0216 - accuracy: 0.9936 - val_loss: 0.0851 - val_accuracy: 0.9805
Epoch 5/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0244 - accuracy: 0.9922 - val_loss: 0.0757 - val_accuracy: 0.9832
Epoch 6/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0139 - accuracy: 0.9956 - val_loss: 0.1344 - val_accuracy: 0.9752
Epoch 7/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0202 - accuracy: 0.9935 - val_loss: 0.1379 - val_accuracy: 0.9779
Epoch 8/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0141 - accuracy: 0.9956 - val_loss: 0.0919 - val_accuracy: 0.9818
Epoch 9/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0125 - accuracy: 0.9962 - val_loss: 0.1184 - val_accuracy: 0.9811
Epoch 10/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0154 - accuracy: 0.9956 - val_loss: 0.1157 - val_accuracy: 0.9832
Epoch 11/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0176 - accuracy: 0.9952 - val_loss: 0.1221 - val_accuracy: 0.9803
Epoch 12/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0101 - accuracy: 0.9976 - val_loss: 0.1170 - val_accuracy: 0.9822
Epoch 13/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0124 - accuracy: 0.9969 - val_loss: 0.1216 - val_accuracy: 0.9846
Epoch 14/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0094 - accuracy: 0.9974 - val_loss: 0.1048 - val_accuracy: 0.9848
Epoch 15/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0067 - accuracy: 0.9982 - val_loss: 0.1130 - val_accuracy: 0.9835
Epoch 16/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0122 - accuracy: 0.9974 - val_loss: 0.1463 - val_accuracy: 0.9835
Epoch 17/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0091 - accuracy: 0.9976 - val_loss: 0.1685 - val_accuracy: 0.9833
Epoch 18/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0110 - accuracy: 0.9977 - val_loss: 0.1224 - val_accuracy: 0.9840
Epoch 19/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0036 - accuracy: 0.9989 - val_loss: 0.1733 - val_accuracy: 0.9838
Epoch 20/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0109 - accuracy: 0.9978 - val_loss: 0.1539 - val_accuracy: 0.9859
Epoch 21/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0074 - accuracy: 0.9982 - val_loss: 0.1791 - val_accuracy: 0.9826
Epoch 22/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0085 - accuracy: 0.9986 - val_loss: 0.2264 - val_accuracy: 0.9830
Epoch 23/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0124 - accuracy: 0.9979 - val_loss: 0.1722 - val_accuracy: 0.9840
Epoch 24/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0089 - accuracy: 0.9984 - val_loss: 0.1472 - val_accuracy: 0.9851
Epoch 25/25
960/960 [==============================] - 3s 3ms/step - loss: 0.0048 - accuracy: 0.9988 - val_loss: 0.2005 - val_accuracy: 0.9847
Test loss: 0.18761441111564636 / Test accuracy: 0.9829999804496765

@xuzhang5788
I ran the code shared and face a different error, please refer to this gist.
Please share a colab gist with the error reported.

I rerun the code, colab's link is here:
https://colab.research.google.com/gist/xuzhang5788/269ada412449fe2007ae0460aa329f47/untitled471.ipynb

https://colab.research.google.com/gist/Saduf2019/5c932f4aeccde525d60a87eb3959def0/untitled471.ipynb

The input data is from Kaggle (https://www.kaggle.com/benedictwilkinsai/mnist-hd5f).

I reran it with tensorflow 2.3.0. I got the same results. However, the training process was far faster than tensorflow 2.5.0.
please see the link
https://colab.research.google.com/gist/xuzhang5788/269ada412449fe2007ae0460aa329f47/untitled471.ipynb

I rerun my code with tensorflow 2.3.0, but I got the same results. please see the link:
https://colab.research.google.com/gist/xuzhang5788/c5ce687ff4b7e26d9b26b7f67c30439c/untitled471.ipynb
However, the training process is far faster than tensorflow 2.5.0
960/960 [==============================] - 8s 9ms/step - loss: 2.3011 - accuracy: 0.1140 - val_loss: 2.3022 - val_accuracy: 0.1060
Epoch 20/25
960/960 [==============================] - 8s 8ms/step - loss: 2.3011 - accuracy: 0.1140 - val_loss: 2.3021 - val_accuracy: 0.1060
Epoch 21/25
960/960 [==============================] - 8s 8ms/step - loss: 2.3011 - accuracy: 0.1140 - val_loss: 2.3021 - val_accuracy: 0.1060
Epoch 22/25
960/960 [==============================] - 8s 8ms/step - loss: 2.3011 - accuracy: 0.1140 - val_loss: 2.3022 - val_accuracy: 0.1060
Epoch 23/25
960/960 [==============================] - 8s 8ms/step - loss: 2.3011 - accuracy: 0.1140 - val_loss: 2.3021 - val_accuracy: 0.1060
Epoch 24/25
960/960 [==============================] - 8s 8ms/step - loss: 2.3011 - accuracy: 0.1140 - val_loss: 2.3022 - val_accuracy: 0.1060
Epoch 25/25
960/960 [==============================] - 8s 9ms/step - loss: 2.3011 - accuracy: 0.1140 - val_loss: 2.3021 - val_accuracy: 0.1060

@xuzhang5788
I am unable to access the "train.hdf5" file, please refer to this gist here.

@Saduf2019
I am sorry that I don't know how to let you access train.hdf5 file. Could you please download the files from kaggle? Link is here.
The input data is from Kaggle (https://www.kaggle.com/benedictwilkinsai/mnist-hd5f).

@Saduf2019
I tried tensorflow-gpu 1.15, it is okay. The output is like this:
Epoch 19/25
48000/48000 [==============================] - 10s 201us/sample - loss: 0.0111 - acc: 0.9974 - val_loss: 0.1634 - val_acc: 0.9837
Epoch 20/25
48000/48000 [==============================] - 10s 201us/sample - loss: 0.0089 - acc: 0.9981 - val_loss: 0.1252 - val_acc: 0.9866
Epoch 21/25
48000/48000 [==============================] - 10s 202us/sample - loss: 0.0092 - acc: 0.9982 - val_loss: 0.1667 - val_acc: 0.9844
Epoch 22/25
48000/48000 [==============================] - 10s 202us/sample - loss: 0.0041 - acc: 0.9991 - val_loss: 0.1959 - val_acc: 0.9822
Epoch 23/25
48000/48000 [==============================] - 10s 203us/sample - loss: 0.0105 - acc: 0.9983 - val_loss: 0.1459 - val_acc: 0.9859
Epoch 24/25
48000/48000 [==============================] - 10s 202us/sample - loss: 0.0130 - acc: 0.9980 - val_loss: 0.1898 - val_acc: 0.9840
Epoch 25/25
48000/48000 [==============================] - 10s 203us/sample - loss: 0.0065 - acc: 0.9986 - val_loss: 0.2155 - val_acc: 0.9848
I think the bugs are only in tensorflow 2.0+

I tried different versions of tensorflow.
works version: 1.15.0, 2.0.0
not works version: 2.1.0, 2.2.0, 2.3.0, 2.5.0
So the bugs appeared after version 2.0.0. please fix it!!

I tried your example with shuffle='batch', False, True options and found that model is not converging for either of them. See the gist
I noticed that shuffle arg takes any string other than 'batch' right now which is confusing.
I will dig deeper and refactor it.

@ymodak
I think there are critical bugs when keras became tensorflow 2's API. Because many people's datasets are too large to fit into memory, so they chose to use hdf5 files. I don't know when it can be fixed. It is been here for almost one month, but I didn't get any response for this.

Lower the learning rate?

Lower the learning rate?

@tensorflowbutler @ymodak @Saduf2019
The problems are still there. When will they be solved?

This issue still exists in Tensorflow 2.7, could you please open this issue in keras-team/keras repo.
To know more see;
https://discuss.tensorflow.org/t/keras-project-moved-to-new-repository-in-https-github-com-keras-team-keras/1999
Thank you!

Closing the issue here since it is moved to keras-team/keras repo, you can track the issue in the above link.

source: gharchive/issue

repo: tensorflow/tensorflow

license_type: permissive

license_source: github-api


text:
Tensorflow 2.3.0 MKL Intel AVX Binary Issue

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): custom code
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): tensorflow-mkl 2.3.0
TensorFlow version (use command below): 2.3.0
Python version: 3.7.9
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:

You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with:

TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

2.3.0
Describe the current behavior
Here's the code:

import tensorflow as tf
import numpy as np
a = np.array([2 , 4, 5])
ap=tf.constant(a)

Here's the warning message:

" I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags."

Moreover, if I run system check mentioned on Intel's website (https://software.intel.com/content/www/us/en/develop/articles/intel-optimization-for-tensorflow-installation-guide.html),
Code:
import tensorflow as tf
major_version = int(tf.__version__.split(".")[0])
if major_version >= 2:
   from tensorflow.python import _pywrap_util_port
   print("MKL enabled:", _pywrap_util_port.IsMklEnabled())
else:
   print("MKL enabled:", tf.pywrap_tensorflow.IsMklEnabled())

I get

MKL enabled: False

Describe the expected behavior
I shouldn't get the warning about AVX because I am using Intel's MKL version. Moreover, surprisingly, if I downgrade tensorflow to 2.1, warning changes to the issue described https://github.com/tensorflow/tensorflow/issues/45632.
Standalone code to reproduce the issue
import tensorflow as tf
import numpy as np
a = np.array([2 , 4, 5])
ap=tf.constant(a)

Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.
CPU Version: Intel i7-10610U CPU @1.8GHz
I have installed Tensorflow in a new environment to ensure that there is no issue with dependencies.

Please add label "comp:mkl", as requested at https://software.intel.com/content/www/us/en/develop/articles/intel-optimization-for-tensorflow-installation-guide.html

"I shouldn't get the warning about AVX because I am using Intel's MKL version.". The warning is not complaining that AVX is not enabled. It is just an informative message saying only for some ops, Intel's oneDNN would be used and AVX or AVX2 would be turned on.

"I shouldn't get the warning about AVX because I am using Intel's MKL version.". The warning is not complaining that AVX is not enabled. It is just an informative message saying only for some ops, Intel's oneDNN would be used and AVX or AVX2 would be turned on.

Thanks for your help. My hypothesis is that if we use Intel's MKL version, we shouldn't get the warning because Tensorflow would be optimized for our Intel processor. So, I believe we shouldn't get the informative message (This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX AVX2). More importantly, given what you have said, we don't know whether we are getting full performance out of our CPU. Also, if we check whether mkl_enabled flag (as per documentation on Intel's website), it shows that MKL is not. I also tested with Tensorflow 2.1: the warning changes, but the flag is enabled. (https://github.com/tensorflow/tensorflow/issues/45632) I think there is an issue with Intel's Tensorflow-mkl package.

@gmatalongthewatchtower
I run following code, get the result:
MKL enabled: True
import tensorflow as tf
major_version = int(tf.__version__.split(".")[0])
if major_version >= 2:
   from tensorflow.python import _pywrap_util_port
   print("MKL enabled:", _pywrap_util_port.IsMklEnabled())
else:
   print("MKL enabled:", tf.pywrap_tensorflow.IsMklEnabled())

Could you check your Tensorflow version?
I install it by:
conda install tensorflow=2.3.0=mkl_py37h0481017_0
If you want to confirm the MKL(oneDNN) is enabled, please set the ENV:
export MKLDNN_VERBOSE=1
You will see following log if Tensorflow-MKL is installed/built successfully.
2020-12-22 12:24:35.981441: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-12-22 12:24:36.014367: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2100000000 Hz
2020-12-22 12:24:36.020244: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55cdf9586130 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-12-22 12:24:36.020286: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-12-22 12:24:36.020435: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.

Input image shape: (100, 224, 224, 3)
dnnl_verbose,info,oneDNN v1.4.0 (commit N/A)
dnnl_verbose,info,cpu,runtime:OpenMP
dnnl_verbose,info,cpu,isa:Intel AVX-512 with Intel DL Boost
dnnl_verbose,info,gpu,runtime:none
dnnl_verbose,exec,cpu,reorder,jit:uni,undef,src_f32::blocked:acdb:f0 dst_f32::blocked:abcd:f0,,,32x3x224x224,49.5181
dnnl_verbose,exec,cpu,reorder,jit:uni,undef,src_f32::blocked:cdba:f0 dst_f32::blocked:Acdb16a:f0,,,64x3x7x7,0.413818
dnnl_verbose,exec,cpu,convolution,jit:avx512_common,forward_training,src_f32::blocked:abcd:f0 wei_f32::blocked:Acdb16a:f0 bia_f32::blocked:a:f0 dst_f32::blocked:aBcd16b:f0,,alg:convolution_direct,mb32_ic3oc64_ih224oh112kh7sh2dh0ph3_iw224ow112kw7sw2dw0pw3,11.1809
dnnl_verbose,exec,cpu,batch_normalization,bnorm_jit:avx512_common,forward_inference,data_f32::blocked:aBcd16b:f0 diff_undef::un

@NeoZhangJianyu
Here's the output of tf version:
tf.__version__
'2.3.0'

Here's the output from MKL code:
import tensorflow as tf
major_version = int(tf.__version__.split(".")[0])
if major_version >= 2:
    from tensorflow.python import _pywrap_util_port
    print("MKL enabled:", _pywrap_util_port.IsMklEnabled())
else:
    print("MKL enabled:", tf.pywrap_tensorflow.IsMklEnabled())

MKL enabled: False

Here's the package name after running conda list command:
tensorflow-mkl            2.3.0                h93d2e19_0
You have told me to set export MKLDNN_VERBOSE=1. This looks like Linux environment variable. How can I set this on Windows 10?
Can you please let me know if you have more questions?

@gmatalongthewatchtower
In windows, set MKLDNN_VERBOSE=1

@NeoZhangJianyu : Thanks again for your help. I did set MKLDNN_VERBOSE=1. How do I verify whether this flag is set? I ran the following code:
import tensorflow as tf
major_version = int(tf.__version__.split(".")[0])
if major_version >= 2:
    from tensorflow.python import _pywrap_util_port
    print("MKL enabled:", _pywrap_util_port.IsMklEnabled())
else:
    print("MKL enabled:", tf.pywrap_tensorflow.IsMklEnabled())

I am still getting MKL enabled: False
I ran the following sample code to test verbosity of mkl:
import tensorflow as tf
from tensorflow import keras

input_A = keras.layers.Input(shape=[5], name="wide_input")
input_B = keras.layers.Input(shape=[6], name="deep_input")
hidden1 = keras.layers.Dense(30, activation="relu")(input_B)
hidden2 = keras.layers.Dense(30, activation="relu")(hidden1)
concat = keras.layers.concatenate([input_A, hidden2])
output = keras.layers.Dense(1, name="output")(concat)
model = keras.models.Model(inputs=[input_A, input_B], outputs=[output])

Here's the output:

[2020-12-23 12:18:25.936282: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.](url)

@gmatalongthewatchtower
As explain of @wei-v-wang , the warning is not complaining that AVX is not enabled.
Could paste more log?

Hello @NeoZhangJianyu, I did respond to @wei-v-wang above. It is true that warning is not complaining, but there are two red flags. 1) We don't know whether we are getting full performance out of our CPU. 2) if you see mkl_enabled flag (as per documentation on Intel's website), it shows that MKL is not enabled.
I am happy to generate log files. Can you please let me know the code and steps to generate these? I am using tf 2.3 on Windows 10.

@gmatalongthewatchtower

check MKL is working in your TF.

Windows:
set MKLDNN_VERBOSE=1
Linux:
export MKLDNN_VERBOSE=1
Execute following python script based on TF.
import numpy as np
import tensorflow as tf

x_in = np.array([[
  [[2], [1], [2], [0], [1]],
  [[1], [3], [2], [2], [3]],
  [[1], [1], [3], [3], [0]],
  [[2], [2], [0], [1], [1]],
  [[0], [0], [3], [1], [2]], ]])
kernel_in = np.array([
 [ [[2, 0.1]], [[3, 0.2]] ],
 [ [[0, 0.3]],[[1, 0.4]] ], ])
x = tf.constant(x_in, dtype=tf.float32)
kernel = tf.constant(kernel_in, dtype=tf.float32)
tf.nn.conv2d(x, kernel, strides=[1, 1, 1, 1], padding='VALID')

you will see the log which show oneDNN is enabled:
2020-12-25 09:20:59.557263: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-12-25 09:20:59.582017: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
dnnl_verbose,info,oneDNN v1.4.0 (commit N/A)
dnnl_verbose,info,cpu,runtime:OpenMP
dnnl_verbose,info,cpu,isa:Intel AVX2
dnnl_verbose,info,gpu,runtime:none
dnnl_verbose,exec,cpu,reorder,simple:any,undef,src_f32::blocked:cdba:f0 dst_f32:p:blocked:Acdb8a:f0,,,2x1x2x2,0.0439
dnnl_verbose,exec,cpu,convolution,jit:avx2,forward_training,src_f32::blocked:abcd:f0 wei_f32:p:blocked:Acdb8a:f0 bia_undef::undef::f0 dst_f32:p:blocked:aBcd8b:f0,,alg:convolution_direct,mb1_ic1oc2_ih5oh4kh2sh1dh0ph0_iw5ow4kw2sw1dw0pw0,0.0589
dnnl_verbose,exec,cpu,reorder,simple:any,undef,src_f32:p:blocked:aBcd8b:f0 dst_f32::blocked:acdb:f0,,,1x2x4x4,0.0295

MKL enabled issue
I reproduce your issue by following code.

import tensorflow as tf
major_version = int(tf.__version__.split(".")[0])
if major_version >= 2:
    from tensorflow.python import _pywrap_util_port
    print("MKL enabled:", _pywrap_util_port.IsMklEnabled())
else:
    print("MKL enabled:", tf.pywrap_tensorflow.IsMklEnabled())

The root cause is tensorflow is MKL version and tensorflow-base is GPU version.
Run conda list, I see:
tensorflow-2.3.0-mkl_py37h936c3e2_0
tensorflow-base-2.3.0-gpu_py37h18d21e4_0

I fix it by remove them and reinstall by:
conda install tensorflow tensorflow-base=2.3.0=mkl_py37h7075554_0
conda list
...
tensorflow                2.3.0           mkl_py37he70e3f7_0    defaults
tensorflow-base           2.3.0           mkl_py37h7075554_0    defaults
...

@NeoZhangJianyu: Thanks so much for your guidance.
1. Generate logs
I did try to run set mkldnn verbose=1 in Windows Command Prompt, but I don't see verbose logs. I believe this set command is not being detected by python. Could you please guide me where should I run this command? I have tried running it by doing all of these below:

on Anaconda Prompt that comes with Anaconda using Administrative privileges.
I use PyCharm. I tried running it from the terminal using Admin privileges.
Anaconda NAvigator -> Environment -> Right click terminal -> run command using Admin privileges.
I googled this and found that we can write a code as well.
I ran the following code and then the code that you gave me:

import mkl
mkl.verbose(1)

I ran the code you gave me, and here's the output:
MKL_VERBOSE Intel(R) MKL 2020.0 Update 2 Product build 20200624 for Intel(R) 64 architecture Intel(R) Advanced Vector Extensions 2 (Intel(R) AVX2) enabled processors, Win 1.80GHz lp64 intel_thread
MKL_VERBOSE SDOT(2,0000020AB5779ED0,1,0000020AB5779ED0,1) 60.51us CNR:OFF Dyn:1 FastMM:1 TID:0  NThr:4
2020-12-26 02:14:32.134646: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

<tf.Tensor: shape=(1, 4, 4, 2), dtype=float32, numpy=
...

Could you please let me know your thoughts?
2. Version-check
I did run conda list.
Here are the tensorflow packages I have:
tensorboard               2.3.0              pyh4dce500_0
tensorboard-plugin-wit    1.6.0                      py_0
tensorflow                2.3.0           mkl_py37h3bad0a6_0
tensorflow-base           2.3.0           eigen_py37h17acbac_0
tensorflow-datasets       1.2.0                    py37_0
tensorflow-estimator      2.3.0              pyheb71bc4_0
tensorflow-metadata       0.14.0             pyhe6710b0_1
tensorflow-mkl            2.3.0                h93d2e19_0
termcolor                 1.1.0                    py37_1

Please note that I haven't done anything special to install tensorflow except following the instructions on Intel's website posted above. I am not sure whether eigen-tensorflow-base package is same as gpu-version.
Could you please guide me? Thanks for your help. I look forward to hearing from you.

@gmatalongthewatchtower
I think your tensorflow release is with problem:
tensoflow is for mkl, but tensorflow-base is for eigen.
I guess it be the bug in conda to install the old tensorflow release.
In newer release installation,  I see it disappear.
Yes.  I have provided the guide in above comments.

Please remove the tensorflow by conda command

Reintall it by:
conda install tensorflow tensorflow-base=2.3.0=mkl_py37h7075554_0

check the tensorflow release.

conda list
...
tensorflow                2.3.0           mkl_py37he70e3f7_0    defaults
tensorflow-base           2.3.0           mkl_py37h7075554_0    defaults
...

test the MKL enable by my comment above.

Thanks @NeoZhangJianyu.
I installed tensorflow as per your suggestion:
Here's conda list
Output:
tensorflow                2.3.0           mkl_py37he70e3f7_0
tensorflow-base           2.3.0           mkl_py37h7075554_0
tensorflow-estimator      2.3.0              pyheb71bc4_0

Test for mkl
Code:
import tensorflow as tf
major_version = int(tf.__version__.split(".")[0])
if major_version >= 2:
    from tensorflow.python import _pywrap_util_port
    print("MKL enabled:", _pywrap_util_port.IsMklEnabled())
else:
    print("MKL enabled:", tf.pywrap_tensorflow.IsMklEnabled())

Output:
MKL enabled: True
MKL vs. non-MKL test:
Now I test whether MKL is doing better than standard tensorflow 2.3 (conda install tensorflow).
Code:
from tensorflow.keras.layers import Input, Dense, LSTM, Bidirectional, Conv1D
from tensorflow.keras.layers import Flatten, Dropout
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
import tensorflow.keras.backend as K
import numpy as np
from time import time

def timeit(func, iterations, *args):
    t0 = time()
    for _ in range(iterations):
        func(*args)
    print("Time/iter: %.4f sec" % ((time() - t0) / iterations))

def make_small_model(batch_shape):
    ipt   = Input(batch_shape=batch_shape)
    x     = Conv1D(128, 400, strides=4, padding='same')(ipt)
    x     = Flatten()(x)
    x     = Dropout(0.5)(x)
    x     = Dense(64, activation='relu')(x)
    out   = Dense(1,  activation='sigmoid')(x)
    model = Model(ipt, out)
    model.compile(Adam(lr=1e-4), 'binary_crossentropy')
    return model

def make_medium_model(batch_shape):
    ipt   = Input(batch_shape=batch_shape)
    x     = Bidirectional(LSTM(512, activation='relu', return_sequences=True))(ipt)
    x     = LSTM(512, activation='relu', return_sequences=True)(x)
    x     = Conv1D(128, 400, strides=4, padding='same')(x)
    x     = Flatten()(x)
    x     = Dense(256, activation='relu')(x)
    x     = Dropout(0.5)(x)
    x     = Dense(128, activation='relu')(x)
    x     = Dense(64,  activation='relu')(x)
    out   = Dense(1,   activation='sigmoid')(x)
    model = Model(ipt, out)
    model.compile(Adam(lr=1e-4), 'binary_crossentropy')
    return model

def make_data(batch_shape):
    return np.random.randn(*batch_shape), np.random.randint(0, 2, (batch_shape[0], 1))

batch_shape = (32, 400, 16)
X, y = make_data(batch_shape)

model_small = make_small_model(batch_shape)
model_small.train_on_batch(X, y)  # skip first iteration which builds graph
timeit(model_small.train_on_batch, 200, X, y)

K.clear_session()  # in my testing, kernel was restarted instead

model_medium = make_medium_model(batch_shape)
model_medium.train_on_batch(X, y)  # skip first iteration which builds graph
timeit(model_medium.train_on_batch, 10, X, y)
#endregion

Without MKL
Out[2]: 0.6387945413589478
Time/iter: 0.1114 sec
Out[2]: 0.6915708780288696
Time/iter: 20.7256 sec

With MKL
Warning message:
2020-12-31 18:32:50.096360: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-12-31 18:32:50.097360: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.

Output:
Out[2]: 0.6798648834228516
Time/iter: 0.1222 sec
Out[2]: 327.92877197265625
<Timed out...I had to stop the kernel after ~50 minutes>

Given above data, I have two questions:

It seems base tensorflow is much faster than MKL version for the above code. Why so? At the time of installing tensorflow, I got the following question from Conda:

The following packages will be DOWNGRADED:

  intel-openmp                                   2020.2-254 --> 2019.4-245

It seems TF is using older library. I am not too sure. Could this explain the slowness?

While I was able to install MKL using the version you provided. Tensorflow  2.4 is already out. In the future, how do I know what version to install?

@gmatalongthewatchtower

The TF with MKL needs to be set optimization setting.
I try following setting in linux, got shorter time than stack TF in small_model (CNN).
But a little longer time than stack TF in medium model (LSTM).

Different model need different setting to reach best performance on CPU.
You could try to adjust it based on your CPU.
set TF_ENABLE_MKL_NATIVE_FORMAT=1
set TF_NUM_INTEROP_THREADS=1
set TF_NUM_INTRAOP_THREADS=4
set  OMP_NUM_THREADS=4
set KMP_BLOCKTIME=1
set KMP_AFFINITY=granularity=fine,compact,1,0

Additional, we recommend to set the physical CPU cores number in above setting, instead of threads number.
for example,  your CPU is  i7-10610U, 4 cores/8 threads.
You should use 4 in above setting.
The conda will upgrade/downgrade the depended package during installing tensorflow.
So,  intel-openmp is downgraded by conda.

In conda, you could use following cmd to search which Tensorflow is supported by conda:

conda search tensorflow

Currently, the latest release is 2.3
But you could build the TF2.4 from source code by bazel.

Yes, you find the cause.
We should try to let the CPU cores are busy to get better performance.

@gmatalongthewatchtower

The TF with MKL needs to be set optimization setting.
I try following setting in linux, got shorter time than stack TF in small_model (CNN).
But a little longer time than stack TF in medium model (LSTM).

Different model need different setting to reach best performance on CPU.
You could try to adjust it based on your CPU.
set TF_ENABLE_MKL_NATIVE_FORMAT=1
set TF_NUM_INTEROP_THREADS=1
set TF_NUM_INTRAOP_THREADS=4
set  OMP_NUM_THREADS=4
set KMP_BLOCKTIME=1
set KMP_AFFINITY=granularity=fine,compact,1,0

Additional, we recommend to set the physical CPU cores number in above setting, instead of threads number.
for example,  your CPU is  i7-10610U, 4 cores/8 threads.
You should use 4 in above setting.
The conda will upgrade/downgrade the depended package during installing tensorflow.
So,  intel-openmp is downgraded by conda.

In conda, you could use following cmd to search which Tensorflow is supported by conda:

conda search tensorflow

Currently, the latest release is 2.3
But you could build the TF2.4 from source code by bazel.

Yes, you find the cause.
We should try to let the CPU cores are busy to get better performance.

Thanks @NeoZhangJianyu.
#1: With above settings, Intel's MKL is much slower.
#2: I searched for tensorflow, and here's what I got. How do I know which flavor of tensorflow to pick? I can decide between Python 3.7 and 3.8, but I am not sure about mkl versions.
tensorflow                     2.3.0 mkl_py37h04bc1aa_0  pkgs/main
tensorflow                     2.3.0 mkl_py37h10aaca4_0  pkgs/main
tensorflow                     2.3.0 mkl_py37h3bad0a6_0  pkgs/main
tensorflow                     2.3.0 mkl_py37h48e11e3_0  pkgs/main
tensorflow                     2.3.0 mkl_py37h856240d_0  pkgs/main
tensorflow                     2.3.0 mkl_py37h936c3e2_0  pkgs/main
tensorflow                     2.3.0 mkl_py37h952ae9f_0  pkgs/main
tensorflow                     2.3.0 mkl_py37he40ee82_0  pkgs/main
tensorflow                     2.3.0 mkl_py37he70e3f7_0  pkgs/main
tensorflow                     2.3.0 mkl_py38h1fcfbd6_0  pkgs/main
tensorflow                     2.3.0 mkl_py38h37f7ee5_0  pkgs/main
tensorflow                     2.3.0 mkl_py38h3c6dea5_0  pkgs/main
tensorflow                     2.3.0 mkl_py38h46e32b0_0  pkgs/main
tensorflow                     2.3.0 mkl_py38h637f690_0  pkgs/main
tensorflow                     2.3.0 mkl_py38h8557ec7_0  pkgs/main
tensorflow                     2.3.0 mkl_py38h8c0d9a2_0  pkgs/main
tensorflow                     2.3.0 mkl_py38ha39cb68_0  pkgs/main
tensorflow                     2.3.0 mkl_py38hd19cc29_0  pkgs/main

3: Also, is there anyway to use Intel GPU?

Thanks @NeoZhangJianyu.
#1: With above settings, Intel's MKL is much slower.
#2: I searched for tensorflow, and here's what I got. How do I know which flavor of tensorflow to pick? I can decide between Python 3.7 and 3.8, but I am not sure about mkl versions.
tensorflow                     2.3.0 mkl_py37h04bc1aa_0  pkgs/main
tensorflow                     2.3.0 mkl_py37h10aaca4_0  pkgs/main
tensorflow                     2.3.0 mkl_py37h3bad0a6_0  pkgs/main
tensorflow                     2.3.0 mkl_py37h48e11e3_0  pkgs/main
tensorflow                     2.3.0 mkl_py37h856240d_0  pkgs/main
tensorflow                     2.3.0 mkl_py37h936c3e2_0  pkgs/main
tensorflow                     2.3.0 mkl_py37h952ae9f_0  pkgs/main
tensorflow                     2.3.0 mkl_py37he40ee82_0  pkgs/main
tensorflow                     2.3.0 mkl_py37he70e3f7_0  pkgs/main
tensorflow                     2.3.0 mkl_py38h1fcfbd6_0  pkgs/main
tensorflow                     2.3.0 mkl_py38h37f7ee5_0  pkgs/main
tensorflow                     2.3.0 mkl_py38h3c6dea5_0  pkgs/main
tensorflow                     2.3.0 mkl_py38h46e32b0_0  pkgs/main
tensorflow                     2.3.0 mkl_py38h637f690_0  pkgs/main
tensorflow                     2.3.0 mkl_py38h8557ec7_0  pkgs/main
tensorflow                     2.3.0 mkl_py38h8c0d9a2_0  pkgs/main
tensorflow                     2.3.0 mkl_py38ha39cb68_0  pkgs/main
tensorflow                     2.3.0 mkl_py38hd19cc29_0  pkgs/main

3: Also, is there anyway to use Intel GPU?

@gmatalongthewatchtower
1.
The optimization setting is depended on your system (HW, OS, SW).
I try the setting in Intel Xeon CPU, Linux. It maybe not adapt to your system.
Please tuning the values to get better performance.
You could check if the CPU is full when running.

The newer is better.

Tensorflow support Intel GPU is coming.
By now, you could use OpenVINO for inference, accelerated by Intel Integrated GPU.

@gmatalongthewatchtower
1.
The optimization setting is depended on your system (HW, OS, SW).
I try the setting in Intel Xeon CPU, Linux. It maybe not adapt to your system.
Please tuning the values to get better performance.
You could check if the CPU is full when running.

The newer is better.

Tensorflow support Intel GPU is coming.
By now, you could use OpenVINO for inference, accelerated by Intel Integrated GPU.

same problem for me, but I tried all above methods but did not work :(

@Tlevi16
What's your CPU type? And what's the test case of your?
Could you provide detailed info of your case? like, test script, optimization setting.

Yes.

if yes, it's
INTEL(R) CORE(TM) i5- 8250U CPU @ 1.60 GHz 1.80 GHz

and 64 bit operating system, windows 10 home

Yes.  It's OK!
Could you share your test script or more detailed info about your test?
I try to reproduce your case in my PC. Then try to find the solution for your case.

`import tensorflow as tf
import numpy as np
a = tf.constant(np.array([1., 2., 3.]))
b = tf.constant(np.array([.4, .5, .6]))
c = tf.tensordot(a, b, 1)
output = c.numpy() `
but when I run this program, I get the error :-
I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.2 AVX AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

what should I do now ????????

pleaseee help me

@Tlevi16
I believe you have met the performance issue in Tensorflow 2.3.0 MKL.
I'd like to answer your question in this issue.
For above error in your simple case, it's another issue.
Is it possible you create another github issue to track it?
I guess your Tensorflow running environment is changed recently.

I am new to tensorflow i just started it today, yeah I changed my tensorflow running environment

pls give a solution

to my problem

ya I'll create another github issue @NeoZhangJianyu , what should I name it as ??

pls reply

@Tlevi16
Welcome to Tensorflow world!
Is it possible to create a new issue of your case?
We want to keep this issue to focus on the original issue.
The discussion will help thousands of  developers for same issue.
Thank you!

ok but what can I name it as ??

I just test your case.
It's passed.
The warning is just warning.
Your CPU supports AVX2. You could ignore it.

ok thx

@gmatalongthewatchtower
Could you please try on latest stable version of TF 2.5  and let us know if this is still an issue.Thanks!

亲，有没有建议的深度学习加速库，step-by—step的指南？ondnn,openvino都没弄成，常常参考资料过时缺东少西。(linux系统)不胜感激

亲，有没有建议的深度学习加速库，step-by—step的指南？ondnn,openvino都没弄成，常常参考资料过时缺东少西。(linux系统)不胜感激

@gmatalongthewatchtower 1. The optimization setting is depended on your system (HW, OS, SW). I try the setting in Intel Xeon CPU, Linux. It maybe not adapt to your system. Please tuning the values to get better performance. You could check if the CPU is full when running.

The newer is better.
Tensorflow support Intel GPU is coming.
By now, you could use OpenVINO for inference, accelerated by Intel Integrated GPU.

亲，有没有建议的深度学习加速库，step-by—step的指南？ondnn,openvino都没弄成，常常参考资料过时缺东少西。(linux系统)不胜感激

@chuangzhidan
Could you create a new github issue for your question? This issue is already closed.
It's OK to ask your question in Chinese. (topic should be in English)
Thank you!

@gmatalongthewatchtower In windows, set MKLDNN_VERBOSE=1

how to set that? in windows

authors: Dreamcouple NeoZhangJianyu Tlevi16 chuangzhidan gmatalongthewatchtower sushreebarsa wei-v-wang


text:
failed to alloc X bytes unified memory; result: CUDA_ERROR_OUT_OF_MEMORY: out of memory

I am trying to run a tensorflow project and I am encountering memory problems on the university HPC cluster. I have to run a prediction job for hundreds of inputs, with differing length. We have GPU nodes with different amount of vmem, so I am trying to set up the scripts in a way that will not crash in any combination of GPU node - input length.
After searching the net for solutions, I played around with TF_FORCE_UNIFIED_MEMORY, XLA_PYTHON_CLIENT_MEM_FRACTION, XLA_PYTHON_CLIENT_PREALLOCATE and TF_FORCE_GPU_ALLOW_GROWTH, and also with tensorflow's set_memory_growth. As I understood, with unified memory, I should be able to use more memory than a GPU has in itself.
This was my final solution (only relevant parts)
os.environ['TF_FORCE_UNIFIED_MEMORY']='1'
os.environ['XLA_PYTHON_CLIENT_MEM_FRACTION']='2.0'
#os.environ['XLA_PYTHON_CLIENT_PREALLOCATE']='false'
os.environ['TF_FORCE_GPU_ALLOW_GROWTH ']='true' # as I understood, this is redundant with the set_memory_growth part

import tensorflow as tf
gpus = tf.config.list_physical_devices('GPU')
if gpus:
  try:
    # Currently, memory growth needs to be the same across GPUs
    for gpu in gpus:
      print(gpu)
      tf.config.experimental.set_memory_growth(gpu, True)
    logical_gpus = tf.config.list_logical_devices('GPU')
    print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
  except RuntimeError as e:
    # Memory growth must be set before GPUs have been initialized
    print(e)

and I submit it on the cluster with --mem=30G (slurm job scheduler) and --gres=gpu:1.
And this is the error my code crashes with. As I understand, it does try to use the unified memory but is failing for some reason.
Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5582 MB memory) -> physical GPU (device: 0, name: GeForce GTX TITAN Black, pci bus id: 0000:02:00.0, compute capability: 3.5)
2021-08-24 09:22:02.053935: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:764] failed to alloc 12758286336 bytes unified memory; result: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-24 09:22:03.738635: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:764] failed to alloc 11482457088 bytes unified memory; result: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-24 09:22:05.418059: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:764] failed to alloc 10334211072 bytes unified memory; result: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-24 09:22:07.102411: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:764] failed to alloc 9300789248 bytes unified memory; result: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-24 09:22:08.784349: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:764] failed to alloc 8370710016 bytes unified memory; result: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-24 09:22:10.468644: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:764] failed to alloc 7533638656 bytes unified memory; result: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-24 09:22:12.150588: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:764] failed to alloc 6780274688 bytes unified memory; result: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-24 09:23:10.326528: W external/org_tensorflow/tensorflow/core/common_runtime/bfc_allocator.cc:272] Allocator (GPU_0_bfc) ran out of memory trying to allocate 4.33GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.

Traceback (most recent call last):
  File "scripts/script.py", line 654, in <module>
    prediction_result, (r, t) = cf.to(model_runner.predict(processed_feature_dict, random_seed=seed), "cpu")
  File "env/lib/python3.7/site-packages/alphafold/model/model.py", line 134, in predict
    result, recycles = self.apply(self.params, jax.random.PRNGKey(random_seed), feat)
  File "env/lib/python3.7/site-packages/jax/_src/traceback_util.py", line 183, in reraise_with_filtered_traceback
    return fun(*args, **kwargs)
  File "env/lib/python3.7/site-packages/jax/_src/api.py", line 402, in cache_miss
    donated_invars=donated_invars, inline=inline)
  File "env/lib/python3.7/site-packages/jax/core.py", line 1561, in bind
    return call_bind(self, fun, *args, **params)
  File "env/lib/python3.7/site-packages/jax/core.py", line 1552, in call_bind
    outs = primitive.process(top_trace, fun, tracers, params)
  File "env/lib/python3.7/site-packages/jax/core.py", line 1564, in process
    return trace.process_call(self, fun, tracers, params)
  File "env/lib/python3.7/site-packages/jax/core.py", line 607, in process_call
    return primitive.impl(f, *tracers, **params)
  File "env/lib/python3.7/site-packages/jax/interpreters/xla.py", line 608, in _xla_call_impl
    *unsafe_map(arg_spec, args))
  File "env/lib/python3.7/site-packages/jax/linear_util.py", line 262, in memoized_fun
    ans = call(fun, *args)
  File "env/lib/python3.7/site-packages/jax/interpreters/xla.py", line 758, in _xla_callable
    compiled = compile_or_get_cached(backend, built, options)
  File "env/lib/python3.7/site-packages/jax/interpreters/xla.py", line 76, in compile_or_get_cached
    return backend_compile(backend, computation, compile_options)
  File "env/lib/python3.7/site-packages/jax/interpreters/xla.py", line 373, in backend_compile
    return backend.compile(built_c, compile_options=options)
jax._src.traceback_util.UnfilteredStackTrace: RuntimeError: Resource exhausted: Out of memory while trying to allocate 4649385984 bytes.

The stack trace below excludes JAX-internal frames.
The preceding is the original exception that occurred, unmodified.

--------------------

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "scripts/script.py", line 654, in <module>
    prediction_result, (r, t) = cf.to(model_runner.predict(processed_feature_dict, random_seed=seed), "cpu")
  File "env/lib/python3.7/site-packages/alphafold/model/model.py", line 134, in predict
    result, recycles = self.apply(self.params, jax.random.PRNGKey(random_seed), feat)
  File "env/lib/python3.7/site-packages/jax/interpreters/xla.py", line 373, in backend_compile
    return backend.compile(built_c, compile_options=options)
RuntimeError: Resource exhausted: Out of memory while trying to allocate 4649385984 bytes.

I would be glad for any ideas how to get it work and use all the available memory.
Thank you!
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): debian 5781
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: -
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 2.3.0
Python version: Python 3.7.3
Bazel version (if compiling from source): -
GCC/Compiler version (if compiling from source): -
CUDA/cuDNN version: 11
GPU model and memory: several GPUs on the cluster (Tesla M60, vmem:8G ; RTX 2080Ti, vmem:10G; Quadro RTX 6000, vmem: 24G)

I would be glad for any ideas how to get it work and use all the available memory.
Thank you!

Hi @gezmi
We see that the issue template has not been filled, could you please do so as it helps us analyse the issue [tf version, steps followed before you ran into this error or stand alone code/colab gist to reproduce the issue faced].Thanks

authors: gezmi mohantym


text:
Segmentation model C++ API's based inference output different from Python tensors

System information

Have I written custom code:No
OS Platform and Distribution (e.g., Linux Ubuntu 18.04):  Ubuntu 18.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): source
TensorFlow version (use command below): v2.3.0
Python version: 3.6.9
Bazel version (if compiling from source): 3.1.0
GCC/Compiler version (if compiling from source): 7.5.0
CUDA/cuDNN version: 10.1-cudnn7
GPU model and memory:

Describe the current behavior
We've trained a Segmentation model using Keras with Tensorflow backend. Now, We are trying to infer by using the TF C++ interface.
I used the below code snippet to convert the .h5 model to a .pb file.
keras_model_path="keras_model.h5"
model = tf.keras.models.load_model(keras_model_path,  compile=False)
model.summary()
tf.saved_model.save(model, "~/tf2_cpp/")

Our python pipeline,
model = tf.keras.models.load_model(keras_model_path,  compile=False)
img = cv2.imread("test.jpg")
resized_img = cv2.resize(img,(224, 224)).reshape([1, 224, 224,3]).astype(np.float32)
output = model.predict(resized_img) #7, 224, 244, 2
mask = tf.argmax(output[0], axis=3) #1, 244, 244

our C++ pipeline,
// Pre Processing
cv::Mat img = cv::imread("test.jpg")
cv::Size s(224,224);
cv::resize(img, img, s, cv::INTER_LINEAR);
tensorflow::Tensor input_tensor(tensorflow::DT_FLOAT, tensorflow::TensorShape({1, height, width, depth}));
float *p = input_tensor.flat<float>().data();
cv::Mat fakeMat(224, 224, CV_32FC3, p);
img.convertTo(fakeMat, CV_32FC3);

// Prediction
const string input_node = "serving_default_input_4:0";
// std::vector<std::pair<string, tensorflow::Tensor>> inputs_data  = {{input_node, input_tensor_mapped}};
std::vector<string> output_nodes = {{"StatefulPartitionedCall:3",
    "StatefulPartitionedCall:2",
    "StatefulPartitionedCall:5",
    "StatefulPartitionedCall:1",
    "StatefulPartitionedCall:4",
    "StatefulPartitionedCall:0",
    "StatefulPartitionedCall:6"}};

std::vector<Tensor> predictions;

this->bundle.GetSession()->Run({{input_node, input_tensor}}, output_nodes, {}, &predictions);

//Post-processing
auto argOps = tensorflow::ops::ArgMax(transposeScope, predictions[1], 3);
std::vector<Tensor> temOutputs;
TF_CHECK_OK(sess.Run({argOps}, &temOutputs));

The tensor value is different when I compare the result between Python and C++. And, C++ argmax API returning zero for all the tensor as well.
Describe the expected behavior
The output expected to be same in both Python and C++. Maybe with some precision changes.

@nullbyte91 ,
In order to expedite the trouble-shooting process, could you please provide a complete code to reproduce the issue.Also
could you please update TensorFlow to the latest stable version v.2.7 and let us know if you are facing the same error. Thanks!

ArgMax doesnt work, I suppose it has a bug, it returns a list of zero for the tensor

authors: nullbyte91 sguelbol tilakrayal