text:
dynamic_embedding does not support tf.RaggedTensor as the input to look up

Describe the feature and the current behavior/state.
First of all, thank you all for making this awesome project!
Right now dynamic embedding seems does not support embedding lookup with ragged tensor. For example, this code snippet for dynamic embedding lookup works well with regular tf.tensor
movie_embeddings = tfra.dynamic_embedding.get_variable(
    name="movie_dynamic_embeddings",
    dim=16,
    initializer=tf.keras.initializers.RandomNormal(-1, 1))

movie_embeddings_shadow = tfra.dynamic_embedding.shadow_ops.ShadowVariable(
    movie_embeddings,
    name='movie_dynamic_embeddings_shadow',
    max_norm=None,
    trainable=True)

movie_ids = tf.random.uniform(shape=[8, 4], minval=0, maxval=15, dtype=tf.int64)

movie_id_weights = tfra.dynamic_embedding.shadow_ops.embedding_lookup(movie_embeddings_shadow, movie_ids)

However, if movie_ids is a ragged tensor, like
ragged_movie_ids = tf.ragged.constant([[1, 2], [3], [4, 5, 6]])
movie_id_weights = tfra.dynamic_embedding.shadow_ops.embedding_lookup(movie_embeddings_shadow, ragged_movie_ids)

doing the embedding lookup failed with the following stack trace
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-20-d8d43f7fc947> in <module>
----> 1 movie_id_weights = tfra.dynamic_embedding.shadow_ops.embedding_lookup(movie_embeddings_shadow, ragged_movie_ids)

/data/miniconda3/envs/env-3.8.8/lib/python3.8/site-packages/tensorflow_recommenders_addons/dynamic_embedding/python/ops/shadow_embedding_ops.py in embedding_lookup(shadow, ids, partition_strategy, name, validate_indices)
    241       containing the values from the params tensor(s) for keys in ids.
    242   """
--> 243   ids = ops.convert_to_tensor(ids)
    244
    245   if shadow.ids.dtype != ids.dtype:

/data/miniconda3/envs/env-3.8.8/lib/python3.8/site-packages/tensorflow/python/profiler/trace.py in wrapped(*args, **kwargs)
    181         with Trace(trace_name, **trace_kwargs):
    182           return func(*args, **kwargs)
--> 183       return func(*args, **kwargs)
    184
    185     return wrapped

/data/miniconda3/envs/env-3.8.8/lib/python3.8/site-packages/tensorflow/python/framework/ops.py in convert_to_tensor(value, dtype, name, as_ref, preferred_dtype, dtype_hint, ctx, accepted_result_types)
   1693
   1694     if ret is None:
-> 1695       ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
   1696
   1697     if ret is NotImplemented:

/data/miniconda3/envs/env-3.8.8/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py in _constant_tensor_conversion_function(v, dtype, name, as_ref)
    341                                          as_ref=False):
    342   _ = as_ref
--> 343   return constant(v, dtype=dtype, name=name)
    344
    345

/data/miniconda3/envs/env-3.8.8/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py in constant(value, dtype, shape, name)
    265     ValueError: if called on a symbolic tensor.
    266   """
--> 267   return _constant_impl(value, dtype, shape, name, verify_shape=False,
    268                         allow_broadcast=True)
    269

/data/miniconda3/envs/env-3.8.8/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py in _constant_impl(value, dtype, shape, name, verify_shape, allow_broadcast)
    277       with trace.Trace("tf.constant"):
    278         return _constant_eager_impl(ctx, value, dtype, shape, verify_shape)
--> 279     return _constant_eager_impl(ctx, value, dtype, shape, verify_shape)
    280
    281   g = ops.get_default_graph()

/data/miniconda3/envs/env-3.8.8/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py in _constant_eager_impl(ctx, value, dtype, shape, verify_shape)
    302 def _constant_eager_impl(ctx, value, dtype, shape, verify_shape):
    303   """Creates a constant on the current device."""
--> 304   t = convert_to_eager_tensor(value, ctx, dtype)
    305   if shape is None:
    306     return t

/data/miniconda3/envs/env-3.8.8/lib/python3.8/site-packages/tensorflow/python/framework/constant_op.py in convert_to_eager_tensor(value, ctx, dtype)
    100       dtype = dtypes.as_dtype(dtype).as_datatype_enum
    101   ctx.ensure_initialized()
--> 102   return ops.EagerTensor(value, ctx.device_name, dtype)
    103
    104

ValueError: TypeError: object of type 'RaggedTensor' has no len()

As a consequence, the high level keras embedding layers built on top of it failed as well, such as

tfra.dynamic_embedding.keras.layers.BasicEmbedding
tfra.dynamic_embedding.keras.layers.SquashedEmbedding

tf.RaggedTensor are usually used to represent features in recommender ranking models such as:

the variable-length list of video_id viewed by a user in the past 3 days
the variable-length list of author_id subscribed by a user  in the past 24 hours

Is there any plan to support embedding lookup with ragged tensors?
Relevant information

Are you willing to contribute it (yes/no): Yes
Are you willing to maintain it going forward? (yes/no): Yes
Is there a relevant academic paper? (if so, where): No
Is there already an implementation in another framework? (if so, where): Not sure, haven't checked hugectr sok
Was it part of tf.contrib? (if so, where): No

Which API type would this fall under (layer, metric, optimizer, etc.)

tfra.dynamic_embedding.shadow_ops.embedding_lookup
tfra.dynamic_embedding.keras.layers.BasicEmbedding
tfra.dynamic_embedding.keras.layers.SquashedEmbedding

Who will benefit with this feature?
Any recommender developer who wishes to use ragged tensor to look up dynamic embedding
Any other info.

@cockroachzl  Thanks for feedback. Unfortunately, embedding_lookup currently only supports Tensor and SparseTensor.  If using RaggedTensor, you'll have to process before lookup.
BTW, what do you expect on the lookup result, if using ragged tensor as input?

Hi @cockroachzl , thank you for your feedback that' very important for us.

Firstly, we recommend you using tfra.dynamic_embedding.embedding_lookup APIs first, for shadow_ops usually were used for secondary development.
Secondly, yes, you're right, tf.RaggedTensor is still not supported, You can reach same target through SparseTensor and tfra.dynamic_embedding.embedding_lookup_sparse or tfra.dynamic_embedding.safe_embedding_lookup_sparse. We designed the tfra.dynamic_embedding.embedding_lookup, tfra.dynamic_embedding.embedding_lookup_sparse and tfra.dynamic_embedding.safe_embedding_lookup_sparse APIs fully followed the same name APIs under the namespace of tf.nn, you could refer to one of them https://www.tensorflow.org/api_docs/python/tf/nn/embedding_lookup_sparse
Last, we try to support the  tf.RaggedTensor as ids in next release.

Thank you again.

@cockroachzl Thanks for feedback. Unfortunately, embedding_lookup currently only supports Tensor and SparseTensor. If using RaggedTensor, you'll have to process before lookup.

Got it, right now I work around this by using regular tf.tensor.

BTW, what do you expect on the lookup result, if using ragged tensor as input?

I expect the look up result to be the same as using tf.keras.layers.Embedding with ragged tensors as the input.
If the input is tf.ragged.constant([[1, 2], [3], [4, 5, 6]]), and the embedding size is 2, then the result of the embedding lookup is like
tf.ragged.constant([
    [[0.1,0.1], [0.2,0.2]],
    [[0.3,0.3]],
    [[0.4,0.4], [0.5,0.5], [0.6,0.6]]
])

Has this issue been solved? May I close it?

authors: Lifann MoFHeka cockroachzl rhdong

repo: tensorflow/recommenders-addons


text:
No way to freeze fused BN stats

When fine-tuning networks trained with BN sometimes we want to freeze and use the accumulated moving averages while allowing the gradients to be backpropagated through the BN layer, but currently there is no way of doing so with fused BN, since when is_training = False the layer gives erroneous gradients. Of course, we could use the batch statistics from the new task to accumulate the stats, but it isn't possible in the case of batch_size = 1.
I understand that due to the nature of the CuDNN kernel it might be hard to implement such feature, but a fused Batch Renorm layer could be a decent compromise, as it uses the moving averages when training as well as during inference.

@zhangyaobit can you comment on this?  Thanks!

Setting is_training to False and at the same time doing gradient computation is expected to get erroneous gradients, because the second and third outputs of FusedBatchNorm are only applicable/valid when is_training is True.
Could you try train with fused batch norm, and fine-tuning with non-fused batch norm?

In object detection and semantic segmentation sometimes the batch_size is set to 1 for the original network, and unfortunately our use case fall under this category. Non-fused batch norm for NCHW is around 6x slower https://github.com/tensorflow/tensorflow/issues/7551#issuecomment-280421351.

Not sure if I understand correctly: are you saying fine tune is done with a batch size of 1 and non-fused batch norm doesn't work with a batch size of 1? And even if it works, it is 6x slower?
How long do you expect fine tune currently takes (on the scale of hours or days? say on GPU)?

Yes, fine tune is done with a batch size of 1 (on each GPU). I am training on a 4x 1080 machine, and fine tuning would take around ~20 hours. Non-fused batch norm does work (gives correct gradients), but it is way too slow and will push the fine tuning time beyond 100 hours. It would be faster to simply retrain the original network in NHWC and fine tune there with non-fused batch norm, which is ironic, considering NCHW is the CuDNN canonic format.

It may make sense to simply forward the population mean and variance to the second and third outputs of FusedBatchNorm when fused=Tue. There is no performance penalty for this, since it is a forwarding instead of a copy.
Before this feature is implemented, you can modify the code here  to set batch_mean and batch_variance to mean and variance when is_training is False. Note that doing this will introduce actual copies (a slight performance penalty), so as ideal as making this change on the C++ side; but this should get you most of the GPU performance.

I marked this as contribution welcome for now. Anyone on the Tensorflow team or external developers are welcome to contribute if interested. Please let me know if you have any questions.

I'm closing this issue, as the freeze mode fused batch norm has been supported now: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/ops/nn_grad.py#L674

Thank you!

@zhangyaobit Thanks for providing the implementation for backward. However, I noticed that the performance of FusedBatchNormGrad when is_training=False is very pool.
I didn't benchmark the op alone, but switching is_training on and off can affect my total training speed by 2.5x. My estimate is that the op itself would have at least 5x difference in speed.
I guess the reason is that the implementation now is not based on fused kernels. This makes it as slow as the non-fused kernels.

@zhangyaobit I used the kernel BatchNormWithGlobalNormalizationGrad for the backward implementation when is_training=False. This improves the speed a lot (my overall training time is then closed to the fused case when is_training=True).
To really use this op there are still some issues:

The op BatchNormWithGlobalNormalizationGrad was marked deprecated. I unmarked it to test but not sure if that's desired. It's a helpful op in this case.
The op doesn't support backward. This will break "grad of grad" unit tests. I think "grad of grad" is less important but it could be implemented if really needed.

I'll send a PR on this.

Hi @ppwwyyxx ,
I notice that currently only NHWC is supported for FusedBatchNormFreezeGrad and tf.transpose is needed in python if the input is NCHW.  This can take enourmous time if batch norm is adopted in large scale and the data format is NCHW.
Do you consider adding support for NCHW?

I don't have time for this.
In fact a thorough benchmark is needed because the performance of the normal reduction ops has been improved a lot since last year. It's quite likely that my naive cuda kernel isn't very performant in NHWC or NCHW compared to a non-fused multi-op implementation.

Thanks for your prompt response.
I have done a quick benchmark using my network. For fused BN, it takes ~1.43 second/batch while for non-fused BN, it takes ~1.32 second/batch regardless if NCHW or NHWC is used. It seems for freezed BN, non-fused is faster.

A quick test shows that fused & non-fused BN have similar backward speed when freezed:
#!/usr/bin/env python
import tensorflow as tf
import time
import os
from tensorflow.python.client import device_lib
local_device_protos = device_lib.list_local_devices()
GPU_MODE = len([x.name for x in local_device_protos if x.device_type == 'GPU'])

N = 64
C = 128
H, W = 128, 128
print("N, C, H, W:", [N, C, H, W])

def benchmark_all(fuse, format):
    shape4d = [N, C, H, W] if format == 'NCHW' else [N, H, W, C]

    tf.reset_default_graph()
    input = tf.get_variable('input', shape=shape4d, dtype=tf.float32)

    scale = tf.get_variable('scale', shape=[C])
    offset = tf.get_variable('offset', shape=[C])
    mean = tf.random_normal(shape=[C])
    var = tf.random_normal(shape=[C]) + 1.

    if fuse:
        output, _, _ = tf.nn.fused_batch_norm(
            input, scale, offset, mean, var, epsilon=1e-5, data_format=format, is_training=False)
    else:
        if format == 'NCHW':
            newshape = [1, C, 1, 1]
            scale = tf.reshape(scale, newshape)
            offset = tf.reshape(offset, newshape)
            mean = tf.reshape(mean, newshape)
            var = tf.reshape(var, newshape)
        output = (input - mean) * (tf.rsqrt(var + 1e-5) * scale) + offset

    forward_op = output.op
    cost = tf.reduce_sum(output)
    backward_op = tf.train.GradientDescentOptimizer(0.1).minimize(cost)

    def benchmark(op, nr_iter=200, nr_warmup=10):
        if not GPU_MODE:
            nr_iter = nr_iter // 10
            nr_warmup = nr_warmup // 5
        for k in range(nr_warmup):
            op.run()
        start = time.perf_counter()
        for k in range(nr_iter):
            op.run()
        end = time.perf_counter()
        itr_per_sec = nr_iter * 1. / (end - start)
        return itr_per_sec

    sess = tf.Session()
    with sess.as_default():
        sess.run(tf.global_variables_initializer())

        spd_forward = benchmark(forward_op)
        print("Fuse={}, Format={}, Forward: {} itr/s".format(fuse, format, spd_forward))
        spd_backward = benchmark(backward_op)
        print("Fuse={}, Format={}, Backward: {} itr/s".format(fuse, format, spd_backward))

if GPU_MODE:
    formats = ['NHWC', 'NCHW']
else:
    formats = ['NHWC']
for format in formats:
    for fuse in [True, False]:
        benchmark_all(fuse, format)

Outputs on GTX1080Ti:
N, C, H, W: [64, 128, 128, 128]
Fuse=True, Format=NHWC, Forward: 100.62081027587591 itr/s
Fuse=True, Format=NHWC, Backward: 61.392495723794156 itr/s
Fuse=False, Format=NHWC, Forward: 100.51494045143419 itr/s
Fuse=False, Format=NHWC, Backward: 60.875319893753236 itr/s
Fuse=True, Format=NCHW, Forward: 261.46341501158184 itr/s
Fuse=True, Format=NCHW, Backward: 44.5398623696861 itr/s
Fuse=False, Format=NCHW, Forward: 71.64600813406135 itr/s
Fuse=False, Format=NCHW, Backward: 48.042794214933586 itr/s

source: gharchive/issue

authors: DeepLearningJohnDoe gongbudaizhe ppwwyyxx tatatodd zhangyaobit

repo: tensorflow/tensorflow

license_type: permissive

license_source: github-api


text:
[Bug] Tensorflow serving loads incorrect model weights when using saved model main_op

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Debian 3.16.36 x86_64
TensorFlow installed from (source or binary): Binary (pip)
TensorFlow version (use command below): Git version 1.4.0-19-ga52c8d9 Release version 1.4.1
Python version: 2.7.9
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:
Exact command to reproduce: See attached gist

Hey there,
I've documented a lot of this bug over on this issue on the tensorflow serving repository, which was closed with the direction to open an issue here.
Essentially, the bug is as follows: I have an embedding layer in Keras that uses some pre-initialized weights. When I export the model for use by TF serving, I note the following behavior:

The keras model itself has no issue outputting the correct results
The exported model itself (upon inspection) has the correct weights
The result values from TF serving are incorrect.

I've narrowed it down to an issue with tensorflow.python.ops.variables.global_variables_initializer, as follows:
When specifying the main_op argument in tensorflow.saved_model.builder.SavedModelBuilder.add_meta_graph_and_variables, if I use tensorflow.saved_model.main_op.main_op() I encounter this issue.
If instead, I use a control flow group that excludes the global variables initializer as follows:
main_op_new = control_flow_ops.group(
            lookup_ops.tables_initializer(),
            variables.local_variables_initializer(),
)

I do not encounter this issue. I've attached the full code for reproducing this issue (with the caveat of needing tensorflow_model_server running) here
Here's the example output with the global variables initializer: https://gist.github.com/zmjjmz/d739cdfa52148eb814450e48cbf8ddb6
If you comment out line 83 of the repro code and uncomment line 84, you should get the correct output as shown here: https://gist.github.com/zmjjmz/9edee5b4eeff94f383122545d80ee55f
Thanks for helpin out

Someone posted a day ago https://github.com/tensorflow/serving/issues/656#issuecomment-352907846:

for me, the difference here disappeared in a recent TF / keras update ... as far as I could understand it it was a difference in TF prediction and TF Serving prediction that existed briefly and was fixed in a recent release

Does that resolve the issue for you?

I was just able to reproduce it with the versions I listed in the issue, which are the latest versions afaict from the Github releases page.
If those are the updated versions then, no -- but maybe I'm missing something newer?

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Debian 3.16.36 x86_64
TensorFlow installed from (source or binary): Binary (pip)
TensorFlow version (use command below): Git version 1.4.0-19-ga52c8d9 Release version 1.4.1
Python version: 2.7.9
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:
Exact command to reproduce: See attached gist

Hey there,
I've documented a lot of this bug over on this issue on the tensorflow serving repository, which was closed with the direction to open an issue here.
Essentially, the bug is as follows: I have an embedding layer in Keras that uses some pre-initialized weights. When I export the model for use by TF serving, I note the following behavior:

The keras model itself has no issue outputting the correct results
The exported model itself (upon inspection) has the correct weights
The result values from TF serving are incorrect.

I've narrowed it down to an issue with tensorflow.python.ops.variables.global_variables_initializer, as follows:
When specifying the main_op argument in tensorflow.saved_model.builder.SavedModelBuilder.add_meta_graph_and_variables, if I use tensorflow.saved_model.main_op.main_op() I encounter this issue.
If instead, I use a control flow group that excludes the global variables initializer as follows:
main_op_new = control_flow_ops.group(
            lookup_ops.tables_initializer(),
            variables.local_variables_initializer(),
)

I do not encounter this issue. I've attached the full code for reproducing this issue (with the caveat of needing tensorflow_model_server running) here
Here's the example output with the global variables initializer: https://gist.github.com/zmjjmz/d739cdfa52148eb814450e48cbf8ddb6
If you comment out line 83 of the repro code and uncomment line 84, you should get the correct output as shown here: https://gist.github.com/zmjjmz/9edee5b4eeff94f383122545d80ee55f
Thanks for helpin out

Thank you for point it out and it helps a lot ^_^

authors: ghtwht jart zmjjmz


text:
Missing functionality in Keras TensorBoard callback

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No.
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
TensorFlow installed from (source or binary): Binary.
TensorFlow version (use command below): 1.5.0 (Keras 2.1.2-tf)
Python version: 3.6
Bazel version (if compiling from source): NA
GCC/Compiler version (if compiling from source): NA
CUDA/cuDNN version: 9.0 / 7.0
GPU model and memory: GTX 1070 8GB
Exact command to reproduce: NA

Describe the problem
The TensorBoard-callback in Keras supports the writing of embeddings to the log, but this is not supported in the TensorFlow version of Keras, even though the doc-string of the TensorFlow version actually lists these parameters.
https://github.com/tensorflow/tensorflow/blob/579125e87af201ae6b6fa872b6dc3f3ecb400de9/tensorflow/python/keras/_impl/keras/callbacks.py#L643-L651
But these are missing from the __init__:
https://github.com/tensorflow/tensorflow/blob/579125e87af201ae6b6fa872b6dc3f3ecb400de9/tensorflow/python/keras/_impl/keras/callbacks.py#L656-L662
This is the original Keras implementation which has e.g. embeddings_freq:
https://github.com/keras-team/keras/blob/ad00676b80556a6354180a1bfa3009d4db316d3e/keras/callbacks.py#L642-L650
There seems to be a difference between the original Keras implementation and the version in TensorFlow. I am wondering if these are somehow developed in parallel so it is actually not the original Keras that is included with TensorFlow?
Thanks!

This functionality currently relies on feature from contrib, while tf.keras is in core TF. Thus it cannot be support by tf.keras yet.

Nagging Assignee @fchollet: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @fchollet: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @fchollet: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @fchollet: It has been 15 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @fchollet: It has been 15 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @fchollet: It has been 108 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @fchollet: It has been 123 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @fchollet: It has been 138 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @fchollet: It has been 153 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @fchollet: It has been 168 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @fchollet: It has been 183 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @fchollet: It has been 198 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @fchollet: It has been 213 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

authors: Hvass-Labs fchollet harshini-gadige tensorflowbutler


text:
TFLite slower with NNAPI on Snapdragon 660

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): n/a
Mobile device: Xiaomi Mi A2, Android 9
TensorFlow installed from: binary
TensorFlow version: 1.13.1
GPU model and memory: Adreno 512

Describe the current behavior
Running the TFLite demo (the link is the HEAD of master branch as of now) on , average inference time for 1 thread, mobilenet v1 quant, on CPU is 70 ms and with NNAPI degrades to 170 ms. For mobilenet v1 float, CPU is around 150 ms, and NNAPI degrades to 800 ms. Choosing GPU (only possible for float model) delivers same 70 ms as CPU/quant.
On the other hand, a weaker Xiaomi Mi A1 device (which got official upgrade to Android 9), NNAPI does provide x2 acceleration (from around 140 ms for quantized net to 70 ms).
Describe the expected behavior
I would expect NNAPI at least not to cause performance degradation. If I understand correctly, the library is smart enough to choose between GPU and CPU.
Other info / logs
A1:

Qualcomm MSM8953 Snapdragon 625
Octa-core 2.0 GHz Cortex-A53
Adreno 506

A2:

Qualcomm SDM660 Snapdragon 660
Octa-core (4x2.2 GHz Kryo 260 & 4x1.8 GHz Kryo 260)
Adreno 512

This is similar to other devices, e.g. https://github.com/tensorflow/tensorflow/issues/15554

I'm experiencing the same problem using an Xiaomi Mi 9(Snapdragon 855).
I'm experiencing significant inference slowdown for some models when using the NNApi.
I've benchmarked using TFLite Model Benchmark, and tried a couple of models from https://www.tensorflow.org/lite/guide/hosted_models
Densenet, Resnet and NASNet are much slower using the NNAPI, while MobileNet and Inception(V3, V4) are much faster.

@alexcohn as you can see in the name, NNAPI is an API. What you get when you call NNAPI depends on underlying implementation(s). If the device you use doesn't have NNAPI driver(s), NNAPI's cpu fallback will be used, see NNAPI doc. NNAPI cpu fallback is not a part of TensorFlow Lite. It's a part of AOSP. That is, this is unlikely a TensorFlow issue. It seems you ran into either NNAPI fallback or bad drivers. If you take a look at TFLite's and NNAPI's source code, you can see that they actually share the same code base for kernels. But they not always synced because NNAPI release time has to be aligned with Android's release time. And there are no way to set something like the number of threads for CPU fallback in NNAPI. Thus, you see performance mismatch.

@freedomtan Thanks for your answer.
So, the conclusion is that my app should include the 'discovery logic'. That is, after install, it must benchmark my specific model for both NNAPI and CPU (and also, the GPU delegate), and decide which delivers best performance on the particular device.
I also came across the Neural Network Transpiler project that purports to Convert a model from tflite to a C++ source code using Android Neural Network API. I am kind of sceptical now after reading your answer. Is there any chance that such static conversion will provide reliably better performance, compared to TFLite CPU or NNAPI option?

@alexcohn Ideally, NNAPI should do it for you. However, as far as I can tell, it's not there yet.
Regarding the Neural Network Transpiler, if it is done right, it should be able to reduce some memory management overhead.

Just coming here to confirm "freedom"'s answer.
NNAPI updates as a part of the yearly Android release cycle, including any CPU optimizations. They share the same performance-impacting CPU code.
So when comparing NNAPI versus TFLite, there are a few factors:

NNAPI is faster due to hardware acceleration outside the CPU
NNAPI is slower on the CPU since it lags TFLite's optimizations by up to a year.

This also includes the fact that while NNAPI provides speedups for the hosted models page ("fully quantized models"), it doesn't for models generated by this [tooling]
(https://www.tensorflow.org/lite/performance/post_training_quantization)("hybrid or weight-only quantized models").

It's good that you saw the speedups on the Xiaomi. For the other device, (I guess the A1?), it sounds like it could have an even older version of Android / NNAPI (Android 8?).

No, I did not see speedup on the newer Xiaomi A2, running with official Android 9. The x2 speedup with NNAPI I saw was on the older Xiaomi A1 device, which recently received an official OTA upgrade to Android 9.
I totally understand that release cycle of NNAPI cannot keep at pace with FTLite. My major disappointment was that it does not seem possible to predict whether using NNAPI with either setUseNNAPI(true) or with addDelegate(NnApiDelegate()) will deliver a substantial improvement or an even more substantial slowdown.

Hey @alexcohn, we're working on some tooling to make it easier to predict/understand NNAPI performance for a given device+model combination. Historically we've relied on vendors to vet the performance of their driver, but there can be surprising performance cliffs in certain cases. Thanks for the feedback; it's definitely an issue that's on our radar and something we want to solve generally to make it easy to hit the fast paths.

I faced the same difficulties having slower inference time when activating the NNAPI on different devices (Google Pixel XL, Google Pixel 3XL, Samsung Galaxy S8, Motorola Moto G5S Plus). Also, I tried to use this to check if there might be a connection between having an AI Accelerator as it is called in the ranking and profiting from turning NNAPI on. But according to the ranking, the Pixel 3 XL should profit from the NNAPI but my tensorflow lite model runs in 48 ms compared to 31 ms without NNAPI.

@Noltibus It must likely is a problem with your model. Have you checked if your ops in your model are compatible with the NNAPI? https://developer.android.com/ndk/guides/neuralnetworks#operations
When you have ops in your graph that are not compatible with the NNAPI, the CPU-fallback is used which will cause multiple graphs to be executed, which in turns causes some major slowdowns.

This is my model shown in Netron:

The only operation I cannot find in the reference is the SPLIT operation, but is that really the bottleneck?

@Noltibus Try removing the split and see if the inference-time using the NNAPI improves relative to CPU. I would guess that the split causes your graph to be divided into two subgraphs, which would slow down inference quite a bit.
Another thing to take into consideration is that hexagons DSP only works in 8bit. If you use a floating point model the network initialization time will increase as the DSP automatically quantize the network parameters. In my experience, DSP-inference on smaller models(smaller inputs, fewer parameters) won't outperform a CPU counterpart.

I will try and report back! Thank you for the input!

I wonder whether there are mainstream devices where NNAPI is faster than GPU delegate. The catch is that the list of ops implemented in GPU is limited. But AFAIK, the list of ops implemented in NNAPI is even shorter.

@alexcohn Definitely there are devices with good NN accelerators and corresponding NN HAL drivers. You may want to check the list @Noltibus mentioned. On my Pixel 3a, running Android Q Beta 4, MobileNet V1 takes around 55 ms (fp16) with GPU delegate and less than 8 ms (quantized int8) with NNAPI delegate to DSP.

Hi @alexcohn,
You can take a look at http://ai-benchmark.com/ranking_detailed to see the speed of different NN models running with NNAPI on various Android devices. For float and quantized models, you should see a huge speed-up on all phones with SDM855, Kirin 980/810 and Helio P90.

@aiff22, I have studied this table a lot, and I don't doubt that some devices have nice acceleration via NNAPI. The issue is that for some devices, same NNAPI makes the inference slower, so we cannot use it everywhere. Another question, is NNAPI always better than GPU delegate on devices with hardware acceleration?

@alexcohn, yes, right now NNAPI is dramatically better than GPU delegate on devices with hardware acceleration. Besides that, the current version of GPU delegate contains a huge number of critical bugs, thus I would recommend using it really carefully and always check the actual outputs that are produced by the model (e.g. NANs are returned for almost any model running on Mali GPUs).
Regarding NNAPI - it won't give you better results if vendor's NNAPI drivers are not present in Android image. It makes sense always use NNAPI only for the following chipsets:

Snapdragon 855 / 730
Helio P90
Kirin 980 / 970 / 810
Exynos 9820

If you have a quantized model, you can enable NNAPI for the next SoCs as well (even if vendor's NNAPI drivers are not present, the results won't be much worse):

Snapdragon 845 / 712 / 710 / 675 / 670 / 665
Helio P70 / P60

In all other cases just use TFLite with CPU and set the number of threads to min(4, number_of_cpu_cores). This will guarantee you the fastest execution speed and the absence of accuracy problems.

Thanks, Andrey. I believe that your answer can close this issue.

I have an Exynos 9810 with Mali G72 and I've never got a NaN (not that I had seen at least). But tbh I never saw an increase in performance when switching the interpreter from cpu to gpu

@alexcohn, yes, right now NNAPI is dramatically better than GPU delegate on devices with hardware acceleration. Besides that, the current version of GPU delegate contains a huge number of critical bugs, thus I would recommend using it really carefully and always check the actual outputs that are produced by the model (e.g. NANs are returned for almost any model running on Mali GPUs).
Regarding NNAPI - it won't give you better results if vendor's NNAPI drivers are not present in Android image. It makes sense always use NNAPI only for the following chipsets:
* Snapdragon 855 / 730

* Helio P90

* Kirin 980 / 970 / 810

* Exynos 9820

If you have a quantized model, you can enable NNAPI for the next SoCs as well (even if vendor's NNAPI drivers are not present, the results won't be much worse):
* Snapdragon 845 / 712 / 710 / 675 / 670 / 665

* Helio P70 / P60

In all other cases just use TFLite with CPU and set the number of threads to min(4, number_of_cpu_cores). This will guarantee you the fastest execution speed and the absence of accuracy problems.

I have an Exynos 9810 with Mali G72 and I never got any NaN (not that I had noticed at least). But tbh I never saw an increase in performance when switching the interpreter from cpu to gpu so I even doubt if that's actually running on gpu(?)

@tgpsantos, if runtime is the same - this usually means that the model contains some unsupported ops, and thus CPU backend is used instead of GPU.

@aiff22 how could i check there is any unsupported op in my tflite model?

@aiff22 also, when i run tflite object detection example in Huawei P20, which equipped with an Kirin 970 CPU, enable nnapi will make inference slower(from 40ms to 200ms)

@allenling I would recommend that you try the benchmark tool found at,
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark
You can find a list of supported ops at https://developer.android.com/ndk/guides/neuralnetworks#operands
Note that for quantized models, as per the example, there are a couple of limitations in NNAPI <= 1.1. For instance, hybrid operators are not delegated to the NNAPI, bilinear interpolation only works when width == height etc. So there are a couple of limitations that could make the execution graph split up into multiple sub-graphs, some executed on an accelerator and some on CPU.
Best way to check this would be to enable --enable_op_profiling=true when running the benchmark util mentioned above.

Coming back to this issue about 6 months later:

The only operation I cannot find in the reference is the SPLIT operation, but is that really the bottleneck?

Now, according to the website, the SPLIT operation is supported, but I still have a worse performance with my Pixel 3 XL running Android 10 and using NNApi compared to not using it. On a Pixel XL running Android 10, the peformance with and without the NNApi is the same. This still sounds like there is an unsupported operation, right? But is the SPLIT operation mentioned in the docs the right one?

@alexcohn, yes, right now NNAPI is dramatically better than GPU delegate on devices with hardware acceleration. Besides that, the current version of GPU delegate contains a huge number of critical bugs, thus I would recommend using it really carefully and always check the actual outputs that are produced by the model (e.g. NANs are returned for almost any model running on Mali GPUs).
Regarding NNAPI - it won't give you better results if vendor's NNAPI drivers are not present in Android image. It makes sense always use NNAPI only for the following chipsets:

Snapdragon 855 / 730
Helio P90
Kirin 980 / 970 / 810
Exynos 9820

If you have a quantized model, you can enable NNAPI for the next SoCs as well (even if vendor's NNAPI drivers are not present, the results won't be much worse):

Snapdragon 845 / 712 / 710 / 675 / 670 / 665
Helio P70 / P60

In all other cases just use TFLite with CPU and set the number of threads to min(4, number_of_cpu_cores). This will guarantee you the fastest execution speed and the absence of accuracy problems.

There is no reliable way of finding out if the phone the app is running on has one of these processors, right?

Keep in mind that even if NNAPI states that a given op is supported, hardware drivers for NNAPI may not support that op, or all types for that op (e.g., the Pixel 3 has a DSP driver for quantized models, but may not accelerate float models).
We're aware of the complexity in predicting how well NNAPI will support a model across the wide permutation of drivers and devices, and are working on some tweaks to the delegate to both avoid regressing performance and help whitelist the fast paths for model/device combinations. This is a major priority for the team over the next few months.

There is no reliable way of finding out if the phone the app is running on has one of these processors, right?

Reliable, yes. Officially documented, probably no.

authors: Noltibus aiff22 alanchiao alexcohn allenling freedomtan jdduke silfverstrom tgpsantos


text:
Variation in Tensorflow execution latency

System information:
Neural Network: 	LeNet 300-100
Target Platform: 	XEON PLATINUM 8000 SERIES - AMAZON AWS EC2 C5 INSTANCE
Supports:  A		VX-512, 72 Cores, 2-4 Threads; Up to 2 Instance Cores and 2 vCPUs
Baseline Framework: 	TensorFlow (AWS Image Version)
AMI ID: 		Deep Learning AMI (Ubuntu) Version 23.1 (ami-07262a45de118922e)
Python version:		p27
Describe the problem:
		To collect baseline results for TensorFlow model of lenet_300_100 ( Find the Accuracy and inference latency). By using MNIST dataset with a batch size of 16 and full(10000) number of batches.

Analysis Tensorflow inference runtime results, There is inconsistent on subsequent sessions. Please review the attached “ref_times” for  inconsistent runtime for various batches.

ref_times.txt
Please review the following logs and advice on this.
Source code / logs:
Time session source code:
with tf.Session(graph=graph, config=config) as sess:
curr_batch = 0
warm_up = 5
for i in range(warm_up):
sess.run(output, feed_dict={input_: zeros})
while True:
batch = input_reader.receive_batch()
if batch is None:
break
start = monotonic.monotonic()
pred = sess.run(output, feed_dict={input_: batch})
sys.stderr.write('Batch Inference Time(s): {}\n'.format(
monotonic.monotonic() - start))
output_writer.output_send(pred)
Please let know need another details? Please review all and advice me on this.

@VinothInspirit ,
Please provide us TF version and also the complete code being used so that we can replicate the issue from our end.Thanks!

Thanks for your response.
I have attached the complete source code for your evaluation, Results of evaluation time text file to understand the inconsistency inference time
And TF version : 1.13.1.
Please review and advice me on this
Results_ref_times.txt

@VinothInspirit,
When i tried executing the last section of the code i faced the following error RuntimeError: Attempted to use a closed Session.Kindly help us to reproduce the issue.Thanks!

#!/usr/bin/env python
# coding: utf-8

import tensorflow as tf
import numpy as np
import sys
from scipy.io import savemat
import matplotlib.pyplot as plt
from tensorflow.examples.tutorials.mnist import input_data

mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)

output_graph =  "./lenet-300.pb"

# Parameters
learning_rate_ini = 0.001
training_epochs = 2
batch_size = 16
display_step = 1
num_batches = 100

# Network Parameters
n_hidden_1 = 300  # 1st layer num features
n_hidden_2 = 100  # 2nd layer num features
n_classes = 10  # MNIST total classes (0-9 digits)

# tf Graph input
tf.reset_default_graph()
with tf.name_scope("input_tensor"):
    x = tf.placeholder("float", [None, 784])
with tf.name_scope("label_tensor"):
    y = tf.placeholder("float", [None, n_classes])

# Create model
def model(_X, _W, _biases):
    layer_1 = tf.nn.relu(tf.add(tf.matmul(_X, _W['fc1']), _biases['fc1']))  # Hidden layer with RELU activation
    layer_2 = tf.nn.relu(tf.add(tf.matmul(layer_1, _W['fc2']), _biases['fc2']))      # Hidden layer with RELU activation
    with tf.name_scope("predictions"):
        pred = tf.matmul(layer_2, _W['out']) + _biases['out']
    return pred

# In[36]:

# Store layers weight & bias
W = {
    'fc1': tf.Variable(tf.random_normal([784, n_hidden_1], stddev=0.01)),
    'fc2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2], stddev=0.01)),
    'out': tf.Variable(tf.random_normal([n_hidden_2, n_classes], stddev=0.01))
}

biases = {
    'fc1': tf.Variable(tf.random_normal([n_hidden_1], stddev=0.01)),
    'fc2': tf.Variable(tf.random_normal([n_hidden_2], stddev=0.01)),
    'out': tf.Variable(tf.random_normal([n_classes], stddev=0.01))
}

import time

if(1):
    # Construct model
    pred = model(x, W, biases)
    loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=pred, labels=y))  # Softmax loss
    cost = loss
    optimizer = tf.train.AdamOptimizer(learning_rate_ini, beta1=0.9, beta2=0.999,
            epsilon=1e-08, use_locking=False).minimize(cost)
    # Test model
    correct_prediction = tf.equal(tf.argmax(pred, 1), tf.argmax(y, 1))
    # Calculate accuracy
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
    # Initializing the variables
    init = tf.initialize_all_variables()
    # Run model
    accuracy_total = None
    config = tf.ConfigProto(device_count={'GPU': 0})
    config.log_device_placement = True
    with tf.Session(config=config) as sess:
        sess.run(init)
        # Training cycle
        for epoch in range(training_epochs):
            avg_loss = 0.
            total_batch = int(mnist.train.num_examples/batch_size)
            # Loop over all batches
            for i in range(total_batch):
                batch_x, batch_y = mnist.train.next_batch(batch_size)
                # Run optimization op (backprop) and cost op (to get loss value)
                accuracy_total, predictions,_, l = sess.run([accuracy,pred, optimizer, loss],
                                                            feed_dict={x: batch_x,
                                                              y: batch_y})
                # Compute average loss
                avg_loss += l / total_batch
            # Display logs per epoch step
            if epoch % display_step == 0:
                print("Epoch:", '%04d' % (epoch+1), "loss=",                     "{:.9f}".format(avg_loss))
        print("Optimization Finished!")
        print("Accuracy : ", accuracy_total)
        for i in range(num_batches):
            batch_x, batch_y = mnist.test.next_batch(batch_size)
            start_time = time.time()
            val_acc = accuracy.eval({x: batch_x , y: batch_y})
            print("Evaluation time batch {} : {}".format(i+1, time.time()-start_time))
            print("Validation Acc : {}".format(val_acc))

I've pasted the indented code. This should work.

@azaks2 do you know off the top of your head what could cause this?

@azaks2 replied offline:

A profile would be nice
We do not generally encourage inference via python, so python gc comes to mind
3 We overlap with the computation of next_batch

Reg:2. We do not generally encourage inference via python, so python gc comes to mind
VinothInspirit: What then would be the tf prescribed way of running(and measuring) inferences?
Reg:3. We overlap with the computation of next_batch
VinothInspirit: We give the inputs here using feed_dict, not by a pipeline, so I'm not sure if sess.run(...) would still overlap with next_batch.

For inference take a look at https://www.tensorflow.org/tfx/guide/serving
One important part is that the process will turn variables into constants.
According to the timeline (comparing slow and fast sess.run) it seems most of the diff comes from the MatMul op. I am not 100% sure but using taskset to run the script (taskset 0x1 python ...) improved the variance for my runs.

slow inference

fast inference

to get timeline
val_acc = sess.run([accuracy],
feed_dict={x: batch_x , y: batch_y},
options=tf.RunOptions(trace_level=tf.RunOptions.SOFTWARE_TRACE),
run_metadata=run_metadata)
...
tl = timeline.Timeline(run_metadata.step_stats)
ctf = tl.generate_chrome_trace_format()
with open(str(i)+'timeline.json', 'w') as f:
f.write(ctf)

authors: VinothInspirit alextp anush-o azaks3


text:
About bias quantization of hexagon delegate

As far as I know, Tensorflow Lite uses input_scale * weight_scale as bias scale to quantize bias tensor to int32. However, hexagon use int32_min/in32_max as min/max to quantize bias tensor.
So how do you handle this difference to hexagon delegation? Below is the conv_2d builder of the hexagon delegation. I don't see where the transformation is.
https://github.com/tensorflow/tensorflow/blob/539b7642a928c7fbfb4d896f650f7e6d79c2a5e0/tensorflow/lite/experimental/delegates/hexagon/builders/conv_2d_builder.cc#L234

Hi @lee-bin
The delegate doesn't use int32_min/max as min max for quantization, it computes them from the scale/zero point data in the tensor
https://github.com/tensorflow/tensorflow/blob/539b7642a928c7fbfb4d896f650f7e6d79c2a5e0/tensorflow/lite/experimental/delegates/hexagon/builders/op_builder.h#L133
The quantization params in the tensor should guarantee the input_scale*weight_scale you mentioned.
Thanks

Thanks. The bias scale, this way, is the same as in lite and hexagon which guarantees the restored bias. It really helped.

authors: karimnosseir lee-bin


text:
Run "bazel-bin/tensorflow/cc/tutorials_example_trainer --use_gpu", got "No such file or directory"

Hi everyone:
Environment info
Ubuntu 14.04 LTS
Installed version of CUDA and cuDNN:
CUDA 7.5, cuDNN v5.1

I try to build tensorflow with CUDA 7.5, cuDNN v5.1 and bazel-0.3.1-installer-linux-x86_64.sh. When I run 'bazel build -c opt --config=cuda //tensorflow/cc:tutorials_example_trainer', it seems good. But when I run 'bazel-bin/tensorflow/cc/tutorials_example_trainer --use_gpu', I got the error 'No such file or directory'. So I find the tutorials_example_trainer in following path:

I tried to run '/home/caffe/.cache/bazel/_bazel_caffe/5954a53028453a61b395a19127f3e1cf/execroot/tensorflow/bazel-out/local_linux-opt/bin/tensorflow/cc/tutorials_example_trainer.runfiles/org_tensorflow/tensorflow/cc/tutorials_example_trainer --use_gpu' and got the result like following:

It seems ok, but I do not know why cann't I run 'bazel-bin/tensorflow/cc/tutorials_example_trainer --use_gpu'.
Can anyone help me ?
Thanks!

I can't reproduce, does this file bazel-bin/tensorflow/cc/tutorials_example_trainer exist in your tensorflow root directory?

@suiyuan2009 It does not exist. I do not know why. I am still checking out something I ignored.

@JohnnyY8 have you tried bazel clean and ./configure then rebuilding?

@suiyuan2009 OK, I will try it now.

@suiyuan2009
I got this:

I think this is the problem, but when I use "sudo", I got "sudo: bazel: command not found". So if I can run bazel with sudo, it may works, right?

@JohnnyY8 run which bazel, and use full bazel bin file path instead bazel

@suiyuan2009 I change the authority of tensorflow file, and it works. Thanks for you help!
BTW, take the authority into account, do people usually git clone tensorflow under the $HOME path or somewhere?

why bazel need sudo authority?

@suiyuan2009 Maybe because I git clone the tensorflow under "/home/common/" path which is made by myself. This path belongs to root user.

This looks like a simple Unix permissions issue, and not a TensorFlow bug.  Please reopen if this is not the case.

I am having the same issue. Steps i followed

bazel clean
./configure on cpu
bazel build --config op gives t:

kush@kush-Lenovo-B40-80:~/machine_learning/deeplearning/tensorflow$ bazel build --config=op
Warning: ignoring LD_PRELOAD in environment.
WARNING: Config values are not defined in any .rc file: op
INFO: Found 0 targets...
INFO: Elapsed time: 2.365s, Critical Path: 0.03s

Now when i try to run
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg

kush@kush-Lenovo-B40-80:~/machine_learning/deeplearning/tensorflow$ bazel-bin/tensorflow/tools/pip_package/build_pip_package
bash: bazel-bin/tensorflow/tools/pip_package/build_pip_package: No such file or directory

But then i tried sudo it gives
sudo: bazel-bin/tensorflow/tools/pip_package/build_pip_package: command not found
bazel is installed in my system. i have checked it.
Can anyone help me on this?


text:
Colab TPU not working in 2.3.0-rc1, "InvalidArgumentError: NodeDef expected inputs 'string' do not match 0 inputs specified"

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Google Colab

TensorFlow installed from (source or binary): Installed using !pip install --upgrade tensorflow==2.3.0-rc1

TensorFlow version (use command below): 2.3.0-rc1

Python version: 3, colab default

GPU model and memory: Colab TPU

Describe the current behavior
When I try to initialize the TPU using
tpu = tf.distribute.cluster_resolver.TPUClusterResolver()
tf.config.experimental_connect_to_cluster(tpu)
tf.tpu.experimental.initialize_tpu_system(tpu)

I get this error
INFO:tensorflow:2.3.0-rc1
INFO:absl:Entering into master device scope: /job:worker/replica:0/task:0/device:CPU:0
INFO:tensorflow:Initializing the TPU system: grpc://10.6.63.186:8470
INFO:tensorflow:Initializing the TPU system: grpc://10.6.63.186:8470
INFO:tensorflow:Clearing out eager caches
INFO:tensorflow:Clearing out eager caches
---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-4-5aaeec42239c> in <module>()
     19 tpu = tf.distribute.cluster_resolver.TPUClusterResolver()
     20 tf.config.experimental_connect_to_cluster(tpu)
---> 21 tf.tpu.experimental.initialize_tpu_system(tpu)
     22 strategy = tf.distribute.experimental.TPUStrategy(tpu)
     23 print('strategy.num_replicas_in_sync', strategy.num_replicas_in_sync)

3 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/tpu/tpu_strategy_util.py in initialize_tpu_system(cluster_resolver)
    109     context.context()._clear_caches()  # pylint: disable=protected-access
    110
--> 111     serialized_topology = output.numpy()
    112
    113     # TODO(b/134094971): Remove this when lazy tensor copy in multi-device

/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py in numpy(self)
   1061     """
   1062     # TODO(slebedev): Consider avoiding a copy for non-CPU or remote tensors.
-> 1063     maybe_arr = self._numpy()  # pylint: disable=protected-access
   1064     return maybe_arr.copy() if isinstance(maybe_arr, np.ndarray) else maybe_arr
   1065

/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py in _numpy(self)
   1029       return self._numpy_internal()
   1030     except core._NotOkStatusException as e:  # pylint: disable=protected-access
-> 1031       six.raise_from(core._status_to_exception(e.code, e.message), None)  # pylint: disable=protected-access
   1032
   1033   @property

/usr/local/lib/python3.6/dist-packages/six.py in raise_from(value, from_value)

InvalidArgumentError: NodeDef expected inputs 'string' do not match 0 inputs specified; Op<name=_Send; signature=tensor:T -> ; attr=T:type; attr=tensor_name:string; attr=send_device:string; attr=send_device_incarnation:int; attr=recv_device:string; attr=client_terminated:bool,default=false; is_stateful=true>; NodeDef: {{node _Send}}

Describe the expected behavior
TPU should initialize like in 2.2.0
Standalone code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate
the problem. If possible, please share a link to Colab/Jupyter/any notebook.
Here is a github colab gist
https://colab.research.google.com/gist/Santosh-Gupta/a4d5459c13b4bb54ace08993f5b174da/tpu_try_2-3-0-rc1.ipynb

@Santosh-Gupta,
This query has already been answered in another issue. Please take a look at this comment from a member of the TensorFlow team. Thanks!

authors: Santosh-Gupta amahendrakar


text:
Can't loading model from checkpoint if use adam optimizer

Tensorflow version:v2.3.0-rc2-23-gb36436b087 2.3.0
Ubuntu 18.04
===================
code:
model.compile(optimizer="adam",
                 loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
                 metrics=["accuracy"])

checkpoint_dir = 'training/'
checkpoint_path = os.path.join(checkpoint_dir,"cp-{epoch:04d}.ckpt")
checkpoint_path = os.path.join(checkpoint_path)
cp_callback = tf.keras.callbacks.ModelCheckpoint(
    filepath=checkpoint_path,
    verbose=1,
    save_weights_only=True,
    save_freq=1000)

model.load_weights(tf.train.latest_checkpoint(os.path.dirname(checkpoint_dir))).expect_partial()

If the optimizer = "adam" , load_weights get a error:
Make sure the slot variables are created under the same strategy scope. This may happen if you're restoring from a checkpoint outside the scope
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-0.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-1.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-1.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-1.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-1.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-1.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-2.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-2.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-3.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-3.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-3.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-3.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-3.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-4.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-4.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-5.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-5.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-5.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-5.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-5.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-6.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-6.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-7.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-7.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-7.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-7.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-7.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-8.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-8.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-9.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-9.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-9.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-9.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-9.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-10.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-10.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-11.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-11.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-11.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-11.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-11.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-12.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-12.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-13.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-13.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-13.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-13.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-13.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-14.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-14.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-15.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-15.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-15.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-15.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-15.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-16.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-16.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-17.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-17.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-17.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-17.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-17.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-18.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-18.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-19.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-19.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-19.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-19.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-19.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-20.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-20.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-21.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-21.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-21.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-21.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-21.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-22.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-22.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-23.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-23.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-23.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-23.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-23.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-24.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-24.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-25.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-25.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-25.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-25.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-25.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-26.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-26.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-27.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-27.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-27.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-27.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-27.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-28.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-28.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-29.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-29.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-29.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-29.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-29.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-30.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-30.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-31.axis
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-31.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-31.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-31.moving_mean
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-31.moving_variance
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-32.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-32.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.iter
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_1
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_2
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.decay
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.learning_rate
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-0.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-0.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-1.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-1.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-2.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-2.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-3.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-3.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-4.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-4.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-5.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-5.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-6.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-6.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-7.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-7.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-8.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-8.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-9.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-9.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-10.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-10.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-11.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-11.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-12.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-12.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-13.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-13.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-14.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-14.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-15.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-15.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-16.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-16.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-17.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-17.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-18.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-18.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-19.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-19.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-20.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-20.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-21.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-21.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-22.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-22.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-23.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-23.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-24.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-24.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-25.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-25.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-26.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-26.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-27.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-27.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-28.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-28.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-29.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-29.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-30.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-30.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-31.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-31.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-32.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-32.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-0.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-0.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-1.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-1.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-2.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-2.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-3.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-3.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-4.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-4.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-5.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-5.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-6.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-6.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-7.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-7.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-8.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-8.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-9.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-9.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-10.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-10.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-11.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-11.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-12.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-12.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-13.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-13.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-14.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-14.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-15.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-15.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-16.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-16.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-17.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-17.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-18.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-18.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-19.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-19.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-20.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-20.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-21.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-21.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-22.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-22.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-23.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-23.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-24.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-24.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-25.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-25.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-26.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-26.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-27.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-27.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-28.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-28.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-29.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-29.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-30.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-30.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-31.gamma
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-31.beta
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-32.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-32.bias
WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.

But if optimizer ='sgm' ,everything is OK.

@smallworld-network-wupeng
Please provide with complete stand alone code for us to replicate, as i am facing a different error with the code shared.
Please find the gist here, or if possible share colab gist with error reported.

full code:

from alfred.dl.tf.common import mute_tf
mute_tf()
import pathlib
import tensorflow as tf
from tensorflow import keras
from datetime import datetime
import numpy as np
import os
from pynvml import *
import multiprocessing
import argparse
import time

os.environ['TF_GPU_THREAD_MODE'] = 'gpu_private'

parser = argparse.ArgumentParser(description="Image classification")
parser.add_argument('--input_dir',type=str,default="data_dir",help='input image directory,default is data_dir')
parser.add_argument('--epochs',type=int,default=500,help='epochs times,default is 500')
parser.add_argument('--batch_size',type=int,default=64,help='batch size,default is 64')
parser.add_argument('--from_cp',type=bool,default=False,help='training from last checkpoint')
args = parser.parse_args()

#获取显卡显存信息
l_time = time.strftime("%Y-%m-%d-%H-%M-%S", time.localtime())
gpus = tf.config.experimental.list_physical_devices('GPU')
nvmlInit()
device_count = nvmlDeviceGetCount()
for i in range(device_count):
    handle = nvmlDeviceGetHandleByIndex(i)
    mem = nvmlDeviceGetMemoryInfo(handle).total // 1024 //1024
    tf.config.experimental.set_virtual_device_configuration(gpus[i], [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=mem*0.9)])
    print('GPU Total Memory:{} limit: {}'.format(mem,mem*0.9))
#获取CPU核心数
cpu_core = multiprocessing.cpu_count()
print('CPU total core: {}'.format(cpu_core))

IMG_HEIGHT = 224
IMG_WIDTH = 224
NUM_WORKERS = cpu_core
batch_size = args.batch_size
epochs = args.epochs
data_dir = args.input_dir
from_cp = args.from_cp

AUTOTUNE = tf.data.experimental.AUTOTUNE

#gpus = tf.config.experimental.list_physical_devices('GPU')
#tf.config.experimental.set_virtual_device_configuration(gpus[0], [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=mem_0*0.9)])
#tf.config.experimental.set_virtual_device_configuration(gpus[1], [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=mem_0*0.9)])

mirrored_strategy = tf.distribute.MirroredStrategy()

train_dir = pathlib.Path(os.path.join(data_dir,'train'))
valid_dir = pathlib.Path(os.path.join(data_dir,'valid'))
image_count = len(list(train_dir.glob('*/*.jpg'))) + len(list(valid_dir.glob('*/*.jpg')))
class_names = np.array(sorted([item.name for item in train_dir.glob('*')]))
TRAIN_STEPS_PER_EPOCH = np.ceil((image_count*0.8/batch_size)-1)
VAL_STEPS_PER_EPOCH = np.ceil((image_count*0.2/batch_size)-1)

def pre_process_image(image):

    image = tf.image.random_flip_left_right(image)
    #image = tf.image.random_hue(image, max_delta=0.05)
    #image = tf.image.random_contrast(image, lower=0.3, upper=1.0)
    image = tf.image.random_brightness(image, max_delta=0.2)
    #image = tf.image.random_saturation(image, lower=0.0, upper=2.0)
    #image = tf.image.rot90(image, np.random.randint(1,4))
    return image

def get_label(file_path):
    # convert the path to a list of path components
    parts = tf.strings.split(file_path, os.path.sep)
    # The second to last is the class-directory
    one_hot = parts[-2] == class_names
    print(class_names)
    # Integer encode the label
    return tf.argmax(one_hot)

def decode_img(img):
    # convert the compressed string to a 3D uint8 tensor
    img = tf.image.decode_jpeg(img, channels=3)
    #img = pre_process_image(img)
    # resize the image to the desired size
    return tf.image.resize(img, [IMG_HEIGHT, IMG_WIDTH])

def process_path(file_path):
    label = get_label(file_path)
    # load the raw data from the file as a string
    img = tf.io.read_file(file_path)
    img = decode_img(img)
    return img, label

def get_dataset(subset):
    if subset == 'train':
        list_ds = tf.data.Dataset.list_files(str(train_dir/'*/*'),shuffle=False)
    if subset == 'valid':
        list_ds = tf.data.Dataset.list_files(str(valid_dir/'*/*'),shuffle=False)
    list_ds = list_ds.shuffle(image_count, reshuffle_each_iteration=False)

    print(subset)
    '''
    log_txt = open(subset+'.txt', 'w')
    for d in ds:
        log_txt.write(str(d) + '\n')
    log_txt.close()
    '''
    ds = list_ds.map(process_path,num_parallel_calls=cpu_core)
    #ds = ds.cache()
    ds = ds.shuffle(buffer_size=1000)
    ds = ds.batch(batch_size)
    ds = ds.prefetch(buffer_size=cpu_core).repeat()
    return ds

train_ds = get_dataset('train')
valid_ds = get_dataset('valid')

#print(AUTOTUNE)
#train_ds = train_ds.cache().prefetch(buffer_size=AUTOTUNE)
#valid_ds = valid_ds.cache().prefetch(buffer_size=AUTOTUNE)

with mirrored_strategy.scope():
    tinydarknet = keras.Sequential([
        keras.layers.Conv2D(16, (3, 3), strides=[1, 1], padding="same", input_shape=(IMG_HEIGHT,IMG_WIDTH, 3)),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.MaxPooling2D((2, 2), strides=(2, 2)),
        keras.layers.Conv2D(32, (3, 3), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.MaxPooling2D((2, 2), strides=(2, 2)),
        keras.layers.Conv2D(16, (1, 1), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(128, (3, 3), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(16, (1, 1), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(128, (3, 3), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.MaxPooling2D((2, 2), strides=(2, 2)),
        keras.layers.Conv2D(32, (1, 1), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(256, (3, 3), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(32, (1, 1), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(256, (3, 3), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.MaxPooling2D((2, 2), strides=(2, 2)),
        keras.layers.Conv2D(64, (1, 1), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(512, (3, 3), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(64, (1, 1), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(512, (3, 3), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(128, (1, 1), strides=[1, 1], padding="same"),
        keras.layers.BatchNormalization(),
        keras.layers.LeakyReLU(alpha=0.1),
        keras.layers.Conv2D(1000, (1, 1)),
        keras.layers.BatchNormalization(),
        keras.layers.AveragePooling2D(),
        keras.layers.Flatten(),
        keras.layers.Dense(1,activation="sigmoid")
    ])

    tinydarknet.compile(optimizer="adam",
                 loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
                 metrics=["accuracy"])

logs = "logs/" + datetime.now().strftime("%Y%m%d-%H%M%S")
tboard_callback = tf.keras.callbacks.TensorBoard(log_dir = logs,
                                                 histogram_freq = 1,
                                                 profile_batch = '500,520')

#save checkpoint

checkpoint_dir = 'training/'
checkpoint_path = os.path.join(checkpoint_dir,"cp-{epoch:04d}.ckpt")
checkpoint_path = os.path.join(checkpoint_path)
cp_callback = tf.keras.callbacks.ModelCheckpoint(
    filepath=checkpoint_path,
    verbose=1,
    save_weights_only=True,
    save_freq=1000)

if from_cp:
    print("checkpoint_path::" + os.path.dirname(checkpoint_dir))
    tinydarknet.load_weights(tf.train.latest_checkpoint(os.path.dirname(checkpoint_dir))).expect_partial()
    #checkpoint = tf.train.Checkpoint(myModel=tinydarknet)
    #checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))

history = tinydarknet.fit(
    train_ds,
    steps_per_epoch=TRAIN_STEPS_PER_EPOCH,
    epochs=epochs,
    validation_data=valid_ds,
    validation_steps=VAL_STEPS_PER_EPOCH,
    workers=NUM_WORKERS,
    callbacks = [tboard_callback,cp_callback]
)

acc = history.history['accuracy']
val_acc = history.history['val_accuracy']

loss=history.history['loss']
val_loss=history.history['val_loss']

epochs_range = range(epochs)

#save model
tinydarknet.save(l_time+"-keras_model")

converter = tf.lite.TFLiteConverter.from_keras_model(tinydarknet)
tflite_model = converter.convert()

with open(l_time+"-tinydarknet.tflite", "w+b") as fp:
    fp.write(tflite_model)
    fp.flush()

@smallworld-network-wupeng
I ran the code shared and face different error,please find the gist here.


text:
tools:summarize_graph fails to open TFHub models due to parsing errors

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 20.04
TensorFlow installed from (source or binary): source
TensorFlow version (use command below): v2.4.0-rc4-71-g582c8d236cb 2.4.0
Python version: Python 3.8.6 (default, Oct  9 2020, 11:59:16)
Bazel version (if compiling from source): Bazelisk version: v1.6.1
GCC/Compiler version (if compiling from source): [GCC 9.3.0] on linux
CUDA/cuDNN version: not relevant
GPU model and memory: not relevant

Describe the current behavior
summarize_graph is not able to load downloaded and extracted models from TFHub e.g. https://tfhub.dev/deepmind/biggan-128/2 due to parsing errors.
$ bazel-bin/tensorflow/tools/graph_transforms/summarize_graph --in_graph='/media/paulgavrikov/DATA/noCNN/model_conversion/biggan/compare_gan_ssgan_128x128_1/saved_model.pb'  --print_structure=true
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/text_format.cc:324] Error parsing text-format tensorflow.GraphDef: 1:1: Invalid control characters encountered in text.
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/text_format.cc:324] Error parsing text-format tensorflow.GraphDef: 1:2: Interpreting non ascii codepoint 154.
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/text_format.cc:324] Error parsing text-format tensorflow.GraphDef: 1:2: Expected identifier, got: �
2021-01-17 12:13:33.563477: E tensorflow/tools/graph_transforms/summarize_graph_main.cc:320] Loading graph '/media/paulgavrikov/DATA/noCNN/model_conversion/biggan/compare_gan_ssgan_128x128_1/saved_model.pb' failed with Can't parse /media/paulgavrikov/DATA/noCNN/model_conversion/biggan/compare_gan_ssgan_128x128_1/saved_model.pb as binary proto
	 (both text and binary parsing failed for file /media/paulgavrikov/DATA/noCNN/model_conversion/biggan/compare_gan_ssgan_128x128_1/saved_model.pb)
2021-01-17 12:13:33.563504: E tensorflow/tools/graph_transforms/summarize_graph_main.cc:322] usage: bazel-bin/tensorflow/tools/graph_transforms/summarize_graph

Describe the expected behavior
The tool should load the model

@paulgavrikov
This issue is more suitable for TensorFlow Hub repo. Please post it on hub repo from here. Thanks!

@paulgavrikov
This issue is more suitable for TensorFlow Hub repo. Please post it on hub repo from here. Thanks!

authors: paulgavrikov ravikyram


text:
Descriptions of tf.RaggedTensor's methods 'xx' are unclear

Documentation bug for: https://www.tensorflow.org/api_docs/python/tf/RaggedTensor?hl=en#and
Description of issue:
Usage examples are unmatched
tf.RaggedTensor has a number of class methods, for example, __abs__, __add__, ... The usage examples for these methods are unmatched. For instance, the example code for __add__ does not contain any ragged tensors, instead the examples are all normal tensors (of type tf.Tensor).
The example code is:
a = tf.constant([True])
b = tf.constant([False])
tf.math.logical_and(a, b)

That is not expected for a tf.RaggedTensor.__add__() method.

@ArrowIntoTheSky
In order to expedite the trouble-shooting process here,Could you please fill the issue template, and provide more details on this?
Thanks!

@sushreebarsa Sorry for the late response. I have updated the description of the issue to include more details. Can you please look into this?

Added a PR #55222 for the fix.

@ArrowIntoTheSky,
A RaggedTensor is a subclass of Tensor. If the method is not redefined, the parent class method is used. Hence the examples given are in dense tensor. Thanks!

Reopening as https://github.com/tensorflow/tensorflow/issues/53828#issuecomment-1101038091 is a comment on #55222 which could be edited to incorporate the comment and add proper ragged tensor description

Fix provided here https://github.com/tensorflow/tensorflow/commit/d0356e76730aee601a91f342d32a9671a487230a contains the example of Ragged Tensors, it is reflecting in the document as well here https://www.tensorflow.org/api_docs/python/tf/RaggedTensor?hl=en&version=nightly#abs

authors: ArrowIntoTheSky gadagashwini mihaimaruseac sachinprasadhs sushreebarsa


text:
pthread cond_wait deadlock on assignment via tf.while_loop

State:

tf 0.12RC0 (installed via pip URL)
Ubuntu 16.04
cuda 8.0 w/cudnn 5.1

Ran python through gdb and it looks like there is a deadlock?
import tensorflow as tf

with tf.Session() as sess:
    index = tf.constant(0)
    values_0 = [tf.constant(0, dtype=tf.int32), tf.zeros([10, 100])]
    cond = lambda i, val: tf.less(i, 10)
    def _update(i, val):
        row = tf.random_normal([1, 100])
        update = val + tf.concat(0, [tf.zeros([i, 100]),
                                     row,
                                     tf.zeros([10-1-i, 100])])
       # Just to verify sizing
        print 'update = ', update.get_shape().as_list(), \
            ' | values_0 = ', values_0[1].get_shape().as_list()

        return [i+i, update]

    _, values_t = tf.while_loop(cond, _update, values_0,
                                parallel_iterations=10)

    print sess.run(values_t)

Is this expected behavior? I want to update different rows of the matrix in parallel and this seemed somewhat logical to me. Since each row is independent it "shouldn't" really collide even if it is in parallel.

Automatically closing due to lack of recent activity. We hope that you were able to resolve it on your own. However, since this is a support issue rather than a bug or feature request, you will probably get more information by posting it on StackOverflow.

@tfboyd : not sure I would say this is a support issue.
tf.while_loop should not deadlock. This is undefined behavior.

I likely chose the wrong words it was a canned response.  I understand it can be frustrating as teams follow different processes and we did not document our action.  We essentially declared a "bug bankruptcy" and closed most issues older than ~6 months after a quick review.
That said if you are seeing this behavior on TF 1.1 or 1.2, I am happy to reopen and try to push it back through Triage again.
While wordy, I cannot stress enough no malice is intended and I read each bug in detail before closing.  I knew I would make a few mistakes and counted on people responding back.  Thank you for the ping.

authors: jramapuram tfboyd


text:
Bazel：command not found

I have install the bazel v0.3.2 per the instruction and it works well on the terminal.but when I was running ./configure, it comes with a result of "Bazel :command not found"

COuld you post more information about what is the operating system, how was bazel installed, also run these commands from your terminal and post the outputs:
python -c "import tensorflow; print(tensorflow.__version__)"
git rev-parse HEAD
bazel version

If you are using Linux or Mac OS, also post this:
echo  ${PATH}
./configure

I am not sure what is the equivalent for PATH in Windows

This looks like a problem with your installation of bazel.
Not really a TensorFlow issue.
Please make sure in the same terminal you are running ./configure, you can run bazel version.
You might need to add the location of bazel binary to your path.
Since this is a bazel installation issue, I am closing it. Please make sure to follow bazel installation documentation to make sure it is installed properly.

authors: gunan louisway zafartahirov


text:
Windows uname -s might return different values

configure#L14 checks if the system is Windows, but doesn't check uname -s values correctly according to the wiki page.
Unless not supported, I think other Windows SDE's should be included in the check.

@drpngx could you please check if #6872 solves it

Yeah, I think that's fine. I'm not sure how far you can go with the build after that. The configuration that works is the cmake build.

I wonder if there is a Windows system with cygwin and uwin in the Jenkins, because if there are none, bugs will not be caught even if enabled in the configure

No, there isn't. We only check the cmake build.

Assigning to @drpngx since it seems that he's reviewing the PR
(Thanks @zafartahirov for the report and PR)

For windows, we currently do native compilation with cmake, (no cygwin).
Bazel setup for windows uses msys2, which is the closest thing we have to cygwin, but that also uses visual studio to compile TF.

Fix submitted for this particular check.

authors: asimshankar drpngx gunan zafartahirov