text:
Neural Shuffle-Exchange Seq2Seq model

Added Neural Shuffle-Exchange model for Seq2Seq problems as described in paper and original code.

Hi @OptimatiumFeles - Can you please fix the lint errors below before we take this in?
Lint errors found:
Invalid import order.
Diff:
    from __future__ import absolute_import
    from __future__ import division
    from __future__ import print_function
    import six
    from tensor2tensor.layers import modalities
    from tensor2tensor.models import basic
    from tensor2tensor.models import bytenet
    from tensor2tensor.models import distillation
    from tensor2tensor.models import evolved_transformer
    from tensor2tensor.models import image_transformer
    from tensor2tensor.models import image_transformer_2d
    from tensor2tensor.models import lstm
    from tensor2tensor.models import mtf_image_transformer
    from tensor2tensor.models import mtf_resnet
    from tensor2tensor.models import mtf_transformer
    from tensor2tensor.models import mtf_transformer2
    from tensor2tensor.models import neural_assistant
    from tensor2tensor.models import neural_gpu
    from tensor2tensor.models import resnet
    from tensor2tensor.models import revnet
    from tensor2tensor.models import shake_shake
    from tensor2tensor.models import slicenet
    from tensor2tensor.models import text_cnn
    from tensor2tensor.models import transformer
    from tensor2tensor.models import vanilla_gan
    from tensor2tensor.models import xception
    from tensor2tensor.models.neural_architecture_search import nas_model
    from tensor2tensor.models.research import adafactor_experiments
    from tensor2tensor.models.research import aligned
    from tensor2tensor.models.research import attention_lm
    from tensor2tensor.models.research import attention_lm_moe
    from tensor2tensor.models.research import autoencoders
    from tensor2tensor.models.research import cycle_gan
    from tensor2tensor.models.research import gene_expression
    from tensor2tensor.models.research import glow
    from tensor2tensor.models.research import lm_experiments
    from tensor2tensor.models.research import moe_experiments
    from tensor2tensor.models.research import multiquery_paper
    from tensor2tensor.models.research import neural_stack
    from tensor2tensor.models.research import rl
+   from tensor2tensor.models.research import shuffle_network
    from tensor2tensor.models.research import similarity_transformer
    from tensor2tensor.models.research import super_lm
    from tensor2tensor.models.research import transformer_moe
    from tensor2tensor.models.research import transformer_nat
    from tensor2tensor.models.research import transformer_parallel
    from tensor2tensor.models.research import transformer_revnet
    from tensor2tensor.models.research import transformer_sketch
    from tensor2tensor.models.research import transformer_symshard
    from tensor2tensor.models.research import transformer_vae
    from tensor2tensor.models.research import universal_transformer
    from tensor2tensor.models.research import vqa_attention
    from tensor2tensor.models.research import vqa_recurrent_self_attention
    from tensor2tensor.models.research import vqa_self_attention
-   from tensor2tensor.models.research import shuffle_network
    from tensor2tensor.models.video import basic_deterministic
    from tensor2tensor.models.video import basic_recurrent
    from tensor2tensor.models.video import basic_stochastic
    from tensor2tensor.models.video import emily
    from tensor2tensor.models.video import epva
    from tensor2tensor.models.video import next_frame_glow
    from tensor2tensor.models.video import savp
    from tensor2tensor.models.video import sv2p
    from tensor2tensor.utils import registry
Copy and pastable ordering:
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import six
from tensor2tensor.layers import modalities
from tensor2tensor.models import basic
from tensor2tensor.models import bytenet
from tensor2tensor.models import distillation
from tensor2tensor.models import evolved_transformer
from tensor2tensor.models import image_transformer
from tensor2tensor.models import image_transformer_2d
from tensor2tensor.models import lstm
from tensor2tensor.models import mtf_image_transformer
from tensor2tensor.models import mtf_resnet
from tensor2tensor.models import mtf_transformer
from tensor2tensor.models import mtf_transformer2
from tensor2tensor.models import neural_assistant
from tensor2tensor.models import neural_gpu
from tensor2tensor.models import resnet
from tensor2tensor.models import revnet
from tensor2tensor.models import shake_shake
from tensor2tensor.models import slicenet
from tensor2tensor.models import text_cnn
from tensor2tensor.models import transformer
from tensor2tensor.models import vanilla_gan
from tensor2tensor.models import xception
from tensor2tensor.models.neural_architecture_search import nas_model
from tensor2tensor.models.research import adafactor_experiments
from tensor2tensor.models.research import aligned
from tensor2tensor.models.research import attention_lm
from tensor2tensor.models.research import attention_lm_moe
from tensor2tensor.models.research import autoencoders
from tensor2tensor.models.research import cycle_gan
from tensor2tensor.models.research import gene_expression
from tensor2tensor.models.research import glow
from tensor2tensor.models.research import lm_experiments
from tensor2tensor.models.research import moe_experiments
from tensor2tensor.models.research import multiquery_paper
from tensor2tensor.models.research import neural_stack
from tensor2tensor.models.research import rl
from tensor2tensor.models.research import shuffle_network
from tensor2tensor.models.research import similarity_transformer
from tensor2tensor.models.research import super_lm
from tensor2tensor.models.research import transformer_moe
from tensor2tensor.models.research import transformer_nat
from tensor2tensor.models.research import transformer_parallel
from tensor2tensor.models.research import transformer_revnet
from tensor2tensor.models.research import transformer_sketch
from tensor2tensor.models.research import transformer_symshard
from tensor2tensor.models.research import transformer_vae
from tensor2tensor.models.research import universal_transformer
from tensor2tensor.models.research import vqa_attention
from tensor2tensor.models.research import vqa_recurrent_self_attention
from tensor2tensor.models.research import vqa_self_attention
from tensor2tensor.models.video import basic_deterministic
from tensor2tensor.models.video import basic_recurrent
from tensor2tensor.models.video import basic_stochastic
from tensor2tensor.models.video import emily
from tensor2tensor.models.video import epva
from tensor2tensor.models.video import next_frame_glow
from tensor2tensor.models.video import savp
from tensor2tensor.models.video import sv2p
from tensor2tensor.utils import registry
[g-bad-import-order]
	//depot/google3/third_party/py/tensor2tensor/models/__init__.py:61

Need 1 blank line before def [g-wrong-blank-lines]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:194

Line too long (100/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:57

Line too long (88/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:59

Line too long (83/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:61

Line too long (82/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:81

Line too long (102/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:82

Line too long (84/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:85

Line too long (91/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:92

Line too long (82/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:95

Line too long (107/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:99

Line too long (107/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:100

Line too long (112/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:116

Line too long (88/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:122

Line too long (92/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:123

Line too long (91/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:130

Line too long (82/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:137

Line too long (82/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:148

Line too long (90/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:155

Line too long (90/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:162

Wrong hanging indentation (add 2 spaces).
        forward_step,
        ^ |
[bad-continuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:168

Wrong hanging indentation (add 2 spaces).
        tf.range(0, n_bits),
        ^ |
[bad-continuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:169

Wrong hanging indentation (add 2 spaces).
        initializer=(block_out, residuals_queue),
        ^ |
[bad-continuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:170

Wrong hanging indentation (add 2 spaces).
        parallel_iterations=1,
        ^ |
[bad-continuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:171

Wrong hanging indentation (add 2 spaces).
        swap_memory=True
        ^ |
[bad-continuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:172

Line too long (95/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:175

Wrong hanging indentation (add 2 spaces).
        reverse_step,
        ^ |
[bad-continuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:179

Wrong hanging indentation (add 2 spaces).
        tf.range(n_bits, n_bits * 2),
        ^ |
[bad-continuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:180

Wrong hanging indentation (add 2 spaces).
        initializer=(forward_last, residuals_queue),
        ^ |
[bad-continuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:181

Wrong hanging indentation (add 2 spaces).
        parallel_iterations=1,
        ^ |
[bad-continuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:182

Wrong hanging indentation (add 2 spaces).
        swap_memory=True
        ^ |
[bad-continuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:183

Line too long (114/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:189

Line too long (95/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:209

Line too long (98/80) [line-too-long]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:210

There should not be a space before a one-line docstring summary [g-space-before-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:1

One-line docstring summary should end with ".", ".)", "?", or "!" [g-short-docstring-punctuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:1

One-line docstring summary should be followed by a blank line [g-no-space-after-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:1

Extraneous space in one-line docstring [g-short-docstring-space]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:19

One-line docstring summary should end with ".", ".)", "?", or "!" [g-short-docstring-punctuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:19

One-line docstring summary should end with ".", ".)", "?", or "!" [g-short-docstring-punctuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:29

One-line docstring summary should end with ".", ".)", "?", or "!" [g-short-docstring-punctuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:39

Missing function docstring [missing-docstring]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:53-63

Missing class docstring [missing-docstring]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:66-138

Old-style class defined. [old-style-class]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:66-138

There should not be a space before a one-line docstring summary [g-space-before-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:78

One-line docstring summary should end with ".", ".)", "?", or "!" [g-short-docstring-punctuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:78

Args section is missing, does not match arg list or is badly indented.

Expected: inputs, suffix, bias_start, in_units, out_units

Found:

Missing: inputs, suffix, bias_start, in_units, out_units

Unexpected:
[g-doc-args]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:77

There should not be a space before a one-line docstring summary [g-space-before-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:86

One-line docstring summary should end with ".", ".)", "?", or "!" [g-short-docstring-punctuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:86

Args section is missing, does not match arg list or is badly indented.

Expected: inputs, suffix, bias_start_reset, in_units, out_units

Found:

Missing: inputs, suffix, bias_start_reset, in_units, out_units

Unexpected:
[g-doc-args]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:85

Use a != b rather than not a == b [g-comparison-negation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:129

Missing class docstring [missing-docstring]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:192-227

There should not be a space before a one-line docstring summary [g-space-before-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:195

One-line docstring summary should be followed by a blank line [g-no-space-after-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:195

Args section is missing, does not match arg list or is badly indented.

Expected: features

Found:

Missing: features

Unexpected:
[g-doc-args]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:194

There should not be a space before a one-line docstring summary [g-space-before-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:214

One-line docstring summary should end with ".", ".)", "?", or "!" [g-short-docstring-punctuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:214

One-line docstring summary should be followed by a blank line [g-no-space-after-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:214

Args section is missing, does not match arg list or is badly indented.

Expected: logits, features

Found:

Missing: logits, features

Unexpected:
[g-doc-args]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:213

Missing function docstring [missing-docstring]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:231-252

Invalid import order.
Diff:
    from __future__ import absolute_import
    from __future__ import division
    from __future__ import print_function
-   import tensorflow as tf
    from tensor2tensor.layers import common_hparams
    from tensor2tensor.utils import registry
    from tensor2tensor.utils import t2t_model
+   import tensorflow as tf
Copy and pastable ordering:
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

from tensor2tensor.layers import common_hparams
from tensor2tensor.utils import registry
from tensor2tensor.utils import t2t_model
import tensorflow as tf
[g-bad-import-order]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:13

Thanks for the PR though :-)

Hi @afrozenator, thank you for taking a look.
I've fixed all mentioned linting errors, hope that it'll be ok now.

A few more, we're almost there:
Lint errors found:
There should not be a space before a one-line docstring summary [g-space-before-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:1

One-line docstring summary should end with ".", ".)", "?", or "!" [g-short-docstring-punctuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:1

One-line docstring summary should be followed by a blank line [g-no-space-after-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:1

There should not be a space before a one-line docstring summary [g-space-before-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:21

There should not be a space before a one-line docstring summary [g-space-before-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:42

There should not be a space before a one-line docstring summary [g-space-before-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:62

One-line docstring summary should end with ".", ".)", "?", or "!" [g-short-docstring-punctuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:62

One-line docstring summary should be followed by a blank line [g-no-space-after-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:62

There should not be a space before a one-line docstring summary [g-space-before-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:84

There should not be a space before a one-line docstring summary [g-space-before-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:98

There should not be a space before a one-line docstring summary [g-space-before-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:130

One-line docstring summary should be followed by a blank line [g-no-space-after-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:130

There should not be a space before a one-line docstring summary [g-space-before-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:136

One-line docstring summary should end with ".", ".)", "?", or "!" [g-short-docstring-punctuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:136

One-line docstring summary should be followed by a blank line [g-no-space-after-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:136

One-line docstring summary should end with ".", ".)", "?", or "!" [g-short-docstring-punctuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:152

There should not be a space before a one-line docstring summary [g-space-before-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:172

One-line docstring summary should end with ".", ".)", "?", or "!" [g-short-docstring-punctuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:172

There should not be a space before a one-line docstring summary [g-space-before-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:207

One-line docstring summary should end with ".", ".)", "?", or "!" [g-short-docstring-punctuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:207

There should not be a space before a one-line docstring summary [g-space-before-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:249

There should not be a space before a one-line docstring summary [g-space-before-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:266

One-line docstring summary should be followed by a blank line [g-no-space-after-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:266

There should not be a space before a one-line docstring summary [g-space-before-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:334

One-line docstring summary should be followed by a blank line [g-no-space-after-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:334

There should not be a space before a one-line docstring summary [g-space-before-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:362

One-line docstring summary should end with ".", ".)", "?", or "!" [g-short-docstring-punctuation]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:362

One-line docstring summary should be followed by a blank line [g-no-space-after-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:362

There should not be a space before a one-line docstring summary [g-space-before-docstring-summary]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:384

Missing Returns section in docstring, or wrong choice of Returns/Yields [g-doc-return-or-yield]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:398

Invalid import order.
Diff:
    from __future__ import absolute_import
    from __future__ import division
    from __future__ import print_function
-   import tensorflow as tf
    from tensor2tensor.layers import common_hparams
    from tensor2tensor.utils import registry
    from tensor2tensor.utils import t2t_model
+   import tensorflow as tf
Copy and pastable ordering:
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

from tensor2tensor.layers import common_hparams
from tensor2tensor.utils import registry
from tensor2tensor.utils import t2t_model
import tensorflow as tf
[g-bad-import-order]
	//depot/google3/third_party/py/tensor2tensor/models/research/shuffle_network.py:15

Errors should be gone now.

Many many thanks for contributing and fixing the lints @OptimatiumFeles !

authors: OptimatiumFeles afrozenator


text:
RNN Cell: different device placement for variables and ops

I want to use RNN cell on multiple GPUs. For maximum performance, I want to place RNN cell variables on CPU and operations (matmul, tanh, etc) on GPU, just like in cifar10 multi gpu example. However, the current implementation of RNN cell seems to only allow same device placement for both variables and ops. Could you enable different device placement?

The existing implementation focuses equally on being readable/understandable/hackable, and being performant. Please see the documentation here for how to place variables/operations on specific devices. Closing this issue as the functionality to do what you want is already present in TensorFlow.

Thanks, I just want to confirm one thing: using the raw RNNCell as it is, I cannot place vars and ops in different devices, is this correct? I understand that I can easily make changes to it myself to give different placement.

That's correct.

@seominjoon Hi, How did you seperate vars and ops in different devices for an RNN cell?

I am also having this issue where I want to place RNN vars on CPU but execute the cell ops on GPU. having to hack around source code is subprime.  it would be nice if vars and ops had a kwarg for soft device placement. then we could do:
to create a cell
cell = tf.contrib.rnn.BasicRNNCell(celldim,device='/cpu:')
to run the cell ops
output,state = cell(x_batch[:,time_step,:], cell_state,device='/gpu:0')

source: gharchive/issue

authors: YiMX andrebeu keveman seominjoon

repo: tensorflow/tensorflow

license_type: permissive

license_source: github-api


text:
The tensorflow tested builds advices wrong bazel version for tensorflow-gpu r1.8

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Linux Ubuntu 16.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
No
TensorFlow installed from (source or binary):
Source
TensorFlow version (use command below):
r1.8
Python version:
2.7
Bazel version (if compiling from source):
0.9.0
GCC/Compiler version (if compiling from source):
4.8.5
CUDA/cuDNN version:
9.0/7.1
GPU model and memory:
GeForce GTX 960M
Exact command to reproduce:
./configure
The output is:
You have bazel 0.9.0 installed.
Please upgrade your bazel installation to version 0.10.0 or higher to build TensorFlow!
Configuration finished

Describe the problem
I tried to build tf r1.8 with gpu with bazel 0.17 but got errors then I checked here to make sure I got the right version, but saw that I need bazel 0.9.0. When trying it with bazel 0.9.0 I get the error above.
I did not find where I could edit the tf website itself to make a PR, so I just opened this issue.
Source code / logs
None

I think 0.17 was a typo.
@lamberta the problem is for Linux, tensorflow-gpu, version 1.8.0, we have bazel 0.9.0.
But it needs to be bazel 0.10.
Where do we update that?

authors: Kaju-Bubanja gunan


text:
tf.compat.v2.summary.scalar has no attribute 'summary_scope'

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes, I have a minimal test file
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): ubuntu 18.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): source
TensorFlow version (use command below):
tf.version
'2.0.0-alpha0'
Python version:
$ python3
Python 3.6.7 (default, Oct 22 2018, 11:32:17)
Bazel version (if compiling from source): binary installer version 0.23
GCC/Compiler version (if compiling from source):
$ gcc --version
gcc (Ubuntu 7.3.0-27ubuntu1~18.04) 7.3.0
CUDA/cuDNN version: Not supported
GPU model and memory: Not supported

You can collect some of this information using our environment capture script
You can also obtain the TensorFlow version with
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
Describe the current behavior
tf.compat.v2.summary.scalar("stddev",_st,step=count,description='stddev')
is being processed as an _api.v1.summary
Describe the expected behavior
tf.compat.v2.summary.scalar("stddev",_st,step=count,description='stddev')
is processed as an _api.v2.summary
Code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate the problem.
tf_sdata = tf.convert_to_tensor(sdata,dtype=tf.float32,name="tf_sdata")
tfdata_stddev = tf.data.Dataset.from_tensor_slices(tf_sdata[:,0])

tfstddev_iter = tfdata_stddev.__iter__()

stats_writer = tf.compat.v2.summary.create_file_writer(logs_path,max_queue=10,flush_millis=120000,
        filename_suffix='.stats',name='stats_writer')

count = 1
for _st in tfstddev_iter:
    tf.compat.v2.summary.scalar("stddev",_st,step=count,description='stddev')
    count += 1

Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
Traceback (most recent call last):
File "./tf2summary_bug.py", line 71, in
tf.compat.v2.summary.scalar("stddev",_st,step=count,description='stddev')
File "/usr/lib/python3/dist-packages/tensorboard/plugins/scalar/summary_v2.py", line 55, in scalar
with tf.summary.summary_scope(
AttributeError: module 'tensorflow._api.v1.summary' has no attribute 'summary_scope'

I think you need to mark the stats_writer as default. Either by calling stats_writer.set_as_default() (do not directly add this call to the creation of the SummaryFileWriter [#25976]) or by setting up the summary scope like this:
with stats_writer.as_default():
    count = 1
    for _st in tfstddev_iter:
        tf.compat.v2.summary.scalar("stddev",_st,step=count,description='stddev')
        count += 1

meyerjo, thank you for your comment.
I tried the context: with stats_writer.as_default():
This produced the same Traceback:
Traceback (most recent call last):
File "./tf2summary_bug.py", line 72, in
tf.compat.v2.summary.scalar("stddev",_st,step=count,description='stddev')
File "/usr/lib/python3/dist-packages/tensorboard/plugins/scalar/summary_v2.py", line 55, in scalar
with tf.summary.summary_scope(
AttributeError: module 'tensorflow._api.v1.summary' has no attribute 'summary_scope'
I do not see why this would need a custom summary scope. The problem seems to be the code is executing in the summary_v2.py code as expected and encountering a v1.summary object that is not expected. I could be wrong. Just thinking out loud.

authors: kshaeffer meyerjo


text:
piecewise_constant_decay not supported by xla: Segmentation fault (core dumped)

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CENTOS 7
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: NA
TensorFlow installed from (source or binary): source
TensorFlow version (use command below): 1.13
Python version: 3.6
Bazel version (if compiling from source): 0.21
GCC/Compiler version (if compiling from source): 6.3.0
CUDA/cuDNN version: NA
GPU model and memory: NA

I am trying to run a model which uses tf.train.piecewise_constant_decay with xla.  Previously this was supported in tf1.10 when I last tried. I am now trying the same script in tf1.13.
I receive a segmentation fault when I run it now. After digging around it seems to be because of type DT_STRING which the const kernel doesn't seem to support.
When I see what operations are supported by the Const kernel,
(bazel run -c opt -- tensorflow/compiler/tf2xla:tf2xla_supported_ops --device=XLA_CPU_JIT) it shows
Const | dtype={bool,complex64,double,float,half,int32,int64,int8,qint32,qint8,quint8,string,uint32,uint64,uint8}

To reproduce this bug, a simple example is:
import tensorflow as tf
from tensorflow.contrib.compiler.xla import compile
import numpy as np

LABELS = 10
HIDDEN_SIZE = 256
BATCH_SIZE = 64
FEAT_DIM = 784

xla_flag = True

def model_fn(features, labels):
    with tf.variable_scope("fc", use_resource=True):
        net = tf.keras.layers.Dense(units=HIDDEN_SIZE,
                                    activation=tf.nn.relu)(features)
        logits = tf.keras.layers.Dense(units=LABELS)(net)
        cross_entropy = tf.reduce_mean(
            tf.nn.softmax_cross_entropy_with_logits(labels=labels,
                                                    logits=logits))
        global_step = tf.train.get_or_create_global_step()
        boundaries = [100000, 110000]
        values = [1.0, 0.5, 0.1]
        learning_rate = tf.train.piecewise_constant_decay(global_step, boundaries, values)
        train_step = tf.train.GradientDescentOptimizer(
            learning_rate, name="final_node").minimize(cross_entropy)
        with tf.control_dependencies([train_step]):
            return tf.identity(cross_entropy, name="results")

x = tf.placeholder(tf.float32, [BATCH_SIZE, FEAT_DIM], name='x')
y = tf.placeholder(tf.float32, [BATCH_SIZE, LABELS], name='y')

if xla_flag:
    (xla_loss,) = compile(model_fn, [x,y])
else:
    xla_loss = model_fn(x,y)

with tf.Session() as sess:
    sess.run(tf.initialize_all_variables())
    ans = sess.run(xla_loss,feed_dict={x:np.ones((BATCH_SIZE,FEAT_DIM)),y:np.ones((BATCH_SIZE,LABELS))})

While looking at the implementation, it seems like it could be because of the name_scope defined at https://github.com/tensorflow/tensorflow/blob/r1.13/tensorflow/python/training/learning_rate_decay_v2.py#L169
Please let me know if there is a solution, or if I am doing something wrong from my side.

@jlebar

Would you be willing to check with a nightly build?  I believe we may have fixed this recently.

It looks like it was fixed in this pr https://github.com/tensorflow/tensorflow/commit/0b85b901b256b7e771b0597a7b3184ce7bf2319e

Closing this issue since its resolved. Feel free to reopen if have any further problems. Thanks!

authors: jlebar vishalsubbiah ymodak


text:
Unable to run native XLA AOT tests

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS Mojave 10.14.4
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A
TensorFlow installed from (source or binary): source
TensorFlow version: r1.12
Python version: python 3.6.3
Bazel version (if compiling from source): 0.18.1
GCC/Compiler version (if compiling from source): Apple LLVM version 10.0.1 (clang-1001.0.46.4)
CUDA/cuDNN version: N/A
GPU model and memory: N/A

Describe the problem
There is a set of native tests in tensorflow/compiler/aot/tests. I am not able to run them using bazel because of a linking error while building the rule tfcompile_test in .../tests/BUILD. Specifically, tfcompile_test depends upon test_graph_tfmatmulandadd_with_profiling and test_graph_tfmatmulandadd which fail to link.
Code to reproduce the issue
In an untouched Tensorflow repo with r1.12 checkout and bazel 0.18.1 installed, run
bazel build //tensorflow/compiler/aot/tests:tfcompile_test
Describe the current behavior
Error message is as follows:
/Users/willxujun/code/tensorflow/tensorflow/compiler/aot/tests/BUILD:225:1: Linking of rule '//tensorflow/compiler/aot/tests:tfcompile_test' failed (Exit 1) Undefined symbols for architecture x86_64:  "___tfcompile_MatMulAndAddCompWithProfiling_HloProfilePrinterData_protobuf_array_contents", referenced from:       MatMulAndAddCompWithProfiling::StaticData()::'lambda'()::operator()() const in tfcompile_test.o   "___tfcompile_MatMulAndAddComp_ProgramShape_protobuf_array_contents", referenced from:       MatMulAndAddComp::StaticData()::'lambda'()::operator()() const in tfcompile_test.o ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) Target //tensorflow/compiler/aot/tests:tfcompile_test failed to build
The problem seems to stem from constructing MatMulAndAddCompWithProfiling or MatMulAndAddComp objects in tfcompile_test.cc. These two classes are graphs compiled by XLA.
Disabling the relevant tests in tfcompile_test.cc makes the issue go away.
Alternatively, in .../aot/tests/BUILD, removing tfcompile_flags from test_graph_tfmatmulandadd, and setting enable_xla_hlo_profiling to False in test_graph_tfmatmulandadd_with_profiling makes the issue go away.
This problem seems to happen to only macOS. I could run the same command without issues in Ubuntu 16.04.
This problem seems to be related to the linking of dependencies. I could successfully run the individual tf_librarys and get the compiled graph classes in bazel_genfiles/, and both undefined symbols are "extern C" variables found in the graphs' header files, and it seems the root definition is not found somehow. As a result, the graph objects cannot be constructed.
Describe the expected behavior
In Ubuntu 16.04, build succeeds with the following:
INFO: Build completed successfully, 3560 total actions
Then a tfcompile_test executable would be produced in bazel_bin/tensorflow/compiler/aot/tests, which can be run successfully.
Other info / logs
N/A

@willxujun Request you to please follow the link, as per the reference the bazel version for 1.12 is 0.15.0

Re-run using Bazel 0.15.0 exhibits identical behaviour

TensorFlow version: r1.12

Does this reproduce at git head?

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

authors: jlebar muddham willxujun ymodak


text:
Android: Failed to resolve: tensorflow-lite

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOS 10.13.2 HighSierra

TensorFlow installed from (source or binary): gradle dependency 'org.tensorflow:tensorflow-lite:0.0.0-nightly'

TensorFlow version: 0.0.0-nightly

Android Studio : 3.2.1

Android compile / targetSdkVersion  = 28

Gradle Version: 4.6

Describe the problem
Tensorflow Lite dependency is not resolved
Provide the exact sequence of commands / steps that you executed before running into the problem
Run assembleDebug command ->
_Could not find tensorflow-lite.aar (org.tensorflow:tensorflow-lite:0.0.0-nightly).
Searched in the following locations:
https://jcenter.bintray.com/org/tensorflow/tensorflow-lite/0.0.0-nightly/tensorflow-lite-0.0.0-nightly.aar_
Any other info / logs

I am receiving this error as well. Assuming this is a nightly build error, but will wait for feedback.

You can try using the explicit version 1.14.0
https://bintray.com/google/tensorflow/tensorflow-lite#

@joelromanpr Thanks it worked!

I can easily replace the dependency, but there are some classes missing in this version
error: cannot find symbol public class TFLiteObjectDetectionAPIModel implements Classifier {
error: cannot find symbol private Interpreter tfLite;
error: cannot find symbo public List

@peterszaszlateral that looks like a different issue. TFLiteObjectDetectionAPIModel is a part of the sample, not of tensorflow-lite.aar.
I'm not able to reproduce the issue with the nightly build, though. Are you able to successfully the build the classification example? https://github.com/tensorflow/examples/tree/master/lite/examples/image_classification/android

@jdduke It all started working again last week at some point + the sample app you specified is also building successfully. Thanks for the help!

Glad it's working!

Hello, I am running times error: “Caused by: org.gradle.internal.resolve.ModuleVersionNotFoundException:  Could not find org. Tensorflow: tensorflow - lite - select - tf - ops: 0.0.0 - the nightly - the SNAPSHOT.
"Is there a good solution?

您可以尝试使用显式版本 1.14.0
https://bintray.com/google/tensorflow/tensorflow-lite#

Hello, how to use this version? Do I need to manually download and import it or do I need to change the library address in the project file?

authors: Ryenum itimmisriis jdduke joelromanpr peterszaszlateral siddhi-droid


text:
[tf.keras] Model.metrics_names no longer works in TensorFlow 2.2.0

System information

Have I written custom code (as opposed to using a stock
example script provided in TensorFlow): Y
OS Platform and Distribution (e.g.,
Linux Ubuntu 16.04):  Mac OS
TensorFlow version (use command below): 2.2.0rc0
Python version: 3.7

Describe the current behavior
Model.metrics_names returns an empty list (see below example) as of TensorFlow 2.2.0 which is not the case in TensorFlow 2.1.0
This is import because it enables people to check that monitored quantities in callbacks exist at compile time. Simple typos val_lss can be caught before training starts as opposed to the end of the first epoch.
Describe the expected behavior
Model.metric returns metrics and loss names (see below example).
Standalone code to reproduce the issue
import tensorflow as tf

inputs = tf.keras.layers.Input(shape=(3,))
outputs = tf.keras.layers.Dense(2, name="out_1")(inputs)
net = tf.keras.models.Model(inputs=inputs, outputs=outputs)
net.compile(optimizer="Adam", loss="mse", metrics=["mae"])

net.metrics_names == []  # TensorFlow 2.2.0rc2
net.metrics_names == ["loss", "mae"] # TensorFlow 2.1.0 ["loss", "mae"]

@briannemsick, In TF 2.2 , you need call fit or train_on_batch before you call  net.metrics_names.

That seems like a design regression. The eventual names are known at compile time.
Knowing the metrics names before called model.fit can prevent issues that would normally occur before the end of the epoch.

Firstly we need to make proper layer in say sequential neural network as if before training the model if done the layering for doing the particular job we can easily use metrics_names as after the fit function or transform function can be used

This change also breaks several callbacks that rely on being able to read the metrics and metrics_names with the on_train_begin() hook. In my case, it breaks the TensorFlow Addons tqdm_progress_bar callback. Any idea as to how I can implement a work-around to this issue. I had to upgrade to TF-2.2-r1 to fix a memory leak bug introduced in TF-2.1 and now I'm stuck with this instead.

@briannemsick Thank you for pointing this out. This is definitely a breaking change but intended. Sorry about the lack of information on this on release notes and docs. We have fixed that now.
We refactored compile/fit quite a bit and with that we made functional and subclassed model behaviors consistent. Because of this you will be able to use some of these model properties only after your model has been trained/evaluated on actual data.

In this case, a warning or exception when these attributes are accessed before train/eval feels warranted.

@pavithrasv Can I ask what is the reasoning behind this change? You mention refactoring to make things consistent, but how was the previous behavior inconsistent?

@briannemsick We are looking into adding a warning as well. Thank you.
@bjtho08 Earlier you will be able to see metrics_name after compile call for functional model but not for subclassed model.

@briannemsick there are constraints because of which we will not be able to add a warning but we have updated the docs with a note and examples to clarify the behavior.


text:
Much worse performance when using mixed precision training (using tensorflow.keras policy)

Hi!
I'm training an image generative resnet-based model using MSE loss and Adam optimizer. I'm using the API tf.data to feed the model as fast as I can. Once I achieved a 100% GPU usage during training, I'm seeking for speeding up the process even more by using the experimental mixed_precision Keras API. I've previously tested this API by running this tutorial and it works as expected, getting an increase in performance of aproximatley 2x. However, when trying it with my model, the performance is much worse than using regular fp32 computation. I suspect there is something wrong with my model, however, I've tried to follow all the TensorCore performance considerations in order to correclty set the model's hyperparameters, such as channels and batch size being divisible by 8 and input images cast to float16 without any success
System information

OS Platform and Distribution: Linux Ubuntu 18.04.4 LTS
TensorFlow installed from (container): tensorflow/tensorflow:nightly-gpu
TensorFlow version (use command below): Tensorflow:  2.3.0-dev20200514
Python version: 3.6.9
CUDA/cuDNN version: V10.1.243 / CUDNN=7.6.4.38-1
GPU model and memory: RTX 2080 8GB

Describe the current behavior
Worse performance using mixed precision Keras API than using regular fp32 computation.
Describe the expected behavior
Performance increases of aproximately 1.5x compared to fp32 computation.
Standalone code to reproduce the issue
Colab
Other info / logs
`Compute dtype: float16
Variable dtype: float32
Keras:  2.2.4-tf
Tensorflow:  2.3.0-dev20200514
Image format:  channels_last
2020-05-14 19:34:27.261423: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-14 19:34:27.261615: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1544] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2080 computeCapability: 7.5
coreClock: 1.815GHz coreCount: 46 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.23GiB/s
2020-05-14 19:34:27.261632: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-14 19:34:27.261652: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-14 19:34:27.261663: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-14 19:34:27.261672: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-14 19:34:27.261682: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-14 19:34:27.261692: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-14 19:34:27.261701: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-14 19:34:27.261732: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-14 19:34:27.261931: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-14 19:34:27.262108: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1686] Adding visible gpu devices: 0
2020-05-14 19:34:27.262119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1085] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-14 19:34:27.262124: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1091]      0
2020-05-14 19:34:27.262127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1104] 0:   N
2020-05-14 19:34:27.262168: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-14 19:34:27.262368: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-14 19:34:27.262551: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1230] Created TensorFlow device (/device:GPU:0 with 6300 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5)
Device : /device:CPU:0
type : CPU
desc :
Device : /device:XLA_CPU:0
type : XLA_CPU
desc :device: XLA_CPU device
Device : /device:XLA_GPU:0
type : XLA_GPU
desc :device: XLA_GPU device
Device : /device:GPU:0
type : GPU
desc :device: 0, name: GeForce RTX 2080, pci bus id: 0000:01:00.0, compute capability: 7.5
2020-05-14 19:34:31.803789: I tensorflow/core/profiler/lib/profiler_session.cc:163] Profiler session started.
2020-05-14 19:34:31.803820: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1365] Profiler found 1 GPUs
2020-05-14 19:34:31.804185: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcupti.so.10.1
2020-05-14 19:34:31.873411: I tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1487] CUPTI activity buffer flushed
Epoch 1/3
2020-05-14 19:34:39.768955: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-14 19:34:40.485835: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-14 19:34:43.260095: I tensorflow/compiler/jit/xla_compilation_cache.cc:314] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.`
Thank you in advance.

@alvarobasi,
On running the code, I am facing an error stating NameError: name 'list_files' is not defined. Please find the gist of it here.
Could you please provide the complete code and the dataset you are using. Thanks!

Sure, here it is. The dataset I´m using is the Celeba dataset loaded using tfds.load().

Hi! Any updates about this? I've noticed you got an error while getting the dataset inside the colab. In my case I got the database just by downloading it from this link , reading all the files and storing their paths in an array and then loading them to a tf.Dataset by using from_tensor_slices method

Can you update the colab to get the Dataset so that it can run successfully? Ideally, you would use tfds or just create a random numpy array with np.rand to generate the data, since all we are debugging here is performance.

Sure, now it runs successfully. You can find the colab here. I created a random sample of aprox 2k matrices with the same shape as the celeba dataset ones. I ran this again on my RTX 2080 and it gave me the same results: much worse performance with mixed_float16...
Thanks in advance.

@alvarobasi, thank you for filing this issue. The slowdown is caused by the last Conv2D in the model. As a temporary workaround, try passing dtype="float32" to the final Conv2D layer of your model.
I can reproduce with a standalone program:
import tensorflow as tf
import time

def bench(dtype, data_format):
  if data_format == 'NHWC':
    x = tf.random.normal((16, 128, 128, 256), dtype=dtype)
    f = tf.random.normal((9, 9, 256, 3), dtype=dtype)
  else:
    x = tf.random.normal((16, 256, 128, 128), dtype=dtype)
    f = tf.random.normal((9, 9, 256, 3), dtype=dtype)

  p = tf.constant(0.)

  # Warmup
  tf.nn.conv2d(x, f, strides=[1, 1, 1, 1], padding='SAME',
               data_format=data_format)

  start = time.time()
  for _ in range(10):
    tf.nn.conv2d(x, f, strides=[1, 1, 1, 1], padding='SAME',
                 data_format=data_format)
  # Synchronize GPU by sending result of computation to CPU
  p = p + 1.
  p.numpy()

  end = time.time()
  print('time for %s %s: %s' % (dtype, data_format, end - start))

bench('float32', 'NHWC')
bench('float32', 'NCHW')
bench('float16', 'NHWC')
bench('float16', 'NCHW')

On my Titan V, the output is:
time for float32 NHWC: 0.06822085380554199
time for float32 NCHW: 0.059490203857421875
time for float16 NHWC: 0.9011242389678955
time for float16 NCHW: 0.05471491813659668

The results are a bit misleading, as TensorFlow silently converts a float32 NHWC Conv2D to NCHW. If I remove the conversion from that file, float32 NHWC is about as slow as float16 NHWC.
@nluehr is it expected for NHWC to be 15x slower than NCHW in some cases with float16 Volta/Turing? If so, what cases is NHWC slower? I think the best approach is to try to detect when NHWC is slower and if so, fallback to NCHW. Note tensor cores are not used since the filter is not a multiple of 8.

@reedwm I ran your script with cudnn v7 on GV100 and I can get similar perf. In addition, I also gave a shot with cudnn v8 and got:
time for float32 NHWC: 0.060801029205322266
time for float32 NCHW: 0.05140805244445801
time for float16 NHWC: 0.09125709533691406
time for float16 NCHW: 0.04851174354553223

This time, the fp16 NHWC becomes <2x slower than NCHW.
After some digging, I found this is more about the algorithm selection rather than tensor core selection. With cudnn v8, by using the tensor core we can see the perf is boosted from 50ms to 9ms for both NHWC/NCHW cudnn algo0. However, the NCHW supports algo5 (an FFT based algorithm) and it costs only 4.8ms. Unfortunately, this algo5 is not supported in NHWC. So, the above fp16 perf comparison is virtually NHWC algo0 vs NCHW algo5.
Similarly, since there is implicit format conversion for fp32, the NHWC/NCHW will both rely on the NCHW algo5 and attain ~5ms.
AFAIK, the FFT algo doesn't support Tensor core. So, tensor core shouldn't matter, when FFT algo is preferred. For this case, I suggest users upgrade to cudnn v8 and at least check if the gap can be closed.

Same poor performance on 2080 Ti + TF compiled from source
time for float32 NHWC: 0.12609028816223145
time for float32 NCHW: 0.08841919898986816
time for float16 NHWC: 1.4761300086975098
time for float16 NCHW: 0.06844806671142578

TF 2.4 uses cudnn 8 and the performance is similar with what @kaixih reported. So this issue is far less severe, as now float16 is less than 2x slower, while before it was between 15x and 20x slower. However, 2x slower is still significant.
Perhaps we should use autotune both NCHW and NHWC and choose the faster one, transposing the input and output if necessary. Ideally grappler would optimizer the data format based on the performance, but I think implementing that is infeasible.
/CC @sanjoy

Yes, for this case, the cuDNN doesn't provide faster NHWC fp16 kernel than its NCHW counterparts. We need to sync with our cuDNN team to see if adding new algorithm or adding auto transformation is possible.

/CC @timshen91

@alvarobasi,
Performance seems to be fine when tf.nn.conv2d is replaced with tf.keras.layers.Conv2D, with Tensorflow Version 2.5. Please find the Gist of the working code. Thanks!

authors: alvarobasi amahendrakar kaixih reedwm rmothukuru sanjoy shkarupa-alex


text:
Quantization Aware Training model has weird inference behavior

Hi,
I have a pretrained detection model I trained in Tensorflow 2.3 with fp32 precision. I used this model's weights as initial pretrained weights for Quantization Aware Training (QAT). During training I could see that training converged and gave logical predictions (I visualized results during training).
When trying to load with Tensorflow the Quantization Aware Training weights for inference the model behaves differently: When applying in inference the same preprocessing + normalization of mean/std, the predictions' probabilities are always around 0.25. Also, all the predictions' bounding boxes in inference are really small: around 15 pixels height/width, while during training I could see predictions having logical bounding boxes that are with varied dimensions and with much higher probabilities. This behavior appears even on images that were used for training so it's not related to overfitting issues.
Technical constraints for my implementation:

I used Tensorflow's tf.quantization.quantize_and_dequantize in specific places in my model: After the activation function which appears in the following pattern: Conv2D --> BatchNormalization --> Activation. I attach here an example for such pattern:

from tensorflow.keras.layers import Conv2D, ZeroPadding2D, LeakyReLU, ReLU, BatchNormalization

def MyConv(x, filters, kernel_size, strides=1, batch_norm=True, is_quantized=False):
    if strides == 1:
        padding = "same"
    else:
        x = ZeroPadding2D(((1, 0), (1, 0)))(x)  # top left half-padding
        padding = "valid"
    x = Conv2D(filters=filters, kernel_size=kernel_size,
               strides=strides, padding=padding,
               use_bias=not batch_norm, kernel_regularizer=l2(0.0005))(x)

    if batch_norm:
        x = BatchNormalization()(x)
        if is_quantized:
            # We'll use The leaky version of Relu6 for stability of low-precision computations
            # We define the Leaky Relu using Relu for conversion constrains to TensorRT later
            x = ReLU(negative_slope=0.1, max_value=6)(x)
        else:
            x = LeakyReLU(alpha=0.1)(x)
    if is_quantized:
        # Using tf syntax of quantization - applying quantization only on the conv op, after the activation
        x = tf.quantization.quantize_and_dequantize(x, input_min=-64, input_max=64, range_given=False)
        # The input_min / input_max should be ignored due to range_given=False
    return x

This function is called when building the model both for training and inference.
The reason I place the quantize_and_dequantize op specifically after the activation function is because this model should be converter to TensorRT later- their QAT guidelines note that this is the place to apply quantization op for QAT, due to the layer fusion TensorRT applies in the conversion and optimization process.
I tried also applying QAT using the Tensorflow Model Optimization Toolkit as describes here, but this resulted with a model that includes unsupported nodes for TensorRT such as ConvInteger. A TF model that uses quantize_and_dequantize produces model that is possible to convert to TRT, but has the issues I described above due to the irregular behavior in TF during inference. Here is an example to the quant ops in Netron for the inference model:

This is a Tensorflow-related issue since the issue appears during inference in Tensorflow with QAT weights.
2) I create the model for inference using the following command:
tf.keras.backend.set_learning_phase(1)
self.model = self._build_model()

where build_model() calls internally also the function I attached here which contains the quantization ops after every Activation function.
Is there any additional command that needs to be applied in inference when using tf.quantize.quantize_and_dequantize ? I remember in TF 1.x there was tf.contrib.create_eval_graph, but I didn't see anything similar to it in TF 2.3.
I prefer a solution that lets me still use tf.quantize.quantize_and_dequantize due to the TensorRT constraints I described (since I can't use Model Optimization Toolkit's quantize_model).
System information

OS Platform and Distribution: Linux Ubuntu 18.04
TensorFlow version: 2.3.0
Python version: 3.6

@xhae  could you triage this issue?

@Xhark Can you take the issue?

Another note, I would like to know if this is a possible cause for issues: When I inspect the onnx converted model from my QAT tensorflow model, I can see the following for all QuantizeLinear and DequantizeLinear nodes: x_scale values are always ~0.5039 (x_zero_point is 0, as it should be). I would have expected this quantization scale to be different for all layers since this is a learnable parameter for quantization op. Is it ok?

I think I've found the issue- it's related to the way I loaded the weights with the relevant flags.
Previously I loaded the weights like this:
keras_model.load_weights(transform_weight_file, by_name=True)

Apparently the weights weren't loaded properly, and no warning or error was raised.
I changed it to:
keras_model.load_weights(transform_weight_file, by_name=False, skip_mismatch=False)

And now the inference values make sense for the QAT model both in probabilities and bounding boxes dimensions.

Hi Would you please give me more context about how did you apply QAT? (code, how much step you train, etc...)
x_scale should changed during the training. (but depends on dataset)
Hi @daverim, do you have any context about TensorRT supports?

It seems this is happening due to a tf2onnx issue- the range_given=False argument is currently ignored:
https://github.com/onnx/tensorflow-onnx/issues/1438#issuecomment-814516723
So the settings in Tensorflow were OK.

Seems that this is the issue -- looks like the scale is calculated from input_min = -64, input_max = 64 (64/127 = .503937). Curious if the rest of this workflow is working correctly?

The rest of the workflow is working correctly :) Only the scale was calculated in an wanted way

Hi,
Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.
The Tensorflow team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.
Please follow the release notes to stay up to date with the latest developments which are happening in the Tensorflow space.

authors: Xhark abattery daverim israelwei tilakrayal xhae