text:
Tensorflow 1.5.0 breaking previously built models

Hello!
I've recently updated to tensorflow version 1.5.0, and suddenly receive an error, that I can't decipher, for code that worked before (in version 1.4.1): Cannot use 'transducer_training/while/rnn/strided_slice' as input to 'gradients/transducer_training/while/rnn/while/Select_1_grad/Select/f_acc' because 'transducer_training/while/rnn/strided_slice' is in a while loop
I've also tried using the softmax_cross_entropy_with_logits function, but that still produced the same error. Here's the stackoverflow post, in case its a coding mistake on my part.
The model is a seq2seq variation.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
TensorFlow installed from (source or binary): Binary
TensorFlow version (use command below): 1.5.0 (previous 1.4.1)
Python version: 2.7
Bazel version (if compiling from source): N/A
GCC/Compiler version (if compiling from source): N/A
CUDA/cuDNN version: N/A
GPU model and memory: N/A
Exact command to reproduce: Copy, paste and run the code

import tensorflow as tf
from tensorflow.contrib.rnn import LSTMCell, LSTMStateTuple
from tensorflow.python.layers import core as layers_core

# NOTE: Time major

# ---------------- Constants Manager ----------------------------
class ConstantsManager(object):
    def __init__(self, input_dimensions, input_embedding_size, inputs_embedded, encoder_hidden_units,
                 transducer_hidden_units, vocab_ids, input_block_size, beam_width):
        assert transducer_hidden_units == encoder_hidden_units, 'Encoder and transducer have to have the same amount' \
                                                                'of hidden units'
        self.input_dimensions = input_dimensions
        self.vocab_ids = vocab_ids
        self.E_SYMBOL = len(self.vocab_ids)
        self.vocab_ids.append('E_SYMBOL')
        self.GO_SYMBOL = len(self.vocab_ids)
        self.vocab_ids.append('GO_SYMBOL')
        self.vocab_size = len(self.vocab_ids)
        self.input_embedding_size = input_embedding_size
        self.inputs_embedded = inputs_embedded
        self.encoder_hidden_units = encoder_hidden_units
        self.transducer_hidden_units = transducer_hidden_units
        self.input_block_size = input_block_size
        self.beam_width = beam_width
        self.batch_size = 1  # Cannot be increased, see paper
        self.log_prob_init_value = 0

# ----------------- Model ---------------------------------------

class Model(object):
    def __init__(self, cons_manager):
        self.var_list = []
        self.cons_manager = cons_manager
        self.max_blocks, self.inputs_full_raw, self.transducer_list_outputs, self.start_block, self.encoder_hidden_init,\
            self.trans_hidden_init, self.logits, self.encoder_hidden_state_new, \
            self.transducer_hidden_state_new, self.train_saver = self.build_full_transducer()

        self.targets, self.train_op, self.loss = self.build_training_step()

    def build_full_transducer(self):
        with tf.variable_scope('transducer_training'):

            embeddings = tf.Variable(tf.random_uniform([self.cons_manager.vocab_size,
                                                        self.cons_manager.input_embedding_size], -1.0, 1.0),
                                     dtype=tf.float32,
                                     name='embedding')
            # Inputs
            max_blocks = tf.placeholder(dtype=tf.int32, name='max_blocks')  # total amount of blocks to go through
            if self.cons_manager.inputs_embedded is True:
                input_type = tf.float32
            else:
                input_type = tf.int32
            inputs_full_raw = tf.placeholder(shape=(None, self.cons_manager.batch_size,
                                                    self.cons_manager.input_dimensions), dtype=input_type,
                                             name='inputs_full_raw')  # shape [max_time, 1, input_dims]
            transducer_list_outputs = tf.placeholder(shape=(None,), dtype=tf.int32,
                                                     name='transducer_list_outputs')  # amount to output per block
            start_block = tf.placeholder(dtype=tf.int32, name='transducer_start_block')  # where to start the input

            encoder_hidden_init = tf.placeholder(shape=(2, 1, self.cons_manager.encoder_hidden_units), dtype=tf.float32,
                                                 name='encoder_hidden_init')
            trans_hidden_init = tf.placeholder(shape=(2, 1, self.cons_manager.transducer_hidden_units), dtype=tf.float32,
                                               name='trans_hidden_init')

            # Temporary constants, maybe changed during inference
            end_symbol = tf.get_variable(name='end_symbol',
                                         initializer=tf.constant_initializer(self.cons_manager.vocab_size),
                                         shape=(), dtype=tf.int32)

            # Turn inputs into tensor which is easily readable#

            inputs_full = tf.reshape(inputs_full_raw, shape=[-1, self.cons_manager.input_block_size,
                                                             self.cons_manager.batch_size,
                                                             self.cons_manager.input_dimensions])

            # Outputs
            outputs_ta = tf.TensorArray(dtype=tf.float32, size=max_blocks)

            init_state = (start_block, outputs_ta, encoder_hidden_init, trans_hidden_init)

            # Initiate cells, NOTE: if there is a future error, put these back inside the body function
            encoder_cell = tf.contrib.rnn.LSTMCell(num_units=self.cons_manager.encoder_hidden_units)
            transducer_cell = tf.contrib.rnn.LSTMCell(self.cons_manager.transducer_hidden_units)

            def cond(current_block, outputs_int, encoder_hidden, trans_hidden):
                return current_block < start_block + max_blocks

            def body(current_block, outputs_int, encoder_hidden, trans_hidden):

                # --------------------- ENCODER ----------------------------------------------------------------------
                encoder_inputs = inputs_full[current_block]
                encoder_inputs_length = [tf.shape(encoder_inputs)[0]]
                encoder_hidden_state = encoder_hidden

                if self.cons_manager.inputs_embedded is True:
                    encoder_inputs_embedded = encoder_inputs
                else:
                    encoder_inputs = tf.reshape(encoder_inputs, shape=[-1, self.cons_manager.batch_size])
                    encoder_inputs_embedded = tf.nn.embedding_lookup(embeddings, encoder_inputs)

                # Build model

                # Build previous state
                encoder_hidden_c, encoder_hidden_h = tf.split(encoder_hidden_state, num_or_size_splits=2, axis=0)
                encoder_hidden_c = tf.reshape(encoder_hidden_c, shape=[-1, self.cons_manager.encoder_hidden_units])
                encoder_hidden_h = tf.reshape(encoder_hidden_h, shape=[-1, self.cons_manager.encoder_hidden_units])
                encoder_hidden_state_t = LSTMStateTuple(encoder_hidden_c, encoder_hidden_h)

                #   encoder_outputs: [max_time, batch_size, num_units]
                encoder_outputs, encoder_hidden_state_new = tf.nn.dynamic_rnn(
                    encoder_cell, encoder_inputs_embedded,
                    sequence_length=encoder_inputs_length, time_major=True,
                    dtype=tf.float32, initial_state=encoder_hidden_state_t)

                # Modify output of encoder_hidden_state_new so that it can be fed back in again without problems.
                encoder_hidden_state_new = tf.concat([encoder_hidden_state_new.c, encoder_hidden_state_new.h], axis=0)
                encoder_hidden_state_new = tf.reshape(encoder_hidden_state_new,
                                                      shape=[2, -1, self.cons_manager.encoder_hidden_units])

                # --------------------- TRANSDUCER --------------------------------------------------------------------
                encoder_raw_outputs = encoder_outputs
                # Save/load the state as one tensor, use encoder state as init if this is the first block
                trans_hidden_state = tf.cond(current_block > 0, lambda: trans_hidden, lambda: encoder_hidden_state_new)
                transducer_amount_outputs = transducer_list_outputs[current_block - start_block]

                # Model building
                helper = tf.contrib.seq2seq.GreedyEmbeddingHelper(
                    embedding=embeddings,
                    start_tokens=tf.tile([self.cons_manager.GO_SYMBOL],
                                         [self.cons_manager.batch_size]),  # TODO: check if this looks good
                    end_token=end_symbol)  # vocab size, so that it doesn't prematurely end the decoding

                attention_states = tf.transpose(encoder_raw_outputs,
                                                [1, 0, 2])  # attention_states: [batch_size, max_time, num_units]

                attention_mechanism = tf.contrib.seq2seq.LuongAttention(
                    self.cons_manager.encoder_hidden_units, attention_states)

                decoder_cell = tf.contrib.seq2seq.AttentionWrapper(
                    transducer_cell,
                    attention_mechanism,
                    attention_layer_size=self.cons_manager.transducer_hidden_units)

                projection_layer = layers_core.Dense(self.cons_manager.vocab_size, use_bias=False)

                # Build previous state
                trans_hidden_c, trans_hidden_h = tf.split(trans_hidden_state, num_or_size_splits=2, axis=0)
                trans_hidden_c = tf.reshape(trans_hidden_c, shape=[-1, self.cons_manager.transducer_hidden_units])
                trans_hidden_h = tf.reshape(trans_hidden_h, shape=[-1, self.cons_manager.transducer_hidden_units])
                trans_hidden_state_t = LSTMStateTuple(trans_hidden_c, trans_hidden_h)

                decoder = tf.contrib.seq2seq.BasicDecoder(
                    decoder_cell, helper,
                    decoder_cell.zero_state(1, tf.float32).clone(cell_state=trans_hidden_state_t),
                    output_layer=projection_layer)

                outputs, transducer_hidden_state_new, _ = tf.contrib.seq2seq.dynamic_decode(decoder,
                                                                                output_time_major=True,
                                                                                maximum_iterations=transducer_amount_outputs)
                logits = outputs.rnn_output  # logits of shape [max_time,batch_size,vocab_size]
                decoder_prediction = outputs.sample_id  # For debugging

                # Modify output of transducer_hidden_state_new so that it can be fed back in again without problems.
                transducer_hidden_state_new = tf.concat(
                    [transducer_hidden_state_new[0].c, transducer_hidden_state_new[0].h],
                    axis=0)
                transducer_hidden_state_new = tf.reshape(transducer_hidden_state_new,
                                                         shape=[2, -1, self.cons_manager.transducer_hidden_units])

                # Note the outputs
                outputs_int = outputs_int.write(current_block - start_block, logits)

                return current_block + 1, outputs_int, encoder_hidden_state_new, transducer_hidden_state_new

            _, outputs_final, encoder_hidden_state_new, transducer_hidden_state_new = \
                tf.while_loop(cond, body, init_state, parallel_iterations=1)

            # Process outputs
            outputs = outputs_final.concat()
            logits = tf.reshape(
                outputs,
                shape=(-1, 1, self.cons_manager.vocab_size))  # And now its [max_output_time, batch_size, vocab]

            # For loading the model later on
            logits = tf.identity(logits, name='logits')
            encoder_hidden_state_new = tf.identity(encoder_hidden_state_new, name='encoder_hidden_state_new')
            transducer_hidden_state_new = tf.identity(transducer_hidden_state_new, name='transducer_hidden_state_new')

        train_saver = tf.train.Saver()  # For now save everything

        return max_blocks, inputs_full_raw, transducer_list_outputs, start_block, encoder_hidden_init,\
            trans_hidden_init, logits, encoder_hidden_state_new, transducer_hidden_state_new, train_saver

    def build_training_step(self):
        targets = tf.placeholder(shape=(None,), dtype=tf.int32, name='targets')
        targets_one_hot = tf.one_hot(targets, depth=self.cons_manager.vocab_size, dtype=tf.float32)

        targets_one_hot = tf.Print(targets_one_hot, [targets], message='Targets: ', summarize=10)
        targets_one_hot = tf.Print(targets_one_hot, [tf.argmax(self.logits, axis=2)], message='Argmax: ', summarize=10)

        stepwise_cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=targets_one_hot,
                                                                         logits=self.logits)
        loss = tf.reduce_mean(stepwise_cross_entropy)
        train_op = tf.train.AdamOptimizer().minimize(loss)
        return targets, train_op, loss

constants_manager = ConstantsManager(input_dimensions=1, input_embedding_size=11, inputs_embedded=False,
                                     encoder_hidden_units=100, transducer_hidden_units=100, vocab_ids=[0, 1, 2],
                                     input_block_size=1, beam_width=5)
model = Model(cons_manager=constants_manager)

I can try and make a smaller fail case if needed.
Thanks!
Nikita

Any updates? @ebrevdo could you take a look?

This is the same issue as described in #17199

Let's close one of these bugs as duplicate.
On Sat, Feb 24, 2018, 8:36 AM nikita68 notifications@github.com wrote:

This is the same issue as described in #17199
https://github.com/tensorflow/tensorflow/issues/17199
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/17045#issuecomment-368240594,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABtim637HnMPacDWU_v-kHHDHbShUUAeks5tYDqBgaJpZM4SG_9N
.


text:
concat puts gradients as 0.0

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
TensorFlow installed from (source or binary): source
TensorFlow version (use command below): 1.5.0
Python version: 2.7
Bazel version (if compiling from source): 0.11.0
GCC/Compiler version (if compiling from source): 5.4.0
CUDA/cuDNN version: CuDA 9.0
GPU model and memory: 4Gb, Nvidia GTX960M
Exact command to reproduce:

Describe the problem
When I write delt to get gradients of pi with respect to v, I get the gradients. Same for rem with respect to v. However, when I ask for gradients of z with respect to v, I get a matrix of 0s, which is clearly not the gradient as z is just the concatenation of pi and rem, both of which have non-zero gradients. Why is this happening?
Source code / logs
def latent_space(v):
	# Calculate pi
	with tf.name_scope('pi'):
		def fn(previous_output,current_input):
			[stick,remaining] = previous_output
			i = current_input
			stick = v[:,i]*remaining
			remaining *= (1-v[:,i])
			return [stick,remaining]

		elems = tf.Variable(tf.range(latent-1))
		[pi,rem] = tf.scan(fn,elems,initializer=[tf.ones([bs]),tf.ones([bs])])
		rem = tf.reshape(rem[-1,:],[1,bs])
		z = tf.concat([pi,rem],axis=0)
		z = tf.transpose(z)
		delt = tf.gradients(z,v)
	return z,delt

Nagging Assignee @tatatodd: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @tatatodd: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @tatatodd: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

This question is better asked on StackOverflow since it is not a bug or feature request. There is also a larger community that reads questions there.
If you think we've misinterpreted a bug, please comment again with a clear explanation, as well as all of the information requested in the issue template. Thanks!

source: gharchive/issue

authors: sarthmit tatatodd tensorflowbutler

repo: tensorflow/tensorflow

license_type: permissive

license_source: github-api


text:
Distributed TensorFlow got error message with the MPI collective Ops units test

Have I written custom code:  None
OS Platform and Distribution: Linux Ubuntu 16.04
Open MPI version: 3.0.0
TensorFlow installed from: b'v1.7.0-rc1-816-g1712002ad0' 1.7.0-rc1
Bazel version: 0.11.1
Python version: 3.5.2
GCC version: 6.3.0
CUDA/cuDNN version: N/A
GPU model and memory: N/A
Exact command to reproduce: mpirun -np 1 python mpi_allreduce_test.py under directory tensorflow/tensorflow/contrib/mpi_collectives
I ran the unit test under directory tensorflow/tensorflow/contrib/mpi_collectives and the run command is mpirun -np 1 python mpi_allreduce_test.py . I got the following error message:
Could someone please help me take a look at this? Thanks!
WARNING:tensorflow:From /ec/fm/disks/nrv_algo_home01/langjian/.tf_17/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Use the retry module or similar alternatives.
Traceback (most recent call last):
  File "mpi_allreduce_test.py", line 23, in <module>
    import tensorflow.contrib.mpi_collectives as mpi
  File "$HOME/.tf_17/lib/python3.5/site-packages/tensorflow/contrib/mpi_collectives/__init__.py", line 128, in <module>
    from tensorflow.contrib.mpi_collectives.python.ops.mpi_ops import init
  File "$HOME/.tf_17/lib/python3.5/site-packages/tensorflow/contrib/mpi_collectives/python/ops/mpi_ops.py", line 29, in <module>
    resource_loader.get_path_to_datafile('_mpi_ops.so'))
  File "$HOME/.tf_17/lib/python3.5/site-packages/tensorflow/contrib/util/loader.py", line 55, in load_op_library
    ret = load_library.load_op_library(path)
  File "$HOME/.tf_17/lib/python3.5/site-packages/tensorflow/python/framework/load_library.py", line 58, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename, status)
  File "$HOME/.tf_17/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 516, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: $HOME/.tf_17/lib/python3.5/site-packages/tensorflow/contrib/mpi_collectives/python/ops/_mpi_ops.so: undefined symbol: _ZN10tensorflow7contrib15mpi_collectives7MPITypeIiEEP15ompi_datatype_tv
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[18657,1],0]
  Exit code:    1
--------------------------------------------------------------------------

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

@tensorflowbutler Thanks a lot for the quick response! See the updated issue template below:
Have I written custom code:  None
OS Platform and Distribution: Linux Ubuntu 16.04
Open MPI version: 3.0.0
TensorFlow installed from: b'v1.7.0-rc1-816-g1712002ad0' 1.7.0-rc1
Bazel version: 0.11.1
Python version: 3.5.2
GCC version: 6.3.0
CUDA/cuDNN version: N/A
GPU model and memory: N/A
Exact command to reproduce: mpirun -np 1 python mpi_allreduce_test.py

@jthestness @poxvoculi : Any suggestions here?
It seems to be complaining about not finding ompi_datatype_t* tensorflow::contrib::mpi_collectives::MPIType<int>(), which should be defined in ring.cc.
Just to be clear: You did build from sources using the instructions in contrib/mpi/README.md, right?

Ok, you're going to need to step through the chain of defining symbols across the libraries to see where there is a missing link. Check that the Tensorflow library you built points to the correct MPI library with ldd <path_to_tf_build>/python/_pywrap_tensorflow_internal.so. If that's correct, then check the mpi_collectives library: ldd <path_to_tf_build>/contrib/mpi_collectives/python/ops/_mpi_ops.so and then objdump -t <path_to_tf_build>/contrib/mpi_collectives/python/ops/_mpi_ops.so | grep ompi_datatype_t. The latter should show something like this (note "F" means the function is defined, you probably have a "U" or the functions are missing):
00000000000265f0 g     F .text	0000000000000008              _ZN10tensorflow7contrib15mpi_collectives7MPITypeIxEEP15ompi_datatype_tv
00000000000265e0 g     F .text	0000000000000008              _ZN10tensorflow7contrib15mpi_collectives7MPITypeIiEEP15ompi_datatype_tv
00000000000265d0 g     F .text	0000000000000008              _ZN10tensorflow7contrib15mpi_collectives7MPITypeIfEEP15ompi_datatype_tv

If the function is not defined ("U" or missing), then something went wrong with the TF build. Be sure that you either opt 'yes' when prompted to configure the build with MPI or set the environment variable TF_NEED_MPI=1 before configuring.
Another thing you could try is to roll back to TF 1.5.0, which we have working internally. (We haven't tested 1.7.0-rc1, and unfortunately, the regressions architecture for TF doesn't permit us to run MPI tests, so changes to TF can break the mpi_collectives code).

mpirun -np 1 python mpi_allreduce_test.py

I run mpirun -np 1 python mpi_allreduce_test.py,the test pass;
but when run mpirun -np 3 python mpi_allreduce_test.py
I got error
1: iter 0
rank 2: My output is 330772870.0
rank 2: Our output is 170543409.0
rank 0: My output is 11908270.0
rank 0: Our output is 170543409.0
2: iter 0
0: iter 0
[e92e09614.em21:08680] Read -1, expected 2362084, errno = 38
[e92e09614.em21:08681] Read -1, expected 2362080, errno = 38
[e92e09614.em21:08679] Read -1, expected 2362080, errno = 38
[e92e09614.em21:08680] Read -1, expected 2362080, errno = 38
[e92e09614.em21:08681] Read -1, expected 2362084, errno = 38
[e92e09614.em21:08679] Read -1, expected 2362080, errno = 38
[e92e09614.em21:08680] Read -1, expected 2362080, errno = 38

I run mpirun -np 1 python mpi_allreduce_test.py ;the test pass.
but when I run mpirun -np 3 python mpi_allreduce_test.py, the test failed
1: iter 0
rank 2: My output is 330772870.0
rank 2: Our output is 170543409.0
rank 0: My output is 11908270.0
rank 0: Our output is 170543409.0
2: iter 0
0: iter 0
[e92e09614.em21:08680] Read -1, expected 2362084, errno = 38
[e92e09614.em21:08681] Read -1, expected 2362080, errno = 38
[e92e09614.em21:08679] Read -1, expected 2362080, errno = 38
[e92e09614.em21:08680] Read -1, expected 2362080, errno = 38
[e92e09614.em21:08681] Read -1, expected 2362084, errno = 38
[e92e09614.em21:08679] Read -1, expected 2362080, errno = 38
[e92e09614.em21:08680] Read -1, expected 2362080, errno = 38

Hi @chengdianxuezi: Your issue looks different from the issue in this thread. Can you please create a new TF Github issue and follow TF's issue creation process (i.e., include information about the version of TF and other software that you are using. See here: https://github.com/tensorflow/tensorflow/issues/new)?

I have the same problem.
tensorflow/contrib/mpi_collectives$ mpirun -np 1 python mpi_allreduce_test.py
Traceback (most recent call last):
File "mpi_allreduce_test.py", line 23, in
import tensorflow.contrib.mpi_collectives as mpi
File "/home/research/virtualenvs/tfenv/lib/python3.5/site-packages/tensorflow/contrib/mpi_collectives/init.py", line 128, in
from tensorflow.contrib.mpi_collectives.python.ops.mpi_ops import init
File "/home/research/virtualenvs/tfenv/lib/python3.5/site-packages/tensorflow/contrib/mpi_collectives/python/ops/mpi_ops.py", line 29, in
resource_loader.get_path_to_datafile('_mpi_ops.so'))
File "$HOME/virtualenvs/tfenv/lib/python3.5/site-packages/tensorflow/contrib/util/loader.py", line 56, in load_op_library
ret = load_library.load_op_library(path)
File "$HOME/virtualenvs/tfenv/lib/python3.5/site-packages/tensorflow/python/framework/load_library.py", line 61, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: $HOME/virtualenvs/tfenv/lib/python3.5/site-packages/tensorflow/contrib/mpi_collectives/python/ops/_mpi_ops.so: undefined symbol: _ZN10tensorflow7contrib15mpi_collectives7MPITypeIiEEP15ompi_datatype_tv
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.

mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:
Process name: [[35887,1],0]
Exit code:    1
when I check ompi_datatype_t in _mpi_ops.so, I got following output:
$ objdump -t $HOME/virtualenvs/tfenv/lib/python3.5/site-packages/tensorflow/contrib/mpi_collectives/python/ops/_mpi_ops.so | grep ompi_datatype_t
0000000000000000         UND	0000000000000000              _ZN10tensorflow7contrib15mpi_collectives7MPITypeIiEEP15ompi_datatype_tv
0000000000000000         UND	0000000000000000              _ZN10tensorflow7contrib15mpi_collectives7MPITypeIxEEP15ompi_datatype_tv
0000000000000000         UND	0000000000000000              _ZN10tensorflow7contrib15mpi_collectives7MPITypeIfEEP15ompi_datatype_tv

I have ubuntu 16.04,
tensorflow branch r1.13, (only mpi is enabled, all other options are disabled on build)
python 3.5,
gcc (Ubuntu 4.8.5-4ubuntu2) 4.8.5,
tested with both bazel 0.19 and 0.21.
tested with OpenMPI 2.1.6, 3.1.2, 4.0.1
not sure how to proceed. Any help would be really appreciated.

@jianyinglang ,
We see that you are using older version of tensorflow (1.x) which is not actively supported. We recommend that you upgrade to latest stable version of tensorflow 2.6.0 and let us know if the issue still persists in newer versions .Thanks!

authors: ahmet-uyar asimshankar chengdianxuezi jianyinglang jthestness tensorflowbutler tilakrayal


text:
MonitoredTrainingSession is not handling the dummy_QueueRunner cleanly?

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):  Linux Fedora Red Hat Enterprise Linux Server release 7.5 (Maipo)
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 1.7.0
Python version: 3.6.3
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: CUDA9.0/ CUDNN7.0
GPU model and memory:  P100 16GB
Exact command to reproduce:  bash run.sh

Describe the problem
I am using the MonitoredTrainingSession to test a simple script (base on this of synchronized distributed training/multi_gpu training. When the training is finished, the session always throws out Exception (CancelledError) in thread QueueRunnerThread.  I suspect this is related to QueueRunner is not handled cleanly when sess.close() is called.
Source code / logs
`REPLICAS_TO_AGGREGATE = 1
def main():
# Configure
config=tf.ConfigProto(allow_soft_placement=True, log_device_placement=False)
config.gpu_options.allow_growth = True
# Server Setup
cluster = tf.train.ClusterSpec({
    'ps':['localhost:2222'],
    'worker':['localhost:2223']
    }) #allows this node know about all other nodes
if FLAGS.job_name == 'ps': #checks if parameter server
    with tf.device('/job:ps/task:0/cpu:0'):
        server = tf.train.Server(cluster,
                        job_name="ps",
                        task_index=FLAGS.task_index,
                        config=config)
        server.join()
else: #it must be a worker server
    is_chief = (FLAGS.task_index == 0) #checks if this is the chief node
    with tf.device('/gpu:%d' % (FLAGS.task_index)):
        server = tf.train.Server(cluster,
                job_name="worker",
                task_index=FLAGS.task_index,
                config=config)

    # Graph
    # ps_device = "/job:ps/task:0/cpu:0"
    worker_device = "/job:worker/task:%d/gpu:%d" % (FLAGS.task_index, FLAGS.task_index)
    with tf.device(tf.train.replica_device_setter(ps_tasks=1,
                   #ps_device=ps_device,
                   worker_device=worker_device)):

        a = tf.Variable(tf.constant(0.,shape=[2]),dtype=tf.float32)
        b = tf.Variable(tf.constant(0.,shape=[2]),dtype=tf.float32)
        c= a+b

        global_step = tf.Variable(0,dtype=tf.int32,trainable=False,name='global_step')
        target = tf.constant(100.,shape=[2],dtype=tf.float32)
        loss = tf.reduce_mean(tf.square(c-target))

        # create an optimizer then wrap it with SynceReplicasOptimizer
        optimizer = tf.train.GradientDescentOptimizer(.0001)
        optimizer1 = tf.train.SyncReplicasOptimizer(optimizer,
                 replicas_to_aggregate=REPLICAS_TO_AGGREGATE, total_num_replicas=REPLICAS_TO_AGGREGATE)

        opt = optimizer1.minimize(loss,global_step=global_step) # averages gradients
        #opt = optimizer1.minimize(REPLICAS_TO_AGGREGATE*loss,
        #                           global_step=global_step) # hackily sums gradients

    # Session
    sync_replicas_hook = optimizer1.make_session_run_hook(is_chief)
    stop_hook = tf.train.StopAtStepHook(last_step=10)
    hooks = [sync_replicas_hook,stop_hook]

    # Monitored Training Session
    with tf.train.MonitoredTrainingSession(master = server.target,
          is_chief=is_chief,
          config=config,
          hooks=hooks,
          stop_grace_period_secs=10) as sess:

        print('Starting training on worker %d'%FLAGS.task_index)
        while not sess.should_stop():
            _,r,gs=sess.run([opt,c,global_step])
            print(r,'step: ',gs,'worker: ',FLAGS.task_index)
            # if is_chief: time.sleep(1)
            time.sleep(1)
        print('Done',FLAGS.task_index)

        time.sleep(10) #grace period to wait before closing session
        #sess.close() # if uncomment this, it will raise error 'Session is already closed' after the CanceledError
    print('Session from worker %d closed cleanly'%FLAGS.task_index)

if name == 'main':
parser = argparse.ArgumentParser()
# Flags for defining the tf.train.ClusterSpec
parser.add_argument(
"--job_name",
type=str,
default="",
help="One of 'ps', 'worker'"
)
# Flags for defining the tf.train.Server
parser.add_argument(
"--task_index",
type=int,
default=0,
help="Index of task within the job"
)
FLAGS, unparsed = parser.parse_known_args()
print(FLAGS.task_index)
main()`
**************** I use the following script to run the code
#!/bin/bash -e export CUDA_VISIBLE_DEVICES=''  python text_multi_gpu.py --job_name="ps" --task_index=0 & export CUDA_VISIBLE_DEVICES='0'  python text_multi_gpu.py --job_name="worker" --task_index=0
****************** Below is part of the log
`[0.15988803 0.15988803] step:  8 worker:  0
[0.17985605 0.17985605] step:  9 worker:  0
Done 0
Exception in thread QueueRunnerThread-dummy_queue-sync_token_q_EnqueueMany:
Traceback (most recent call last):
File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib64/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/home/python3_env/lib64/python3.6/site-packages/tensorflow/python/training/queue_runner_impl.py", line 268, in _run
coord.request_stop(e)
File "/home/python3_env/lib64/python3.6/site-packages/tensorflow/python/training/coordinator.py", line 213, in request_stop
six.reraise(*sys.exc_info())
File "/home/python3_env/lib64/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/home/python3_env/lib64/python3.6/site-packages/tensorflow/python/training/queue_runner_impl.py", line 252, in _run
enqueue_callable()
File "/home/python3_env/lib64/python3.6/site-packages/tensorflow/python/client/session.py", line 1249, in _single_operation_run
self._call_tf_sessionrun(None, {}, [], target_list, None)
File "/home/python3_env/lib64/python3.6/site-packages/tensorflow/python/client/session.py", line 1420, in _call_tf_sessionrun
status, run_metadata)
File "/home/python3_env/lib64/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 516, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.CancelledError: Step was cancelled by an explicit call to "Session::Close()".
Session from worker 0 closed cleanly`

Sorry, I'm having a little trouble understanding the problem here-- you should not need to explicitly close the session in the with ... block, so the version with sess.close() commented out is the correct one. Is there a problem still with that version?

@karmel
Yes, the problem shown above still happens when "sess.close()" is commented.
If "sess.close()" is uncommented, there is another error as you can expect in addition to the Error I shown above.

@karmel
any clue? Thank for your help.

Nagging Assignee @karmel: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Unfortunately, it is hard to debug from a distance. Is there any reason you are using the deprecated Queue API rather than the newer Dataset API

I have the same issue, any ideas?

It has been 15 days with no activity and the awaiting response label was assigned. Is this still an issue?

It has been 29 days with no activity and the awaiting response label was assigned. Is this still an issue?

It has been 14 days with no activity and the awaiting response label was assigned. Is this still an issue?

@tsoi2 how do you fix this problem? I met the same problem..


text:
FusedBatchNorm problem during benchmark test

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: None
TensorFlow installed from (source or binary): 1.9
TensorFlow version (use command below): 1.9
Python version: 3.6
Bazel version (if compiling from source): 0.11.1 (for covert and benchmark the model)
GCC/Compiler version (if compiling from source): None, installed from build
CUDA/cuDNN version: Not used
GPU model and memory: Not used
Exact command to reproduce:

Problem description
I'm trying to convert a piece of mobilenetV1 (using slim model) from the master repository https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md; In particular I have added 2 simple convolution after the MobilenetV1/Conv2d_8_pointwise/Conv2D. Obviously the number of parameters is more less than the original MobilenetV1. Without train the model, I have initialized all the variables and save them into a freezed model. Now If I visualize the model with tensorboard is the same of that downloaded from the link above until the MobilenetV1/Conv2d_8_pointwise/Conv2D (It compare the Conv2D and the FusedBatchNorm, Gamma, Beta, var and so on). The problems cames when I try to benchmark the model using bazel: the original one is more faster than mine (FLOPs estimated 1.14B VS 3.31B) and also in summaries of the original model the FusedBatchNorm Node type doesn't compare instead in mine model yes. Where am I doing wrong? And there is also the possibility of combining the conv2D and the batch norm layer?
logs
------------------------- ORIGINAL MOBILENETV1 ---------------------------------------------------------------

...stat_summarizer.cc:468] 	             [Node type]	  [count]	  [avg ms]	    [avg %]	    [cdf %]	  [mem KB]	[times called]
...stat_summarizer.cc:468] 	                  Conv2D	       15	    52.027	    56.168%	    56.168%	 12447.652	       15
...stat_summarizer.cc:468] 	   DepthwiseConv2dNative	       13	    15.413	    16.640%	    72.808%	  7905.664	       13
...stat_summarizer.cc:468] 	                     Mul	       27	    12.018	    12.975%	    85.783%	     0.000	       27
...stat_summarizer.cc:468] 	                     Add	       27	    11.806	    12.746%	    98.529%	     0.000	       27
...stat_summarizer.cc:468] 	                   Relu6	       27	     1.163	     1.256%	    99.784%	     0.000	       27
...stat_summarizer.cc:468] 	                   Const	       83	     0.120	     0.130%	    99.914%	     0.000	       83
...stat_summarizer.cc:468] 	                 Softmax	        1	     0.027	     0.029%	    99.943%	     0.000	        1
...stat_summarizer.cc:468] 	                 AvgPool	        1	     0.022	     0.024%	    99.967%	     4.096	        1
...stat_summarizer.cc:468] 	                    NoOp	        1	     0.007	     0.008%	    99.974%	     0.000	        1
...stat_summarizer.cc:468] 	                 BiasAdd	        1	     0.007	     0.008%	    99.982%	     0.000	        1
...stat_summarizer.cc:468] 	                 Squeeze	        1	     0.006	     0.006%	    99.988%	     0.000	        1
...stat_summarizer.cc:468] 	                 _Retval	        1	     0.005	     0.005%	    99.994%	     0.000	        1
...stat_summarizer.cc:468] 	                    _Arg	        1	     0.004	     0.004%	    99.998%	     0.000	        1
...stat_summarizer.cc:468] 	                Identity	        1	     0.002	     0.002%	   100.000%	     0.000	        1

---------------------------- MY MODEL ------------------------------------------------------------------------

...stat_summarizer.cc:468] 	             [Node type]	  [count]	  [avg ms]	    [avg %]	    [cdf %]	  [mem KB]	[times called]
...stat_summarizer.cc:468] 	          FusedBatchNorm	       18	   151.477	    48.975%	    48.975%	    72.704	       18
...stat_summarizer.cc:468] 	                  Conv2D	       11	   138.036	    44.630%	    93.605%	 15457.344	       11
...stat_summarizer.cc:468] 	   DepthwiseConv2dNative	        8	    18.148	     5.868%	    99.472%	  9300.352	        8
...stat_summarizer.cc:468] 	                    Relu	       18	     1.581	     0.511%	    99.984%	     0.000	       18
...stat_summarizer.cc:468] 	                   Const	       29	     0.038	     0.012%	    99.996%	     0.000	       29
...stat_summarizer.cc:468] 	                 _Retval	        1	     0.005	     0.002%	    99.997%	     0.000	        1
...stat_summarizer.cc:468] 	                    _Arg	        1	     0.004	     0.001%	    99.999%	     0.000	        1
...stat_summarizer.cc:468] 	                    NoOp	        1	     0.004	     0.001%	   100.000%	     0.000	        1

OK, I have noted the differen Relu activation but I do not think that's the problem for the fusedbatchnorm layer..

Nagging Assignee @tatatodd: It has been 15 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @tatatodd: It has been 30 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @tatatodd: It has been 44 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @tatatodd: It has been 59 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @tatatodd: It has been 75 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @tatatodd: It has been 90 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @tatatodd: It has been 105 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @tatatodd: It has been 120 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

authors: DavideCatto tensorflowbutler


text:
Confusion matrix and printing CNN kernel with TF

Please go to Stack Overflow for help and support:
https://stackoverflow.com/questions/tagged/tensorflow
If you open a GitHub issue, here is our policy:

It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).
The form below must be filled out.
It shouldn't be a TensorBoard issue. Those go here.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary):
TensorFlow version (use command below):
Python version:
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:
Exact command to reproduce:

You can collect some of this information using our environment capture script:
https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh
You can obtain the TensorFlow version with
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
Describe the problem
Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.
Source code / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

Hello. Please reopen if this is a real issue.

Hi !!
I do not how to reopen the issue !!! and I need really help ...
Le jeu. 27 sept. 2018 à 00:47, Amit Patankar notifications@github.com a
écrit :

Closed #22545 https://github.com/tensorflow/tensorflow/issues/22545.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/22545#event-1869764003,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AlYs1TSUwfPc0SgW5rNlhKo7LA22O9EBks5ufBH8gaJpZM4W7ZKh
.

@Issam-Jmili I have reopened the issue. What do you need help with?

Hi thanks soooo muuuch
what I need is just to to visualize the confusion matrix and 5 kernels of
the first convolutional layer.
I am following the tutorial provided on the website of tensorflow to
retrain flower_photos by transfer knowledge using the Inception V3 here the
code in the retrain.py and it works.
So what I have to change or add in this file to visualize the confusion
matrix and 5 kernels of the first convolutional layer.
Like I have mentioned I am new in this field and it is my first with python
... and it is emergency and thanks again
Le jeu. 27 sept. 2018 à 18:25, Amit Patankar notifications@github.com a
écrit :

@Issam-Jmili https://github.com/Issam-Jmili I have reopened the issue.
What do you need help with?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/22545#issuecomment-425172950,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AlYs1cl6Qrln4CjWX-Xd98OoiS5dfjQaks5ufQn8gaJpZM4W7ZKh
.

Hi @Issam-Jmili unfortunately please keep GitHub for the following as per our policy:

It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).

This question is better suited for Stack Overflow.
That being said I'll provide some advice in passing. Here is some information on producing a confusion matrix, and as for seeing the 5 kernels of the first layer are you using Keras with Eager?

about the code ot tf.confusion.matrix can you tell me where should I put in
the main code of retrain.py please
and about the kernels I am just using tensorflow as virtualenv and what I
konw is keras uses tensorflow as backend to be to be specific I do not use
Keras with Eager ...
thank you by the way for you help it is so kind
Le jeu. 27 sept. 2018 à 23:40, Amit Patankar notifications@github.com a
écrit :

Hi @Issam-Jmili https://github.com/Issam-Jmili unfortunately please
keep GitHub for the following as per our policy:

It must be a bug, a feature request, or a significant problem with
documentation (for small docs fixes please send a PR instead).

This question is better suited for Stack Overflow.
That being said I'll provide some advice in passing. Here
https://www.tensorflow.org/api_docs/python/tf/confusion_matrix is some
information on producing a confusion matrix, and as for seeing the 5
kernels of the first layer are you using Keras with Eager?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/22545#issuecomment-425263960,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AlYs1VTPGEXN7PyuvQUP19LaFkxHPmu6ks5ufVP2gaJpZM4W7ZKh
.

Can you link me your tutorial I'm curious as to what you are using. Once again this would be much better if you posted it on Stack Overflow.

I did post my question on Stack Overflow and no help :\
this is the tutorial that I am following from the website of tensorflow but
they didn't mention how to show the confusion matrix wich it is my project
you see ...
https://www.tensorflow.org/hub/tutorials/image_retraining
Le ven. 28 sept. 2018 à 00:11, Amit Patankar notifications@github.com a
écrit :

Can you link me your tutorial I'm curious as to what you are using. Once
again this would be much better if you posted it on Stack Overflow.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/22545#issuecomment-425269566,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AlYs1a6WNP7sR9f6mY3LpGx4KRZLCmHoks5ufVs0gaJpZM4W7ZKh
.

@Issam-Jmili You can retrieve the predictions from output and plot confusion matrix using a third party tool such as sklearn, seaborn or matlab. Kernels are specified by you, so any of those tools would make visualization work. Hope this helps.

Closing this issue, feel free to reopen if any additional questions.

authors: Issam-Jmili av8ramit wt-huang


text:
problem with installation

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MAC Yosemite 10.10.5 (14F27)

TensorFlow installed from (source or binary):source

TensorFlow version: 1.13.1

Python version: Python 3.6.8 :: Anaconda, Inc.

Installed using virtualenv? pip? conda?: pip

GCC/Compiler version (if compiling from source):

CUDA/cuDNN version:

GPU model and memory: 4 GB 1600 MHz DDR3

Describe the problem
Provide the exact sequence of commands / steps that you executed before running into the problem
Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
Although tensorflow is installed and pip install tensorflow shows that it is already satisfied, but I cannot import tensorflow library
File "/Users/botaduisenbay/Desktop/tester.py", line 9, in
import tensorflow as tf
File "/Users/botaduisenbay/anaconda3/lib/python3.6/site-packages/tensorflow/init.py", line 24, in
from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
File "/Users/botaduisenbay/anaconda3/lib/python3.6/site-packages/tensorflow/python/init.py", line 49, in
from tensorflow.python import pywrap_tensorflow
File "/Users/botaduisenbay/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "/Users/botaduisenbay/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "/Users/botaduisenbay/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "/Users/botaduisenbay/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "/Users/botaduisenbay/anaconda3/lib/python3.6/imp.py", line 243, in load_module
return load_dynamic(name, filename, file)
File "/Users/botaduisenbay/anaconda3/lib/python3.6/imp.py", line 343, in load_dynamic
return _load(spec)
ImportError: dlopen(/Users/botaduisenbay/anaconda3/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so, 6): no suitable image found.  Did find:
/Users/botaduisenbay/anaconda3/lib/python3.6/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so: truncated mach-o error: segment __LINKEDIT extends to 365926664 which is past end of file 154042368
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors
for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.

duplicate #28545

authors: botastark ymodak


text:
Loading Keras model: TypeError: init() missing 1 required positional argument: 'fn'

Installed with pip install tensorflow-gpu, and also tried with pip install tensorflow which yielded the same issue.
tensorflow 1.14
Keras-Applications 1.0.8
Keras-Preprocessing 1.1.0

After saving model with tf.keras.models.save_model(model=self.predict_model, filepath=path), we try to load with tf.keras.models.load_model(path, custom_objects={'tf': tf}) and it yields the error that I reported in the title.
What possible conflict of packages may I have? Or is this a known bug?

All packages in the environment:
Package              Version  Location
-------------------- -------- ------------------------
absl-py              0.7.1
astor                0.8.0
backcall             0.1.0
boto3                1.9.184
botocore             1.12.184
cycler               0.10.0
decorator            4.4.0
docutils             0.14
gast                 0.2.2
google-pasta         0.1.7
grpcio               1.22.0
h5py                 2.9.0
ipython              7.6.1
ipython-genutils     0.2.0
jedi                 0.14.0
jmespath             0.9.4
joblib               0.13.2
Keras-Applications   1.0.8
Keras-Preprocessing  1.1.0
kiwisolver           1.1.0
Markdown             3.1.1
matplotlib           3.1.1
networkx             2.3
nltk                 3.4.4
numpy                1.16.4
pandas               0.24.2
parso                0.5.0
pexpect              4.7.0
pickleshare          0.7.5
pip                  19.1.1
prompt-toolkit       2.0.9
protobuf             3.8.0
ptyprocess           0.6.0
Pygments             2.4.2
pyparsing            2.4.0
python-dateutil      2.8.0
pytz                 2019.1
PyYAML               5.1.1
recordclass          0.11.1
s3transfer           0.2.1
scikit-learn         0.21.2
scipy                1.3.0
seaborn              0.9.0
setuptools           41.0.1
six                  1.12.0
sklearn              0.0
tensorboard          1.14.0
tensorflow           1.14.0
tensorflow-estimator 1.14.0
termcolor            1.1.0
torch                1.1.0
tqdm                 4.32.2
traitlets            4.3.2
urllib3              1.25.3
wcwidth              0.1.7
Werkzeug             0.15.4
wheel                0.33.4
wordninja            0.1.5
wrapt                1.11.2

In order to expedite the trouble-shooting process, please provide a code snippet to reproduce the issue reported here. Thanks!

Just create any Keras' model, save it and load with the commands I mentioned

@JoaoLages
Can you please check attached zip folder.I am able to save and load the Keras folder.Please, let me know if i am wrong.Thanks!
keras.zip

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

The problem occurs when saving a Keras model (save_model function) in tensorflow-gpu==1.14.0 and loading it in tensorflow==1.14 (load_model function)

@JoaoLages
I tried with tensorflow-gpu==1.14.0 and able to save and load successfully.Please, find the gist here.Thanks!

@ravikyram can you try to load it with tensorflow==1.14? (and not tensorflow-gpu)

@JoaoLages
Loaded using !pip install tensorflow==1.14in the below attachment . Request you to have a look on this and let me know if the issue still persists.Thanks!
keras.zip

I am closing the issue since I am not able to reproduce this error out of my environment.

Solved!
self.model = tf.keras.models.load_model(model_file, custom_objects={'tf': tf}, compile=False) - not able to use the loss function but works for prediction

authors: JoaoLages ravikyram


text:
Desynchronized zipped datasets when using tf.data.experimental.ignore_errors

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Linux Ubuntu 18.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
na
TensorFlow installed from (source or binary):
binary
TensorFlow version (use command below):
v2.0.0-rc2-26-g64c3d38 2.0.0
Python version:
Python 3.7.4
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
cuda 10.0
GPU model and memory:
P100 / 16GB

You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with: 1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" 2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"
Describe the current behavior
When an error is caught using the tf.data.experimental.ignore_errors on a zipped dataset, only the faulty dataset drops an element. The datasets are therefore desynchronized.
Describe the expected behavior
Datasets should stay synchronized by dropping an element from both datasets.
Code to reproduce the issue
good_dataset = tf.data.Dataset.from_tensor_slices([1., 2., 0., 4.])
bad_dataset = good_dataset.map(lambda x: tf.debugging.check_numerics(1. / x, "error"))

dataset = tf.data.Dataset.zip((bad_dataset, good_dataset))
dataset = dataset.apply(tf.data.experimental.ignore_errors())

for bad, good in dataset:
    print(float(good), float(bad))

1.0 1.0
2.0 0.5
0.0 0.25

Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

I could reproduce the issue with Tf 2.0.0. Please see the gist here. Thanks!

This is working as intended. The good dataset dropped 0.0 while the bad dataset dropped 1 / 0.0.
@scharron What output would you expect for your example?

@jsimsa Oops sorry I went too fast, I fixed the code to reproduce
(if zipping good then bad it works, but the reverse does not)

I think the issue is that, in case of zip, when error or end-of-sequence encountered the remaining components are not "flushed out".  So out-of-sync happens when ignore_errors is in play.
Created a PR #33887, think this could fix the issue.

Thanks !
Wouldn't it be possible and useful to also issue a warning if the tf.data.experimental.ignore_errors is applied to any Dataset used as input to tf.data.Dataset.zip.
This could also result in datasets desynchronization if any of the zipped datasets drops any item, and in my opinion shouldn't be the behaviour of a zip function.

authors: gadagashwini jsimsa scharron yongtang


text:
Documentation unclear tf.batch_to_space

URL(s) with the issue:
https://www.tensorflow.org/api_docs/python/tf/batch_to_space
Description of issue (what needs changing):

Example needs to be added.
crops part in Args: section is very difficult to understand. Needs to be better formatted (put into code).

Clear description:
Documentation is unclear, no example is shown. Also, there is one extremely long paragraph with code in between in the crops section under args.
Parameters defined
Are all parameters defined and formatted correctly?
=> Not well formatted.
Returns defined
Are return values defined?
=> Yes
Raises listed and defined
Are the errors defined?
=> No
Usage example
Is there a usage example?
=> No
Submit a pull request?
=> No

@nikochiko Do you want to work on improving this section? I could not understand from the opened issue if you have only reported an issue or also proposed to work with it? In later case, I can join you in re-formatting the doc for this topic.

@copperwiring Sure, I would like to work on it. But I will have to look into its technical details before that. If you have any resources, it would be great if you share them. I'll try to learn something about it from code in the meanwhile.

@oanush Please assign this to me and @copperwiring . Will open a PR soon.

@nikochiko Feel free to raise a PR and reference this issue. Thanks!

@ymodak this issue be used for Google Code-in as a Documentation and research challenge?

@dubesar I had started working on the formatting part of the issue and was planning to make a PR till tomorrow. The job is mostly to

Sync the docs formatting with its r1 version.
Make the documentation easier to understand, editing it and adding "Examples:" and "Raises:" sections.
I can make some minor edits with the language to make it more suitable as a Google Code-In challenge.
@ymodak , @dubesar ?

@nikochiko Yeah that would be great if some part of this issue is available for Google Code-in

@dubesar See the referenced PR and a comment on it. I have made some additional changes and made the task more clear.

PR merged. Closing issue.

authors: copperwiring dubesar nikochiko ymodak


text:
ImportError: cannot import name 'load_from_saved_model'

Please make sure that this is a bug. As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock
example script provided in TensorFlow): na
OS Platform and Distribution (e.g.,
Linux Ubuntu 16.04): colab/ubuntu
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if
the issue happens on mobile device:
TensorFlow installed from (source or
binary): - TensorFlow version (use command below): pip
Python version: - Bazel
version (if compiling from source):1.4.0
GCC/Compiler version (if compiling from
source):
CUDA/cuDNN version: - GPU model and memory:

You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with: 1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" 2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"
Describe the current behavior
Error raised once "import tensorflow as tf"
from tensorflow.python.keras.saving.saved_model import load_from_saved_model
ImportError: cannot import name 'load_from_saved_model'
It seems like it happens today, which "import tensorflow" will raise this error once I "pip install tensorflow-gpu==1.14.0".
Describe the expected behavior
Try to run "from tensorflow.python.keras.saving.saved_model import load_from_saved_model
"
It should import correctly without any error.
Standalone code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate
the problem. If possible, please share a link to Colab/Jupyter/any notebook.
Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.

BTW, this is the colab link: https://colab.research.google.com/drive/1Jr0j_I_npKQxcQ1ggJQFHTqoJHmt9Bqk

try tf.saved_model.load ortf.keras.models.load_model

@Sanqiang I was able to import tensorflow using tensorflow 1.15. Can you please install tensorflow -gpu 1.15. Thanks!

Close ticket.
It is because my origiinal TF2.0 is not removed cleanly.

authors: Sanqiang gowthamkpr maxkaustav


text:
wide_n_deep model deal with large dataset got ValueError: GraphDef cannot be larger than 2GB.

wide_n_deep seems like unable to fit large dataset, when I put 1 million data, I got:
Traceback (most recent call last):
  File "search_click.py", line 207, in <module>
    tf.app.run()
  File "/usr/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv))
  File "search_click.py", line 204, in main
    train_and_eval()
  File "search_click.py", line 181, in train_and_eval
    m.fit(input_fn=lambda: input_fn(df_train), steps=FLAGS.train_steps)
  File "/usr/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 182, in fit
    monitors=monitors)
  File "/usr/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/estimators/estimator.py", line 458, in _train_model
    summary_writer=graph_actions.get_summary_writer(self._model_dir))
  File "/usr/lib/python2.7/site-packages/tensorflow/contrib/learn/python/learn/graph_actions.py", line 76, in get_summary_writer
    graph=ops.get_default_graph())
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/summary_io.py", line 113, in __init__
    self.add_graph(graph=graph, graph_def=graph_def)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/training/summary_io.py", line 204, in add_graph
    true_graph_def = graph.as_graph_def(add_shapes=True)
  File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2117, in as_graph_def
    raise ValueError("GraphDef cannot be larger than 2GB.")
ValueError: GraphDef cannot be larger than 2GB.

def input_fn(df):
  """Input builder function."""
  # Creates a dictionary mapping from each continuous feature column name (k) to
  # the values of that column stored in a constant Tensor.
  continuous_cols = {k: tf.constant(df[k].values) for k in CONTINUOUS_COLUMNS}
  # Creates a dictionary mapping from each categorical feature column name (k)
  # to the values of that column stored in a tf.SparseTensor.
  categorical_cols = {k: tf.SparseTensor(
      indices=[[i, 0] for i in range(df[k].size)],
      values=df[k].values,
      shape=[df[k].size, 1])
                      for k in CATEGORICAL_COLUMNS}
  # Merges the two dictionaries into one.
  feature_cols = dict(continuous_cols)
  feature_cols.update(categorical_cols)
  # Converts the label column into a constant Tensor.
  label = tf.constant(df[LABEL_COLUMN].values)
  # Returns the feature columns and the label.
  return feature_cols, label

Is there a replacement of tf.constant and tf.SparseTensor, so we can load a batch once a time?
Any help would be appreciated!

@xinyuwuhenZZ You can use partial_fit() found here.

Thanks, but seems like no difference between fit() and partial_fit().

Thanks, but seems like no difference between fit() and partial_fit().

You are right. The implementation is quite different now and many functionalities may not work anymore. Perhaps @martinwicke can advise on this better.

Large datasets should not be put as constants into the graph. Use queues. You can check on the tutorials page, or on StackOverflow on how to use it.

@martinwicke Can you give an example? Thank you very much. I am a freshman to tensorflow.

Please check out this howto: https://www.tensorflow.org/how_tos/reading_data/

@martinwicke Thank you for your time. How to edit the function input_fn below to fix the ValueError: GraphDef cannot be larger than 2GB? Can you give another version of function input_fn? Thanks a lot. I am not very clear of how to do this edition with the link https://www.tensorflow.org/how_tos/reading_data/ after trying a lot with a lot of errors.
def input_fn(df):
  """Input builder function."""
  # Creates a dictionary mapping from each continuous feature column name (k) to
  # the values of that column stored in a constant Tensor.
  continuous_cols = {k: tf.constant(df[k].values) for k in CONTINUOUS_COLUMNS}
  # Creates a dictionary mapping from each categorical feature column name (k)
  # to the values of that column stored in a tf.SparseTensor.
  categorical_cols = {k: tf.SparseTensor(
      indices=[[i, 0] for i in range(df[k].size)],
      values=df[k].values,
      shape=[df[k].size, 1])
                      for k in CATEGORICAL_COLUMNS}
  # Merges the two dictionaries into one.
  feature_cols = dict(continuous_cols)
  feature_cols.update(categorical_cols)
  # Converts the label column into a constant Tensor.
  label = tf.constant(df[LABEL_COLUMN].values)
  # Returns the feature columns and the label.
  return feature_cols, label

@martinwicke Thank you for your time. How to edit the function input_fn below to fix the ValueError: GraphDef cannot be larger than 2GB? Can you give another version of function input_fn? Thanks a lot. I am not very clear of how to do this edition with the link https://www.tensorflow.org/how_tos/reading_data/ after trying a lot with a lot of errors.
def input_fn(df):
  """Input builder function."""
  # Creates a dictionary mapping from each continuous feature column name (k) to
  # the values of that column stored in a constant Tensor.
  continuous_cols = {k: tf.constant(df[k].values) for k in CONTINUOUS_COLUMNS}
  # Creates a dictionary mapping from each categorical feature column name (k)
  # to the values of that column stored in a tf.SparseTensor.
  categorical_cols = {k: tf.SparseTensor(
      indices=[[i, 0] for i in range(df[k].size)],
      values=df[k].values,
      shape=[df[k].size, 1])
                      for k in CATEGORICAL_COLUMNS}
  # Merges the two dictionaries into one.
  feature_cols = dict(continuous_cols)
  feature_cols.update(categorical_cols)
  # Converts the label column into a constant Tensor.
  label = tf.constant(df[LABEL_COLUMN].values)
  # Returns the feature columns and the label.
  return feature_cols, label

Please ask this type of question on StackOverflow.
Avoid calling tf.constant. Use tf.batch instead.
On Sat, Dec 31, 2016 at 20:37 zhangrong notifications@github.com wrote:

@martinwicke https://github.com/martinwicke Thank you for your time.
How to edit the function input_fn below to fix the ValueError: GraphDef
cannot be larger than 2GB? Can you give another version of function
input_fn? Thanks a lot. I am not very clear of how to do this edition
with the link https://www.tensorflow.org/how_tos/reading_data/ after
trying a lot with a lot of errors.
def input_fn(df):
"""Input builder function."""
Creates a dictionary mapping from each continuous feature column name (k) to
the values of that column stored in a constant Tensor.
continuous_cols = {k: tf.constant(df[k].values) for k in CONTINUOUS_COLUMNS}
Creates a dictionary mapping from each categorical feature column name (k)
to the values of that column stored in a tf.SparseTensor.
categorical_cols = {k: tf.SparseTensor(
  indices=[[i, 0] for i in range(df[k].size)],

  values=df[k].values,

  shape=[df[k].size, 1])

                  for k in CATEGORICAL_COLUMNS}

Merges the two dictionaries into one.
feature_cols = dict(continuous_cols)
feature_cols.update(categorical_cols)
Converts the label column into a constant Tensor.
label = tf.constant(df[LABEL_COLUMN].values)
Returns the feature columns and the label.
return feature_cols, label
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/3813#issuecomment-269892070,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAjO_ZESr8LhQ9I2u-OpN1ugiGQ0VYIQks5rNy2YgaJpZM4JkJL4
.

Please ask this type of question on StackOverflow.
Avoid calling tf.constant. Use tf.batch instead.
On Sat, Dec 31, 2016 at 20:37 zhangrong notifications@github.com wrote:

@martinwicke https://github.com/martinwicke Thank you for your time.
How to edit the function input_fn below to fix the ValueError: GraphDef
cannot be larger than 2GB? Can you give another version of function
input_fn? Thanks a lot. I am not very clear of how to do this edition
with the link https://www.tensorflow.org/how_tos/reading_data/ after
trying a lot with a lot of errors.
def input_fn(df):
"""Input builder function."""
Creates a dictionary mapping from each continuous feature column name (k) to
the values of that column stored in a constant Tensor.
continuous_cols = {k: tf.constant(df[k].values) for k in CONTINUOUS_COLUMNS}
Creates a dictionary mapping from each categorical feature column name (k)
to the values of that column stored in a tf.SparseTensor.
categorical_cols = {k: tf.SparseTensor(
  indices=[[i, 0] for i in range(df[k].size)],

  values=df[k].values,

  shape=[df[k].size, 1])

                  for k in CATEGORICAL_COLUMNS}

Merges the two dictionaries into one.
feature_cols = dict(continuous_cols)
feature_cols.update(categorical_cols)
Converts the label column into a constant Tensor.
label = tf.constant(df[LABEL_COLUMN].values)
Returns the feature columns and the label.
return feature_cols, label
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/3813#issuecomment-269892070,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAjO_ZESr8LhQ9I2u-OpN1ugiGQ0VYIQks5rNy2YgaJpZM4JkJL4
.

authors: guozhizou martinwicke renwaiyouren terrytangyuan xinyuwuhenZZ


text:
Can't load saved keras model.h5

Please make sure that this is a bug. As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): windows 10
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): pip
TensorFlow version (use command below): 2.1.0
Python version: 3.7
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: 10.2 & 10.1
GPU model and memory:

You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with:

TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the current behavior
When loading the model with keras.models.load_model the activation functions are not recognized. This happens only when activation functions are not builtin strings
Describe the expected behavior
To load the model with success
Standalone code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate
the problem. If possible, please share a link to Colab/Jupyter/any notebook.
import tensorflow as tf

layer = tf.keras.layers.Dense(1, activation=tf.keras.layers.PReLU(alpha_initializer='random_uniform', alpha_regularizer=None, alpha_constraint=None, shared_axes=None),\
                            name='layerX', kernel_initializer=tf.keras.initializers.he_normal())
model = tf.keras.Sequential(layer)
model.compile("adam", "binary_crossentropy", ["accuracy"])
model.fit([[1]], [1])
model.save("keras_model.h5", save_format='h5' ")
model = tf.keras.models.load_model("keras_model.h5")

EDIT: After reading the docs I saved the model with save_format h5, first time I tried without any format (the default is tf for tensorflow 2.0+)
Instead I found a workaround: I can load the model if I save it with (default) save_format='tf'
Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.
  File "C:\Users\Teo\OneDrive\Licenta\main.py", line 138, in <module>
    main()
  File "C:\Users\Teo\OneDrive\Licenta\main.py", line 135, in main
    model = keras.models.load_model(os.path.join(dataset_path, "keras_model.h5"))
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\saving\save.py", line 146, in load_model
    return hdf5_format.load_model_from_hdf5(filepath, custom_objects, compile)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\saving\hdf5_format.py", line 168, in load_model_from_hdf5
    custom_objects=custom_objects)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\saving\model_config.py", line 55, in model_from_config
    return deserialize(config, custom_objects=custom_objects)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\layers\serialization.py", line 106, in deserialize
    printable_module_name='layer')
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\utils\generic_utils.py", line 303, in deserialize_keras_object
    list(custom_objects.items())))
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\sequential.py", line 377, in from_config
    custom_objects=custom_objects)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\layers\serialization.py", line 106, in deserialize
    printable_module_name='layer')
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\utils\generic_utils.py", line 305, in deserialize_keras_object
    return cls.from_config(cls_config)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\base_layer.py", line 519, in from_config
    return cls(**config)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\layers\core.py", line 1082, in __init__
    self.activation = activations.get(activation)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\activations.py", line 450, in get
    identifier, printable_module_name='activation')
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\utils\generic_utils.py", line 292, in deserialize_keras_object
    config, module_objects, custom_objects, printable_module_name)
  File "C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\utils\generic_utils.py", line 250, in class_and_config_for_serialized_keras_object
    raise ValueError('Unknown ' + printable_module_name + ': ' + class_name)
ValueError: Unknown activation: PReLU

@teodor440
Request you to share colab link or simple standalone code to reproduce the issue in our environment.It helps us in localizing the issue faster.Thanks!

@teodor440
Request you to share colab link or simple standalone code to reproduce the issue in our environment.It helps us in localizing the issue faster.Thanks!

https://colab.research.google.com/drive/1qZ8RuI-oNoTsD2uokHZztqeCb20RX3wR#scrollTo=_gMOL9xUPGha
I also updated the code in the issue

I have tried in colab with TF 2.1.0 , 2.2-rc3 and was able to reproduce the issue. Please, find the gist here.Thanks!

@teodor440 I was able to reproduce the issue with save_format='h5'. However, when I used save_format = 'tf' everything worked as expected. model.predict before saving and after loading are exactly same. Please take a look at the gist here. Thanks!

I am not familiar with the implementation of tensorflow, but seems like there is a problem with how tf sees keras models
I can recall the same value error when trying to convert the model to an estimator with  keras.estimator.model_to_estimator

Ok so the problem was that I specified the activations in the layers as something different than a string. For some reason or another this confuses the operations made on keras models by tf

@teodor440 You can use "tf" format as it is working well. But, this is still a bug with "h5" format. I will reopen the issue. Thanks!

You can see now at the collab link that the issue with saving h5 model doesn't persist anymore if using separate layers for dense and activation. So basically this is a workaround for the problem

Closing this issue since the associated PR has been merged and the issue is fixed. Thanks

authors: jvishnuvardhan ravikyram teodor440 ymodak


text:
about attention_seq2seq function without embedding

we already have a function tf.nn.seq2seq.embedding_attention_seq2seq(), but if i want to use an embedding trained from other model, the function is not convenient, do we have a function like tf.nn.seq2seq.attention_seq2seq() that need an embedding para or the inputs embedded.
Thanks!

it seem that the seq2seq folder have different classes, but i haven't try.
let me know when you solve the problem xD

This does not appear to be a bug report, but rather a general question.  Please can you reask on StackOVerflow.

authors: Mev-z prb12 wagamamaz


text:
tf.TensorArray.stack() fails inside tf.function when dtype is tf.uint32 or tf.uint64

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): v2.2.0-rc4-8-g2b96f3662b 2.2.0
Python version: 3.7.7
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: none
GPU model and memory: N/A

Describe the current behavior
When used in a tf.function, calling stack() in a tf.TensorArray object fails with the following error when the dtype is tf.uint32 or tf.uint64.

tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument:  Unsupported type in DataTypeToPrimitiveType: 'variant'
[[node TensorArrayV2Stack/TensorListStack (defined at ta_error.py:21) ]]
(1) Invalid argument:  Unsupported type in DataTypeToPrimitiveType: 'variant'
[[node TensorArrayV2Stack/TensorListStack (defined at ta_error.py:21) ]]
[[TensorArrayV2Stack/TensorListStack/_6]]
0 successful operations.
0 derived errors ignored. [Op:__inference_test_12]

Describe the expected behavior
Should not fail, but return a stacked tensor. Works correctly when the dtype is not tf.uint32 or tf.uint64. Also works correctly when executed in eager mode.
Standalone code to reproduce the issue
@tf.function
def test():
  ta = tf.TensorArray(tf.uint64, 1)
  ta = ta.write(0, 123)
  return ta.stack()

I have tried in colab with TF version 2.2 ,nightly version(2.3.0-dev20200604) and was able to reproduce the issue.Please,find the gist here.Thanks!

I think this issue is now fixed with tf-nightly. Closing issue. Please reopen if still there is a problem


text:
bazel can not compile Tensorflow with GCC 6.2.1

When I launch
"bazel build  -c opt --jobs=10 tensorflow/tools/pip_package:build_pip_package"
to compile tensorflow from source ,it crashed at:
"external/com_googlesource_code_re2/BUILD:11:1: C++ compilation of rule '@com_googlesource_code_re2//:re2' failed: gcc failed: error executing command /usr/bin/gcc -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wl,-z,-relro,-z,now -B/usr/bin -B/usr/bin -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-canonical-system-headers ... (remaining 32 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
external/com_googlesource_code_re2/re2/dfa.cc: In constructor 're2::DFA::State::State()':
external/com_googlesource_code_re2/re2/dfa.cc:95:10: error: unknown array size in delete...
"
Here are some relative infos:
1)bazel label: 0.3.1- (@non-git)
2)gcc (GCC) 6.2.1 20160830
3)tensorflow is the latest master branch
Can you tell me how to configure bazel with clang(not cross compile) ? Thanks.

https://github.com/google/re2/issues/102  or https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70932

Looks related to a gcc bug. Assigning an owner from our side to track this until the gcc bug is fixed.

This was fixed in https://github.com/tensorflow/tensorflow/commit/7ea4e2f

authors: MaeThird andydavis1 jart


text:
ValueError: Shapes (None, 1) and (None, 10) are incompatible

I have 40 rows and 15000 columns
EEG data with attaching lebel
Epoch 1/10
WARNING:tensorflow:Layer dense_2 is casting an input tensor from dtype float64 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because its dtype defaults to floatx.

If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.

To change all layers to have dtype float64 by default, call `tf.keras.backend.set_floatx('float64')`. To change just this layer, pass dtype='float64' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-33-601f7224d647> in <module>()
----> 1 model.fit(X_train, y_train, epochs=10, batch_size=32)

10 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
    971           except Exception as e:  # pylint:disable=broad-except
    972             if hasattr(e, "ag_error_metadata"):
--> 973               raise e.ag_error_metadata.to_exception(e)
    974             else:
    975               raise

ValueError: in user code:

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:806 train_function  *
        return step_function(self, iterator)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:796 step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:1211 run
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2585 call_for_each_replica
        return self._call_for_each_replica(fn, args, kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2945 _call_for_each_replica
        return fn(*args, **kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:789 run_step  **
        outputs = model.train_step(data)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:749 train_step
        y, y_pred, sample_weight, regularization_losses=self.losses)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/compile_utils.py:204 __call__
        loss_value = loss_obj(y_t, y_p, sample_weight=sw)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/losses.py:149 __call__
        losses = ag_call(y_true, y_pred)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/losses.py:253 call  **
        return ag_fn(y_true, y_pred, **self._fn_kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/util/dispatch.py:201 wrapper
        return target(*args, **kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/losses.py:1535 categorical_crossentropy
        return K.categorical_crossentropy(y_true, y_pred, from_logits=from_logits)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/util/dispatch.py:201 wrapper
        return target(*args, **kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/backend.py:4687 categorical_crossentropy
        target.shape.assert_is_compatible_with(output.shape)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/tensor_shape.py:1134 assert_is_compatible_with
        raise ValueError("Shapes %s and %s are incompatible" % (self, other))

    ValueError: Shapes (None, 1) and (None, 10) are incompatible

@connect2robiul
Please, fill issue template.
Request you to share colab link or simple standalone code to reproduce the issue in our environment.It helps us in localizing the issue faster. Thanks!

@connect2robiul
Please, fill issue template.
Request you to share colab link or simple standalone code to reproduce the issue in our environment.It helps us in localizing the issue faster. Thanks!

https://colab.research.google.com/drive/1nLY9ucV9UV6Avp7k4obKaGRZL21i7wLI?usp=sharing

@connect2robiul
Can you help us with sample data to reproduce the issue in our environment.Which TF version you are using?
Thanks!

@connect2robiul
Can you help us with sample data to reproduce the issue in our environment.Which TF version you are using?
Thanks!

I can not share my dataset. I am using default colab TF which is 2.3.0

I see you are using tf.data.Dataset.from_tensor_slices() to convert the pandas.Dataframe into a tf.data dataset which is the correct approach.
https://www.tensorflow.org/tutorials/load_data/pandas_dataframe
model.fit accepts input data types such as numpy array, tensor or tf dataset.

Looks like you were successful in finding a workaround for this.
I see you are using tf.data.Dataset.from_tensor_slices() to convert the pandas.Dataframe into a tf.data dataset which is the correct approach.
https://www.tensorflow.org/tutorials/load_data/pandas_dataframe
model.fit accepts input data types such as numpy array, tensor or tf dataset.

same error through I change to to model.fit(X_train.to_numpy(), y_train.to_numpy(), epochs=10, batch_size=32)

@connect2robiul Can you share a minimal example to reproduce this with minimal dummy data?

@connect2robiul Can you share a minimal example to reproduce this with minimal dummy data?

I got a different error on the same line; code link <https://colab.research.google.com/drive/1UAiDGFQ14EhEzYDnb7yIsJPD0SVL58Rn?usp=sharing>
And it's editable

@connect2robiul
Can you help us with sample data to reproduce the issue in our environment.Which TF version you are using?
Thanks!

I got a different error on the same line; code link https://colab.research.google.com/drive/1UAiDGFQ14EhEzYDnb7yIsJPD0SVL58Rn?usp=sharing
And it's editable

The colab you shared is different from the previously shared where we were dealing with csv data frame and converting it into tf.data.Datasets.
However in the current colab we may want to change loss=binary_crossentropy since the label is in binary and set correct input data and target shapes.

The colab you shared is different from the previously shared where we were dealing with csv data frame and converting it into tf.data.Datasets.
However in the current colab we may want to change loss=binary_crossentropy since the label is in binary and set correct input data (47, 120000) and target data (47,) shapes.

I run the same code here https://colab.research.google.com/drive/1pYOmiNKCOA4LkIr2MbQlqfHbyvvcWb-U?usp=sharing. But still has an error on the fit function

The colab you shared is different from the previously shared where we were dealing with csv data frame and converting it into tf.data.Datasets.
However in the current colab we may want to change loss=binary_crossentropy since the label is in binary and set correct input data (47, 120000) and target data (47,) shapes.

I run the same code here < https://colab.research.google.com/drive/1pYOmiNKCOA4LkIr2MbQlqfHbyvvcWb-U?usp=sharing>. But still has an error on the fit function

For your last colab example:
model.add(Dense(11, activation='softmax'))

authors: bhack connect2robiul ravikyram ymodak


text:
gpu out of mempry

i difined a simple model use tensorflow2.0, but it run slowly. then i code gpustat in terminal, i found gpu out of memory. i run the same code use tensorflow tensorflow1.15, it runs normaly, and gpu memory cost correctly. the model defined as follow:
model = keras.Sequential([
layers.Reshape(target_shape=(28 * 28,), input_shape=(28, 28)),
layers.Dense(100, activation='relu'),
layers.Dense(100, activation='relu'),
layers.Dense(10)])
what should i do?

@daeing
Can you try with latest TF stable version 2.4 or nightly version and see if the performance issue still persists.If the issue still present ,request you to share colab link or simple standalone code to reproduce the issue in our environment.It helps us in localizing the issue faster. Thanks!

@daeing
Can you try with latest TF stable version 2.4 or nightly version and see if the performance issue still persists.If the issue still present ,request you to share colab link or simple standalone code to reproduce the issue in our environment.It helps us in localizing the issue faster. Thanks!

Thank you for your replying. I trid tensorflow2.4 in windows, it runs normaly.
During I deal with this question. I installed cuda 9, cuda 10.0, cuda 10.1, and cuda 11 my driver suggest me install. Then I created virtual environment in anaconda, such as tensorflow115, tensorflow20, tensorflow22, and use pip install tensorflow-gpu==1.15/ pip install tensorflow-gpu==2.0 / pip install tensorflow==2.2. After that, i enter the environment,  I code tf.test.is_gpu_available() it didn't work, no answer(True, False).  Can you give me some suggestion?

@daeing
Please, see tested build configurations from here and gpu installation guide for windows from here and see if it helps you.Thanks!

authors: daeing ravikyram


text:
XLA HLO profiling not supported on GPU

Please make sure that this is a feature request. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template
System information

TensorFlow version (you are using): v2.5.0-rc3-213-ga4dfb8d1a71 2.5.0
Are you willing to contribute it (Yes/No): Yes

Describe the feature and the current behavior/state.
when enable hlo profile via export XLA_FLAGS="--xla_hlo_profile", log says
2021-09-14 17:15:45.256221: E tensorflow/compiler/xla/service/gpu/gpu_compiler.cc:893] --xla_hlo_profile for GPU is unsupported.

Will this change the current api? How?
No
Who will benefit with this feature?
Machine learning system researchers/engineers would benefit from this feature when profiling machine learning models.
Any Other info.

Hi @jvishnuvardhan ,Could you please look at this issue .

Hi,
Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.
The Tensorflow team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.
Please follow the release notes to stay up to date with the latest developments which are happening in the Tensorflow space.

authors: donglinz mohantym tilakrayal


text:
saved model regression in 2.11

Click to expand!
Issue Type
Bug
Have you reproduced the bug with TF nightly?
No
Source
binary
Tensorflow Version
2.11.0
Custom Code
Yes
OS Platform and Distribution
Linux Ubuntu 20.04.5 LTS
Mobile device
No response
Python version
3.8
Bazel version
No response
GCC/Compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current Behaviour?
When calling a method from a loaded tensorflow saved model, I get this exception -

      1 imported = tf.saved_model.load("saved_model")
----> 2 imported(inputs={"x": [1.]}, training=True)

/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py in _call_attribute(instance, *args, **kwargs)
    730
    731 def _call_attribute(instance, *args, **kwargs):
--> 732   return instance.__call__(*args, **kwargs)
    733
    734

/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/traceback_utils.py in error_handler(*args, **kwargs)
    151     except Exception as e:
    152       filtered_tb = _process_traceback_frames(e.__traceback__)
--> 153       raise e.with_traceback(filtered_tb) from None
    154     finally:
    155       del filtered_tb

/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/polymorphic_function/function_spec.py in canonicalize_function_inputs(self, args, kwargs)
    517         missing_args = [self._arg_names[i] for i in sorted(missing_arg_indices)]
    518         if len(missing_args) == 1:
--> 519           raise TypeError(f"{self.signature_summary()} missing 1 required "
    520                           f"argument: {missing_args[0]}.")
    521         else:

TypeError: f(self) missing 1 required argument: self.

This used to work with tenforflow version 2.6.0 and prior.
Standalone code to reproduce the issue
import tensorflow as tf

class CustomModel(tf.keras.Model):

  def __init__(self):
    super().__init__()

  def call(self, inputs, training):
    print('Tracing with', inputs)
    return inputs

model = CustomModel()

model.__call__ = tf.function(model.__call__)

print('Saving model...')

tf.saved_model.save(model, "saved_model", signatures=model.__call__.get_concrete_function(
            inputs={"x": tf.TensorSpec(shape=[1,], dtype=tf.float32)}, training=tf.TensorSpec(shape=None, dtype=tf.bool, name="training")
        ))

imported = tf.saved_model.load("saved_model")
imported(inputs={"x": [1.]}, training=True)

Relevant log output
No response

i can get it to work if I define the "call" method -
import tensorflow as tf

class CustomModel(tf.keras.Model):

  def __init__(self):
    super().__init__()

  def call(self, inputs, training):
    print('Tracing with', inputs)
    return inputs

  def __call__(self, *args, **kwargs):

        return super().__call__(*args, **kwargs)

model = CustomModel()

since the __call__ method is defined on the base keras model class, it should not be needed to do this.

@saswatac
Could you please try model.__call__ = tf.function(model.call) instead of model.__call__ = tf.function(model.__call__). By using model.__call__ = tf.function(model.call) I was able to execute the given code without error on colab using TF v2.11. Please find the gist here for reference.
Thank you!

Thanks @tiruk007, it works. Are there any potential downsides to using this? What would be the difference between this solution, and the one I posted in the comment above.

@saswatac
I am calling the call function from the subclass model where you are calling the function from the parental class which is tf.keras.Model. Could you please close the issue if it is resolved ?
Thank you !

thanks for your response.


text:
Have mnist.py be controlled by tf.set_random_seed() for testing purposes.

It might be helpful to make the internal random generator within mnist.py be connected to tf.set_random_seed().  Since calling this function is part of the first tutorials to tensorflow, it can be confusing if users try to obtain reproducible results by setting tf.set_random_seed() and seeing that it has no effect (because mnist.py does not comprehend this).

Created pull request with code changes to supersede this issue.