text:
README should include an example of a build command with explicit component paths

System information

TensorFlow version: 2.3.1
Doc Link: https://github.com/tensorflow/java (the README.md file)

Describe the documentation issue
The README.md file should provide a build example for an arbitrary environment, which contains all of the required components, but not necessarily configured as the default choices. This is often the case with ML development workstations, which have multiple versions of CUDA, cuDNN, gcc, and friends, since different projects require the developer to use different versions for reproducibility reasons.
This example uses Bazelisk to manage multiple Bazel versions.
TF_CUDA_COMPUTE_CAPABILITIES will require #195 to be merged.
TF_CUDA_COMPUTE_CAPABILITIES=3.5,3.7,6.1,7.0,7.5 \
GCC_HOST_COMPILER_PATH=/usr/bin/gcc-8 \
CC=/usr/bin/gcc-8 \
CXX=/usr/bin/g++-8 \
TF_CUDA_PATHS=/usr/local/cuda-10.2.89,/usr/local/cudnn-10.2-7.6.5.32 \
CUDA_TOOLKIT_PATH=/usr/local/cuda-10.2.89 \
CUDNN_INSTALL_PATH=/usr/local/cudnn-10.2-7.6.5.32 \
USE_BAZEL_VERSION=3.1.0 \
TMP=/tmp \
mvn install -Dmaven.test.skip=true -Djavacpp.platform.extension=-gpu

We welcome contributions by users. Will you be able to update submit a PR (use the doc style guide) to fix the doc Issue?
No.

Since 99% of the native build is just building tensorflow, configuration should be handled in the same way (i.e. the configure script, or anything else).  This is mentioned in #195, and clarified in the latest commit.  Is there any reason that wouldn't work for you?

It's not that it's completely impossible to find out the practical information required to create a custom build, it's just that it would be incredibly easy to make finding this information convenient.
As a result, hundreds or thousands of individual users wouldn't need to spend time searching for and learning this trivia, which they'll probably never need again, just to accomplish the task of building the binaries for compute capabilities other than 3.5 and 7.0.

You don't have to go hunting for any information though, you just clone tensorflow and run the configuration script.  The reason we (and tensorflow) delegate to that is any hard-coded paths we provide will be wrong more often than they are right, so the script tries its best to autodetect them.  Plus a lot of the configuration has been moved to the bazelrc files in the latest tensorflow release, so there should be even less required configuration.
What exactly is so hard to find?

It is possible though to simply use the TF archive that Bazel downloads and unzips just before building TF Java, instead of cloning the repo.
I'm wondering if it would work to run the configure script from that unzipped archive during the Maven build directly, e.g. by passing a parameter like -Dnative.build.custom=true. I never tried running a blocking goal that requires user input in Maven though, is that possible?
Or we simply add a new script in the Java repo that invoke Bazel to download the archive and run configure on it, bypassing our default .bazelrc config.

Or we simply add a new script in the Java repo that invoke Bazel to download the archive and run configure on it, bypassing our default .bazelrc config.

I like this option much more.  It would just be our own version of the configure script.

authors: karllessard mikaelhg rnett

repo: tensorflow/java


text:
Examples in optvis readme don't work

Maybe the documentation is outdated, but it looks like several of the examples are using methods that do not seem to be implemented:
https://github.com/tensorflow/lucid/tree/master/lucid/optvis
e.g.
obj = objectives.channel("mixed4a_pre_relu", 2)
param_f = lambda: tf.concat([
    param.rgb_sigmoid(param.naive([1, 128, 128, 3])),
    param.fancy_colors(param.naive([1, 128, 128, 8])/1.3),
    param.rgb_sigmoid(param.laplacian_pyramid([1, 128, 128, 3])/2.),
    param.fancy_colors(param.laplacian_pyramid([1, 128, 128, 8])/2./1.3),
], 0)
render_vis(model, obj, param_f)

I have the same problem. Seems the rgb_sigmoid function is replaced by to_valid_rgb(t, sigmoid=True)? I may be wrong about this.

@colah updated the documentation in 75a0d88. Does that help? I'll close for now, but feel free to reopen if you notice we missed a spot.
@lindaterlouw that is correct! (Feel free to leave out the sigmoid=True, which is a default parameter.)
Sorry for the breaking API changes! We try to be careful, but right now the API has not yet fully stabilized.

source: gharchive/issue

authors: Quasimondo lindaterlouw ludwigschubert

repo: tensorflow/lucid

license_type: permissive

license_source: bigquery


text:
Add transposedConv2D (#596)

TansposedConv2D layer is using internal API conv2DBackpropInput.
It makes creating custom transposed convolutional layers outside the library hard.
This PR exposes that API with new name transposedConv2D.

I restored conv2DBackpropInput and added transposedConv2D.
Now TransposedConv*D layers use transposedConv2D and _vjpConv2D* funcs use conv2DBackpropInput.

Could you please add some tests for transposedConv2D and its derivative? You can adapt existing tests for conv2D.

It looks there's no tests for conv2D. But there are tests for Conv2D instead.
Also TransposedConv2D has its test, but it didn't have test of its gradients, so I just added it.
By the way I got two questions.

_vjpConv2DBackpropInput and some other functions are @usableFromInline.
Is there any reason to prefer @usableFromInline to @inlinable?

transposedConv2D has argument shape: Tensor<Int32>. It seems we can change its type to TensorShape.
Do you think it's good idea?

@t-ae all shape variables in the operators file for convolutional operator are Tensor<Int32> because the backend generated ops need them in that format, so making it a TensorShape is not a good idea.

all shape variables in the operators file for convolutional operator are Tensor

They are internal but transposedConv2D is public.
We'd better decide its signature from the point of view of users.
According to tf.nn.conv2d_transpose documentation,

output_shape: A 1-D Tensor representing the output shape of the deconvolution op.

TensorShape is always 1-D so user can't pass wrong parameter.
I don't know if there's case rank of shape is not 4. If there's no such case, it should be rather (Int, Int, Int, Int) than TensorShape.

@dan-zheng
The discussion about the type of shape is left over.
Do you have any comments?

source: gharchive/pull-request

repo: tensorflow/swift-apis

license_source: github-api


text:
shuffle a tensor along an axis

I'm having trouble shuffling a tensor along a specified axis. I tried using a shuffled array of indices to do this, like so:
var indices: [Int] = Array(0...cardinality)
indices.shuffle()
return data[indices]

The problem is that S4TF does not like integer arrays being used as indices. I get the error argument type '[Int]' does not conform to expected type 'TensorRangeExpression'

I'm having trouble shuffling a tensor along a specified axis. I tried using a shuffled array of indices to do this, like so:
var indices: [Int] = Array(0...cardinality)
indices.shuffle()
return data[indices]

The problem is that S4TF does not like integer arrays being used as indices. I get the error argument type '[Int]' does not conform to expected type 'TensorRangeExpression'

There's actually a name for this shuffling operation: Tensor.transposed(permutation: [Int]).
Here's how to use it:
import TensorFlow
let data = Tensor<Float>(randomNormal: [1, 2, 3, 4, 5])
var indices: [Int] = Array(0..<data.rank)
indices.shuffle()
print(data.shape)
print(data.transposed(permutation: indices).shape)
// [1, 2, 3, 4, 5]
// [4, 1, 5, 3, 2]

Swift supports type extensions, so you can define Tensor.randomPermutation for convenience:
import TensorFlow
extension Tensor {
    var randomPermutation: Tensor {
        var permutation = Array(0..<rank)
        permutation.shuffle()
        return transposed(permutation: permutation)
    }
}
let data = Tensor<Float>(randomNormal: [1, 2, 3, 4, 5])
print(data.shape)
print(data.randomPermutation.shape)
// [1, 2, 3, 4, 5]
// [4, 1, 5, 3, 2]

Hope this helps!

Thanks for the help! I ran into 2 problems trying to implement this.
First, I'm not entirely sure where the indices attribute is coming from. I'm using the latest toolchain (0.7) and am getting error: value of type 'Tensor<Float>' has no member 'indices' when I run your first example.
Also, it looks like this code would shuffle the order of each axis of a tensor. My goal is to shuffle along an axis. For example:
a = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] shuffled along the 0th axis could be [[4, 5, 6], [1, 2, 3], [7, 8, 9]], and shuffled along the 1th axis could be [[2, 1, 3], [5, 4, 6], [8, 7, 9]]. This is like what tf.random.shuffle does

First, I'm not entirely sure where the indices attribute is coming from. I'm using the latest toolchain (0.7) and am getting error: value of type 'Tensor<Float>' has no member 'indices' when I run your first example.

Sorry! This was a typo that I just fixed. In Tensor.randomPermutation, Array(indices) should be Array(shape.indices).

Also, it looks like this code would shuffle the order of each axis of a tensor. My goal is to shuffle along an axis. For example:
a = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] shuffled along the 0th axis could be [[4, 5, 6], [1, 2, 3], [7, 8, 9]], and shuffled along the 1th axis could be [[2, 1, 3], [5, 4, 6], [8, 7, 9]]. This is like what tf.random.shuffle does

Aha, yes. It sounds like you want a "shuffled" function that is different than randomPermutation - sorry I misunderstood your prompt. I think the implementation is a bit more involved, but I have faith you can figure it out!

Reopening, as the original question (a "shuffle" function like tf.random.shuffle) has not yet been answered.

This can be done with _Raw.randomShuffle if you only shuffle on the first
dimension and don’t need the indices anywhere else.
Otherwise it can be done with_Raw.gatherV2 where you specify the axis and
provide the indices.
I am on mobile but can post examples later.
Seems also common enough that adding version outside of _Raw that is
differentials would be handy.
On Sat, Feb 29, 2020 at 5:59 AM Dan Zheng notifications@github.com wrote:

Reopened #394 https://github.com/tensorflow/swift/issues/394.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/swift/issues/394?email_source=notifications&email_token=AAKFCDQOKP3YGYDVNILL5HTRFFUK5A5CNFSM4K4QTJM2YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOW7EAZMQ#event-3083340978,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKFCDQF44GUVPL4NJVH6DLRFFUK5ANCNFSM4K4QTJMQ
.

--
Michael Kowalski
+61 4 2337 9338

An example with _Raw.randomShuffle:
import TensorFlow
let x = Tensor<Int32>(shape: [5, 2], scalars: Array<Int32>(0 ..< 10))
let y = _Raw.randomShuffle(value: x, seed: 1, seed: 2)
print(y) // [[8, 9], [2, 3], [4, 5], [6, 7], [0, 1]]

Even with the seeds I couldn't make this deterministic which surprised me.  So the output shows the shuffling but will be different for each run.
An example with _Raw.gatherV2:
import TensorFlow
let x = Tensor<Int32>(shape: [5, 2], scalars: Array<Int32>(0 ..< 10))
let y = _Raw.gatherV2(params: x, indices: Tensor<Int32>([1, 0]), axis: Tensor<Int32>(1))
print(y) // [[1, 0], [3, 2], [5, 4], [7, 6], [9, 8]]

I will do it.

here's a quick function that does exactly the same as tf.random.shuffle() but also takes an axis dimension
def tf_shuffle_axis(value, axis=0, seed=None, name=None):
    perm = list(range(tf.rank(value)))
    perm[axis], perm[0] = perm[0], perm[axis]
    value = tf.random.shuffle(tf.transpose(value, perm=perm))
    value = tf.transpose(value, perm=perm)
    return value

authors: FarisHijazi dan-zheng kiranchhetri1 mikowals noahtren

repo: tensorflow/swift


text:
How to decode against WMT ENDE model via Tensorflow Serving

Somewhat similar to https://github.com/tensorflow/tensor2tensor/issues/344, I have my model served by TensorFlow Serving API, how can I translate an English sentence against this model now?
Model created by setting export_saved_model=True during training, and served by tensorflow_model_server --model_base_path=/the/path/ --port=9000.

So I've gone further into the abyss, trying to figure this out, and have gotten to the point of sending a predict request to my served model.
Took quite a bit of digging through the t2t source code but I've encoded the text as input_ids = vocabulary.encode(text) and converted to array by np.array(x).astype(np.int32).
I construct request as request.inputs['inputs'].CopyFrom(         tf.contrib.util.make_tensor_proto(input_ids, dtype=tf.int32, shape=????)) but don't know what to put in for shape, from going through t2t's code I've found reshape calls of either [1, 299, 299, -1] or [1, -1, 1, 1], neither of these work and the error I receive is ValueError: Too many elements provided. Needed at most -89401, but received 4 and ValueError: Too many elements provided. Needed at most 1, but received 4 respectively. I've guessed that the max elements it requires is determined by the multiplication of elements 1 and 2 in the arrays. If I increase these, the request just hangs.
By leaving shape unspecified the error is AbortionError(code=StatusCode.INVALID_ARGUMENT, details="Expected multiples argument to be a vector of length 2 but got length 5
Any help would be much appreciated.
R.

What is the result of saved_model_cli show?
In my case,
  request.inputs['inputs'].CopyFrom(tf.contrib.util.make_tensor_proto([[[[1]]], [[[2]]]]))

worked for this model
# saved_model_cli show --dir /path/to/saved/model/dir --tag_set serve --signature_def serving_default
The given SavedModel SignatureDef contains the following input(s):
inputs['inputs'] tensor_info:
    dtype: DT_INT32
    shape: (-1, 1, 1, 1)
    name: ExpandDims_2:0
The given SavedModel SignatureDef contains the following output(s):
outputs['outputs'] tensor_info:
    dtype: DT_INT32
    shape: (-1, -1)
    name: strided_slice_8:0
Method name is: tensorflow/serving/predict

The response is
(outputs {
  key: "outputs"
  value {
    dtype: DT_INT32
    tensor_shape {
      dim {
        size: 2
      }
      dim {
        size: 9
      }
    }
    int_val: 5
    int_val: 3
    int_val: 1
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 5
    int_val: 3
    int_val: 1
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
  }
}

The server side log is
2017-10-20 06:55:26.749815: I external/org_tensorflow/tensorflow/core/kernels/logging_ops.cc:79] beam_decode batch_size=[2]

Now I wonder how I can input a sequence of multiple words in batch size 1, not 2 sequences of one word.
And I also wonder how the beam size, etc. can be specified in a request.

Hi @ninotoshi,
I get the same output for saved_model_cli show and running request.inputs['inputs'].CopyFrom(tf.contrib.util.make_tensor_proto([[[[1]]], [[[2]]]])) does successfully return a response
outputs {
  key: "outputs"
  value {
    dtype: DT_INT32
    tensor_shape {
      dim {
        size: 2
      }
      dim {
        size: 19
      }
    }
    int_val: 337
    int_val: 468
    int_val: 226
    int_val: 1
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 3
    int_val: 1
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
    int_val: 0
  }
}

In your make_tensor_proto call what does [[[[1]]], [[[2]]]] encode? I have been getting inputs_ids for string 'hello world', and putting that directly in make_tensor_proto, which gives me errors as it's an array.
Cheers,
R.

[[[[1]]], [[[2]]]] is just test data. I assume this must be encoded from a list of strings such as ['hello world', 'foo bar baz'] with encode() like
https://github.com/tensorflow/tensor2tensor/blob/f1b75861d8c9927fbc13643a6d58b60d2f3d08b0/tensor2tensor/data_generators/text_encoder_test.py#L132
using a vocabulary file (e.g. vocab.enfr.32768).
Because of
inputs['inputs'] tensor_info:
    dtype: DT_INT32
    shape: (-1, 1, 1, 1)

it seems make_tensor_proto only accepts a list of integers such as [[[[1]]], [[[2]]]] where -1 in shape means a variable length.
In a translation problem, the input should be (-1, -1). I don't know why the shape is (-1, 1, 1, 1).
It is also painful to convert text to integers to make a request and convert integers in a response to text on the client side.

Hi,
I've followed the encoding, as is done in the decoding.py file https://github.com/tensorflow/tensor2tensor/blob/f1b75861d8c9927fbc13643a6d58b60d2f3d08b0/tensor2tensor/utils/decoding.py#L428
Following from that you can see how they encode the numpy array. Neither the input_ids array, or the inputs array as encoded in the above file, work in the make_tensor_proto function.
Furthermore, setting shape in make_tensor_proto to (-1, 1, 1, 1), gives varying errors like ValueError: Too many elements provided. Needed at most -1, but received 4.
I agree that it's not ideal having to convert text to integers on client side and the opposite with the result, but if there was documentation on how to do it, then it would be fine. I opened this issue 10 days ago, and am not much closer to a solution than I was, when I started.
R.

Just to let you know: we're looking at it too, but got sidetracked by other things. Converting to ints on the client allows to do tokenization (which is really hard in TF graphs), or did you have sth else in mind?

Hi,
I don't have a problem with converting to ints on the client at all, in fact, I expected that to be the case when I looked into serving one of these models.
All that I am concerned with, is how to send the array of ints that represent a string to a served model. I've tried to see if I can figure it out from T2T's source code, but have gotten only so far.
I think I'm correct in how I'm encoding the string, following the decoding.py file as I mentioned previously, just need to find out how to send that to the server, and following from that, how to decode the result received back to a string.
It's not a pressing concern for me, as I've looked into alternatives for machine translation in the meantime, but I would like to get this running also at some point too.
Thanks,
R.

@lukaszkaiser if tokenizing on the server side is not easy at all with TF graphs, does this mean that the best route to serve a model + (tok / detok layers) is to go with an adhoc API ?

Not sure what you mean by adhoc API, but yes, you'll have to encode, then send your request, then decode the response.
And just fyi, the serving input function is defined here:
https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/data_reader.py#L369

If you're able to get the predict result from the server, you can use something along these lines to convert the results to a user readable string of target language text:
    output_ids = decoding._save_until_eos(result["outputs"].flatten())
    output_pieces = targets_vocab.decode_list(output_ids)
    output_token = [{"text": piece} for piece in output_pieces]
    output = targets_vocab.decode(output_ids)

This requires a private method that I'd like to make public _save_until_eos that the binaries use internally to convert token indices.
I have a webserver wrapping t2t that does all this internally that I'm hoping to check in soonish as a demonstration.

Hi is there any update on this? Like @robertBrnnn , I am struggling with encoding the input to the translation server. Following the discussion above, I have created preprocessing layers to tokenize the input sentence, and then encoded to integer using vocabulary. Now how do I convert this integer array to request input?

authors: fozziethebeat lukaszkaiser ninotoshi robertBrnnn rsepassi vince62s vishalnus


text:
AttributeError: module 'tensorflow' has no attribute 'constant'

Please go to Stack Overflow for help and support:
http://stackoverflow.com/questions/tagged/tensorflow
If you open a GitHub issue, here is our policy:

It must be a bug or a feature request.
The form below must be filled out.
It shouldn't be a TensorBoard issue. Those go here.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
TensorFlow installed from (source or binary):
TensorFlow version (use command below):
Bazel version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:
Exact command to reproduce:

You can collect some of this information using our environment capture script:
https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh
You can obtain the TensorFlow version with
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
Describe the problem
Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.
Source code / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

I apologize but I am having a hard time understanding what the problem is, where the problem is, and what version it affects. Please resubmit and pay attention to the issue template (https://github.com/tensorflow/tensorflow/issues/new) . Please provide all the information it asks. Thank you.

repo: tensorflow/tensorflow


text:
[Feature Request] Feed tensors to feed_dict

I think that a nice step towards "dynamic computation graphs" is the possibility to feed tensors to placeholders.
I am building an application which creates a network.
I create my training and testing batches with tf.train.batch.
However, the training batches are unlimited (so no epoch limit to the tf.train.slice_input_producer) but I would like to also test during training so to only test one epoch on the testing data.
For both the training and testing images, I have a preprocessing pipeline which also uses a computation graph, but is different for testing and for training.
Now the problem is that I have to create the testing batch every time I want to test (because I want to test for 1 epoch, and I cannot otherwise limit the input producing...).
So it would definitely help, to be able to feed the training and testing batch respectively to the network input.
My current solution is to create the network every time I make the switch from training to testing with the new network inputs. But this takes quite an (unnecessary) long amount of time...
Is there perhaps another way to "link" two graphs in tensorflow?

It doesn't really make sense to feed a Tensor object. Each tensor represents output of a an op and can correspond to different values, whereas numpy array is a specific value. Instead of feed_dict you can use standard Python API to indicate that you want tensor to be an input to another node.  A common scenario is to only use one graph, and have different parts of the graph correspond to training vs testing parts of the network.

authors: AndreiCostinescu yaroslavvb


text:
Weights and biases not being updated.

(I've already tried stackoverflow)
I've made this neural net to figure out whether a house is a good buy or a bad buy. For some reasons the code is not updating weights and biases. My loss stays same. This is my code:
import pandas as pd
import tensorflow as tf

data = pd.read_csv("E:/workspace_py/datasets/good_bad_buy.csv")

features = data.drop(['index', 'good buy'], axis = 1)
lbls = data.drop(['index', 'area', 'bathrooms', 'price', 'sq_price'], axis = 1)

features = features[0:20]
lbls = lbls[0:20]

print(features)
print(lbls)
n_examples = len(lbls)

# Model

# Hyper parameters

epochs = 100
learning_rate = 0.1
batch_size = 1

input_data = tf.placeholder('float', [None, 4])
labels = tf.placeholder('float', [None, 1])

weights = {
			'hl1': tf.Variable(tf.random_normal([4, 10])),
			'hl2': tf.Variable(tf.random_normal([10, 10])),
			'hl3': tf.Variable(tf.random_normal([10, 4])),
			'ol': tf.Variable(tf.random_normal([4, 1]))
			}

biases = {
			'hl1': tf.Variable(tf.random_normal([10])),
			'hl2': tf.Variable(tf.random_normal([10])),
			'hl3': tf.Variable(tf.random_normal([4])),
			'ol': tf.Variable(tf.random_normal([1]))
			}

hl1 = tf.nn.relu(tf.add(tf.matmul(input_data, weights['hl1']), biases['hl1']))
hl2 = tf.nn.relu(tf.add(tf.matmul(hl1, weights['hl2']), biases['hl2']))
hl3 = tf.nn.relu(tf.add(tf.matmul(hl2, weights['hl3']), biases['hl3']))
ol = tf.nn.sigmoid(tf.add(tf.matmul(hl3, weights['ol']), biases['ol']))

loss = tf.reduce_mean((labels - ol)**2)
train = tf.train.AdamOptimizer(learning_rate).minimize(loss)

sess = tf.Session()
sess.run(tf.global_variables_initializer())

iterations = int(n_examples/batch_size)

for epoch_no in range(epochs):
    ptr = 0
	for iteration_no in range(iterations):
		epoch_input = features[ptr:ptr+batch_size]
		epoch_label = lbls[ptr: ptr+batch_size]
		ptr = ptr + batch_size
		_, err = sess.run([train, loss], feed_dict={input_data: features, labels: lbls})
	print("Error at epoch ", epoch_no, ": ", err)

print(sess.run(ol, feed_dict={input_data: [[2104, 3, 399900, 190.0665]]}))

This is the dataset:
Features:

    area  bathrooms   price    sq_price
0   2104          3  399900  190.066540
1   1600          3  329900  206.187500
2   2400          3  369000  153.750000
3   1416          2  232000  163.841808
4   3000          4  539900  179.966667
5   1985          4  299900  151.083123
6   1534          3  314900  205.280313
7   1427          3  198999  139.452698
8   1380          3  212000  153.623188
9   1494          3  242500  162.315930
10  1940          4  239999  123.710825
11  2000          3  347000  173.500000
12  1890          3  329999  174.602645
13  4478          5  699900  156.297454
14  1268          3  259900  204.968454
15  2300          4  449900  195.608696
16  1320          2  299900  227.196970
17  1236          3  199900  161.731392
18  2609          4  499998  191.643542
19  3031          4  599000  197.624546

labels:

    good buy
0        1.0
1        0.0
2        1.0
3        0.0
4        1.0
5        0.0
6        0.0
7        1.0
8        0.0
9        0.0
10       1.0
11       1.0
12       1.0
13       1.0
14       0.0
15       1.0
16       0.0
17       1.0
18       1.0
19       1.0

Any suggestions on how to fix this?

This is a stackoverflow question. When double posting please provide link to your stackoverflow entry  i.e. https://stackoverflow.com/questions/46264133/weights-and-biases-not-updating-in-tensorflow
(which I had to go searching for).

authors: ParmuSingh aselle


text:
TypeError: Cannot interpret feed_dict key as Tensor: Can not convert a NoneType into a Tensor.

Error:
/usr/lib64/python3.6/site-packages/h5py/init.py:34: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
2018-07-27 08:35:51.967246: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: FMA
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1075, in _run
subfeed, allow_tensor=True, allow_operation=False)
File "/usr/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3590, in as_graph_element
return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
File "/usr/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3679, in _as_graph_element_locked
types_str))
TypeError: Can not convert a NoneType into a Tensor.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "--------------.py", line 397, in
tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "/usr/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "---------------------.py", line 88, in main
print(prediction(e))
File "---------------------.py", line 361, in prediction
model_prediction = sess.run(pred_label, feed_dict={x: subject_npy, y: y_const_npy, keep_prob: 1.0})
File "/usr/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/usr/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1078, in _run
'Cannot interpret feed_dict key as Tensor: ' + e.args[0])
TypeError: Cannot interpret feed_dict key as Tensor: Can not convert a NoneType into a Tensor.

I've tried what some stackoverflow posts recommended, e.g. keeping the feed_dict placeholder names as x and y, when either saving & restoring the graph or just making new placeholders.
My objective: I'm trying to restore and run an lstm.
My code:
"""setup for training-------(no issue with training)"""
out_weights = tf.Variable(tf.random_normal([num_units,n_classes]), name = "out_weights")
out_bias = tf.Variable(tf.random_normal([n_classes]), name = "out_bias")
x = tf.placeholder("float",[None,time_steps,n_input])
y = tf.placeholder("float",[None,n_classes])
keep_prob = tf.placeholder("float", shape=())
input = tf.unstack(x ,time_steps,1, name = "inputs")
lstm_layer = rnn.BasicLSTMCell(num_units,forget_bias=1)
outputs, _ = rnn.static_rnn(lstm_layer,input,dtype="float32")
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=prediction,labels=y))
tf.summary.scalar('cross_entrophy', loss)
opt = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(loss)
pred_label = tf.argmax(prediction,1)
actual_label = tf.argmax(y,1)
correct_prediction = tf.equal(pred_label, actual_label)
accuracy = tf.reduce_mean(tf.cast(correct_prediction,tf.float32))
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
saver = tf.train.Saver()
for i in range(N):
sess.run(, feed_dict = {x: batch[0], y: batch[1], keep_prob: 0.5})
if i%100==0:
saver.save(sess, "path/model.ckpt")
"""LOADING  lstm"""
with tf.Session() as sess:
from_saver = tf.train.import_meta_graph('path/model.ckpt.meta')
from_saver.restore(sess, tf.train.latest_checkpoint('path_to_summaries_dir'))
lstm_graph = tf.get_default_graph()
out_weights = lstm_graph.get_tensor_by_name("out_weights:0")
out_bias = lstm_graph.get_tensor_by_name("out_bias:0")
x = tf.placeholder("float",[time_steps,n_input])
y = tf.placeholder("float",[n_classes])
keep_prob = keep_prob = tf.placeholder("float", shape=())
inputs = lstm_graph.get_tensor_by_name("inputs:0")
prediction = lstm_graph.get_tensor_by_name("prediction:0") + out_bias
pred_label = tf.argmax(prediction,1)
"""RUNNING MODEL"""
with tf.Session(graph=lstm_graph) as sess:
sess.run(tf.global_variables_initializer())
model_prediction = sess.run(pred_label, feed_dict={x: subject_npy, y: y_const_npy, keep_prob: 1.0})

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce
Mobile device

authors: ECE-Engineer tensorflowbutler


text:
lose some early epoch checkpoint file during training

Have I written custom code: Y
OS Platform and Distribution: centos release 6.5
TensorFlow installed from: pip install
TensorFlow version: 1.40
Bazel version: N/A
CUDA/cuDNN version: Cuda 8.0
GPU model and memory: N/A
Exact command to reproduce: N/A
Mobile device: N/A
I found myself losing some checkpoint in the target dir after training. For example, I lose 1-13 epoch checkpoint files after 60-epoch training.
I used tf.train.Saver and tf.train.Supervisor in my code:
tf.train.Supervisor(             is_chief=is_chief,             logdir=FLAGS.log_dir,             global_step=global_step,             saver=tf.train.Saver(var_list=var_list,max_to_keep=80,sharded=True),             save_model_secs=FLAGS.save_model_secs,             save_summaries_secs=FLAGS.save_summary_secs,             init_op=tf.global_variables_initializer(),             local_init_op=tf.group(tf.local_variables_initializer(),tf.tables_initializer())
where save_model_secs=1800, save_summary_secs=1800
FYI, it costs me about 30 minutes for training 1 epoch.WHY?

@suzirui123 Hi, just to confirm, have you looked into protocol buffer to check the list of checkpoints ?

@harshini-gadige I'm using distributed_training, could you please tell me where to find the protocol buffer you mentioned?

@harshini-gadige Do you mean freeze the graph to check the pb of checkpoints? I haven't tried that.

@aselle  - Hi, could you please look into this issue ?

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!


text:
Continuously differentiable eigendecomposition

System information

TensorFlow version (you are using): 1.12
Are you willing to contribute it (Yes/No): Yes, but I'd probably need some guidance.

Describe the feature and the current behavior/state.
Currently, when any two eigenvalues are equal, the tf.gradients( tf.self_adjoint_eig(matrix), matrix)  function returns NaN.
It was proposed to return a subgradient (https://github.com/tensorflow/tensorflow/issues/12853) in this case, which equals v_k^T*v_k for kth eigenvalue. However, there exists a procedure which allows to generate continuously differentiable eigendecomposition. This allows to have calculate true gradient (of a particluar decomposition) as soon as eigenbasis exists. Intuitively, if we choose a particular consistent way to choose eigenvectors from subspace, the eigendecomposition would be well defined.
Possibly related issue (same about SVD): https://github.com/tensorflow/tensorflow/issues/13641
Will this change the current api? How? The eigendecomposition will probably change in cases it is not unique.
Who will benefit with this feature? People dealing with optimizations based on eigenvalues/eigenvectors.

ping

@rmlarsen as far as I understand, you are the prime developer of linalg related stuff. Could you mentor me on this one or propose someone who could mentor me?

@Randl,
Sorry for the delayed response. Can you please let us know if tf.linalg.eigh is the API that you are looking for? Thanks!

Yes, I think it was fixed by introducing Lorenz broadening here: https://github.com/tensorflow/tensorflow/pull/33808/files#diff-589f3bb3ae164fc660ae23ebe4eceabf5700612ecef44785c47ce788bd8e4584R712

authors: Randl rmothukuru


text:
[TFLite] Feature request: Add support for DepthToSpace op

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):Ubuntu 16.04
TensorFlow installed from (source or binary):source
TensorFlow version (or github SHA if from source):1.13

Provide the text output from tflite_convert
...  Converting unsupported operation: DepthToSpace

Any other info / logs
DepthToSpace op has been well used for implementing subpixel convolution (aka pixel shuffle) like this super-resolution task.
I know SELECT_TF_OPS option. But its binary size is too huge for mobile application.
Would you like to add DepthToSpace to built-in ops?

@aselle @stakemura  Hi, I am interested in using the depth_to_space op. Is there a workaround for this in the meantime till it is added to the list of supported ops? I am able to convert my model to tflite by specifying
converter.target_ops = [tf.lite.OpsSet.TFLITE_BUILTINS,tf.lite.OpsSet.SELECT_TF_OPS]
converter.allow_custom_ops = True # this is for DEPTH_TO_SPACE

However, the interpreter fails with the error:
ValueError: Didn't find custom op for name 'DEPTH_TO_SPACE' with version 1
Registration failed.

Hey! When will DepthToSpace be added?

If depth_to_space isn’t supported even now, Can you tell me an alternative to depth_to_space which I can convert to tflite.

@stakemura,
Sorry for the delayed response. Can you please let us know if tfl.depth_to_space (TFL::DepthToSpaceOp) is the functionality that you are looking for? Thanks!

authors: mouryareddy purva98 rmothukuru stakemura tejaskhot


text:
libtensorflow.dll leaks icu symbols on Windows

System information

OS Platform and Distribution: Windows 10
TensorFlow installed from (source or binary): Self-compiled with bazel & from https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-windows-x86_64-1.12.0.zip
TensorFlow version: 1.12.0 & 1.13.0-rc2

Describe the problem
libtensorflow.dll exports a ton of symbols from icu. They should be private to the dll.
Any other info / logs
C:\Users\mfischer\Desktop>dumpbin /exports tensorflow-1.12.0.dll
Microsoft (R) COFF/PE Dumper Version 14.16.27027.1
Copyright (C) Microsoft Corporation.  All rights reserved.

Dump of file tensorflow-1.12.0.dll

File Type: DLL

  Section contains the following exports for libtensorflow.so

    00000000 characteristics
    5BDBAEF0 time date stamp Fri Nov  2 02:57:04 2018
        0.00 version
           1 ordinal base
        3078 number of functions
        3078 number of names

    ordinal hint RVA      name

          1    0 01841AA0 ??0?$MaybeStackArray@D$0CI@@icu_62@@AEAA@AEBV01@@Z
          2    1 023BC120 ??0?$MaybeStackArray@D$0CI@@icu_62@@QEAA@$$QEAV01@@Z
          3    2 023BC180 ??0?$MaybeStackArray@D$0CI@@icu_62@@QEAA@H@Z
          4    3 023BC200 ??0?$MaybeStackArray@D$0CI@@icu_62@@QEAA@XZ
          5    4 023C5F90 ??0Appendable@icu_62@@QEAA@AEBV01@@Z
          6    5 023C5F90 ??0Appendable@icu_62@@QEAA@XZ
          7    6 0245DC70 ??0BreakIterator@icu_62@@IEAA@AEBV01@@Z
          8    7 0245DCD0 ??0BreakIterator@icu_62@@IEAA@AEBVLocale@1@0@Z
          9    8 0245DD20 ??0BreakIterator@icu_62@@IEAA@XZ
         10    9 02399020 ??0ByteSink@icu_62@@QEAA@XZ
         11    A 02454B40 ??0BytesDictionaryMatcher@icu_62@@QEAA@AEBV01@@Z
         12    B 02454B70 ??0BytesDictionaryMatcher@icu_62@@QEAA@PEBDHPEAUUDataMemory@@@Z
         13    C 023CC750 ??0BytesTrie@icu_62@@AEAA@PEAXPEBX@Z
         14    D 023CC770 ??0BytesTrie@icu_62@@QEAA@AEBV01@@Z
         15    E 023CC7A0 ??0BytesTrie@icu_62@@QEAA@PEBX@Z
         16    F 0245BA10 ??0BytesTrieBuilder@icu_62@@QEAA@AEAW4UErrorCode@@@Z
         17   10 02458DB0 ??0CStr@icu_62@@QEAA@AEBVUnicodeString@1@@Z
         18   11 023D3F10 ??0CacheKeyBase@icu_62@@QEAA@AEBV01@@Z
         19   12 023D3F30 ??0CacheKeyBase@icu_62@@QEAA@XZ
         20   13 02459A80 ??0CanonicalIterator@icu_62@@QEAA@AEBVUnicodeString@1@AEAW4UErrorCode@@@Z
         21   14 02399030 ??0Char16Ptr@icu_62@@QEAA@$$T@Z
         22   15 02399030 ??0Char16Ptr@icu_62@@QEAA@PEAG@Z
         23   16 02399030 ??0Char16Ptr@icu_62@@QEAA@PEA_S@Z
         24   17 02399030 ??0Char16Ptr@icu_62@@QEAA@PEA_W@Z
         25   18 02459090 ??0CharString@icu_62@@QEAA@$$QEAV01@@Z
         26   19 023BC220 ??0CharString@icu_62@@QEAA@AEBV01@AEAW4UErrorCode@@@Z
         27   1A 023BC270 ??0CharString@icu_62@@QEAA@PEBDHAEAW4UErrorCode@@@Z
         28   1B 023BC2C0 ??0CharString@icu_62@@QEAA@VStringPiece@1@AEAW4UErrorCode@@@Z
         29   1C 023BC320 ??0CharString@icu_62@@QEAA@XZ
         30   1D 02459730 ??0CharacterIterator@icu_62@@IEAA@AEBV01@@Z
         31   1E 02459760 ??0CharacterIterator@icu_62@@IEAA@H@Z
         32   1F 02459790 ??0CharacterIterator@icu_62@@IEAA@HH@Z
...

@mika-fischer Could you explain more details on the problem and its context? Thanks!

I'm not sure what to say. libtensorflow.dll exports symbols for applications to use. It should only export the TF... symbols. Instead it also exports all of the symbols of its internal(!) dependency libicu. This is almost certainly a mistake.
The status quo creates problems when trying to link libtensorflow.dll into an application that already uses libicu because then the icu symbols would be present twice.
The build process needs to be fixed to not export those symbols. Looking at http://userguide.icu-project.org/packaging#TOC-Link-to-ICU-statically all you have to do is most probably define the C preprocessor symbol: U_STATIC_IMPLEMENTATION...

I think @hamatake did something to address this in 7e090f6a5412fd5f22b5b75ca7ae7844c3d025e9, but perhaps the fix didn't completely solve the problem?

Actually 7e090f6 should fix this issue. But it definitely is not fixed in https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-windows-x86_64-1.13.1.zip
Are there nightly master libtensorflow zips that I could check to see if it's fixed in master?

I'm not sure if we publish nightlies for that build.
@av8ramit I noticed that you made a change internally (209826891) that caches artifacts for a libtensorflow CPU build on Windows, but do you know if these are published anywhere?

Well, it's easy enough to check: dumpbin /exports tensorflow.dll will print all exported symbols.

@mrry that was for the release packages unfortunately.

Unfortunately that change was rolled back because it didn't actually fix
the build in question. This dropped off my radar for awhile but it's back
on. I've reproduced the link failure on a windows machine and am working on
a fix.
On Fri, Mar 1, 2019 at 3:35 PM Amit Patankar notifications@github.com
wrote:

@mrry https://github.com/mrry that was for the release packages
unfortunately.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/26084#issuecomment-468847701,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEnCCPY-80fZC5C1w7tZqQ9VGBOr6DApks5vSbkjgaJpZM4bPnbn
.

This should be fixed by https://github.com/tensorflow/tensorflow/commit/025aa34d77b0ba1c9035cfa7b3c7f98bcccb1435#diff-76ffb659afb259c45a6303f170a6613a

Thanks @hamatake!

authors: av8ramit hamatake jvishnuvardhan mika-fischer mrry


text:
[TF 2.0 API Docs] tf.math.argmax

Please make sure that this is a documentation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:doc_template
System information

TensorFlow version:  2.0
Doc Link: https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/math/argmax

Describe the documentation issue

Usage example
No usage example is provided.

Visuals, if Applicable
No visuals are included.

Raises listed and defined
No raises listed

Much like #25802, documentation for tf.math.argmax is created from a generated file python/ops/gen_math_ops.py; a link to the file that generates python/ops/gen_math_ops.py would be handy for users.
Related files to be updated: tensorflow/core/api_def/base_api/api_def_ArgMax.pbtxt, /tensorflow/core/ops/math_ops.cc
We welcome contributions by users. Will you be able to update submit a PR (use the doc style guide) to fix the doc Issue?

@mandroid6 Can you create single issue with all the similar modifications? It is too much overhead to work on 8 similar issues and it will be better for everyone (including you). Could you combine rest of the seven similar issues into one issue? Thanks!

@jvishnuvardhan Yes sure,my understanding was to create individual issues so that new GSoC students could easily pick them up as their first task 😃.
Will combine them into a single issue by tomorrow!

@mandroid6 Are you going to create single issue? please let me know so that i can close other 8 similar issues. Thanks!

@jvishnuvardhan still waiting to combine them as new students have already started working on individual issues. Please wait till tomorrow. Thanks

@mandroid6 Thanks for your support.

@Mandar-Shinde Can I take reference from here for adding examples just like it is mentioned in this Issue - https://github.com/tensorflow/tensorflow/issues/25802

Closing this in favor of #26532 as work has been done on #26540 for both this and that


text:
Building TensorFlow on RaspberryPi

I'm trying to build TensorFlow on a Raspberry Pi (OS : Raspbian Jessie), but I get this error :
http://pastebin.com/zUfP4xsP
I use this command to build TensorFlow :
sudo bazel build -c opt --local_resources 1024,1.0,1.0 --verbose_failures --ignore_unsupported_sandboxing --copt="-mfpu=neon" tensorflow/tools/pip_package:build_pip_package
Any help ?

@petewarden  - do you know what the current status is of TF on RaspberryPi?

It's a bit tricky to build using Bazel, since you need a USB drive plugged in so that in can swap beyond the 1GB of RAM on the Pi. @samjabrahams has a great rundown here:
https://github.com/samjabrahams/tensorflow-on-raspberry-pi
If you only need to run models, not train them, we're have makefile support for the core inference part of TensorFlow here:
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/makefile
https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/pi_examples

Closing since this isn't a specific bug, and RaspberryPi is not an offically supported platform (afaik).
It makes sense to redirect discussion to https://github.com/samjabrahams/tensorflow-on-raspberry-pi

Awesome- I hadn't seen the makefile folders in contrib yet- I'll add in that information to the Raspberry Pi repo after I give it a look-over.


text:
cannot find symbol class Fillwhere T is a type-variable:T extends Object declared in class Zeros

System information

Have I written custom code: no
OS Platform and Distribution: Windows 10
Mobile device: Huawei P9 Lite (Android 6.0)
TensorFlow installed from: source but i did not build it, read below
TensorFlow version: 1.13.1 but on anaconda and in my situation i'm not using any installation read below.
Python version: 3.6.8 same thing.
Bazel version (if compiling from source): 0.21.0
CUDA/cuDNN version: 10.0 / 7.3.1
GPU model and memory: GEFORCE 840M 4GB

Current behavior
I'm using Tensorflow's android example but it would fail to compile unless i changed nativeBuildSystem from 'bazel' to 'none' in build.gradle.
The expected behavior
I must use a compiler because if i don't, then Object Tracking feature wouldn't be available in the app.
Code to reproduce the issue
I cloned the repo (but did not build it) and got bazel up and running, i then opened the android example in Android Studio, that is all to reproduce this i guess. I'm only using the android example do i need to build tensorflow from source because i've struggled with that for months.
All Errors
cannot find symbol class Fillwhere T is a type-variable:T extends Object declared in class Zeros
cannot find symbol class Fillwhere T is a type-variable:T extends Object declared in class Zeros
cannot find symbol variable Fillwhere T is a type-variable:T extends Object declared in class Zeros

I have the same problem, is  there  anyone that has found a solution?

Is there a reason you cannot use TensorFlow Lite? In general, TensorFlow Mobile development is being deprecated, and its build system is not well-maintained, particularly on the gradle (non-bazel) end.

It has been 14 days with no activity and the awaiting response label was assigned. Is this still an issue?

It has been 29 days with no activity and the awaiting response label was assigned. Is this still an issue?

It has been 44 days with no activity and the awaiting response label was assigned. Is this still an issue?

We are closing this issue for now due to lack of activity. Please comment if this is still an issue for you. Thanks!

authors: iseegr8tfuldeadppl jdduke nachovoss tensorflowbutler


text:
add_update in cross-replica mode is broken (BatchNormalization layer impossible to use)

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Archlinux
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): v1.12.1-3374-g9eb67b17bf 2.0.0-dev20190605
Python version: 3.6
CUDA/cuDNN version: 10
GPU model and memory: 1080 Ti

Describe the current behavior
I expect to do a forward pass with a model with a BachNormalization layer in training mode, when using the tf.distribuite.MirroredStrategy but I can't, because it reises the following exception:

RuntimeError: add_update was called in a cross-replica context. This is not expected. If you require this feature, please file an issue.

Why it is not expected?
Describe the expected behavior
It should work.
The commit that introduced this behavior is: https://github.com/tensorflow/tensorflow/commit/316cd57883166e6a0b4c2d0eaacebddad7675b39#diff-8eb7e20502209f082d0cb15119a50413
Code to reproduce the issue
import tensorflow as tf

model = tf.keras.Sequential(
    [
        tf.keras.layers.Dense(10),
        tf.keras.layers.BatchNormalization(),
        tf.keras.layers.Dense(1),
    ]
)

strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
    out = model(tf.zeros((1, 10)), training=True)

I was able to reproduce the issue on Colab with TensorFlow version 2.0.0-dev20190605.

does this have been resolved yet?

I met same question recently，does this have been resolved yet？

Has this been resolved?  I can't use tf.distribute.MirroredStrategy with BatchNormalization in a training loop

Hi @galeone, when using custom training loops with DistributionStrategy you have to use experimental_run_v2, please see: https://www.tensorflow.org/guide/distribute_strategy#using_tfdistributestrategy_with_custom_training_loops

HI @omalleyt12 , I'm my tests it still continues to fail, even when using experimental_run_v2.
If the system has more then one GPU (and thus the distribution strategy can really distribute the computation) the same exception is raised even if I change the code in this way:
import tensorflow as tf

strategy = tf.distribute.MirroredStrategy()
with strategy.scope():

  model = tf.keras.Sequential([
        tf.keras.layers.Dense(10),
        tf.keras.layers.BatchNormalization(),
        tf.keras.layers.Dense(1),
    ])

  def forward():
    return model(tf.zeros((1, 10)), training=True)

  print(strategy.experimental_run_v2(forward, args=()))

I hit the same problem when I use batch normalization, too.
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/layers/normalization.py", line 659, in call
    outputs = self._fused_batch_norm(inputs, training=training)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/layers/normalization.py", line 556, in _fused_batch_norm
    self.add_update(mean_update)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 1113, in add_update
    '`add_update` was called in a cross-replica context. This is not '
RuntimeError: `add_update` was called in a cross-replica context. This is not expected. If you require this feature, please file an issue

The problem happens while Model.build()-s so there is no way to try experimental_run_v2 for me.

@galeone @vmarkovtsev this should be fixed in the latest nightly, could you please check and confirm?

@vmarkovtsev if it's not fixed for your use case, could you provide a simple repro?

I confirm that this issue is fixed now. Thank you @omalleyt12 !

Thanks!

@omalleyt12 I met the same problem in 2.0.0rc.

@3fen can you share a simple reproduction that is failing?

@omalleyt12  it is similary with the main thread.
model = tf.keras.Sequential(
    [
        tf.keras.layers.Dense(10),
        tf.keras.layers.BatchNormalization(),
        tf.keras.layers.Dense(1),
    ]
)

strategy = tf.distribute.MirroredStrategy()
with strategy.scope():
    out = model(tf.zeros((1, 10)), training=True)
    print(out)

Outupt:

RuntimeError: add_update was called in a cross-replica context. This is not expected. If you require this feature, please file an issue.

@3fen , when using custom training loops with DistributionStrategy you have to use experimental_run_v2, please see: https://www.tensorflow.org/guide/distribute_strategy#using_tfdistributestrategy_with_custom_training_loops

@omalleyt12 Got it, thanks for the confirmation.

I hit the same problem when I use batch normalization, too.
i am using tensorflow 2.0 b1..

@DAEHEESHIN v2.0 b1 is ancient, we are discussing the current nightly here.

I'm having the same issue as @vmarkovtsev using tf2 and tf-nightly when I use tf.function and get_concrete_function:
tf.function(train_model.__call__, autograph=False).get_concrete_function(                 tf.TensorSpec([batch_size_per_replica, *flags_obj.image_dimension], tf.float32),                 ...            )

@gdj0nes Could you share a larger code snippet that repros this?

@omalleyt12 i'm working on making something I can share, just wanted to make sure there wasn't already a clear fix. It seems like it's an issue with setting the training parameter.

@omalleyt12, here is an example snippet using tf-nightly:
import tensorflow as tf
from tensorflow.keras import layers

class Model(tf.Module):

    def __init__(self):
        super().__init__()

        with self.name_scope:
            self.layers = [
                layers.Conv2D(10, (3, 3)),
                layers.BatchNormalization()
            ]

    def no_param_call(self, input):
        x = input
        for layer in self.layers:
            x = layer(x)

        return x

    def param_call(self, input):
        x = input
        for layer in self.layers:
            x = layer(x, training=True)

        return x

def app():
    strategy = tf.distribute.MirroredStrategy()
    image_dimension = []
    with strategy.scope():
        model = Model()
        no_param_forward_fn = tf.function(model.no_param_call, autograph=False).get_concrete_function(
                tf.TensorSpec([1, 64, 64, 3], tf.float32)
        )
        print('no param call succeeded')
        param_forward_fn = tf.function(model.param_call, autograph=False).get_concrete_function(
            tf.TensorSpec([1, 64, 64, 3], tf.float32)
        )
        print('param call succeeded')

if __name__ == '__main__':
    app()

Output:
2019-10-28 13:20:34.225201: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fbc6e390880 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2019-10-28 13:20:34.225217: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
WARNING:tensorflow:There are non-GPU devices in `tf.distribute.Strategy`, not using nccl allreduce.
no param call succeeded
Traceback (most recent call last):
  File "/Users/{{omitted}}/scratches/scratch_62.py", line 48, in <module>
    app()
  File "/Users/{{omitted}}/scratches/scratch_62.py", line 42, in app
    tf.TensorSpec([1, 64, 64, 3], tf.float32)
  File "/Users/{{omitted}}lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 902, in get_concrete_function
    self._initialize(args, kwargs, add_initializers_to=initializers)
  File "/Users/{{omitted}}lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 497, in _initialize
    *args, **kwds))
  File "/Users/{{omitted}}lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 2365, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/Users/{{omitted}}lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 2673, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/Users/{{omitted}}lib/python3.6/site-packages/tensorflow_core/python/eager/function.py", line 2563, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/Users/{{omitted}}lib/python3.6/site-packages/tensorflow_core/python/framework/func_graph.py", line 958, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/Users/{{omitted}}lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py", line 439, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/Users/gareth/Library/Preferences/PyCharm2019.3/scratches/scratch_62.py", line 27, in param_call
    x = layer(x, training=True)
  File "/Users/{{omitted}}lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 778, in __call__
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "/Users/{{omitted}}lib/python3.6/site-packages/tensorflow_core/python/keras/layers/normalization.py", line 695, in call
    outputs = self._fused_batch_norm(inputs, training=training)
  File "/Users/{{omitted}}lib/python3.6/site-packages/tensorflow_core/python/keras/layers/normalization.py", line 592, in _fused_batch_norm
    self.add_update(mean_update)
  File "/Users/{{omitted}}lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "/Users/{{omitted}}lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 1208, in add_update
    '`add_update` was called in a cross-replica context. This is not '
RuntimeError: `add_update` was called in a cross-replica context. This is not expected. If you require this feature, please file an issue.```

Indeed, this issue has returned.

I find this issue to be very tricky.
The core of the problem comes from creating the batchnorm layer with training=True under the strategy scope. If you look at the code, it checks whether training is True and indeed calls add_update. This comes out only when you try to subclass a Keras model.
I have found a workaround though: wrap your subclassed model in another one:
my_model = Model(...)
tf.keras.Model(inputs=[tf.keras.layers.Input(shape=...)], outputs=my_model.call(training=True))

That wrapping gives an additional bonus of being able to build() and properly summary() the model.

I guess this issue should be re-opened, I'm facing again the same issue

Same issue for me!
It only occurs if I use tf.keras.callbacks.ModelCheckpoint, no problem otherwise...

@omalleyt12 - are you looking into this?
@rchao could this one of the issues you've fixed recently with checkpointing?

@guptapriya Thanks for reopening this.
For your information, this issue does not seems to occur for the .h5 format, only the TF format is is affected, as pointed out in my recent issue (which is a duplicate of this one, sorry):

Note: no error if we use .h5 instead of TF checkpoint format. However, just a few hours ago, @karmel and @jvishnuvardhan recommended me to use TF format (because .h5 can cause issues, okay fine). So, which one should I use finally? None is working? I am lost...

@netw0rkf10w @vmarkovtsev  @galeone What tf version are you using? This should be fixed in the nightly

Recently we have submitted a few fixes related to using batch norm layer with tf.distribute strategies. If you could try with the nightly and let us know if it's fixed it'd be great.

I've updated my tf to tf-nightly (2.1.0-dev20200104) and this problem still exists.
Any idea?

@galeone @vmarkovtsev  @netw0rkf10w  Issue seems to be fixed in latest TF version. Please confirm and close the issue if it is resolved for you. Thanks!

Any updates regarding this issue? Thanks!

Closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

In TF 2.1.0 the issue is not fixed... anybody help me ?
Is this issue fixed in TF 2.2 ?

@taki0112 I don't understand. Why didn't you simply upgrade to TF 2.2 and see if it works for you?

authors: 3fen DAEHEESHIN XuChunqiao achandraa davesc galeone gdj0nes guptapriya hassanhub luvwinnie netw0rkf10w omalleyt12 rchao saikumarchalla taki0112 vmarkovtsev


text:
Error in Android Example

Hello,
I want to tell that I am using source code which I got from tensorflow.com/lite website. Now I am frustrated to solve this error. Please see screenshot and let me know. how to fix it and use this. When I start the app then it takes seconds to take to crash my app.

@KanwarpartapSinghBimrah ,
Please provide details about what platform you are using (operating system, architecture). Also include your TensorFlow version.
In order to expedite the trouble-shooting process, please provide complete code snippet to reproduce the issue reported here in accessible format. Thanks

@oanush Now it's working fine and I fix all the errors. if in future I need help I let you know.

Thank you for the feedback.Closing since the issue is resolved.

authors: KanwarpartapSinghBimrah oanush


text:
error: can't copy 'tensorflow/python/ops/gen_sparse_ops.py': doesn't exist or not a regular file

After I compiled the source code of TF0.10 (the latest version) using command "bazel build -c opt --config=cuda //tensorflow/tools/pip_package:build_pip_package", I tried to generate the .whl package using the following command:
"bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg"
However, errors occured:
Thu Aug 11 20:10:14 EDT 2016 : === Using tmpdir: /tmp/tmp.r1OX0vuucW
/tmp/tmp.r1OX0vuucW ~/bxl/tf1.0
Thu Aug 11 20:10:14 EDT 2016 : === Building wheel
error: can't copy 'tensorflow/python/ops/gen_sparse_ops.py': doesn't exist or not a regular file
Anyone knows how to solve this? Thx

I had the same error, but for me there was already an error during the build process. Check if you have the same problem (missing /usr/local/cuda/ dependencies) and then fix the problem by adding a line to the file third_party/gpus/crosstool/CROSSTOOL, see here: https://github.com/tensorflow/tensorflow/issues/3589#issuecomment-236430359
This fix works for me using both CUDA Toolkit version 7.5 and 8.0 RC.

No, I don't have the cuda error.
Is there any idea to solve the problem in generating whl package?

Can you show us the full output of all your build commands?

I found that the build is not successful and has the following error: @girving
ERROR: /home/dl/bxl/tf-0815/tensorflow/contrib/rnn/BUILD:46:1: undeclared inclusion(s) in rule '//tensorflow/contrib/rnn:python/ops/_lstm_ops_gpu':
this rule is missing dependency declarations for the following files included by 'tensorflow/contrib/rnn/kernels/lstm_ops_gpu.cu.cc':
'/usr/local/cuda-7.5/include/cuda_runtime.h'
'/usr/local/cuda-7.5/include/host_config.h'
'/usr/local/cuda-7.5/include/builtin_types.h'
'/usr/local/cuda-7.5/include/device_types.h'
'/usr/local/cuda-7.5/include/host_defines.h'
'/usr/local/cuda-7.5/include/driver_types.h'
'/usr/local/cuda-7.5/include/surface_types.h'
'/usr/local/cuda-7.5/include/texture_types.h'
'/usr/local/cuda-7.5/include/vector_types.h'
'/usr/local/cuda-7.5/include/channel_descriptor.h'
'/usr/local/cuda-7.5/include/cuda_runtime_api.h'
'/usr/local/cuda-7.5/include/cuda_device_runtime_api.h'
'/usr/local/cuda-7.5/include/driver_functions.h'
'/usr/local/cuda-7.5/include/vector_functions.h'
'/usr/local/cuda-7.5/include/vector_functions.hpp'
'/usr/local/cuda-7.5/include/common_functions.h'
'/usr/local/cuda-7.5/include/math_functions.h'
'/usr/local/cuda-7.5/include/math_functions.hpp'
'/usr/local/cuda-7.5/include/math_functions_dbl_ptx3.h'
'/usr/local/cuda-7.5/include/math_functions_dbl_ptx3.hpp'
'/usr/local/cuda-7.5/include/cuda_surface_types.h'
'/usr/local/cuda-7.5/include/cuda_texture_types.h'
'/usr/local/cuda-7.5/include/device_functions.h'
'/usr/local/cuda-7.5/include/device_functions.hpp'
'/usr/local/cuda-7.5/include/device_atomic_functions.h'
'/usr/local/cuda-7.5/include/device_atomic_functions.hpp'
'/usr/local/cuda-7.5/include/device_double_functions.h'
'/usr/local/cuda-7.5/include/device_double_functions.hpp'
'/usr/local/cuda-7.5/include/sm_20_atomic_functions.h'
'/usr/local/cuda-7.5/include/sm_20_atomic_functions.hpp'
'/usr/local/cuda-7.5/include/sm_32_atomic_functions.h'
'/usr/local/cuda-7.5/include/sm_32_atomic_functions.hpp'
'/usr/local/cuda-7.5/include/sm_35_atomic_functions.h'
'/usr/local/cuda-7.5/include/sm_20_intrinsics.h'
'/usr/local/cuda-7.5/include/sm_20_intrinsics.hpp'
'/usr/local/cuda-7.5/include/sm_30_intrinsics.h'
'/usr/local/cuda-7.5/include/sm_30_intrinsics.hpp'
'/usr/local/cuda-7.5/include/sm_32_intrinsics.h'
'/usr/local/cuda-7.5/include/sm_32_intrinsics.hpp'
'/usr/local/cuda-7.5/include/sm_35_intrinsics.h'
'/usr/local/cuda-7.5/include/surface_functions.h'
'/usr/local/cuda-7.5/include/surface_functions.hpp'
'/usr/local/cuda-7.5/include/texture_fetch_functions.h'
'/usr/local/cuda-7.5/include/texture_fetch_functions.hpp'
'/usr/local/cuda-7.5/include/texture_indirect_functions.h'
'/usr/local/cuda-7.5/include/texture_indirect_functions.hpp'
'/usr/local/cuda-7.5/include/surface_indirect_functions.h'
'/usr/local/cuda-7.5/include/surface_indirect_functions.hpp'
'/usr/local/cuda-7.5/include/device_launch_parameters.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/Core'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/util/DisableStupidWarnings.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/util/Macros.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/util/MKL_support.h'
'/usr/local/cuda-7.5/include/cuda_fp16.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/util/Constants.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/util/Meta.h'
'/usr/local/cuda-7.5/include/math_constants.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/util/ForwardDeclarations.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/util/StaticAssert.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/util/XprHelper.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/util/Memory.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/NumTraits.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/MathFunctions.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/GenericPacketMath.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/arch/CUDA/Half.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/arch/CUDA/PacketMathHalf.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/arch/CUDA/TypeCasting.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/arch/CUDA/PacketMath.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/arch/CUDA/MathFunctions.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/arch/Default/Settings.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/functors/TernaryFunctors.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/functors/BinaryFunctors.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/functors/UnaryFunctors.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/functors/NullaryFunctors.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/functors/StlFunctors.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/functors/AssignmentFunctors.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/DenseCoeffsBase.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/DenseBase.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/plugins/BlockMethods.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/MatrixBase.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/plugins/CommonCwiseUnaryOps.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/plugins/CommonCwiseBinaryOps.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/plugins/MatrixCwiseUnaryOps.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/plugins/MatrixCwiseBinaryOps.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/EigenBase.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Product.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/CoreEvaluators.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/AssignEvaluator.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Assign.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/ArrayBase.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/plugins/ArrayCwiseUnaryOps.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/plugins/ArrayCwiseBinaryOps.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/util/BlasUtil.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/DenseStorage.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/NestByValue.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/ReturnByValue.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/NoAlias.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/PlainObjectBase.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Matrix.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Array.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/CwiseTernaryOp.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/CwiseBinaryOp.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/CwiseUnaryOp.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/CwiseNullaryOp.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/CwiseUnaryView.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/SelfCwiseBinaryOp.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Dot.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/StableNorm.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Stride.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/MapBase.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Map.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Ref.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Block.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/VectorBlock.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Transpose.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/DiagonalMatrix.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Diagonal.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/DiagonalProduct.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Redux.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Visitor.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Fuzzy.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/IO.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Swap.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/CommaInitializer.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/GeneralProduct.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Solve.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Inverse.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/SolverBase.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/PermutationMatrix.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Transpositions.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/TriangularMatrix.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/SelfAdjointView.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/products/GeneralBlockPanelKernel.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/products/Parallelizer.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/ProductEvaluators.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/products/GeneralMatrixVector.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/products/GeneralMatrixMatrix.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/SolveTriangular.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/products/GeneralMatrixMatrixTriangular.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/products/SelfadjointMatrixVector.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/products/SelfadjointMatrixMatrix.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/products/SelfadjointProduct.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/products/SelfadjointRank2Update.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/products/TriangularMatrixVector.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/products/TriangularMatrixMatrix.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/products/TriangularSolverMatrix.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/products/TriangularSolverVector.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/BandMatrix.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/CoreIterators.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/ConditionEstimator.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/BooleanRedux.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Select.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/VectorwiseOp.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Random.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Replicate.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/Reverse.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/ArrayWrapper.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/GlobalFunctions.h'
'/home/dl/bxl/fix-0815/eigen-eigen-6f952374ef2b/Eigen/src/Core/util/ReenableStupidWarnings.h'
'/usr/local/cuda-7.5/include/curand_kernel.h'
'/usr/local/cuda-7.5/include/curand.h'
'/usr/local/cuda-7.5/include/curand_discrete.h'
'/usr/local/cuda-7.5/include/curand_precalc.h'
'/usr/local/cuda-7.5/include/curand_mrg32k3a.h'
'/usr/local/cuda-7.5/include/curand_mtgp32_kernel.h'
'/usr/local/cuda-7.5/include/cuda.h'
'/usr/local/cuda-7.5/include/curand_mtgp32.h'
'/usr/local/cuda-7.5/include/curand_philox4x32_x.h'
'/usr/local/cuda-7.5/include/curand_globals.h'
'/usr/local/cuda-7.5/include/curand_uniform.h'
'/usr/local/cuda-7.5/include/curand_normal.h'
'/usr/local/cuda-7.5/include/curand_normal_static.h'
'/usr/local/cuda-7.5/include/curand_lognormal.h'
'/usr/local/cuda-7.5/include/curand_poisson.h'
'/usr/local/cuda-7.5/include/curand_discrete2.h'.
nvcc warning : option '--relaxed-constexpr' has been deprecated and replaced by option '--expt-relaxed-constexpr'.
nvcc warning : option '--relaxed-constexpr' has been deprecated and replaced by option '--expt-relaxed-constexpr'.
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 103.576s, Critical Path: 94.31s

@martinwicke Another undeclared inclusion; do you know if this is fixed?  @gclouding Can you try with the latest source to see if it's different?

The problem was finally solved by adding the library path of cuda and eigen in the cxx_builtin_include_directory in the file  ./third_party/gpus/crosstool/CROSSTOOL @girving

This issue looks like it is now solved?  Please reopen if the problem is still present.

authors: gclouding girving prb12 tomrunia


text:
Segmentation fault with nightly-gpu docker

Please make sure that this is a bug. As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock
example script provided in TensorFlow): No
OS Platform and Distribution (e.g.,
Linux Ubuntu 16.04): Ubuntu 18.04.3 LTS
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if
the issue happens on mobile device: No
TensorFlow installed from (source or
binary): - TensorFlow version (use command below): Nightly-gpu (2.2)
Python version: - Bazel
version (if compiling from source): Python 3.6.9
GCC/Compiler version (if compiling from
source): N/A
CUDA/cuDNN version: - GPU model and memory: RTX 2080 Ti 11GB CUDA 10.1

You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with: 1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" 2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"
Describe the current behavior
Segmentation fault (core dumped) mid-train after calling model.fit()
I use a callback to check memory and it seems like may be there is a memory leak.
Describe the expected behavior
Not have this happen
Standalone code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate
the problem. If possible, please share a link to Colab/Jupyter/any notebook.
Cannot provide, i have a large dataset (10gb for training inputs).
I can train fine on my local 1060 6gb card with same amount of CPU memory (32GB). But it seems like this docker image has a problem with a memory leak.
Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.
No logs, just this error.

@ben-arnao,
In order to expedite the trouble-shooting process, we need the reproducible code.
Is it possible to provide the minimal code snippet with a sample dataset. Thanks!

Any updates regarding this issue? Thanks!

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

authors: amahendrakar ben-arnao


text:
python program spitting out a tensorflow error of "ImportError: DLL Load failed while importing"

System information

Windows 10 pro 64bit ver 2004
TensorFlow version: 2.2.0
Python version: 3.8.3
Installed using PIP
CUDA ver: 10.0
GPU model and memory: NVIDIA GTX 1070 8GB
-My CPU does not support the AVX Instruction set

when attempting to use a python program, I am greeted with this error (Copied Verbatim below)
(side note, I'm not great with python or most of the dev tools I've attempted to use to download all of the required libraries, so forgive me if this has a relatively simple solution.)
This occurs when I use the CMD prompt to execute the main.py file for this program. to do this, I typed this command
Python main.py
after a few moments the error is triggered. the prompt remains blank until the error traceback appears.
(I can give more specifics to the best of my ability, but this is all of the information that I am aware that I have to help diagnose)
(this is the entire error verbatim)
Error when importing libraries:  Traceback (most recent call last):
File "G:\APPLICATIONS\Python\3.8.3\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "G:\APPLICATIONS\Python\3.8.3\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "G:\APPLICATIONS\Python\3.8.3\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "G:\APPLICATIONS\Python\3.8.3\lib\imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "G:\APPLICATIONS\Python\3.8.3\lib\imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: DLL load failed while importing _pywrap_tensorflow_internal: A dynamic link library (DLL) initialization routine failed.
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors
for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.
Some Python libraries are missing. You can install all required libraries by running in the command line 'pip install -r requirements.txt'

is there a way around using the 2.2.0 version of TF without the AVX instruction set? right now I just want to see this work, processing time and efficiency do not matter at the moment.

@SugoiShades
What is make/model of your cpu?
I suspect your cpu model does not support AVX instructions sets.See hardware requirements
Make sure to download the latest microsoft visual c++ redistributable from here.
.Also, please follow the instructions from to install from Tensorflow website.
TensorFlow release binaries version 1.6 and higher are prebuilt with AVX instruction sets.
Therefore on any CPU that does not have these instruction sets, either CPU or GPU version of TF will fail to load.
If your CPU model does not support AVX instruction sets. You can still use TensorFlow with the alternatives given below:
Try Google Colab to use TensorFlow.
The easiest way to use TF will be to switch to google colab. You get pre-installed latest stable TF version. Also you can use pip install to install any other preferred TF version.
It has an added advantage since you can you easily switch to different hardware accelerators
(cpu, gpu, tpu) as per the task.
All you need is a good internet connection and you are all set.
Try to build TF from sources by changing CPU optimization flags.
Thanks!

@ravikyram My CPU Definitely doesn't support AVX (its a Core i7 980) I but you answered my question, thank you.

@SugoiShades
Please, close this thread if your query has been answered.Thanks!

I am closing this issue  since the query has been answered. Thanks!

authors: SugoiShades ravikyram


text:
windows build issues(finally made it)

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): windows 10 x64
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A
TensorFlow installed from (source or binary): source
TensorFlow version: 2.4.0
Python version: 3.8.3 x64
Installed using virtualenv? pip? conda?: N/A
Bazel version (if compiling from source): 3.4.1
GCC/Compiler version (if compiling from source): visual studio 2019
CUDA/cuDNN version: 11.0/8.0.1
GPU model and memory: RTX2070 GDDR6 8GB

Describe the problem
many problems with windows build, and I made a few fixes
Provide the exact sequence of commands / steps that you executed before running into the problem

cudnn_stub  dependency problem ( https://github.com/tensorflow/tensorflow/issues/41057 )
I simply add dependency in that BUILD

then thrust version unmatch error
I delete thrust version - cub version comparison codes from
bazel-out\x64_windows-opt\bin\external\local_config_cuda\cuda\cuda\include\thrust\system\cuda\config.h

finally, big old numpy problem
this has been very long and unsolvable error, I made same fix as before( simply pip uninstall numpy ( uninstall conda numpy ) and pip install numpy )

and still there are too many "ignoring unknown options" from cl
anyway now I succeeded build and have my wheel file
Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

anyway now I succeeded build and have my wheel file

@alanpurple,
Is this still an issue? Please feel free to close the issue if resolved. Thanks!

this is 4 issues, 3 I fixed myself,  1(warnings) I can't fix

think of this post as kinda PR since I made fixes

@alanpurple,
In this case, could you please submit a new PR for the fix. Thanks!

@amahendrakar
Is conda numpy problem unsolvable?

Is conda numpy problem unsolvable?

Could you please describe this issue and provide the steps to reproduce it, so that we can look into it? Thanks!

@amahendrakar
#35977

@alanpurple,
Can we close this issue since it is already being tracked there? Thanks!

@amahendrakar
ok

@surak I believe this is what he did:
https://github.com/tensorflow/tensorflow/pull/41822/commits/d480a3ad713e50057a2b1a611e7cacd6bbca1eb5
Didn't realize that cudnn_version.h was already defined somewhere in the file. Anyhow, tried it again with fresh source for 2.3.0 and with this change and it builds successfully on Windows now.

I still get a directory without the correct file. All I get is
In file included from tensorflow/stream_executor/cuda/cudnn_stub.cc:15:
bazel-out/k8-py2-opt/bin/external/local_config_cuda/cuda/_virtual_includes/cudnn_header/third_party/gpus/cudnn/cudnn.h:59:10: fatal error: cudnn_version.h: No such file or directory
   59 | #include "cudnn_version.h"
      |          ^~~~~~~~~~~~~~~~~
compilation terminated.
SUBCOMMAND: # //tensorflow/core:function_ops_op_lib [action 'Compiling tensorflow/core/ops/function_ops.cc']
(cd /tmp/eb-w9wkEo/tmpSnNZgZ-bazel-build/execroot/org_tensorflow && \

And this directory has
ls -la bazel-out/k8-py2-opt/bin/external/local_config_cuda/cuda/_virtual_includes/cudnn_header/third_party/gpus/cudnn
total 12
drwxrwxr-x 2 strube1 jusers 4096 Aug 25 19:27 .
drwxrwxr-x 3 strube1 jusers 4096 Aug 25 19:27 ..
lrwxrwxrwx 1 strube1 jusers  139 Aug 25 19:27 cudnn.h -> /tmp/eb-w9wkEo/tmpSnNZgZ-bazel-build/execroot/org_tensorflow/bazel-out/k8-py2-opt/bin/external/local_config_cuda/cuda/cudnn/include/cudnn.h

I don't really understand Bazel's syntax. Can you help me with that?

@isleong @alanpurple turns out that it's somewhere else. cudnn 8 has different headers from that cudnn.h and bazel fails to identify the correct version. It's at third_party/gpus/cuda_configure.bzl line 1164:
This is the problematic part:
    if [int(x) for x in cuda_config.cudnn_version.split(".")] < [8, 0]:
      cudnn_headers = ["cudnn.h"]
    else:
      cudnn_headers = ["cudnn_adv_infer.h",
                       "cudnn_adv_train.h",
                       "cudnn_cnn_infer.h",
                       "cudnn_cnn_train.h",
                       "cudnn_ops_infer.h",
                       "cudnn_ops_train.h",
                       "cudnn.h",
                       "cudnn_version.h",
                      ]

This cudnn version is somehow wrong.
This is my patch:
--- third_party/gpus/cuda_configure.bzl.orig	2020-08-26 18:55:08.405838123 +0200
+++ third_party/gpus/cuda_configure.bzl		2020-08-26 19:24:47.466666178 +0200
@@ -1136,10 +1136,11 @@
         out_dir = "cuda/bin",
     ))

-    if [int(x) for x in cuda_config.cudnn_version.split(".")] < [8, 0]:
-      cudnn_headers = ["cudnn.h"]
-    else:
-      cudnn_headers = ["cudnn_adv_infer.h",
+    #if [int(x) for x in cuda_config.cudnn_version.split(".")] < [8, 0]:
+    #  cudnn_headers = ["cudnn.h"]
+    #else:
+    cudnn_headers = ["cudnn_adv_infer.h",
+                       "cudnn_backend.h",
                        "cudnn_adv_train.h",
                        "cudnn_cnn_infer.h",
                        "cudnn_cnn_train.h",

This will break with older cuDNNs, but turns out that because check fails, I really don't care. I only need this version of TensorFlow for the A100 gpus, which only support CUDA 11 and cuDNN 8+ anyway.

authors: alanpurple amahendrakar isleong surak


text:
Confusing error in gradient of assign_add + while_loop + gather

Describe the current behavior
Autograph produces a static graph with the wrong number of inputs when a trainable variable is in a separate class. I also get a warning about converting a sparse op to a dense one, which seems to be related (while to gather, problem goes away when removed).
Describe the expected behavior
The function should execute without errors.
Standalone code to reproduce the issue
https://colab.research.google.com/drive/1x36cCyKsSxsOCw1Z4mnR4AzbwDSuXzTY#scrollTo=O6GzPt2gZiPF

Turns out it was unrelated to the secondary class, I've updated the colab accordingly

Interestingly, the error actually seems to go away if you switch to gather_nd
https://colab.research.google.com/drive/1Pa8g79qxnzlzf0oUM0cEtDoJrOUZSICd#scrollTo=cJDdoS6OR_Fc

I am able to replicate this issue, please find the gist here

It looks like certain inputs are indeed mismatched in the graph, but the error happens in the gradient ops. The error message about incorrect number of inputs looks like a red herring. Will update the bug title a bit.

Actually, I think assign_add is unrelated, it seems to fail with any graph op that requires the variable, eg
https://colab.research.google.com/drive/11MZorVrToy1bng_GfgmSWwum4z6kUvEL#scrollTo=wGyr_8aK0iv5

I think this commit fixes the bug (colab), should I close the issue @mdanatg?

Yep looks like the issue is resolved and can be closed. Thanks!

authors: Saduf2019 kentslaney mdanatg