text:
USE-QA v3 to KerasLayer with trainable = True fail on fine-tune the model

(i) I define USE-QA for response and question encodes
USE_qa_response =  hub.KerasLayer('use-model/use-qa-model',                        input_shape=[],                        dtype=tf.string,                       output_key='outputs',                       signature='response_encoder',                        trainable=True)
USE_qa_question =  hub.KerasLayer('use-model/use-qa-model',                        input_shape=(),                        dtype=tf.string,                       output_key='outputs',                       signature='question_encoder',                        trainable=True)
(ii) Then, create a network for fine-tune the USE-QA
encode1 = USE_qa_response({'input': input1, 'context': context1})
  encode2 = USE_qa_question(questions)
(iii) After that, the error show
ValueError: Setting hub.KerasLayer.trainable = True is unsupported when calling a SavedModel signature.

Environment:
tensorflow == 2.4.1 tensorflow-hub == 0.11.0
If I change trainable = False it works! but the USE-QA won't fine-tune or change any parameter inside the model.
What should I do?

@mrpeerat,
The behavior seems to be expected as per the documentation of Arguments of Hub Keras Layer, mentioned below:

trainable | Optional. A boolean controlling whether this layer is trainable. Must not be set to True when using a signature (raises ValueError), including the use of legacy TF1 Hub format.

Since you are using signature='response_encoder', the argument, trainable should be set to False.

@mrpeerat,
The behavior seems to be expected as per the documentation of Arguments of Hub Keras Layer, mentioned below:

trainable | Optional. A boolean controlling whether this layer is trainable. Must not be set to True when using a signature (raises ValueError), including the use of legacy TF1 Hub format.

Since you are using signature='response_encoder', the argument, trainable should be set to False.

But if we set trainable = False the USE-QA will update the value or parameter inside the model?
Or what if we delete signature='response_encoder', I can't imagine how to implement fine-tune the USE-QA without the signature parameter, any suggestion?

Here is more information on why trainable = True is not supported when calling a SavedModel signature: https://github.com/tensorflow/hub/blob/a452309e69a9ec85641534a84016e92007785451/tensorflow_hub/keras_layer.py#L259
https://github.com/tensorflow/hub/issues/428#issuecomment-647191531 discusses a very similar problem and potential workarounds. Please reopen if you have further questions.

authors: maringeo mrpeerat rmothukuru

repo: tensorflow/hub


text:
sketch-rnn: Time to train

Hello, thanks for this amazing algorithm!
I am writing a Pytorch version of sketch-rnn, and I would like to know what is the average time of training with a simple laptop's GPU. (using the same default hyperparameters, and only training to generate samples from the cat.npz dataset)
Also, what is approximatively the minimal number of epoch in order to get recognizable drawing generations?

Ok !
Thanks a lot for the tips !

You should be able to see some results (i.e. it should draw something that look like crabs if you crab), after 10-20K steps. We set the default number of steps close to infinity to allow us to control manually when we want to stop training, by looking at the TensorBoard graphs and the various reported losses. In practice once the validation loss curves stop decreasing we stop the training
The convergence depends on the data, and also the regularization scheme you use, and size of the network, so can vary around 50K-500K steps even.

Ok. That helps a lot. Many thanks @hardmaru

The loss function for the decoder is somewhat non standard, so you may encounter some challenges if you want to implement in a high level framework such as Keras, but I think it should be possible.
If you manage to write the model in Keras in the future, please let us know so we can refer your implementation to Keras users or other interested parties.

Ok sure.. We're making an attempt. Will share if we get some good results.

@hardmaru @alexis-jacq @falaktheoptimist @alexiskattan
I implemented Sketch-RNN in keras.
The repo can be found here.
Any comments are welcomed.
Cheers

source: gharchive/issue

authors: KKeishiro alexis-jacq falaktheoptimist hardmaru

repo: tensorflow/magenta

license_type: permissive

license_source: github-api


text:
T2T model's loss method calculates incorrect values?

Hi, I have a question regarding this piece of code in the T2T model class:
  def loss(self, logits, features):
    if isinstance(logits, dict):
      losses = {}
      for k, v in logits.iteritems():
        losses[k] = self._loss_single(v, features)
      return tf.add_n([n / d for n, d in logits.values()])
    else:
      return self._loss_single(logits, features)

So if the logits argument is a dict, then the return value is a sum of the logit values instead of losses dictionary values. But the method self._loss_single should output the numerator and denumerator.
Is this correct?
For the corresponding lines of code see https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/utils/t2t_model.py#L335-L336

Nice catch, thanks Adrian!

Nice catch, thanks Adrian!


text:
tf.matching_files order of returned files

As far as I can tell from https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/platform/file_system.cc, the order of filenames returned by tf.matching_files can be non-determinstic.
If that is correct, it would be nice if that were stated in the documentation (and also for Dataset.list_files and train.match_filenames_once).
Even better would be to guarantee alphabetical order, but I am not sure about the performance overhead that would incur.
This would enable to process files given as e.g.
A/1.png, A/2.png, ... and B/1.png, ... jointly by doing to match_files followed by a zip.

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

repo: tensorflow/tensorflow


text:
TensorFlow1.5.0 absl.flags._exceptions.UnparsedFlagAccessError in Jupyter Notebook

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 1.5.0
Python version:  3.6
Bazel version (if compiling from source): n/a
GCC/Compiler version (if compiling from source): n/a
CUDA/cuDNN version: 9.0/7.0.5
GPU model and memory:GeForce 950M(4Gb)
Exact command to reproduce:

Describe the problem
I run the function cifar10.maybe_download_and_extract() of tensorflow/models/tutorials/image/cifar10.py file in python command and jupyter notebook .The code runs successfully in python command,  but it causes absl.flags._exceptions.UnparsedFlagAccessErrorfrom cifar10.FLAGS.data_dir in jupyter notebook.
The jupyter notebook has already configured the tensorflow kernel, other tensorflow examples could run smoothly on jupyter notebook.
Significant code in cifar10.py file is:
import tensorflow as tf
import cifar10_input
FLAGS = tf.app.flags.FLAGS

# Basic model parameters.
tf.app.flags.DEFINE_integer('batch_size', 128,
                        """Number of images to process in a batch.""")
tf.app.flags.DEFINE_string('data_dir', '/tmp/cifar10_data',
                       """Path to the CIFAR-10 data directory.""")
tf.app.flags.DEFINE_boolean('use_fp16', False,
                        """Train the model using fp16.""")
# .....
def maybe_download_and_extract():
    """Download and extract the tarball from Alex's website."""
    dest_directory = FLAGS.data_dir
    if not os.path.exists(dest_directory):
        os.makedirs(dest_directory)
# ......

The exception happened from the code statement dest_directory = FLAGS.data_dir.
The successful output in python command is as follows:
>>> cifar10.maybe_download_and_extract()
>> Downloading cifar-10-binary.tar.gz 100.0%
Successfully downloaded cifar-10-binary.tar.gz 170052171 bytes.

The detailed error in jupyter notebook is as follows:
UnrecognizedFlagError                     Traceback (most recent call last)
<ipython-input-6-ddaee26bfd14> in <module>()
----> 1 cifar10.maybe_download_and_extract()
~/models/tutorials/image/cifar10/cifar10.py in maybe_download_and_extract()
381 def maybe_download_and_extract():
382   """Download and extract the tarball from Alex's website."""
--> 383   dest_directory = FLAGS.data_dir
384   if not os.path.exists(dest_directory):
385     os.makedirs(dest_directory)
~/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/platform/flags.py in
__getattr__(self, name)
82     # a flag.
83     if not wrapped.is_parsed():
---> 84       wrapped(_sys.argv)
85     return wrapped.__getattr__(name)
86
~/anaconda3/envs/tensorflow/lib/python3.6/site-packages/absl/flags/_flagvalues.py in __call__(self, argv, known_only)
628       suggestions = _helpers.get_flag_suggestions(name, list(self))
629       raise _exceptions.UnrecognizedFlagError(
--> 630           name, value, suggestions=suggestions)
631
632     self.mark_as_parsed()
UnrecognizedFlagError: Unknown command line flag 'f'

Source code / logs
#git clone https://github.com/tensorflow/models.git
#cd models/tutorials/image/cifar10
import cifar10
cifar10.maybe_download_and_extract()

I do not have much experience with Jupyter myself.
Maybe @caisq or @random-forests may know, or may know who to redirect this to?

Nagging Awaiting TensorFlower: It has been 14 days with no activity and the awaiting tensorflower label was assigned. Please update the label and/or status accordingly.

Nagging TensorFlower: It has been 14 days with no activity and the awaiting tensorflower label was assigned. Please update the label and/or status accordingly.

Nagging Assignee @skye: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @caisq: It has been 37 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @caisq: It has been 52 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @caisq: It has been 67 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @caisq: It has been 82 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @caisq: It has been 97 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @caisq: It has been 127 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @caisq: It has been 142 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @caisq: It has been 157 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @caisq: It has been 172 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @caisq: It has been 187 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @caisq: It has been 202 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @caisq: It has been 217 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

authors: bignamehyp jhljx tensorflowbutler


text:
Tensorflow failed due to error LNK2019 when build with MSVC

System information
•	Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
N/A
•	OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Windows server 2016
•	TensorFlow installed from (source or binary):
Source
•	TensorFlow version (use command below):
Master branch latest revison
•	Python version:
Anaconda 4.1.1 (Python 3.5 64-bit)
•	Bazel version (if compiling from source):
N/A
•	GCC/Compiler version (if compiling from source):
VS2017 15.5.7
•	CUDA/cuDNN version:
NVidia CUDA Toolkit 8.0
NVidia CUDNN 5.1
•	GPU model and memory:
N/A
•	Exact command to reproduce:
N/A
Describe the problem:
Tensorflow failed to build due to the error LNK2019 and error LNK1120. This should be tensorflow source issue, could you please help take a look at this? Thanks!
The failures like:
The whole log file please see attachment.
[log_x64_build.log](https://github.com/tensorflow/tensorflow/files/1942013/log_x64_build.log)
(Link target) ->
         c_api.cc.obj : error LNK2019: unresolved external symbol "public: __cdecl tensorflow::CppShapeInferenceResult_HandleShapeAndType::CppShapeInferenceResult_HandleShapeAndType(void)" (??0CppShapeInferenceResult_HandleShapeAndType@tensorflow@@QEAA@XZ) referenced in function "protected: class tensorflow::CppShapeInferenceResult_HandleShapeAndType * __cdecl google::protobuf::internal::RepeatedPtrFieldBase::Add<class google::protobuf::RepeatedPtrField<class tensorflow::CppShapeInferenceResult_HandleShapeAndType>::TypeHandler>(class tensorflow::CppShapeInferenceResult_HandleShapeAndType *)" (??$Add@VTypeHandler@?$RepeatedPtrField@VCppShapeInferenceResult_HandleShapeAndType@tensorflow@@@protobuf@google@@@RepeatedPtrFieldBase@internal@protobuf@google@@IEAAPEAVCppShapeInferenceResult_HandleShapeAndType@tensorflow@@PEAV45@@Z) [D:\Tensorflow\build_x64\tensorflow.vcxproj]
         c_api.cc.obj : error LNK2019: unresolved external symbol "protected: __cdecl tensorflow::CppShapeInferenceResult_HandleShapeAndType::CppShapeInferenceResult_HandleShapeAndType(class google::protobuf::Arena *)" (??0CppShapeInferenceResult_HandleShapeAndType@tensorflow@@IEAA@PEAVArena@protobuf@google@@@Z) referenced in function "protected: class tensorflow::CppShapeInferenceResult_HandleShapeAndType * __cdecl google::protobuf::internal::RepeatedPtrFieldBase::Add<class google::protobuf::RepeatedPtrField<class tensorflow::CppShapeInferenceResult_HandleShapeAndType>::TypeHandler>(class tensorflow::CppShapeInferenceResult_HandleShapeAndType *)" (??$Add@VTypeHandler@?$RepeatedPtrField@VCppShapeInferenceResult_HandleShapeAndType@tensorflow@@@protobuf@google@@@RepeatedPtrFieldBase@internal@protobuf@google@@IEAAPEAVCppShapeInferenceResult_HandleShapeAndType@tensorflow@@PEAV45@@Z) [D:\Tensorflow\build_x64\tensorflow.vcxproj]
         c_api.cc.obj : error LNK2019: unresolved external symbol "private: void __cdecl tensorflow::CppShapeInferenceResult_HandleShapeAndType::_slow_mutable_shape(void)" (?_slow_mutable_shape@CppShapeInferenceResult_HandleShapeAndType@tensorflow@@AEAAXXZ) referenced in function TFE_GetResourceHandleShapeAndType [D:\Tensorflow\build_x64\tensorflow.vcxproj]
         c_api.cc.obj : error LNK2019: unresolved external symbol "public: __cdecl tensorflow::CppShapeInferenceResult_HandleData::CppShapeInferenceResult_HandleData(void)" (??0CppShapeInferenceResult_HandleData@tensorflow@@QEAA@XZ) referenced in function TFE_GetResourceHandleShapeAndType [D:\Tensorflow\build_x64\tensorflow.vcxproj]
         c_api.cc.obj : error LNK2019: unresolved external symbol "public: virtual __cdecl tensorflow::CppShapeInferenceResult_HandleData::~CppShapeInferenceResult_HandleData(void)" (??1CppShapeInferenceResult_HandleData@tensorflow@@UEAA@XZ) referenced in function TFE_GetResourceHandleShapeAndType [D:\Tensorflow\build_x64\tensorflow.vcxproj]
         D:\Tensorflow\build_x64\Release\tensorflow.dll : fatal error LNK1120: 5 unresolved externals [D:\Tensorflow\build_x64\tensorflow.vcxproj]

Repro steps:

git clone https://github.com/tensorflow/tensorflow D:\Tensorflow\src
pushd D:\Tensorflow
set PreferredToolArchitecture=x64
set rel=Release
set CUDNN_HOME="C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\cuda"
set PY=C:\ProgramData\Anaconda3
set CL=/FS /permissive-
cmake D:\Tensorflow\src\tensorflow\contrib\cmake -A x64 -DCMAKE_BUILD_TYPE=Release -DPYTHON_EXECUTABLE=C:\ProgramData\Anaconda3\python.exe -DPYTHON_LIBRARIES=C:\ProgramData\Anaconda3\libs\python36.lib -DSWIG_EXECUTABLE=D:\Tensorflow\swigwin-3.0.12\swig.exe -Dtensorflow_BUILD_PYTHON_TESTS=ON -Dtensorflow_BUILD_SHARED_LIB=ON
MSBuild /m /p:Configuration=Release;Platform=x64 /p:WindowsTargetPlatformVersion=10.0.16299.0 tensorflow.sln /t:Rebuild

@gunan Can you take a look at this? I don't have a Windows device for reproductions.

There are some known build issues with visual studio 2017. It may just be those.
I dont have a better answer at the moment as we currently use VS 2015 update 3 to build TF on windows.

Hi @gunan. Thanks. I found that this issue can be reproduced from master revision f196351c. Hope it will help solve this issue.

Nagging Assignees @gunan, @angersson: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignees @gunan, @angersson: It has been 31 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

I sent a few fixes and was able to successfully build with vs 2017, but I forgot to close the issues.
Please reopen if you still run into the same problem.

Thank you for fixing this issue, @gunan. I tried your fix and Tensorflow works well.

authors: angersson gunan shanshan0309 tensorflowbutler


text:
No module named 'sparse_dot_topn

Hello all, trying to install the "sparse_dot_topn" package in python using pip install sparse_dot_topn in my mac terminal but failing.How else can i go about this?

Try building it. https://github.com/ing-bank/sparse_dot_topn

This question is better asked on StackOverflow since it is not a bug or feature request. There is also a larger community that reads questions there.
If you think we've misinterpreted a bug, please comment again with a clear explanation, as well as all of the information requested in the issue template. Thanks!

Within Mac anvironment you can download the wheel for:
https://pypi.org/project/sparse-dot-topn/

authors: Culeshovi gtarasconi mas24 reedwm


text:
how to get reg_loss  in eager mode

https://github.com/tensorflow/tensorflow/blob/9e0b05bbc4bb88d1b34fb2147429dc4ad7bd25cd/tensorflow/contrib/eager/python/examples/densenet/densenet_test.py#L105
In this demo, the Conv2D have regluarizer configure but when it computed loss, there is only softmax_loss!!
where reg_loss?????

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce
Mobile device

Hi Johnson,
Thanks a lot for reaching out! Yash wrote this example model, so I'm not totally sure of the details. But giving it a glance I think you're correct. To account for the regularization terms we would need to sum over all the terms in model.losses. I will confirm with Yash on this and get to you real soon.
Chen

Hi Johnson,
We have updated the code. As a bit of explanation, model.losses returns a list of loss tensors (this is what tf.keras.Model do, not eager specific), so we need only add this to the cross entropy within the tape context.
Hope this helped!
Chen

@lxuechen Thank you very much. I will try it now.

authors: Johnson-yue lxuechen tensorflowbutler


text:
RuntimeError: TOCO failed see console for info.

Please go to Stack Overflow for help and support:
https://stackoverflow.com/questions/tagged/tensorflow
If you open a GitHub issue, here is our policy:

It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).
The form below must be filled out.
It shouldn't be a TensorBoard issue. Those go here.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary):
TensorFlow version (use command below):
Python version:
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:
Exact command to reproduce:

You can collect some of this information using our environment capture script:
https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh
You can obtain the TensorFlow version with
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
Describe the problem
Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.
Source code / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

Please provide details about what platform you are using (operating system, architecture). Also include your TensorFlow version. Also, did you compile from source or install a binary?
Make sure you also include the exact command if possible to produce the output included in your test case. If you are unclear what to include see the issue template displayed in the Github new issue template.
We ask for this in the issue submission template, because it is really difficult to help without that information. Thanks!

Closing this issue due to lack of activity. Feel free to reopen this issue when new information is available.
Thanks!.


text:
ERROR: Config value monolithic is not defined in any .rc file

Hello TensorFlow help team,
I am trying to build Tensorflow for Raspberry pi using instructions given on
https://www.tensorflow.org/install/source_rpi
I hit to following error:
make[1]: Leaving directory `/tmp/openblas_src'
Building for the Pi Two/Three, with NEON acceleration
WARNING: The following rc files are no longer being read, please transfer their contents or import their path into one of the standard rc files:
/workspace/tools/bazel.rc
INFO: Options provided by the client:
Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'build' from /etc/bazel.bazelrc:
Inherited 'common' options: --color=yes
INFO: Reading rc options for 'build' from /etc/bazel.bazelrc:
'build' options: --verbose_failures --spawn_strategy=standalone --genrule_strategy=standalone
INFO: Reading rc options for 'build' from /workspace/.tf_configure.bazelrc:
'build' options: --action_env PYTHON_BIN_PATH=/usr/bin/python3 --action_env PYTHON_LIB_PATH=/usr/local/lib/python3.4/dist-packages --python_path=/usr/bin/python3 --define with_jemalloc=true --define with_gcp_support=true --define with_hdfs_support=true --define with_aws_support=true --define with_kafka_support=true --action_env TF_NEED_OPENCL_SYCL=0 --action_env TF_NEED_CUDA=0 --action_env TF_DOWNLOAD_CLANG=0 --define grpc_no_ares=true
ERROR: Config value monolithic is not defined in any .rc file
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
INFO: Invocation ID: 4d7baf49-d76f-47e7-9bbb-f2bc6304fe21
Could you please help me on this?
Thanks..

Can you please provide the following information asked by the template: (relevant in your case)
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary):
TensorFlow version:
Python version:
Installed using virtualenv? pip? conda?:
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:

Describe the problem
Provide the exact sequence of commands / steps that you executed before running into the problem
Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Docker on Mac running ubuntu:14.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: (Building for pi on Mac using Docker using instructions given on https://www.tensorflow.org/install/source_rpi)
TensorFlow installed from (source or binary): source
TensorFlow version: r1.11
Python version: 3.4
Installed using virtualenv? pip? conda?: No
Bazel version (if compiling from source): 0.19.2
GCC/Compiler version (if compiling from source): 4.8
CUDA/cuDNN version: -
GPU model and memory: -

Hello,
I am facing the same issue. Tried to install for python 2.7 and its giving me the same error.
Did you figure out any solution to this problem ?
Thanks you.

@pranoot Not yet, still working on different options.

I got the solution.
Bazel 0.20.0 was getting install which created the problem.
How to solve this problem:

Remove previously created docker images (docker system prune -a)
Remove line rm -rf /usr/local/bin/bazel from /tensorflow/tools/ci_build/install/install_pi_python3_toolchain.sh because bazel 0.15.0 is installed there and we don't want to remove it.
remote "bazel" install from /tensorflow/tools/ci_build/install/install_pi_python3_toolchain.sh (last line).

Hello,
Thanks for the help. I tried to do it for python 2.7 but the build failed again.
Although your advice solved "ERROR: Config value monolithic is not defined in any .rc file" this bug.
Thanks a lot again.
Regards

With Bazel 0.15.2, I'm able to build it.

What was the resolution for this? I"m using the Docker / cross-compilation instructions via https://www.tensorflow.org/install/source_rpi with r1.12 branch.

@alexellis Yes, It's for source build for RPI and I had the same error when i compiled r1.12 branch.

Hey @anilknayak
Do you have the libtensorflow.so and other artefacts hosted somewhere?

No, If you need I can publish it.
On Tue, Apr 2, 2019 at 12:31 PM Rushabh Nagda notifications@github.com
wrote:

Hey @anilknayak https://github.com/anilknayak
Do you have the libtensorflow.so and other artefacts hosted somewhere?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/24104#issuecomment-479159415,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEqtwwKtOqMyDWhw9V0pGTAoVoYAhx6Uks5vc7AogaJpZM4Y9YLl
.

--
Anil Kumar Nayak
Software Engineer - Computer Vision, Deep Learning and Machine Learning
Omnify.ai Inc
1250 Borregas Avenue
Sunnyvale, CA
-- Never give up, there is always a way, find it out.

@anilknayak that's be very helpful. thanks.

Find the Tensorflow artifacts for ARM7 in this drive
https://drive.google.com/open?id=1PBFFMCK87FHuiEhJaOqHJDFnxtDUeO9Y
On Tue, Apr 2, 2019 at 9:10 PM Rushabh Nagda notifications@github.com
wrote:

@anilknayak https://github.com/anilknayak that's be very helpful.
thanks.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/24104#issuecomment-479329845,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEqtw4QH8nV0OL3d4pMT1p9xFbK9YMUSks5vdCmkgaJpZM4Y9YLl
.

--
Anil Kumar Nayak
Software Engineer - Computer Vision, Deep Learning and Machine Learning
Omnify.ai Inc
1250 Borregas Avenue
Sunnyvale, CA
-- Never give up, there is always a way, find it out.

authors: Kishwar alexellis anilknayak pranoot rky0930 rushTENm rushabhnagda11 ymodak


text:
TFRecordDataset constructor does not support globbing pattern

URL(s) with the issue:
Better performance with data: https://www.tensorflow.org/guide/data_performance#structure_of_an_input_pipeline
Description of issue (what needs changing):
The code example provided instantiate a tf.data.TFRecordDataset passing a globbing pattern: "/path/to/dataset/train-*.tfrecord" while it does not support it.
The example should be updated, relying on tf.data.Dataset.list_files.
- dataset = tf.data.TFRecordDataset("/path/to/dataset/train-*.tfrecord")
+ dataset = tf.data.TFRecordDataset(tf.data.Dataset.list_files("/path/to/dataset/train-*.tfrecord"))

Submit a pull request?
I will submit a pull request soon.

This has been fixed with tensorflow/docs#1111.


text:
Check failed: dim_size >= 1 (0 vs. 1)

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):  Ubuntu16.04
TensorFlow installed from (source or binary): binary
TensorFlow version (or github SHA if from source): 12.0

Command used to run the converter or code if you’re using the Python API
bazel run --config=opt tensorflow/contrib/lite/toco:toco -- \
  --input_file=/install_bar/frozen_inference_graph_quantization.pb \
  --input_format=TENSORFLOW_GRAPHDEF  --output_format=TFLITE \
  --output_file=./frozen_inference_graph_quantization.tflite --inference_type=FLOAT \
  --input_data_types =FLOAT --input_arrays=["image_tensor"] \
  --output_arrays="detection_boxes","detection_scores","detection_classes","num_detections","raw_detection_boxes","raw_detection_scores" \
  --input_shapes=1,299,299,3

The output from the converter invocation
2019-11-14 02:55:54.079426: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1127] Op node missing output type attribute: SecondStagePostprocessor/map/TensorArray_2
2019-11-14 02:55:54.079449: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1080] Converting unsupported operation: Enter
2019-11-14 02:55:54.079540: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1080] Converting unsupported operation: Enter
2019-11-14 02:55:54.079595: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1080] Converting unsupported operation: Enter
2019-11-14 02:55:54.079641: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1080] Converting unsupported operation: Enter
2019-11-14 02:55:54.079742: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1080] Converting unsupported operation: LoopCond
2019-11-14 02:55:54.079824: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1127] Op node missing output type attribute: SecondStagePostprocessor/map/while/LoopCond
2019-11-14 02:55:54.079927: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1080] Converting unsupported operation: TensorArrayReadV3
2019-11-14 02:55:54.080043: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1080] Converting unsupported operation: Enter
2019-11-14 02:55:54.080130: I tensorflow/contrib/lite/toco/import_tensorflow.cc:189] Unsupported data type in placeholder op: 20
2019-11-14 02:55:54.080154: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1080] Converting unsupported operation: Enter
2019-11-14 02:55:54.080274: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1080] Converting unsupported operation: TensorArrayReadV3
2019-11-14 02:55:54.080337: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1080] Converting unsupported operation: Enter
2019-11-14 02:55:54.080365: I tensorflow/contrib/lite/toco/import_tensorflow.cc:189] Unsupported data type in placeholder op: 20
2019-11-14 02:55:54.080381: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1080] Converting unsupported operation: Enter
2019-11-14 02:55:54.080610: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1080] Converting unsupported operation: TensorArrayWriteV3
2019-11-14 02:55:54.080645: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1127] Op node missing output type attribute: SecondStagePostprocessor/map/while/TensorArrayWrite/TensorArrayWriteV3
2019-11-14 02:55:54.080667: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1080] Converting unsupported operation: Enter
2019-11-14 02:55:54.080696: I tensorflow/contrib/lite/toco/import_tensorflow.cc:189] Unsupported data type in placeholder op: 20
2019-11-14 02:55:54.080727: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1080] Converting unsupported operation: Exit
2019-11-14 02:55:54.080834: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1080] Converting unsupported operation: TensorArraySizeV3
2019-11-14 02:55:54.080855: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1127] Op node missing output type attribute: SecondStagePostprocessor/map/TensorArrayStack/TensorArraySizeV3
2019-11-14 02:55:54.080950: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1080] Converting unsupported operation: TensorArrayGatherV3
2019-11-14 02:55:54.138743: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before Removing unused ops: 2132 operators, 3712 arrays (0 quantized)
2019-11-14 02:55:54.229959: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After Removing unused ops pass 1: 1999 operators, 3467 arrays (0 quantized)
2019-11-14 02:55:54.334917: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before general graph transformations: 1999 operators, 3467 arrays (0 quantized)
2019-11-14 02:55:54.385257: F tensorflow/contrib/lite/toco/graph_transformations/resolve_constant_slice.cc:59] Check failed: dim_size >= 1 (0 vs. 1)
Aborted (core dumped)

**Also, please include a link to the saved model or GraphDef**

Put link here or attach to the issue.

**Failure details**
If the conversion is successful, but the generated model is wrong,
state what is wrong:
- Producing wrong results and/or decrease in accuracy
- Producing correct results, but the model is slower than expected (model generated from old converter)

**Any other info / logs**

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

Any solution to this issue, please?

@Julius-ZCJ We see that you are using older version of tensorflow .Many bug have been fixed in latest version.  We recommend that you upgrade to latest stable version of tensorflow 2.6.0 and let us know if the issue still persists in newer versions .Thanks!

authors: Julius-ZCJ SirPhemmiey sushreebarsa


text:
WHere is staging package in tf2?

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary):
TensorFlow version (use command below):
Python version:3.6
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:

You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with: 1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" 2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"
Describe the current behavior
Hi there, i want to use the StagingArea structure by using 'from tensorflow.contrib.staging import StagingArea' in old version. How could i use it in tf2, and i see the definition is here :tensorflow/python/ops/data_flow_ops.py
Describe the expected behavior
Code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate the problem.
Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

@YunchuZhang tensorflow.contrib.staging uses StagingArea class as shown here and also yo can find staging area class here
In Tensorflow 2.0 the you can import StagingArea as follows
from tensorflow.python.ops.data_flow_ops import StagingArea
You can refer to the following line for more understanding. Thanks!

Thank you so much!!!
On Fri, Nov 22, 2019 at 6:28 PM gowthamkpr notifications@github.com wrote:

@YunchuZhang https://github.com/YunchuZhang tensorflow.contrib.staging
uses StagingArea class as shown here
https://github.com/tensorflow/tensorflow/blob/590d6eef7e91a6a7392c8ffffb7b58f2e0c8bc6b/tensorflow/contrib/staging/__init__.py#L20
and also yo can find staging area class here
https://github.com/tensorflow/tensorflow/blob/590d6eef7e91a6a7392c8ffffb7b58f2e0c8bc6b/tensorflow/python/ops/data_flow_ops.py#L1770
In Tensorflow 2.0 the you can import StagingArea as follows
from tensorflow.python.ops.data_flow_ops import StagingArea
You can refer to the following line
https://github.com/tensorflow/tensorflow/blob/1cf0898dd4331baf93fe77205550f2c2e6c90ee5/tensorflow/python/ops/data_flow_ops.py#L1769
for more understanding. Thanks!
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/34475?email_source=notifications&email_token=AIAVGNNQ7SW6SQDTGILXWLDQVBTLFA5CNFSM4JP5KE3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE7FCBY#issuecomment-557732103,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AIAVGNICVJ7KRR57PHBMKF3QVBTLFANCNFSM4JP5KE3A
.

Hi there,
do you know which function or class could be used to change the StagingArea
in tf2?
Currently, i want to use tf.Variable in tf2 not placeholder to be put in
stagingarea like this:
self.staging_tf = StagingArea(
dtypes=[tf.float32 for _ in self.stage_shapes.keys()],
shapes=list(self.stage_shapes.values()))
self.buffer_ph_tf = [tf.Variable(tf.ones(shape=shape)) for shape in
self.stage_shapes.values()
tf.compat.v1.placeholder(tf.float32, shape=shape) for shape in
self.stage_shapes.values()
]
self.stage_op = self.staging_tf.put(self.buffer_ph_tf)
batch = self.staging_tf.get()
it comes error:
File "/home/robertmu/baselines/baselines/her/ddpg.py", line 345, in
_create_network
batch = self.staging_tf.get()
File
"/home/robertmu/miniconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_core/python/ops/data_flow_ops.py",
line 1924, in get
return self.__internal_get(fn, name)
File
"/home/robertmu/miniconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_core/python/ops/data_flow_ops.py",
line 1890, in __internal_get
return self._get_return_value(ret, indices)
File
"/home/robertmu/miniconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_core/python/ops/data_flow_ops.py",
line 1740, in _get_return_value
tensors = self._create_device_transfers(tensors)
File
"/home/robertmu/miniconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_core/python/ops/data_flow_ops.py",
line 1717, in _create_device_transfers
curr_device_scope = control_flow_ops.no_op().device
AttributeError: 'NoneType' object has no attribute 'device
On Fri, Nov 22, 2019 at 9:19 PM YUNCHU ZHANG yunchu@ucla.edu wrote:

Thank you so much!!!
On Fri, Nov 22, 2019 at 6:28 PM gowthamkpr notifications@github.com
wrote:

@YunchuZhang https://github.com/YunchuZhang tensorflow.contrib.staging
uses StagingArea class as shown here
https://github.com/tensorflow/tensorflow/blob/590d6eef7e91a6a7392c8ffffb7b58f2e0c8bc6b/tensorflow/contrib/staging/__init__.py#L20
and also yo can find staging area class here
https://github.com/tensorflow/tensorflow/blob/590d6eef7e91a6a7392c8ffffb7b58f2e0c8bc6b/tensorflow/python/ops/data_flow_ops.py#L1770
In Tensorflow 2.0 the you can import StagingArea as follows
from tensorflow.python.ops.data_flow_ops import StagingArea
You can refer to the following line
https://github.com/tensorflow/tensorflow/blob/1cf0898dd4331baf93fe77205550f2c2e6c90ee5/tensorflow/python/ops/data_flow_ops.py#L1769
for more understanding. Thanks!
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/34475?email_source=notifications&email_token=AIAVGNNQ7SW6SQDTGILXWLDQVBTLFA5CNFSM4JP5KE3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE7FCBY#issuecomment-557732103,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AIAVGNICVJ7KRR57PHBMKF3QVBTLFANCNFSM4JP5KE3A
.

@YunchuZhang Please post this question in stack overflow as github is only meant for issues related to bug/performance, build/install, feature requests and bugs. Thanks!

It has been 14 days with no activity and the awaiting response label was assigned. Is this still an issue?

authors: YunchuZhang gowthamkpr tensorflowbutler


text:
ImportError: DLL load failed: The specified module could not be found.

Hi,
I keep hitting an import error shown below
setup is Windows 10 64 bit - Processor	Intel(R) Core(TM) i7-7820HQ - GPU Nividia Quadro M1200,
Running Spyder version 4.0 (tried 4.01 also) with python 3.7.6 (also tried 3.8) via virtual env  with tensorflow 2.  Despite a weeks worth of various combinations I am having no luck. Any help much appreciated
Home setup works fine but I need it working on my work PC. I have a local install as I don't have admin rights.
ImportError: Traceback (most recent call last):
File "C:\Users\user.user\Python 3.7.6 64bit\3.7.6\lib\site-packages\tensorflow_core\python\pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\Users\user.user\Python 3.7.6 64bit\3.7.6\lib\site-packages\tensorflow_core\python\pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\Users\user.user\Python 3.7.6 64bit\3.7.6\lib\site-packages\tensorflow_core\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "C:\Users\user.user\AppData\Local\Programs\Python\Python37\lib\imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "C:\Users\user.user\AppData\Local\Programs\Python\Python37\lib\imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: DLL load failed: The specified module could not be found.
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors
for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.

Closing as duplicate.
https://github.com/tensorflow/tensorflow/issues/36167#issuecomment-577886156
Just to sample over 100 similar issues: #36167 #36151 #36138 #36054 #36045 #36020 #36003 #35988 #35903 #35880 #35865 #35805 #35789 #35773 #35772 #35767 #35766 #35749 #35721 #35618 #35204
Please make sure you do a search in the future.


text:
Center Loss in TF 2.0

I am trying to implement center loss in Tensorflow 2.0 by referring to code in following repositories:

https://github.com/handongfeng/MNIST-center-loss

https://github.com/mjDelta/keras-center-loss-MNIST

https://github.com/Kakoedlinnoeslovo/center_loss

I noticed that some arguments in the method "self.add_update((self.centers,new_centers),x)" when creating a custom layer, are deprecated. The code (linked above) does not throw any error, but the centers are never updated when run in Tensorflow 2.0.
You can find the related description for the problem here:
https://github.com/tensorflow/tensorflow/blob/v2.1.0/tensorflow/python/keras/engine/base_layer.py#L1170-L1259
I am trying to update the weights of the layer after every forward pass so I can use it as tf.Variable which can persist the updated centroids for the next iteration.
Any help on this would be appreciated.
Thanks

@suraj-maniyar
Thanks for reporting the issue. Can you please share the simple standalone code to reproduce the issue. It helps us in localizing the issue faster. Thanks!

It has been 14 days with no activity and the awaiting response label was assigned. Is this still an issue?

@suraj-maniyar
Any update on this issue please. Thanks!

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

Is there a solution to this? I'm also facing exactly the same problem.

To update this, we need to use

tf.keras.backend.update(x, new_x)

as the argument to the add_update function, as it expects an update operation.

authors: pgaggar ravikyram suraj-maniyar tensorflowbutler


text:
How does the "tensorflow/compiler/mlir/tensorflow/ir/tf_generated_ops.td" file generated?

How does the "tensorflow/compiler/mlir/tensorflow/ir/tf_generated_ops.td" file generated?

@jpienaar Hi, is there any script in TF to generate the contents in "tensorflow/compiler/mlir/tensorflow/ir/tf_generated_ops.td" ?

This question is better asked on StackOverflow since it is not a bug or feature request. There is also a larger community that reads questions there and provide better and faster support for such issues. Thanks!

OK. Thanks anyway.

authors: ai-jit ravikyram


text:
Saving/Loading Subclassed Models TF2+/keras

Update-template I don't really see how to fill in this template.
This template is for miscellaneous issues not covered by the other issue categories.
For questions on how to work with TensorFlow, or support for problems that are not verified bugs in TensorFlow, please go to StackOverflow.
If you are reporting a vulnerability, please use the dedicated reporting process.
For high-level discussions about TensorFlow, please post to discuss@tensorflow.org, for questions about the development or internal workings of TensorFlow, or if you would like to know how to contribute to TensorFlow, please post to developers@tensorflow.org.
Hello,
I am having a lot of difficulties saving a subclassed model and reloading it. I have uploaded the current files I'm working on (specifically, policy.py) to https://github.com/ryanmaxwell96/trpo_fractal5NN. Now, I know you are supposed to be able to save things under the SavedModel format so that is what I've been trying to follow. But I'm still getting the error seen here:
'''
Traceback (most recent call last):
File "train.py", line 411, in
main(**vars(args))
File "train.py", line 367, in main
policy_model.save('policy_model')
File "/home/ryan/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/network.py", line 975, in save
signatures, options)
File "/home/ryan/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/save.py", line 115, in save_model
signatures, options)
File "/home/ryan/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/saved_model/save.py", line 74, in save
save_lib.save(model, filepath, signatures, options)
File "/home/ryan/.local/lib/python3.6/site-packages/tensorflow_core/python/saved_model/save.py", line 883, in save
_ = _SaveableView(checkpoint_graph_view)
File "/home/ryan/.local/lib/python3.6/site-packages/tensorflow_core/python/saved_model/save.py", line 164, in init
self.checkpoint_view.objects_ids_and_slot_variables())
File "/home/ryan/.local/lib/python3.6/site-packages/tensorflow_core/python/training/tracking/graph_view.py", line 418, in objects_ids_and_slot_variables
object_names[obj] = _object_prefix_from_path(path)
File "/home/ryan/.local/lib/python3.6/site-packages/tensorflow_core/python/training/tracking/graph_view.py", line 64, in _object_prefix_from_path
for trackable in path_to_root))
File "/home/ryan/.local/lib/python3.6/site-packages/tensorflow_core/python/training/tracking/graph_view.py", line 64, in
for trackable in path_to_root))
File "/home/ryan/.local/lib/python3.6/site-packages/tensorflow_core/python/training/tracking/graph_view.py", line 57, in _escape_local_name
return (name.replace(_ESCAPE_CHAR, _ESCAPE_CHAR + _ESCAPE_CHAR)
AttributeError: 'NoneType' object has no attribute 'replace'
'''
In line 90 of the policy.py file is where I'm trying to save the subclassed model "self.trpo" after having called train_on_batch and predict_on_batch. I have absolutely no idea what the error is telling me.
Any help would be greatly appreciated!
Thanks,
Ryan

@ryanmaxwell96
Request you to fill issue template.Also, let us know Tensorflow version you are using.
Request you to share colab link or simple standalone code to reproduce the issue in our environment.It helps us in localizing the issue faster.Thanks!

Here is the colab file with the issue I was referring to. https://colab.research.google.com/drive/1dD9QKj9-NzdCECBWKkkihS18Iqmlf5-o?usp=sharing
For the prompts it asks for, input the following in this order:
CartPoleBulletEnv-v1
n
1.0

Ahh bummer. It gave me the error the first time through, but now is giving me another error I haven't seen before.

Ok so I think I may see a problem. I used the model.summary() command to get:
trpo summary
Model: "trpo"

Layer (type)                 Output Shape              Param #
policy_nn (PolicyNN)         multiple                  4393

log_prob (LogProb)           multiple                  0

kl_entropy (KLEntropy)       multiple                  0
Total params: 4,394
Trainable params: 4,393
Non-trainable params: 1

None
I have no idea why it says None at the end, but I'm guessing this is the problem. Any way to remove it?

@ryanmaxwell96
Request you to grant me the access for the colab link you have shared. Thanks!

I think I gave it. Do you have access now?

@ryanmaxwell96
I am not able to reproduce the issue.Please, find the gist here.Please, help with the reproducible code .It helps us in localizing the issue faster.Thanks!

Sorry for the round-about way of doing this, as I don't really know how to use colab very well. But you need to upload these (https://github.com/ryanmaxwell96/trpo_w_subclassing/upload) files to the colab as well as use !pip install pybullet to get the errors that I've been talking about.

Also, don't forget this part:
For the prompts it asks for, input the following in this order:
CartPoleBulletEnv-v1
n
1.0

This should show the errors I've been getting:
https://colab.research.google.com/drive/1dD9QKj9-NzdCECBWKkkihS18Iqmlf5-o#scrollTo=Ft_-1u1cnjW2

@ryanmaxwell96 I am not able to reproduce this issue. Please share us a minimal reproducible code to reproduce this. Thanks!

If you follow the steps that I listed: upload these (https://github.com/ryanmaxwell96/trpo_w_subclassing/upload) files to the colab, use !pip install pybullet, and then type these (CartPoleBulletEnv-v1, n, 1.0) in when prompted, you should get the errors I was referring to.
I have shared the train colab file with you that should show the errors.

@ryanmaxwell96 I tried to reproduce this issue but I am not able to. Please find my gist here

authors: gowthamkpr ravikyram ryanmaxwell96


text:
Trouble running Tensorflow on Xcode 8 beta 5

I get the following message, I don't know if this will be fixed in an upcoming version or if I need to add more configuration (it's running fine on the latest Xcode 7 though):
Undefined symbols for architecture arm64:
  "_deflate", referenced from:
      tensorflow::io::ZlibOutputBuffer::Deflate(int) in libtensorflow-core.a(zlib_outputbuffer.o)
  "_inflate", referenced from:
      tensorflow::io::ZlibInputBuffer::Inflate() in libtensorflow-core.a(zlib_inputbuffer.o)
  "_deflateInit2_", referenced from:
      tensorflow::io::ZlibOutputBuffer::ZlibOutputBuffer(tensorflow::WritableFile*, int, int, tensorflow::io::ZlibCompressionOptions const&) in libtensorflow-core.a(zlib_outputbuffer.o)
  "_inflateEnd", referenced from:
      tensorflow::io::ZlibInputBuffer::~ZlibInputBuffer() in libtensorflow-core.a(zlib_inputbuffer.o)
      tensorflow::io::ZlibInputBuffer::~ZlibInputBuffer() in libtensorflow-core.a(zlib_inputbuffer.o)
  "_deflateEnd", referenced from:
      tensorflow::io::ZlibOutputBuffer::Close() in libtensorflow-core.a(zlib_outputbuffer.o)
  "_inflateInit2_", referenced from:
      tensorflow::io::ZlibInputBuffer::ZlibInputBuffer(tensorflow::RandomAccessFile*, unsigned long, unsigned long, tensorflow::io::ZlibCompressionOptions const&) in libtensorflow-core.a(zlib_inputbuffer.o)
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

That looks like we might need to add 'zlib' to the list of libraries to link against. On Linux, you can just add -lz to the linker command line, but I'm not sure how to do that with Xcode.

I believe I've got this fixed now in the latest code. Can you give this a try and let me know?

Now I get the error
/Users/Kevin/Developer/tensorflow/third_party/eigen3/unsupported/Eigen/CXX11/Tensor:1:10: 'unsupported/Eigen/CXX11/Tensor' file not found
on
#include "unsupported/Eigen/CXX11/Tensor"

@KevinAyuque
Did you fix your first problem
ld:symbol(s) not found for architecture arm64?
I have the same error now.

Is this still broken with the latest version?

Closing due to inactivity. Feel free to re-open if you would like us to look again.

I have the the same error with the latest code, any body have a fix?

@liliang027 getting the same error, could you fix it? :D

authors: KevinAyuque ainurb drpngx liliang027 petewarden shucaoxiaobo


text:
_0_QueueInput/input_queue: Skipping cancelled enqueue attempt with queue not closed

I am trying to start the training mask rcnn of tensorpack:
python train.py --config DATA.BASEDIR=/home/n/car/car MODE_FPN=True "DATA.VAL=('balloon_val',)"  "DATA.TRAIN=('balloon_train',)" TRAIN.BASE_LR=1e-3 TRAIN.EVAL_PERIOD=0 "TRAIN.LR_SCHEDULE=[1000]" "PREPROC.TRAIN_SHORT_EDGE_SIZE=[600,1200]" TRAIN.CHECKPOINT_PERIOD=1 DATA.NUM_WORKERS=1 --load COCO-MaskRCNN-R50FPN2x.npz --logdir log_full

And exactly after epoch 20 Iam getting:
2020-07-08 10:21:42.621628: W tensorflow/core/kernels/queue_base.cc:277] _0_QueueInput/input_queue: Skipping cancelled enqueue attempt with queue not closed

Environment:
sys.platform          linux
Python                3.6.10 |Anaconda, Inc.| (default, May  8 2020, 02:54:21) [GCC 7.3.0]
Tensorpack            v0.10.1-0-g8f831349
Numpy                 1.19.0
TensorFlow            1.13.1/b'v1.13.1-0-g6612da8951'
TF Compiler Version   4.8.5
TF CUDA support       True
TF MKL support        False
Nvidia Driver         /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.440.33.01
CUDA                  /usr/local/cuda-10.0/lib64/libcudart.so.10.0.130
CUDNN                 /usr/local/cuda-10.0/lib64/libcudnn.so.7.5.1
NCCL
CUDA_VISIBLE_DEVICES  Unspecified
GPU 0,1,2,3,4,5,6,7   Tesla K80
Free RAM              475.14/480.07 GB
CPU Count             32
cv2                   4.2.0
msgpack               1.0.0
python-prctl          False

I have tried changing:
 MODE_FPN=True/False

However with "False" Im getting worse results.
Here is the link
What is happening ? And how to fix it ?

@Adblu,
In order to expedite the trouble-shooting process, could you please provide the complete code to reproduce the issue reported here.
Also, please update TensorFlow to v1.15 or v2.2 and check if you are facing the same error. Thanks!

authors: Adblu amahendrakar


text:
CUBLAS_STATUS_NOT_INITIALIZED

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): binary
TensorFlow version: tf-nightly
Python version: 3.8.5
Installed using virtualenv? pip? conda?: conda
Bazel version (if compiling from source): n/a
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: 10.1 / 7.6.5
GPU model and memory: GeFroce 2080 Ti (X2)

Describe the problem
I'm struggling to get off the ground here with multi-gpu tensorflow.  I've eventually gotten stuck with"cublas not initialized."  Thank you for having a look.
Provide the exact sequence of commands / steps that you executed before running into the problem
print("Num GPUs Available: ", len(tf.config.experimental.list_physical_devices('GPU')))
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
Create 2 virtual GPUs with 1GB memory each
try:
tf.config.experimental.set_virtual_device_configuration(
gpus[0],
[tf.config.experimental.VirtualDeviceConfiguration(memory_limit=1024),
tf.config.experimental.VirtualDeviceConfiguration(memory_limit=1024)])
logical_gpus = tf.config.experimental.list_logical_devices('GPU')
print(len(gpus), "Physical GPU,", len(logical_gpus), "Logical GPUs")
except RuntimeError as e:
# Virtual devices must be set before GPUs have been initialized
print(e)
mirrored_strategy = tf.distribute.MirroredStrategy()
(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()
x_train = x_train[:,:,:,np.newaxis] / 255.0
x_test = x_test[:,:,:,np.newaxis] / 255.0
y_train = to_categorical(y_train)
y_test = to_categorical(y_test)
with mirrored_strategy.scope():
model4 = Sequential()
model4.add(Conv2D(filters=64, kernel_size=2, padding='same', activation='relu', input_shape=(28,28, 1)))
model4.add(Conv2D(filters=64, kernel_size=2))
model4.add(MaxPooling2D(pool_size=2))
model4.add(Flatten())
model4.add(Dense(10, activation='softmax'))
model4.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
model4.summary()
model4.fit(x_train, y_train, epochs=2, validation_split=0.1)
_, test_acc = model4.evaluate(x_test, y_test)
print(test_acc)
model4.save_weights('cnn.h5')
Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
(base_gpu) C:\Users\Mo\BTC>python test.py
2020-08-12 23:09:37.482332: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-08-12 23:09:40.326028: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2020-08-12 23:09:40.483740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:67:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2020-08-12 23:09:40.497852: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 1 with properties:
pciBusID: 0000:68:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2020-08-12 23:09:40.515569: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-08-12 23:09:40.530217: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-08-12 23:09:40.545759: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-08-12 23:09:40.552588: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-08-12 23:09:40.566705: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-08-12 23:09:40.574656: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-08-12 23:09:40.597844: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-08-12 23:09:40.602568: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0, 1
Num GPUs Available:  2
2020-08-12 23:09:40.606521: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-08-12 23:09:40.672282: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x227a512f230 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-08-12 23:09:40.700825: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-08-12 23:09:41.131362: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:67:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2020-08-12 23:09:41.146897: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 1 with properties:
pciBusID: 0000:68:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 11.00GiB deviceMemoryBandwidth: 573.69GiB/s
2020-08-12 23:09:41.183979: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2020-08-12 23:09:41.228784: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-08-12 23:09:41.251706: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2020-08-12 23:09:41.259688: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2020-08-12 23:09:41.303717: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2020-08-12 23:09:41.328316: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2020-08-12 23:09:41.353029: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-08-12 23:09:41.379921: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0, 1
2020-08-12 23:09:42.687780: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-12 23:09:42.695670: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0 1
2020-08-12 23:09:42.700946: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N N
2020-08-12 23:09:42.723277: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 1:   N N
2020-08-12 23:09:42.747771: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1024 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:67:00.0, compute capability: 7.5)
2020-08-12 23:09:42.782663: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 1024 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:67:00.0, compute capability: 7.5)
2020-08-12 23:09:42.814118: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 8582 MB memory) -> physical GPU (device: 1, name: GeForce RTX 2080 Ti, pci bus id: 0000:68:00.0, compute capability: 7.5)
2020-08-12 23:09:42.864267: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x22831d21960 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-08-12 23:09:42.889142: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
2020-08-12 23:09:42.914052: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (1): GeForce RTX 2080 Ti, Compute Capability 7.5
2 Physical GPU, 3 Logical GPUs
WARNING:tensorflow:NCCL is not supported when using virtual GPUs, fallingback to reduction to one device
Model: "sequential"

Layer (type)                 Output Shape              Param #
conv2d (Conv2D)              (None, 28, 28, 64)        320

conv2d_1 (Conv2D)            (None, 27, 27, 64)        16448

max_pooling2d (MaxPooling2D) (None, 13, 13, 64)        0

flatten (Flatten)            (None, 10816)             0

dense (Dense)                (None, 10)                108170
Total params: 124,938
Trainable params: 124,938
Non-trainable params: 0

2020-08-12 23:09:44.300510: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:118] None of the MLIR optimization passes are enabled (registered 1)
Epoch 1/2
2020-08-12 23:09:47.355213: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2020-08-12 23:09:47.364005: E tensorflow/stream_executor/cuda/cuda_blas.cc:225] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2020-08-12 23:09:47.387331: E tensorflow/stream_executor/cuda/cuda_blas.cc:225] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2020-08-12 23:09:47.431316: E tensorflow/stream_executor/cuda/cuda_blas.cc:225] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2020-08-12 23:09:47.473173: E tensorflow/stream_executor/cuda/cuda_blas.cc:225] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2020-08-12 23:09:47.491439: E tensorflow/stream_executor/cuda/cuda_blas.cc:225] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2020-08-12 23:09:47.499352: E tensorflow/stream_executor/cuda/cuda_blas.cc:225] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2020-08-12 23:09:47.539511: E tensorflow/stream_executor/cuda/cuda_blas.cc:225] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2020-08-12 23:09:47.569205: E tensorflow/stream_executor/cuda/cuda_blas.cc:225] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2020-08-12 23:09:47.593318: E tensorflow/stream_executor/cuda/cuda_blas.cc:225] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED
2020-08-12 23:09:47.605302: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2020-08-12 23:09:47.643972: E tensorflow/stream_executor/cuda/cuda_dnn.cc:325] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2020-08-12 23:09:47.643972: E tensorflow/stream_executor/cuda/cuda_dnn.cc:325] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2020-08-12 23:09:47.673588: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Error retrieving driver version: Unimplemented: kernel reported driver version not implemented on Windows
2020-08-12 23:09:47.649853: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Error retrieving driver version: Unimplemented: kernel reported driver version not implemented on Windows
2020-08-12 23:09:47.700174: E tensorflow/stream_executor/cuda/cuda_dnn.cc:325] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2020-08-12 23:09:47.723925: E tensorflow/stream_executor/cuda/cuda_dnn.cc:325] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2020-08-12 23:09:47.746358: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Error retrieving driver version: Unimplemented: kernel reported driver version not implemented on Windows
2020-08-12 23:09:47.769343: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Error retrieving driver version: Unimplemented: kernel reported driver version not implemented on Windows
2020-08-12 23:09:47.819798: E tensorflow/stream_executor/cuda/cuda_dnn.cc:325] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2020-08-12 23:09:47.844975: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Error retrieving driver version: Unimplemented: kernel reported driver version not implemented on Windows
2020-08-12 23:09:47.870709: E tensorflow/stream_executor/cuda/cuda_dnn.cc:325] Could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
2020-08-12 23:09:47.880941: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Error retrieving driver version: Unimplemented: kernel reported driver version not implemented on Windows
Traceback (most recent call last):
File "test.py", line 55, in
model4.fit(x_train, y_train, epochs=2, validation_split=0.1)
File "C:\Users\Mo\anaconda3\envs\base_gpu\lib\site-packages\tensorflow\python\keras\engine\training.py", line 103, in _method_wrapper
return method(self, *args, **kwargs)
File "C:\Users\Mo\anaconda3\envs\base_gpu\lib\site-packages\tensorflow\python\keras\engine\training.py", line 1102, in fit
tmp_logs = self.train_function(iterator)
File "C:\Users\Mo\anaconda3\envs\base_gpu\lib\site-packages\tensorflow\python\eager\def_function.py", line 787, in call
result = self._call(*args, **kwds)
File "C:\Users\Mo\anaconda3\envs\base_gpu\lib\site-packages\tensorflow\python\eager\def_function.py", line 847, in _call
return self._stateless_fn(*args, **kwds)
File "C:\Users\Mo\anaconda3\envs\base_gpu\lib\site-packages\tensorflow\python\eager\function.py", line 2922, in call
return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
File "C:\Users\Mo\anaconda3\envs\base_gpu\lib\site-packages\tensorflow\python\eager\function.py", line 1853, in _filtered_call
return self._call_flat(
File "C:\Users\Mo\anaconda3\envs\base_gpu\lib\site-packages\tensorflow\python\eager\function.py", line 1933, in _call_flat
return self._build_call_outputs(self._inference_function.call(
File "C:\Users\Mo\anaconda3\envs\base_gpu\lib\site-packages\tensorflow\python\eager\function.py", line 552, in call
outputs = execute.execute(
File "C:\Users\Mo\anaconda3\envs\base_gpu\lib\site-packages\tensorflow\python\eager\execute.py", line 59, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.UnknownError: 4 root error(s) found.
(0) Unknown:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node replica_2/sequential/conv2d/Conv2D (defined at C:\Users\Mo\anaconda3\envs\base_gpu\lib\threading.py:932) ]]
[[div_no_nan_1/ReadVariableOp_4/_88]]
(1) Unknown:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node replica_2/sequential/conv2d/Conv2D (defined at C:\Users\Mo\anaconda3\envs\base_gpu\lib\threading.py:932) ]]
(2) Unknown:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node replica_2/sequential/conv2d/Conv2D (defined at C:\Users\Mo\anaconda3\envs\base_gpu\lib\threading.py:932) ]]
[[gradient_tape/replica_1/sequential/conv2d/BiasAdd/BiasAddGrad/_144]]
(3) Unknown:  Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node replica_2/sequential/conv2d/Conv2D (defined at C:\Users\Mo\anaconda3\envs\base_gpu\lib\threading.py:932) ]]
[[AddN_2/_132]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_3014]

@chocolate-skyr
Can you see the issues with similar error #35029, #9489 and see if it helps you.Thanks!

Thank you.  What ended up working fixing my issue was:
reinstalling python with conda which downgraded the cuda toolkit / cudnn to appropriate versions

@chocolate-skyr
Glad to know your issue was resolved.Please, close this thread as your issue was resolved. Thanks!

authors: chocolate-skyr ravikyram


text:
Concrete Function output shape sometimes changes after save/load cycle

Output of environment capture script:
== check python ===================================================
python version: 3.7.8
python branch:
python build version: ('default', 'Aug 10 2020 13:15:25')
python compiler version: Clang 10.0.0 (clang-1000.10.44.4)
python implementation: CPython

== check os platform ===============================================
os: Darwin
os kernel version: Darwin Kernel Version 17.7.0: Thu Dec 20 21:47:19 PST 2018; root:xnu-4570.71.22~1/RELEASE_X86_64
os release version: 17.7.0
os platform: Darwin-17.7.0-x86_64-i386-64bit
linux distribution: ('', '', '')
linux os distribution: ('', '', '')
mac version: ('10.13.6', ('', '', ''), 'x86_64')
uname: uname_result(system='Darwin', node='...', release='17.7.0', version='Darwin Kernel Version 17.7.0: Thu Dec 20 21:47:19 PST 2018; root:xnu-4570.71.22~1/RELEASE_X86_64', machine='x86_64', processor='i386')
architecture: ('64bit', '')
machine: x86_64

== are we in docker =============================================
No

== compiler =====================================================
Apple LLVM version 10.0.0 (clang-1000.10.44.4)
Target: x86_64-apple-darwin17.7.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

== check pips ===================================================
numpy                         1.18.5
protobuf                      3.10.0
tensorflow                    2.3.1
tensorflow-estimator          2.3.0

== check for virtualenv =========================================
False

== tensorflow import ============================================
tf.version.VERSION = 2.3.1
tf.version.GIT_VERSION = v2.3.0-54-gfcc4b966f1
tf.version.COMPILER_VERSION = 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.46.4)

== env ==========================================================
LD_LIBRARY_PATH is unset
DYLD_LIBRARY_PATH is unset

== nvidia-smi ===================================================
./tf_env_collect.sh: line 147: nvidia-smi: command not found

== cuda libs  ===================================================

== tensorflow installed from info ==================
Name: tensorflow
Version: 2.3.1
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author-email: packages@tensorflow.org
License: Apache 2.0
Location: .../python3.7/site-packages
Required-by:

== python version  ==============================================
(major, minor, micro, releaselevel, serial)
(3, 7, 8, 'final', 0)

== bazel version  ===============================================

Describe the current behavior
The structured_outputs of a concrete function might not match after saving and restoring a model containing that function via tf.saved_model. In the example below, the function doesn't change the shape of the tensor, but, when requesting a concrete function for a tensor with shape [None, 3], a more specific output [7, 3] is returned instead.
Describe the expected behavior
The output shape should not be more specific than the true output.
Standalone code to reproduce the issue
Reproduction script and stdout of for i in $(seq 10); do echo $i; python repro.py; done in this Gist: https://gist.github.com/gmacon/057cc64bf849c8d65974daec56a037b7
Other info / logs
Initially, I thought this bug was deterministic, but it's clearly actually random. Based on running the reproduction script in a loop, it appears to work correctly about half of the time. Question: does this have something to do with the traversal order of a randomized hash table?
Edit again: I initially thought it was deterministic because it was happening every time in the system I'm building. However, I was reusing the same serialized model over and over in those tests. It appears to work in about half the saves, not half the loads.
~As far as I can tell, this was working correctly in Tensorflow 2.2.~ Edit: I'm not sure about that any more.
Edit again again: I reproduced this in Tensorflow 2.0.0.

I am able to replicate the issue, please find the gist here.

@gmacon Can you try with @tf.function(input_signature=(tf.TensorSpec(shape=[None, 3], dtype=tf.float32),))?

Providing an explicit input_signature seems to work, as far as I can tell. However, that doesn't really help me, because my actual system uses keyword arguments with boolean values as switches, like this more complicated reproducer, and it's my understanding that input_signature can't be used in that situation.

Yes working with kwargs is still in the todo list.
Why do you require to trace with t1(np.ones((7, 3), dtype=np.float32))?

Providing some explicit data is required to tickle the bug. If you remove that, it will never manifest. In my real system the "provide some data" step is really "train the model with initial data", so I don't see an obvious way to avoid providing any example data to the model.
I don't think there's anything special about 7, it's a "random" number in this situation.

@gmacon If you insert
print(t1.__call__.pretty_printed_concrete_signatures())
You will see:
ConcreteFunction __call__(x)
  Args:
    x: float32 Tensor, shape=(None, 3)
  Returns:
    float32 Tensor, shape=(None, 3)
__call__(x)
  Args:
    x: float32 Tensor, shape=(7, 3)
  Returns:
    float32 Tensor, shape=(7, 3)

When you save the model the saved concrete function could be in any order. E.g.
Tensor("StatefulPartitionedCall:0", shape=(7, 3), dtype=float32)
Tensor("StatefulPartitionedCall_1:0", shape=(None, 3), dtype=float32)
Tensor("StatefulPartitionedCall_2:0", shape=(None, 3), dtype=float32)

The logic is that it takes the first saved concrete function with a "compatible" in a loop over all the saved concrete functions:
https://github.com/tensorflow/tensorflow/blob/4e281e9a211729a9a57df04dad22d15c988f2919/tensorflow/python/saved_model/function_deserialization.py#L246-L253
And the compatibility check is:
https://github.com/tensorflow/tensorflow/blob/4e281e9a211729a9a57df04dad22d15c988f2919/tensorflow/python/saved_model/function_deserialization.py#L92-L119
So as the order of the concrete functions change  on save you could enter in any compatible concrete function.

/cc @andresusanopinto

Was able to reproduce your issue in Tf Nightly 2.6.0-dev20210530, please find the gist here. Thanks!

Hi,
Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.
The Tensorflow team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.
Please follow the release notes to stay up to date with the latest developments which are happening in the Tensorflow space.

The project I was observing this problem on has ended, so I don't personally care about this problem any more, but I just checked against 2.18.0-dev20240816, and this issue still exists with the same reproducer.

authors: Saduf2019 bhack gmacon sachinprasadhs tilakrayal


text:
[AArch64] error: inlining failed in call to 'always_inline' 'uint8x16_t vaeseq_u8(uint8x16_t, uint8x16_t)': target specific option mismatch

I did a build today and got hit by same issue as in #41409 one. I use manylinux2014 image for builds.
top commit:
commit 4e8bba06da6c3212c1d4e19ef86c42d905d47c7a (grafted, HEAD -> master, origin/master, origin/HEAD)
Author: TensorFlower Gardener <gardener@tensorflow.org>
Date:   Mon Aug 30 03:37:48 2021 -0700

    Merge pull request #50961 from nouiz:upstream-cudaMallocAsync-no_stats_error

    PiperOrigin-RevId: 393736407
    Change-Id: I31f5ba2b885683cc587a30cdce70e0f20ffefef9

Error:
      /opt/rh/devtoolset-10/root/usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++11' -MD -MF bazel-out/aarch64-o
pt/bin/external/com_google_absl/absl/random/internal/_objs/randen_hwaes_impl/randen_hwaes.pic.d '-frandom-seed=bazel-out/aarch64-opt/bin/external/com_google_absl/absl/random/internal/_objs/randen_hwaes_impl/randen_hwaes.pic.o' -fPIC -iquoteexternal/com_google_absl -iquotebazel-
out/aarch64-opt/bin/external/com_google_absl -w -DAUTOLOAD_DYNAMIC_KERNELS '-std=c++14' -Wall -Wextra -Wcast-qual -Wconversion-null -Wformat-security -Wmissing-declarations -Woverlength-strings -Wpointer-arith -Wundef -Wunused-local-typedefs -Wunused-result -Wvarargs -Wvla -Wwr
ite-strings -DNOMINMAX -Wno-pass-failed -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/com_google_absl/absl/random/internal/randen_hwaes.cc -o bazel-out/aarch64-opt/bin/external
/com_google_absl/absl/random/internal/_objs/randen_hwaes_impl/randen_hwaes.pic.o)
    Execution platform: @local_execution_config_platform//:platform
    In file included from external/com_google_absl/absl/random/internal/randen_hwaes.cc:229:
    /opt/rh/devtoolset-10/root/usr/lib/gcc/aarch64-redhat-linux/10/include/arm_neon.h: In function 'Vector128 {anonymous}::AesRound(const Vector128&, const Vector128&)':
    /opt/rh/devtoolset-10/root/usr/lib/gcc/aarch64-redhat-linux/10/include/arm_neon.h:12332:1: error: inlining failed in call to 'always_inline' 'uint8x16_t vaesmcq_u8(uint8x16_t)': target specific option mismatch
    12332 | vaesmcq_u8 (uint8x16_t data)
          | ^~~~~~~~~~
    external/com_google_absl/absl/random/internal/randen_hwaes.cc:255:20: note: called from here
      255 |   return vaesmcq_u8(vaeseq_u8(state, uint8x16_t{})) ^ round_key;
          |          ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    In file included from external/com_google_absl/absl/random/internal/randen_hwaes.cc:229:
    /opt/rh/devtoolset-10/root/usr/lib/gcc/aarch64-redhat-linux/10/include/arm_neon.h:12318:1: error: inlining failed in call to 'always_inline' 'uint8x16_t vaeseq_u8(uint8x16_t, uint8x16_t)': target specific option mismatch
    12318 | vaeseq_u8 (uint8x16_t data, uint8x16_t key)
          | ^~~~~~~~~
    external/com_google_absl/absl/random/internal/randen_hwaes.cc:255:20: note: called from here
      255 |   return vaesmcq_u8(vaeseq_u8(state, uint8x16_t{})) ^ round_key;
          |          ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    At global scope:
    cc1plus: note: unrecognized command-line option '-Wno-pass-failed' may have been intended to silence earlier diagnostics
    Target //tensorflow/tools/pip_package:build_pip_package failed to build
    INFO: Elapsed time: 5671.673s, Critical Path: 500.29s
    INFO: 9206 processes: 996 internal, 8210 local.

Originally posted by @hrw in https://github.com/tensorflow/tensorflow/issues/41409#issuecomment-908364569

Hi @hrw,
Could you please  fill the template  as it helps us analyse the issue [tf version, steps followed before you ran into this error or stand alone code/colab gist to reproduce the issue faced].Thanks

System information

OS Platform and Distribution: manylinux2014 container (so CentOS 7)
TensorFlow installed from (source or binary): source
TensorFlow version: git HEAD
Python version: 3.9
Installed using virtualenv? pip? conda?: not installed
Bazel version: 3.7.2
GCC/Compiler version: gcc (GCC) 10.2.1 20210130 (Red Hat 10.2.1-11)
CUDA/cuDNN version: none
GPU model and memory: none

Provide the exact sequence of commands / steps that you executed before running into the problem
$ git clone --depth 1 https://git.linaro.org/ci/job/configs.git
$ cp configs/ldcg-python-manylinux-tensorflow/build-manylinux2014-wheels.sh .
$ chmod 755 build-manylinux2014-wheels.sh
$ echo '  - "git"' >> configs/ldcg-python-manylinux-tensorflow/ansible/vars/vars.yml
$ docker run --rm -u root --security-opt seccomp=unconfined -v $PWD:/tmp/workspace quay.io/pypa/manylinux2014_aarch64 /tmp/workspace/build-manylinux2014-wheels.sh

This then installs Ansible and runs set of playbooks to build Tensorflow. Which means:

installation of proper Bazel version (3.7.2 from github release in this case)
installation of required system packages
creation of virtualenv
installation of numpy, h5py and other Python packages
clone of HEAD of tensorflow repo (depth=1)
configure build
run build to get Python wheel

Steps 3-7 are done in a loop over requested Python versions. This time I used only 3.9 as I needed build of git version to build Tensorflow/io package.

https://ci.linaro.org/job/ldcg-python-manylinux-tensorflow-nightly/80/consoleText for example build log

Hi @Saduf2019 ,Could you please look into this issue?

Having the same issue. Building Tensorflow Lite pip package on Ubuntu 20.04 with
sudo CI_DOCKER_EXTRA_PARAMS="-e CI_BUILD_PYTHON=python3.7 -e CROSSTOOL_PYTHON_INCLUDE_PATH=/usr/include/python3.7"   tensorflow/tools/ci_build/ci_build.sh PI-PYTHON37   tensorflow/lite/tools/pip_package/build_pip_package_with_bazel.sh aarch64
Here is the output:
SUBCOMMAND: # @com_google_protobuf//:protobuf_lite [action 'Compiling com_google_protobuf/src/google/protobuf/any_lite.cc', configuration: b6fcde5caf516b18393a5aaba5217f71c7f0fce6771445fc2ee2d8aad33523ea, execution platform: @local_execution_config_platform//:platform]
(cd /root/tensorflow/bazel-ci_build-cache/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/execroot/org_tensorflow && \
  exec env - \
    LD_LIBRARY_PATH='' \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
    PWD=/proc/self/cwd \
    PYTHON_BIN_PATH=/usr/local/bin/python3.7 \
    PYTHON_LIB_PATH=/usr/lib/python3/dist-packages \
    TF2_BEHAVIOR=1 \
  /root/tensorflow/bazel-ci_build-cache/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/aarch64_linux_toolchain/bin/aarch64-linux-gnu-gcc -fstack-protector -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections -isystem /root/tensorflow/bazel-ci_build-cache/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/aarch64_linux_toolchain/lib/gcc/aarch64-linux-gnu/8.3.0/include -isystem /root/tensorflow/bazel-ci_build-cache/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/aarch64_linux_toolchain/lib/gcc/aarch64-linux-gnu/8.3.0/include-fixed -isystem /root/tensorflow/bazel-ci_build-cache/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/aarch64_linux_toolchain/aarch64-linux-gnu/include/c++/8.3.0/ -isystem /root/tensorflow/bazel-ci_build-cache/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/aarch64_linux_toolchain/aarch64-linux-gnu/libc/usr/include/ -isystem /usr/include/python3.7m -isystem /usr/include/ -MD -MF bazel-out/aarch64-opt/bin/external/com_google_protobuf/_objs/protobuf_lite/any_lite.pic.d '-frandom-seed=bazel-out/aarch64-opt/bin/external/com_google_protobuf/_objs/protobuf_lite/any_lite.pic.o' -fPIC -iquote external/com_google_protobuf -iquote bazel-out/aarch64-opt/bin/external/com_google_protobuf -isystem external/com_google_protobuf/src -isystem bazel-out/aarch64-opt/bin/external/com_google_protobuf/src -w -DAUTOLOAD_DYNAMIC_KERNELS -O3 '-std=c++14' -DHAVE_PTHREAD -DHAVE_ZLIB -Woverloaded-virtual -Wno-sign-compare -Wno-unused-function -Wno-write-strings -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -no-canonical-prefixes -fno-canonical-system-headers -c external/com_google_protobuf/src/google/protobuf/any_lite.cc -o bazel-out/aarch64-opt/bin/external/com_google_protobuf/_objs/protobuf_lite/any_lite.pic.o)
ERROR: /root/tensorflow/bazel-ci_build-cache/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/com_google_absl/absl/random/internal/BUILD.bazel:323:11: C++ compilation of rule '@com_google_absl//absl/random/internal:randen_hwaes_impl' failed (Exit 1): aarch64-linux-gnu-gcc failed: error executing command
  (cd /root/tensorflow/bazel-ci_build-cache/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/execroot/org_tensorflow && \
  exec env - \
    LD_LIBRARY_PATH='' \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
    PWD=/proc/self/cwd \
    PYTHON_BIN_PATH=/usr/local/bin/python3.7 \
    PYTHON_LIB_PATH=/usr/lib/python3/dist-packages \
    TF2_BEHAVIOR=1 \
  /root/tensorflow/bazel-ci_build-cache/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/aarch64_linux_toolchain/bin/aarch64-linux-gnu-gcc -fstack-protector -g0 -O2 -DNDEBUG -ffunction-sections -fdata-sections -isystem /root/tensorflow/bazel-ci_build-cache/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/aarch64_linux_toolchain/lib/gcc/aarch64-linux-gnu/8.3.0/include -isystem /root/tensorflow/bazel-ci_build-cache/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/aarch64_linux_toolchain/lib/gcc/aarch64-linux-gnu/8.3.0/include-fixed -isystem /root/tensorflow/bazel-ci_build-cache/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/aarch64_linux_toolchain/aarch64-linux-gnu/include/c++/8.3.0/ -isystem /root/tensorflow/bazel-ci_build-cache/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/aarch64_linux_toolchain/aarch64-linux-gnu/libc/usr/include/ -isystem /usr/include/python3.7m -isystem /usr/include/ -MD -MF bazel-out/aarch64-opt/bin/external/com_google_absl/absl/random/internal/_objs/randen_hwaes_impl/randen_hwaes.pic.d '-frandom-seed=bazel-out/aarch64-opt/bin/external/com_google_absl/absl/random/internal/_objs/randen_hwaes_impl/randen_hwaes.pic.o' -fPIC -iquote external/com_google_absl -iquote bazel-out/aarch64-opt/bin/external/com_google_absl -w -DAUTOLOAD_DYNAMIC_KERNELS -O3 '-std=c++14' -Wall -Wextra -Wcast-qual -Wconversion-null -Wformat-security -Wmissing-declarations -Woverlength-strings -Wpointer-arith -Wundef -Wunused-local-typedefs -Wunused-result -Wvarargs -Wvla -Wwrite-strings -DNOMINMAX -Wno-pass-failed -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -no-canonical-prefixes -fno-canonical-system-headers -c external/com_google_absl/absl/random/internal/randen_hwaes.cc -o bazel-out/aarch64-opt/bin/external/com_google_absl/absl/random/internal/_objs/randen_hwaes_impl/randen_hwaes.pic.o)
Execution platform: @local_execution_config_platform//:platform
In file included from external/com_google_absl/absl/random/internal/randen_hwaes.cc:229:
/root/tensorflow/bazel-ci_build-cache/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/aarch64_linux_toolchain/lib/gcc/aarch64-linux-gnu/8.3.0/include/arm_neon.h: In function 'Vector128 {anonymous}::AesRound(const Vector128&, const Vector128&)':
/root/tensorflow/bazel-ci_build-cache/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/aarch64_linux_toolchain/lib/gcc/aarch64-linux-gnu/8.3.0/include/arm_neon.h:12440:1: error: inlining failed in call to always_inline 'uint8x16_t vaesmcq_u8(uint8x16_t)': target specific option mismatch
 vaesmcq_u8 (uint8x16_t data)
 ^~~~~~~~~~
external/com_google_absl/absl/random/internal/randen_hwaes.cc:255:20: note: called from here
   return vaesmcq_u8(vaeseq_u8(state, uint8x16_t{})) ^ round_key;
          ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from external/com_google_absl/absl/random/internal/randen_hwaes.cc:229:
/root/tensorflow/bazel-ci_build-cache/.cache/bazel/_bazel_root/eab0d61a99b6696edb3d2aff87b585e8/external/aarch64_linux_toolchain/lib/gcc/aarch64-linux-gnu/8.3.0/include/arm_neon.h:12426:1: error: inlining failed in call to always_inline 'uint8x16_t vaeseq_u8(uint8x16_t, uint8x16_t)': target specific option mismatch
 vaeseq_u8 (uint8x16_t data, uint8x16_t key)
 ^~~~~~~~~
external/com_google_absl/absl/random/internal/randen_hwaes.cc:255:20: note: called from here
   return vaesmcq_u8(vaeseq_u8(state, uint8x16_t{})) ^ round_key;
          ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Target //tensorflow/lite/python/interpreter_wrapper:_pywrap_tensorflow_interpreter_wrapper failed to build
INFO: Elapsed time: 547.661s, Critical Path: 65.94s
INFO: 4323 processes: 1428 internal, 2895 local.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

Commits tried:
18794fe8889572b91cee62184ad3e71ef6464d51
2509a6e82e6eb95888de697539845089923c23d5
current master

Related issue https://github.com/abseil/abseil-cpp/issues/662.

@hwrod,
@cfRod

Can you try tensorflow-aarch64.patch.gz patch ?
In basic is a backport from absl upstream that adress exactly this issue.

If works we can PR it into tensorflow.

will check tomorrow

It is kind of chicken and egg.
To install Tensorflow 2.7 you need tensorflow-io-gcs-filesystem.
To build tensorflow-io-gcs-filesystem you need Tensorflow package. Good that 0.20 needs TF 2.6.0+.
I have something built but need to clean my hacks and submit them to tensorflow/io (several already got merged).

So those changes will be in tensorflow/io?... and do we need TensorFlow then to pick the right version of tensorflow-io-gcs-filesystem?

@cfRod no - this issue is about Tensorflow. Once it is fixed we can build Tensorflow 2.7-git.
Installing it would require tensorflow-io-gcs-filesystem which is from tensorflow/io which is other project. And I sent several aarch64 related changes/issues there already.

Build successful with the patch (@cbalint13 )
INFO: Elapsed time: 1090.835s, Critical Path: 255.11s
INFO: 4664 processes: 204 internal, 4460 local.
INFO: Build completed successfully, 4664 total actions

Thanks for the confirmation @cfRod !

It seems to pass olso on my side too.
I try merge into upstream via a new PR#51806 .

@hrw > no - this issue is about Tensorflow.
I was referring to the issue on tensorflow-io-gcs-filesystem actually but thanks for the update. Is it just tensorflow/io#1441 that is remaining?
@cbalint13 thanks for the PR.

Depends on build environment. I do builds in manylinux2014 so https://github.com/tensorflow/io/issues/1510 is a problem too (due to gcc 10).

https://snapshots.linaro.org/ldcg/python-cache/ has tensorflow-io packages if someone needs.
versioned as "0.20.0"  but they are git HEAD ones.

Thanks for the package @hrw

authors: AIWintermuteAI cbalint13 cfRod hrw mohantym


text:
Missing docker images for 2.5.2 and 2.5.3

Click to expand!
Issue Type
Bug
Source
source
Tensorflow Version
2.5.2 and 2.5.3
Custom Code
No
OS Platform and Distribution
No response
Mobile device
No response
Python version
No response
Bazel version
No response
GCC/Compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current Behaviour?
The docker images for 2.5.2 and 2.5.3 are missing here:
https://hub.docker.com/r/tensorflow/tensorflow
How can I obtain these images? I need the GPU images.

Standalone code to reproduce the issue
sudo docker pull tensorflow/tensorflow:2.5.2-gpu
sudo docker pull tensorflow/tensorflow:2.5.3-gpu

Relevant log output
No response

Hi @Lotte1990 ,
Could you please refer to the link here where there is complete list of  images and let us know if it is helpful ?
Thank you!

@SuryanarayanaY The docker images are not listed in the link you provided. 2.5.0 and 2.5.1 are present, but 2.5.2 and 2.5.3 are missing. Please add these docker images. I cannot upgrade beyond these versions because my models behave differently from 2.6 onwards.

@Lotte1990 ,
The issue put up forward. Could you please confirm whether you used the images 2.5.2 or 2.5.3 earlier.Whether they exists earlier ? I couldn't get track of those images.

Hi @Lotte1990 , Usually when there are not much difference between 2.5.1 to 2.5.2 & 2.5.3, these images will not be uploaded in docker hub, you can use 2.5.1 or you can use the 2.6.0 version, if you are looking for specific changes made in 2.5.2 & 2.5.3.
Refer the similar issue here. Thanks!

@sachinprasadhs I cannot upgrade to 2.6.0 and beyond because my models behave differently and I do need the changes made in 2.5.2 and 2.5.3. Please upload these versions.

@angerson , @learning-to-play , Could you please look into this issue. Thanks!

@angerson @learning-to-play @sachinprasadhs Any updates on this?

We too are reliant upon TF 2.5.3.  There is a bug that breaks our stack in 2.5.1 that was fixed by 2.5.3 and our model does not work with 2.6+.  Please post a TF 2.5.3 container and help users avoid a pip install tensorflow==2.5.3 upgrade to a possibly (less optimized) pypi package.

Unfortunately 2.5 is significantly out of our current range of support.  It is unlikely we would create new containers for it.

authors: Lotte1990 MichaelHudgins SuryanarayanaY qhaas sachinprasadhs


text:
Segfault when running tensorflow.python.ops.math_ops.sobol_sample

Click to expand!
Issue Type
Bug
Have you reproduced the bug with TF nightly?
Yes
Source
source
Tensorflow Version
2.10.0
Custom Code
Yes
OS Platform and Distribution
Ubuntu 22.04
Mobile device
No response
Python version
3.9
Bazel version
No response
GCC/Compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current Behaviour?
Segmentation fault on very large input arguments.

Standalone code to reproduce the issue
import tensorflow as tf
import os
import numpy as np
from tensorflow.python.ops import math_ops
try:
  arg_0 = 5
  arg_1 = 125091515651
  dtype = None
  out = math_ops.sobol_sample(arg_0,arg_1,dtype=dtype,)
except Exception as e:
  print("Error:"+str(e))

### Relevant log output

_No response_</details>

@nimashiri Could you please have a look at the outcome I faced while reproducing this issue on tf-nightly. Please find the gist as well.
Error:{{function_node __wrapped__SobolSample_device_/job:localhost/replica:0/task:0/device:CPU:0}} num_results*dim must be less than 2147483647 [Op:SobolSample]

Thank you!

@nimashiri Could you please have a look at the outcome I faced while reproducing this issue on tf-nightly. Please find the gist as well.
Error:{{function_node __wrapped__SobolSample_device_/job:localhost/replica:0/task:0/device:CPU:0}} num_results*dim must be less than 2147483647 [Op:SobolSample]

Thank you!

This issue is for 2.10.

authors: nimashiri sushreebarsa