text:
Pip install can't find tensorflow2.0.0rc0

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: Nope
TensorFlow installed from (source or binary): wheel file via pip
TensorFlow version: tensorflow==2.0.0rc0
Python version: python3.7
Installed using virtualenv? pip? conda?: pip
Bazel version (if compiling from source): not source
GCC/Compiler version (if compiling from source): not source
CUDA/cuDNN version: not GPU
GPU model and memory: not GPU

Describe the problem
Provide the exact sequence of commands / steps that you executed before running into the problem
When you pip install tensorflow==2.0.0rc0, you get an error message saying that pip can't find a matching version of the distribution that't given.
Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
Step 5/11 : RUN /usr/local/bin/algorithmia-build
 ---> Running in 229b8273366c
Collecting algorithmia<2.0,>=1.0.0 (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/12/ae/38a82aae155a42261621eaba0a878959a0af93bf2d70d0ba98048ddaa3d2/algorithmia-1.2.0-py2.py3-none-any.whl
Requirement already satisfied: six in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (1.12.0)
Collecting pip==19.2.2 (from -r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/8d/07/f7d7ced2f97ca3098c16565efbe6b15fafcba53e8d9bdb431e09140514b0/pip-19.2.2-py2.py3-none-any.whl (1.4MB)
Collecting tensorflow-gpu==2.0.0rc0 (from -r requirements.txt (line 4))
  Could not find a version that satisfies the requirement tensorflow-gpu==2.0.0rc0 (from -r requirements.txt (line 4)) (from versions: 1.13.0rc1, 1.13.0rc2, 1.13.1, 1.13.2, 1.14.0rc0, 1.14.0rc1, 1.14.0, 2.0.0a0, 2.0.0b0, 2.0.0b1)
No matching distribution found for tensorflow-gpu==2.0.0rc0 (from -r requirements.txt (line 4))
You are using pip version 18.1, however version 19.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Removing intermediate container 229b8273366c
The command '/bin/sh -c /usr/local/bin/algorithmia-build' returned a non-zero code: 1

Looks like tensorflow2.0.0rc0 moved away from the manylinux1 tag to the manylinux2010 tag.
The manylinux2010 tag is only supported w/ pip version 19 and above.
To fix this bug/issue, update pip via:
pip install -U pip

I'm using pip 19.2.2 and I get the same error.

Run this command, this worked for me.
pip install --upgrade tensorflow-gpu==2.0.0-rc0

@besirkurtulmus,
For Tensorflow CPU version try,
pip install --upgrade tensorflow==2.0.0-rc0
Let us know if that helps. Thanks!

@besirkurtulmus,
For Tensorflow CPU version try,
pip install --upgrade tensorflow==2.0.0-rc0
Let us know if that helps. Thanks!

This now works. Did you guys make a fix?

I am having this exact issue right now
OS - Windows 7 64 bit
PIP version 19.2.3
Python version 3.7.4
pip install --upgrade tensorflow==2.0.0-rc0 Collecting tensorflow==2.0.0-rc0   ERROR: Could not find a version that satisfies the requirement tensorflow==2.0.0-rc0 (from versions: none) ERROR: No matching distribution found for tensorflow==2.0.0-rc0

Can you try pip install --upgrade -v tensorflow==2.0.0rc0 and post the full log?

https://pastebin.com/QKMjueDJ

Just to be sure, can you also try python3 -m pip install --upgrade pip && python3 -m pip install --upgrade -v tensorflow==2.0.0rc0 and post the full log there?
I have a nagging feeling that pip is running from a different interpreter than the one you're installing TF from

Powershell says python3 isnt a command i can run...

`PS C:\Users\Seva> python --version
Python 3.7.4
PS C:\Users\Seva> python3
The term 'python3' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spe
lling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:8

python3 <<<<

CategoryInfo          : ObjectNotFound: (python3:String) [], CommandNotFoundException
FullyQualifiedErrorId : CommandNotFoundException

PS C:\Users\Seva>`
really odd

Having the exact same issue.

I figure it's because I only have py -3 available, so running python is equivalent to python3
Regardless, I ran that command as python3 -m pip install --upgrade pip && python3 -m pip install --upgrade -v tensorflow==2.0.0rc0 and got the same error, other than pip telling me it was already up to date

Tried that too.

These all seem issues with the environment. I'd suggest looking at the detailed logs from python -m pip install --upgrade -vvv tensorflow==2.0.0rc0 (3 vs, the maximum verbosity level).

@charmscale, Did you get a chance to look @mihaimaruseac's comment. Thanks!

I am also getting similar issue...

python -m pip install --upgrade pip && python -m pip install --upgrade setuptools && python -m pip install --upgrade tensorflow
If this still fails, add -vvv just before tensorflow and post the full log as attachment, not as image

Thanks for suggestion, but it didn't worked. Please find below attached logs : -
https://pastebin.com/wmHbFzqn

Try python -c ‘import pip; print(pip.pep425tags.get_supported())

I got all the same issues as everyone else and this is the output I got.
Microsoft Windows [Version 10.0.18362.295]
(c) 2019 Microsoft Corporation. Med enerett.
C:\Users[USER] > pip debug --verbose
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.
pip version: pip 19.2.3 from d:\program files (x86)\python 3.6.8\lib\site-packages\pip (python 3.6)
sys.version: 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 23 2018, 23:31:17) [MSC v.1916 32 bit (Intel)]
sys.executable: d:\program files (x86)\python 3.6.8\python.exe
sys.getdefaultencoding: utf-8
sys.getfilesystemencoding: utf-8
locale.getpreferredencoding: cp1252
sys.platform: win32
sys.implementation:
name: cpython
Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
Compatible tags: 13
cp36-cp36m-win32
cp36-none-win32
py3-none-win32
cp36-none-any
cp3-none-any
py36-none-any
py3-none-any
py35-none-any
py34-none-any
py33-none-any
py32-none-any
py31-none-any
py30-none-any**

@Indseta it's because you're using a 32 bits CPU and we're releasing for 64 bits CPUs

But by system has 64 bit OS

python -m pip install --upgrade pip && python -m pip install --upgrade setuptools && python -m pip install --upgrade tensorflow
If this still fails, add -vvv just before tensorflow and post the full log as attachment, not as image

https://pastebin.com/kE0WMwzA

pip debug --verbose

https://pastebin.com/dUTzee3Z
I am trying to install tensorflow in a virtualenvironment through anaconda prompt. I am a beginner so please suggest accordingly.

I think in both of these cases you installed a 32 bits Python. Please install a 64 bits one if your CPU is 64 bits.
For 32 bits support see #32315

Thanks for your suggestion.
Will same solution also work for libraries OpenCV and face_recognition ?
Because both libraries are having similar installation issues on my system or any other solution will work for them to get fixed. Thanks.

Recommend StackOverflow for this, no idea if same solution would work there and cannot extrapolate.

Thanks for your suggestion.
Will same solution also work for libraries OpenCV and face_recognition ?
Because both libraries are having similar installation issues on my system or any other solution will work for them to get fixed. Thanks.

If you are having issues with face_recognition because of mtcnn module i suggest using my fix, it works for tensorflow 2.0.
https://github.com/reliefs/mtcnn-tensorflow2
https://github.com/ipazc/mtcnn/issues/47

Looks like original issue reported here is resolved. Please feel free to reopen if issue still persists. Thanks!

I reinstalled python and was able to run the two commands for upgrading pip and then installing tensorflow.

I think in both of these cases you installed a 32 bits Python. Please install a 64 bits one if your CPU is 64 bits.
For 32 bits support see #32315

This ans is right please check python version

I was also facing the same issue, essentially the reason was tensorflow wheel file is not available for 32 bit system. They also not planning to publish 32 bit version this year, as it takes lots of effort at their end and it was not in their planning.
So, I solved it by changing my installation of python 32 bit to python 64 bit.

As mentioned: if you really have a 32bits CPU (unlikely as these are very old), see #32315
If your CPU is 64 bits, please install a 64 bits Python.

https://github.com/tensorflow/tensorflow/issues/32627

On my windows machine, I was facing same issue while running pip install tensorflow==2.0.0-rc1 though I was on latest version of pip and on 64-bit distribution for python.
Oddly enough, I face this issue when I am running the pip install inside a virtual environment. If i deactivate from my virtual env. I do not get this issue.
Will continue to investigate if I could find anything more.
Thanks

@rsjain1978 Can you check if you have both 32 bits and 64 bits binaries in your system? It's very likely that somehow the virtualenv python is 32 bits.
Alternatively, you can compare output of pip debug --verbose outside and inside the virtualenv. That should point out these differences very quickly.

C:\Users\User\Desktop\number_recognition\venv\Scripts\python.exe "C:/Users/User/Desktop/number_recognition/number _recog.py"
Using TensorFlow backend.
Traceback (most recent call last):
File "C:\Users\User\Desktop\number_recognition\venv\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\Users\User\Desktop\number_recognition\venv\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\Users\User\Desktop\number_recognition\venv\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "C:\Users\User\AppData\Local\Programs\Python\Python38\lib\imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "C:\Users\User\AppData\Local\Programs\Python\Python38\lib\imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: DLL load failed while importing _pywrap_tensorflow_internal: The specified module could not be found.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/User/Desktop/number_recognition/number recog.py", line 6, in
from keras.preprocessing.image import ImageDataGenerator
File "C:\Users\User\AppData\Local\Programs\Python\Python38\lib\site-packages\keras_init.py", line 3, in
from . import utils
File "C:\Users\User\AppData\Local\Programs\Python\Python38\lib\site-packages\keras\utils_init_.py", line 6, in
from . import conv_utils
File "C:\Users\User\AppData\Local\Programs\Python\Python38\lib\site-packages\keras\utils\conv_utils.py", line 9, in
from .. import backend as K
File "C:\Users\User\AppData\Local\Programs\Python\Python38\lib\site-packages\keras\backend_init_.py", line 1, in
from .load_backend import epsilon
File "C:\Users\User\AppData\Local\Programs\Python\Python38\lib\site-packages\keras\backend\load_backend.py", line 90, in
from .tensorflow_backend import *
File "C:\Users\User\AppData\Local\Programs\Python\Python38\lib\site-packages\keras\backend\tensorflow_backend.py", line 5, in
import tensorflow as tf
File "C:\Users\User\Desktop\number_recognition\venv\lib\site-packages\tensorflow_init_.py", line 41, in
from tensorflow.python.tools import module_util as module_util
File "C:\Users\User\Desktop\number_recognition\venv\lib\site-packages\tensorflow\python_init.py", line 50, in
from tensorflow.python import pywrap_tensorflow
File "C:\Users\User\Desktop\number_recognition\venv\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 69, in
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "C:\Users\User\Desktop\number_recognition\venv\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\Users\User\Desktop\number_recognition\venv\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\Users\User\Desktop\number_recognition\venv\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "C:\Users\User\AppData\Local\Programs\Python\Python38\lib\imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "C:\Users\User\AppData\Local\Programs\Python\Python38\lib\imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: DLL load failed while importing _pywrap_tensorflow_internal: The specified module could not be found.
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors
for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.
Process finished with exit code 1

authors: Crazychicken563 Indseta akanyaani anujmca aquib1011 besirkurtulmus charmscale gadagashwini lordvissu mihaimaruseac moraywheel nonlin rahulsrivastava1519 reliefs rsjain1978 vibwipro


text:
Build error due to missing dependency declarations in double-conversion

System information

OS Platform and Distribution : CentOS-8.0
TensorFlow installed from (source or binary): source
TensorFlow version: 2.0
Python version: 3.7.4
Installed using virtualenv? pip? conda?: from source using spack
Bazel version (if compiling from source): 0.26.1
GCC/Compiler version (if compiling from source): gcc-7.4.0
CUDA/cuDNN version: CUDA@10.1.243/cuDNN@7.6.5.32-10.1
GPU model and memory: 2x NVIDIA Quadro P4000

Describe the problem
I used spack to build all dependencies for tensorflow including bazel. As per the build logs from spack, the following procedure was attempted to install tensorflow from source :
'./configure'

'bazel' '--nohome_rc' '--nosystem_rc' '--output_user_root=/tmp/spack/tf' 'build' '--color=no' '--jobs=16' '--config=opt' '--config=cuda' '--config=noaws' '--config=nohdfs' '--config=noignite' '--config=nokafka' '--config=v2' '--cxxopt=-D_GLIBCXX_USE_CXX11_ABI=0' '//tensorflow/tools/pip_package:build_pip_package'

Full error log from spack : error_log
The error comes from '@double_conversion//:double-conversion':  :
[sajid@xrmlite ~]$ cat error_log | grep -a3b3 ERROR
80532-INFO: Deleting stale sandbox base /tmp/spack/tf/ba2957b26f360b54b039cbd58767fa4e/sandbox
80621-[6 / 38] [Prepa] BazelWorkspaceStatusAction stable-status.txt
80683-[48 / 1,263] [Prepa] Executing genrule @local_config_nccl//:nccl-files [for host] ... (6 actions, 5 running)
80792:ERROR: /tmp/spack/tf/ba2957b26f360b54b039cbd58767fa4e/external/double_conversion/BUILD.bazel:12:1: undeclared inclusion(s) in rule '@double_conversion//:double-conversion':
80965-this rule is missing dependency declarations for the following files included by 'external/double_conversion/double-conversion/double-conversion.cc':
81115-  '/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.4.0/gettext-0.20.1-yjkttko6qwne6igwp2cabop2d4p2g3ff/include/libintl.h'
81256-[75 / 1,297] Executing genrule @local_config_cuda//cuda:cuda-lib [for host]; 2s local ... (14 actions, 2 running)
[sajid@xrmlite ~]$

Additional Info :
The full dependency tree for building py-tensoflow is show below. As can be seen, gettext is a dependency for python built from source. This python is both a dependency of bazel and the python we want to eventually link py-tensorflow against.
[sajid@xrmlite ~]$ spack spec -I  py-tensorflow cuda_arch=61  ^/ppi2muw  ^cuda@10.1.243 ^python@3.7.4 %gcc@7.4.0

.....

Concretized
--------------------------------
 -   py-tensorflow@2.0.0%gcc@7.4.0~android~aws~computecpp+cuda cuda_arch=61 ~dynamic_kernels+gcp~gdr~hdfs~ignite~ios~jemalloc~kafka~mkl~monolithic~mpi+nccl~ngraph~numa~opencl patches=c49766a976e5c25c3827036828df0a2630e511bd783ac9cdcc6fc13068b22fac ~rocm~tensorrt~verbs~xla arch=linux-centos8-broadwell
[+]      ^bazel@0.26.1%gcc@7.4.0 patches=3e6448a0dde42bbd72568d29c5646d370dd62ca300cdd10a630908c086844167,75fad08d2a118372ed86696832942c7903bb716af28e5af969d8e20857655cf4,aa926467d3fc2bcd338ccc6355bc9f56adfd18dd3b4e1813a4ce8daee9e34600 arch=linux-centos8-broadwell
[+]          ^jdk@1.8.0_131-b11%gcc@7.4.0 arch=linux-centos8-broadwell
[+]          ^python@3.7.4%gcc@7.4.0+bz2+ctypes+dbm+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4~uuid+zlib arch=linux-centos8-broadwell
[+]              ^bzip2@1.0.8%gcc@7.4.0+shared arch=linux-centos8-broadwell
[+]              ^expat@2.2.9%gcc@7.4.0+libbsd arch=linux-centos8-broadwell
[+]                  ^libbsd@0.10.0%gcc@7.4.0 arch=linux-centos8-broadwell
[+]              ^gdbm@1.18.1%gcc@7.4.0 arch=linux-centos8-broadwell
[+]                  ^readline@8.0%gcc@7.4.0 arch=linux-centos8-broadwell
[+]                      ^ncurses@6.1%gcc@7.4.0~symlinks~termlib arch=linux-centos8-broadwell
[+]              ^gettext@0.20.1%gcc@7.4.0+bzip2+curses+git~libunistring+libxml2+tar+xz arch=linux-centos8-broadwell
[+]                  ^libxml2@2.9.9%gcc@7.4.0~python arch=linux-centos8-broadwell
[+]                      ^libiconv@1.16%gcc@7.4.0 arch=linux-centos8-broadwell
[+]                      ^xz@5.2.4%gcc@7.4.0 arch=linux-centos8-broadwell
[+]                      ^zlib@1.2.11%gcc@7.4.0+optimize+pic+shared arch=linux-centos8-broadwell
[+]                  ^tar@1.32%gcc@7.4.0 arch=linux-centos8-broadwell
[+]              ^libffi@3.2.1%gcc@7.4.0 arch=linux-centos8-broadwell
[+]              ^openssl@1.1.1d%gcc@7.4.0+systemcerts arch=linux-centos8-broadwell
[+]              ^sqlite@3.30.1%gcc@7.4.0~column_metadata+fts~functions~rtree arch=linux-centos8-broadwell
[+]      ^cuda@10.1.243%gcc@7.4.0 arch=linux-centos8-broadwell
[+]      ^cudnn@7.6.5.32-10.1-linux-x64%gcc@7.4.0 arch=linux-centos8-broadwell
[+]      ^curl@7.63.0%gcc@7.4.0~darwinssl~gssapi~libssh~libssh2~nghttp2 arch=linux-centos8-broadwell
[+]      ^nccl@2.4.8-1%gcc@7.4.0 patches=42778c78eb9875dacddf5eca20f7f6a077773fcbee41e51174f81b3143684b6d arch=linux-centos8-broadwell
[+]      ^py-absl-py@0.7.0%gcc@7.4.0 arch=linux-centos8-broadwell
[+]          ^py-setuptools@41.4.0%gcc@7.4.0 arch=linux-centos8-broadwell
[+]          ^py-six@1.12.0%gcc@7.4.0 arch=linux-centos8-broadwell
[+]      ^py-astor@0.8.0%gcc@7.4.0 patches=edc5eeddabe153b08e938f52edaeb2d880ee3128082967f310db0f98510fe6e0 arch=linux-centos8-broadwell
[+]      ^py-gast@0.2.2%gcc@7.4.0 arch=linux-centos8-broadwell
[+]      ^py-google-pasta@0.1.8%gcc@7.4.0 arch=linux-centos8-broadwell
[+]      ^py-grpcio@1.25.0%gcc@7.4.0 arch=linux-centos8-broadwell
[+]          ^c-ares@1.15.0%gcc@7.4.0 build_type=RelWithDebInfo arch=linux-centos8-broadwell
[+]              ^cmake@3.15.5%gcc@7.4.0~doc+ncurses+openssl+ownlibs patches=3387faf4a71efe81c0fa17410b270ca7d352081ac88d2322df3da9bb6a6a3f2d ~qt arch=linux-centos8-broadwell
[+]          ^py-cython@0.29.13%gcc@7.4.0 arch=linux-centos8-broadwell
[+]      ^py-keras-applications@1.0.8%gcc@7.4.0 arch=linux-centos8-broadwell
[+]      ^py-keras-preprocessing@1.1.0%gcc@7.4.0 arch=linux-centos8-broadwell
[+]      ^py-numpy@1.17.4%gcc@7.4.0+blas+lapack arch=linux-centos8-broadwell
[+]          ^openblas@0.3.7%gcc@7.4.0+avx2~avx512 cpu_target=auto ~ilp64+pic+shared threads=none ~virtual_machine arch=linux-centos8-broadwell
[+]      ^py-opt-einsum@3.1.0%gcc@7.4.0 arch=linux-centos8-broadwell
[+]      ^py-protobuf@3.11.0%gcc@7.4.0~cpp arch=linux-centos8-broadwell
[+]      ^py-termcolor@1.1.0%gcc@7.4.0 arch=linux-centos8-broadwell
[+]      ^py-wheel@0.33.1%gcc@7.4.0 arch=linux-centos8-broadwell
[+]      ^py-wrapt@1.11.2%gcc@7.4.0 arch=linux-centos8-broadwell
[+]      ^swig@4.0.0%gcc@7.4.0 arch=linux-centos8-broadwell
[+]          ^pcre@8.42%gcc@7.4.0~jit+multibyte+utf arch=linux-centos8-broadwell
[+]          ^pkgconf@1.6.3%gcc@7.4.0 arch=linux-centos8-broadwell

The environment for building py-tensorflow is described in spack-build-env.txt.
Contents of .bazelrc
The spack build recipe for py-tensorlfow.
Downstream tracker : https://github.com/spack/spack/issues/14105

I attempted this again after adding a few debug flags in the build recipe as per https://github.com/spack/spack/pull/14177/files. This time, I see a similar error where nasm complains about a PyObject in the header eval.h from the python standard library. I'm mystified as to how building nasm, which doesn't even depend on python according to the BUILD.bazel could raise this error.
I'm sharing the following debug files :

command.log
java.log
spack-build.out
BUILD from both local_config_python and local_config_cc

available at this link : https://northwestern.box.com/s/yxhh3uzj72nj3nqn81xcvkozwzqslrhs
I asked for an explanation of Bazel's behavior via --explain=explainlogfile.txt' but I didn't see any such file in the build directory. Where should I look for this file ?
Is there a way to not let Bazel build any of the dependencies for tensorflow and use local installations instead ?

I recommend reaching out to spack developers, or the spack package maintainer.
We do not maintain any spack related documentation.
For using system installed dependencies, @perfinion may have some pointers.

Hi @gunan, I'm a Spack developer and I personally maintain the TensorFlow and PyTorch packages in Spack.
I don't believe the problem above is specific to Spack, and should be possible to reproduce outside of Spack. If we are doing something wrong in the Spack package, let me know and I can fix it.

this rule is missing dependency declarations for the following files included by

The above error (which references a system header) usually means the bazel c++ toolchain was misconfigured. Bazel did not detect the path, in this case /home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.4.0/gettext-0.20.1-yjkttko6qwne6igwp2cabop2d4p2g3ff/ a part of the C++ toolchain.
The way we make bazel use a different toolchain than system default gcc is to create a toolchain definition here: https://github.com/tensorflow/tensorflow/tree/master/third_party/toolchains/preconfig/ubuntu16.04/gcc7_manylinux2010
You may need to do that, and direct the bazel executions under spack to use that, or there may be an easier way to redirect bazel using some environment variables (which you have to check with bazel team).

I added the cc_toolchain_config.bzl in the folder as well, available here. Looking closely at the toolchain config, I do see that it includes a bunch of libraries :

    tool_paths = [
        tool_path ( name= "ar", path= "/usr/bin/ar" ),
        tool_path ( name= "ld", path= "/home/sajid/packages/spack/lib/spack/env/ld" ),
        tool_path ( name= "cpp", path= "/home/sajid/packages/spack/lib/spack/env/cpp" ),
        tool_path ( name= "gcc", path= "/home/sajid/packages/spack/lib/spack/env/gcc/gcc" ),
        tool_path ( name= "dwp", path= "/usr/bin/dwp" ),
        tool_path ( name= "gcov", path= "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.2.1/gcc-9.2.0-yaowtafvg5f7v3f2en47gpta5b6ucvlq/bin/gcov" ),
        tool_path ( name= "nm", path= "/usr/bin/nm" ),
        tool_path ( name= "objcopy", path= "/usr/bin/objcopy" ),
        tool_path ( name= "objdump", path= "/usr/bin/objdump" ),
        tool_path ( name= "strip", path= "/usr/bin/strip" ),
    ]

    cxx_builtin_include_directories = [
"/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.2.1/gcc-7.3.0-uhzsamzf47moqf6gu2exxqsyg5gw2pbx/lib/gcc/x86_64-pc-linux-gnu/7.3.0/include",
    "/usr/local/include",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.2.1/gcc-7.3.0-uhzsamzf47moqf6gu2exxqsyg5gw2pbx/include",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.2.1/gcc-7.3.0-uhzsamzf47moqf6gu2exxqsyg5gw2pbx/lib/gcc/x86_64-pc-linux-gnu/7.3.0/include-fixed",
    "/usr/include",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/openssl-1.1.1d-4bo5gg2eookuk2ubczsa4ciaz5c4mbwz/include",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/bzip2-1.0.8-5zisvxvepuimx27rqxi53srnkqj6jqlk/include",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cudnn-7.6.5.32-10.1-linux-x64-sabespboxtmnfvkohzmmackhv7tjvbqb/include",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cuda-10.1.243-wciaq2rrrkc4w6vcraocheah55wrekha/include",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/gdbm-1.18.1-ek2j6ymdsfdhwtcsgm5typyeqr2d427s/include",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/readline-8.0-sdz3prqngfpcyiooqriil7iyy3mp77nr/include",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libxml2-2.9.9-s5wjt2encrutai46rptb2qdbuqm74rcj/include/libxml2",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libffi-3.2.1-mhdz37flhbznp5bla7dxtvtuwqxlzdpr/lib/libffi-3.2.1/include",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/gettext-0.20.1-3x3th3xdaosnjd7ttly2mym5hukdqln5/include",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/include/python3.7m",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/ncurses-6.1-bnt5ehudftgrg5smastj4ejhjvyxfjmj/include",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/xz-5.2.4-oc7pfgyjo477pshkef56rkv4ovz7ykei/include",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/nccl-2.4.8-1-phuasbwfolmvjlo5mmizcuz4vqeuayhk/include",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/expat-2.2.9-v4k636wrhszysgefq2sijry72mj5l2au/include",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libbsd-0.10.0-jhowpdxutwgnbbpqv5dl3ifkfmrmuck5/include",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/curl-7.63.0-jrvggtmsfzn64chuvcenb7a4g7lznfvj/include",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/sqlite-3.30.1-kifllx2du5fe2yqreol2ldfk7zzqrhm2/include",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libiconv-1.16-qeeepkzh7y5dnj75fidj2abx23x6nwbl/include",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/zlib-1.2.11-ah6qvpjr2sam2rpn527ivwmz4vyfcqbf/include",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.2.1/gcc-7.3.0-uhzsamzf47moqf6gu2exxqsyg5gw2pbx/include/c++/7.3.0",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.2.1/gcc-7.3.0-uhzsamzf47moqf6gu2exxqsyg5gw2pbx/include/c++/7.3.0/x86_64-pc-linux-gnu",
    "/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.2.1/gcc-7.3.0-uhzsamzf47moqf6gu2exxqsyg5gw2pbx/include/c++/7.3.0/backward"
    ]

If I understand correctly, these include paths shouldn't be in the compiler toolchain definition, right ?

What order are those include paths checked? Maybe /usr/include or /usr/local/include are overriding the other directories we add.
For the record, we use the following patch to bazel (and corresponding environment variable) to inject these include paths into the build:
--- a/tools/cpp/unix_cc_configure.bzl
+++ b/tools/cpp/unix_cc_configure.bzl
@@ -145,11 +145,18 @@ def get_escaped_cxx_inc_directories(repository_ctx, cc, lang_flag, additional_fl
     else:
         inc_dirs = result.stderr[index1 + 1:index2].strip()

-    return [
+    default_inc_directories = [
         _prepare_include_path(repository_ctx, _cxx_inc_convert(p))
         for p in inc_dirs.split("\n")
     ]

+    env = repository_ctx.os.environ
+    if "SPACK_INCLUDE_DIRS" in env:
+        for path in env["SPACK_INCLUDE_DIRS"].split(":"):
+            default_inc_directories.append(path)
+
+    return default_inc_directories
+
 def _is_compiler_option_supported(repository_ctx, cc, option):
     """Checks that `option` is supported by the C compiler. Doesn't %-escape the option."""
     result = repository_ctx.execute([

Looking at this more closely, I see that the exact command that is failing is
 (cd /tmp/spack/tf/3d4affddb55a47900a919f49b073413c/execroot/org_tensorflow && \
  exec env - \
    PATH=/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/lib/spack/env/gcc:/home/sajid/packages/spack/lib/spack/env/case-insensitive:/home/sajid/packages/spack/lib/spack/env:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cuda-10.1.243-wciaq2rrrkc4w6vcraocheah55wrekha/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/py-wheel-0.33.1-op2hwc4hqsgmc4qhevj3tq3tvw7civj2/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/py-numpy-1.17.4-uxmerwuy33filyrx4sw5rkfawty2vlup/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/curl-7.63.0-jrvggtmsfzn64chuvcenb7a4g7lznfvj/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/swig-4.0.0-mvastxbkyr3gnidrgierbrulpj6eys5u/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/py-numpy-1.17.4-uxmerwuy33filyrx4sw5rkfawty2vlup/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/jdk-1.8.0_231-b11-ynnz6lmhhw5mngtkw74hwcnn5ungicak/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/bazel-0.26.1-fgoxd6mcj3iyn7z43j7fq7dyjb5okpm7/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/py-setuptools-41.4.0-ctklysjoeronag3sfh44ptqfrcs6cuov/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/py-setuptools-41.4.0-ctklysjoeronag3sfh44ptqfrcs6cuov/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cuda-10.1.243-wciaq2rrrkc4w6vcraocheah55wrekha/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/py-wheel-0.33.1-op2hwc4hqsgmc4qhevj3tq3tvw7civj2/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/py-numpy-1.17.4-uxmerwuy33filyrx4sw5rkfawty2vlup/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/curl-7.63.0-jrvggtmsfzn64chuvcenb7a4g7lznfvj/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/swig-4.0.0-mvastxbkyr3gnidrgierbrulpj6eys5u/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/bazel-0.26.1-fgoxd6mcj3iyn7z43j7fq7dyjb5okpm7/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/py-setuptools-41.4.0-ctklysjoeronag3sfh44ptqfrcs6cuov/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.2.1/gcc-9.2.0-yaowtafvg5f7v3f2en47gpta5b6ucvlq/bin:/home/sajid/packages/spack/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin \
    PWD=/proc/self/cwd \
    SPACK_CC=/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.2.1/gcc-7.3.0-uhzsamzf47moqf6gu2exxqsyg5gw2pbx/bin/gcc \
    SPACK_CC_RPATH_ARG=-Wl,-rpath, \
    SPACK_COMPILER_IMPLICIT_RPATHS=/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.2.1/gcc-7.3.0-uhzsamzf47moqf6gu2exxqsyg5gw2pbx/lib/gcc/x86_64-pc-linux-gnu/7.3.0:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.2.1/gcc-7.3.0-uhzsamzf47moqf6gu2exxqsyg5gw2pbx/lib64 \
    SPACK_COMPILER_SPEC=gcc@7.3.0 \
    SPACK_CXX=/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.2.1/gcc-7.3.0-uhzsamzf47moqf6gu2exxqsyg5gw2pbx/bin/g++ \
    SPACK_CXX_RPATH_ARG=-Wl,-rpath, \
    SPACK_DEBUG_LOG_DIR=/home/sajid \
    SPACK_DEBUG_LOG_ID=py-tensorflow-a73jdej \
    SPACK_DTAGS_TO_ADD=--disable-new-dtags \
    SPACK_DTAGS_TO_STRIP=--enable-new-dtags \
    SPACK_ENV_PATH=/home/sajid/packages/spack/lib/spack/env:/home/sajid/packages/spack/lib/spack/env/case-insensitive:/home/sajid/packages/spack/lib/spack/env/gcc \
    SPACK_F77=/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.2.1/gcc-7.3.0-uhzsamzf47moqf6gu2exxqsyg5gw2pbx/bin/gfortran \
    SPACK_F77_RPATH_ARG=-Wl,-rpath, \
    SPACK_FC=/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.2.1/gcc-7.3.0-uhzsamzf47moqf6gu2exxqsyg5gw2pbx/bin/gfortran \
    SPACK_FC_RPATH_ARG=-Wl,-rpath, \
    SPACK_INCLUDE_DIRS=/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/openssl-1.1.1d-4bo5gg2eookuk2ubczsa4ciaz5c4mbwz/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/bzip2-1.0.8-5zisvxvepuimx27rqxi53srnkqj6jqlk/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cudnn-7.6.5.32-10.1-linux-x64-sabespboxtmnfvkohzmmackhv7tjvbqb/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cuda-10.1.243-wciaq2rrrkc4w6vcraocheah55wrekha/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/gdbm-1.18.1-ek2j6ymdsfdhwtcsgm5typyeqr2d427s/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/readline-8.0-sdz3prqngfpcyiooqriil7iyy3mp77nr/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libxml2-2.9.9-s5wjt2encrutai46rptb2qdbuqm74rcj/include/libxml2:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libffi-3.2.1-mhdz37flhbznp5bla7dxtvtuwqxlzdpr/lib/libffi-3.2.1/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/gettext-0.20.1-3x3th3xdaosnjd7ttly2mym5hukdqln5/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/include/python3.7m:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/ncurses-6.1-bnt5ehudftgrg5smastj4ejhjvyxfjmj/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/xz-5.2.4-oc7pfgyjo477pshkef56rkv4ovz7ykei/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/nccl-2.4.8-1-phuasbwfolmvjlo5mmizcuz4vqeuayhk/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/expat-2.2.9-v4k636wrhszysgefq2sijry72mj5l2au/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libbsd-0.10.0-jhowpdxutwgnbbpqv5dl3ifkfmrmuck5/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/curl-7.63.0-jrvggtmsfzn64chuvcenb7a4g7lznfvj/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/sqlite-3.30.1-kifllx2du5fe2yqreol2ldfk7zzqrhm2/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libiconv-1.16-qeeepkzh7y5dnj75fidj2abx23x6nwbl/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/zlib-1.2.11-ah6qvpjr2sam2rpn527ivwmz4vyfcqbf/include \
    SPACK_LINKER_ARG=-Wl, \
    SPACK_LINK_DIRS=/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/openssl-1.1.1d-4bo5gg2eookuk2ubczsa4ciaz5c4mbwz/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/bzip2-1.0.8-5zisvxvepuimx27rqxi53srnkqj6jqlk/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cudnn-7.6.5.32-10.1-linux-x64-sabespboxtmnfvkohzmmackhv7tjvbqb/lib64:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cuda-10.1.243-wciaq2rrrkc4w6vcraocheah55wrekha/targets/x86_64-linux/lib/stubs:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cuda-10.1.243-wciaq2rrrkc4w6vcraocheah55wrekha/lib64:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/gdbm-1.18.1-ek2j6ymdsfdhwtcsgm5typyeqr2d427s/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/readline-8.0-sdz3prqngfpcyiooqriil7iyy3mp77nr/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libxml2-2.9.9-s5wjt2encrutai46rptb2qdbuqm74rcj/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libffi-3.2.1-mhdz37flhbznp5bla7dxtvtuwqxlzdpr/lib64:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libffi-3.2.1-mhdz37flhbznp5bla7dxtvtuwqxlzdpr/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/gettext-0.20.1-3x3th3xdaosnjd7ttly2mym5hukdqln5/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/ncurses-6.1-bnt5ehudftgrg5smastj4ejhjvyxfjmj/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/xz-5.2.4-oc7pfgyjo477pshkef56rkv4ovz7ykei/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/nccl-2.4.8-1-phuasbwfolmvjlo5mmizcuz4vqeuayhk/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/expat-2.2.9-v4k636wrhszysgefq2sijry72mj5l2au/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libbsd-0.10.0-jhowpdxutwgnbbpqv5dl3ifkfmrmuck5/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/curl-7.63.0-jrvggtmsfzn64chuvcenb7a4g7lznfvj/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/sqlite-3.30.1-kifllx2du5fe2yqreol2ldfk7zzqrhm2/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libiconv-1.16-qeeepkzh7y5dnj75fidj2abx23x6nwbl/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/zlib-1.2.11-ah6qvpjr2sam2rpn527ivwmz4vyfcqbf/lib \
    SPACK_ROOT=/home/sajid/packages/spack \
    SPACK_RPATH_DIRS=/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/py-tensorflow-2.0.0-a73jdejmdsiybrq3ocyrzxhq4zbv2cjz/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/py-tensorflow-2.0.0-a73jdejmdsiybrq3ocyrzxhq4zbv2cjz/lib64:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/openssl-1.1.1d-4bo5gg2eookuk2ubczsa4ciaz5c4mbwz/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/bzip2-1.0.8-5zisvxvepuimx27rqxi53srnkqj6jqlk/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cudnn-7.6.5.32-10.1-linux-x64-sabespboxtmnfvkohzmmackhv7tjvbqb/lib64:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cuda-10.1.243-wciaq2rrrkc4w6vcraocheah55wrekha/targets/x86_64-linux/lib/stubs:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cuda-10.1.243-wciaq2rrrkc4w6vcraocheah55wrekha/lib64:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/gdbm-1.18.1-ek2j6ymdsfdhwtcsgm5typyeqr2d427s/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/readline-8.0-sdz3prqngfpcyiooqriil7iyy3mp77nr/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libxml2-2.9.9-s5wjt2encrutai46rptb2qdbuqm74rcj/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libffi-3.2.1-mhdz37flhbznp5bla7dxtvtuwqxlzdpr/lib64:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libffi-3.2.1-mhdz37flhbznp5bla7dxtvtuwqxlzdpr/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/gettext-0.20.1-3x3th3xdaosnjd7ttly2mym5hukdqln5/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/ncurses-6.1-bnt5ehudftgrg5smastj4ejhjvyxfjmj/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/xz-5.2.4-oc7pfgyjo477pshkef56rkv4ovz7ykei/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/nccl-2.4.8-1-phuasbwfolmvjlo5mmizcuz4vqeuayhk/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/expat-2.2.9-v4k636wrhszysgefq2sijry72mj5l2au/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libbsd-0.10.0-jhowpdxutwgnbbpqv5dl3ifkfmrmuck5/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/curl-7.63.0-jrvggtmsfzn64chuvcenb7a4g7lznfvj/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/sqlite-3.30.1-kifllx2du5fe2yqreol2ldfk7zzqrhm2/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libiconv-1.16-qeeepkzh7y5dnj75fidj2abx23x6nwbl/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/zlib-1.2.11-ah6qvpjr2sam2rpn527ivwmz4vyfcqbf/lib \
    SPACK_SHORT_SPEC='py-tensorflow@2.0.0%gcc@7.3.0~android~aws~computecpp+cuda cuda_arch=61 ~dynamic_kernels+gcp~gdr~hdfs~ignite~ios~jemalloc~kafka~mkl~monolithic~mpi+nccl~ngraph~numa~opencl patches=c49766a976e5c25c3827036828df0a2630e511bd783ac9cdcc6fc13068b22fac ~rocm~tensorrt~verbs~xla arch=linux-centos8-broadwell/a73jdej' \
    SPACK_SYSTEM_DIRS=/bin:/usr/bin:/usr/local/bin:/bin64:/usr/bin64:/usr/local/bin64:/include:/usr/include:/usr/local/include:/lib:/usr/lib:/usr/local/lib:/lib64:/usr/lib64:/usr/local/lib64:/:/usr:/usr/local \
    SPACK_TARGET_ARGS='-march=broadwell -mtune=broadwell' \
  external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/host/bin/external/nasm/_objs/nasm/outelf.d '-frandom-seed=bazel-out/host/bin/external/nasm/_objs/nasm/outelf.o' -DHAVE_SNPRINTF -DHAVE_SYS_TYPES_H -iquote external/nasm -iquote bazel-out/host/bin/external/nasm -iquote external/bazel_tools -iquote bazel-out/host/bin/external/bazel_tools -isystem external/nasm/asm -isystem bazel-out/host/bin/external/nasm/asm -isystem external/nasm/include -isystem bazel-out/host/bin/external/nasm/include -isystem external/nasm/output -isystem bazel-out/host/bin/external/nasm/output -isystem external/nasm/x86 -isystem bazel-out/host/bin/external/nasm/x86 -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fPIE -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -fno-omit-frame-pointer -no-canonical-prefixes -fno-canonical-system-headers -DNDEBUG -g0 -O2 -ffunction-sections -fdata-sections -g0 '-march=native' -w '-std=c99' -c external/nasm/output/outelf.c -o bazel-out/host/bin/external/nasm/_objs/nasm/outelf.o)

This is failing because bazel wants to build nasm/output/outelf.c and this file has a #include eval.h. Now, as per the nasm build recipe for bazel which is at tensorflow/third_party/nasm/BUILD.bazel shows that the includes for nasm are :
    includes = [
        "asm",
        "include",
        "output",
        "x86",
    ],

and the folder asm does indeed have the relevant header eval.h. Despite being given the right definitions in the BUILD.bazel recipe, for some reason bazel completely ignores this header, searches either $PATH or cxx_builtin_include_directories to try and include a completely irrelevant header (that of eval.h from python standard library instead !).
I'm not sure how we can debug this, since it appears to me that bazel isn't following the bazel recipe.
PS: Even the official bazel docs say that one should explicitly list the include locations which is the case here.

@s-sajid-ali,
Can you try installing the latest stable version of tensorflow i.e 2.6.0 and lets us know if the issue still persists. You can follow this guide to build from source. Thanks!

This issue was fixed in spack: https://github.com/spack/spack/pull/16077.

source: gharchive/issue

authors: adamjstewart gunan s-sajid-ali sanatmpa1

repo: tensorflow/tensorflow

license_type: permissive

license_source: github-api


text:
TensorFlow Lite C++ image classification demo result error

I follow the guidelines of TensorFlow Lite C++ image classification demo. Firstly I clone TensorFlow and use to build label_image for Android armv8 by bazel 1.2.1.

bazel build -c opt --cxxopt=-std=c++11 --config=android_arm64
//tensorflow/lite/examples/label_image:label_image

When I run the sample in the guide. I got the error result like below:

Loaded model ./mobilenet_v1_1.0_224.tflite
resolved reporter
INFO: Initialized TensorFlow Lite runtime.
invoked
average time: 52.036 ms
-nan: 1000 1000:toilet tissue, toilet paper, bathroom tissue
-nan: 999 999:ear, spike, capitulum
-nan: 998 998:bolete
-nan: 997 997:hen-of-the-woods, hen of the woods, Polyporus frondosus, Grifola frondosa
-nan: 996 996:earthstar

According to the guide, the right result is below:

Loaded model /tmp/mobilenet_v1_1.0_224.tflite
resolved reporter
invoked
average time: 68.12 ms
0.860174: 653 653:military uniform
0.0481017: 907 907:Windsor tie
0.00786704: 466 466:bulletproof vest
0.00644932: 514 514:cornet, horn, trumpet, trump
0.00608029: 543 543:drumstick

BTW, The TensorFlow version is the latest commit of the master.
`
commit 27e6c7b49f4558dfc4bd59a9c492bf4f390a77da
Author: A. Unique TensorFlower gardener@tensorflow.org
Date:   Mon Dec 16 01:03:06 2019 -0800
compat: Update forward compatibility horizon to 2019-12-16

PiperOrigin-RevId: 285720630
Change-Id: Ib744d5f7de70a6c6d73dd1a386712e404d0c2b99

`

I got the right result by checkout the tag v2.0.0

Loaded model ./mobilenet_v1_1.0_224.tflite
resolved reporter
INFO: Initialized TensorFlow Lite runtime.
invoked
average time: 76.076 ms
0.860174: 653 653:military uniform
0.048102: 907 907:Windsor tie
0.00786704: 466 466:bulletproof vest
0.00644931: 514 514:cornet, horn, trumpet, trump
0.0060803: 543 543:drumstick

Closing this issue since its resolved. Thanks!

authors: jianlongf ymodak


text:
Annoying Documentation for Tensorflow 2.2.0

I saw the implementations of tf.reduce_mean, tf.reduce_sum but its not there in the documentation for tf 2.2.0. It is really annoying as it created a giant confusion in our code. Please keep documentation and implmentation in sync.

Hi @y12uc231
tf.reduce_mean and tf.reduce_sum are both included in the documentation. Can you please link to the implementations you are referring to, and provide more information on the discrepancy you are seeing between documentation and implementation? Thank you.

Hi @nikitamaia  : If you check the links you posted, they are tf.math.reduce_mean not tf.reduce_mean.

If you click on the "View aliases" button at the top of the docs, you will see that tf.reduce_sum is an alias for tf.math.reduce_sum, so you can use either.

Thanks!
Quick feedback : Based on my personal experience, documentation of tensorflow is incredibly complicated compared any other frameworks I use like pytorch or mxnet.

Thanks for the feedback, @y12uc231. If you run into any confusion, or find something in the docs you think would benefit from some clarity, please do file an issue.


text:
Strange Out Of Memory error when trying example Colab

When I try to run the officially provided Colab
not changing any row, everything goes fine until the training model = question_answer.create(train_data, model_spec=spec)
As soon as training process started, after a while it begins to consume memory with big chunks (e.g. memory consumption increases by 2GB per 2-3 seconds) and shortly it crashes the Colab environment as it throws Out of Memory error. I think this is some sort of a bug, because otherwise this will not be official example, or it would be documented somewhere in the example that the free Colab environment with 16GB ram is not enough to run this example.
I also tried to run the example with much smaller dataset, but the same problem in the same way is being reproduced. So I am almost sure that this is a bug in the library.

@ando0689
We see that the issue template has not been filled, could you please do so as it helps us analyse the issue [tf version, steps followed before you ran into this error or stand alone code to reproduce the issue faced]
Van you share your colab gist where the error is present.

Hi. I am sorry for that, but actually I don't know what to fill there.
I opened this issue under the "Other" tag, as I guess it's not relevant what OS I use etc, because I just run the Official Tensorflow Lite Model Maker example Colab  without changing anything, just running each cell one by one, and on the cell called "Customize the TensorFlow Model" I get out of memory error.
Is it relevant who runs the example or on what machine etc. if I just run it in Google Colab?

I didn't provide enough information in this issue, so I close this and provided the full information here: https://github.com/tensorflow/tensorflow/issues/45541


text:
Mismatch between TFLearn documentation and tutorial

Hi all,
The changes to the metric creation in Estimator in tflearn don't match the current documentation on monitors. For example:
https://www.tensorflow.org/versions/r0.11/tutorials/monitors/index.html
suggests:
validation_metrics = {"accuracy": tf.contrib.metrics.streaming_accuracy,
                      "precision": tf.contrib.metrics.streaming_precision,
                      "recall": tf.contrib.metrics.streaming_recall}
validation_monitor = tf.contrib.learn.monitors.ValidationMonitor(
    test_set.data,
    test_set.target,
    every_n_steps=50,
    metrics=validation_metrics)

wheras
https://github.com/tensorflow/tensorflow/blob/r0.11/tensorflow/contrib/learn/python/learn/estimators/rnn.py
contains:
# Single head metrics.
      if isinstance(predictions, dict):
        raise ValueError(
            'Metrics passed provide only name, no prediction, '
            'but predictions are dict. '
            'Metrics: %s, Targets: %s.' % (metrics, targets))

What related GitHub issues or StackOverflow threads have you found by searching the web for your problem?
None
Environment info
Operating System:
Max OSX El Capitan
Installed version of CUDA and cuDNN:
N/A - cpu version

The output from python -c "import tensorflow; print(tensorflow.__version__)".

0.11.0rc0
If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)
https://gist.github.com/alexshires/d9674c58e352ad81be43f5f7da74c7e1
What other attempted solutions have you tried?
Looking up MetricSpec configuration - no examples or help available
Logs or other output that would be helpful
(If logs are large, please upload as attachment or provide link).
https://gist.github.com/alexshires/5310a751e9b767fbd3595570d618c342

Thanks for the detailed report @alexshires !
@martinwicke @ispirmustafa : IIRC, you guys were looking to switch to MetricSpec from a dict (a274c662b3f090193ead4138791896ffb65d680e). Would you be the right persons to fix the tutorial?

@sandersk, @ispirmustafa, looks like the monitors tutorial references an old form of metrics. Should we fix it, or are you making a new one for hooks anyway?

@martinwicke, since we need to update regardless, I think it would be ideal to replace the Monitors tutorial with a new one on Hooks. I will work with @ispirmustafa on that.

That's perfect, thanks!
On Thu, Oct 27, 2016 at 2:35 PM, Sanders Kleinfeld <notifications@github.com

wrote:

@martinwicke https://github.com/martinwicke, since we need to update
regardless, I think it would be ideal to replace the Monitors tutorial with
a new one on Hooks. I will work with @ispirmustafa
https://github.com/ispirmustafa on that.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/5005#issuecomment-256776051,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAjO_WRgrPyv4F2-2Otn2M05lWeP-hRmks5q4RkTgaJpZM4KYhc_
.

I made a quick and dirty hack to the example code on my own fork for those who (like me) got stuck because of this: https://github.com/dbfr3qs/tensorflow/commit/e1006cf8307662e1984dba6560e67e246a7ba46c

We're still working on new tutorial content, but this particular issue is out of date.

authors: alexshires asimshankar dbfr3qs martinwicke sandersk


text:
Gradient computation through tf.linalg.eigh does not work with complex-valued input in XLA

Click to expand!
Issue Type
Bug
Source
source
Tensorflow Version
2.9.1
Custom Code
Yes
OS Platform and Distribution
No response
Mobile device
No response
Python version
No response
Bazel version
No response
GCC/Compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current Behaviour?
Computing the gradient through tf.linalg.eigh() fails in XLA when the input of the function is complex-valued (tf.complex64).

Standalone code to reproduce the issue
import tensorflow as tf

h_real = tf.Variable([[1., 1.], [1., 1.]], dtype=tf.float32, trainable=True)
h_imag = tf.Variable([[0.0, 1.], [-1., 0.0]], dtype=tf.float32, trainable=True)

@tf.function(jit_compile=True)
def test():
    with tf.GradientTape() as tape:
        # Build Hernitian matrix
        h = tf.complex(h_real, h_imag)
        # Compute singular values
        s,_ = tf.linalg.eigh(h)
        loss = tf.math.real(s[0])
    grad = tape.gradient(loss,[h_real,h_imag])
    return grad

g = test()

Relevant log output
InvalidArgumentError: Binary op add with different element types: f32[2,2] and c64[].
	 [[{{node gradient_tape/add}}]] [Op:__inference_test_245]

@sachinprasadhs,
I was able to reproduce the issue on tensorflow v2.8, v2.9 and nightly. Kindly find the gist of it here.

Hi,
Thank you for opening this issue. Since this issue has been open for a long time, the code/debug information for this issue may not be relevant with the current state of the code base.
The Tensorflow team is constantly improving the framework by fixing bugs and adding new features. We suggest you try the latest TensorFlow version with the latest compatible hardware configuration which could potentially resolve the issue. If you are still facing the issue, please create a new GitHub issue with your latest findings, with all the debugging information which could help us investigate.
Please follow the release notes to stay up to date with the latest developments which are happening in the Tensorflow space.

authors: Venkat6871 faycalaa tilakrayal


text:
tf.data.Dataset.map() of tf.data.Dataset feat. tf.RaggedTensor causes broken for-loop

Click to expand!
Issue Type
Bug
Have you reproduced the bug with TF nightly?
No
Source
binary
Tensorflow Version
2.9.2
Custom Code
Yes
OS Platform and Distribution
Windows 10, WSL2 Ubuntu 20.04
Mobile device
No response
Python version
3.8.10
Bazel version
No response
GCC/Compiler version
No response
CUDA/cuDNN version
11.2
GPU model and memory
No response
Current Behaviour?
When using a tf.data.Dataset for tf.RaggedTensors, the .map() functionality seems to break the ability to iterate over the datapoints via for-loop, causing an internal TypeError on the for-loop itself.
The expected behaviour would be for the program not to throw a TypeError as the same function works if used via a simple for-loop over the data instead of the .map() functionality.

Standalone code to reproduce the issue
import tensorflow as tf
ragged_data=tf.ragged.constant([[[[1,2],[3,4],[5,6]], [[1,2],[3,4],[5,6]], [[1,2],[3,4],[5,6]]], [[[1,2],[3,4],[5,6]], [[1,2],[3,4],[5,6]], [[1,2],[3,4],[5,6]]]])
ds=tf.data.Dataset.from_tensor_slices(ragged_data)
def test_fn(ragged_data):
    for x in ragged_data:
        continue
    return ragged_data
ds.map(test_fn)

Relevant log output
TypeError: Input 'y' of 'Less' Op has type int64 that does not match type int32 of argument 'x'.

@sushreebarsa
I was able to reproduce the issue on Colab using TF v2.11 . Please find the gist here for reference.
Thank you!

Hi, @Tens0rFlaw
Apologize for the delay and I was able to replicate the issue on Google Colab with Tensorflow==2.11 and it's working as expected, for your reference I have added gist-file  and it seems like you're using python map() instead of tf.map_fn() and for your reference I have added official documentation for tf.map_fn() which may help you. If issue still persists please let us know or Could you please confirm if this issue is resolved for you ? Please feel free to close the issue if it is resolved ? Thank you!

Hi @gaikwadrahul8
that indeed resolves the issue.
Thank you for you answer!


text:
Potential memory leak with SymbolicTensor

Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
Yes
Source
source
TensorFlow version
2.15
Custom code
Yes
OS platform and distribution
Manjaro Linux
Mobile device
No response
Python version
3.11.5
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
12.3.52
GPU model and memory
No response
Current behavior?
Hello,
I am working on a TinyML NAS framework, i.e. throughout the execution of my code, hundreds if not thousands of models are created and trained. I have come across a problem that has been starving my system of memory after a couple of days of execution. By using tracemalloc I have been able to see that the main contributor appears to be the symbolic tensor created when creating a Conv2D layer. Maybe I am missing something basic in terms of garbage collection in my code but over time the demo code will eventually consume all system memory.
I have also tried tf.keras.backend.clear_session() and gc.collect() but neither help.
Any help would be appreciated.
Cheers
Standalone code to reproduce the issue
import gc
import tracemalloc, sys, linecache, os
import numpy as np
import tensorflow as tf
from tensorflow import keras

EPOCHS = 5
BS = 512
TEST_LOOPS = 10000

def start_tracemalloc():
    tracemalloc.start()

def display_top(snapshot, key_type="lineno", limit=5):
    snapshot = snapshot.filter_traces(
        (
            tracemalloc.Filter(False, "<frozen importlib._bootstrap>"),
            tracemalloc.Filter(False, "<unknown>"),
        )
    )
    top_stats = snapshot.statistics(key_type)

    print("Top %s lines" % limit)
    for index, stat in enumerate(top_stats[:limit], 1):
        frame = stat.traceback[0]
        # replace "/path/to/module/file.py" with "module/file.py"
        filename = os.sep.join(frame.filename.split(os.sep)[-2:])
        print(
            "#%s: %s:%s: %.1f KiB" % (index, filename, frame.lineno, stat.size / 1024)
        )
        line = linecache.getline(frame.filename, frame.lineno).strip()
        if line:
            print("    %s" % line)

    other = top_stats[limit:]
    if other:
        size = sum(stat.size for stat in other)
        print("%s other: %.1f KiB" % (len(other), size / 1024))
    total = sum(stat.size for stat in top_stats)
    print("Total allocated size: %.1f KiB" % (total / 1024))

def display_snapshot():
    snapshot = tracemalloc.take_snapshot()
    display_top(snapshot)

def create_model() -> keras.models.Model:
    inputs = keras.layers.Input(shape=(28, 28, 1))
    x = keras.layers.Conv2D(32, kernel_size=(3, 3), padding="valid")(inputs)
    x = keras.layers.MaxPooling2D(pool_size=(2, 2), strides=None)(x)
    x = keras.layers.Flatten()(x)
    x = keras.layers.Dense(128, activation=tf.nn.relu)(x)
    x = keras.layers.Dropout(0.2)(x)
    outputs = keras.layers.Dense(10, activation=tf.nn.softmax)(x)

    model = keras.models.Model(inputs=inputs, outputs=outputs)
    model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

    return model

def train_model(model, train_images, train_labels):
    model.fit(train_images, train_labels, epochs=EPOCHS, batch_size=BS)

def main() -> int:
    mnist = keras.datasets.mnist
    (train_images, train_labels), (test_images, test_labels) = mnist.load_data()

    train_images = train_images.reshape(train_images.shape[0], train_images.shape[1], train_images.shape[2], 1)
    train_images = train_images.astype(np.float32) / 255.0

    start_tracemalloc()

    for i in range(TEST_LOOPS):
        tf.keras.backend.clear_session()
        gc.collect()

        model = create_model()
        train_model(model, train_images, train_labels)
        display_snapshot()
        del model

if __name__ == '__main__':
    sys.exit(main())

Relevant log output
Epoch 1/5
...
Top 5 lines
#1: <frozen abc>:123: 1856.2 KiB
#2: python3.11/linecache.py:137: 606.4 KiB
    lines = fp.readlines()
#3: framework/ops.py:245: 197.9 KiB
    return pywrap_tf_session.PyTensor.__new__(
#4: <frozen importlib._bootstrap_external>:729: 158.6 KiB
#5: framework/ops.py:1211: 137.3 KiB
    self._gradient_function = None
952 other: 1407.6 KiB
Total allocated size: 4364.1 KiB
Epoch 1/5
...
Top 5 lines
#1: <frozen abc>:123: 1846.9 KiB
#2: python3.11/linecache.py:137: 620.8 KiB
    lines = fp.readlines()
#3: framework/ops.py:245: 382.3 KiB
    return pywrap_tf_session.PyTensor.__new__(
#4: framework/ops.py:1211: 264.9 KiB
    self._gradient_function = None
#5: framework/ops.py:1161: 254.7 KiB
    self = Operation(c_op, SymbolicTensor)
947 other: 2160.5 KiB
Total allocated size: 5530.0 KiB
Epoch 1/5
...
Top 5 lines
#1: <frozen abc>:123: 1843.9 KiB
#2: python3.11/linecache.py:137: 620.8 KiB
    lines = fp.readlines()
#3: framework/ops.py:245: 568.5 KiB
    return pywrap_tf_session.PyTensor.__new__(
#4: framework/ops.py:1211: 394.1 KiB
    self._gradient_function = None
#5: framework/ops.py:1161: 378.9 KiB
    self = Operation(c_op, SymbolicTensor)
949 other: 2702.3 KiB
Total allocated size: 6508.5 KiB

@SuryanarayanaY I was able to replicate the issue reported here. Thank you!

Being discussed in Keras. #19058

@SuryanarayanaY I posted a similar thing on Keras as I was not sure if it was a TF problem or a Keras problem

Any news @SuryanarayanaY? I just have a research paper waiting on this bugfix >.<

authors: SuryanarayanaY alxhoff sushreebarsa


text:
Feature request: LU Decomposition

I suggest adding LU decomposition with partial pivoting to Tensorflow. The implementation must support rectangular matrices as well, and not only square matrices. Tensorflow already uses LU decomposition for solving linear systems of equations.
LU decomposition can be used as an efficient algorithm for finding the range of a matrix and its low rank approximation. The advantages over QR and SVD are that LU is more computationally efficient and is very suited for GPUs giving a significant speed boost to many computational tasks.
See the following papers:

Li, H., Linderman, G. C., Szlam, A., Stanton, K. P., Kluger, Y., & Tygert, M. (2017). Algorithm 971: An Implementation of a Randomized Algorithm for Principal Component Analysis. ACM Transactions on Mathematical Software (TOMS), 43(3), 28
Shabat, G., Shmueli, Y., Aizenbud, Y., & Averbuch, A. (2016). Randomized LU decomposition. Applied and Computational Harmonic Analysis.
Li, H., Kluger, Y., & Tygert, M. (2016). Randomized algorithms for distributed computation of principal component analysis and singular value decomposition. arXiv preprint arXiv:1612.08709.

@rmlarsen for comment

@rmlarsen already implemented tf.matrix_solve with lu decomposition:
https://github.com/tensorflow/tensorflow/blob/b10f50ff15944badb7262a207f6628dfa52d6a9d/tensorflow/core/kernels/matrix_solve_op.cc
It would be nice to expose the decomposition, like tf.cholesky_solve, so that it can be reused on the next solve if A doesn't change.

Yeah, but from what I remember from Eigen's documentation (perhaps I am wrong), their LU decomposition works only for square matrices. An LU decomposition for rectangular matrices is also very important similar to Scipy and MKL.

Both would be useful - my current use case is for square matrices. This could also be taken from cuSOLVER like cholesky.
I don't have time to work on this, but it would involve copying the cholesky op and primarily changing potrf to getrf. Writing a gradient would be the hard bit.

Le 27 mai 2017 à 20:35, gilgils notifications@github.com a écrit :
Yeah, but from what I remember from Eigen's documentation (perhaps I am wrong), their LU decomposition works only for square matrices. An LU decomposition for rectangular matrices is also very important similar to Scipy and MKL.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

It has been 14 days with no activity and the awaiting tensorflower label was assigned. Please update the label and/or status accordingly.

Nagging Awaiting TensorFlower: It has been 14 days with no activity and the awaiting tensorflower label was assigned. Please update the label and/or status accordingly.

I am still working on the pull request #16185 for parts of this request with @rmlarsen

Nagging Assignee: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@tensorflowbutler @rmlarsen sorry for the late, I am still working on the requirements from the review and merging the code conflicts. Please give me more time. Thanks.

Nagging Assignee @rmlarsen: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@tensorflowbutler fixed the conflicts and test bench errors after pulling and merging the latest tf master branch. working on other enhancements.

Nagging Assignee @rmlarsen: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 77 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 92 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Hi @rmlarsen
I posted a related question in the (closed) pull request https://github.com/tensorflow/tensorflow/pull/16185
I am currently actively working on that.
Regards,

Nagging Assignee @rmlarsen: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 29 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 44 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 59 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 74 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Hi, I think the majority part of the LU ops is done in the PR #21885 lu_op.cc. Probably, need to add more tests in lu_op_test.py. Thanks

Nagging Assignee @rmlarsen: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Hi @harshini-gadige, the pull request #21885 is still under review. thanks.

Nagging Assignee @rmlarsen: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 29 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 44 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 59 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 74 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@gilgils As mentioned by @rmlarsen I think this was resolved by LU op developed. I am closing this issue but open a new issue if you think Imade a mistake. Thanks!

authors: gilgils jvishnuvardhan mccajm tensorflowbutler yaroslavvb zhuangh


text:
can't install tenserflow on ubuntu 16.04 via pip!!

Exception:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 342, in run
prefix=options.prefix_path,
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_set.py", line 784, in install
**kwargs
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_install.py", line 851, in install
self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
File "/usr/local/lib/python2.7/dist-packages/pip/req/req_install.py", line 1064, in move_wheel_files
isolated=self.isolated,
File "/usr/local/lib/python2.7/dist-packages/pip/wheel.py", line 345, in move_wheel_files
clobber(source, lib_dir, True)
File "/usr/local/lib/python2.7/dist-packages/pip/wheel.py", line 316, in clobber
ensure_dir(destdir)
File "/usr/local/lib/python2.7/dist-packages/pip/utils/init.py", line 83, in ensure_dir
os.makedirs(path)
File "/usr/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/tensorflow_gpu-0.12.1.dist-info'

Closing as a duplicate of #7063. Please don't send duplicate issues. If you want to edit the description, use the pencil icon on the upper right corner.

authors: amanckc drpngx


text:
QR decomposition is slow

We are doing a bunch of of QR decompositions in numpy. I did preliminary investigation of moving them to TF, but TF version is slow compared to numpy.
Below is a benchmark script that runs QR decomposition of 4096x4096 matrix. It took 7.3 seconds in TF and 1.93 in numpy MKL. Numpy MKL is the default numpy that comes when installing Anaconda.
version: HEAD from last week, built with --config=cuda --config=opt
cpu: 32 core Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
https://github.com/yaroslavvb/stuff/blob/master/tiny_runs/qr_test.py
Note that pip install --upgrade $TF_BINARY_URL will overwrite MKL numpy with OpenBLAS numpy that is actually slower than TF version. The way to check is to look at np.__config__.show() and look for strings like mkl_intel_lp64. You can get MKL version back by uninstalling numpy and doing conda install numpy
@rmlarsen

@rmlarsen is there some planned work, or should I mark this as contributions welcome?

@yaroslavvb The current OP is using the single-threaded (CPU) Householder QR in Eigen. Does the MKL version run multi-threaded?

Yes, MKL version seems to be using 800-900 CPU

I'll working on this!

I find MKL is already supported in the latest tf code...

@yaroslavvb , I both installed mkl_dnn 2018 and full mkl 2018, add
`#ifdef INTEL_MKL
#define EIGEN_USE_MKL_ALL
#endif // INTEL_MKLtotensorflow/core/kernels/qr_op_impl.h, run bazel build --config=opt --config=cuda --define=using_mkl=true //tensorflow/tools/pip_package:build_pip_packageto build, gotTF QR on 4096 by 4096 matrix in 2.59 seconds
numpy QR on 4096 by 4096 matrix in 2.64 seconds`.

Oh interesting! PS, I thought latest MKL is 2017 update 2. At least that's the version I get when I download latest official Intel for Python distribution from Intel.
You can get MKL version using this snippet
import ctypes
import numpy as np

def mklVersion():
    ver = np.zeros(199, dtype=np.uint8)
    mkl = ctypes.cdll.LoadLibrary("libmkl_rt.so")
    mkl.MKL_Get_Version_String(ver.ctypes.data_as(ctypes.c_char_p), 198)
    return ver[ver != 0].tostring()

# mklVersion()

Can you also use SVD from MKL in TF? The MKL version seems to be pretty efficient and the TF version had some correctness issues -- https://github.com/tensorflow/tensorflow/issues/8905
https://github.com/yaroslavvb/stuff/blob/master/svd_benchmark.py

@yaroslavvb , I use 2018 beta version of MKL only for building c++ code. I find tf use BDCSVD which doesn't have LAPACKE version.

@yaroslavvb What's the status here?

@girving I haven't tested it, but @suiyuan2009 says this can be fixed with 3 line change to qr_op_impl.h

@suiyuan2009 Interested in submitting a PR? :)

@girving It seems this issue can be closed since #11116, which fixed but didn't close this issue automatically, is merged.

Please remove the assignee, as this issue is inviting external contributions. Otherwise, remove the contributions welcome label. Thank you.

Closing per qmick@'s last comment.

QR is still very slow using intel optimized tf2.0 from anaconda. QR calls don't use mkl routine (shown by set export MKL_VERBOSE=1), for float64 random uniform matrix with size 4096 * 4096, the time in mkl linked numpy and tf2.0 cpu is 2.35s vs 28.5s on 2*Xeon 5120 CPU. tf QR  (Eigen implementation) is more than 10 times slower than mkl.
Actually the PR enabling MKL for QR op is reverted in https://github.com/tensorflow/tensorflow/pull/12219. It seems that only some basic ops can be patched by intel mkl via mkl-dnn. And mkl support for eigen (-DEIGEN_USE_MKL_ALL) is not supported at all for current tensorflow setup.
Therefore, for now, so called "intel optimized" or "mkl enabled" tensorflow is very limited in terms of  mkl operations, only matrix multiplication or convolution types of things are actually optimized by mkl-dnn. QR, SVD and other involved ops are still using Eigen implementation which is slow since it is default to single thread.

authors: annarev girving qmick refraction-ray rmlarsen suiyuan2009 tensorflowbutler yaroslavvb


text:
Unexpected behaviour training custom Siamese Network

Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
No
Source
source
TensorFlow version
tf 2.15.1
Custom code
Yes
OS platform and distribution
Linux Mint 21.2 Cinnamon
Mobile device
No response
Python version
3.10
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current behavior?
I am trying to train a custom built Siamese Network, following the Keras documentation closely, only modifying the architecture and other things as needed.
What I'm trying to do differently is load the data in a different manner. Due to the dimensions of the dataset I will be working with, I can't store it all at once in arrays in memory as is done in the example.
I tried loading it iteratively to the model to be trained in two ways:

Using a tf.data.Dataset that will load the data as needed
Building a custom training loop that would only fetch the data batch by batch

To test these, I put together an example script (supplied below) to test these approaches and how they perform in comparison to supplying the whole dataset to the model from an array in memory
Surprisingly, both of my methods produce a weird behaviour, where the trained model simply outputs the same class for every instance (classifying the image pairs as being distinct), while with the first method the model trains well and converges to a good solution. I don't understand if I'm doing something wrong, I have checked my produced tf.data.Dataset and they have the exact same data as the in-memory arrays (as it's supposed to) and I am doing things correctly according to the documentation as far as I am aware.  Can you reproduce the issue, and if yes, understand if I'm doing something wrong or if this behaviour is related to something internal of Tensorflow/Keras that I am not aware of?
Standalone code to reproduce the issue
import time
import sys
import numpy as np
import itertools
from tensorflow.keras.layers import *
from tensorflow.keras.models import Sequential, Model
from random import shuffle, random

tf.random.set_seed(1234)
np.random.seed(1234)

np.set_printoptions(threshold=sys.maxsize)

mnist = tf.keras.datasets.mnist

(X_train, y_train), (X_test, y_test) = mnist.load_data()

print(X_train.shape, y_train.shape, X_test.shape, y_test.shape)

img_A_input = Input((28, 28, 3), name='img_A_input')
img_B_input = Input((28, 28, 3), name='img_B_input')

def euclidean_distance(vects):
    """Find the Euclidean distance between two vectors.

    Arguments:
        vects: List containing two tensors of same length.

    Returns:
        Tensor containing euclidean distance
        (as floating point value) between vectors.
    """

    x, y = vects
    sum_square = tf.math.reduce_sum(tf.math.square(x - y), axis=1, keepdims=True)
    return tf.math.sqrt(tf.math.maximum(sum_square, tf.keras.backend.epsilon()))

def loss(margin=1):
    """Provides 'contrastive_loss' an enclosing scope with variable 'margin'.

    Arguments:
        margin: Integer, defines the baseline for distance for which pairs
                should be classified as dissimilar. - (default is 1).

    Returns:
        'contrastive_loss' function with data ('margin') attached.
    """

    # Contrastive loss = mean( (1-true_value) * square(prediction) +
    #                         true_value * square( max(margin-prediction, 0) ))
    def contrastive_loss(y_true, y_pred):
        """Calculates the contrastive loss.

        Arguments:
            y_true: List of labels, each label is of type float32.
            y_pred: List of predictions of same length as of y_true,
                    each label is of type float32.

        Returns:
            A tensor containing contrastive loss as floating point value.
        """

        square_pred = tf.math.square(y_pred)
        margin_square = tf.math.square(tf.math.maximum(margin - (y_pred), 0))
        return tf.math.reduce_mean((1 - y_true) * square_pred + (y_true) * margin_square)

    return contrastive_loss

cnn = tf.keras.Sequential([
    tf.keras.layers.Conv2D(32, (2, 2), activation="tanh", padding="same"),
    tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),
    tf.keras.layers.Dropout(0.3),

    tf.keras.layers.Conv2D(64, (2, 2), activation="tanh", padding="same"),
    tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),
    tf.keras.layers.Dropout(0.3),

    tf.keras.layers.Conv2D(128, (2, 2), activation="tanh", padding="same"),
    tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),
    tf.keras.layers.Dropout(0.3),

    tf.keras.layers.Conv2D(256, (2, 2), activation="tanh", padding="same"),
    tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),
    tf.keras.layers.Dropout(0.3),

    tf.keras.layers.GlobalAveragePooling2D(),
    tf.keras.layers.Dense(64, activation='tanh')
])

feature_vector_A = cnn(img_A_input)
feature_vector_B = cnn(img_B_input)

merge_layer = tf.keras.layers.Lambda(euclidean_distance, output_shape=(1,))(
    [feature_vector_A, feature_vector_B]
)
normal_layer = tf.keras.layers.BatchNormalization()(merge_layer)

output = Dense(1, activation='sigmoid')(normal_layer)

model = Model(inputs=[img_A_input, img_B_input], outputs=output)

random_indices = np.random.choice(X_train.shape[0], 500, replace=False)
X_train_sample, y_train_sample = X_train[random_indices], y_train[random_indices]

random_indices = np.random.choice(X_test.shape[0], 200, replace=False)
X_test_sample, y_test_sample = X_test[random_indices], y_test[random_indices]

def make_paired_dataset(X,y):
    X_pairs, y_pairs = [], []

    tuples = [(x1, y1) for x1, y1 in zip(X,y)]

    for t in itertools.product(tuples, tuples):
        img_A, label_A = t[0]
        img_B, label_B = t[1]

        img_A = tf.expand_dims(img_A, -1)
        img_A = tf.image.grayscale_to_rgb(img_A)

        img_B = tf.expand_dims(img_B, -1)
        img_B = tf.image.grayscale_to_rgb(img_B)

        new_label = float(label_A == label_B)

        X_pairs.append([img_A, img_B])
        y_pairs.append(new_label)

    pairs = [(x, y) for x, y in zip(X_pairs, y_pairs)]
    shuffle(pairs)

    X_pairs = np.array([x for x, _ in pairs])
    y_pairs = np.array([y for _, y in pairs])

    return X_pairs, y_pairs

def generate_paired_samples_dev(X, y):
    tuples = [(x1, y1) for x1, y1 in zip(X, y)]

    for t in itertools.product(tuples, tuples):
        img_A, label_A = t[0]
        img_B, label_B = t[1]

        img_A = tf.expand_dims(img_A, -1)
        img_A = tf.image.grayscale_to_rgb(img_A)

        img_B = tf.expand_dims(img_B, -1)
        img_B = tf.image.grayscale_to_rgb(img_B)

        new_label = float(label_A == label_B)
        yield [img_A, img_B], new_label

X_train_pairs, y_train_pairs = make_paired_dataset(X_train_sample, y_train_sample)
X_test_pairs, y_test_pairs = make_paired_dataset(X_test_sample, y_test_sample)

train_dataset = tf.data.Dataset.from_generator(
    generate_paired_samples_dev,
    args=(X_train_sample, y_train_sample),
    output_signature=(
        tf.TensorSpec(shape=(2,) + (28, 28, 3), dtype=tf.int32),
        tf.TensorSpec(shape=(), dtype=tf.float32)
    )
).map(lambda x, y: ({'img_A_input': x[0], 'img_B_input': x[1]}, y))
train_dataset = train_dataset.batch(batch_size=32)
train_dataset = train_dataset.prefetch(tf.data.AUTOTUNE)

val_dataset = tf.data.Dataset.from_generator(
    generate_paired_samples_dev,
    args=(X_test_sample, y_test_sample),
    output_signature=(
        tf.TensorSpec(shape=(2,) + (28, 28, 3), dtype=tf.int32),
        tf.TensorSpec(shape=(), dtype=tf.float32)
    )
).map(lambda x, y: ({'img_A_input': x[0], 'img_B_input': x[1]}, y))
val_dataset = val_dataset.batch(batch_size=32)
val_dataset = val_dataset.prefetch(tf.data.AUTOTUNE)

model.compile(loss=loss(), optimizer=tf.keras.optimizers.Adam(learning_rate=0.001), metrics=['accuracy'])
model.summary()

class_weight = {0: 0.1,
                1: 0.9}

weights = model.get_weights()

"""
    Training with all the data pairs stored in arrays
"""
model.fit(x=[X_train_pairs[:, 0, :, :], X_train_pairs[:, 1, :, :]], # numbers to differentiate the input images for each subnetwork
          y=y_train_pairs,
          validation_data=([X_test_pairs[:, 0, :, :], X_test_pairs[:, 1, :, :]], y_test_pairs),
          epochs=10, batch_size=32, class_weight=class_weight, verbose=2)

print(model.evaluate(x=[X_test_pairs[:, 0, :, :], X_test_pairs[:, 1, :, :]], y=y_test_pairs, batch_size=32, verbose=2))

model.set_weights(weights) # just to reset the initial state without any training

"""
    Training with all the data using the tf.data.Dataset class, so that the data isn't all kept in memory at the same time
"""
model.fit(train_dataset,
          validation_data=val_dataset,
          epochs=10, class_weight=class_weight, verbose=2)
print(model.evaluate(x=[X_test_pairs[:, 0, :, :], X_test_pairs[:, 1, :, :]], y=y_test_pairs, batch_size=32, verbose=2))

model.set_weights(weights) # just to reset the initial state without any training

batch_size = 32
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
loss_fn = loss()

train_acc_metric = tf.keras.metrics.Accuracy()
val_acc_metric = tf.keras.metrics.Accuracy()

"""
    Custom training loop to train the model by batch. This is just a demo where the final use case is to only have in memory
    the file paths for the images, and load only each batch of images when needed, since the whole dataset wouldn't fit in memory
"""
for epoch in range(10):
    tmp = [(x, y) for x, y in zip(X_train_pairs, y_train_pairs)]
    shuffle(tmp)
    X_train_pairs = np.array([x for x, _ in tmp])
    y_train_pairs = np.array([y for _, y in tmp])

    print("Starting epoch " + str(epoch + 1))
    start_time = time.time()
    for idx in range((len(y_train_pairs) // batch_size)):
        batch_x = X_train_pairs[idx * batch_size: (idx + 1) * batch_size]
        batch_y = y_train_pairs[idx * batch_size: (idx + 1) * batch_size]
        with tf.GradientTape() as tape:
            preds = model([batch_x[:, 0, :, :], batch_x[:, 1, :, :]], training=True)
            loss = loss_fn(batch_y, preds)
        grads = tape.gradient(loss, model.trainable_weights)
        optimizer.apply_gradients(zip(grads, model.trainable_weights))
        train_acc_metric.update_state(batch_y, preds)

        # Log every 200 batches.
        if idx % 200 == 0:
            print(
                "Training loss (for one batch) at step %d: %.4f"
                % (idx + 1, float(loss))
            )
            print("Seen so far: %s samples" % ((idx + 1) * batch_size))

    # Display metrics at the end of each epoch.
    train_acc = train_acc_metric.result()
    print("Training acc over epoch: %.4f" % (float(train_acc),))

    # Reset training metrics at the end of each epoch
    train_acc_metric.reset_states()

    # Run a validation loop at the end of each epoch.
    for idx in range((len(y_test_pairs) // batch_size)):
        batch_x = X_test_pairs[idx * batch_size: (idx + 1) * batch_size]
        batch_y = y_test_pairs[idx * batch_size: (idx + 1) * batch_size]
        val_preds = model([batch_x[:, 0, :, :], batch_x[:, 1, :, :]], training=False)
        val_acc_metric.update_state(batch_y, val_preds)

    val_acc = val_acc_metric.result()
    val_acc_metric.reset_states()
    print("Validation acc: %.4f" % (float(val_acc),))
    print("Time taken: %.2fs" % (time.time() - start_time))

Relevant log output
First method training:
Epoch 1/10
7813/7813 - 113s - loss: 0.0109 - accuracy: 0.9274 - val_loss: 0.0238 - val_accuracy: 0.9714 - 113s/epoch - 14ms/step
Epoch 2/10
7813/7813 - 111s - loss: 0.0021 - accuracy: 0.9847 - val_loss: 0.0168 - val_accuracy: 0.9813 - 111s/epoch - 14ms/step
Epoch 3/10
7813/7813 - 111s - loss: 0.0016 - accuracy: 0.9877 - val_loss: 0.0239 - val_accuracy: 0.9728 - 111s/epoch - 14ms/step
Epoch 4/10
7813/7813 - 111s - loss: 0.0015 - accuracy: 0.9890 - val_loss: 0.0159 - val_accuracy: 0.9823 - 111s/epoch - 14ms/step
Epoch 5/10
7813/7813 - 111s - loss: 0.0014 - accuracy: 0.9895 - val_loss: 0.0191 - val_accuracy: 0.9786 - 111s/epoch - 14ms/step
Epoch 6/10
7813/7813 - 111s - loss: 0.0014 - accuracy: 0.9895 - val_loss: 0.0210 - val_accuracy: 0.9768 - 111s/epoch - 14ms/step
Epoch 7/10
7813/7813 - 111s - loss: 0.0012 - accuracy: 0.9909 - val_loss: 0.0205 - val_accuracy: 0.9771 - 111s/epoch - 14ms/step
Epoch 8/10
7813/7813 - 111s - loss: 0.0012 - accuracy: 0.9908 - val_loss: 0.0184 - val_accuracy: 0.9792 - 111s/epoch - 14ms/step
Epoch 9/10
7813/7813 - 111s - loss: 0.0012 - accuracy: 0.9911 - val_loss: 0.0177 - val_accuracy: 0.9798 - 111s/epoch - 14ms/step
Epoch 10/10
7813/7813 - 111s - loss: 0.0010 - accuracy: 0.9922 - val_loss: 0.0182 - val_accuracy: 0.9786 - 111s/epoch - 14ms/step
1250/1250 - 4s - loss: 0.0182 - accuracy: 0.9786 - 4s/epoch - 4ms/step
[0.01822792738676071, 0.9785500168800354]
1250/1250 - 41s - loss: 0.0182 - accuracy: 0.9786 - 41s/epoch - 32ms/step
[0.018227916210889816, 0.9785500168800354]

Second method training:
Epoch 1/10
2813/2813 - 43s - loss: 0.0237 - accuracy: 0.8624 - val_loss: 0.0925 - val_accuracy: 0.8972 - 43s/epoch - 15ms/step
Epoch 2/10
2813/2813 - 42s - loss: 0.0193 - accuracy: 0.8945 - val_loss: 0.0926 - val_accuracy: 0.8972 - 42s/epoch - 15ms/step
Epoch 3/10
2813/2813 - 42s - loss: 0.0192 - accuracy: 0.8945 - val_loss: 0.0924 - val_accuracy: 0.8972 - 42s/epoch - 15ms/step
Epoch 4/10
2813/2813 - 42s - loss: 0.0193 - accuracy: 0.8945 - val_loss: 0.0924 - val_accuracy: 0.8972 - 42s/epoch - 15ms/step
Epoch 5/10
2813/2813 - 42s - loss: 0.0192 - accuracy: 0.8945 - val_loss: 0.0925 - val_accuracy: 0.8972 - 42s/epoch - 15ms/step
Epoch 6/10
2813/2813 - 42s - loss: 0.0192 - accuracy: 0.8945 - val_loss: 0.0924 - val_accuracy: 0.8972 - 42s/epoch - 15ms/step
Epoch 7/10
2813/2813 - 42s - loss: 0.0192 - accuracy: 0.8945 - val_loss: 0.0926 - val_accuracy: 0.8972 - 42s/epoch - 15ms/step
Epoch 8/10
2813/2813 - 42s - loss: 0.0193 - accuracy: 0.8945 - val_loss: 0.0924 - val_accuracy: 0.8972 - 42s/epoch - 15ms/step
Epoch 9/10
2813/2813 - 42s - loss: 0.0193 - accuracy: 0.8945 - val_loss: 0.0925 - val_accuracy: 0.8972 - 42s/epoch - 15ms/step
Epoch 10/10
2813/2813 - 42s - loss: 0.0192 - accuracy: 0.8945 - val_loss: 0.0925 - val_accuracy: 0.8972 - 42s/epoch - 15ms/step
704/704 - 2s - loss: 0.0925 - accuracy: 0.8972 - 2s/epoch - 4ms/step
[0.09247653186321259, 0.8971555829048157]
704/704 - 23s - loss: 0.0925 - accuracy: 0.8972 - 23s/epoch - 32ms/step
[0.09247657656669617, 0.8971555829048157]

Third method training:
(I won't put the log here since it's long and doesn'r add much information, the loss just keeps jumping between 0.05 and 1 randomly, without converging as the number of epochs passed increases)

Hi @Carl0smvs ,
Sorry for the dealy, Please post this issue on keras-team/keras repo. as this issue is more related to keras
Thank you!

I will, thank you!

Hi @Carl0smvs ,
Could you please close this issue if it has already been raised in the Keras repository?
Thank you!


text:
Distributed mode hangs on in local mode

Looks like the workers do not start their server when using the estimator API and running in local-distributed mode. The minimalist code down helps reproduce: runs fine with only one process, but when launched with TF_CONFIG={"cluster": {"ps":["localhost:5040"], "worker":["localhost:5041"]}, "task":{"type":"ps","index":0}} (resp. worker), it hangs on. After investigating, looks like in
tensorflow/contrib/learn/python/learn/experiment.py l.250-258:
# Start the server, if needed. It's important to start the server before
# we (optionally) sleep for the case where no device_filters are set.
# Otherwise, the servers will wait to connect to each other before starting
# to train. We might as well start as soon as we can.
config = self._estimator.config
if (config.environment != run_config.Environment.LOCAL and
    config.environment != run_config.Environment.GOOGLE and
    config.cluster_spec and config.master):
  self._start_server()

the server is not started when the environment is local, no matter distributed or not. When I force the _start_server() to be executed though, everything works just fine.
What related GitHub issues or StackOverflow threads have you found by searching the web for your problem?
I wrote a stackoverflow thread here: http://stackoverflow.com/questions/43076035/tensorflow-minimalist-program-fails-on-distributed-mode
Environment info
Operating System: Windows-64 bit
Installed version of CUDA and cuDNN: none, only on CPU
(please attach the output of ls -l /path/to/cuda/lib/libcud*):
If installed from binary pip package, provide:

A link to the pip package you installed: from install in the website (pip install --upgrade tensorflow)
The output from python -c "import tensorflow; print(tensorflow.__version__)". 1.0.1

If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)
import numpy as np
import tensorflow as tf
from tensorflow.contrib.learn.python.learn import learn_runner
from tensorflow.contrib import layers

DATA_SIZE=10
DIMENSION=5

def generate_input_fn():
    def _input_fn():
        mid = int(DATA_SIZE/2)
        data = np.array([np.ones(DIMENSION) if x < mid else -np.ones(DIMENSION) for x in range(DATA_SIZE)])
        labels = ['0' if x < mid else '1' for x in range(DATA_SIZE)]
        table = tf.contrib.lookup.string_to_index_table_from_tensor(tf.constant(['0', '1']))
        label_tensor = table.lookup(tf.convert_to_tensor(labels, dtype=tf.string))
        return dict(zip(['features'], [tf.convert_to_tensor(data, dtype=tf.float32)])), label_tensor
    return _input_fn

def build_estimator(model_dir):
    features = layers.real_valued_column('features', dimension=DIMENSION)
    return tf.contrib.learn.LinearClassifier(
        feature_columns=[features],
        model_dir=model_dir)

def generate_exp_fun():
    def _exp_fun(output_dir):
        return tf.contrib.learn.Experiment(
            build_estimator(output_dir),
            train_input_fn=generate_input_fn(),
            eval_input_fn=generate_input_fn(),
            train_steps=1000)
    return _exp_fun

if __name__ == '__main__':
    tf.logging.set_verbosity(tf.logging.DEBUG)
    learn_runner.run(generate_exp_fun(), 'job_dir')

What other attempted solutions have you tried?
Logs or other output that would be helpful
(If logs are large, please upload as attachment or provide link).
If server is not started, hangs on with log: INFO:tensorflow:Create CheckpointSaverHook.

@mrry, is distributed mode expected to work on Windows?

The distributed-mode tests pass on Windows, so it is expected to work.

having the same issue trying to use learn.Experiment in a distributed fashion, creating separate processes on a local Ubuntu 16.04, tensorflow 1.1 gpu install.  any progress on this issue?

@lw394 haven't seen any progress but I did not have time yet to investigate more thoroughly though - wanted to have a glance to the unit tests and see if something makes a difference from my script and there tests (might be something wrong that we do... ). If you have access to sources, you can patch just like I proposed to make it work while waiting for a proper fix and a better understanding.

@vbod thanks, I commented out config.environment != run_config.Environment.LOCAL and, and that appears to work.  I wonder why starting the server is prevented when using the Experiment class with a cluster definition locally.

@mrry What's the status of this issue?

It sounds like there might be a bug in the  environment-sensing part of the tf.contrib.learn.Experiment code. I believe it's in a state of flux, but @xiejw might know if there are any expected issues.

This is intended. For environment in run_config,  'local' means not distributed. We probably should document this better.
For distributed training, environment should be set as 'cloud' in the TF_CONFIG. This covers the cases that all nodes in the cluster are localhost process. From implementation perspective, it is hard to define "local distributed" precisely.
Hope this clarifies.

That took care of it, thank you.

It has been 14 days with no activity and the awaiting tensorflower label was assigned. Please update the label and/or status accordingly.

authors: aselle girving lw394 mrry tensorflowbutler vbod xiejw


text:
Has complex MVN been implemented?

I have below error for my code , where muC and SigmaC are complex vector and matrix
dist = tf.contrib.distributions.MultivariateNormalFull(muC, SigmaC)

TypeError: Value passed to parameter 'input' has DataType complex64 not in list of allowed values: float64, float32

I tried to implemented the pdf of MVN by myself, but it looks like tf.matrix_inverse also don't support complex tensor.

What's more generally the intended scope of support for complex operations in tensorflow?

Automatically closing due to lack of recent activity. Since this issue is old at this point, please reopen the issue if it still occurs when tried with the latest version of Tensorflow. Thank you.

authors: 4d55397500 DanqingZ itsmeolivia


text:
MKL: Fix for mkl_input conversion for MKL DNN.

Fix also enables elementwise operations in MKL

@gunan The Windows Cmake test failure doesn't seem to be due to this PR. Let  me know if you need any thing else.

source: gharchive/pull-request