text:
Add deprecation warning

Per https://github.com/tensorflow/addons/pull/1980

@fsx950223 @abhichou4
You are owners of some files modified in this pull request.
Would you kindly review the changes whenever you have the time to?
Thank you very much.

Sounds good to me. Should we revert the change and add the warning?

Tzu-wei, sorry, I think you're right. Let's revert #1980 and remove the argument after next release, what do you think?

Sounds good to me. Should we revert the change and add the warning?

Tzu-wei, sorry, I think you're right. Let's revert #1980 and remove the argument after next release, what do you think?

Sure. Let's do it.

LGTM! Thanks. I didn't weighed in #1980 because IMO end user would always use core image ops as a ref, hence for the end user that argument shouldn't have been in there in the first place. Sorry for overlooking it though.

@WindQAQ I think we can merge this now or are there any more changes that need to be made?

Sure let's merge it :-)

authors: AakashKumarNain WindQAQ bot-of-gabrieldemarmiesse facaiy


text:
Fix FilterResponseNormalization save

Description
Fixes #2334
Type of change

[x] Bug fix
[ ] New Tutorial
[ ] Updated or additional documentation
[x] Additional Testing
[ ] New Activation and the changes conform to the activation contribution guidelines
[ ] New Callback and the changes conform to the callback contribution guidelines
[ ] New Image addition and the changes conform to the image op contribution guidelines
[ ] New Layer and the changes conform to the layer contribution guidelines
[ ] New Loss and the changes conform to the loss contribution guidelines
[ ] New Metric and the changes conform to the metric contribution guidelines
[ ] New Optimizer and the changes conform to the optimizer contribution guidelines
[ ] New RNN Cell and the changes conform to the rnn contribution guidelines
[ ] New Seq2seq addition and the changes conform to the seq2seq contribution guidelines
[ ] New Text addition and the changes conform to the text op contribution guidelines

Checklist:

[x] I've properly formatted my code according to the guidelines

[ ] By running Black + Flake8
[ ] By running pre-commit hooks

[ ] This PR addresses an already submitted issue for TensorFlow Addons
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] This PR contains modifications to C++ custom-ops

How Has This Been Tested?
Passes existing and new tests.

@smokrow
You are owner of some files modified in this pull request.
Would you kindly review the changes whenever you have the time to?
Thank you very much.

@smokrow
You are owner of some files modified in this pull request.
Would you kindly review the changes whenever you have the time to?
Thank you very much.

source: gharchive/pull-request

authors: WindQAQ bot-of-gabrieldemarmiesse

repo: tensorflow/addons

license_type: permissive

license_source: github-api


text:
Simplify exclude_from_weight_decay implementation

Description
This PR changes the handling of exclude_from_weight_decay to ensure that the regex is only evaluated once per step since currently it get's called within a distributed merge_call. I implemented this when trying to debug a performance regression which turned out to be in my code, but I think this still is a nice cleanup to have since it consolidates the handling of decay_var_list and exclude_from_weight_decay.
Type of change

[x] Bug fix
[ ] New Tutorial
[ ] Updated or additional documentation
[ ] Additional Testing
[ ] New Activation and the changes conform to the activation contribution guidelines
[ ] New Callback and the changes conform to the callback contribution guidelines
[ ] New Image addition and the changes conform to the image op contribution guidelines
[ ] New Layer and the changes conform to the layer contribution guidelines
[ ] New Loss and the changes conform to the loss contribution guidelines
[ ] New Metric and the changes conform to the metric contribution guidelines
[ ] New Optimizer and the changes conform to the optimizer contribution guidelines
[ ] New RNN Cell and the changes conform to the rnn contribution guidelines
[ ] New Seq2seq addition and the changes conform to the seq2seq contribution guidelines
[ ] New Text addition and the changes conform to the text op contribution guidelines

Checklist:

[x] I've properly formatted my code according to the guidelines

[x] By running Black + Flake8
[x] By running pre-commit hooks

[ ] This PR addresses an already submitted issue for TensorFlow Addons
[ ] I have made corresponding changes to the documentation
[ ] I have added tests that prove my fix is effective or that my feature works
[ ] This PR contains modifications to C++ custom-ops

How Has This Been Tested?
I adapted the unittests that where directly relying on the internal functions that had been modified.

@philjd
You are owner of some files modified in this pull request.
Would you kindly review the changes whenever you have the time to?
Thank you very much.

authors: bot-of-gabrieldemarmiesse lgeiger


text:
Try to build with Cuda 10.2 on Linux

DO NOT MERGE, this is just a test for now

Hi, I was also looking into cuda 10.2 support. Even if I installed cuda 10.2 on a local machine, and successfully built from source. TF2.3 (dso_loader.cc) is still trying to load cuda 10.1 as it's hard coded. Need to symlink cuda10.2 as 10.1.

Hi @roywei ,
As we have discussed with @frankfliu during our last meeting, we try to keep the build of the native libraries as close as what is being delivered in the Python wheels and try to avoid tampering with the source code of the main TF repository, since it would be hard for us to guarantee any stability by doing so.
We also have discussed starting to have additional "experimental" builds, in which case no stability will be guaranteed and users would be well aware of it. Cuda 10.2-enabled can be one of them. But right now, we already have a lot of issues with our build platform (GitHub actions) and adding more stress on it is probably not something that we will look at in the near future. So I would say that having custom builds (where you can patch the code as you want) is probably still the best approach until TensorFlow supports Cuda 10.2 officially.
There is already this issue reported in TensorFlow core where you can share your ideas as well: https://github.com/tensorflow/tensorflow/issues/38194

authors: karllessard roywei

repo: tensorflow/java

license_source: bigquery


text:
Nsynth - RuntimeError: Folder must contain .wav or .npy files.

I'm getting this error when trying to run the nsynth example. Here the input and the error message. Any ideas? The folder definitely has wav files in it.
(magenta) tate-2:jupyter-notebooks tatecarson$ nsynth_generate \
> --checkpoint_path= /Users/tatecarson/Documents/tensorflow/magenta/pre-trained/wavenet-ckpt/model.ckpt-200000.data-00000-of-00001 \
> --source_path= '/Users/tatecarson/Desktop/Kea' \
> --save_path= /private/tmp/nsynth  \
> --batch_size=4
Traceback (most recent call last):
  File "/Users/tatecarson/miniconda2/envs/magenta/bin/nsynth_generate", line 11, in <module>
    sys.exit(console_entry_point())
  File "/Users/tatecarson/miniconda2/envs/magenta/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/nsynth_generate.py", line 89, in console_entry_point
    tf.app.run(main)
  File "/Users/tatecarson/miniconda2/envs/magenta/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/Users/tatecarson/miniconda2/envs/magenta/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/nsynth_generate.py", line 57, in main
    raise RuntimeError("Folder must contain .wav or .npy files.")
RuntimeError: Folder must contain .wav or .npy files.

That's odd, these are the relevant lines of code from nsynth_generate.py. It explicitly is looking for files ending in ".wav" so you can make sure that's the case. you could also try throwing in a print statement to see if it's picking up the files properly.
    files = tf.gfile.ListDirectory(source_path)
    exts = [os.path.splitext(f)[1] for f in files]
    if ".wav" in exts:
      postfix = ".wav"
    elif ".npy" in exts:
      postfix = ".npy"
    else:
      raise RuntimeError("Folder must contain .wav or .npy files.")

(magenta) tate-2:~ tatecarson$ nsynth_generate \
> --checkpoint_path= /Users/tatecarson/Documents/tensorflow/magenta/pre-trained/wavenet-ckpt/model.ckpt-200000.data-00000-of-00001 \
> --source_path= /Users/tatecarson/Desktop/Kea \
> --save_path= /private/tmp/nsynth  \
> --batch_size=4
source path is /Users/tatecarson
Traceback (most recent call last):
  File "/Users/tatecarson/miniconda2/envs/magenta/bin/nsynth_generate", line 11, in <module>
    sys.exit(console_entry_point())
  File "/Users/tatecarson/miniconda2/envs/magenta/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/nsynth_generate.py", line 92, in console_entry_point
    tf.app.run(main)
  File "/Users/tatecarson/miniconda2/envs/magenta/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/Users/tatecarson/miniconda2/envs/magenta/lib/python2.7/site-packages/magenta/models/nsynth/wavenet/nsynth_generate.py", line 60, in main
    raise RuntimeError("Folder must contain .wav or .npy files.")
RuntimeError: Folder must contain .wav or .npy files.
(magenta) tate-2:~ tatecarson$

See source path is /Users/tatecarson . Why does that not = --source_path= /Users/tatecarson/Desktop/Kea ?

I think you have a space after you equals sign. It should work if you get rid of the whitespaces.
--source_path= /Users/tatecarson/Desktop/Kea \
-->
--source_path=/Users/tatecarson/Desktop/Kea \

Fixed, thank you!

source: gharchive/issue

authors: jesseengel tatecarson

repo: tensorflow/magenta


text:
tensorflow-0.7.1-cp27-none-any.whl is not a supported wheel on this platform.

For bugs/issues, please fill in the following.  The more information you
provide, the more likely we can help you.
Environment info
Operating System:
yosemite os x
If installed from binary pip package, provide:

Which pip package you installed.
The output from python -c "import tensorflow; print(tensorflow.version)".

If installed from sources, provide the commit hash:
Steps to reproduce

What have you tried?
1.source ~/tensorflow/bin/activate
2.pip install --upgrade https://storage.googleapis.com/tensorflow/mac/tensorflow-0.7.1-cp27-none-any.whl
Logs or other output that would be helpful
(If logs are large, please upload as attachment).
tensorflow-0.7.1-cp27-none-any.whl is not a supported wheel on this platform.

What do you get when you run
pip -- version

?

pip 7.1.2 from /usr/local/lib/python3.5/site-packages (python 3.5)

Try installing the python3 wheel, not the python2 one.

(https://storage.googleapis.com/tensorflow/mac/tensorflow-0.7.1-cp35-none-any.whl)

How i didn't get? can u drop me command?

Exception:
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/shutil.py", line 538, in move
os.rename(src, real_dst)
PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.5/site-packages/external/init.py' -> '/var/folders/55/crqk_b7n0pn13rh7c0kwp5300000gn/T/pip-o8eda0ef-uninstall/usr/local/lib/python3.5/site-packages/external/init.py'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/pip/basecommand.py", line 211, in main
status = self.run(options, args)
File "/usr/local/lib/python3.5/site-packages/pip/commands/install.py", line 311, in run
root=options.root_path,
File "/usr/local/lib/python3.5/site-packages/pip/req/req_set.py", line 640, in install
requirement.uninstall(auto_confirm=True)
File "/usr/local/lib/python3.5/site-packages/pip/req/req_install.py", line 716, in uninstall
paths_to_remove.remove(auto_confirm)
File "/usr/local/lib/python3.5/site-packages/pip/req/req_uninstall.py", line 125, in remove
renames(path, new_path)
File "/usr/local/lib/python3.5/site-packages/pip/utils/init.py", line 315, in renames
shutil.move(old, new)
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/shutil.py", line 553, in move
os.unlink(src)
PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.5/site-packages/external/init.py'

Sounds like an issue with permissions -- consider installing in a virtualenv, as described on the install guide on tensorflow.org

Getting the same thing:
PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.4/dist-packages/tensorflow-0.7.1.dist-info'
Trying to install in a conda env. The python 2 version works, but not python 3. It seems to want to install globally instead of in the env.

Thanks  I resolved it, i installed globally instead in envirnoment

authors: maheshwarLigade mwlow vrv

repo: tensorflow/tensorflow


text:
Retraining the lower layers of the inception model for a better classifier.

https://github.com/tensorflow/tensorflow/blob/4e75ae1f1e8c6479cfa86fde1a940453945e6671/tensorflow/examples/image_retraining/retrain.py#L26
Hi, I am trying to use the inception model to learn to classify between about 200 different object types.  I tried the approach mentioned here, which seems to only retrain the last few layer of the model, and I can't get about ~ 60% accuracy even when I am training the model. I imagine if I pass on the gradient and change the weights in the lower layers, that I will be able to improve the accuracy of the model when applied to my data set.
How do I go about doing this ? Any guidance on this being done else where would be extremely helpful.

The retrain.py example adds a new layer to the end of an existing model. Whether a single layer is sufficient, or whether the number of parameters chosen is sufficient, or how much data you should have to train effectively, or what the most effective way to train your classifier of 200 objects is are broad questions which aren't a matter of a TensorFlow bug or feature request :)
Apologies, but I'm going to close this issue out since the TensorFlow github issues are meant for bugs and feature requests.

I wasn't asking a broad question of creating my own model. I was asking how can I pass the gradients down to the lower layers of the Inception Model that is provided by tensorflow. Only the last layers is retrained, how can I retrain the lower layers of an existing model ?
I apologize if I was not clear earlier.

authors: asimshankar scm-ns


text:
Faster R-CNN: too many resources requested for launch

I am trying to deploy the pretrained Faster-RCNN Inception V2 from the object detection API on a Jetson TX2.
I am running CUDA 8, cuDNN 6 and have tested with both TF 1.3 and 1.5 in a Jupyter Notebook environment.
When I monitor the GPU memory it starts out by having 4.8 GB free and when launching these fills up immediately. When I run on my GTX1060 6 GB GPU I have effectively the same amount of memory free but are having no issues running.
Smaller models as SSD MobileNet runs without problems.
From tests performed today, I can supply the following dumps.
Jupyter Notebook terminal output:
2018-01-17 16:16:19.584106: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:859] ARM64 does not support NUMA - returning NUMA node zero
2018-01-17 16:16:19.584261: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1206] Found device 0 with properties:
name: NVIDIA Tegra X2 major: 6 minor: 2 memoryClockRate(GHz): 1.3005
pciBusID: 0000:00:00.0
totalMemory: 7.67GiB freeMemory: 4.97GiB
2018-01-17 16:16:19.584312: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1300] Adding visible gpu device 0
2018-01-17 16:16:20.824479: I tensorflow/core/common_runtime/gpu/gpu_device.cc:987] Creating TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4437 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
2018-01-17 16:17:09.816477: E tensorflow/stream_executor/cuda/cuda_driver.cc:1080] failed to synchronize the stop event: CUDA_ERROR_LAUNCH_FAILED
2018-01-17 16:17:09.816703: E tensorflow/stream_executor/cuda/cuda_timer.cc:54] Internal: error destroying CUDA event in context 0x7f001959b0: CUDA_ERROR_LAUNCH_FAILED
2018-01-17 16:17:09.816771: E tensorflow/stream_executor/cuda/cuda_timer.cc:59] Internal: error destroying CUDA event in context 0x7f001959b0: CUDA_ERROR_LAUNCH_FAILED
2018-01-17 16:17:09.816912: E tensorflow/stream_executor/cuda/cuda_dnn.cc:2456] failed to enqueue convolution on stream: CUDNN_STATUS_EXECUTION_FAILED
2018-01-17 16:17:10.174651: E tensorflow/stream_executor/event.cc:33] error destroying CUDA event in context 0x7f001959b0: CUDA_ERROR_LAUNCH_FAILED
2018-01-17 16:17:10.174772: E tensorflow/stream_executor/event.cc:33] error destroying CUDA event in context 0x7f001959b0: CUDA_ERROR_LAUNCH_FAILED
2018-01-17 16:17:10.174806: E tensorflow/stream_executor/event.cc:33] error destroying CUDA event in context 0x7f001959b0: CUDA_ERROR_LAUNCH_FAILED
2018-01-17 16:17:10.174836: E tensorflow/stream_executor/event.cc:33] error destroying CUDA event in context 0x7f001959b0: CUDA_ERROR_LAUNCH_FAILED
2018-01-17 16:17:10.174865: E tensorflow/stream_executor/event.cc:33] error destroying CUDA event in context 0x7f001959b0: CUDA_ERROR_LAUNCH_FAILED

Error dump from printout inside the notebook:
Exception in thread Thread-4:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "<ipython-input-5-a51933cd03d8>", line 19, in worker
    im, t_elapsed = detect_objects(frame_rgb, sess, detection_graph)
  File "<ipython-input-4-6c8da66803e2>", line 19, in detect_objects
    feed_dict={image_tensor: image_np_expanded})
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 895, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1128, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1344, in _do_run
    options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1363, in _do_call
    raise type(e)(node_def, op, message)
InternalError: cuDNN launch failure : input shape([1,64,138,256]) filter shape([3,3,64,192])
	 [[Node: FirstStageFeatureExtractor/InceptionV2/InceptionV2/Conv2d_2c_3x3/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](FirstStageFeatureExtractor/InceptionV2/InceptionV2/Conv2d_2b_1x1/Relu, FirstStageFeatureExtractor/InceptionV2/Conv2d_2c_3x3/weights/read/_47__cf__53)]]
	 [[Node: BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/SortByField/Equal/_883 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_10508...ield/Equal", tensor_type=DT_BOOL, _device="/job:localhost/replica:0/task:0/device:CPU:0"](^_cloopBatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/non_max_suppression/iou_threshold/_1)]]

Caused by op u'FirstStageFeatureExtractor/InceptionV2/InceptionV2/Conv2d_2c_3x3/Conv2D', defined at:
  File "/usr/lib/python2.7/threading.py", line 774, in __bootstrap
    self.__bootstrap_inner()
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "<ipython-input-5-a51933cd03d8>", line 10, in worker
    tf.import_graph_def(od_graph_def, name='')
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/util/deprecation.py", line 316, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/importer.py", line 548, in import_graph_def
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3176, in create_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1617, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InternalError (see above for traceback): cuDNN launch failure : input shape([1,64,138,256]) filter shape([3,3,64,192])
	 [[Node: FirstStageFeatureExtractor/InceptionV2/InceptionV2/Conv2d_2c_3x3/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](FirstStageFeatureExtractor/InceptionV2/InceptionV2/Conv2d_2b_1x1/Relu, FirstStageFeatureExtractor/InceptionV2/Conv2d_2c_3x3/weights/read/_47__cf__53)]]
	 [[Node: BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/SortByField/Equal/_883 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_10508...ield/Equal", tensor_type=DT_BOOL, _device="/job:localhost/replica:0/task:0/device:CPU:0"](^_cloopBatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/non_max_suppression/iou_threshold/_1)]]

Output of tegrastats at the point of error:
RAM 3151/7851MB (lfb 915x4MB) cpu [2%@345,100%@2034,99%@2034,1%@348,3%@348,6%@349] EMC 5%@1866 APE 150 GR3D 0%@114
RAM 3151/7851MB (lfb 915x4MB) cpu [0%@345,100%@1981,100%@1988,3%@348,5%@348,4%@349] EMC 5%@1866 APE 150 GR3D 0%@114
RAM 3152/7851MB (lfb 915x4MB) cpu [2%@345,100%@2021,100%@2021,4%@348,5%@348,2%@349] EMC 5%@1866 APE 150 GR3D 0%@114
RAM 3152/7851MB (lfb 915x4MB) cpu [2%@345,100%@2035,100%@2034,3%@349,4%@348,2%@348] EMC 5%@1866 APE 150 GR3D 0%@114
RAM 3152/7851MB (lfb 915x4MB) cpu [1%@345,100%@2016,100%@2019,2%@345,1%@349,3%@348] EMC 5%@1866 APE 150 GR3D 0%@114
RAM 3181/7851MB (lfb 898x4MB) cpu [21%@806,100%@2021,56%@2024,8%@499,10%@500,3%@500] EMC 5%@1866 APE 150 GR3D 24%@114
RAM 3210/7851MB (lfb 887x4MB) cpu [8%@345,100%@2018,32%@2026,7%@345,24%@345,13%@349] EMC 5%@1866 APE 150 GR3D 99%@114
RAM 3327/7851MB (lfb 838x4MB) cpu [2%@1573,100%@1987,31%@1992,35%@1574,13%@1575,5%@1573] EMC 5%@1866 APE 150 GR3D 8%@114
RAM 3578/7851MB (lfb 758x4MB) cpu [19%@1806,100%@2080,0%@2035,7%@2035,2%@2035,56%@1727] EMC 5%@1866 APE 150 GR3D 10%@114
RAM 3732/7851MB (lfb 715x4MB) cpu [2%@345,100%@2034,83%@2035,5%@348,21%@345,2%@346] EMC 7%@1866 APE 150 GR3D 99%@624
RAM 3732/7851MB (lfb 715x4MB) cpu [94%@2036,100%@2035,97%@2034,87%@1987,13%@2035,1%@2035] EMC 4%@1866 APE 150 GR3D 43%@1032
RAM 3659/7851MB (lfb 727x4MB) cpu [2%@653,81%@2022,20%@2027,28%@652,2%@655,4%@655] EMC 3%@1866 APE 150 GR3D 0%@114
RAM 3661/7851MB (lfb 727x4MB) cpu [1%@345,100%@2033,0%@2035,1%@346,2%@348,3%@349] EMC 3%@1866 APE 150 GR3D 0%@114
RAM 3661/7851MB (lfb 727x4MB) cpu [2%@345,100%@2035,0%@2034,0%@348,3%@348,0%@348] EMC 2%@1866 APE 150 GR3D 0%@114
RAM 3661/7851MB (lfb 727x4MB) cpu [3%@345,100%@2034,0%@2035,1%@348,1%@348,3%@348] EMC 2%@1866 APE 150 GR3D 0%@114
RAM 3661/7851MB (lfb 727x4MB) cpu [2%@345,100%@2034,0%@2034,2%@348,4%@348,1%@348] EMC 2%@1866 APE 150 GR3D 0%@114
RAM 3661/7851MB (lfb 727x4MB) cpu [4%@345,100%@1988,0%@1987,2%@346,2%@345,1%@345] EMC 2%@1866 APE 150 GR3D 9%@114
RAM 3661/7851MB (lfb 727x4MB) cpu [4%@345,100%@2026,0%@2026,1%@347,0%@348,3%@348] EMC 2%@1866 APE 150 GR3D 0%@114
RAM 3661/7851MB (lfb 727x4MB) cpu [8%@345,100%@2024,0%@2028,5%@345,8%@345,1%@345] EMC 2%@1866 APE 150 GR3D 0%@114

As you can see the RAM are nowhere near full at the moment of the error.
Can anybody suggest a solution to this?

@JesperChristensen89 I also got this problem.  Have you solved it? any progress?

@samuel1208 No, unfortunately not. Let me know if you have any progress on the issue.

Same problem happened to me. If this cannot be solved, MaskRCNN cannot be used on Jetson TX2 either. Any progress?

also hanging on this error for a while.
My mask rcnn consumes 1.8GB in total, but is not able to run on the Jetson TX2 gpu, only on cpu.

I finally make it work on tx2 but using mxnet....said story

Any updates on the same?


text:
Tensorflow 1.5.0 manylinux binary requires Numpy 1.14.1

On Travis-CI using Python 3.6 Numpy 1.13.2 is installed. Trying to use the tensorflow manylinux binary from PyPi will result in:
RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb

A log from a broken build is here:
https://travis-ci.org/TwentyBN/ionn/jobs/344881411
Fixing the .travis.yml like so:
https://github.com/TwentyBN/ionn/commit/24ced51bb26d78aa5fed265ed9196f47da4d7d6d
Produces a green build like here:
https://travis-ci.org/TwentyBN/ionn/jobs/346796560
Reading the build log, Numpy is updated like so:
$ pip install -U numpy
Collecting numpy
  Downloading numpy-1.14.1-cp36-cp36m-manylinux1_x86_64.whl (12.2MB)
    100% |████████████████████████████████| 12.2MB 57kB/s
Installing collected packages: numpy
  Found existing installation: numpy 1.13.3
    Uninstalling numpy-1.13.3:
      Successfully uninstalled numpy-1.13.3
Successfully installed numpy-1.14.1

Please set the minimum version for Tensorflow 1.5.0 on Python 3.6 to Numpy 1.14.1. Thanks!

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

Have I written custom code : no
OS Platform and Distribution : Linux - Travis-CI - Trusty
TensorFlow installed from : PyPi
TensorFlow version: 1.5.0 (Python 3.6)
Bazel version : N/A
CUDA/cuDNN version : N/A
GPU model and memory : N/A
Exact command to reproduce : N/A

Nagging Assignee @bignamehyp: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

I'm not sure we are officially support travis-CI. So what's the current minimum version for Tensorflow 1.5.0 on Python 3.6?

1.14.1

@yifeif  can you please take a look? Thanks

Nagging Assignee @bignamehyp: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @bignamehyp: It has been 30 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @bignamehyp: It has been 45 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Thanks for reporting @esc. Sorry missed this one. From TF 1.6+, we have updated the numpy dependency to >=1.13.3. Closing this.

authors: bignamehyp esc tensorflowbutler yifeif


text:
Could anybody help check this problem? I've been stuck in two days and didn't find the solution.

I'm using the tf.Dataset API as follows: Suppose I have 10 *.tfrecords files (named trn-001.tfrecords,trn-002.tfrecords,etc.). There are 128 samples in each of these tfrecords. Suppose the samples in the first 5 tfrecords belong to class 0 and the remaining 5 tfrecords belong to class 1. In my case, the data provider should first random select 2 tfrecords for each class from those 5 tfrecords, then select fixed number of samples from each of the selected tfrecords and combine them to generate one balanced batch data to feed into the network for training. For example, for class 0, select trn-000.tfrecords and trn-001.tfrecords; for class 1, select trn-006.tfrecords and trn-009.tfrecords; then for each of the four selected tfrecords, randomly select 1 sample from each of them and combine the 1+1+1+1 samples to generate one batch data. My script to initialize the dataset operations is like this:
trn0_filenames = ['trn-000.tfrecords','trn-001.tfrecords','trn-002.tfrecords','trn-003.tfrecords','trn-004.tfrecords']
trn1_filenames = ['trn-005.tfrecords','trn-006.tfrecords','trn-007.tfrecords','trn-008.tfrecords','trn-009.tfrecords']

half_batch_size = int(BATCH_SIZE / 2)
trn0_count = len(trn0_filenames)
trn1_count = len(trn1_filenames)
features0_ops = []
labels0_ops = []
for trn0_filename in trn0_filenames:
    feature_op, label_op, = dataset_input_from_tfrecords([trn0_filename], batch_size=1, num_epochs=50000, shuffle=True, image_mean=None)
    features0_ops.append(feature_op)
    labels0_ops.append(label_op)

features1_ops = []
labels1_ops = []
for trn1_filename in trn1_filenames:
    feature_op, label_op = dataset_input_from_tfrecords([trn1_filename], batch_size=1, num_epochs=50000, shuffle=True, image_mean=None)
    features1_ops.append(feature_op)
    label_ops.append(label_op)

Then during training, the script to generate the batch data is like this:
features = np.zeros((BATCH_SIZE, HEIGHT, WIDTH, 3), dtype=np.float32)
labels = np.zeros((BATCH_SIZE,), dtype=np.int64)

while not coord.should_stop():
    ind0 = np.random.choice(trn0_count, half_batch_size)
    ind1 = np.random.choice(trn1_count, half_batch_size)
    batch_idx = 0
    for idx in ind0:
        temp1,temp2 = sess.run([features0_ops[idx],
                                            labels0_ops[idx]])
        features[batch_idx] = temp1[0]
        labels[batch_idx] = temp2[0]
        batch_idx+=1
    for idx in ind1:
        temp1,temp2 = sess.run([features1_ops[idx],
                                labels1_ops[idx]])
        features[batch_idx] = temp1[0]
        labels[batch_idx] = temp2[0]
        batch_idx+=1

and then feed the batch data into network for training:
summary,_,loss_total,loss,loss_reg = sess.run([summary_op, train_op,
                               loss_total_op,loss_op,loss_reg_op],
                              feed_dict={x_tensor: features, y_tensor: labels})

The problem is: during training, I found that the memory usage of the process is consistently increasing. Please check the following screenshot. The process 3208 is the running process. At first, the memory usage is about 10%, but at the moment when I taken this picture, the memory usage has increased to 41.6%. So my question is: is this a memory problem? What's wrong? I cannot find the problem. If the memory usage is increasing to some certain amount, the computer will be halt down then I need to restart.

Can you please share the code for the dataset_input_from_tfrecords() function, as well as what exact version of TensorFlow you're using?

Hi @mrry , if I use the ordinary version without randomly use the python list to save the ops (feature_op, label_op), and use the following:
trn_filenames = trn0_filenames +trn1_filenames
features_op, labels_op = dataset_input_from_tfrecords(filenames, batch_size=4, num_epochs=50000, shuffle=True, image_mean=None)

Then everything works fine, which means the dataset_input_from_tfrecords is fine. But in my case I want to make the class distribution balanced and just one sample from one tfrecords file. I'm not sure if the Dataset API has this functionality, thus I generated many Datasets operations and did the work like that described above.

@mrry Sorry, the problem may not lie in the Dataset API. Maybe lie in the following:  After one epoch, I conduct the prediction on validation set like this:
if step%NUM_TRAIN_STEPS_ONE_EPOCH==0:
    val_features_op, val_labels_op = dataset_input_from_tfrecords(val_filenames, batch_size=1, num_epochs=50000, shuffle=True, image_mean=None)
    ....
    val_features, val_labels = sess.run([val_features_op, val_labels_op])
    ....

So the val_features_op, val_labels_op will add to the whole graph for many times during the loop, right? Maybe this is the problem.

@mrry , that's the problem. After I commented the validation code, the memory usage is reasonable during training. Sorry for disturbing you!

Thanks for confirming! Yes, that call to dataset_input_from_tfrecords() inside the training loop would cause the graph to grow.

@mrry , thanks a lot for your confirmation!


text:
GPU: tf.Session() needs 1-3 minutes at first start

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Linux powerai 4.4.0-109-generic #132-Ubuntu SMP Tue Jan 9 20:00:40 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

TensorFlow installed from (source or binary):
source (R1.10)

TensorFlow version (use command below):
R1.10

Python version:
3.5.2

Bazel version (if compiling from source):
1.15.2

GCC/Compiler version (if compiling from source):
5.4.0 20160609

CUDA/cuDNN version:
CUDA 9.0, cuDNN 7.1

GPU model and memory:
4 x  Tesla P100-SXM2-16GB

Exact command to reproduce:
import tensorflow as tf
sess = tf.Session()

Describe the problem
Every first time I start a tensorflow session, it needs 1-3 minutes to finish.
It freezes at the point "2018-09-04 08:32:04.920181: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0, 1, 2, 3"
After waiting and restarting the python script it works in 1-5 seconds.
Source code / logs
There is no additional source code or loginformation.

Its likely jit compiling the binary PTX up to your device's compute capacity. See for e.g. https://devtalk.nvidia.com/default/topic/810047/ptx-compute-2-0-not-forward-compatible-with-gtx-970-/

I don't know how this link should solve the issue ?

Tensorflow binaries are compiled against specific compute capabilities (3.5 and 5.2 IIRC). If you your device is a different compute capability, the PTX is jit-compiled to your device's compute capability. Since there is a great deal of CUDA code, this initially takes some time, but I believe its a once off thing.

It's 6.0

Seeing the 970 cause the JIT rebuild in cycles, as if the delay is something that get's GC'd. Kind of adds some lame delays waiting for the GPUs to be accessible.

authors: Tockra sjperkins toddpi314


text:
Decorated function tried to create variables on non-first call

I am training a TM model using transfer and whin I executing the fit(dataset, epochs=3) I got the following error
`---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-38-c3f14a31606c> in <module>
      2 EPOCHS = 1
      3
----> 4 model.fit(dataset, epochs=EPOCHS)

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
    726         max_queue_size=max_queue_size,
    727         workers=workers,
--> 728         use_multiprocessing=use_multiprocessing)
    729
    730   def evaluate(self,

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py in fit(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, **kwargs)
    322                 mode=ModeKeys.TRAIN,
    323                 training_context=training_context,
--> 324                 total_epochs=epochs)
    325             cbks.make_logs(model, epoch_logs, training_result, ModeKeys.TRAIN)
    326

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py in run_one_epoch(model, iterator, execution_function, dataset_size, batch_size, strategy, steps_per_epoch, num_samples, mode, training_context, total_epochs)
    121         step=step, mode=mode, size=current_batch_size) as batch_logs:
    122       try:
--> 123         batch_outs = execution_function(iterator)
    124       except (StopIteration, errors.OutOfRangeError):
    125         # TODO(kaftan): File bug about tf function and errors.OutOfRangeError?

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py in execution_function(input_fn)
     84     # `numpy` translates Tensors to values in Eager mode.
     85     return nest.map_structure(_non_none_constant_value,
---> 86                               distributed_function(input_fn))
     87
     88   return execution_function

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py in __call__(self, *args, **kwds)
    455
    456     tracing_count = self._get_tracing_count()
--> 457     result = self._call(*args, **kwds)
    458     if tracing_count == self._get_tracing_count():
    459       self._call_counter.called_without_tracing()

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py in _call(self, *args, **kwds)
    518         # Lifting succeeded, so variables are initialized and we can run the
    519         # stateless function.
--> 520         return self._stateless_fn(*args, **kwds)
    521     else:
    522       canon_args, canon_kwds = \

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py in __call__(self, *args, **kwargs)
   1820   def __call__(self, *args, **kwargs):
   1821     """Calls a graph function specialized to the inputs."""
-> 1822     graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
   1823     return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
   1824

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py in _maybe_define_function(self, args, kwargs)
   2148         graph_function = self._function_cache.primary.get(cache_key, None)
   2149         if graph_function is None:
-> 2150           graph_function = self._create_graph_function(args, kwargs)
   2151           self._function_cache.primary[cache_key] = graph_function
   2152         return graph_function, args, kwargs

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   2039             arg_names=arg_names,
   2040             override_flat_arg_shapes=override_flat_arg_shapes,
-> 2041             capture_by_value=self._capture_by_value),
   2042         self._function_attributes,
   2043         # Tell the ConcreteFunction to clean up its graph once it goes out of

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    913                                           converted_func)
    914
--> 915       func_outputs = python_func(*func_args, **func_kwargs)
    916
    917       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py in wrapped_fn(*args, **kwds)
    356         # __wrapped__ allows AutoGraph to swap in a converted function. We give
    357         # the function a weak reference to itself to avoid a reference cycle.
--> 358         return weak_wrapped_fn().__wrapped__(*args, **kwds)
    359     weak_wrapped_fn = weakref.ref(wrapped_fn)
    360

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py in distributed_function(input_iterator)
     71     strategy = distribution_strategy_context.get_strategy()
     72     outputs = strategy.experimental_run_v2(
---> 73         per_replica_function, args=(model, x, y, sample_weights))
     74     # Out of PerReplica outputs reduce or pick values to return.
     75     all_outputs = dist_utils.unwrap_output_dict(

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/distribute/distribute_lib.py in experimental_run_v2(self, fn, args, kwargs)
    758       fn = autograph.tf_convert(fn, ag_ctx.control_status_ctx(),
    759                                 convert_by_default=False)
--> 760       return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    761
    762   def reduce(self, reduce_op, value, axis):

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/distribute/distribute_lib.py in call_for_each_replica(self, fn, args, kwargs)
   1785       kwargs = {}
   1786     with self._container_strategy().scope():
-> 1787       return self._call_for_each_replica(fn, args, kwargs)
   1788
   1789   def _call_for_each_replica(self, fn, args, kwargs):

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/distribute/distribute_lib.py in _call_for_each_replica(self, fn, args, kwargs)
   2130         self._container_strategy(),
   2131         replica_id_in_sync_group=constant_op.constant(0, dtypes.int32)):
-> 2132       return fn(*args, **kwargs)
   2133
   2134   def _reduce_to(self, reduce_op, value, destinations):

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/autograph/impl/api.py in wrapper(*args, **kwargs)
    290   def wrapper(*args, **kwargs):
    291     with ag_ctx.ControlStatusCtx(status=ag_ctx.Status.DISABLED):
--> 292       return func(*args, **kwargs)
    293
    294   if inspect.isfunction(func) or inspect.ismethod(func):

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py in train_on_batch(model, x, y, sample_weight, class_weight, reset_metrics)
    262       y,
    263       sample_weights=sample_weights,
--> 264       output_loss_metrics=model._output_loss_metrics)
    265
    266   if reset_metrics:

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_eager.py in train_on_batch(model, inputs, targets, sample_weights, output_loss_metrics)
    313     outs = [outs]
    314   metrics_results = _eager_metrics_fn(
--> 315       model, outs, targets, sample_weights=sample_weights, masks=masks)
    316   total_loss = nest.flatten(total_loss)
    317   return {'total_loss': total_loss,

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_eager.py in _eager_metrics_fn(model, outputs, targets, sample_weights, masks)
     72         masks=masks,
     73         return_weighted_and_unweighted_metrics=True,
---> 74         skip_target_masks=model._prepare_skip_target_masks())
     75
     76   # Add metric results from the `add_metric` metrics.

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py in _handle_metrics(self, outputs, targets, skip_target_masks, sample_weights, masks, return_weighted_metrics, return_weighted_and_unweighted_metrics)
   2061           metric_results.extend(
   2062               self._handle_per_output_metrics(self._per_output_metrics[i],
-> 2063                                               target, output, output_mask))
   2064         if return_weighted_and_unweighted_metrics or return_weighted_metrics:
   2065           metric_results.extend(

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py in _handle_per_output_metrics(self, metrics_dict, y_true, y_pred, mask, weights)
   2012       with K.name_scope(metric_name):
   2013         metric_result = training_utils.call_metric_function(
-> 2014             metric_fn, y_true, y_pred, weights=weights, mask=mask)
   2015         metric_results.append(metric_result)
   2016     return metric_results

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_utils.py in call_metric_function(metric_fn, y_true, y_pred, weights, mask)
   1065
   1066   if y_pred is not None:
-> 1067     return metric_fn(y_true, y_pred, sample_weight=weights)
   1068   # `Mean` metric only takes a single value.
   1069   return metric_fn(y_true, sample_weight=weights)

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/metrics.py in __call__(self, *args, **kwargs)
    191     from tensorflow.python.keras.distribute import distributed_training_utils  # pylint:disable=g-import-not-at-top
    192     return distributed_training_utils.call_replica_local_fn(
--> 193         replica_local_fn, *args, **kwargs)
    194
    195   @property

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/distribute/distributed_training_utils.py in call_replica_local_fn(fn, *args, **kwargs)
   1133     with strategy.scope():
   1134       return strategy.extended.call_for_each_replica(fn, args, kwargs)
-> 1135   return fn(*args, **kwargs)
   1136
   1137

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/metrics.py in replica_local_fn(*args, **kwargs)
    174     def replica_local_fn(*args, **kwargs):
    175       """Updates the state of the metric in a replica-local context."""
--> 176       update_op = self.update_state(*args, **kwargs)  # pylint: disable=not-callable
    177       with ops.control_dependencies([update_op]):
    178         result_t = self.result()  # pylint: disable=not-callable

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/utils/metrics_utils.py in decorated(metric_obj, *args, **kwargs)
     73
     74     with tf_utils.graph_context_for_symbolic_tensors(*args, **kwargs):
---> 75       update_op = update_state_fn(*args, **kwargs)
     76     if update_op is not None:  # update_op will be None in eager execution.
     77       metric_obj.add_update(update_op)

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/metrics.py in update_state(self, y_true, y_pred, sample_weight)
    579         y_pred, y_true)
    580
--> 581     matches = self._fn(y_true, y_pred, **self._fn_kwargs)
    582     return super(MeanMetricWrapper, self).update_state(
    583         matches, sample_weight=sample_weight)

<ipython-input-37-0d57172496e6> in accuracy(y_true, y_pred)
      8     # ensure labels have shape (batch_size, MAX_LENGTH - 1)
      9     y_true = tf.reshape(y_true, shape=(-1, MAX_LENGTH - 1))
---> 10     accuracy = tf.metrics.SparseCategoricalAccuracy()(y_true, y_pred)
     11     return accuracy
     12

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/metrics.py in __init__(self, name, dtype)
    762   def __init__(self, name='sparse_categorical_accuracy', dtype=None):
    763     super(SparseCategoricalAccuracy, self).__init__(
--> 764         sparse_categorical_accuracy, name, dtype=dtype)
    765
    766

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/metrics.py in __init__(self, fn, name, dtype, **kwargs)
    552       **kwargs: The keyword arguments that are passed on to `fn`.
    553     """
--> 554     super(MeanMetricWrapper, self).__init__(name=name, dtype=dtype)
    555     self._fn = fn
    556     self._fn_kwargs = kwargs

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/metrics.py in __init__(self, name, dtype)
    452     """
    453     super(Mean, self).__init__(
--> 454         reduction=metrics_utils.Reduction.WEIGHTED_MEAN, name=name, dtype=dtype)
    455
    456

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/metrics.py in __init__(self, reduction, name, dtype)
    291     with ops.init_scope():
    292       self.total = self.add_weight(
--> 293           'total', initializer=init_ops.zeros_initializer)
    294       if reduction in [metrics_utils.Reduction.SUM_OVER_BATCH_SIZE,
    295                        metrics_utils.Reduction.WEIGHTED_MEAN]:

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/metrics.py in add_weight(self, name, shape, aggregation, synchronization, initializer, dtype)
    271         collections=[],
    272         synchronization=synchronization,
--> 273         aggregation=aggregation)
    274
    275   ### End: For use by subclasses ###

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py in add_weight(self, name, shape, dtype, initializer, regularizer, trainable, constraint, partitioner, use_resource, synchronization, aggregation, **kwargs)
    520         collections=collections_arg,
    521         synchronization=synchronization,
--> 522         aggregation=aggregation)
    523     backend.track_variable(variable)
    524

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/training/tracking/base.py in _add_variable_with_custom_getter(self, name, shape, dtype, initializer, getter, overwrite, **kwargs_for_getter)
    742         dtype=dtype,
    743         initializer=initializer,
--> 744         **kwargs_for_getter)
    745
    746     # If we set an initializer and the variable processed it, tracking will not

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in make_variable(name, shape, dtype, initializer, trainable, caching_device, validate_shape, constraint, use_resource, collections, synchronization, aggregation, partitioner)
    137       synchronization=synchronization,
    138       aggregation=aggregation,
--> 139       shape=variable_shape if variable_shape else None)
    140
    141

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/ops/variables.py in __call__(cls, *args, **kwargs)
    256   def __call__(cls, *args, **kwargs):
    257     if cls is VariableV1:
--> 258       return cls._variable_v1_call(*args, **kwargs)
    259     elif cls is Variable:
    260       return cls._variable_v2_call(*args, **kwargs)

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/ops/variables.py in _variable_v1_call(cls, initial_value, trainable, collections, validate_shape, caching_device, name, variable_def, dtype, expected_shape, import_scope, constraint, use_resource, synchronization, aggregation, shape)
    217         synchronization=synchronization,
    218         aggregation=aggregation,
--> 219         shape=shape)
    220
    221   def _variable_v2_call(cls,

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/ops/variables.py in getter(**kwargs)
     63
     64   def getter(**kwargs):
---> 65     return captured_getter(captured_previous, **kwargs)
     66
     67   return getter

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/distribute/distribute_lib.py in creator(next_creator, *args, **kwargs)
   2046     def creator(next_creator, *args, **kwargs):
   2047       _require_strategy_scope_strategy(strategy)
-> 2048       return next_creator(*args, **kwargs)
   2049
   2050     self._var_creator_scope = variable_scope.variable_creator_scope(creator)

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/ops/variables.py in getter(**kwargs)
     63
     64   def getter(**kwargs):
---> 65     return captured_getter(captured_previous, **kwargs)
     66
     67   return getter

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/distribute/distribute_lib.py in creator(next_creator, *args, **kwargs)
   2046     def creator(next_creator, *args, **kwargs):
   2047       _require_strategy_scope_strategy(strategy)
-> 2048       return next_creator(*args, **kwargs)
   2049
   2050     self._var_creator_scope = variable_scope.variable_creator_scope(creator)

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/ops/variables.py in getter(**kwargs)
     63
     64   def getter(**kwargs):
---> 65     return captured_getter(captured_previous, **kwargs)
     66
     67   return getter

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/distribute/distribute_lib.py in creator(next_creator, *args, **kwargs)
   2046     def creator(next_creator, *args, **kwargs):
   2047       _require_strategy_scope_strategy(strategy)
-> 2048       return next_creator(*args, **kwargs)
   2049
   2050     self._var_creator_scope = variable_scope.variable_creator_scope(creator)

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/ops/variables.py in getter(**kwargs)
     63
     64   def getter(**kwargs):
---> 65     return captured_getter(captured_previous, **kwargs)
     66
     67   return getter

/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/eager/def_function.py in invalid_creator_scope(*unused_args, **unused_kwds)
    411       """Disables variable creation."""
    412       raise ValueError(
--> 413           "tf.function-decorated function tried to create "
    414           "variables on non-first call.")
    415

ValueError: tf.function-decorated function tried to create variables on non-first call.

Please fill in template and provide a short/minimal reproducing example. I already added backticks to the error paste to make it readable, though it is extremely long.

It has been 14 days with no activity and the awaiting response label was assigned. Is this still an issue?

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

I'm having the same issue. Any updates?

@ryanmaxwell96, Please open new issue and provide all the information asked in the template. Thanks


text:
Cmake issue with cuDNN; Docker container doesn't seem to have cuDNN?

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04

TensorFlow installed from (source or binary): Docker, docker run --gpus all -it --rm tensorflow/tensorflow:latest-gpu-py3

TensorFlow version: 2.0.0

Python version: 3.6.8

Installed using virtualenv? pip? conda?: Docker

Bazel version (if compiling from source): n/a

GCC/Compiler version (if compiling from source): n/a

CUDA/cuDNN version: nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

cudNN: does not seem to come with Docker image. I am trying to use
sudo docker cp libcudnn7_7.6.5.32-1+cuda10.0_amd64.deb [container ID]:/ && sudo docker cp libcudnn7-dev_7.6.5.32-1+cuda10.0_amd64.deb [container ID]:/ && sudo docker cp libcudnn7-doc_7.6.5.32-1+cuda10.0_amd64.deb [container ID]:/ on the host then
dpkg -i libcudnn7_7.6.5.32-1+cuda10.0_amd64.deb && dpkg -i libcudnn7-dev_7.6.5.32-1+cuda10.0_amd64.deb && dpkg -i libcudnn7-doc_7.6.5.32-1+cuda10.0_amd64.deb

GPU model and memory: RTX 2060 6GB

Describe the problem
If I don't install cuDNN manually, dlib compiles "successfully" but will not use CUDA. If I install cudNN via dpkg, I get this error during the build:
-- Using CMake version: 3.10.2
-- Compiling dlib version: 19.18.99
-- Found system copy of libpng: /usr/lib/x86_64-linux-gnu/libpng.so;/usr/lib/x86_64-linux-gnu/libz.so
-- Found system copy of libjpeg: /usr/lib/x86_64-linux-gnu/libjpeg.so
-- Looking for cuDNN install...
-- Found cuDNN: /usr/lib/x86_64-linux-gnu/libcudnn.so
-- Enabling CUDA support for dlib.  DLIB WILL USE CUDA
-- C++11 activated.
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
CUDA_cublas_LIBRARY (ADVANCED)
linked by target "dlib" in directory /home/project/dlib/dlib
CUDA_curand_LIBRARY (ADVANCED)
linked by target "dlib" in directory /home/project/dlib/dlib
CUDA_cusolver_LIBRARY (ADVANCED)
linked by target "dlib" in directory /home/project/dlib/dlib
-- Configuring incomplete, errors occurred!
See also "/home/project/dlib/build/CMakeFiles/CMakeOutput.log".
See also "/home/project/dlib/build/CMakeFiles/CMakeError.log".
root@c7f1e218bc6f:/home/project/dlib/build#
I also tried
sudo docker cp cudnn-10.0-linux-x64-v7.6.5.32.tgz [c7f1e218bc6f]:/
tar -xzvf cudnn-10.0-linux-x64-v7.6.5.32.tgz && cp cuda/include/cudnn.h /usr/local/cuda/include && cp cuda/lib64/libcudnn* /usr/local/cuda/lib64 && chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib64/libcudnn*
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/include export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64
Provide the exact sequence of commands / steps that you executed before running into the problem
sudo docker run --gpus all -it --rm tensorflow/tensorflow:latest-gpu-py3 #run container
python -c "import tensorflow as tf; print(tf.reduce_sum(tf.random.normal([1000, 1000])))" #hello world
note: this^ runs just fine.
sudo docker cp libcudnn7_7.6.5.32-1+cuda10.0_amd64.deb [container ID]:/ && sudo docker cp libcudnn7-dev_7.6.5.32-1+cuda10.0_amd64.deb [container ID]:/ && sudo docker cp libcudnn7-doc_7.6.5.32-1+cuda10.0_amd64.deb [container ID]:/
dpkg -i libcudnn7_7.6.5.32-1+cuda10.0_amd64.deb && dpkg -i libcudnn7-dev_7.6.5.32-1+cuda10.0_amd64.deb && dpkg -i libcudnn7-doc_7.6.5.32-1+cuda10.0_amd64.deb
apt update && apt upgrade && apt install git cmake libx11-dev nano cd home && mkdir project && cd project  git clone https://github.com/davisking/dlib.git #setup dlib
cd dlib && mkdir build; cd build; cmake ..; cmake --build .
Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.
CMakeOutput.log: https://pastebin.com/QC1kdejw
CMakeError.log: https://pastebin.com/JfQc02Qb

Can you try using bazel instead of cmake and let us know how it progresses. Thanks!

@IntegralWorks
Any update on this issue please. Thanks!

@ravikyram
Sorry, new to GitHub and I'm busy this week with examinations. I hope this
message reaches you; I'm replying on email.
Unfortunately I decided using Docker for my project was not worth it and I
ended up successfully installing dlib with CUDA support on bare
metal/regular install.
I noticed that the key difference between my regular install and the Docker
container is that the container seems to refuse multiple versions of cuDNN.
If you're interested in how this issue turns out I highly suggest finding
someone who needs cuDNN in their project and wants to use the Tensorflow
GPU container.
Even stranger is that nvcc works out the box for the Tensorflow container,
whereas for a manual install the user-developer must meticulously follow
nVidia's documentation to the letter, such as adding things to .bashrc and
the such. But cuDNN seems broken even upon installation either through the
tar file or the deb package. With a properly, meticulously installed
regular CUDA installation cuDNN works seamlessly--the point I'm trying to
make is I might so bold as to suggest the Tensorflow GPU container might
have a semi-broken CUDA installation. Of course, it's more likely that I
just don't know how to install cuDNN on the Tensorflow GPU container
properly--but there doesn't seem to be any documentation(!) on this edge
case(?) I don't see how it could be an edge case if dlib isn't some obscure
library.
tl;dr
I have no idea how to use Bazel, I've been trained to use gcc, clang, cmake
etc. All compilation methods I know of cause dlib to *detect CUDA, only to
say "cuDNN is broken, sorry". Therefore I want to emphasize that right now
dlib does **not compile with GPU support on the Tensorflow GPU Docker
Container, *despite the fact that intuitively you'd expect it to just be
plug-and-play, at least after installing cuDNN via the documented
instructions on nVidia's websites.
On Wed, Nov 20, 2019 at 12:03 AM ravikyram notifications@github.com wrote:

@IntegralWorks https://github.com/IntegralWorks
Any update on this issue please. Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/34216?email_source=notifications&email_token=AGOWRXBB73ZPW3VJ6DYPP4LQUTHJRA5CNFSM4JMTBNM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEQZ4HA#issuecomment-555851292,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AGOWRXDE2CIGFGYNR43XHW3QUTHJRANCNFSM4JMTBNMQ
.

@IntegralWorks
CUDA and cuDNN can be installed by following these instructions. Thanks!

@IntegralWorks
Any update on this issue please. Thanks!

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

authors: IntegralWorks ravikyram


text:
tf.reduce_mean crashes TensorFlow Lite

System information

Ubuntu 18.04 4.15.0-66-generic x86_64
TensorFlow installed from (source or binary): official wheel
TensorFlow version (use command below): v2.0.0-rc2-26-g64c3d38 2.0.0
Python version: 3.7
CUDA/cuDNN version: N/A

Describe the current behavior
The program SIGABRTs (see the code below).
Describe the expected behavior
The program does not crash.
Code to reproduce the issue

Code
from time import time

import numpy as np
import tensorflow as tf
from tensorflow.lite.python.interpreter import load_delegate

def measure(size, rep, tpu):
    @tf.function(input_signature=[tf.TensorSpec([size] * 2, tf.float32)] * 2)
    def bench_func(a, b):
        x = tf.linalg.matmul(a, b)
        return tf.reduce_mean(x)

    def gen_input_samples():
        i = np.identity(size, np.float32)
        yield [-i, i]
        yield [i, i]

    converter = tf.lite.TFLiteConverter.from_concrete_functions([bench_func.get_concrete_function()])
    converter.optimizations = [tf.lite.Optimize.DEFAULT]
    converter.representative_dataset = gen_input_samples
    converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
    converter.inference_input_type = tf.uint8
    converter.inference_output_type = tf.uint8
    tflite_model = converter.convert()
    delegates = [load_delegate("libedgetpu.so.1.0")] if tpu else []
    interpreter = tf.lite.Interpreter(model_content=tflite_model, experimental_delegates=delegates)
    interpreter.allocate_tensors()
    input_details = interpreter.get_input_details()
    output_details = interpreter.get_output_details()
    tensor_in1 = interpreter.tensor(input_details[0]["index"])
    tensor_in2 = interpreter.tensor(input_details[1]["index"])
    tensor_out = interpreter.tensor(output_details[0]["index"])
    inp1 = np.random.rand(size, size) - 0.5
    inp2 = np.random.rand(size, size) - 0.5
    tensor_in1()[:] = inp1
    tensor_in2()[:] = inp2
    now = time()
    for _ in range(rep):
        interpreter.invoke()
    return time() - now

print(measure(2048, 1, tpu=False))

Other info / logs

Log
2019-11-20 10:43:18.538007: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2019-11-20 10:43:18.546197: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2019-11-20 10:43:18.546227: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (vadim-XPS-13-9380): /proc/driver/nvidia/version does not exist
2019-11-20 10:43:18.546569: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-11-20 10:43:18.572569: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 1992000000 Hz
2019-11-20 10:43:18.573251: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x48fd5a0 executing computations on platform Host. Devices:
2019-11-20 10:43:18.573290: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
2019-11-20 10:43:18.671265: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2019-11-20 10:43:18.671336: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2019-11-20 10:43:18.690246: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: graph_to_optimize
2019-11-20 10:43:18.690274: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718]   function_optimizer: function_optimizer did nothing. time = 0.005ms.
2019-11-20 10:43:18.690277: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718]   function_optimizer: function_optimizer did nothing. time = 0ms.
2019-11-20 10:43:18.700228: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2019-11-20 10:43:18.700397: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2019-11-20 10:43:18.720772: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:716] Optimization results for grappler item: graph_to_optimize
2019-11-20 10:43:18.720800: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718]   constant folding: Graph size after: 6 nodes (0), 5 edges (0), time = 0.895ms.
2019-11-20 10:43:18.720822: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:718]   constant folding: Graph size after: 6 nodes (0), 5 edges (0), time = 0.072ms.
INFO: Initialized TensorFlow Lite runtime.
INFO: Initialized TensorFlow Lite runtime.
Aborted (core dumped)

gdb bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffff7a24801 in __GI_abort () at abort.c:79
#2  0x00007fff3c18b616 in TfLiteStatus tflite::ops::builtin::reduce::EvalMean<(tflite::ops::builtin::reduce::KernelType)0>(TfLiteContext*, TfLiteNode*) ()
   from /usr/local/lib/python3.7/dist-packages/tensorflow_core/lite/python/interpreter_wrapper/_tensorflow_wrap_interpreter_wrapper.so
#3  0x00007fff3c1f0577 in tflite::Subgraph::Invoke() ()
   from /usr/local/lib/python3.7/dist-packages/tensorflow_core/lite/python/interpreter_wrapper/_tensorflow_wrap_interpreter_wrapper.so
#4  0x00007fff3c1f222a in tflite::Interpreter::Invoke() ()
   from /usr/local/lib/python3.7/dist-packages/tensorflow_core/lite/python/interpreter_wrapper/_tensorflow_wrap_interpreter_wrapper.so
#5  0x00007fff3c0bff58 in tflite::interpreter_wrapper::InterpreterWrapper::Invoke() ()
   from /usr/local/lib/python3.7/dist-packages/tensorflow_core/lite/python/interpreter_wrapper/_tensorflow_wrap_interpreter_wrapper.so
#6  0x00007fff3c0be054 in _wrap_InterpreterWrapper_Invoke ()
   from /usr/local/lib/python3.7/dist-packages/tensorflow_core/lite/python/interpreter_wrapper/_tensorflow_wrap_interpreter_wrapper.so
#7  0x00000000005b0462 in _PyMethodDef_RawFastCallKeywords () at ../Objects/call.c:694
#8  0x00000000005cc860 in _PyCFunction_FastCallKeywords (kwnames=<optimized out>, nargs=<optimized out>, args=0x7fff6c1216c0, func=
    <built-in method InterpreterWrapper_Invoke of module object at remote 0x7fff6c11ed18>) at ../Objects/call.c:730
#9  call_function.lto_priv () at ../Python/ceval.c:4568
#10 0x000000000051add9 in _PyEval_EvalFrameDefault () at ../Python/ceval.c:3093
#11 0x00000000005b0eac in PyEval_EvalFrameEx (throwflag=0, f=
    Frame 0x7fff6c121540, for file /usr/local/lib/python3.7/dist-packages/tensorflow_core/lite/python/interpreter_wrapper/tensorflow_wrap_interpreter_wrapper.py, line 109, in Invoke (self=<InterpreterWrapper(this=<SwigPyObject at remote 0x7fff74aca030>) at remote 0x7fff741f1c88>)) at ../Python/ceval.c:547
#12 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at ../Objects/call.c:283
#13 _PyFunction_FastCallKeywords () at ../Objects/call.c:408
#14 0x00000000005cc6ea in call_function.lto_priv () at ../Python/ceval.c:4616
#15 0x000000000051add9 in _PyEval_EvalFrameDefault () at ../Python/ceval.c:3093
#16 0x00000000005b0eac in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0x7fff6c125048, for file /usr/local/lib/python3.7/dist-packages/tensorflow_core/lite/python/interpreter.py, line 453, in invoke (self=<Interpreter(_model_content=b' \x00\x00\x00TFL3\x00\x00\x00\x00\x00\x00\x12\x00\x1c\x00\x04\x00\x08\x00\x0c\x00\x10\x00\x14\x00\x00\x00\x18\x00\x12\x00\x00\x00\x03\x00\x00\x00\x00&\x00\x00\xf0 \x00\x00\xd8 \x00\x00<\x00\x00\x00\x04\x00\x00\x00\x01\x00\x00\x00\x0c\x00\x00\x00\x08\x00\x0c\x00\x04\x00\x08\x00\x08\x00\x00\x00\x08\x00\x00\x00\t\x00\x00\x00\x13\x00\x00\x00min_runtime_version\x00\n\x00\x00\x00\x90 \x00\x00\x80\x00\x00\x00p\x00\x00\x00d\x00\x00\x00L\x00\x00\x004\x00\x00\x00,\x00\x00\x00$\x00\x00\x00\x1c\x00\x00\x00\x04\x00\x00\x00\xaa\xff\xff\xff\x04\x00\x00\x00\x05\x00\x00\x001.6.0\x00\x00\x00\xa4\xda\xff\xff\xa8\xda\xff\xff\xac\xda\xff\xff\xca\xff\xff\xff\x04\x00\x00\x00\x08\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\xde\xff\xff\xff\x04\x00\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\xd8\xda\xff\xff\x04\x00\x06\x00\x04\x00\x00\x00\x00\x00\x06\x00\x...(truncated)) at ../Python/ceval.c:547
#17 function_code_fastcall (globals=<optimized out>, nargs=<optimized out>, args=<optimized out>, co=<optimized out>) at ../Objects/call.c:283
#18 _PyFunction_FastCallKeywords () at ../Objects/call.c:408
#19 0x0000000000516d6a in call_function (kwnames=0x0, oparg=<optimized out>, pp_stack=0x7fffffffd010) at ../Python/ceval.c:4616
#20 _PyEval_EvalFrameDefault () at ../Python/ceval.c:3110
#21 0x00000000005cd202 in PyEval_EvalFrameEx (throwflag=0,
    f=Frame 0xf20238, for file bench.py, line 40, in measure (rep=1, tpu=False, bench_func=<Function(_lock=<_thread.lock at remote 0x7fffe26e7b98>, _python_function=<function at remote 0x7fffe05bf9d8>, _function_spec=<FunctionSpec(_fullargspec=<FullArgSpec at remote 0x7fff74b27c88>, _is_method=False, _default_values=None, _args_to_indices={'a': 0, 'b': 1}, arg_names=['a', 'b'], vararg_name=None, _arg_indices_to_default_values={}, _input_signature=(<TensorSpec at remote 0x7fff74ab6dc8>, <...>), _flat_input_signature=(<...>, <...>)) at remote 0x7ffff6614710>, _autograph=True, _experimental_autograph_options=None, experimental_relax_shapes=False, _created_variables=[], _stateful_fn=<Function(_python_function=<function at remote 0x7fff74ac2158>, _function_spec=<FunctionSpec(_fullargspec=<FullArgSpec at remote 0x7fff74b27cf8>, _is_method=False, _default_values=None, _args_to_indices={'a': 0, 'b': 1}, arg_names=['a', 'b'], vararg_name=None, _arg_indices_to_default_values={}, _input_signature=(...), _flat_input_signature=...(truncated)) at ../Python/ceval.c:547

tf.reduce_sum() does not crash.

@vmarkovtsev, Please paste the code snippet to reproduce the reported issue here. Thanks!

@gadagashwini But it's already here.

@gadagashwini But it's already here.

Sorry I didn't check. My bad. Thanks!

Could you please try this on the nightly which you can install with pip install --upgrade tf-nightly (after uninstalling the regular version). It works for me on the nightly but not on the release version.

@vmarkovtsev This was resolved in TF2.2. Can you please verify once and close the issue? Thanks!

I'm getting a similar issue now with TF 2.9. The crash happens in NextIndex called by EvalMeanReferenceOps

authors: aselle feliwir gadagashwini jvishnuvardhan vmarkovtsev


text:
TensorFlow Feature Columns Fail for Keras Model Functional API

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Databricks 6.3- Linux version 4.4.0-1100-aws (buildd@lgw01-amd64-030) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.12) ) #111-Ubuntu SMP Wed Dec 4 12:20:15 UTC 2019 (Ubuntu 4.4.0-1100.111-aws 4.4.203)
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below):

2020-02-02 17:40:54.646833: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2020-02-02 17:40:54.648818: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.6
v2.1.0-rc2-17-ge5bf8de 2.1.0

Python version: 3.7.3
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source): GCC 7.3.0
CUDA/cuDNN version: 10.1
GPU model and memory: AWS p2.xlarge
Exact command to reproduce:

source_column = tf.feature_column.numeric_column("source_name")
categorical_column = tf.feature_column.bucketized_column(source_column, [0, 10, 100])
embedding_column = tf.feature_column.embedding_column(categorical_column, 7)
feature_columns = [ embedding_column ]

name_schema = tf.feature_column.make_parse_example_spec(feature_columns)
inputs = {
  name: keras.layers.Input(shape=schema.shape, name=name, dtype=schema.dtype)
  for name, schema in name_schema.items()
}

dense_features = keras.layers.DenseFeatures(feature_columns)
column_tensors = {}
dense_tensor = dense_features(inputs, cols_to_output_tensors=column_tensors)

Describe the problem

Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.

I believe it should be possible to use Feature Columns with Keras Model Functional API.  Most feature columns work, however, it appears that some combinations of feature columns like the above embedding - bucketized - numeric columns fail.
It is not clear how to work around this error or apply the suggested wrapping.
Source code / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

The above code fails with:
ValueError                                Traceback (most recent call last)
<command-1166953> in <module>
     12 dense_features = keras.layers.DenseFeatures(feature_columns)
     13 column_tensors = {}
---> 14 dense_tensor = dense_features(inputs, cols_to_output_tensors=column_tensors)

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    803               kwargs.pop('mask')
    804             inputs, outputs = self._set_connectivity_metadata_(
--> 805                 inputs, outputs, args, kwargs)
    806           self._handle_activity_regularization(inputs, outputs)
    807           self._set_mask_metadata(inputs, outputs, input_masks)

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py in _set_connectivity_metadata_(self, inputs, outputs, args, kwargs)
   2012     # This updates the layer history of the output tensor(s).
   2013     self._add_inbound_node(
-> 2014         input_tensors=inputs, output_tensors=outputs, arguments=arguments)
   2015     return inputs, outputs
   2016

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py in _add_inbound_node(self, input_tensors, output_tensors, arguments)
   2042         input_tensors=input_tensors,
   2043         output_tensors=output_tensors,
-> 2044         arguments=arguments)
   2045
   2046     # Update tensor history metadata.

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/node.py in __init__(self, outbound_layer, inbound_layers, node_indices, tensor_indices, input_tensors, output_tensors, arguments)
    120       if base_layer_utils.needs_keras_history(
    121           tensor_argument, ignore_call_context=True):
--> 122         base_layer_utils.create_keras_history(tensor_argument)
    123
    124     # Add nodes to all layers involved.

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in create_keras_history(tensors)
    185     keras_tensors: The Tensors found that came from a Keras Layer.
    186   """
--> 187   _, created_layers = _create_keras_history_helper(tensors, set(), [])
    188   return created_layers
    189

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in _create_keras_history_helper(tensors, processed_ops, created_layers)
    247               constants[i] = backend.function([], op_input)([])
    248       processed_ops, created_layers = _create_keras_history_helper(
--> 249           layer_inputs, processed_ops, created_layers)
    250       name = op.name
    251       node_def = op.node_def.SerializeToString()

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in _create_keras_history_helper(tensors, processed_ops, created_layers)
    247               constants[i] = backend.function([], op_input)([])
    248       processed_ops, created_layers = _create_keras_history_helper(
--> 249           layer_inputs, processed_ops, created_layers)
    250       name = op.name
    251       node_def = op.node_def.SerializeToString()

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in _create_keras_history_helper(tensors, processed_ops, created_layers)
    247               constants[i] = backend.function([], op_input)([])
    248       processed_ops, created_layers = _create_keras_history_helper(
--> 249           layer_inputs, processed_ops, created_layers)
    250       name = op.name
    251       node_def = op.node_def.SerializeToString()

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in _create_keras_history_helper(tensors, processed_ops, created_layers)
    247               constants[i] = backend.function([], op_input)([])
    248       processed_ops, created_layers = _create_keras_history_helper(
--> 249           layer_inputs, processed_ops, created_layers)
    250       name = op.name
    251       node_def = op.node_def.SerializeToString()

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in _create_keras_history_helper(tensors, processed_ops, created_layers)
    247               constants[i] = backend.function([], op_input)([])
    248       processed_ops, created_layers = _create_keras_history_helper(
--> 249           layer_inputs, processed_ops, created_layers)
    250       name = op.name
    251       node_def = op.node_def.SerializeToString()

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in _create_keras_history_helper(tensors, processed_ops, created_layers)
    221             'Sparse ops are not supported with functional models with built-in '
    222             'layer wrapping. Please wrap the sparse ops in a Lambda layer like'
--> 223             ': \n{lambda_example}\n'.format(lambda_example=lambda_example))
    224
    225       # Recursively set `_keras_history`.

ValueError: Sparse ops are not supported with functional models with built-in layer wrapping. Please wrap the sparse ops in a Lambda layer like:

        weights_mult = lambda x: tf.sparse.sparse_dense_matmul(x, weights)
        output = tf.keras.layers.Lambda(weights_mult)(input)

On Tensorflow 1.15, I get the following error:
ValueError                                Traceback (most recent call last)
<command-1167185> in <module>
     12 dense_features = keras.layers.DenseFeatures(feature_columns)
     13 column_tensors = {}
---> 14 dense_tensor = dense_features(inputs, cols_to_output_tensors=column_tensors)

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs)
    879               kwargs.pop('mask')
    880             inputs, outputs = self._set_connectivity_metadata_(
--> 881                 inputs, outputs, args, kwargs)
    882           self._handle_activity_regularization(inputs, outputs)
    883           self._set_mask_metadata(inputs, outputs, input_masks)

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py in _set_connectivity_metadata_(self, inputs, outputs, args, kwargs)
   2041     # This updates the layer history of the output tensor(s).
   2042     self._add_inbound_node(
-> 2043         input_tensors=inputs, output_tensors=outputs, arguments=arguments)
   2044     return inputs, outputs
   2045

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py in _add_inbound_node(self, input_tensors, output_tensors, arguments)
   2071         input_tensors=input_tensors,
   2072         output_tensors=output_tensors,
-> 2073         arguments=arguments)
   2074
   2075     # Update tensor history metadata.

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/node.py in __init__(self, outbound_layer, inbound_layers, node_indices, tensor_indices, input_tensors, output_tensors, arguments)
    120       if base_layer_utils.needs_keras_history(
    121           tensor_argument, ignore_call_context=True):
--> 122         base_layer_utils.create_keras_history(tensor_argument)
    123
    124     # Add nodes to all layers involved.

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in create_keras_history(tensors)
    182     keras_tensors: The Tensors found that came from a Keras Layer.
    183   """
--> 184   _, created_layers = _create_keras_history_helper(tensors, set(), [])
    185   return created_layers
    186

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in _create_keras_history_helper(tensors, processed_ops, created_layers)
    229               constants[i] = backend.function([], op_input)([])
    230       processed_ops, created_layers = _create_keras_history_helper(
--> 231           layer_inputs, processed_ops, created_layers)
    232       name = op.name
    233       node_def = op.node_def.SerializeToString()

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in _create_keras_history_helper(tensors, processed_ops, created_layers)
    229               constants[i] = backend.function([], op_input)([])
    230       processed_ops, created_layers = _create_keras_history_helper(
--> 231           layer_inputs, processed_ops, created_layers)
    232       name = op.name
    233       node_def = op.node_def.SerializeToString()

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in _create_keras_history_helper(tensors, processed_ops, created_layers)
    229               constants[i] = backend.function([], op_input)([])
    230       processed_ops, created_layers = _create_keras_history_helper(
--> 231           layer_inputs, processed_ops, created_layers)
    232       name = op.name
    233       node_def = op.node_def.SerializeToString()

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in _create_keras_history_helper(tensors, processed_ops, created_layers)
    229               constants[i] = backend.function([], op_input)([])
    230       processed_ops, created_layers = _create_keras_history_helper(
--> 231           layer_inputs, processed_ops, created_layers)
    232       name = op.name
    233       node_def = op.node_def.SerializeToString()

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in _create_keras_history_helper(tensors, processed_ops, created_layers)
    229               constants[i] = backend.function([], op_input)([])
    230       processed_ops, created_layers = _create_keras_history_helper(
--> 231           layer_inputs, processed_ops, created_layers)
    232       name = op.name
    233       node_def = op.node_def.SerializeToString()

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in _create_keras_history_helper(tensors, processed_ops, created_layers)
    229               constants[i] = backend.function([], op_input)([])
    230       processed_ops, created_layers = _create_keras_history_helper(
--> 231           layer_inputs, processed_ops, created_layers)
    232       name = op.name
    233       node_def = op.node_def.SerializeToString()

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in _create_keras_history_helper(tensors, processed_ops, created_layers)
    229               constants[i] = backend.function([], op_input)([])
    230       processed_ops, created_layers = _create_keras_history_helper(
--> 231           layer_inputs, processed_ops, created_layers)
    232       name = op.name
    233       node_def = op.node_def.SerializeToString()

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in _create_keras_history_helper(tensors, processed_ops, created_layers)
    229               constants[i] = backend.function([], op_input)([])
    230       processed_ops, created_layers = _create_keras_history_helper(
--> 231           layer_inputs, processed_ops, created_layers)
    232       name = op.name
    233       node_def = op.node_def.SerializeToString()

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in _create_keras_history_helper(tensors, processed_ops, created_layers)
    229               constants[i] = backend.function([], op_input)([])
    230       processed_ops, created_layers = _create_keras_history_helper(
--> 231           layer_inputs, processed_ops, created_layers)
    232       name = op.name
    233       node_def = op.node_def.SerializeToString()

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in _create_keras_history_helper(tensors, processed_ops, created_layers)
    229               constants[i] = backend.function([], op_input)([])
    230       processed_ops, created_layers = _create_keras_history_helper(
--> 231           layer_inputs, processed_ops, created_layers)
    232       name = op.name
    233       node_def = op.node_def.SerializeToString()

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer_utils.py in _create_keras_history_helper(tensors, processed_ops, created_layers)
    227           else:
    228             with ops.init_scope():
--> 229               constants[i] = backend.function([], op_input)([])
    230       processed_ops, created_layers = _create_keras_history_helper(
    231           layer_inputs, processed_ops, created_layers)

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/backend.py in __call__(self, inputs)
   3642     return nest.pack_sequence_as(
   3643         self._outputs_structure,
-> 3644         [x._numpy() for x in outputs],  # pylint: disable=protected-access
   3645         expand_composites=True)
   3646

/databricks/python/lib/python3.7/site-packages/tensorflow_core/python/keras/backend.py in <listcomp>(.0)
   3642     return nest.pack_sequence_as(
   3643         self._outputs_structure,
-> 3644         [x._numpy() for x in outputs],  # pylint: disable=protected-access
   3645         expand_composites=True)
   3646

ValueError: Cannot convert a Tensor of dtype resource to a NumPy array.

Creating the dense_tensor with tf.compat.v1.feature_column.input_layer fails later when used.
The following code runs successfully.
column_tensors = {}
dense_tensor = tf.compat.v1.feature_column.input_layer(
  inputs,
  feature_columns,
  cols_to_output_tensors=column_tensors
)

However, the same failure exists when the dense_tensor is used:
output_1 = keras.layers.Dense(64, activation='relu')(dense_tensor)

Can feature_columns be used with Keras models?

Could you please refer tho this link , in case you are still unable to resolve the issue,
Could you please provide us with simple standalone code to reproduce the issue in our environment, Thanks

@Saduf2019 , I am a bit confused.  The referenced document is from 2018.  It refers to tensorflow.python.feature_column.feature_column_v2.FeatureLayer, which I don't think exists anymore.  A more recent document at https://www.tensorflow.org/tutorials/structured_data/feature_columns#create_a_feature_layer refers to tf.keras.layers.DenseFeatures, which I use.
For standalone code, I created a Google Colab at https://colab.research.google.com/drive/1DVfpU6EoEz5vkMqLxH7JTXHALpa3VoOs .  Is that what you were looking for?
Also, I believe it is important to note that this doesn't work on TensorFlow 2 and TensorFlow 1.  I'll try to add the tag.

@jeisinge I ran with tf-nightly and it did throw only deprecation warning and clearly mentioned to use new feature_column API. For example changing tf.feature_column.numeric_column to feature_column.numeric_column. Before that we need to import feature_column API as from tensorflow import feature_column. Please check the tutorial on TF website.
Here is gist for y/our reference. Thanks!
Please close the issue if it was resolved for you. Thanks!

@jvishnuvardhan , I am not an expert in Python.  What is the difference between:
import tensorflow as tf

source_column = tf.feature_column.numeric_column("source_name")
...

and
from tensorflow import feature_column

source_column = feature_column.numeric_column("source_name")
...

?
Are you saying that this defect is fixed in the nightly build?  If so, were you able to reproduce in TF 2.1 or TF 1.15?

I attempted to install tf-nightly in the colar with !pip install tf-nightly.  It also failed.
Also, I did try the from tensorflow import feature_column.  That failed as well.

@jeisinge did you try running my colab gist? Thanks!

Yeah. It fails.
On Thu, Feb 20, 2020, 11:58 AM Vishnuvardhan Janapati <
notifications@github.com> wrote:

@jeisinge https://github.com/jeisinge did you try running my colab
gist? Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/36424?email_source=notifications&email_token=ACDUX2IORQSOELGNCEBTPMTRD3AC5A5CNFSM4KO3C5K2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMPOARI#issuecomment-589226053,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACDUX2OEXALJPVVFT6WIYNDRD3AC5ANCNFSM4KO3C5KQ
.

@jeisinge It is strange. my Colab gist should run without an issue. Can you please share the error you are facing? Are you running my gist as it is or did you modify anything? Are you running it in Chrome or any other browser? Still not sure what is the root-cause. Thanks!

I reran the gist and it succeeded.  However, it appears to be on TF 1.15 due to a misspelling on install.   After upgrading to TF 2 or TF Nightly, it appears to fail on a rerun with the same error:
ValueError                                Traceback (most recent call last)

<ipython-input-6-bed319bac6ce> in <module>()
      1 dense_features = keras.layers.DenseFeatures(feature_columns)
      2 column_tensors = {}
----> 3 dense_tensor = dense_features(inputs, cols_to_output_tensors=column_tensors)

10 frames

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer_utils.py in _create_keras_history_helper(tensors, processed_ops, created_layers)
    220             'Sparse ops are not supported with functional models with built-in '
    221             'layer wrapping. Please wrap the sparse ops in a Lambda layer like'
--> 222             ': \n{lambda_example}\n'.format(lambda_example=lambda_example))
    223
    224       # Recursively set `_keras_history`.

ValueError: Sparse ops are not supported with functional models with built-in layer wrapping. Please wrap the sparse ops in a Lambda layer like:

        weights_mult = lambda x: tf.sparse.sparse_dense_matmul(x, weights)
        output = tf.keras.layers.Lambda(weights_mult)(input)

This might be due to restarting the Python Interpereter.  I believe you can see the gist at https://colab.research.google.com/gist/jeisinge/90718acccf943c16b61ec80a2ff3cb66/tensorflow-feature-columns-fail-for-keras-model-functional-api.ipynb .
In general, I had to run cell 1.  Click the Restart button that appears, rerun everything.

@jeisinge  Could you please check with Nightly version and let us know if the issue still persists.Thanks!

I just tested on TF 2.3 and TF Nightly.  TF 2.3 fails, but TF Nightly works.  Also, the resulting model can be serialized.  Thank you!

authors: Saduf2019 jeisinge jvishnuvardhan saikumarchalla


text:
Official tutorial (Text classification with an RNN )code running error !!!

In the official text tutorial of tensorflow: Text classification with an RNN (https://tensorflow.google.cn/tutorials/text/text_classification_rnn),
train_dataset = train_dataset.padded_batch(BATCH_SIZE)
test_dataset = test_dataset.padded_batch(BATCH_SIZE)

When I run the code there is an error:
    train_dataset = train_dataset.padded_batch(BATCH_SIZE)
TypeError: padded_batch() missing 1 required positional argument: 'padded_shapes'

My tensorflow version: 2.1.0rc2
My python version: 3.7
What should be the correct answer？

@AITutorials,
I was able to run the code from the above tutorial without any issues. Please find the gist of it here. Thanks!

Any updates regarding this issue? Thanks!

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

You need tensorflow version 2.2.0 to make the code work.
Since 2.2.0, the padded_shapes argument is not mandatory.
You can see a start marked note here https://tensorflow.google.cn/tutorials/text/text_classification_rnn

I have the same problem today.

authors: AITutorials PratsBhatt amahendrakar mathslingo


text:
ImportError: cannot import name 'tf2' which type of error is that                                          tensorflow version=1.12 and python version=3.7

Please go to Stack Overflow for help and support:
https://stackoverflow.com/questions/tagged/tensorflow
If you open a GitHub issue, here is our policy:

It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).
The form below must be filled out.
It shouldn't be a TensorBoard issue. Those go here.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary):
TensorFlow version (use command below):
Python version:
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:
Exact command to reproduce:

You can collect some of this information using our environment capture script:
https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh
You can obtain the TensorFlow version with:
python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the problem
Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.
Source code / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

TF 1.12 is too old. Recommend switching to 1.15, 2.0, 2.1 or 2.2 (1.15 or 2.2 are the best)
You have not filled the issue template at all. Hence, closing the issue. Please open a new one if you still have issues on the recommended versions and please flll in issue template when doing so.

TF 1.12 is too old. Recommend switching to 1.15, 2.0, 2.1 or 2.2 (1.15 or 2.2 are the best)
You have not filled the issue template at all. Hence, closing the issue. Please open a new one if you still have issues on the recommended versions and please flll in issue template when doing so.

Hi, I'm having the same error even though i'm using tf 1.15... I just can't find any solution...
I opened a new issue: https://github.com/tensorflow/models/issues/8572

authors: mihaimaruseac sohartma suraj9741


text:
How can I enable aws during running configure?

@tensorflower-gardener @gunan
Environment:
Ubuntu 16.04
python 3.6
bazel 0.26.0
I built tensorflow 1.14 from source. I want to enable aws, but I did not find anyway.
How can I enable it?

I built tensorflow 1.14 from source. I want to enable aws, but I did not find anyway.
How can I enable it?

@ibrahimLearning,
Could you please elaborate the issue you are facing or the use case you're trying to implement. Thanks!

Unfortunately 1.14 was not properly released. Can you attempt 1.15 instead?
Also, what do you mean by enabling AWS? S3 filesystem?

what do you mean by enabling AWS? S3 filesystem?

I just want this:
Do you wish to build TensorFlow with Amazon AWS Platform support? [Y/n]: Y
Amazon AWS Platform support will be enabled for TensorFlow.
but I did not find this, during TF1.14 installation.

The exact question as specified in that snippet has been introduced in tensorlfow==1.10:
https://github.com/tensorflow/tensorflow/blob/4dcfddc5d12018a5a0fdca652b9221ed95e9eb23/configure.py#L1468-L1469
And got removed by the time tensorflow==1.12 got released.
They got removed in https://github.com/tensorflow/tensorflow/commit/d56c298f1ef14b5a738e1e0b7bbc66fcd736be3e#diff-ade1d3e4b7c35655f854151d899df62b and since then these filesystems are enabled by default if supported by the operating system.

Closing issue, as AWS is now default enabled, except on windows.
On windows, I don't think our AWS or GCP support ever worked.

authors: amahendrakar gunan ibrahimLearning mihaimaruseac