text:
Unable to convert MRCNN model to .tflite model

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS High Sierra 10.13.4
TensorFlow installed from (source or binary): source
TensorFlow version (use command below):
python3 -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" /anaconda3/lib/python3.5/site-packages/h5py/__init__.py:34: FutureWarning: Conversion of the second argument of issubdtype from floattonp.floatingis deprecated. In future, it will be treated asnp.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters b'v1.7.0-1844-ga0edcf60f7' 1.7.0
Python version: 3.5
Bazel version (if compiling from source):
bazel version Build label: 0.10.0-homebrew Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar Build time: Wed Jan 10 02:02:06 +50057 (1517480013726) Build timestamp: 1517480013726 Build timestamp as int: 1517480013726
GCC/Compiler version (if compiling from source):
gcc --version Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk/usr/include/c++/4.2.1 Apple LLVM version 9.0.0 (clang-900.0.39.2) Target: x86_64-apple-darwin17.5.0 Thread model: posix InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
CUDA/cuDNN version:
N/A
GPU model and memory:
N/A
Exact command to reproduce:
bazel-bin/tensorflow/contrib/lite/toco/toco   --input_file=./mobile_mrcnn.pb --input_format=TENSORFLOW_GRAPHDEF  --output_format=TFLITE --output_file=/tmp/mobilenet_v1_1.0_224.tflite --inference_type=FLOAT --input_arrays=input_image,input_image_meta --output_arrays=output_node0,output_node1,output_node2,output_node3,output_node4,output_node5,output_node6 --input_shapes=1,224,224,3:1,89

You can collect some of this information using our environment capture script:
https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh
You can obtain the TensorFlow version with
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
Describe the problem
When trying to convert an MRCNN from a frozen graph (.pb) file to (.tflite) using the tensorflow toco script, I get an Abort trap: 6 error with no explanation. Any advice on how to debug/add unsupported ops/functionality or just what went wrong would be great.
Source code / logs
Error:
bazel-bin/tensorflow/contrib/lite/toco/toco   --input_file=./mobile_mrcnn.pb --input_format=TENSORFLOW_GRAPHDEF  --output_format=TFLITE --output_file=/tmp/mobilenet_v1_1.0_224.tflite --inference_type=FLOAT --input_arrays=input_image,input_image_meta --output_arrays=output_node0,output_node1,output_node2,output_node3,output_node4,output_node5,output_node6 --input_shapes=1,224,224,3:1,89  2018-04-15 15:48:31.302680: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: ResizeNearestNeighbor 2018-04-15 15:48:31.303544: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: ResizeNearestNeighbor 2018-04-15 15:48:31.303972: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: ResizeNearestNeighbor 2018-04-15 15:48:31.319684: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: NonMaxSuppression 2018-04-15 15:48:31.319847: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Round 2018-04-15 15:48:31.319897: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Equal 2018-04-15 15:48:31.319908: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Where 2018-04-15 15:48:31.319921: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: GatherNd 2018-04-15 15:48:31.319966: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: CropAndResize 2018-04-15 15:48:31.319983: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Equal 2018-04-15 15:48:31.319993: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Where 2018-04-15 15:48:31.320006: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: GatherNd 2018-04-15 15:48:31.320048: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: CropAndResize 2018-04-15 15:48:31.320065: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Equal 2018-04-15 15:48:31.320074: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Where 2018-04-15 15:48:31.320087: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: GatherNd 2018-04-15 15:48:31.320131: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: CropAndResize 2018-04-15 15:48:31.320149: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Equal 2018-04-15 15:48:31.320159: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Where 2018-04-15 15:48:31.320180: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: GatherNd 2018-04-15 15:48:31.320370: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: CropAndResize 2018-04-15 15:48:31.416940: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: GatherNd 2018-04-15 15:48:31.416977: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: GatherNd 2018-04-15 15:48:31.417344: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Rint 2018-04-15 15:48:31.417372: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Where 2018-04-15 15:48:31.417419: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Unique 2018-04-15 15:48:31.417470: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: TensorArrayV3 2018-04-15 15:48:31.417531: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: TensorArrayScatterV3 2018-04-15 15:48:31.417551: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: TensorArrayV3 2018-04-15 15:48:31.417578: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Enter 2018-04-15 15:48:31.417590: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Enter 2018-04-15 15:48:31.417621: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Enter 2018-04-15 15:48:31.417635: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: LoopCond 2018-04-15 15:48:31.417673: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Enter 2018-04-15 15:48:31.417685: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Enter 2018-04-15 15:48:31.417709: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: TensorArrayReadV3 2018-04-15 15:48:31.417719: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Enter 2018-04-15 15:48:31.417729: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Equal 2018-04-15 15:48:31.417738: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Where 2018-04-15 15:48:31.417785: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Enter 2018-04-15 15:48:31.417807: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Enter 2018-04-15 15:48:31.417838: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: NonMaxSuppression 2018-04-15 15:48:31.417853: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Enter 2018-04-15 15:48:31.417942: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: PadV2 2018-04-15 15:48:31.417955: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Enter 2018-04-15 15:48:31.417969: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: TensorArrayWriteV3 2018-04-15 15:48:31.418017: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Exit 2018-04-15 15:48:31.418028: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: TensorArraySizeV3 2018-04-15 15:48:31.418069: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: TensorArrayGatherV3 2018-04-15 15:48:31.418957: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Where 2018-04-15 15:48:31.419022: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: DenseToDenseSetOperation 2018-04-15 15:48:31.419054: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: SparseToDense 2018-04-15 15:48:31.419385: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Round 2018-04-15 15:48:31.419425: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Equal 2018-04-15 15:48:31.419434: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Where 2018-04-15 15:48:31.419445: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: GatherNd 2018-04-15 15:48:31.419498: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: CropAndResize 2018-04-15 15:48:31.419525: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Equal 2018-04-15 15:48:31.419545: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Where 2018-04-15 15:48:31.419557: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: GatherNd 2018-04-15 15:48:31.419597: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: CropAndResize 2018-04-15 15:48:31.419612: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Equal 2018-04-15 15:48:31.419620: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Where 2018-04-15 15:48:31.419654: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: GatherNd 2018-04-15 15:48:31.419725: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: CropAndResize 2018-04-15 15:48:31.419756: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Equal 2018-04-15 15:48:31.419766: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: Where 2018-04-15 15:48:31.419793: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: GatherNd 2018-04-15 15:48:31.419850: I tensorflow/contrib/lite/toco/import_tensorflow.cc:1268] Converting unsupported operation: CropAndResize 2018-04-15 15:48:31.445467: F tensorflow/contrib/lite/toco/tooling_util.cc:822] Check failed: d >= 1 (0 vs. 1) Abort trap: 6
Source:
https://github.com/matterport/Mask_RCNN/blob/master/mrcnn/model.py

Hi @asimshankar,
It seems to be failing at CHECK_GE(d, 1);
in
`// Apply checks to arrays individually (for-each fashion).
//
// Check consistency of array fields, check name.
void CheckEachArray(const Model& model) {
for (const auto& array_entry : model.GetArrayMap()) {
const auto& array = array_entry.second;
if (array->has_shape()) {
for (int d : array->shape().dims()) {
CHECK_GE(d, 1);
}
}
// It's OK to have a buffer or an alloc, but not both.
// (Since allocs are for transient arrays without a buffer).
CHECK(!array->buffer || !array->alloc);
if (array->buffer) {
// If there is a buffer, its type should be consistent with data_type.
CHECK(array->buffer->type == array->data_type);
// The presence of a fixed buffer should imply the presence of a fixed
// shape.
CHECK(array->has_shape());
// The shape flat-size should agree with the buffer length.
CHECK_EQ(array->buffer->Length(),
RequiredBufferSizeForShape(array->shape()));
}
// Check name.  Either "name_with_suffix_8", "name_with_port:3", but not
// "name_with_both:3_8".
const string& name = array_entry.first;
auto colon_pos = name.find_first_of(":");
if (colon_pos != string::npos) {
  CHECK_EQ(name.substr(colon_pos + 1).find_first_not_of("0123456789"),
           string::npos)
      << "Array name must only have digits after colon";
}
CHECK_GT(colon_pos, 0)
    << "First character of array name must not be a colon.";

}
}`
Is the error produced when converting this instance of the CropAndResize operation? Also, are there any tools you recommend for narrowing down the error, ie. valgrind or some toco specific util?

Using the order of operations, I'm pretty sure I've been able to narrow down to which crop_and_resize operation is causing the issue (assuming it is that function), and have seen that it might be the 4th tensor param that has 0 dim. Any suggestions on how to debugs? @andrehentz @aselle @asimshankar @suharshs

After some digging, I found that it was due to
 AllocateTransientArrays(model, kDefaultTransientDataAlignment);
from
if (SupportsPreallocatedWorkspace(output_format)) {
   AllocateTransientArrays(model, kDefaultTransientDataAlignment);
   LogDump(kLogLevelModelChanged, "AFTER ALLOCATION", *model);
}

in the void Transform(const TocoFlags& toco_flags, Model* model) call in ToolMain
and not from
Import(toco_flags, model_flags, graph_def_contents); in ToolMain.
Here's my output:
... dimension 0size: 1 dimension 0size: 4 2018-05-22 21:55:46.128895: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before dequantization graph transformations: 564 operators, 1155 arrays (0 quantized) from Transform from AllocateTransientArrays allocating mrcnn_detection/map/while/NextIteration from AllocateTransientArray from TransientArraySize dimension 0size: 1 dimension 1size: 1 dimension 2size: 1 dimension 3size: 0 2018-05-22 21:55:46.141249: F tensorflow/contrib/lite/toco/tooling_util.cc:664] Check failed: dim >= 1 (0 vs. 1) Abort trap: 6
Is this expected with the while op since it is not implemented for lite yet and is it alright if I comment out the SupportsPreallocatedWorkspace(output_format) conditional block?

Nagging Assignees @aselle, @suharshs: It has been 29 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@Cpruce
Hi Do you find any solution for this specific issues? it seems like I have a similar issue.
F tensorflow/contrib/lite/toco/tooling_util.cc:664] Check failed: dim >= 1 (0 vs. 1)

@offthewallace
My workaround was to comment out the SupportsPreallocatedWorkspace(output_format) if clause since this is likely an unintentional fault. However, I haven't continued further since my model would need several ops implemented for lite.

I apologize, but your link to the source code is now broken. Could you fix it, and I will take a look.

Hi, I have similar problem with SSD_MobileNet trained with tensorflow object detection API.

@aselle https://github.com/matterport/Mask_RCNN/blob/master/mrcnn/model.py

Nagging Assignees @aselle, @suharshs: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Thanks, I'll take a look when I can.

Hi, I have same problem with SSD_MobileNet trained with tensorflow object detection API.

I too am seeing this with ssd_mobilenet in tf od api. Same message, different location

Nagging Assignees @aselle, @suharshs: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignees @aselle, @suharshs: It has been 29 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignees @aselle, @suharshs: It has been 44 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignees @aselle, @suharshs: It has been 59 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

+1 to this question "Is this expected with the while op since it is not implemented for lite yet and is it alright if I comment out the SupportsPreallocatedWorkspace(output_format) conditional block?" I'm running into the same thing with a custom model. Thanks!

Nagging Assignees @aselle, @suharshs: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

This might be because "while" is being used.
What is happening is that the conversion is trying to calculate the size of a given tensor based on its shape, which is [1, 1, 1, 0]. That means the tensor has no elements and the memory planning code doesn't like that. It is possible that shape inference in the presence of "while" yields this type of zero-element tensor.
We are working to get some form of flow control working in TF Lite. Until then I'm afraid it will be quite hard to convert models that contain "while" ops.

Nagging Assignees @aselle, @suharshs: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

For CropAndResize op or other unsupported ops, please look at this reference
As we work toward fleshing out the builtin op library for TensorFlow Lite, we've been working on an experimental feature that allows using select TensorFlow ops from within the TensorFlow Lite runtime. The goal is to help reduce some of the friction for using models that rely on ops not yet natively supported by TensorFlow Lite (at the cost of increased binary size). This feature requires opting in during model conversion, as well as adding an additional dependency. More details can be found here.
Feedback is very much appreciated (either via GitHub or directly via tflite@tensorflow.org), and we'll be adding and refining functionality over the coming weeks. Cheers.

Thanks, @achowdhery . Did you mean for "here" in "More details can be found here." to be a link we can look at?

Closing due to lack of activity. There has been significant progress on improving conversion support over the past year.

authors: Cpruce MohammadMoradi achowdhery andrehentz aselle jdduke lauriebyrum offthewallace tensorflowbutler xiaofansong


text:
Profiling Tensorflow C++ with nvidia profiler/tf.Profiler

Please go to Stack Overflow for help and support:
https://stackoverflow.com/questions/tagged/tensorflow
If you open a GitHub issue, here is our policy:

It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).
The form below must be filled out.
It shouldn't be a TensorBoard issue. Those go here.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: n/a
TensorFlow installed from (source or binary): bazel
TensorFlow version (use command below): 1.8
Python version:2.7
Bazel version (if compiling from source): 0.14.1
GCC/Compiler version (if compiling from source): 5.4.0 20160609
CUDA/cuDNN version: 9/7.1
GPU model and memory: 1080ti/11gb dual
Exact command to reproduce:
nvprof bazel-bin/tensorflow/examples/label_image/label_image from Link
works, but when i run nvvp it throws me an error, that application profiled cannot be found. I am looking at an TF alternative tf.Profiler , but it doesn't have any documentation for C++ api

You can collect some of this information using our environment capture script:
https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh
You can obtain the TensorFlow version with
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
Describe the problem
Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.
Source code / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

Nagging Assignee @shivaniag: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Switched to Timeline.

source: gharchive/issue

authors: dhingratul tensorflowbutler

repo: tensorflow/tensorflow

license_type: permissive

license_source: github-api


text:
ValueError: Op type not registered 'NcclAllReduce' in binary running on 35712d892b7a.

Please go to Stack Overflow for help and support:
https://stackoverflow.com/questions/tagged/tensorflow
If you open a GitHub issue, here is our policy:

It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).
The form below must be filled out.
It shouldn't be a TensorBoard issue. Those go here.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes.
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: No
TensorFlow installed from (source or binary): Binary
TensorFlow version (use command below): v1.10.0-0-g656e7a2b34 1.10.0
Python version: 3.5
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: based on nvidia-docker tensorflow/tensorflow:1.10.0-gpu-py3 image.
GPU model and memory: Titan Xp 11GB 2 pcs.
Exact command to reproduce: gen_nccl_ops.nccl_all_reduce( ... )

You can collect some of this information using our environment capture script:
https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh
You can obtain the TensorFlow version with
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
Describe the problem
Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.
Hi, I'm trying to implement multi-gpu tensorflow code.
I am trying to use gen_nccl_ops.nccl_all_reduce to sum values from each GPU, however, the code raises ValueError written below.
ValueError: Op type not registered 'NcclAllReduce' in binary running on 35712d892b7a. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) tf.contrib.resamplershould be done before importing the graph, as contrib ops are lazily registered when the module is first accessed. while building NodeDef 'Tower_0/Network/NcclAllReduce'
As suggested by the error message, I tried to access tf.contrib.nccl at the beginning of the whole code files.
import tensorflow as tf
tf.contrib.nccl

# ... Code continues ... #

But it doesn't work!
I checked NCCL is installed using
export NCCL_DEBUG=VERSION
, and got NCCL version 2.2.13+cuda9.2.
I'm using nvidia-docker image tensorflow/tensorflow:1.10.0-gpu-py3.
I confirmed nccl.all_sum() is working.
Is there anyway to import NCCL library before graph establishment?
Thanks.
Source code / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.
# Gradient Penalty on real distribution
self.gradient = tf.gradients(tf.reduce_sum(self.D[real][logits]), [self.x])[0]
self.slope_square = tf.reduce_sum(tf.square(self.gradient), reduction_indices=[1, 2, 3])

if self.tower_config is None:
    penalty = tf.reduce_mean(self.slope_square)
    self.gradient_penalty = self.C_gradient * penalty
else:
    # Utilize NCCL
    shared_name = tf.get_variable_scope().name.\
        replace(self.tower_config.name, self.tower_config.prefix.format("NCCL"))
    penalty = gen_nccl_ops.nccl_all_reduce(
        input=self.slope_square,
        reduction="sum",
        num_devices=self.tower_config.num_devices,
        shared_name=shared_name
    ) / (1.0 / self.tower_config.num_devices)
    self.gradient_penalty = self.C_gradient * penalty

Try:
sudo apt install libnccl2=2.2.13-1+cuda9.0  libnccl-dev=2.2.13-1+cuda9.0
Tensorflow is built with CUDA 9.0 and you have the NCCL for CUDA 9.2 installed

Hi,
Thanks for the solution.
I tried to apply your solution, it doesn't work.
I removed current container and I reinstalled tensorflow/tensorflow:1.10.0-gpu-py3 image.
Right after installation, I only can find libnccl2 as shown below and code was not working.
root@28838711efd4:/notebooks# apt list | grep nccl

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libnccl2/now 2.2.13-1+cuda9.0 amd64 [installed,local]

So, I tried to install libnccl-dev as follows.
root@28838711efd4:/notebooks# apt update
root@28838711efd4:/notebooks# apt install libnccl-dev=2.2.13-1+cuda9.0
root@28838711efd4:/notebooks# apt list | grep nccl

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libhttpasyncclient-java/xenial 4.1-1 all
libnccl-dev/unknown 2.2.13-1+cuda9.2 amd64 [upgradable from: 2.2.13-1+cuda9.0]
libnccl1/unknown 1.2.3-1+cuda8.0 amd64
libnccl2/unknown 2.2.13-1+cuda9.2 amd64 [upgradable from: 2.2.13-1+cuda9.0]
libvncclient1/xenial-updates,xenial-security 0.9.10+dfsg-3ubuntu0.16.04.2 amd64
libvncclient1-dbg/xenial-updates,xenial-security 0.9.10+dfsg-3ubuntu0.16.04.2 amd64
python-ncclient/xenial 0.4.7-1 all

However, same problem.
Did I installed libnccl2 and libnccl-dev properly?

Hi,
Thanks for the solution.
I tried to apply your solution, it doesn't work.
I removed current container and I reinstalled tensorflow/tensorflow:1.10.0-gpu-py3 image.
Right after installation, I only can find libnccl2 as shown below and code was not working.
root@28838711efd4:/notebooks# apt list | grep nccl

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libnccl2/now 2.2.13-1+cuda9.0 amd64 [installed,local]

So, I tried to install libnccl-dev as follows.
root@28838711efd4:/notebooks# apt update
root@28838711efd4:/notebooks# apt install libnccl-dev=2.2.13-1+cuda9.0
root@28838711efd4:/notebooks# apt list | grep nccl

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libhttpasyncclient-java/xenial 4.1-1 all
libnccl-dev/unknown 2.2.13-1+cuda9.2 amd64 [upgradable from: 2.2.13-1+cuda9.0]
libnccl1/unknown 1.2.3-1+cuda8.0 amd64
libnccl2/unknown 2.2.13-1+cuda9.2 amd64 [upgradable from: 2.2.13-1+cuda9.0]
libvncclient1/xenial-updates,xenial-security 0.9.10+dfsg-3ubuntu0.16.04.2 amd64
libvncclient1-dbg/xenial-updates,xenial-security 0.9.10+dfsg-3ubuntu0.16.04.2 amd64
python-ncclient/xenial 0.4.7-1 all

However, same problem.
Did I installed libnccl2 and libnccl-dev properly?

Test

Try:
sudo apt install libnccl2=2.2.13-1+cuda9.0 libnccl-dev=2.2.13-1+cuda9.0
Tensorflow is built with CUDA 9.0 and you have the NCCL for CUDA 9.2 installed

Hi,
Thanks for the solution.
I tried to apply your solution, it doesn't work.
I removed current container and I reinstalled tensorflow/tensorflow:1.10.0-gpu-py3 image.
Right after installation, I only can find libnccl2 as shown below and code was not working.
root@28838711efd4:/notebooks# apt list | grep nccl

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libnccl2/now 2.2.13-1+cuda9.0 amd64 [installed,local]

So, I tried to install libnccl-dev as follows.
root@28838711efd4:/notebooks# apt update
root@28838711efd4:/notebooks# apt install libnccl-dev=2.2.13-1+cuda9.0
root@28838711efd4:/notebooks# apt list | grep nccl

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libhttpasyncclient-java/xenial 4.1-1 all
libnccl-dev/unknown 2.2.13-1+cuda9.2 amd64 [upgradable from: 2.2.13-1+cuda9.0]
libnccl1/unknown 1.2.3-1+cuda8.0 amd64
libnccl2/unknown 2.2.13-1+cuda9.2 amd64 [upgradable from: 2.2.13-1+cuda9.0]
libvncclient1/xenial-updates,xenial-security 0.9.10+dfsg-3ubuntu0.16.04.2 amd64
libvncclient1-dbg/xenial-updates,xenial-security 0.9.10+dfsg-3ubuntu0.16.04.2 amd64
python-ncclient/xenial 0.4.7-1 all

However, same problem.
Did I installed libnccl2 and libnccl-dev properly?

Have you solved this problem? I have the same problem。

Not yet.. Still waiting for response.

If you solve it, I hope you can make the solution public. If I solve it, I will tell you. 0.0

If you solve it, I hope you can make the solution public. If I solve it, I will tell you. 0.0

Sure. I'll post the solution as soon as I get the answer.

hey，bro：
maybe you can try to add this code:
from tensorflow.contrib.nccl.python.ops import nccl_ops
nccl_ops._maybe_load_nccl_ops_so()

hey bro:
Maybe you can try this code：
from tensorflow.contrib.nccl.python.ops import nccl_ops
nccl_ops._maybe_load_nccl_ops_so()

hey bro:
Maybe you can try this code：
from tensorflow.contrib.nccl.python.ops import nccl_ops
nccl_ops._maybe_load_nccl_ops_so()

Okay, your solution works for importing NCCL library. Thanks!
However, after building network, sess.run() doesn't give any response and just stay in that line. I think some of codes are in infinite loop or some of GPU devices are not returning value.
I'll notice you after I fix all the things!

Looks like the original issue was resolved by importing the nccl_ops library.  Closing this for now, please re-open if the issue persists.

I am facing the same problem, did you solve the problem? What is the final solution? @skang29


text:
A bug of tfdbg : Non-OK-status: env->NewWritableFile(file_path, &f) status: Not found: Failed to create a NewWriteableFile:

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Win10
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:no
TensorFlow installed from (source or binary):pip install tensorflow
TensorFlow version (use command below):1.10.0
Python version: 3.6.5
Bazel version (if compiling from source):no
GCC/Compiler version (if compiling from source):no
CUDA/cuDNN version:no
GPU model and memory:no

When I use tfdbg, I input "run" in the console, the error info  like this:
2018-11-09 17:07:18.381547: F T:\src\github\tensorflow\tensorflow\core\debug\debug_io_utils.cc:621] Non-OK-status: env->NewWritableFile(file_path, &f) status: Not found: Failed to create a NewWriteableFile: C:\Users\winter\AppData\Local\Temp\tfdbg_312ol3zm/tfdbg_device,job_localhost,replica_0,task_0,device_CPU_0/train_step/gradients/bi-lstm/bidirectional_rnn/fw/fw/while/TensorArrayWrite/TensorArrayWriteV3_grad/TensorArrayReadV3/Const_0_DebugIdentity_1541754438373045 : ϵͳ�Ҳ���ָ����·����
; No such process
Process finished with exit code -1073740791 (0xC0000409)

Perhaps I know the reason， the path and file name is too long . It beyonds the windows 10 limit. So it cannot create .

Perhaps I know the reason， the path and file name is too long . It beyonds the windows 10 limit. So it cannot create .

Feel free to close if you find the reason. Or please share sample code snippet which helps us to reproduce the error and look into the issue.

Closing this as a root cause is found.

Although there is a problem with Windows not allowing to create a folder or file with such a large name, it should be addressed in Tensorflow, since it is nothing that we can do to resolve it.

Although there is a problem with Windows not allowing to create a folder or file with such a large name, it should be addressed in Tensorflow, since it is nothing that we can do to resolve it.

However, I found out someone has a workaround on Windows side here. But it is still nice to have a solution on Tensorflow side.

authors: SilenceWinter harshini-gadige mmkamani7


text:
java.lang.RuntimeException: Native TF methods not found; check that the correct native libraries are present in the APK.

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:NA
TensorFlow installed from (source or binary): source
TensorFlow version:1.6
Python version:3.6
Installed using virtualenv? pip? conda?:pip
Bazel version (if compiling from source): 0.16.1
GCC/Compiler version (if compiling from source): -
CUDA/cuDNN version: -
GPU model and memory: -

Describe the problem
Provide the exact sequence of commands / steps that you executed before running into the problem
bazel build -c opt //tensorflow/examples/android:libtensorflow_demo.so --crosstool_top=//external:android/crosstool --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --cpu=armeabi-v7a --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0"
Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

Hello, can you please try bazel 0.9.0  ?

authors: msymp sthuthyraj


text:
Tensorflow 2.0: please do not deprecate important functions!!!

According to Tensorflow 2.0, many important functions are deprecated, such as:
tf.layers.dense()
tf.layers.dropout()
tf.layers.flatten()
tf.layers.batch_normalization()
...
Since these functions are very widely used, and also very helpful for building complicted models, would you please keep them in the future versions?

You have to use tf.keras.layers instead of tf.layers to build the models.

there is no dense(), dropout(), etc. to use, no matter keras or layers

You have to use the functional version of the layers in Keras. There are no dropout, dense and so on layers, because there are the keras layers that you can instantiate and then call:
https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/layers
For example:
tf.layers.dropout becomes tf.keras.layers.Dropout(rate)(input)

Yup, @galeone is right.. TensorFlow is trying to standardize on tf.keras as the direction of general model building.
@chwang85 Also checkout https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/nn/dropout

Sorry, I am not asking for "how to write a TF program".
I believe it is ridiculous to deprecate these functional layers.
Keeping them in TF 2.0 is not a burden, and compatible with TF 1.x, why do you insist in deprecating them?

"Deprecated" means they are still there but will go away on a subsequent release (that is, in 2.0).
So, you can still use them in TF 1.x, but they won't be available in TF 2.x and later. The deprecated message is just to notify you and give you time to update.
Closing this and the duplicated #26151 and #26144

Why not deprecate tf.Session()?

It goes away too

In TF 1.13, when tf.layers.dense() is used, there is a "deprecate" warning, but tf.Session() is OK.
By the way, I think it would be better if the difference between TF 1.x and TF 2.x is like that between Python 2.x and Python 3.x

tf.session will be deprecated later, after the community switches to using TF2.x. Until then, it will still be available for some scenarios, since TF2.x has a very abrupt change in the way TF code is executed (from graph mode to eager mode; tf.session is needed for graph mode even in TF2.x).
tf.layers.dense has a 100% functional equivalent. It has been there for a long time and the decision to deprecate it and favor the object oriented approach has been taken a long time ago. There is no reason to keep both available, so that's why you get this warning. Please understand that APIs change and the design choice has been taken a very long time ago and now it is too late to go back. That ship has sailed for too long.

This decision is not reasonable at all, because the new components, tf.keras.layers, still has a lot of problems in combining with other tf components...
tf.keras.layers even does not support variable scope, see #27016 for example

To share variables using Keras layers you have to build a Keras Model - is a different way of organizing the code and to think.
Once you built the model you have to reuse the object. Hence don't share variables using scopes, but share variables using the model.trainable_variables attribute.
While I was migrating my codebase to TF2 I faced your same issues and I wrote a blog post about this: https://pgaleone.eu/tensorflow/keras/2019/01/19/keras-not-yet-interface-to-tensorflow/
I also wrote a second article about how to migrate the models from tf1 to tf2, while still using tf1: https://pgaleone.eu/tensorflow/gan/2018/11/04/tensorflow-2-models-migration-and-new-design/
(therefore there are still sessions that in tf2 are going to be removed)
In practice, TF2 requires to re-learn how to use the framework if you're used to working with the static graph + session execution.

thanks
i have noticed that the change between TF 1.x and TF 2.x is too big to simply modify tf.layers to tf.keras.lyaers
instead, it requiress a totally different way of thinking and designing...
i will stick on the graph + session ways

By the way, before deprecating old components, would you please complete the current functionality? See #27042. It seems that TF is very good at deprecating widely used functions, but poor at improving its weak points

Can someone explain me. If I need to use  tf.keras.layers instead of tf.layers to build the models, and tf.keras.model.compile to build graph, and tf.keras.model.fit to train instead tf.graph and tf.Session.
Then why I need Tensorflow at all? Maybe I should simply use Keras?
@mihaimaruseac @galeone @mandroid6 @ry

Keras is a high level API for TensorFlow, which will be able to solve most
of the model building and training needs. Still it isn't an exhaustive API
which has all basic ops available in TensorFlow.
Most importantly Keras uses TensorFlow for building the acyclic graphs or
now the eager execution context.
All tf.keras.layers or tf.keras.model uses TensorFlow in the backend. You
can very well still directly use the low level TF ops, but why to when you
have a more simple abstraction available.
On Thu, 28 Mar 2019 at 1:40 PM, OgnjenTanovic notifications@github.com
wrote:

Can someone explain me. If I need to use tf.keras.layers instead of
tf.layers to build the models, and tf.keras.model.compile to build graph,
and tf.keras.model.fit to train instead tf.graph and tf.Session.
Then why I need Tensorflow at all? Maybe I should simply use Keras?
@mihaimaruseac https://github.com/mihaimaruseac @galeone
https://github.com/galeone @mandroid6 https://github.com/mandroid6 @ry
https://github.com/ry
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/26844#issuecomment-477487523,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ASrYl4eMUJGFfpsRHYeXiaIb2_IPAc6pks5vbHjigaJpZM4b6T0L
.

When will you deprecate keras?
When will you deprecate Tensorflow?
When will you deprecate Google?

TensorFlow is giving up a lot of its unique features. As a TensorFlow user since 2015, I feel very sorry about that. Keras should never take the place of TensorFlow.

I was just working on https://github.com/sympy/sympy/pull/17103 and was curious to find why deprecation has been done. I googled and reached this issue.
So, well many of my doubts are clear, but using, tf.compat.v1.Session instead of tf.Session is not clear(Is this related to namespace?), I will investigate more on this, though if any one can provide an answer on this, then it will be really helpful.
PS - BTW, if the end result for tf.layers and tf.keras.layers is same, then I think its a one line change,
Create an alias for use, tf = tf.keras and most of the effort for changes can be avoided. This trick can be included somewhere in the README or warnings, probably you might have done that. Well that's just a suggestion, it can be immature, and many of you might have thought it.
Thanks for reading.

I was just working on adapting to some deprecation warnings and was curious to find why deprecation has been done. I googled and reached this issue.
So, well many of my doubts are clear, but using, tf.compat.v1.Session instead of tf.Session is not clear(Is this related to namespace?), I will investigate more on this, though if any one can provide an answer on this, then it will be really helpful.
PS - BTW, if the end result for tf.layers and tf.keras.layers is same, then I think its a one line change,
Create an alias for use, tf = tf.keras and most of the effort for changes can be avoided. This trick can be included somewhere in the README or warnings, probably you might have done that. Well that's just a suggestion, it can be immature, and many of you might have thought it.
Edit - Found https://www.tensorflow.org/beta/guide/migration_guide, nice guide. Though tf.Session -> tf.compat.v1.Session is not clear.
Thanks for reading.

Regarding tf.Session: TF 2.0 is eager by default, in that world Session is useless. It is being moved to compat.v1 to have some backward compatibility: instead of requiring you to rewrite your entire script immediately you can just change an import (or use a script to rename tf.Session into tf.compat.v1.Session) and then rewrite at your convenience (though, at one point you should rewrite).
Regarding tf.layers: tf top level namespace was too huge, making autocomplete and human understanding very hard. We decided to split it into components.
Note that doing tf = tf.keras will result into issues tf.math.sin will be tf.keras.math.sin which will result in AttributeError: Module 'keras' has no attribute 'math'.

@mihaimaruseac Thank you for the clarification. In fact, I read this proposal, many of my doubts are clear. I would say, it's a very good idea to migrate from Sessions -> Functions. Though I am not familiar with the terminology but examples in the proposal made the issue very clear.

It is being moved to compat.v1 to have some backward compatibility: instead of requiring you to rewrite your entire script immediately you can just change an import (or use a script to rename tf.Session into tf.compat.v1.Session) and then rewrite at your convenience (though, at one point you should rewrite).

Ah! I was thinking of that. So, are there any plans to remove compat somewhere in future?

Note that doing tf = tf.keras will result into issues tf.math.sin will be tf.keras.math.sin which will result in AttributeError: Module 'keras' has no attribute 'math'.

I see, doing, tf = tf.keras is really very immature, well I came across an automation script to make the code compatible with tensorflow 2.0 in   https://www.tensorflow.org/beta/guide/migration_guide.

What is the future of frozen graph? Currently there is no way to freeze a graph in TF2.0. Will it be supported in the near future?

@czgdp1807 Yes, at one point (probably nearing TF 3.0 but don't quote me on that as I might be wrong w.r.t timeline) compat.v1 will be removed
The migration guide is really useful and recommended
@leimao I suggest opening a new issue to discuss that, I have no idea to be honest and if there's a new issue an owner of that part of the code can respond. Thanks

@mihaimaruseac I expect that release of tf 3.0 is not very near, right? I expect it at least after 1 year and I hope that no concrete alternative is in the pipeline for compat.v1, if there is, then please let me know, so that I can make updates to my sympy PR to keep that part of the code base stable for a longterm.

Yes, TF 3.0 is going to be far in the future. There are no plans for it at the moment (we still have to actually make a release for TF 2.0, we are currently in beta)

I think the problem here is that people have no clue how to use tf.keras.layers.Dense without building a Keras model.
Even the official document for tf.keras.layers.Dense has only examples using the Keras model. So naturally, people see this and wonder "How do I add the inputs?" They come to the conclusion that tf.keras.layers.Dense cannot be used with pure TensorFlow and then start using tf.layers.dense
If we could add examples of how to use tf.keras.layers.Dense with pure TensorFlow this would solve the issue. I am guessing the general expression is :
output = tf.keras.layers.<LAYER_TYPE>(size, activation, ...)(inputs)

The solution to this problem:
Switch to PyTorch, you will not regret it.
torch.nn.Linear has been there since the v0 and continues to work ever since.
With TF there have been tf.nn, tf.slim, tf.contrib, tf.layers, tf.keras, I wonder what next... Every year they announce a new, final API...
Unless your company forces you to use TF there is no real reason to stay. Really.

How does this affect Tensorflow.js?

authors: DollarAkshay OgnjenTanovic PiotrDabkowski chwang85 czgdp1807 erikchwang galeone leimao mandroid6 mihaimaruseac sbrl


text:
tf.data.Dataset allow change of structure

System information

TensorFlow version (you are using): v2
Are you willing to contribute it (Yes/No): only python code

Describe the feature and the current behavior/state.
My current use case is as follows:
I get a dataset from tensorflow_datasets.load() where the structure is (image, label), e.g. MNIST. I would like to only use the image information of that dataset. Therefore I tried to find a way to "unpack" that dataset from a dict into a single element (so no key required to work on it).
# Currently

dataset = tfds.load('MNIST', split='train')
for d in dataset:
  # Now d has the "shape" {'image': tensor, 'label': tensor}

# What I would like to do
dataset = tfds.load('MNIST', split='train')
dataset = tf.data.Dataset.unpack(dataset, 'image')
for d in dataset:
  # Now d is a tensor

When the dataset is unpacked I can also do further transformations on it, e.g. filter, batch, ...
Will this change the current api? How? No
Who will benefit with this feature? Users of tensorflow_dataset I assume.
Any Other info.

It could also be seen as a kind of subset, where when the subset size is 1 it unpacks the dict to a tensor.
This would allow selecting multiple keys as a subset from a tf.data.Dataset.

Hi @sleighsoft you can use the map transformation to do what you want:
dataset = tfds.load('MNIST', split='train')
dataset = dataset.map(lambda x: x['image'])
# now dataset only contains sequence of images

authors: jsimsa sleighsoft


text:
Readmea for tensorboard not on website

GitHub issues are for bugs / installation problems / feature requests.
For general support from the community, see StackOverflow.
To make bugs and feature requests more easy to find and organize, we close issues that are deemed
out of scope for GitHub Issues and point people to StackOverflow.
For bugs or installation issues, please provide the following information.
The more information you provide, the more easily we will be able to offer
help and advice.
Environment info
Operating System: Browser issue with readme tensorboard
Installed version of CUDA and cuDNN:
(please attach the output of ls -l /path/to/cuda/lib/libcud*):
If installed from binary pip package, provide:

not tensorproblem but website

If installed from sources, provide the commit hash:
Steps to reproduce

Go to https://www.tensorflow.org/versions/r0.9/how_tos/summaries_and_tensorboard/index.html and click on the readme tensorboard. Give you 404 not found error.

What have you tried?

nothing, its on the website

Logs or other output that would be helpful
(If logs are large, please upload as attachment).

Closing as duplicate of #2813


text:
Memory leak with tf.shuffle, doesn't release buffer memory

System information

OS Platform and Distribution Linux Ubuntu 16.04
Python: 2.7.17 |Anaconda, Inc.| (default, Oct 21 2019, 19:04:46) [GCC 7.3.0]
Tensorflow: 1.12.0
Numpy: 1.16.5
GPU: GeForce RTX 2080 Ti
CUDA: 9.2

Describe the current behavior
CPU memory gradually increase after each epoch until the program restarts, i suspect that dataset.shuffle doesn't release the buffer memory. Tested with tf 1.15, same situation.
Code to reproduce the issue
import numpy as np
import tensorflow as tf
class ASRDataGenerator(object):
    def __init__(self,num):
        self.num = num
    def __call__(self):
        for i in range(self.num):
            for j in range(106):
                yield 'a','b',np.random.randn(100,120)

class TFASRDataSet(object):
    def __init__(self,num,batch_size):

        self.num = num
        self.batch_size = batch_size
        self.asrDataGenerator = ASRDataGenerator(num)

    def setDataSetIterator(self):

        dataset = tf.data.Dataset.from_generator(self.asrDataGenerator, (tf.string,tf.string,tf.float32))
        dataset = dataset.shuffle(30000)
        dataset = dataset.map(lambda s1,s2,feat: [s1,s2,feat])
        dataset = dataset.batch(self.batch_size, drop_remainder=True)
        self.iterator = dataset.make_initializable_iterator()

test_tfASRDataSet = TFASRDataSet(248,192)
test_tfASRDataSet.setDataSetIterator()
test_iter = test_tfASRDataSet.iterator
test_next = test_iter.get_next()

run_config = tf.ConfigProto()
run_config.gpu_options.allow_growth = True
run_config.allow_soft_placement = True

with tf.Session(config=run_config) as sess:

    for i in range(100):

        sess.run(test_iter.initializer)

        while True:
            try:
                loss_list = sess.run([test_next])
                print(len(loss_list[0]))
            except tf.errors.OutOfRangeError:
                print("train epoch %d finish" % (i+1))
                break

I could replicate the issue with Tf 1.15.
Please take a look at the gist. Thanks!

@gadagashwini the user program does not provide evidence of the issue, so I am not sure what do you mean by "replicating" the issue.
When I modify the program with tracking of memory use:
...
def view_used_mem():
  used_mem = psutil.virtual_memory().used
  print("used memory: {} Mb".format(used_mem / 1024 / 1024))

def main(argv):
  del argv

  test_tfASRDataSet = TFASRDataSet(248, 192)
  test_tfASRDataSet.setDataSetIterator()
  test_iter = test_tfASRDataSet.iterator
  test_next = test_iter.get_next()

  run_config = tf.ConfigProto()
  run_config.gpu_options.allow_growth = True
  run_config.allow_soft_placement = True

  with tf.Session(config=run_config) as sess:

    for i in range(100):

      sess.run(test_iter.initializer)

      while True:
        try:
          loss_list = sess.run([test_next])
        except tf.errors.OutOfRangeError:
          print('train epoch %d finish' % (i + 1))
          view_used_mem()
          break
...

and run it using internal TF 1.15 build, the memory is not increasing between epochs.
Furthermore, I have confirmed that the shuffle buffer is properly disposed of between epochs by enabling logging of tf.data iterator constructors / destructors:
...
I0102 13:32:09.063783   34490 dataset.h:887] Iterator::Model constructor
I0102 13:32:09.063821   34490 dataset.h:887] Iterator::Model::MapAndBatch constructor
I0102 13:32:09.063834   34490 dataset.h:887] Iterator::Model::MapAndBatch::Shuffle constructor
I0102 13:32:09.064474   34490 dataset.h:891] Iterator::Model::MapAndBatch::Shuffle destructor
I0102 13:32:09.064498   34490 dataset.h:891] Iterator::Model::MapAndBatch destructor
I0102 13:32:09.064547   34490 dataset.h:891] Iterator::Model destructor
I0102 13:32:09.100376   34570 dataset.h:887] Iterator::Model::MapAndBatch::Shuffle::FlatMap constructor
I0102 13:32:09.100427   34570 dataset.h:887] Iterator::Model::MapAndBatch::Shuffle::FlatMap::FromTensor constructor
I0102 13:32:09.101287   34570 dataset.h:887] Iterator::Model::MapAndBatch::Shuffle::FlatMap[0]::Generator constructor
I0102 13:32:19.100455   34570 shuffle_dataset_op.cc:185] Filling up shuffle buffer (this may take a while): 10188 of 30000
I0102 13:32:29.100570   34570 shuffle_dataset_op.cc:185] Filling up shuffle buffer (this may take a while): 20548 of 30000
I0102 13:32:34.942694   34570 dataset.h:891] Iterator::Model::MapAndBatch::Shuffle::FlatMap[0]::Generator destructor
I0102 13:32:34.944426   34570 dataset.h:891] Iterator::Model::MapAndBatch::Shuffle::FlatMap::FromTensor destructor
I0102 13:32:34.944455   34570 dataset.h:887] Iterator::Model::MapAndBatch::Shuffle::FlatMap constructor
I0102 13:32:34.944461   34570 dataset.h:891] Iterator::Model::MapAndBatch::Shuffle::FlatMap destructor
I0102 13:32:34.944477   34570 dataset.h:887] Iterator::Model::MapAndBatch::Shuffle::FlatMap::FromTensor constructor
I0102 13:32:34.946282   34570 dataset.h:891] Iterator::Model::MapAndBatch::Shuffle::FlatMap::FromTensor destructor
I0102 13:32:34.946300   34570 dataset.h:891] Iterator::Model::MapAndBatch::Shuffle::FlatMap destructor
I0102 13:32:34.946307   34570 shuffle_dataset_op.cc:234] Shuffle buffer filled.
train epoch 2 finish
used memory: 17154.9375 Mb
I0102 13:32:36.206103   34490 dataset.h:887] Iterator::Model constructor
I0102 13:32:36.206153   34490 dataset.h:887] Iterator::Model::MapAndBatch constructor
I0102 13:32:36.206179   34490 dataset.h:887] Iterator::Model::MapAndBatch::Shuffle constructor
I0102 13:32:36.206920   34490 dataset.h:891] Iterator::Model::MapAndBatch::Shuffle destructor
I0102 13:32:36.206948   34490 dataset.h:891] Iterator::Model::MapAndBatch destructor
I0102 13:32:36.206960   34490 dataset.h:891] Iterator::Model destructor
I0102 13:32:36.213317   34616 dataset.h:887] Iterator::Model::MapAndBatch::Shuffle::FlatMap constructor
I0102 13:32:36.213364   34616 dataset.h:887] Iterator::Model::MapAndBatch::Shuffle::FlatMap::FromTensor constructor
I0102 13:32:36.214274   34616 dataset.h:887] Iterator::Model::MapAndBatch::Shuffle::FlatMap[0]::Generator constructor
I0102 13:32:46.213390   34616 shuffle_dataset_op.cc:185] Filling up shuffle buffer (this may take a while): 10597 of 30000
I0102 13:32:56.214073   34616 shuffle_dataset_op.cc:185] Filling up shuffle buffer (this may take a while): 20791 of 30000
I0102 13:33:01.736376   34616 dataset.h:891] Iterator::Model::MapAndBatch::Shuffle::FlatMap[0]::Generator destructor
I0102 13:33:01.736460   34616 dataset.h:891] Iterator::Model::MapAndBatch::Shuffle::FlatMap::FromTensor destructor
I0102 13:33:01.736486   34616 dataset.h:887] Iterator::Model::MapAndBatch::Shuffle::FlatMap constructor
I0102 13:33:01.736495   34616 dataset.h:891] Iterator::Model::MapAndBatch::Shuffle::FlatMap destructor
I0102 13:33:01.736521   34616 dataset.h:887] Iterator::Model::MapAndBatch::Shuffle::FlatMap::FromTensor constructor
I0102 13:33:01.736606   34616 dataset.h:891] Iterator::Model::MapAndBatch::Shuffle::FlatMap::FromTensor destructor
I0102 13:33:01.736621   34616 dataset.h:891] Iterator::Model::MapAndBatch::Shuffle::FlatMap destructor
I0102 13:33:01.736632   34616 shuffle_dataset_op.cc:234] Shuffle buffer filled.
train epoch 3 finish
used memory: 17118.4609375 Mb
I0102 13:33:02.945749   34490 dataset.h:887] Iterator::Model constructor
I0102 13:33:02.945802   34490 dataset.h:887] Iterator::Model::MapAndBatch constructor
I0102 13:33:02.945827   34490 dataset.h:887] Iterator::Model::MapAndBatch::Shuffle constructor
I0102 13:33:02.946734   34490 dataset.h:891] Iterator::Model::MapAndBatch::Shuffle destructor
I0102 13:33:02.946763   34490 dataset.h:891] Iterator::Model::MapAndBatch destructor
I0102 13:33:02.946774   34490 dataset.h:891] Iterator::Model destructor
I0102 13:33:02.973794   34643 dataset.h:887] Iterator::Model::MapAndBatch::Shuffle::FlatMap constructor
I0102 13:33:02.973845   34643 dataset.h:887] Iterator::Model::MapAndBatch::Shuffle::FlatMap::FromTensor constructor
I0102 13:33:02.974667   34643 dataset.h:887] Iterator::Model::MapAndBatch::Shuffle::FlatMap[0]::Generator constructor
I0102 13:33:12.974602   34643 shuffle_dataset_op.cc:185] Filling up shuffle buffer (this may take a while): 10238 of 30000
I0102 13:33:22.973838   34643 shuffle_dataset_op.cc:185] Filling up shuffle buffer (this may take a while): 20354 of 30000
I0102 13:33:28.369214   34643 dataset.h:891] Iterator::Model::MapAndBatch::Shuffle::FlatMap[0]::Generator destructor
I0102 13:33:28.369300   34643 dataset.h:891] Iterator::Model::MapAndBatch::Shuffle::FlatMap::FromTensor destructor
I0102 13:33:28.369320   34643 dataset.h:887] Iterator::Model::MapAndBatch::Shuffle::FlatMap constructor
I0102 13:33:28.369330   34643 dataset.h:891] Iterator::Model::MapAndBatch::Shuffle::FlatMap destructor
I0102 13:33:28.369357   34643 dataset.h:887] Iterator::Model::MapAndBatch::Shuffle::FlatMap::FromTensor constructor
I0102 13:33:28.369455   34643 dataset.h:891] Iterator::Model::MapAndBatch::Shuffle::FlatMap::FromTensor destructor
I0102 13:33:28.369468   34643 dataset.h:891] Iterator::Model::MapAndBatch::Shuffle::FlatMap destructor
I0102 13:33:28.369478   34643 shuffle_dataset_op.cc:234] Shuffle buffer filled.
train epoch 4 finish
used memory: 17145.87109375 Mb

The same issue is also with TensorFlow 2.0. After each iteration of the dataset, the buffer get filled again and doesn't release the old buffer.

Can confirm the same issue exists with Tensorflow 2.3

@kamalkraj and @evancasey please create a separate issue with instructions on how to reproduce (and evidence that leads you to believe that the shuffle buffer is not released. You can for instance run your workload with --vmodule=dataset=2 to check whether the shuffle dataset iterator (which owns the buffer) is destructed at the end of each epoch. As per my response from January 2nd, I am not able to reproduce the issue with the instructions posted on this issue.

@jsimsa could you please provide information about how to use this flag or where we can find information about it? Thanks

@jsimsa we would like to reproduce the issue using vmodule=dataset=2 but can not figure out how to provide that flag in tensorflow 2.0. Please advise.

Based on this, I believe that executing your program with the environment variable TF_CPP_VMODULE set to dataset=2 should have the desired effect.

@jsimsa here's a google collab where our (@evancasey and I) issue can be confirmed by looking at the runtime logs. No 'Shuffle buffer filled message' appears and there are some constructors that seem to not be destroyed. https://colab.research.google.com/drive/1cjRs3FjEXtJBnhELkbsf_ueiS2U3pZ_C?usp=sharing

Hi, so the issue happens over multiple epochs / passes over the dataset.
But also there are multiple shuffles happening here (one over the record
names, and one over the flattened batches). Looking at the logs, I count 3
shuffle constructors and only 2 shuffle destructors. I'm not sure why there
are 3 shuffle constructors in the first place, but it appears that one of
them is not being destructed.
On Thu, Sep 3, 2020 at 7:05 PM Jiri Simsa notifications@github.com wrote:

@vipermu https://github.com/vipermu your colab runs a single epoch of
training, which means the shuffle transformation (and its underlying
buffer) will be created once at the beginning and then destroyed (so there
should certainly not be any leaking of shuffle buffer) ... when I ran your
repro using a standalone Python binary with logging enabled, I receive the
following log output:
I0903 16:01:32.081323 1038857 dataset.cc:495] Iterator::Model constructor
I0903 16:01:32.081376 1038857 dataset.cc:495] Iterator::Model::Prefetch constructor
I0903 16:01:32.081398 1038857 dataset.cc:495] Iterator::Model::Prefetch::BatchV2 constructor
I0903 16:01:32.081406 1038857 dataset.cc:495] Iterator::Model::Prefetch::BatchV2::Shuffle constructor
I0903 16:01:32.100963 1039269 dataset.cc:495] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap constructor
I0903 16:01:32.100995 1039269 dataset.cc:495] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2 constructor
I0903 16:01:32.101010 1039269 dataset.cc:495] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle constructor
I0903 16:01:32.301694 1039272 dataset.cc:495] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle::ParallelInterleaveV4 constructor
I0903 16:01:32.301746 1039272 dataset.cc:495] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle::ParallelInterleaveV4::TensorSlice constructor
I0903 16:01:32.402904 1039313 dataset.cc:495] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle::ParallelInterleaveV4[0]::Zip constructor
I0903 16:01:32.402943 1039313 dataset.cc:495] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle::ParallelInterleaveV4[0]::Zip[0]::FlatMap constructor
I0903 16:01:32.402955 1039313 dataset.cc:495] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle::ParallelInterleaveV4[0]::Zip[0]::FlatMap::TensorSlice constructor
I0903 16:01:32.406277 1039313 dataset.cc:495] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle::ParallelInterleaveV4[0]::Zip[1]::TensorSlice constructor
I0903 16:01:32.406596 1039316 dataset.cc:495] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle::ParallelInterleaveV4[0]::Zip[0]::FlatMap[0]::TFRecord constructor
I0903 16:01:32.419646 1039316 dataset.cc:499] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle::ParallelInterleaveV4[0]::Zip[0]::FlatMap[0]::TFRecord destructor
I0903 16:01:32.419682 1039316 dataset.cc:499] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle::ParallelInterleaveV4[0]::Zip[0]::FlatMap::TensorSlice destructor
I0903 16:01:32.419698 1039316 dataset.cc:499] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle::ParallelInterleaveV4[0]::Zip[1]::TensorSlice destructor
I0903 16:01:32.419709 1039316 dataset.cc:499] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle::ParallelInterleaveV4[0]::Zip[0]::FlatMap destructor
I0903 16:01:32.419722 1039316 dataset.cc:499] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle::ParallelInterleaveV4[0]::Zip destructor
I0903 16:01:32.422107 1039272 dataset.cc:495] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle::ParallelInterleaveV4 constructor
I0903 16:01:32.422194 1039272 dataset.cc:499] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle::ParallelInterleaveV4::TensorSlice destructor
I0903 16:01:32.422208 1039272 dataset.cc:499] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle::ParallelInterleaveV4 destructor
I0903 16:01:32.422235 1039272 dataset.cc:495] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle::ParallelInterleaveV4::TensorSlice constructor
I0903 16:01:32.422360 1039272 dataset.cc:499] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle::ParallelInterleaveV4::TensorSlice destructor
I0903 16:01:32.422374 1039272 dataset.cc:499] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle::ParallelInterleaveV4 destructor
I0903 16:01:32.623497 1039269 dataset.cc:495] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap[0]::TensorSlice constructor
I0903 16:01:32.915468 1039269 dataset.cc:499] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap[0]::TensorSlice destructor
I0903 16:01:32.956434 1039269 dataset.cc:495] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap[1]::TensorSlice constructor
I0903 16:01:33.213744 1039269 dataset.cc:499] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap[1]::TensorSlice destructor
I0903 16:01:33.216261 1039269 dataset.cc:499] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle destructor
I0903 16:01:33.216274 1039269 dataset.cc:499] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2 destructor
I0903 16:01:33.216282 1039269 dataset.cc:495] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap constructor
I0903 16:01:33.216284 1039269 dataset.cc:499] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap destructor
I0903 16:01:33.216294 1039269 dataset.cc:495] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2 constructor
I0903 16:01:33.216301 1039269 dataset.cc:495] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle constructor
I0903 16:01:33.384797 1039269 dataset.cc:499] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2::Shuffle destructor
I0903 16:01:33.384828 1039269 dataset.cc:499] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap::ParallelMapV2 destructor
I0903 16:01:33.384833 1039269 dataset.cc:499] Iterator::Model::Prefetch::BatchV2::Shuffle::FlatMap destructor
which indicates that every shuffle constructor is paired with destructor.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/34942#issuecomment-686807529,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAOP756RQ6HC2OVDGLCBBZLSEAOMLANCNFSM4JX5N3SA
.

--
Cadmium
cadmium.app | evan@cadmium.app
+1 (303) 808-2955

Oops, my mistake there I just noticed the third shuffle destructor. Disregard that last comment. Anyway, can you try running this for more than one epoch? I have a feeling that the issue only happens some of the time, but it will always happen at least once over a few epochs

There are multiple shuffles per epoch because your input pipeline contains multiple shuffles -- one before flat_map and one after flat_map. The one before flat_map will result in a shuffle buffer being created and destroyed for each input element of flat_map (as opposed to once per epoch).
I ran your example for 100 epochs using my standalone binary repro and do not see memory increase. In other words, I cannot reproduce your issue.
Have you confirmed that the issue goes away when you remove the shuffle buffer? If so, can you extend your colab to run the epoch within a loop and print the amount of memory allocated at the end of each loop?

Hi There,
We are checking to see if you still need help on this, as you are using an older version of tensorflow which is officially considered end of life . We recommend that you upgrade to the latest 2.x version and let us know if the issue still persists in newer versions. Please open a new issue for any help you need against 2.x, and we will get you the right help.
This issue will be closed automatically 7 days from now. If you still need help with this issue, please provide us with more information.

I have recently investigated the memory growth observed for OSS version of TensorFlow when shuffle is used. The conclusion of my investigation is that the memory growth is related to poor performance of memory allocator (TensorFlow uses system malloc by default). In my experiments, switching to use TCMalloc (details below) resulted in constant memory usage.
For the evaluation, I used the following simple input pipeline:
import tensorflow as tf
import psutil

dataset = tf.Dataset.range(int(1e7))
iterator = dataset.shuffle(int(1e7)).batch(int(1e6))

for _ in iterator:
  used_mem = psutil.virtual_memory().used
  print("used memory: {} Mb".format(used_mem / 1024 / 1024))

When executed on workstation, it produces the following output:
$ python example.py

used memory: 19853.52734375 Mb
used memory: 19905.6484375 Mb
used memory: 19958.109375 Mb
used memory: 20014.796875 Mb
used memory: 20064.8359375 Mb
used memory: 20061.375 Mb
used memory: 20117.23828125 Mb
used memory: 20172.8515625 Mb
used memory: 20228.18359375 Mb
used memory: 20278.62890625 Mb

I then installed tcmalloc using sudo apt-get install libtcmalloc-minimal4 and used it for the same program, as follows:
LD_PRELOAD=/path/to/libtcmalloc_minimal.so.4 python example.py

used memory: 19291.0859375 Mb
used memory: 19307.90234375 Mb
used memory: 19315.859375 Mb
used memory: 19315.859375 Mb
used memory: 19315.875 Mb
used memory: 19317.8671875 Mb
used memory: 19311.14453125 Mb
used memory: 19317.3515625 Mb
used memory: 19317.34765625 Mb
used memory: 19316.96484375 Mb

Not only the gradual memory growth disappeared, but the program also ran 2x faster.

did TF 2.5 release solved this issue for anyone ?

I am on 2.7 and still have the problem.

Same issue with TensorFlow 2.9 in the Kaggle kernel.

authors: Owaiskhan9654 azzeddineCH evancasey gadagashwini jsimsa kamalkraj kindernerd pubkey tensorflowbutler vipermu


text:
retrying_file_system does not use exponential backoff

In tensorflow-0.10.0rc0/tensorflow/core/platform/cloud/retrying_file_system.cc,
the retry logic does not insert a delay between successive attempts.
Status CallWithRetries(const std::function<Status()>& f) {
int attempts = 0;
while (true) {
attempts++;
auto status = f();
if (!IsRetriable(status) || attempts >= kMaxAttempts) {
return status;
}
LOG(ERROR) << "The operation resulted in an error and will be retried: "
<< status.ToString();
}
}

Yes-  this retry logic does look pretty basic.
@davidj-github  Could you please explain a bit more about the problem you are having.
@rinugun  Is there another backoff mechanism at a lower level (e.g. in an RPC layer?) or does this just retry 4 times without sleeping?

I was reviewing the code and noticed that it does not conform to Google's recommendation to insert a delay between retries.

@davidj-github I agree this is not usually very desirable behavior.  If it's not causing you an immediate problem, then I assume there's no great urgency here?  I'll let the authors of this module comment further.

That's correct, there is no backoff, it simply retries 3 times, so that there is at most 4 attempts including the original one.
The underlying layers do not attempt any retries.
The reason for the lack of a backoff was that the actual Google Cloud Storage errors don't require it -- very infrequently GCS can return an internal error for a read or write operation and an immediate retry typically succeeds.
That said, I don't mind having a backoff if it's recommended.

I would like to work on this

Updating CallWithRetries to do exponential backoff #4501 .
Could you all please review?

Schlumberger-Private
The Cloud storage documentation for exponential backoff recommends adding a random term to the sleep time. See https://cloud.google.com/storage/docs/exponential-backoff
From: Balachander Ramachandran [mailto:notifications@github.com]
Sent: Tuesday, September 20, 2016 10:36 PM
To: tensorflow/tensorflow tensorflow@noreply.github.com
Cc: David Judson DJudson@slb.com; Mention mention@noreply.github.com
Subject: [Ext] Re: [tensorflow/tensorflow] retrying_file_system does not use exponential backoff (#4066)
Updating CallWithRetries to do exponential backoff #4501https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_tensorflow_tensorflow_pull_4501&d=CwMCaQ&c=uGuXJ43KPkPWEl2imVFDmZQlhQUET7pVRA2PDIOxgqw&r=7r_lgzfgEAaM8ykY8TCz9g&m=1VZnPLf14kYsDIRR9L1U0Fbl0Mf2CCC0FDUL4Im_wcA&s=wMRcLblp4zkClmvOBbhBMqSZyuKK3CXrbBaX9p43BoE&e= .
Could you all please review?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_tensorflow_tensorflow_issues_4066-23issuecomment-2D248502364&d=CwMCaQ&c=uGuXJ43KPkPWEl2imVFDmZQlhQUET7pVRA2PDIOxgqw&r=7r_lgzfgEAaM8ykY8TCz9g&m=1VZnPLf14kYsDIRR9L1U0Fbl0Mf2CCC0FDUL4Im_wcA&s=XxeUsnSilLQMvLPmR3DAa1j5359E7fWd_Vx0QCxcml0&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ATdHG-5F0xVwTbyWpqRS5JI4Z1fpiVkVvMks5qsKYmgaJpZM4JuaF2&d=CwMCaQ&c=uGuXJ43KPkPWEl2imVFDmZQlhQUET7pVRA2PDIOxgqw&r=7r_lgzfgEAaM8ykY8TCz9g&m=1VZnPLf14kYsDIRR9L1U0Fbl0Mf2CCC0FDUL4Im_wcA&s=2LRsfLEwulq84LcMiXDzcqj-icQnZwIhsDCmZyjiQ8Y&e=.

@davidj-github @rinugun
Proposed changes to incorporate the above random sleep for milliseconds
#include <chrono>
#include <random>
#include <thread>

void WaitBeforeRetry(const int delay_seconds) {
    std::random_device rd;
    std::mt19937 gen(rd());
    std::uniform_int_distribution<> dist(1, 1000);

    const int delay_milliseconds = delay_seconds * 1000;
    const int random_milliseconds = dist(gen);
    const int delay = std::min(delay_to_milliseconds + random_milliseconds,
                                                kMaximumBackoffSeconds);

    std::this_thread::sleep_for(std::chrono::milliseconds(delay));
}

Status CallWithRetries(const std::function<Status()>& f,
                       const int initial_delay_seconds) {
  int retries = 0;
  while (true) {
    auto status = f();
    if (!IsRetriable(status) || retries >= kMaxRetries) {
      return status;
    }
    LOG(ERROR) << "The operation resulted in an error and will be retried: "
               << status.ToString();
    WaitBeforeRetry(initial_delay_seconds << retries);
    retries++;
  }
}

@davidj-github @rinugun @prb12 what do you guys think?

I'm curious, is random delay better than the fixed delay that was
implemented in https://github.com/tensorflow/tensorflow/pull/4501? The
random sub-second sleep seems to be adding a particular complexity to the
retry logic, I wonder if this is justified and what the expected wins are.
On Sat, Sep 24, 2016 at 6:43 PM, Balachander Ramachandran <
notifications@github.com> wrote:

@davidj-github https://github.com/davidj-github @rinugun
https://github.com/rinugun @prb12 https://github.com/prb12 what do
you guys think?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/4066#issuecomment-249397475,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABnZstqsh-CSWZmdEk4EI9lQ0CKYIw1Zks5qtdHYgaJpZM4JuaF2
.

From the Google Cloud storage documentation -

random_number_milliseconds is a random number of milliseconds less than or equal to 1000. This helps to avoid cases where many clients get synchronized by some situation and all retry at once, sending requests in synchronized waves. The value of random_number_milliseconds is recalculated after each retry request.

I could only find this text in Google Analytics docs (
https://developers.google.com/analytics/devguides/reporting/core/v4/errors),
is that also mentioned somewhere for the Google Cloud Storage docs?
Anyways, I agree that it might be potentially useful. I have a few comments
re the code snippet:

not sure  is supported in the TF code, but you could use
Env::Default()->SleepForMilliseconds() from env.h instead
other minor comments, probably could be discussed during a code review

Thanks
Alex
On Sat, Sep 24, 2016 at 8:38 PM, Balachander Ramachandran <
notifications@github.com> wrote:

From the Google Cloud storage documentation -
random_number_milliseconds is a random number of milliseconds less than or
equal to 1000. This helps to avoid cases where many clients get
synchronized by some situation and all retry at once, sending requests in
synchronized waves. The value of random_number_milliseconds is recalculated
after each retry request.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/4066#issuecomment-249400674,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABnZsg-D-NAn5ghCaT9yHj-szBG8u23Sks5qtey6gaJpZM4JuaF2
.

It is documented here - https://cloud.google.com/storage/docs/exponential-backoff

@davidj-github @rinugun @prb12 have updated the PR(#4501) to include the random millisecond delay. Please review!

shouldnt this be marked closed?

authors: davidj-github moontails prb12 rinugun


text:
Better Instruction/Insight as to how models.load_model() custom objects are loaded (doc/feature request/bug?)

Thank you for submitting a TensorFlow documentation issue. Per our GitHub
policy, we only address code/doc bugs, performance issues, feature requests, and
build/installation issues on GitHub.
The TensorFlow docs are open source! To get involved, read the documentation
contributor guide: https://www.tensorflow.org/community/contribute/docs
URL(s) with the issue:
https://www.tensorflow.org/api_docs/python/tf/keras/models/load_model?hl=no
It may be better to have a separate URL for new documentation.
Description of issue (what needs changing):
This could be treated potentially as a feature request and/or bug as the issues encountered may or may not be a result of things out of my control. Better documentation is requested regardless.
The documentation on how to load models that contain custom objects is lacking as to prerequisites to make them loadable.
The only thing that is clear is that a dict of the objects of interest must be provided. It isn't clear what they keys are supposed to be. It is also unclear as to how the objects are loaded and what happens at load time.
When running this code
    def load_model(self, model_name, model_path):
        """Reload tensorflow session for saved model. Called by djinn.load,

        Args:
            model_path (str, optional): Location of model if different than
                       location set during training.
            model_name (str, optional): Name of model if different than
                       name set during training.

        Returns:
            Object: djinn regressor model.

        """
        # self.__sess = {}
        self.__models={}
        for p in range(0, self.__n_trees):
            tf.keras.backend.clear_session()
            # tf.reset_default_graph()
            # new_saver = \
            # tf.train.import_meta_graph('%s%s_tree%s.ckpt.meta'%(model_path,model_name,p))
            # self.__sess[p] = tf.Session()
            # new_saver.restore(self.__sess[p], '%s%s_tree%s.ckpt'%(model_path,model_name,p))
            self.__models[p] = tf.keras.models.load_model('%s%s_tree%s.ckpt'%(model_path,model_name,p),custom_objects={'WB_Init': WB_Init})
            # self.__models[p] = tf.keras.models.load_model('%s%s_tree%s.ckpt'%(model_path,model_name,p))
            print("Model %s restored"%p)

where WB_Init is defined in another file as
class WB_Init(tf.keras.initializers.Initializer):
    def __init__(self,dat=None,name=None):
        self.dat = dat
        self.name = name
        # print(type(dat))
        # print(dat)

    def __call__(self,shape,dtype):

        if not isinstance(self.dat,tf.Tensor):
            a = tf.convert_to_tensor(self.dat,dtype=tf.float32,name=self.name)
        else:
            # a = self.dat.value()
            a = self.dat

        return a

and the file doing the calling is indeed loading the object with
try:
    from djinn_fns import tree_to_nn_weights, tf_dropout_regression, \
                    get_hyperparams, tf_continue_training, WB_Init
except:
    from djinn.djinn_fns import tree_to_nn_weights, tf_dropout_regression, \
                    get_hyperparams, tf_continue_training, WB_Init

WB_Init is used to initialize weights and biases that have been predefined by other functions and need to be trainable. I have avoided the constant initializer because there is no immediate documentation to suggest the values produced by it are capable of being trained (another documentation clarification maybe).
When trying load in an already trained model I'm greeted with the error traceback:
Traceback (most recent call last):

  File "C:\Users\Michael\Desktop\Code Projects\DJINN TF2\tests\djinn_regression_example.py", line 81, in <module>
    m=model.predict(x_test) #returns the median prediction if more than one tree

  File "..\djinn\djinn.py", line 383, in predict
    return self.bayesian_predict(x_test, None, random_state)

  File "..\djinn\djinn.py", line 341, in bayesian_predict
    if(self.__models == None): self.load_model(self.modelname, self.modelpath)

  File "..\djinn\djinn.py", line 294, in load_model
    self.__models[p] = tf.keras.models.load_model('%s%s_tree%s.ckpt'%(model_path,model_name,p),custom_objects={'WB_Init': WB_Init})

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\keras\saving\save.py", line 182, in load_model
    return hdf5_format.load_model_from_hdf5(filepath, custom_objects, compile)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\keras\saving\hdf5_format.py", line 178, in load_model_from_hdf5
    custom_objects=custom_objects)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\keras\saving\model_config.py", line 55, in model_from_config
    return deserialize(config, custom_objects=custom_objects)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\keras\layers\serialization.py", line 175, in deserialize
    printable_module_name='layer')

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\keras\utils\generic_utils.py", line 358, in deserialize_keras_object
    list(custom_objects.items())))

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\keras\engine\functional.py", line 617, in from_config
    config, custom_objects)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\keras\engine\functional.py", line 1214, in reconstruct_from_config
    process_node(layer, node_data)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\keras\engine\functional.py", line 1162, in process_node
    output_tensors = layer(input_tensors, **kwargs)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py", line 757, in __call__
    self._maybe_build(inputs)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py", line 2098, in _maybe_build
    self.build(input_shapes)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\keras\layers\core.py", line 1178, in build
    trainable=True)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\keras\engine\base_layer_v1.py", line 448, in add_weight
    caching_device=caching_device)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\training\tracking\base.py", line 750, in _add_variable_with_custom_getter
    **kwargs_for_getter)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\keras\engine\base_layer_utils.py", line 145, in make_variable
    shape=variable_shape if variable_shape else None)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\ops\variables.py", line 260, in __call__
    return cls._variable_v1_call(*args, **kwargs)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\ops\variables.py", line 221, in _variable_v1_call
    shape=shape)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\ops\variables.py", line 199, in <lambda>
    previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\ops\variable_scope.py", line 2597, in default_variable_creator
    shape=shape)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\ops\variables.py", line 264, in __call__
    return super(VariableMetaclass, cls).__call__(*args, **kwargs)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\ops\resource_variable_ops.py", line 1518, in __init__
    distribute_strategy=distribute_strategy)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\ops\resource_variable_ops.py", line 1651, in _init_from_args
    initial_value() if init_from_fn else initial_value,

  File "..\djinn\djinn_fns.py", line 873, in __call__
    a = tf.convert_to_tensor(self.dat,dtype=tf.float32,name=self.name)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 1382, in convert_to_tensor_v2
    as_ref=False)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 1499, in convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\framework\constant_op.py", line 338, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\framework\constant_op.py", line 264, in constant
    allow_broadcast=True)

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\framework\constant_op.py", line 282, in _constant_impl
    allow_broadcast=allow_broadcast))

  File "C:\Users\Michael\anaconda3\lib\site-packages\tensorflow\python\framework\tensor_util.py", line 444, in make_tensor_proto
    raise ValueError("None values not supported.")

ValueError: None values not supported.

where it is clear as to the "precise reason" why load is failing. That reason is because the WB_Init.init has a default dat=None
which when WB_Init.call is used returns a None value which isn't a support return type for the initialization process. This is what happens when saved to HDF5 format. The tensorflow format breaks in a different manner at save time with the final error being FailedPreconditionError: ./reg_djinn_test_tree0.ckpt is not a directory.
However, it clearly demonstrates that a fresh call of a WB_Init object is made at load time which doesn't make sense to me because the model has already been trained. It should have weights to load. The initializer shouldn't be run as far as I know. Setting compile=False doesn't fix this problem. If I just wanted to load the model for inference, I could potentially try to completely reconstruct the model from some other pickled custom object that hold the layer topology and then just use load_weights, but this is a horribly dirty solution. Using this workaround also doesn't let me save the optimizer state so that the model training can be picked-up at the most recent optimizer state.
There is another hypothetical fix where WB_Init is fed the saved weights at load time, but I would have no idea of how to make this happen.
There is also the highly non-preferred way of initializing the network at construction by filling it the models with random numbers and loading the pre-calculated weights after construction (and compilation?) of the model. This defeats the purpose of providing an initializer option at construction time though.
Clear description
Please provide more a more clear description about how to make custom intializers (or any custom object ) and, more importantly, how to load custom objects (dict keys?) and the prerequisites for making the model and/or object loadable.
This problem and problems like this have been recurring. Some of them don't make sense to me, some are TF1 and don't apply, and some simply don't fix my issue. Some have simply been bugs.
https://github.com/tensorflow/tensorflow/issues/39752 ( this is the one that actually demonstrated how to use the 'custom_object' option )
https://github.com/tensorflow/tensorflow/issues/42778
https://github.com/qqwweee/keras-yolo3/issues/48  (referenced in the link above)
https://github.com/tensorflow/tensorflow/issues/33646
https://github.com/titu1994/keras-efficientnets/issues/3
https://github.com/tensorflow/tensorflow/issues/26061
It clearly could use better documentation.
I will provide a full script upon request, and if it is possible, I'd prefer to supply it privately. I'm currently trying to convert a TF1/(TF2 with compatibility) model to a TF2 model that uses Keras. It will (almost certainly) break even if this problem is fixed because the conversion is in progress. The break should be different in nature though if this problem is fixed.
Correct links
N/A
Parameters defined
N/A
Returns defined
N/A
Raises listed and defined
N/A
Usage example
See above, but it isn't usable without the rest of the files.
Submit a pull request?
No

@EnderWiggin14
Could you please let us know the tf version on which this issue is faced and share a  colab gist with the error reported.

I'm using TF==2.30.
Oddly, the code broke where is wasn't breaking before because of something to do with metrics as I had passed "callbacks=tf.keras.callbacks.BaseLogger()" and it was choking on metrics. This doesn't happen with Spyder IDE.
Here is colab gist. I couldn't figure out how to use regular .py files without giving access to my whole google drive, so this will have to do.
https://colab.research.google.com/drive/1f-eQ22tKMc4o8j3gV81mS2ZIlk-_x34T?usp=sharing

@EnderWiggin14
I do not see the error "None values not supported." in the gist shared, as reported in the issue. I ran the code on nightly as well and do not face any error.

@Saduf2019
Are you referencing the possibility of using tf.TensorShape? I have added a bypass statement such that
`
def call(self,shape,dtype):
if self.dat is None:
    a = tf.TensorShape(shape)
else:
    a = tf.convert_to_tensor(self.dat,dtype=tf.float32,name=self.name)

return a

It didn't work. I'm met with the errorTypeError: Cannot convert a TensorShape to dtype: <dtype: 'float32'>` which, from what I read about TensorShape, this is more or less what I expected.
Otherwise, I think you have misunderstood the problem. I'm aware of the reason as to why convert_to_tensor is failing. I'm well aware that None type isn't supported. The problem is that I'm trying to use a custom a initializer which because it tries to run the initializer at load time which fails. I have tried to default to a 2x2 array, but that causes an incompatibility with the first layer.
The two ways I can think of how to fix this are

Somehow give a fully valid Tensor of correct shape and dtype without overwriting the existing weights
Feed the correct weights as the input at load time.

Neither of these things appear to be easily accomplished in Python (i.e. without messing with source code)
I'm looking for how a person is supposed to be able to create a custom initializer, save the model that uses it, and re-load the model without having a problem.
I'm not looking initialize with a supplied initializer and then overwriting the weights with the correct weights. This defeats the purpose of being able to

@Saduf2019
What Tensorflow version does colab currently default to?
Attached is a screen shot of the error I got from colab. It might be worded a bit differently, but it is the same error.

@EnderWiggin14 It defaults to tensorflow 2.3.0 now

Hi @EnderWiggin14 ,
The problem is here:
 def __init__(self,dat=None,...):

To make a non-trivial object keras-serializable it should define get_config and from_config methods, as described here
Since you haven't defined those methods, when keras reconstructs your objects it calls WB_Init with no arguments, so dat takes the default value of None.
Also note in this case instead of defining an initializer, you should consider just setting the variable value after creating the layer (otherwise the initializer needs to remember the initial value, which seems unnecessary).


text:
OP_REQUIRES failed at reshape_op.h:57 : Invalid argument: Size 1 must be non-negative, not -9

1.environment: windows10, cuda10.1, cudnn7.6.5 and tensorflow2.2.0
2.my problem:
i wanted to save my total model with tf.saved_model.save , and i used tf.squeeze and tf.reshape together. Then, when i used model.signatures["serving_default"](inputs) some bugs happened.
But, if i used tf.keras.models.load_model('./saved_model') to save model, everything is ok when predicting.
3.part of my codes:
# save model
raw_prediction = tf.squeeze(raw_prediction)
shape = tf.shape(raw_prediction)
height = shape[0]
width = shape[1]
shape_float = tf.cast(shape[0:2], dtype=tf.float32)
raw_prediction = keras.backend.reshape(raw_prediction, shape=(height, width, num_anchors, -1))

# predict
model = tf.saved_model.load('./saved_model')
model.signatures["serving_default"](inputs)  # bugs happen when i predicted

4.bugs:
 OP_REQUIRES failed at reshape_op.h:57 : Invalid argument: Size 1 must be non-negative, not -9
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  Size 1 must be non-negative, not -9
	 [[{{node StatefulPartitionedCall/model_1/tf_op_layer_RealDiv_14/RealDiv_14-1-ReshapeNHWCToNCHW-LayoutOptimizer}}]]
	 [[StatefulPartitionedCall/model_1/tf_op_layer_GatherV2_5/GatherV2_5/_18]]
  (1) Invalid argument:  Size 1 must be non-negative, not -9
	 [[{{node StatefulPartitionedCall/model_1/tf_op_layer_RealDiv_14/RealDiv_14-1-ReshapeNHWCToNCHW-LayoutOptimizer}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference_signature_wrapper_10761]

Function call stack:
signature_wrapper -> signature_wrapper

if i don't use tf.squeeze, codes are operating correctly.
# save model
# raw_prediction = tf.squeeze(raw_prediction)
shape = tf.shape(raw_prediction)
height = shape[1]
width = shape[2]
shape_float = tf.cast(shape[1:3], dtype=tf.float32)
raw_prediction = keras.backend.reshape(raw_prediction, shape=(height, width, num_anchors, -1))

@HirataYurina,
In order to expedite the trouble-shooting process, could you please provide the complete code to reproduce the issue reported here and the dataset you are using.
Also, please update TensorFlow to v2.3 and check if you are still facing the same issue. Thanks!

authors: HirataYurina amahendrakar


text:
tf.linalg.triangular_solve different behaviour on CPU vs GPU

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
https://colab.research.google.com/drive/1uTUzEQXDR6JvCkFQyS-3n2pAUKAH-TLk?usp=sharing#scrollTo=BcSb0SDQ5nI0
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04 (also Colab)
TensorFlow installed from (source or binary): source
TensorFlow version (use command below): 2.4.3 (also Colab)
Python version: 3.7.10

Describe the current behavior
When trying to apply tf.linalg.triangular_solve on an invalid input matrix (in my case I included np.nan in the matrix) - different behaviour is observed when running on CPU vs GPU. On CPU - the functions returns NaNs, whereas on GPU an InvalidArgumentError occurs
Describe the expected behavior
Consistent behaviour between CPU & GPU
Standalone code to reproduce the issue
https://colab.research.google.com/drive/1uTUzEQXDR6JvCkFQyS-3n2pAUKAH-TLk?usp=sharing#scrollTo=BcSb0SDQ5nI0

@qiq208,
As per the colab link provided, with CPU we get InvalidArgumentError and with GPU, it returns NaNs whereas in the issue description you have mentioned in the opposite way.

I am able to reproduce the issue reported, in colab with both TF 2.6.0 and nightly 2.8.0-dev20211101. Please find the gist here