text:
Consider increasing timeout_factor for PyAudio in the Onsets and Frames realtime Python demo

The Python demo for OnF realtime was crashing on me, because of a timeout issue in PyAudio.
Since I heard about problems with PyAudio on macOS Catalina, I first went down the wrong path, instead of just increasing the timeout, and struggled until I got the right idea. The current value for timeout_factor is 12 and I found that I need at least 24 for the demo to work (and it works perfectly fine after). Since I have a reasonably fast machine, I thought other people might have the same problem and that it might be sensible to increase the default timeout_factor (also because there's not a big downside to doing that).
Tagging @cghawthorne as likely the codeowner. Feel free to disagree and close this. Thank you for all your work!

@mtyka any thoughts on this?

I've run into this issue also. Happy to increase it if that helps.

authors: cghawthorne janek mtyka

repo: tensorflow/magenta


text:
sparse_matmul_op_test fails on ppc64le

Please go to Stack Overflow for help and support:
https://stackoverflow.com/questions/tagged/tensorflow
If you open a GitHub issue, here is our policy:

It must be a bug or a feature request.
The form below must be filled out.
It shouldn't be a TensorBoard issue. Those go here.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): RHEL/SLES ppc64le
TensorFlow installed from (source or binary): source
TensorFlow version (use command below): master
Python version: 2.7.5
Bazel version (if compiling from source): 0.4.5
CUDA/cuDNN version:
GPU model and memory:
Exact command to reproduce: bazel test  --test_output=errors //tensorflow/core/kernels:sparse_matmul_op_test

You can collect some of this information using our environment capture script:
https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh
You can obtain the TensorFlow version with
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
Describe the problem
Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.
Test fails since the pbroadcast_* functions in tensorflow/core/kernels/sparse_matmul_op.h are not implemented for PowerPC (has SSE versions), resulting in incorrect values received by BroadcastPacketTest function in the test case code (tensorflow/core/kernels/sparse_matmul_op_test.cc) , log of failure in below section
Source code / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.
INFO: From Testing //tensorflow/core/kernels:sparse_matmul_op_test:
==================== Test output for //tensorflow/core/kernels:sparse_matmul_op_test:
Running main() from test_main.cc
[==========] Running 4 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 4 tests from SparseMatmulOpTest
[ RUN      ] SparseMatmulOpTest.BroadcastPacketTest
[0.170094 0.170094 0.170094 0.170094] != [  0.170094    0.14922 -0.0823886   0.026985], differences: [         0 -0.0208738  -0.252482  -0.143109]
tensorflow/core/kernels/sparse_matmul_op_test.cc:257: Failure
Value of: areApprox(ref, data2, PacketSize)
Actual: false
Expected: true
[  FAILED  ] SparseMatmulOpTest.BroadcastPacketTest (1 ms)
[ RUN      ] SparseMatmulOpTest.InterleavePacketTest
[       OK ] SparseMatmulOpTest.InterleavePacketTest (0 ms)
[ RUN      ] SparseMatmulOpTest.Bfloat16ExpandTest
[       OK ] SparseMatmulOpTest.Bfloat16ExpandTest (0 ms)
[ RUN      ] SparseMatmulOpTest.Bfloat16LoadTest
[       OK ] SparseMatmulOpTest.Bfloat16LoadTest (0 ms)
[----------] 4 tests from SparseMatmulOpTest (1 ms total)
[----------] Global test environment tear-down
[==========] 4 tests from 1 test case ran. (1 ms total)
[  PASSED  ] 3 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] SparseMatmulOpTest.BroadcastPacketTest
1 FAILED TEST
Target //tensorflow/core/kernels:sparse_matmul_op_test up-to-date:
bazel-bin/tensorflow/core/kernels/sparse_matmul_op_test
INFO: Elapsed time: 14.711s, Critical Path: 14.07s
//tensorflow/core/kernels:sparse_matmul_op_test                          FAILED in 1 out of 2 in 0.0s
/root/.cache/bazel/_bazel_root/68a62076e91007a7908bc42a32e4cff9/execroot/tensorflow/bazel-out/local-opt/testlogs/tensorflow/core/kernels/sparse_matmul_op_test/test.log
Executed 1 out of 1 test: 1 fails locally.

@vaibhavsood Thanks for filing the issue and the PR!  I'm marking this as contributions welcome, and we'll close this out once your PR gets merged.

Thanks!

source: gharchive/issue

authors: tatatodd vaibhavsood

repo: tensorflow/tensorflow

license_type: permissive

license_source: github-api


text:
why save model and deploy in android device, the outputs are not the same as in ubuntu?

OS Platform and Distribution :Ubuntu 14.04.5 LTS   && Android 8.0
TensorFlow installed from source
TensorFlow version :1..4.0
Python version : 2.7.6
Bazel version :0.4.5
GCC/Compiler version:4.8.4
CUDA/cuDNN version:8.0
GPU model and memory: GTX1080, 8G
Exact command to reproduce:
step 1. clone code from https://github.com/davidsandberg/facenet
step 2. in the file src/compare.py, add the follow code after line 90, after align.detect_face.create_mtcnn
output_node_names=['pnet/prob1','pnet/conv4-2/BiasAdd','pnet/conv1/BiasAdd','rnet/prob1','rnet/conv5-2/conv5-2','onet/prob1','onet/conv6-2/conv6-2','onet/conv6-3/conv6-3']
output_graph_def = tf.graph_util.convert_variables_to_constants(sess, sess.graph.as_graph_def(), output_node_names) #sess.graph_def,
with tf.gfile.FastGFile("mtcnn.pb", mode = 'wb') as f:
f.write(output_graph_def.SerializeToString())
step 3. run the command:  python src/compare.py ./data/20170512-110547 ./data/images/Anthony_Hopkins_0001.jpg ./data/images/Anthony_Hopkins_0002.jpg
will create the model file mtcnn.pb
step 4 deploy the file mtcnn.pb to android app, validate with the file Anthony_Hopkins_0001.jpg, indeed it can fetch the results for the outputs such as 'pnet/prob1','pnet/conv4-2/BiasAdd',  but the values are difference with the results from the facenet project run on ubuntu,
the query is what is the cause to the difference? the  way to create the mtcnn.pb is wrong? still need to optimize it to adapt android device? or there is something wrong with Tensorflow for mobile device?
the follow is the log show the difference:
with the same input:00.28515625,-0.24609375,-0.59765625
but the output is difference
Android output
Line 5555: 01-26 11:34:20.793 I/lxr     (22967): img00.28515625,-0.24609375,-0.59765625
Line 5795: 01-26 11:34:21.326 I/lxr     (22967): mapWidth 70 mapHeight 70
Line 5796: 01-26 11:34:21.327 I/lxr     (22967): outValue:0.9998832,1.16751995E-4
Line 5797: 01-26 11:34:21.327 I/lxr     (22967): outReg:-0.068167016,-0.2052449,0.06884944,0.1512082
Ubuntu output
img_y (1, 150, 150, 3)
img_y0 [ 0.28515625 -0.24609375 -0.59765625]
out0 shape (1, 70, 70, 4)
out1 shape (1, 70, 70, 2)
out0 [-0.07926445 -0.20101449  0.06468102  0.16017048]
out1 [  9.99792397e-01   2.07666759e-04]

Nagging Assignee: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

We have a trained quantized graph deployed on ubuntu/ios/android. And android results in COMPLETELY different (incorrect) output.

I had found the root cause by myself: the input data need to be transposed,.
and  there still a little difference for the precious between ubuntu and android device.

Oh good to see you're unstuck.

I have same problem ,did you transposed the input in python or android? the input in android is an one dimensional array, which in python is [1,160,160,3],i want to know how did you do it,thanks very much

authors: drpngx eaigner leesino tensorflowbutler zfs1993


text:
CUDNN_STATUS_ALLOC_FAILED

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no, it's on examples/tutorial/mnist,  mnist_deep.py
Bazel version: N/A
Mobile device: N/A
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): windows 10 x64 latest build
TensorFlow installed from (source or binary): binary prebuilt
TensorFlow version (use command below): 1.11.0rc1
Python version: 3.6.6
CUDA/cuDNN version: 9.0/7.2.1
GPU model and memory: GTX1050Ti GDDR5 4GB
Exact command to reproduce:

python mnist_deep.py
E tensorflow/stream_executor/cuda/cuda_dnn.cc:353] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED

other codes are fine-working as before

no problem with my laptop, which has geforce mx150 GDDR5 2GB

all this happens after I installed geforce driver 411.63
Source code / logs
PS D:\VSWorkspace\tensorflow\tensorflow\examples\tutorials\mnist> python .\mnist_deep.py
WARNING:tensorflow:From .\mnist_deep.py:130: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
WARNING:tensorflow:From C:\Anaconda3\lib\site-packages\tensorflow\contrib\learn\python\learn\datasets\mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future versi
on.
Instructions for updating:
Please write your own downloading logic.
WARNING:tensorflow:From C:\Anaconda3\lib\site-packages\tensorflow\contrib\learn\python\learn\datasets\mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future vers
ion.
Instructions for updating:
Please use tf.data to implement this functionality.
Extracting /tmp/tensorflow/mnist/input_data\train-images-idx3-ubyte.gz
WARNING:tensorflow:From C:\Anaconda3\lib\site-packages\tensorflow\contrib\learn\python\learn\datasets\mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future vers
ion.
Instructions for updating:
Please use tf.data to implement this functionality.
Extracting /tmp/tensorflow/mnist/input_data\train-labels-idx1-ubyte.gz
Extracting /tmp/tensorflow/mnist/input_data\t10k-images-idx3-ubyte.gz
Extracting /tmp/tensorflow/mnist/input_data\t10k-labels-idx1-ubyte.gz
WARNING:tensorflow:From C:\Anaconda3\lib\site-packages\tensorflow\contrib\learn\python\learn\datasets\mnist.py:290: DataSet.init (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future ve
rsion.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
Saving graph to: C:\Users\alanp\AppData\Local\Temp\tmpxpinc32v
2018-09-20 22:10:32.455130: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2018-09-20 22:10:32.641187: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1411] Found device 0 with properties:
name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.4175
pciBusID: 0000:01:00.0
totalMemory: 4.00GiB freeMemory: 3.30GiB
2018-09-20 22:10:32.659090: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1490] Adding visible gpu devices: 0
2018-09-20 22:10:33.556406: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-09-20 22:10:33.565294: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977]      0
2018-09-20 22:10:33.571150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0:   N
2018-09-20 22:10:33.578487: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1103] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3013 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus
id: 0000:01:00.0, compute capability: 6.1)
2018-09-20 22:10:36.376573: E tensorflow/stream_executor/cuda/cuda_dnn.cc:353] Could not create cudnn handle: CUDNN_STATUS_ALLOC_FAILED

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Bazel version
Mobile device

@tensorflowbutler  updated

@zheng-xq @zhangyaobit @alextp Could you comment or reassign?

@azaks2 do you know if there is a known issue with this driver version?

I would upgrade to cuDNN 7.3 (7.2 is no longer supported on the NVIDIA site and we are going to downgrade to 7.1.4 for our builds but anything over 7.1 will work).  The 411 driver is really new and likely more aligned with CUDA 10 but I have no data points even on Linux at this time as to combinations I know work.
My advice (with no data points I am sorry):

Upgrade to cuDNN 7.3
Consider downgrading to an older Device Driver like 396 but I realized that is really hard on windows as it tries to keep you on the latest possible version.  I would likely do this as a last resort.

@tfboyd
with cudnn 7.3, same
but thank you anyway, I'd rather stick with ubuntu this moment and leave this pc for a while

For people searching the keyword CUDNN_STATUS_ALLOC_FAILED
and finding this issue. You may check if there is another process allocating the gpu. In my case i got this error on terminal after previously running tensorflow from a jupyter notebook

authors: alanpurple alextp facaiy mjohenneken tensorflowbutler tfboyd


text:
Add demo gif to README

Disclaimer: This is a bot
It looks like your repo is trending. The github_trending_videos Instgram account automatically shows the demo gifs of trending repos in Github.
Your README doesn't seem to have any demo gifs. Add one and the next time the parser runs it will pick it up and post it on its Instagram feed. If you don't want to just close this issue we won't bother you again.

Your link is dead mister robot buddy


text:
Reduced redundancy in Estimator api

System information

TensorFlow version (you are using): 1.10+
Are you willing to contribute it (Yes/No): yes

Describe the feature and the current behavior/state.
The TensorFlow estimator api, while a welcomed addition, seems to add a lot of redundancy to the workflow as well as unnecessary complexity. While a prominent goal of the estimator API is to decouple the input pipelines from the model, unfortunately this is not fully realized due to this redundancy and complexity.
Example
Suppose we wish to make a custom estimator that has an input_fn which reads from TF Record files via tf.data.TFRecordDataset(files) and further, we wish to save our trained model so we can use it later (i.e. we need a serving_input_reciever_fn).
For simplicity lets assume for a single "example" that after our input pipeline we are left with a tensor with shape [2,3], perhaps:
[[0,0,1],[1,0,0]]

and let us say that contextually we will refer to this feature as "main_input".
Then we now need to wrap this into a (Sequence)Example (NOTE: see this S.O. post for the lack of clarity the documentation provides on how to best do this)
# all of input pipeline here

# processed data
main_input = np.array([
    [0, 0, 1],
    [1, 0, 0],
])

# wrap each channel in Feature then wrap all in FeatureList
main_input_feat_list = tf.train.FeatureList(
    feature=[
        tf.train.Feature(float_list=tf.train.FloatList(value=value))
        for value in main_input
    ]
)

# wrap in SequenceExample
example = tf.train.SequenceExample(
    context={
        # other stuff could go here
    },
    feature_lists=tf.train.FeatureLists(feature_list={
        'main_input': main_input_feat_list,
        # other stuff could go here
    })
)

# write to record
with tf.python_io.TFRecordWriter('demo_example.tfrecord') as writer:
    writer.write(example.SerializeToString())

from here, we now need to unwrap this from TF Record in our Estimator's input_fn using a different api:

def parse_record(record):
    return tf.parse_single_sequence_example(
        record,
        context_features={
            # get context features here
        },
        sequence_features={
            'main_input': tf.FixedLenSequenceFeature((3, ), tf.float32), # <---- REDUNDANT
            # get other sequence features here
        }
    )

def input_fn(files:list=['demo_example.tfrecord'], params:dict):
    dataset = tf.data.TFRecordDataset(files).map(lambda record: parse_record(record))
    # reformat dataset to return feature, label pairs
    return dataset

Now if we want to use tf.estimator.BestExporter we need a serving_input_receiver_fn so lets go ahead and get that written:
def serving_input_receiver_fn():
    batch_size = None
    main_input = tf.placeholder(tf.float32, (batch_size, 2, 3), name='main_input') # <---- REDUNDANT

    features = {'main_input': main_input}

    # unclear what the difference is between features and receiver tensors
    receiver_tensors = features
    return  tf.estimator.export.ServingInputReceiver(features, receiver_tensors)

exporter = tf.estimator.BestExporter(
    name='best_exporter',
    serving_input_receiver_fn= serving_input_receiver_fn,
    exports_to_keep=3
)

So while the input pipeline should be decoupled from the model, the model's first layer at the very least is defined by the input. Therefore I should not have to three different times, in three different ways tell TensorFlow what my input features are.

        # 1st to write to TF Records
        feature_lists=tf.train.FeatureLists(feature_list={
            'main_input':  tf.train.FeatureList(
                feature=[
                    tf.train.Feature(float_list=tf.train.FloatList(value=value))
                    for value in main_input
                ]
            ),
            # other stuff could go here
        })

        # then to get it out of TF Records with an asymmetric api
        #...
        'main_input': tf.FixedLenSequenceFeature((3, ), tf.float32)
        #...

        # and once more I have to define placeholders for the features in the serving_input_receiver_fn
        main_input = tf.placeholder(tf.float32, (batch_size, 2, 3), name='main_input') # <---- REDUNDANT

Proposal to solve
In this colab I demonstrate the fledgling idea of how to solve this, named "FIO" (Feature Input / Output) whereby instead of having to redefine everything many times over with different apis depending on the case, you can define everything once in a data "schema" e.g.
# features here are of an actual example (not batched)
features = {
    'my-feature': 'hi',
    'seq': np.array([
        # ch1, ch2, ch3
        [   1,   1,  1], # element 1
        [   2,   2,  2], # element 2
        [   3,   3,  3], # element 3
        [   4,   5,  6]  # element 4
    ])
}

# here we specify what is needed to encode and decode from `(Sequence)Example` and `TF Records`

SCHEMA = {
    'my-feature': {'length': 'fixed', 'dtype': tf.string,  'shape': []},
    'seq': {
        'length': 'fixed',
        'dtype': tf.int64,
        'shape': [4, 3],
        'encode': 'channels',
        'channel_names': ['A', 'B', 'C'],
        'data_format': 'channels_last'
    }
}

then

# define our converter
fio = FIO(SCHEMA, etype='example')

# write to example
file = 'example.tfrecord'

example = fio.to_example(features)
with tf.python_io.TFRecordWriter(file) as writer:
    writer.write(example.SerializeToString())

# use a sequence example instead
fio = FIO(SCHEMA, etype='sequence_example', sequence_features=['seq'])
file = 'sequence_example.tfrecord'
with tf.python_io.TFRecordWriter(file) as writer:
    writer.write(example.SerializeToString())

where in both cases we can read from records with
tf.data.TFRecordDataset(DATASET_FILENAMES).map(lambda r: fio.from_record(r))

Thus I propose that TensorFlow add's a new component to the estimator class, which would be the bride between the input pipelines / serving input functions and the model_fn. This would be something like the data schema.
A user can define a data schema instance once and import it into the input pipeline to write their inputs to TF Record, import the same schema into their Estimator's input_fn to read from TF Records (or change the TF Record file format slightly so that you do not have define how it should be read, like how I do not have to define the keys in a json file to read a json file) and then once more it can be passed to (or automatically used by the Estimator) for the serving_input_receiver_fn.
I believe letting users declaratively, upfront, define what the Estimator accepts in terms of data and then not forcing them to, at least trice, re-write what the data is, would be a great simplifier and clean up the Estimator api
Will this change the current api? How?
Technically, everything "under the hood" could remain the same. The changes would be made to tf.data to introduce a Schema class which would work as a unifier across the multiple ways of encoding / decoding tf/np tensors to / from TF Records. In addition, the Estimator class would be updated to accept a tf.data.Schema instance, from which it would automate the redundancy described above.
Who will benefit with this feature?
Anyone who uses the Estimator api, anyone who uses TF Records, anyone who acknowledges that TensorFlow's documentation is far from complete and examples are needed to bridge how the "high" level apis quite often interface directly with lower (or even the lowest) level apis (e.g. Estimators which deal with Records or users who want to have an exporter which requires a ServingInputReceiver)
Any Other info.
Please look at FIO and generalize this to work as described above.
See these S.O. Posts and their linked Colab's to follow my journey in trying to decipher what should be a high level api, but it turns out to be fairly tethered to lower features.
optimal way to store tensors at TF Records
recovering TF Records
early stopping
what are serving_input_receiver_fn
using an estimator after training
defining estimator spec throws error

@wt-huang thoughts?

@karmel thoughts?

In TensorFlow 2.0, we have consolidated and simplified saving/serving logic. Please see https://github.com/tensorflow/community/blob/master/rfcs/20190509-keras-saved-model.md , and specifically the section on signatures.

@karmel I do not see how that is relevant to the redundancy of converting data to Examples, writing to Records, reading from Records, and parsing Records discussed in this issue.

It seems that one of the primary endpoints is the ServingInputReceiver, which is being de-emphasized. More broadly, in 2.0, many of these usability issues are resolved using the Keras framework, so we are reticent to change the existing behavior of the Estimator framework. For your purposes, I would recommend hosting the suggested class in your lib, with some added functionality to unwrap and rewrap for passing through estimator.

@karmel I understand that the emphasis is to move to TF 2.0..
However, I hope you can also acknowledge why that may not be a sufficient answer.
To my knowledge (and I may very well be wrong), TF 2.0 is still in beta. The key benefits of estimators (e.g. serving and scalable performance), are not fully mirrored in this major release. Further, this is about standardizing the interface of moving data into and out of tf records, which the keras api does not solve.
Some high level experimental classes like CsvDataset will help in some cases, but there will still be a need for many to work at the example / record level and fixing the inconsistency or at least bolstering the documentation would be in everyone's best interest.
Consider this StackOverflow question:  https://stackoverflow.com/questions/52035692/tensorflow-v1-10-store-images-as-byte-strings-or-per-channel
There is a multitude of ways of encoding the same tensor into an (Sequence)Example and no where is it stated which way is most optimal for performance, storage, or recovery. Further still, SequenceExample seems to be a superset of the Example functionality (give "context_feature" and "sequence_features"). So it could even be argued to drop Examples all together.
Unless you (TF as an organization) plans on closing of TF Records from the user, then the major release of 2.0 will not solve this issue.
I think, perhaps, this issue could be renamed as "Reduced redundancy in TF (Sequence)Example, TF Records, and parsing thereof".  I initially stated it as relating to the Estimator API as that was the context in which I was using tf records.

@SumNeuron,
Sorry for the delayed response. In the Tensorflow Version 2.x, since we use TF Keras predominantly and don't use Estimators much, can you please let us know if this Feature is still relevant? Thanks!

authors: SumNeuron karmel rmothukuru


text:
Euclidean Distance Transform

Given the importance and wide use of the distance transform of binary images in different computer vision applications, I wonder if it would be possible to add a Euclidean distance transform with gradient support.
There exists very nice O(n) solutions already implemented in C/C++. For example. Opencv has implemented this paper here.
Here's another fast implementation in PyDIP.
This would be a tremendously useful feature. Thanks!

@martinwicke @ebrevdo @rmlarsen do you know if there's a TensorFlower who'd be interested in this?

I am  a beginner contributer?Is this suitable  for  first timer?Which  lebel  for  first time or beginner in  tensorflow?

Unfortunately I do not think this is a good project for a beginner. This is probably best for someone with experience writing TensorFlow ops.

Are there any updates on this? This will be an amazing feature when available!

Following :)

Hey, guys. Should be done within this week. I'll update this thread once I open a PR.

Thank you so much for your great work.

Is there an update on this feature by any chance? Thanks!

@skye Sorry about the delay, I've created a PR with a barebones implementation. I'll add faster algorithms and test cases over the next few hours.

Great job !  How would we be able to access this custom_op ?

@ahatamiz
The changes haven't been merged into the main repository yet, so clone my repository, and switch to this branch: euclidean-distance-transform.
Here's the build guide for tensorflow_addons.
Then you can use the transform as such:
from tensorflow_addons.custom_ops.image import euclidean_dist_transform

x = ... # binary image
y = euclidean_dist_transform(x)

Note:

x must be a binary image (0 and 1 only)
x must be of type tf.uint8. Use tf.cast if it isn't.
x must be of shape [N, H, W, 1] or [H, W, 1]
The dtype of y can be set like this: y = euclidean_dist_transform(x, dtype=tf.float32). tf.float32 is the default.
The current algorithm is of quadratic complexity O(N^2) where N is the number of pixels. I'm working on the faster O(N) algorithm right now, and I'll push it soon.
The Op is currently CPU only.
The Op is not differentiable, and running tf.gradients(y) will return [None].

Let me know if you find any problems or issues with the Op.

@Squadrick
That's amazing ! this op will be tremendously helpful for many researchers and I am sure people will come to appreciate its importance soon !
The only concern though is the fact that Op is non-differentiable. It would be great if we could have gradients defined for this op.
In many applications, this op will be needed for connecting two layers in a sense that it initializes the second layer with the output of the first layer. In essense, both of the layers have learn-able parameters thus not being able to back-propagate the of error through this op would limit its use.

@Squadrick awesome! Once it's ready with full test cases etc., maybe ping @sofienbouaziz to take a look.

@skye Sure thing.
@ahatamiz Couldn’t really figure out what the gradient of the operation would be, considering the fact that the function is defined on binary images. If you could help out with that part, I can implement fairly easily.

@Squadrick Thanks for the contribution! Just quickly parsing through the code I have the same comment than @ahatamiz that this op is non-differentiable and it would be great if we could back-propagate through it. The algorithm currently used is O(n^2)  is the goal to then implement a O(n) algorithm?

@sofienbouaziz I'm working on the O(n) algorithm right now, I'll push it within a day.
Like I mentioned before, I'm not sure how this op can be differentiable considering the fact that the op is defined on binary images.

@Squadrick Awesome for the O(n) version! Let me think about how to differentiate this op during the week-end. Just out of curiosity do you think it would be possible to develop this op purely in python?

@sofienbouaziz I think @Squadrick is right that gradients make no sense for a binary input. However, there is an generalization for arbitrary functions (that you and @ataiya implemented?) that if I understand correctly should be differentiable. If there was an op for this you would also get the binary version for free by composing it with an indicator function.
I'm not sure if the generalized version would do exactly what @ahatamiz wants, but it would be very nice to have an op for.

As an experiment I tried implementing a simple version of the generalized transform in python, and it is indeed differentiable. With a careful choice of indicator function it seems you can even "learn" a binary image by supervising its distance transform, so it might be suitable for use as a layer in a network as @ahatamiz suggested.
Below is an example showing the gradients. Here, the mask is learned from an L2 loss between the DT of the mask and the DT of the ground truth mask.

Thanks @Squadrick for you great work on this and thanks @sofienbouaziz for taking a look into this. Also thanks for your thoughtful contribution to the discussion @drebain
@Squadrick, you are right in the sense that the op may not be differentiable if we were to utilize on binary images. However,  what I had in mind is very close the implementation by @drebain. One can pass the output of sigmoid, before thresholding, to a (generalized) distance transform layer and still allow for gradient flow. This is useful since in most vision applications, the output of sigmoid (or softmax) can be fed into this layer for more interesting operation down-stream.
The implementation by @drebain is definitely correct, but unfortunately it runs out of memory on bigger images ( like 256 by 256 ) on a GPU like Titan XP and also is not as fast, due to its O(n^2) complexity.

According to its documentation, the default O(n) L2 implementation in OpenCV uses the multi-pass algorithm based on the generalized transform, so if @Squadrick has implemented that, then I think all that would need to be done is to expose a version of the op that doesn't apply the thresholding indicator function.

@ahatamiz
The changes haven't been merged into the main repository yet, so clone my repository, and switch to this branch: euclidean-distance-transform.
Here's the build guide for tensorflow_addons.
Then you can use the transform as such:
from tensorflow_addons.custom_ops.image import euclidean_dist_transform

x = ... # binary image
y = euclidean_dist_transform(x)

Note:

x must be a binary image (0 and 1 only)
x must be of type tf.uint8. Use tf.cast if it isn't.
x must be of shape [N, H, W, 1] or [H, W, 1]
The dtype of y can be set like this: y = euclidean_dist_transform(x, dtype=tf.float32). tf.float32 is the default.
The current algorithm is of quadratic complexity O(N^2) where N is the number of pixels. I'm working on the faster O(N) algorithm right now, and I'll push it soon.
The Op is currently CPU only.
The Op is not differentiable, and running tf.gradients(y) will return [None].

Let me know if you find any problems or issues with the Op.

@Squadrick . I cloned the branch you mentioned and built from the source, but it seems like we are missing a file:
/addons-euclidean-distance-transform/tensorflow_addons/custom_ops/image/python/_image_ops.so  cannot open shared object file: No such file or directory
Also, as @drebain  mentioned, I think it would be helpful to also allow for feeding logits before sigmoid or softmax and not just binary images ( for a back-propagating error ) . Currently, this layer seems to only accept binary images.

@ahatamiz I haven't gotten to fully review the PR, but I recognize that error and you likely just need to move out of the src directory after pip install. The .so files aren't built until bazel compile and your current working directory does not contain them

@seanpmorgan Yes. You were right and it worked.
Also, it seems like this layer requires TensorFlow version >= 2. Would it be possible to allow older versions like 1.9 for now ?  @Squadrick

@ahatamiz I haven't gotten to fully review the PR, but I recognize that error and you likely just need to move out of the src directory after pip install. The .so files aren't built until bazel compile and your current working directory does not contain them

Thanks @seanpmorgan . However I also have an issue with version incompatblity as only Tensorflow 2.0.0 is allowed. I have not yet transferred everything to Tensorflow 2.0.0 and was hoping to get it working on older versions.

@ahatamiz Ah, thats a bit of a problem. Addons is only scoped to be TF 2.x+ and it'll take a lot of convincing to get us to reconsider that. You can however make it work on your own fork of the repo by altering this line:
https://github.com/tensorflow/addons/blob/master/tensorflow_addons/__init__.py#L54

@drebain Very cool colab! Indeed the code you are referring to is an old implementation I did of this particular paper and as you pointed out it is differentiable. Yes, you are right that the gradient doesn't make sense in the case of binary image, however I was thinking we could do something similar than what has been done for quantization where the forward pass would be computed on the binary image and the backward pass on some sort of relaxation.

I am closing this issue as the Euclidean Distance Transform op was merged. If updates are needed let's reopen one with the necessary details.
@Squadrick Thanks for this very useful op.

Thanks you @Squadrick  and @sofienbouaziz .
Is it also possible to provide GPU support for this op ?

@ahatamiz GPU support for TensorFlow Addons is currently WIP. Ref #294. Once this is merged, I'll add GPU support.

Hi,
I just want to confirm.  Can the current implementation of the TF addons Euclidean Distance Transform be used in a loss function?  I noticed the discussion above on the issue of gradients when the distance transform is applied on binary images.  Was this resolved?

@emckenzi123 No. The current implementation of euclidean distance transform does not have a gradient defined.

@emckenzi123 If you do not need to run on large images, the code in my notebook from above ( https://colab.research.google.com/drive/1-Fykc6FVi3pUtFx7spP3OAdn267TKwbD ) might work as a loss function. You do need to allow arbitrary real values for pixels (not just binary) in order for the gradients of a distance transform to be meaningful though.

@drebain I tried your posted method (thank you for making that available) however I ran into an OOM error since my images are large.  I looked into the paper you linked to, and found a python version here.  It uses Cython, and I naively tried to just use this code in a training loop's loss, without success (no training of weights).  I went through and tried to replace ops with tensorflow ops to see if I could get something, but that just ended up being really slow.  I should note that if I use the Cython code outside of tensorflow, it's super fast.  I could try to keep working on this, or do you guys think it's actually non-trivial to turn the linked python code into something TF can use in a loss function?

@emckenzi123 I haven't used Cython and TF together before, but I think you could write a wrapper around the code you linked that manually defines the gradient using tf.py_function and tf.custom_gradient. If I'm reading correctly that the code gives you the index of the pixel corresponding to the minimum of d(p,q)+f(q) you should be able to compute the gradient. That would let you keep the Cython implementation I think, while still being differentiable.

Thank you so much for your help.  I defined the gradient to be 2 * sqrt(distance transform), since the derivative of (p-q)**2 is 2 * (p-q).
In case someone in the future runs into something similar, I have used @drebain 's previous colab notebook as a template to demonstrate the code for this.  The link is here

authors: FrankSD Squadrick ahatamiz drebain emckenzi123 seanpmorgan sendeb skye sofienbouaziz


text:
TF2.0 distribution strategy hanging on keras

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): NO
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS Linux release 7.5.1804
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: NA
TensorFlow installed from (source or binary): source
from tf2.0 branch
TensorFlow version (use command below): 2.0.0a0
Python version: 3.5.2
Bazel version (if compiling from source): 19.2
GCC/Compiler version (if compiling from source): 6.3
CUDA/cuDNN version: NA
GPU model and memory: NA

Describe the current behavior
Trying to run simple example on cpu in distributed mode according to the documentation.  tf2.0 documentation
If I run the code using only python py_file.py then it is running fine.
If I run it by setting TF_CONFIG environment variable with only 2 different node or even a single node then it is hanging.
It hangs after generating this information:
2019-03-27 11:44:19.963538: I tensorflow/core/platform/cpu_feature_guard.cc:145] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance critical operations:  AVX512F FMA
To enable them in non-MKL-DNN operations, rebuild TensorFlow with the appropriate compiler flags.
2019-03-27 11:44:19.976632: I tensorflow/core/common_runtime/process_util.cc:92] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
WARNING: Logging before flag parsing goes to stderr.
W0327 11:44:19.979773 47985184916416 deprecation.py:506] From /panfs/users/mdkamruz/anaconda3/envs/py352_otf2.0_source/lib/python3.5/site-packages/tensorflow/python/ops/init_ops.py:1257: calling VarianceScaling.init (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
W0327 11:44:20.015144 47985184916416 distribute_coordinator.py:825] eval_fn is not passed in. The worker_fn will be used if an "evaluator" task exists in the cluster.
W0327 11:44:20.015259 47985184916416 distribute_coordinator.py:829] eval_strategy is not passed in. No distribution strategy will be used for evaluation.
2019-03-27 11:44:20.021537: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:250] Initialize GrpcChannelCache for job worker -> {0 -> localhost:3002}
2019-03-27 11:44:20.027697: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:359] Started server with target: grpc://localhost:3002
2019-03-27 11:44:20.027718: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:363] Server already started (target: grpc://localhost:3002)
W0327 11:44:20.095507 47985184916416 distribute_coordinator.py:825] eval_fn is not passed in. The worker_fn will be used if an "evaluator" task exists in the cluster.
W0327 11:44:20.095634 47985184916416 distribute_coordinator.py:829] eval_strategy is not passed in. No distribution strategy will be used for evaluation.
W0327 11:44:20.097250 47985184916416 distributed_training_utils.py:933] ModelCheckpoint callback is not provided. Workers will need to restart training if any fails.
Describe the expected behavior
It should run smoothly as described in the documentation.
Code to reproduce the issue
Python code:
dataset = tf.data.Dataset.from_tensors(([1.], [1.])).repeat(100).shuffle(buffer_size=10000).batch(10)
# single node
ds_strategy=tf.distribute.MirroredStrategy()
# cross_device_ops=tf.distribute.HierarchicalCopyAllReduce()
# multinode
# ds_strategy = tf.distribute.experimental.MultiWorkerMirroredStrategy()
with ds_strategy.scope():
    model = tf.keras.Sequential([tf.keras.layers.Dense(1, input_shape=(1,))])
    model.compile(loss='mse', optimizer='sgd')
    model.fit(dataset,epochs=200)

TF_CONFIG environment variable:
export TF_CONFIG='{
    "cluster": {
          "worker": ["IP_ADDR:3001"]
          },
    "task": {"type": "worker", "index": 0}
}'
# for single node multi worker
export TF_CONFIG='{
    "cluster": {
          "worker": ["IP_ADDR:3001","IP_ADDR:3002"]
          },
    "task": {"type": "worker", "index": 0}
}'
# for multinode
# node 1 and similar for node 2
export TF_CONFIG='{
    "cluster": {
          "worker": ["IP_ADDR_1:3001","IP_ADDR_2:3002"]
          },
    "task": {"type": "worker", "index": 0}
}'

Other info / logs
Running on cpu, machine don't have any gpu

According to the provided document.
tf.distribute.MirroredStrategy support synchronous distributed training on multiple GPUs on one machine.
So running dist strat on cpu does not make sense...

According to the provided document.
tf.distribute.MirroredStrategy support synchronous distributed training on multiple GPUs on one machine.
So running dist strat on cpu does not make sense...

The nccl leads to tensorflow hang all the time no matter local training or multiworker training.

The tutorial you have referenced works fine using google colab. Please give it a try using google colab. Thanks!

The tutorial you have referenced works fine using google colab. Please give it a try using google colab. Thanks!

It seems that google colab do not provide GPU, so TensorFlow will not use nccl library to reduce, it can not  repeat the problem.

You can select gpu accelerator under edit > notebook settings > hardware accelerator

But there is only one GPU in google colab, in which case tensorflow will not choose nccl to do reduction.

Not sure if the same issue, but I also experienced frequent hanging when using MultiWorkerMirroredStrategy with nccl. TF 2.0.0-alpha0 uses nccl 2.3.5-5. I modified the tensorflow source to use 2.3.7-1 instead and that seems to have fixed the issue. The release notes for 2.3.7-1 mentions

Fixed a hang during bootstrap due to socket reuse.

I used master branch source code to build tensorflow 2.0 and seems it is working. Hence closing the issue.

authors: Keepmoving-ZXY guoshimin lcycoding md-k-sarker ymodak


text:
Op request tensorflow lite

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
TensorFlow installed from (source or binary):
TensorFlow version (or github SHA if from source):

Provide the text output from tflite_convert
Some of the operators in the model are not supported by the standard TensorFlow Lite runtime. If those are native TensorFlow operators, you might be able to use the extended runtime by passing --enable_select_tf_ops, or by setting target_ops=TFLITE_BUILTINS,SELECT_TF_OPS when calling tf.lite.TFLiteConverter(). Otherwise, if you have a custom implementation for them you can disable this error with --allow_custom_ops, or by setting allow_custom_ops=True when calling tf.lite.TFLiteConverter(). Here is a list of builtin operators you are using: ADD, CAST, CONCATENATION, CONV_2D, DEPTHWISE_CONV_2D, EXP, EXPAND_DIMS, FILL, GATHER, GREATER, GREATER_EQUAL, LESS, LOGICAL_AND, LOGISTIC, MAXIMUM, MINIMUM, MUL, PACK, PAD, RANGE, RESHAPE, RESIZE_BILINEAR, SELECT, SHAPE, SLICE, SPLIT, SQUEEZE, STRIDED_SLICE, SUB, SUM, TILE, TOPK_V2, TRANSPOSE, UNPACK, WHERE, ZEROS_LIKE. Here is a list of operators for which you will need custom implementations: Enter, Exit, LoopCond, Merge, NonMaxSuppressionV3, Switch, TensorArrayGatherV3, TensorArrayReadV3, TensorArrayScatterV3, TensorArraySizeV3, TensorArrayV3, TensorArrayWriteV3.

Also, please include a link to a GraphDef or the model if possible.
Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

@kishorekumarsingh,
Provide the exact sequence of commands / steps that you executed before running into the problem. And also, provide Tensorflow version. Thanks!

This is the command that I used:
tflite_convert  --graph_def_file=frozen_inference_graph.pb  --output_file=new_graph.tflite  --input_format=TENSORFLOW_GRAPHDEF  --output_format=TFLITE  --input_shape=1,224,224,3  --input_array=image_tensor  --output_array=detection_boxes,detection_scores,detection_classes,num_detections  --inference_input_type==QUANTIZED_UINT8 --inference_type=FLOAT  ----mean_values=128  --std_dev_values=128
My tensorflow version :
1.14.0

@kishorekumarsingh
Is this still an issue

Can you try this: https://www.tensorflow.org/lite/guide/ops_select
Feel free to reopen if you run into any issue.


text:
Can not use large dimension in Embedding layer on GPU(s).

I am using TF2.0 latest nightly build and I am trying to train LSTM model for text classification on very large dataset of 16455928 sentences. For embedding layer in the model, I have a vocab size of 366856 and I used 1000 as embedding dimension value in it, on which the 2 GPUs(Tesla T4 from Google) ran out of memory.
Since I can not lower the size of vocabulary (maybe there is a way), so I used lower value for embedding dimension (100) on which the model starts training. Now my question is if there is a way I can use higher value of embedding dimension?. Maybe by putting set of layers of my model on different GPUs, if so then what is the way in TF2.0? Also, will using more number GPUs help? Thank you!

Hi @rishabhsahrawat,
Could you be more precise as to when the issue occurs exactly, and what your data processing pipeline looks like (i.e. do you have the entire dataset loaded in memory, or do you iteratively load and discard pieces of it)?
I would not be surprised if your issue was similar to the one I reported in #30952 (which has not been picked up as of now...), but maybe it is a simpler dataflow management issue. In the former case, you might want to try to run after disabling Eager execution (which is not a fix to the issue, but could be a work-around...).

Hi @pandrey-fr , thank you for your questions. I am actually using tf.data.Dataset (reading a csv file using make_csv_dataset). After that I created an encoder using tfds.features.text.TokenTextEncoder which gives a vocabulary size of 366856 (unique words). This vocab size is then given to Embedding layer of my model along with the embedding size of 1000 which on running the model throws huge log of error but following lines were interesting for me

Resource exhausted:  OOM when allocating tensor with shape[366856, 1000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

So, if I lower the embedding size to <=100, then everything works without any error.
I can not say maybe my the problem is because of memory leak and I hope they fix it soon. Anyways in the meantime, I will try running after disabling Eager execution.

So, if I lower the embedding size to <=100, then everything works without any error.

To be clear, does this include the fitting process of the model? If so, then it could just be that the data is indeed too large for the amount of RAM on your GPU. Could you describe your GPU's specs (name, RAM)?
You could also try running your code without enabling GPU use (run `tf.config.experimental.set_visible_devices([], 'GPU') at the beginning of your code) and monitor how much RAM is being used to get a notion of how much memory your model and data require, and whether this is a stable amount throughout the training cycle or an increasing one (which would make it similar to my issue).

Maybe by putting set of layers of my model on different GPUs, if so then what is the way in TF2.0?
I had not noticed this part of the question, but this could be a strategy. I believe you then need to use the tf.device context manager when defining your operations to force their placement on this or that GPU, e.g. something that would look like:

# list available GPUS, make sure you have at least two
gpus = tf.config.experimental.list_logical_devices('GPU')
assert len(gpus) >= 2
# place the embedding layer on the first GPU
with tf.device(gpus[0]):
    embedded = tf.keras.layers.Embedding(input_dim, output_dim)(inputs)
# place the rest of the model on the second GPU
with tf.device(gpus[1]):
    output = some_layers_stack(embedded)

Yes, this happens after calling model.fit, if that was your question and my error log is also similar like the one you have mentioned in your issue. I am using 2 Tesla T4 GPUs each with 16 GB memory from Google.
As per your suggestion, I will run it on CPU and see the memory consumption. However, I want to share one information (not sure if this will be helpful). Earlier, I was trying to shuffle the dataset using dataset.shuffle()which fills up the buffer and memory of CPU before running for the first epoch. Eventually, this buffer filling up process uses all 16 GB RAM of my CPU and after this nothing runs. I even raised an issue here. Not sure if this is a normal behaviour.
At the end you have mentioned a sample code for placing model layers on different GPUs, I have a question regarding this. Do you think it will work similarly like tf.distribute.MirroredStrategy() from here? Regardless, I will try this approach too. Thank you!

Earlier, I was trying to shuffle the dataset using dataset.shuffle()which fills up the buffer and memory of CPU before running for the first epoch. Eventually, this buffer filling up process uses all 16 GB RAM of my CPU and after this nothing runs.

Okay, so basically your dataset (multiplied by embedding dimension) is huge, and the embedding vectors representing it cannot be all loaded in memory (either CPU or GPU) at the same time. For your model to run, you therefore need to keep the amount of data loaded at any given time under 16 GB (which is the limit on both your CPU and GPU, notwithstanding the idea of using multiple parallel devices).
If we make the (possibly strong) assumption that there is no actual TensorFlow issue, this should be achievable (with a large embedding dimension) by tweaking your tf.data.Dataset object - basically, you want to load a small part of the data (which makes up for a few batches), shuffle and padded-batch it, feed it to your model, discard it and go on doing the same with the next bit. Are you still using the dataset from the tutorial? If so, I will try to write you a bit of code later on tonight.
Now, once this will be dealt with, we should watch for any memory increase during training (despite the dataflow), which would indicate an issue similar to mine, but hopefully your problem only comes from loading too much data at once.

At the end you have mentioned a sample code for placing model layers on different GPUs, I have a question regarding this. Do you think it will work similarly like tf.distribute.MirroredStrategy() from here?

From what I understand, tf.distribute.MirroredStrategy() creates a copy of your model on each available GPU, so as to run parallel fitting on various batches. If that is indeed the case, the amount of memory used on the first GPU should not decrease; instead, this strategy will boost training runtime by distributing the processing of training samples on parallel copies of the model (and still somehow gather unified weights update - I guess each would treat part of the batch, then gradients would be computed based on an aggregate of local losses, or maybe there is some rougher locally-computed updates aggregations scheme ; I have not looked into the details).
What I was suggesting (and I have absolutely no idea whether that would work) was to put distinct parts of the architecture on the various devices. It might not be possible, or induce data transmission overheads that greatly slow computations (and possibly do not yield the memory gain I had in mind) - so, if you try experimenting in that direction, I would be glad to hear about the results!

Yes, I am still using the same way for loading the dataset so padded_batch
etc. except shuffling since it doesn’t work for me as I said.
I will surely try out the way to divide the layers of the model and hope it
will work. I will keep you updated about that.
Thank you again for your suggestions and help.
-Rishabh
On Tuesday, July 30, 2019, Paul Andrey notifications@github.com wrote:

Earlier, I was trying to shuffle the dataset using dataset.shuffle()which
fills up the buffer and memory of CPU before running for the first epoch.
Eventually, this buffer filling up process uses all 16 GB RAM of my CPU and
after this nothing runs.
Okay, so basically your dataset (multiplied by embedding dimension) is
huge, and the embedding vectors representing it cannot be all loaded in
memory (either CPU or GPU) at the same time. For your model to run, you
therefore need to keep the amount of data loaded at any given time under 16
GB (which is the limit on both your CPU and GPU, notwithstanding the idea
of using multiple parallel devices).
If we make the (possibly strong) assumption that there is no actual
TensorFlow issue, this should be achievable (with a large embedding
dimension) by tweaking your tf.data.Dataset object - basically, you want
to load a small part of the data (which makes up for a few batches),
shuffle and padded-batch it, feed it to your model, discard it and go on
doing the same with the next bit. Are you still using the dataset from the
tutorial
https://www.tensorflow.org/beta/tutorials/load_data/text#split_the_dataset_into_text_and_train_batches?
If so, I will try to write you a bit of code later on tonight.
Now, once this will be dealt with, we should watch for any memory increase
during training (despite the dataflow), which would indicate an issue
similar to mine, but hopefully your problem only comes from loading too
much data at once.
At the end you have mentioned a sample code for placing model layers on
different GPUs, I have a question regarding this. Do you think it will work
similarly like tf.distribute.MirroredStrategy() from here?
From what I understand, tf.distribute.MirroredStrategy() creates a copy
of your model on each available GPU, so as to run parallel fitting on
various batches. If that is indeed the case, the amount of memory used on
the first GPU should not decrease; instead, this strategy will boost
training runtime by distributing the processing of training samples on
parallel copies of the model (and still somehow gather unified weights
update - I guess each would treat part of the batch, then gradients would
be computed based on an aggregate of local losses, or maybe there is some
rougher locally-computed updates aggregations scheme ; I have not looked
into the details).
What I was suggesting (and I have absolutely no idea whether that would
work) was to put distinct parts of the architecture on the various devices.
It might not be possible, or induce data transmission overheads that
greatly slow computations (and possibly do not yield the memory gain I had
in mind) - so, if you try experimenting in that direction, I would be glad
to hear about the results!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/31162?email_source=notifications&email_token=ADZMMPDPP5VBX4VVGG66O2DQCBOIPA5CNFSM4IH3FFHKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3EL4SQ#issuecomment-516472394,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADZMMPBBTUPGO7Z6PYH6QL3QCBOIPANCNFSM4IH3FFHA
.

--
Sent from Gmail iPhone

Hi Rishabh,
Looking further at the tutorial (whose dataset is way smaller than yours, if I understand well) and doing a few tests on my own, I must say I am surprised by the issues you are encountering...
Normally, setting a reasonable buffer_size argument when using shuffle should solve memory issues associated with that part of the data pipeline. As for the embeddings matrix, it is indeed rather big, but when I tried allocating a similarly-shaped one on my system (using alternatively the GPU's 4GB dedicated memory or the 16 GB of RAM I have at my disposal) it fitted with just a slight warning about its size (basically it takes a bit less than 2 GB of memory space, which is a lot for a single weights matrix but should be tractable given your config).
Is there any chance you could share your dataset and code with me (optionally via a private channel) so that I can have a look? I suspect there might be something wrong in the dataflow that would explain your going out of memory, but I could be overly optimistic.
At any rate, an alternative way of decreasing the embeddings' dimensionality would be to decrease the number of tokens in your vocabulary, e.g. using a WordPiece tokenizer to break uncommon tokens down to knows ones (including phonetic tokens if needed). This way you might end up with a matrix with (way) less rows, which might also be a good thing for your modeling (depending on the amount of rarely-used tokens in the dataset).
Best,
Paul

Hi Paul, thank you for your response. My dataset size is 16455928 elements/sentences, so in order to make sure nothing that shuffling is perfect I must use either the same or greater value as the size of my full dataset for buffer_size (mentioned here) otherwise there might be some elements that the model might see never or more than once during an epoch.
I am stunned that it fit for you and just using 2 GB of memory. Just to be clear this is how my Embedding layers looks like tf.keras.layers.Embedding(input_dim = 366856, output_dim=150, input_length = 4) giving 55028550 parameters.
I am sorry but I can not share the code or dataset, as it does not belong to me. :(
Your suggestion for using WordPiece. After reading about it quickly I think it is more or less similar to stemming words i.e. playing, played etc. -> play. I am already applying Lemmatization and Stemming during preprocessing dataset.
Regards,
Rishabh

@rishabhsahrawat Will it be possible to create minimal reproducible code and share with us to move faster. Thanks!

My dataset size is 16455928 elements/sentences, so in order to make sure that shuffling is perfect I must use either the same or greater value as the size of my full dataset for buffer_size (mentioned here) otherwise there might be some elements that the model might see never or more than once during an epoch.

That is correct, however I guess an imperfect shuffling might be better than no shuffling at all, if the dataset comprises some order - but it might be already shuffled "by nature", or could probably be shuffled on disk outside of TensorFlow. I leave it up to you to see what suits best your usecase!

I am stunned that it fit for you and just using 2 GB of memory. Just to be clear this is how my Embedding layer looks like tf.keras.layers.Embedding(input_dim = 366856, output_dim=1000, input_length = 4) giving 366857000 parameters. This results in error. However, tf.keras.layers.Embedding(input_dim = 366856, output_dim=100, input_length = 4) giving 36685700 parameters works for me.

I just tested again using your exact code; I agree on the number of parameters, but on RAM it still takes "only" 1.4 ~ 1.5 GB of RAM, at least after instantiation and feeding it a Tensor (to ensure the weights are indeed built)... But maybe there is something somewhere in your code that triggers the creation of multiple copies of the weights?

I am sorry but I can not share the code or dataset, as it does not belong to me. :(

No problem, that was to be expected.

Your suggestion for using WordPiece. After reading about it quickly I think it is more or less similar to stemming words

That is indeed partially similar, but there is also the breaking down to phonetics of uncommon tokens that, in my short experience, can greatly decrease vocabulary size - but again, this is merely an abstract suggestion and it is up to you to see whether it suits your needs given your data and application context :)
To conclude, it seems that you are encountering a memory issue, whose relying on the mere size of your model on a bug somewhere is still unsure...
Edit: after writing the code below I tried running it and I did get out of GPU memory during training (the first batch went well, the second triggered an allocation error).
@gadagashwini I leave it up to Rishabh to provide with more details about the model he is using, notably as to the output dimensionality, but I guess a first approximation would be:
import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Embedding(input_dim=366856, output_dim=1000, input_length=4),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(1000)),
    tf.keras.layers.Dense(500, activation='relu'),
    tf.keras.layers.Dense(64, activation='softmax')
])
model.compile('adam', 'sparse_categorical_crossentropy', ['sparse_categorical_accuracy'])

mock_inputs = tf.random.uniform((64, 4), 1, 366856, tf.int64)
mock_target = tf.random.uniform((64, 1), 0, 64, tf.int64)

model.fit(mock_inputs, mock_target, batch_size=32, epochs=5)

Hi @gadagashwini , I can not share the dataset unfortunately but I can share the model architecture example. These example layers also, from here throws same errors about memory full, GPU exhausted etc. Updated layers look like this-
model = tf.keras.Sequential()
model.add(tf.keras.layers.Embedding(input_dim = 366856, output_dim = 1000, input_length = 4))
model.add(tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(256,dropout = 0.5, recurrent_dropout = 0.5,kernel_regularizer = keras.regularizers.l2(0.001))))
model.add(tf.keras.layers.Dense(128, activation='relu'))
model.add(tf.keras.layers.Dense(685731, activation='softmax'))
model.compile(optimizer=tf.keras.optimizers.Adam(),
	          loss='sparse_categorical_crossentropy',
	          metrics=['accuracy'])

If I use output_dim <=150 in Embedding layer, then training starts without any problems. This was all on 2 GPUs.
Now I have access to 2 more GPUs (now 4 in total) and if I use the same model architecture the error has changed to this-
  File "model_with_tfsplit.py", line 99, in <module>
    history = model.fit(train_data, epochs=20, validation_steps = steps ,validation_data=test_data , callbacks = [callback_for_saving])
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 710, in fit
    use_multiprocessing=use_multiprocessing)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/training_distributed.py", line 680, in fit
    steps_name='steps_per_epoch')
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/training_arrays.py", line 296, in model_iteration
    batch_outs = f(actual_inputs)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/distribute/distributed_training_utils.py", line 836, in execution_function
    return [out.numpy() for out in distributed_function(input_fn)]
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/eager/def_function.py", line 417, in __call__
    self._initialize(args, kwds, add_initializers_to=initializer_map)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/eager/def_function.py", line 360, in _initialize
    *args, **kwds))
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/eager/function.py", line 1749, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/eager/function.py", line 2053, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/eager/function.py", line 1939, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 795, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/eager/def_function.py", line 310, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 785, in wrapper
    raise e.ag_error_metadata.to_exception(type(e))
RuntimeError: in converted code:
    relative to /home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python:
    keras/distribute/distributed_training_utils.py:825 distributed_function  *
        outputs = strategy.experimental_run_v2(
    distribute/distribute_lib.py:753 experimental_run_v2
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    keras/engine/training.py:976 train_on_batch  *
        outputs = training_eager.train_on_batch(
    keras/engine/training_eager.py:303 train_on_batch
        output_loss_metrics=output_loss_metrics))
    keras/engine/training_eager.py:251 _process_single_batch
        model.optimizer.apply_gradients(zip(grads, trainable_weights))
    keras/optimizer_v2/optimizer_v2.py:435 apply_gradients
        self._create_slots(var_list)
    keras/optimizer_v2/adam.py:148 _create_slots
        self.add_slot(var, 'v')
    keras/optimizer_v2/optimizer_v2.py:587 add_slot
        initial_value=initial_value)
    ops/variables.py:258 __call__
        return cls._variable_v2_call(*args, **kwargs)
    ops/variables.py:252 _variable_v2_call
        shape=shape)
    ops/variables.py:63 getter
        return captured_getter(captured_previous, **kwargs)
    distribute/distribute_lib.py:1394 create_colocated_variable
        return next_creator(*args, **kwargs)
    ops/variables.py:63 getter
        return captured_getter(captured_previous, **kwargs)
    distribute/shared_variable_creator.py:69 create_new_variable
        v = next_creator(*args, **kwargs)
    ops/variables.py:63 getter
        return captured_getter(captured_previous, **kwargs)
    distribute/distribute_lib.py:1306 creator_with_resource_vars
        return self._create_variable(*args, **kwargs)
    distribute/mirrored_strategy.py:512 _create_variable
        values.SyncOnReadVariable, *args, **kwargs)
    distribute/distribute_lib.py:2272 create_mirrored_variable
        value_list = real_mirrored_creator(devices, *args, **kwargs)
    distribute/mirrored_strategy.py:504 _real_mirrored_creator
        v = next_creator(*args, **kwargs)
    ops/variables.py:63 getter
        return captured_getter(captured_previous, **kwargs)
    eager/def_function.py:348 variable_capturing_scope
        lifted_initializer_graph=lifted_initializer_graph, **kwds)
    ops/variables.py:260 __call__
        return super(VariableMetaclass, cls).__call__(*args, **kwargs)
    eager/def_function.py:140 __init__
        initial_value() if init_from_fn else initial_value,
    distribute/mirrored_strategy.py:463 initial_value_fn
        return array_ops.identity(init_value)
    util/dispatch.py:180 wrapper
        return target(*args, **kwargs)
    ops/array_ops.py:92 identity
        copied = input._copy()  # pylint: disable=protected-access
    framework/ops.py:921 _copy
        new_tensor = self._copy_nograd(ctx, device_name)
    framework/ops.py:914 _copy_nograd
        new_tensor = self._copy_to_device(context=ctx._handle, device=device_name)

    RuntimeError: Error copying tensor to device: . Dst tensor is not initialized.
When I use output_dim <= 150, it starts training without errors.
Now, I just found out if I use output_dim = 200, the error is same like on 2 GPUs with output_dim = 1000. Seems like the error log is changing also. :(

Hi @pandrey-fr , I just tried your dummy model. As you mentioned, training on single GPU also triggered allocation memory problem one time during epoch 2  like yours and one time during epoch 6. The CPU RAM it takes near 2-3 GBs but as soon as the fit model fit function is executed, the RAM usage increase and stops near 14-15 GBs.
I also then tried training it on multiple GPUs after adding MirroredStrategy() scope. On execution, it does use all my GPUs but then throws Assertion error as follows, even before starting first epoch.
  File "<stdin>", line 1, in <module>
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 710, in fit
    use_multiprocessing=use_multiprocessing)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/training_distributed.py", line 614, in fit
    epochs=epochs)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 2214, in _distribution_standardize_user_data
    assert isinstance(x, dataset_ops.DatasetV2)
AssertionError

Hi @rishabhsahrawat,
Thank you for the details and feedback ; let us hope the issue gets picked up by someone who can clarify what is going on, and especially disentangle expected behaviours from potential bugs.
Just a side note, but as your model's output dimension if the same as the input embedding one (I guess you are doing some kind of language modelling task, e.g. predicting masked or next tokens in a sentence), you might want to re-use the embedding matrix as a kernel on your output layer (this is notably what is done in Transformer models - well, not in the Google tutorial, but in the paper and reference implementations) ; this greatly diminishes the number of trainable weights :)
(note that you have to tweak a little to do that, either with a custom layer or the functional API, but it really is not much of a difficulty)

Just a side note, but as your model's output dimension if the same as the input embedding one (I guess you are doing some kind of language modelling task, e.g. predicting masked or next tokens in a sentence), you might want to re-use the embedding matrix as a kernel on your output layer

I am not sure if I understood it correctly, but I think you mean creating something like a 'word2vec' model? If not, can you also share some helpful link to understand it better?
You are right, I am doing tokens prediction.

UPDATE
I just tested the same model with output_dim= 1000 and also 150 on my Macbook Pro with 16 GB RAM and it trains without any problem. Of course the training is slow since it is only CPU. Now I don't what to do. I think the problems are somewhere in MirrorStrategy() and also training on GPUs.

Just a side note, but as your model's output dimension if the same as the input embedding one (I guess you are doing some kind of language modelling task, e.g. predicting masked or next tokens in a sentence), you might want to re-use the embedding matrix as a kernel on your output layer

I am not sure if I understood it correctly, but I think you mean creating something like a 'word2vec' model? If not, can you also share some helpful link to understand it better?
You are right, I am doing tokens prediction.
Thank you!

What I mean is doing something like this (with a possibly more-general implementation):
class SharedKernelSoftmax(tf.keras.layers.Layer):
    def __init__(self, kernel, bias_initializer='zeros'):
        self.kernel = kernel
        self.bias = self.add_weight(
            name='bias', shape=(tf.shape(kernel)[1],), dtype=kernel.dtype,
            initializer=bias_initializer
        )

    def call(self, inputs, **kwargs):
        output = tf.keras.backend.dot(inputs, self.kernel) + self.bias
        return tf.nn.softmax(output, axis=-1)

And using such a layer (instantiated by being passed a shared embedding matrix of the output vocabulary) instead of the final softmax in your model. That being said, I know see that your input and output layers have different dimensions (different vocabulary, I guess?), so this is actually not relevant here (it would be if your input and output vocabulary were the same, or if you used a sequence-to-sequence model where tokens in the output vocabulary are also embedded before being passed to a decoder model). Sorry!

I just tested the same model with output_dim= 1000 and also 150 on my Macbook Pro with 16 GB RAM and it trains without any problem. Of course the training is slow since it is only CPU. Now I don't what to do. I think the problems are somewhere in MirrorStrategy() and also training on GPUs.

Interesting... Could you monitor RAM usage and share some indications as to the amount of memory used when training on CPU?

Thank you for clarifying my confusion. As you have noticed the input and output layers do not have same dimensions. So it will not be useful, however since I am tokenizing the data so I am saving it as a token file which I can load later without tokenizing the whole dataset on each run.
I am now training the model with output_dim 1000 on CPU and I can see most of the memory is taken but the unused memory is changing to sometimes 3 MB, 600 MB and sometimes even 3 GBs. It is still running the first epoch right now. The %CPU usage is going even upto 1000. I am using top command.

@ymodak @gadagashwini I just found out if I train on 2 GPUs then it takes 1s per step to train but when I use all 4 GPUs it takes 2s per step. I think it should train faster on more GPUs.
I am following this for placing on particular devices.

If you use distribution strategy, could you  try tf.distribute.experimental.CentralStorageStrategy?

Hi @yuefengz , thank you for your suggestion. I tried this by implementing it in the same way as MirroredStrategy so like
central_strategy = tf.distribute.experimental.CentralStorageStrategy()
with central_strategy.scope():
--model def & compilation--

Read about it from here too.
Unfortunately, I am receiving following error, please correct me if my implementation is wrong.
Traceback (most recent call last):
  File "model_with_tfsplit.py", line 82, in <module>
    model.add(tf.keras.layers.Embedding(vocab_size, 150, input_length = 4))
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/training/tracking/base.py", line 458, in _method_wrapper
    result = method(self, *args, **kwargs)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/sequential.py", line 175, in add
    layer(x)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 687, in __call__
    self._maybe_build(inputs)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 2005, in _maybe_build
    self.build(input_shapes)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/utils/tf_utils.py", line 296, in wrapper
    output_shape = fn(instance, input_shape)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/layers/embeddings.py", line 134, in build
    constraint=self.embeddings_constraint)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 2175, in __setattr__
    if val.trainable:
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/ops/variables.py", line 475, in trainable
    raise NotImplementedError
NotImplementedError
2019-08-06 08:11:01.889111: I tensorflow/core/common_runtime/eager/execute.cc:585] Executing op DestroyResourceOp in device /job:localhost/replica:0/task:0/device:CPU:0

The error log is huge, so sharing only the last part of it. If you require, I can share the full log.

I have a question from all of you, @ymodak @gadagashwini @yuefengz , do you think if I use Tf1.14 I can achieve the task of training embedding layer with bigger dimension on multiple GPUs efficiently without any problems that I am facing in TF2.0?. If yes, then could you please share some helpful links for converting the code into TF 1.x. I am new in TF 1.x. Thank you!

I have another question. I want to continue training after loading the lastly saved model for which I am defining tf.keras.models.load_mode(my_model.h5) inside mirror.scope(). Since I already compiled it during training earlier so I don't compile it again as also mentioned here. On execution, I am receiving following error
File "model_with_tfsplit.py", line 94, in <module>
    model =tf.keras.models.load_model('TF_model_onfull_2_03.h5') # Loading for retraining
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/saving/save.py", line 138, in load_model
    return hdf5_format.load_model_from_hdf5(filepath, custom_objects, compile)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 187, in load_model_from_hdf5
    model._make_train_function()
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 2015, in _make_train_function
    params=self._collected_trainable_weights, loss=self.total_loss)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py", line 500, in get_updates
    grads = self.get_gradients(loss, params)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/optimizer_v2/optimizer_v2.py", line 391, in get_gradients
    grads = gradients.gradients(loss, params)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/ops/gradients_impl.py", line 158, in gradients
    unconnected_gradients)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/ops/gradients_util.py", line 541, in _GradientsHelper
    for x in xs
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/distribute/values.py", line 716, in handle
    raise ValueError("`handle` is not available outside the replica context"
ValueError: `handle` is not available outside the replica context or a `tf.distribute.Strategy.update()` call

Even if I choose a single GPU, it still ends up with the same error. If I do not use mirror.scope() then the training starts from beginning like it has never trained.

@rishabhsahrawat You might want to try tf.compat.v1.disable_v2_behavior() in TF2.0; basically, it will allow you to 1) see if the issues are caused by v2 behaviors (in which case you can then play with the finer mechanisms-disabling compatibility instructions in order to understand where the problem emerges from) and 2) see which parts of your code actively depend on v2 behaviors (which will thus raise exceptions and require some code changes).
In my limited experience, if you are using the keras API, you do not need to change that much things to run code compatible with TF1.14. The main changes are Eager being disabled (which disables a few things and requires stricter implementation of some others, notably as to type casting, which is somewhat more flexible with Eager enabled), and the necessity, when not using keras, to manually set up placeholders, session objects and initialization instructions; but the latter points are automated through keras backend, so you probably will not have to go through them.
Good luck, and if you run into specific problems, do ask for support :-)

Hi @ymodak any updates on the issue?. I am still struggling with it.

For the problem with CentralStorageStrategy, could you show a colab or something to reproduce the issue?

With the latest nightly build, I am able to use larger dimension (max 1000) but not bigger. Also, after raising the issue I started building the model in Tf1.14 and it also has the same issue only so value upto 400 works but not higher.

Hi Rishabh,
Generally, an embedding layer in Keras uses more RAM than you would expect. To get the RAM requirements do: no. tokens * no. dimensions * dtype * approx. 15. So for 400k tokens, 1k dimensions, 32 bit dtype you would need about 1.6 * 15 which is about 24 GB. This probably loads on the CPU because it is able to use your swapfile. If the weights were simply in a NumPy array it would only take up about 1.6 GB but Keras / TF is really greedy. I really wish it were less greedy because it would be so much faster to start up a model =(
Maybe PyTorch is better?

@rishabhsahrawat Closing as this is not a bug, but feature request for supporting sharded embeddings in sync training. Please raise a feature request as its working intended. Thanks!.

Hi @yuefengz , thank you for your suggestion. I tried this by implementing it in the same way as MirroredStrategy so like
central_strategy = tf.distribute.experimental.CentralStorageStrategy()
with central_strategy.scope():
--model def & compilation--

Read about it from here too.
Unfortunately, I am receiving following error, please correct me if my implementation is wrong.
Traceback (most recent call last):
  File "model_with_tfsplit.py", line 82, in <module>
    model.add(tf.keras.layers.Embedding(vocab_size, 150, input_length = 4))
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/training/tracking/base.py", line 458, in _method_wrapper
    result = method(self, *args, **kwargs)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/sequential.py", line 175, in add
    layer(x)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 687, in __call__
    self._maybe_build(inputs)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 2005, in _maybe_build
    self.build(input_shapes)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/utils/tf_utils.py", line 296, in wrapper
    output_shape = fn(instance, input_shape)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/layers/embeddings.py", line 134, in build
    constraint=self.embeddings_constraint)
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 2175, in __setattr__
    if val.trainable:
  File "/home/rishabh/.local/lib/python2.7/site-packages/tensorflow_core/python/ops/variables.py", line 475, in trainable
    raise NotImplementedError
NotImplementedError
2019-08-06 08:11:01.889111: I tensorflow/core/common_runtime/eager/execute.cc:585] Executing op DestroyResourceOp in device /job:localhost/replica:0/task:0/device:CPU:0

The error log is huge, so sharing only the last part of it. If you require, I can share the full log.

I have got the same error when using CentralStorageStrategy (2.4.1). It can be reproduced by changing the strategy in https://github.com/tensorflow/docs/blob/master/site/en/tutorials/distribute/parameter_server_training.ipynb.
@yuefengz has it been fixed?

@liyinhgqw @yuefengz do you fix it? I  got the same error changing MirroredStrategy to CentralStorageStrategy because of OOM, on python3.7, tensorflow2.4, linux.

authors: gadagashwini grofte liying9611 liyinhgqw pandrey-fr rishabhsahrawat saikumarchalla yuefengz


text:
Failed to build pip package with syntax error

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): centos
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): source
TensorFlow version: master branch
Python version: 3.6
Installed using virtualenv? pip? conda?:
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:

Describe the problem
when I try to build the tensorflow,with command
bazel-bin/tensorflow/tools/pip_package/build_pip_package $build_dir

I got the error
bazel-bin/tensorflow/tools/pip_package/build_pip_package: line 263: syntax error near unexpected token `done'

Provide the exact sequence of commands / steps that you executed before running into the problem
bazel-bin/tensorflow/tools/pip_package/build_pip_package $build_dir

Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

Ohhh, there goes a hot fix already
https://github.com/tensorflow/tensorflow/commit/75a63b0527bb0ca9604a4e3f5a941e8f689c5fc9


text:
Distribution Strategy running variables: NCCL error crashes CUDA

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary):
TensorFlow version (use command below): 2.0.0
Python version: 3.7.4
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: 10.0/7.6.2
GPU model and memory: Titan XP 12 gb x 8

Describe the current behavior
When using distribution strategy, the following code runs in graph mode, but fails in eager mode resulting in an NCCL error. Since I need persistent Variables to normalize my loss, the loss function was implemented as a callable keras layer, instantiated as a property of the keras model for which is compute the loss.
class HierarchicalLoss(tf.keras.layers.Layer):
    def __init__(self):
        super(HierarchicalLoss, self).__init__()

        self.running_N = {}
        self.running_sum = {}
        self.running_mean = {}

    def build(self, input_shapes):
            for layer in input_shapes['nodes']:
                var_scope = ['local_delta_loss' +  "_" + layer]

                self.running_sum[var_scope] = tf.Variable(
                    initial_value = 1.0,
                    dtype = tf.float64,
                    name = var_scope + '_' + 'running_sum',
                    trainable = False,
                    aggregation = tf.VariableAggregation.SUM)

                self.running_N[var_scope] = tf.Variable(
                    initial_value = 1,
                    dtype = tf.int64,
                    name = var_scope + '_' + 'running_N',
                    trainable = False,
                    aggregation = tf.VariableAggregation.SUM)
    def call(self, model_outputs):
        total_loss = 0.0
        for layer in model_output['nodes']:
            loss, sum, N = compute_loss_and_running_values(model_outputs[layer])
            self.running_N[layer].assign_add(N)
            self.running_sum[layer].assign_add(sum
            total_loss += 0.0
        return total_loss * self.running_N/self.running_sum

2019-10-31 00:57:17.308231: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at nccl_ops.cc:104 : Unknown: Error invoking NCCL: unhandled cuda error
2019-10-31 00:57:17.308341: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Unknown: Error invoking NCCL: unhandled cuda error
         [[{{node NcclAllReduce_7}}]]
         [[NcclAllReduce_5/_26]]
2019-10-31 00:57:17.308471: W tensorflow/core/common_runtime/base_collective_executor.cc:217] BaseCollectiveExecutor::StartAbort Unknown: Error invoking NCCL: unhandled cuda error
         [[{{node NcclAllReduce_7}}]]

If I change the assign_add call to +=, the model runs in eager mode, but yields the following error in graph mode
TypeError: An op outside of the function building code is being passed
a "Graph" tensor. It is possible to have Graph tensors
leak out of the function building context by including a
tf.init_scope in your function building code.
For example, the following function will fail:
  @tf.function
  def has_init_scope():
    my_constant = tf.constant(1.)
    with tf.init_scope():
      added = my_constant * 2

Is there a preferred way to have persistent variables within your model that are aggregated across multiple GPUs?
Describe the expected behavior
Code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate the problem.
Other info / logs
NCCL debug info
node05-ccncluster:220944:221234 [1] NCCL INFO NET/Socket : Using [0]enp96s0f0:10.102.2.200<0> [1]enp134s0:192.168.4.105<0>
node05-ccncluster:220944:221234 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).
node05-ccncluster:220944:221234 [1] NCCL INFO NET/IB : No device found.
NCCL version 2.4.7+cudaCUDA_MAJOR.CUDA_MINOR
node05-ccncluster:220944:224168 [1] NCCL INFO Setting affinity for GPU 1 to 0f,ff000fff
node05-ccncluster:220944:224167 [0] NCCL INFO Setting affinity for GPU 0 to 0f,ff000fff
node05-ccncluster:220944:224167 [0] NCCL INFO Channel 00 :    0   1
node05-ccncluster:220944:224167 [0] NCCL INFO Ring 00 : 0[0] -> 1[1] via P2P/direct pointer
node05-ccncluster:220944:224168 [1] NCCL INFO Ring 00 : 1[1] -> 0[0] via P2P/direct pointer
node05-ccncluster:220944:224167 [0] NCCL INFO Using 256 threads, Min Comp Cap 6, Trees disabled
node05-ccncluster:220944:224167 [0] NCCL INFO comm 0x7f7e38001b30 rank 0 nranks 2 cudaDev 0 nvmlDev 0 - Init COMPLETE
node05-ccncluster:220944:224168 [1] NCCL INFO comm 0x7f7e3c0010b0 rank 1 nranks 2 cudaDev 1 nvmlDev 1 - Init COMPLETE
node05-ccncluster:220944:224164 [0] NCCL INFO Launch mode Group/CGMD
node05-ccncluster:220944:224189 [0] NCCL INFO Setting affinity for GPU 0 to 0f,ff000fff
node05-ccncluster:220944:224190 [0] NCCL INFO Setting affinity for GPU 0 to 0f,ff000fff
node05-ccncluster:220944:224189 [0] NCCL INFO Channel 00 :    0   1
node05-ccncluster:220944:224189 [0] NCCL INFO Ring 00 : 0 -> 1 via P2P/common device
node05-ccncluster:220944:224190 [0] NCCL INFO Ring 00 : 1 -> 0 via P2P/common device
node05-ccncluster:220944:224189 [0] NCCL INFO Using 256 threads, Min Comp Cap 6, Trees disabled
node05-ccncluster:220944:224189 [0] NCCL INFO comm 0x7f7e44001550 rank 0 nranks 2 cudaDev 0 nvmlDev 0 - Init COMPLETE
node05-ccncluster:220944:224190 [0] NCCL INFO comm 0x7f7e30001040 rank 1 nranks 2 cudaDev 0 nvmlDev 0 - Init COMPLETE

node05-ccncluster:220944:224164 [0] external/nccl_archive/src/enqueue.cc:74 NCCL WARN Cuda failure 'invalid device ordinal'
node05-ccncluster:220944:224164 [0] NCCL INFO external/nccl_archive/src/enqueue.cc:175 -> 1
node05-ccncluster:220944:224164 [0] NCCL INFO external/nccl_archive/src/enqueue.cc:437 -> 1

After some debugging, I think this might have to do with attempting to aggregate the int64?

Hi - can you provide the entire model + training code that you're using above? That will help us reproduce and debug the issue.
The all reduce is most likely triggered from the .assign_add() which will try to aggregate the delta before applying to the mirrored variables. But it's not clear to me why it fails. (because similar things happen for gradient aggregation as well and presumably those are not failing for you?)
When you eager vs graph mode in TF 2.0, what do you mean? perhaps sharing your training code for either case would help. thanks

@mjlbach Can you please provide the details @guptapriya is asking?
Please close the issue If the issue was already resolved. Thanks!

I think it was resolved. Closing due to lack of recent activity. Please open new ticket if you see similar issue. Thanks!

authors: guptapriya jvishnuvardhan mjlbach