text:
[Optimizers] Simplified the Adam optimizer.

@rxwei Do you know why the typechecker fails even for not too long/complex expressions and we have to break them down?

Operator type checking has long been very problematic in Swift.

Sounds good! Will do that and add new ones after being done with the refactoring.

@rxwei what about the max(_:_:) and abs(_:) operations? We currently do not have automatically derived protocols for them, right?

We do not, unfortunately.

@rxwei Would it be easy to add derived conformances for Comparable similar to ElementaryFunctions? There are a few optimizers that use max accumulators (e.g., AdaMax and AMSGrad, for which I have an upcoming PR) and this would help a lot with having simplified implementations for them too.

Comparable requires Boolean operators to return a single Bool, so it’s not the right abstraction for our use case.

Yes that's right. For now I simplified AdaMax as much as I could and added an implementation of AMSGrad that's also still using key-paths for some part of it. We can think about what protocols we would need to support simplifying these optimizers later on.

We should consider pitching a PointwiseComparable protocol on the Swift forums. It can be applied to SIMD types too.

I agree. I also feel that automatically deriving protocol conformances keeps coming up very frequently (it also has come up multiple times in some research projects I haven't published yet) and it is currently very tedious having to modify the compiler to support such cases. Do you think it may be a good time to try and come up with a Swift-side way of doing this and implement it at least as part of S4TF? It will be hard to pitch it early on because there are many opinions on this topic, but if we come up with a clean practical way that works for us and implement it, I think we'll also gain a better understanding of what it takes.
Maybe I'm biased because I'm used to defining type classes to achieve my goals (they're super powerful in Scala), but I do believe that they are a very important feature that we keep bumping into.

A better, general conformance derivation mechanism would certainly be amazing. My thoughts:

A robust design seems difficult, given Swift's limited metaprogramming support. I imagine a full design would involve reflection and metaprogramming in some form. Scala typeclass derivation could be inspiration.
Besides derivation, there are other problems, like "actually designing/iterating on the right protocols" (e.g. the newly added PointwiseMultiplicative, and maybe PointwiseComparable). To me, designing the right protocols is a more pressing issue, especially given Evolution challenges, like source/ABI compatibility. Hardcoding derived conformances isn't too much effort (<1 day). Until a general derivation mechanism exists, we can view "automated conformances" as a detail.

I would love to see solid math protocols land before general conformance derivation! A fully algebraic design probably isn't realistic for Swift. I hope we can find some middle ground that unblocks our use cases (e.g. aggregate math) and improves the current status in Swift (e.g. maybe removing Magnitude from Numeric) while remaining suitable for Evolution.

@dan-zheng is right, we should think about the right protocol design first. Currently there are harder problems to deal with: backporting, retroactive refinement without breaking ABI, etc.  This needs us to pitch these designs on Swift forums.

Thanks @dan-zheng for the detailed response. I see your points. I guess I was also motivated by other uses of derived conformances in code outside S4TF where it gets inconvenient to have to use a fork of the Swift compiler. But yes, figuring out the right protocols does seem like an appropriate first steps. I haven't looked in too much detail into the ones you currently have, but will go through them and see if I can make any useful suggestions.
One suggestion I have for now is that it would be nice to allow for something like @noDerivative that allows us to mark stored properties for which the ElementaryFunctions operations act as an identity function, for example. More generally, I believe we should think about what it would mean for being able to opt-out for certain stored properties. Haven't thought this through much but I think it will come up with more complicated layers or layers that store side information (e.g., shape information about their inputs and outputs).
Also, a side comment is that I noticed that VectorProtocol is not being derived automatically for structs with no stored properties. Why is that?

Also, a side comment is that I noticed that VectorProtocol is not being derived automatically for structs with no stored properties. Why is that?

This is because VectorProtocol cannot infer VectorSpaceScalar from a struct that has no stored properties. Hard-coding this inference to use something like Float seems unprincipled.

I see. However it also causes a usability issue. Layers with no parameters (e.g., activation functions if defined as layers) cannot be used to build models that will later be trained using our current optimizers. That is because it won't be possible to derive VectorProtocol for the layers that use them. We should probably work around this somehow in a principled manner.

I see. However it also causes a usability issue. Layers with no parameters (e.g., activation functions if defined as layers) cannot be used to build models that will later be trained using our current optimizers. That is because it won't be possible to derive VectorProtocol for the layers that use them. We should probably work around this somehow in a principled manner.

We can add a typealias VectorSpaceScalar in those layers for now.

We would also have to explicitly declare conformance to VectorProtocol though, right?

Yes. Ideally, we should be able to declare all of these in protocol Layer and expect derived conformances to work. But I'm not sure whether that will work today.

I've been working on a PR that does that but currently the @noDerivative example I mentioned above is blocking this. For example, MaxPool1D cannot conform to VectorProtocol because it has integer stored properties. Do you have any ideas for how to handle this?

One possible solution would be to separate Layer from its parameters and have all layers have a stored property called parameters and a Parameter associated type which has to conform to Differentiable, VectorProtocol, etc.

We shouldn't need layers to conform to VectorProtocol. Instead, we want their TangentVector to conform. Ideally you should be able to just extend those layers's TangentVector to conform, for example:
extension MaxPool1D.TangentVector: VectorProtocol {
    public typealias VectorSpaceScalar = Float
}

However, this is currently blocked on this bug that @dan-zheng is investigating.
By "defining it in the protocol", I meant something like:
public protocol Layer: Differentiable where TangentVector: VectorProtocol {
    typealias VectorSpaceScalar = Float
}

One possible solution would be to separate Layer from its parameters and have all layers have a stored property called parameters and a Parameter associated type which has to conform to Differentiable, VectorProtocol, etc.

This is something we experimented with a year ago, called Parameterized. The solution was abandoned because any Parameter associated type would make a model with multiple parameter types be either impossible or really complicated to define. We concluded that it wasn't a future-proof solution, so we moved away from that model.

Thanks for the explanation! That totally makes sense. Regarding the current bug, defining VectorSpaceScalar in the protocol itself does not resolve the issue for its TangentVector. However, adding a property like @usableFromInline internal var _vectorSpace = 0.0 to the related structs resolved this. So, after the initializers and the random seed PRs are merged, I'll open a new PR that adds constraints to layers so they can be used with the new optimizers. As a temporary workaround for parameter-less layers, I will define this protocol:
public protocol ParameterlessLayer: Layer {
    @usableFromInline
    internal var _vectorSpace = 0.0
}

and add a TODO along with a bug filing to remove this once Dan fixes the underlying issue you mentioned. How does that sound?

Thanks for the explanation! That totally makes sense. Regarding the current bug, defining VectorSpaceScalar in the protocol itself does not resolve the issue for its TangentVector. However, adding a property like @usableFromInline internal var _vectorSpace = 0.0 to the related structs resolved this. So, after the initializers and the random seed PRs are merged, I'll open a new PR that adds constraints to layers so they can be used with the new optimizers. As a temporary workaround for parameter-less layers, I will define this protocol:
public protocol ParameterlessLayer: Layer {
    @usableFromInline
    internal var _vectorSpace = 0.0
}

and add a TODO along with a bug filing to remove this once Dan fixes the underlying issue you mentioned. How does that sound?

I don't feel this is the best solution. Stored properties cannot be defined in protocols, so even if you add a protocol requirement, every conforming type will have to define such a stored property to satisfy this requirement. We are not yet prioritizing a fix for that bug, so it is likely that this suboptimal pattern will last longer.
I'll suggest a different solution: One can always define a custom TangentVector, so it should be trivial to define one for those parameter-less layers. I believe all you need to add in those structs is:
struct TangentVector: Differentiable, VectorProtocol {}

Oh right. Good point. Ok I’ll look into this when I get back home tonight.
On Sun, Jun 30, 2019 at 8:19 PM Richard Wei notifications@github.com
wrote:

Thanks for the explanation! That totally makes sense. Regarding the
current bug, defining VectorSpaceScalar in the protocol itself does not
resolve the issue for its TangentVector. However, adding a property like @usableFromInline
internal var _vectorSpace = 0.0 to the related structs resolved this. So,
after the initializers and the random seed PRs are merged, I'll open a new
PR that adds constraints to layers so they can be used with the new
optimizers. As a temporary workaround for parameter-less layers, I will
define this protocol:
public protocol ParameterlessLayer: Layer {
@usableFromInline
internal var _vectorSpace = 0.0
}
and add a TODO along with a bug filing to remove this once Dan fixes the
underlying issue you mentioned. How does that sound?
I don't feel this is the best solution. Stored properties cannot be
defined in protocols, so even if you add a protocol requirement, every
conforming type will have to define such a stored property to satisfy this
requirement. We are not yet prioritizing a fix for that bug, so it is
likely that this suboptimal pattern will last longer.
I'll suggest a different solution: One can always define a custom
TangentVector, so it should be trivial to define one for those
parameter-less layers. I believe all you need to add in those structs is:
struct TangentVector: Differentiable, VectorProtocol {}
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/swift-apis/pull/306?email_source=notifications&email_token=AAJ4EXDU7NQXYHBYQSSB4CLP5FEQ7A5CNFSM4H4HYG72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY4WWZY#issuecomment-507079527,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAJ4EXF5QF3DIXCROIQI5KTP5FEQ7ANCNFSM4H4HYG7Q
.

@rxwei one issue that comes up is that we need AllDifferentiableVariables == TangentVector which causes a failure when I try what you suggested earlier.

I also get the following error when I use the suggested approach:
Assertion failed: (paramMember && "Could not find corresponding parameter member"), function operator(), file /Users/eaplatanios/Development/GitHub/swift-source-temp/swift/lib/Sema/DerivedConformanceDifferentiable.cpp, line 279.
Stack dump:
0.      Program arguments: /Library/Developer/Toolchains/swift-tensorflow-LOCAL-2019-06-30-a.xctoolchain/usr/bin/swift -frontend -c /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Bindings/EagerExecution.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Bindings/RawOpsGenerated.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Bindings/TFTensorOperation.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Context.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/ArrayOps.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/DataTypes.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/DifferentialOperators.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/Execution.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/LazyTensorOperation.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/LazyTensorTFFunctionBuilder.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/LazyTensorTrace.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/PythonConversion.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/Runtime.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/ShapedArray.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/StringTensor.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/Tensor.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/TensorGroup.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/TensorHandle.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/TensorProtocol.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/TensorShape.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/Threading.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Core/Utilities.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Initializers.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layer.swift -primary-file /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Convolutional.swift -primary-file /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Core.swift -primary-file /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Embedding.swift -primary-file /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Initialization.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Normalization.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Pooling.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Recurrent.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Layers/Upsampling.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Loss.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Operators/Basic.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Operators/Comparison.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Operators/Dataset.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Operators/Math.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Operators/NN.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Optimizers/MomentumBased.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Optimizers/Optimizer.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Optimizers/SGD.swift /Users/eaplatanios/Development/GitHub/swift-apis/Sources/TensorFlow/Random.swift -emit-module-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Convolutional~partial.swiftmodule -emit-module-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Core~partial.swiftmodule -emit-module-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Embedding~partial.swiftmodule -emit-module-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Initialization~partial.swiftmodule -emit-module-doc-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Convolutional~partial.swiftdoc -emit-module-doc-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Core~partial.swiftdoc -emit-module-doc-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Embedding~partial.swiftdoc -emit-module-doc-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Initialization~partial.swiftdoc -emit-dependencies-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Convolutional.d -emit-dependencies-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Core.d -emit-dependencies-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Embedding.d -emit-dependencies-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Initialization.d -emit-reference-dependencies-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Convolutional.swiftdeps -emit-reference-dependencies-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Core.swiftdeps -emit-reference-dependencies-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Embedding.swiftdeps -emit-reference-dependencies-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Initialization.swiftdeps -target x86_64-apple-macosx10.10 -enable-objc-interop -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk -I /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug -I /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib -F /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks -enable-testing -g -module-cache-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/ModuleCache -swift-version 4.2 -Onone -D SWIFT_PACKAGE -D DEBUG -color-diagnostics -enable-anonymous-context-mangled-names -parse-as-library -module-name TensorFlow -o /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Convolutional.swift.o -o /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Core.swift.o -o /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Embedding.swift.o -o /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/TensorFlow.build/Layers/Initialization.swift.o -index-store-path /Users/eaplatanios/Development/GitHub/swift-apis/.build/x86_64-apple-macosx/debug/index/store -index-system-modules
1.      Swift version 5.1-dev (LLVM 7d54a7c711, Swift f62bd5687d)
0  swift                    0x000000010a86c955 llvm::sys::PrintStackTrace(llvm::raw_ostream&) + 37
1  swift                    0x000000010a86b998 llvm::sys::RunSignalHandlers() + 248
2  swift                    0x000000010a86cf48 SignalHandler(int) + 264
3  libsystem_platform.dylib 0x00007fff759bcb5d _sigtramp + 29
4  libsystem_platform.dylib 0x00000001173a9938 _sigtramp + 2711539192
5  libsystem_c.dylib        0x00007fff758766a6 abort + 127
6  libsystem_c.dylib        0x00007fff7583f20d basename_r + 0
7  swift                    0x000000010abb0d13 deriveBodyDifferentiable_move(swift::AbstractFunctionDecl*, void*) (.cold.9) + 35
8  swift                    0x0000000107609ff7 deriveBodyDifferentiable_move(swift::AbstractFunctionDecl*, void*) + 2279
9  swift                    0x0000000107708883 swift::TypeChecker::typeCheckAbstractFunctionBody(swift::AbstractFunctionDecl*) + 83
10 swift                    0x0000000107729453 typeCheckFunctionsAndExternalDecls(swift::SourceFile&, swift::TypeChecker&) + 275
11 swift                    0x0000000107729c1d swift::performTypeChecking(swift::SourceFile&, swift::TopLevelContext&, swift::OptionSet<swift::TypeCheckingFlags, unsigned int>, unsigned int, unsigned int, unsigned int, unsigned int, unsigned int) + 1005
12 swift                    0x0000000106e62d71 swift::CompilerInstance::parseAndCheckTypesUpTo(swift::CompilerInstance::ImplicitImports const&, swift::SourceFile::ASTStage_t)::$_1::operator()(swift::SourceFile&) const + 81
13 swift                    0x0000000106e5e74b swift::CompilerInstance::parseAndCheckTypesUpTo(swift::CompilerInstance::ImplicitImports const&, swift::SourceFile::ASTStage_t) + 507
14 swift                    0x0000000106e5e04a swift::CompilerInstance::performSemaUpTo(swift::SourceFile::ASTStage_t) + 618
15 swift                    0x0000000106c4be92 performCompile(swift::CompilerInstance&, swift::CompilerInvocation&, llvm::ArrayRef<char const*>, int&, swift::FrontendObserver*, swift::UnifiedStatsReporter*) + 1730
16 swift                    0x0000000106c4a7f1 swift::performFrontend(llvm::ArrayRef<char const*>, char const*, void*, swift::FrontendObserver*) + 3025
17 swift                    0x0000000106bf2bf9 main + 729
18 libdyld.dylib            0x00007fff757d13d5 start + 1
19 libdyld.dylib            0x0000000000000077 start + 2323836067

Nevermind, I worked out a solution and will be opening a PR soon. :)

Done in #322 .

authors: dan-zheng eaplatanios rxwei

repo: tensorflow/swift-apis


text:
Avoid unnecessary tensor materialization in tensor-scalar op derivatives.

Add derivatives for (Tensor, Scalar) -> Tensor operations differentiable with respect to only the Tensor argument, to avoid unnecessary tensor materialization in pullbacks.
Background
Currently, (Tensor, Scalar) -> Tensor operations like Tensor * Scalar have one derivative defined with respect to both both lhs and rhs:
extension Tensor where Scalar: TensorFlowFloatingPoint {
  @inlinable
  @derivative(of: * wrt: (lhs, rhs))
  static func _vjpMultiply(lhs: Tensor, rhs: Scalar) -> (
    value: Tensor, pullback: (Tensor) -> (Tensor, Scalar)
  ) {
    return (lhs * rhs, { v in (v * rhs, (v * lhs).sum().scalarized()) })
  }
}

What if we only care about the derivative with respect to the lhs Tensor, like in x * 3?
Currently, differentiation generates "subset parameter derivative thunks" which discard unused pullback results. The thunk for _vjpMultiply wrt only lhs (the Tensor) calls the user-defined derivative wrt both lhs and rhs, so it's not avoiding the work of computing the derivative wrt rhs (the Scalar).
Idea
This patch adds extra derivatives for (Tensor, Scalar) -> Tensor operations differentiable with respect to only the Tensor argument:
  // This derivative for `(lhs: Tensor) * (rhs: Scalar)` copies the one above, but is
  // only with respect to only `lhs`.
  @inlinable
  @derivative(of: *, wrt: lhs)
  static func _vjpMultiply(lhs: Tensor, rhs: Scalar) -> (
    value: Tensor, pullback: (Tensor) -> Tensor
  ) {
    return (lhs * rhs, { v in v * rhs })
  }

This avoids unnecessary work to compute the derivative with respect to Scalar arguments, which involves calling Tensor.scalarized(), thereby triggering X10 tensor materialization.
Future directions
It would be cool if differentiation could generate "efficient subset parameter derivative thunks" from user-defined derivatives that do actually avoid extra work, without affecting usability.

@pschuh confirmed that this patch indeed avoids unnecessary materialization.

@pschuh pointed out that this optimization is relevant for all (Tensor, Scalar) -> Tensor operations, not just Tensor * Scalar. Updated.

Many thanks! This unblocks BERT on XLA 🎉

source: gharchive/pull-request

authors: dan-zheng texasmichelle

license_type: permissive

license_source: github-api


text:
Link to Linux and Mac OS broken in TF for Java README

The link to Linux and Mac OS under section Building from Source is broken.

The Linux should point to:
https://www.tensorflow.org/install/install_sources#PrepareLinux
Added a PR #12145 for the fix.

source: gharchive/issue

authors: deepali-c yongtang

repo: tensorflow/tensorflow


text:
tf.graph_util.extract_sub_graph should raise a better error message

Hi,
I used tf.graph_util.extract_sub_graph to get a subgraph reaching a particular dest node. My dest_node is say conv2d_transpose_7. I by mistake gave it as a string in the second parameter. Python parses String too as a list and it raised an error AssertionError: c is not in graph. The extract_sub_graph should first check if the second parameter isinstace of a List so that it would be convenient to debug this kind of error in the future. A screenshot is attached.
Thanks,

Sounds reasonable, I can work on it.

Marking as contributions welcome since @facaiy is working on it

authors: Sentient07 facaiy reedwm


text:
tensorflow.python.framework.errors_impl.NotFoundError: ; No such file or directory

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Mac OS Sierra 10.12.6
TensorFlow installed from (source or binary):
source
TensorFlow version (use command below):
1.4.0
Python version:
python3
Bazel version (if compiling from source):
sorry, I don't know
GCC/Compiler version (if compiling from source):
sorry, I don't know
CUDA/cuDNN version:
sorry, I don't know
GPU model and memory:
radeon pro 560 4G

Describe the problem
I try to use object detection using tensorflow API
I follow this youtube
https://www.youtube.com/watch?v=JR8CmWyh2E8
but there is a error
Source code / logs
MacBook-Pro-de-Jongwun:~ jongwuni$ cd /Users/jongwuni/Documents/Jongwunibang/Neural_network/models
MacBook-Pro-de-Jongwun:models jongwuni$ cd object_detection
MacBook-Pro-de-Jongwun:object_detection jongwuni$ python3 train.py --logtostderr --train_dir="/Users/jongwuni/Documents/Jongwunibang/Neural_network/models/object_detection/training" --pipline_config_path="/Users/jongwuni/Documents/Jongwunibang/Neural_network/models/object_detection/training/faster_rcnn_inception_resnet_v2_atrous_coco.config"
Traceback (most recent call last):
File "train.py", line 163, in
tf.app.run()
File "/anaconda/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "train.py", line 106, in main
overwrite=True)
File "/anaconda/lib/python3.6/site-packages/tensorflow/python/lib/io/file_io.py", line 385, in copy
compat.as_bytes(oldpath), compat.as_bytes(newpath), overwrite, status)
File "/anaconda/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: ; No such file or directory

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Exact command to reproduce

@jongwun  did you fix the issue?
Thanks

I met the same issue. Has anyone fixed this yet? Thanks

@thanhtu19392  are you using external HDD as a path or storage?

I fixed the error. It's an error from my side. Thanks anw

@thanhtu19392 I'm having this error - how did you fix it?

authors: ProjectDent isrugeek jongwun tensorflowbutler thanhtu19392


text:
Feature request: Create a function that handles errors for tf.train.SessionRunHook()

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Windows 10 Pro 64-bit (10.0, Build 16299)
TensorFlow installed from (source or binary):
Binary
TensorFlow version (use command below):
1.6.0
Python version:
3.6.4
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:
Exact command to reproduce:

You can collect some of this information using our environment capture script:
https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh
You can obtain the TensorFlow version with
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
Describe the problem
I would like an operation for tf.train.SessionRunHook() that executes when an error is thrown from the sess.run() (it should be able to act upon a single error/multiple errors (in a list)/all errors. The reason I would like this, is because there exists use cases where a hook action should be performed only when an error is raised.
My current example of this use case is the switch between training and validation loops. I am, in a tf.train.MonitoredTrainingSession() trying to run a training loop and a validation loop that switches between each other. In order to achieve this, I'm using a tf.data.Dataset.iterator that runs in a while loop until a tf.errors.OutOfRangeError is thrown; breaking the while loop and moving on to the next. When this exception is thrown, I would like to perform hook actions, such as saving epoch summaries, print logging on the terminal, etc. My current way of doing this is to write these actions manually in the exception clause, as shown below. It would, however, be cleaner and easier if these actions could be performed in the hooks themselves when an error is thrown.
Source code / logs
Example code I am using currently:
import tensorflow as tf
from functools import partial

def create_reset_metric(metric, scope='reset_metrics', **metric_args):
    with tf.variable_scope(scope) as scope:
        metric_op, update_op = metric(**metric_args)
        vars = tf.contrib.framework.get_variables(
                scope, collection=tf.GraphKeys.LOCAL_VARIABLES
            )
        reset_op = tf.variables_initializer(vars)
    return metric_op, update_op, reset_op

dataset_train = tf.data.Dataset.range(100)
iterator_train = dataset_train.make_initializable_iterator()
next_elem_train = iterator_train.get_next()
mean_batch_train, mean_update_train, mean_reset_train = create_reset_metric(
                                                            metric=tf.metrics.mean,
                                                            scope='reset_metrics_train',
                                                            values=next_elem_train)
summary_train = tf.summary.scalar('train_summary', mean_update_train, collections=['train'])

dataset_test = tf.data.Dataset.range(50)
iterator_test = dataset_test.make_initializable_iterator()
next_elem_test = iterator_test.get_next()
mean_batch_test, mean_update_test, mean_reset_test = create_reset_metric(
                                                            metric=tf.metrics.mean,
                                                            scope='reset_metrics_test',
                                                            values=next_elem_test)
summary_test = tf.summary.scalar('test_summary', mean_update_test, collections=['test'])

merged_train_summary_op = tf.summary.merge_all('train')
merged_test_summary_op = tf.summary.merge_all('test')

def step_fn(fetches, feed_dict, step_context):
    return step_context.session.run(fetches=fetches, feed_dict=feed_dict)

with tf.summary.FileWriter('./tmp/train_test_switch') as writer:
    with tf.train.MonitoredTrainingSession() as sess:
        epoch_step = 0
        while not sess.should_stop():
            sess.run_step_fn(partial(step_fn, iterator_train.initializer, {}))
            while True:
                try:
                    summary_train_, _ = sess.run([merged_train_summary_op, next_elem_train])
                except tf.errors.OutOfRangeError:
                    writer.add_summary(summary_train_, epoch_step)
                    sess.run_step_fn(partial(step_fn, mean_reset_train, {}))
                    break

            sess.run_step_fn(partial(step_fn, iterator_test.initializer, {}))
            while True:
                try:
                    summary_test_, _ = sess.run([merged_test_summary_op, next_elem_test])
                except tf.errors.OutOfRangeError:
                    writer.add_summary(summary_test_, epoch_step)
                    sess.run_step_fn(partial(step_fn, mean_reset_test, {}))
                    break
            print("epoch_step:", epoch_step)
            epoch_step += 1

Example of how I want the tf.train.MonitoredTrainingSession() to look (the beginning is the same as before):
...

with tf.summary.FileWriter('./tmp/train_test_switch') as writer:
    error_catching_hook = ErrorCatchingHook(...)
    with tf.train.MonitoredTrainingSession(hooks=[error_catching_hook]) as sess:
        epoch_step = 0
        while not sess.should_stop():
            sess.run_step_fn(partial(step_fn, iterator_train.initializer, {}))
            error_catching_hook.is_training() # Set flag to call correct params when OutOfRangeError
            while True:
                try:
                    summary_train_, _ = sess.run([merged_train_summary_op, next_elem_train])
                except tf.errors.OutOfRangeError:
                    # error_catching_hook calls writer.add_summary(...)
                    # error_catching_hook calls mean_reset_train
                    break

            sess.run_step_fn(partial(step_fn, iterator_test.initializer, {}))
            error_catching_hook.is_testing() # Set flag to call correct params when OutOfRangeError
            while True:
                try:
                    summary_test_, _ = sess.run([merged_test_summary_op, next_elem_test])
                except tf.errors.OutOfRangeError:
                    # error_catching_hook calls writer.add_summary(...)
                    # error_catching_hook calls mean_reset_test
                    break
            print("epoch_step:", epoch_step)
            epoch_step += 1

I realize that this might not be the best example of a use case for this, but I hope you get my point, and I'm sure there are other use cases for this.
After having read my code above, I also hope that this could provide a solution for others trying to gather and summarize epoch-wise data. A common pattern of summarizing train/test data is, after all, to summarize an epoch average after each epoch. This feels like an unnecessarily painful implementation task today, and I hope that this request of mine can at least bring down the pain a little bit. Hopefully, this can lead to a tf.train.EndOfSetHook() or similar, that is only called when a dataset reaches its end!

Nagging Assignee @rohan100jain: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rohan100jain: It has been 16 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rohan100jain: It has been 16 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rohan100jain: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rohan100jain: It has been 93 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rohan100jain: It has been 108 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rohan100jain: It has been 124 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@ispirmustafa to comment on feature request feasibility.

handling errors with hooks may not have that straightforward/clear usage. Instead I would recommend using multiple sessions something like following:
You can run some ops at the end of training by using end function of SessionRunHook
with MonitoredTrainingSession(...) as mon_sess:
  while not mon_sess.should_stop():
    mon_sess.run(train_op)

with MonitoredTrainingSession(...) as mon_sess:
  while not mon_sess.should_stop():
    mon_sess.run(eval_op)

Please let us know this works or not

It has been 14 days with no activity and the awaiting response label was assigned. Is this still an issue?

handling errors with hooks may not have that straightforward/clear usage. Instead I would recommend using multiple sessions something like following:
You can run some ops at the end of training by using end function of SessionRunHook
with MonitoredTrainingSession(...) as mon_sess:
  while not mon_sess.should_stop():
    mon_sess.run(train_op)

with MonitoredTrainingSession(...) as mon_sess:
  while not mon_sess.should_stop():
    mon_sess.run(eval_op)

Please let us know this works or not

Hi,
I would like to use multiple session in order to evaluate on a validation dataset during training.
I created the evaluation monitored session as a nested session of the training monitored session:
timer = tf.train.SecondOrStepTimer(...)
with MonitoredTrainingSession(...) as train_session:

  train_session.run(train_op)

  if timer.should_trigger_for_step(..):

    with MonitoredTrainingSession(...) as eval_session:
      eval_session.run(eval_op)

The two graphs share weights, but because I want to produce different summaries for train and evaluation, each monitored-training-session has its own scaffold object. For the training session I have an init function, but for the evaluation session I don't want the weights to be initialised. If I don't declare an init_fn in the scaffold object, I get this error:
  File "/Users/adva/anaconda/lib/python3.5/site-packages/tensorflow/python/training/session_manager.py", line 287, in prepare_session
    init_fn(sess)
  File "/Users/adva/anaconda/lib/python3.5/site-packages/tensorflow/python/training/monitored_session.py", line 163, in <lambda>
    self._init_fn = lambda sess: init_fn(self, sess)
TypeError: callback() takes 1 positional argument but 2 were given

How can I use multiple sessions this way?
Thank you!
Adva

Hi @advaza ,
For your approach you don't need to have two session. You should be able to call eval_op on train_session. That will keep the weights same.

Thanks @ispirmustafa ! I figured it out eventually. The error I got was because I changed the code from slim supervisor to monitored training session, and I had to define the init_fn differently.
I ended up doing:
timer = tf.train.SecondOrStepTimer(...)
with MonitoredTrainingSession(...) as train_session:

  while not train_session.should_stop():

    _, global_step_value = train_session.run([train_op, global_step])

    if timer.should_trigger_for_step(step=global_step_value):

      timer.update_last_triggered_step(step=global_step_value)

      train_session.run(eval_iterator.initializer) # Initializable Iterator

      while True:
        try:
          train_session.run(eval_op)

        except tf.errors.OutOfRangeError:
          break

authors: advaza aforslow ispirmustafa rohan100jain tensorflowbutler


text:
get ctc_greedy_decoder results error C-api ?

System information

Have I written custom code
OS Platform and Distribution (Linux Ubuntu 16.04):
TensorFlow installed from (source or binary): source
TensorFlow version (use command below): 1.12
Python version: 3.6
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: not available
GPU model and memory: not available

Python code:
self.seqLen = tf.placeholder(tf.int32, [None])
decoder = tf.nn.ctc_greedy_decoder(inputs=ctcIn3dTBC, sequence_length=self.seqLen)
Python result:
([SparseTensorValue(indices=array([[ 0,  0],        [ 0,  1],        [ 0,  2],        [ 0,  3],        [ 1,  0],        [ 1,  1],        [ 1,  2],        [ 1,  3],        [ 2,  0],        [ 2,  1],        [ 2,  2],        [ 2,  3],        [ 3,  0],        [ 3,  1],        [ 3,  2],        [ 3,  3],        [ 4,  0],        [ 4,  1],        [ 4,  2],        [ 4,  3],        [ 5,  0],        [ 5,  1],        [ 5,  2],        [ 5,  3],        [ 6,  0],        [ 6,  1],        [ 6,  2],        [ 6,  3],        [ 7,  0],        [ 7,  1],        [ 7,  2],        [ 7,  3],        [ 8,  0],        [ 8,  1],        [ 8,  2],        [ 8,  3],        [ 9,  0],        [ 9,  1],        [ 9,  2],        [ 9,  3],        [10,  0],        [10,  1],        [10,  2],        [10,  3],        [11,  0],        [11,  1],        [11,  2],        [11,  3],        [12,  0],        [12,  1],        [12,  2],        [12,  3],        [13,  0],        [13,  1],        [13,  2],        [13,  3],        [14,  0],        [14,  1],        [14,  2],        [14,  3],        [15,  0],        [15,  1],        [15,  2],        [15,  3],        [16,  0],        [16,  1],        [16,  2],        [16,  3],        [17,  0],        [17,  1],        [17,  2],        [17,  3],        [18,  0],        [18,  1],        [18,  2],        [18,  3],        [19,  0],        [19,  1],        [19,  2],        [19,  3],        [20,  0],        [20,  1],        [20,  2],        [20,  3],        [21,  0],        [21,  1],        [21,  2],        [21,  3],        [22,  0],        [22,  1],        [22,  2],        [22,  3],        [23,  0],        [23,  1],        [23,  2],        [23,  3],        [24,  0],        [24,  1],        [24,  2],        [24,  3],        [25,  0],        [25,  1],        [25,  2],        [25,  3],        [26,  0],        [26,  1],        [26,  2],        [26,  3],        [27,  0],        [27,  1],        [27,  2],        [27,  3],        [28,  0],        [28,  1],        [28,  2],        [28,  3],        [29,  0],        [29,  1],        [29,  2],        [29,  3],        [30,  0],        [30,  1],        [30,  2],        [30,  3],        [31,  0],        [31,  1],        [31,  2],        [31,  3],        [32,  0],        [32,  1],        [32,  2],        [32,  3],        [33,  0],        [33,  1],        [33,  2],        [33,  3],        [34,  0],        [34,  1],        [34,  2],        [34,  3],        [35,  0],        [35,  1],        [35,  2],        [35,  3],        [36,  0],        [36,  1],        [36,  2],        [36,  3],        [37,  0],        [37,  1],        [37,  2],        [37,  3],        [38,  0],        [38,  1],        [38,  2],        [38,  3],        [39,  0],        [39,  1],        [39,  2],        [39,  3],        [40,  0],        [40,  1],        [40,  2],        [40,  3],        [41,  0],        [41,  1],        [41,  2],        [41,  3],        [42,  0],        [42,  1],        [42,  2],        [42,  3],        [43,  0],        [43,  1],        [43,  2],        [43,  3],        [44,  0],        [44,  1],        [44,  2],        [44,  3],        [45,  0],        [45,  1],        [45,  2],        [45,  3],        [46,  0],        [46,  1],        [46,  2],        [46,  3],        [47,  0],        [47,  1],        [47,  2],        [47,  3],        [48,  0],        [48,  1],        [48,  2],        [48,  3],        [49,  0],        [49,  1],        [49,  2],        [49,  3]], dtype=int64), values=array([53, 59, 64, 57, 53, 59, 64, 57, 53, 59, 64, 57, 53, 59, 64, 57, 53,        59, 64, 57, 53, 59, 64, 57, 53, 59, 64, 57, 53, 59, 64, 57, 53, 59,        64, 57, 53, 59, 64, 57, 53, 59, 64, 57, 53, 59, 64, 57, 53, 59, 64,        57, 53, 59, 64, 57, 53, 59, 64, 57, 53, 59, 64, 57, 53, 59, 64, 57,        53, 59, 64, 57, 53, 59, 64, 57, 53, 59, 64, 57, 53, 59, 64, 57, 53,        59, 64, 57, 53, 59, 64, 57, 53, 59, 64, 57, 53, 59, 64, 57, 53, 59,        64, 57, 53, 59, 64, 57, 53, 59, 64, 57, 53, 59, 64, 57, 53, 59, 64,        57, 53, 59, 64, 57, 53, 59, 64, 57, 53, 59, 64, 57, 53, 59, 64, 57,        53, 59, 64, 57, 53, 59, 64, 57, 53, 59, 64, 57, 53, 59, 64, 57, 53,        59, 64, 57, 53, 59, 64, 57, 53, 59, 64, 57, 53, 59, 64, 57, 53, 59,        64, 57, 53, 59, 64, 57, 53, 59, 64, 57, 53, 59, 64, 57, 53, 59, 64,        57, 53, 59, 64, 57, 53, 59, 64, 57, 53, 59, 64, 57], dtype=int64), dense_shape=array([50,  4], dtype=int64))], array([[-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455],        [-297.2455]], dtype=float32))
Graph-def:
483: CTCGreedyDecoder type: CTCGreedyDecoder device:  number inputs: 2 number outputs: 4 Number inputs: 2 0 type : TF_FLOAT 1 type : TF_INT32 Number outputs: 4 0 type : TF_INT64  dims: 2 [-1,2] 1 type : TF_INT64  dims: 1 [-1] 2 type : TF_INT64  dims: 1 [2] 3 type : TF_FLOAT  dims: 2 [50,1]
C-API result:
raw ouput:
0 0
1 0
2 0
3 0
4 0
5 0
6 0
7 0
8 0
9 0
10 0
11 0
12 0
13 0
14 0
15 0
16 0
17 0
18 0
19 0
20 0
21 0
22 0
23 0
24 0
25 0
26 0
27 0
28 0
29 0
30 0
31 0
32 0
33 0
34 0
35 0
36 0
37 0
38 0
39 0
40 0
41 0
42 0
43 0
44 0
45 0
46 0
47 0
48 0
49 0
-1759315024 1332898372
-1746890864 -1804888032
-1759345360 -1759345360
-1759317136 -1759317136
-1759345360 -1759345360
-1759343376 -1759343376
-572662307 -572662307
-572662307 -572662307
-572662307 2104715895
544096522 544498024
1702258025 1768778100
1881173615 1919251557
1696623983 1735289188
1953722216 779711087
1700929652 1701078371
1126195488 1701999975
1684365933 1953722184
1952540788 1886724211
1919902496 1651336480
1948280686 1852793632
1769234802 544370502
744844400 1886330995
1684370293 1920230770
1685091616 1918988320
544436837 1931502962
543236199 1768910955
1918988298 980644453
1869835877 1767994478
1701344288 1768186981
1701601889 1952804193
1931505524 1953068832
1634732645 1936876916
1835343981 543649385
544433524 1752440935
1634887521 1768778100
1629515375 1835562089
1970107747 540701554
1919906670 1852399988
543516788 1852400740
543517794 1634497901
544175136 1769414757
1879729512 1919251557
1696623983 1735289188
1936028769 1948280686
1919377764 1835627632
544108393 1752459634
1701601889 1310734949
1948280431 1818386804
544502645 543236200
544106784 1347690528
1768186981 1734960750
544108393 1864900719
1936024681 1767859564
1601009006 540701540
1864397413 1935962721
1752637551 543516788
1852400740 1936026722
1986618400 1752369710
540697705 1768319348
1752375398 544370534
1701867296 170815087
1684627301 544761188
544434536 1852383333
1651328288 1632397166
1768320623 1852795252
537554804 1667592816
667177 1768319348
1752375398 1081240139
-1804996768 -1804848800
1684627301 544761188
544434536 1852383333
1651328288 1632397166
1768320623 1852795252
537554804 1667592816
667177 74672654
544096522 544498024
1886330995 1769234810
1835102817 1852383347
1713392962 1684365933
1967988782 1919945317
1646290021 1718513475
1431327845 1852400740
1886330996 1702043764
1752440944 1667592818
1684301154 1818386804
1734960750 778989417
1635280160 1752440876
1936269424 1869881444
1819042164 1952804193
1952540788 1634692128
1836020326 1801676136
1700929652 1948279072
543649385 175335712
1684370549 1835102817
1092631155 544370547
1768843617 1763730792
1696623713 1735289188
1881171308 1919251557
1702065440 1700949349
1869351527 1937055859
543516788 1953525536
1869182049 1769107303
1836018954 541139002
1663070831 1852403305
1852383333 1835343980
543649385 1869422693
1869881441 544106784
1852400740 1886743407
543649385 1296122945
2053729641 1818304622
778922100 1953063791
1948270880 1868767346
1735289198 1768843552
1651336480 1948280686
1818588704 544433513
1763730803 1684301154
1802465132 1852404597
1145118821 1835627632
544108393 1752459634
1701601889 1310734949
1948280431 1818386804
544502645 543236200
544106784 1347690528
1768186981 1734960750
544108393 1864900719
1936024681 1767859564
1601009006 540701540
1864397413 1935962721
1752637551 543516788
1852400740 1936026722
1986618400 1752369710
540697705 1768319348
1752375398 544370534
1701867296 170815087
1684627301 544761188
544434536 1852383333
1651328288 1632397166
1768320623 1852795252
537554804 1667592816
667177 0
0 175335936
544096522 544498024
1886330995 1769234810
1835102817 1852383347
1713392962 1684365933
1967988782 1919945317
1646290021 1718513475
1431327845 1852400740
1886330996 1702043764
1752440944 1667592818
I'am blocked to correctly get  result of ctc_greedy_decoder in c-API, please help?
Thanks.

I found the problem. I forgot to normalize the input batch data.


text:
[TF1.13] gate order in tf.keras.layers.cudnnlstm

There is a kernel [input_dim, hid_dim * 4] matrix and recurrent kernel [hid_dim, hid_dim * 4] matrix in cudnnlstm.
My question is what is the order of gate input, forget, new, output in the kernel and recurrent kernel?
[input, forget, new, output]
I would really appreciate it since I am transfering weights from BasicLSTMCell kernel matrix.
I would also like to confirm the gate order in BasicLSTMCell kernel [input_dim + hid_dim, hid_dim * 4] is
[input, new, forget, output]
Thank you!

@edchengg Could you provide more details about the issue and context? Thanks!

@gadagashwini
I am referring to
https://github.com/tensorflow/tensorflow/blob/r1.13/tensorflow/python/keras/layers/cudnn_recurrent.py
L473:
params = recurrent._canonical_to_params(    # pylint: disable=protected-access         weights=[             self.kernel[:, :self.units],             self.kernel[:, self.units:self.units * 2],             self.kernel[:, self.units * 2:self.units * 3],             self.kernel[:, self.units * 3:],             self.recurrent_kernel[:, :self.units],             self.recurrent_kernel[:, self.units:self.units * 2],             self.recurrent_kernel[:, self.units * 2:self.units * 3],             self.recurrent_kernel[:, self.units * 3:],         ],         biases=[             self.bias[:self.units],             self.bias[self.units:self.units * 2],             self.bias[self.units * 2:self.units * 3],             self.bias[self.units * 3:self.units * 4],             self.bias[self.units * 4:self.units * 5],             self.bias[self.units * 5:self.units * 6],             self.bias[self.units * 6:self.units * 7],             self.bias[self.units * 7:],         ],         shape=self._vector_shape)
It is not clear which gate(input, forget, new, output) params is self.kernel[:, :] referring to in the cudnnLSTM.
Thank you.

There are 3 fused weights in keras LSTM layer, kernel, recurrent_kernel, and bias. Each of the fused weights are in IFCO order (input, forget, carry, output).
BasicLSTMCell is existing TF code have two fused weights: kernel and bias. The kernel here is the combination of kernel and recurrent kernel as keras layer, which is concat on dim 0. More importantly, the order of the fused kernel is ICFO (input, carry, forget, output).
If you want to converting your BasicLSTMCell weights to keras format, you need to first split the kernel into 2 parts, and also switch the carry and forget gate.

I have solved this issue by myself.
The gate order in basiclstmcell is [input, carry, forget, output]
the gate order in tf.keras.layers.cudnnlstm is [input, forget, carry, output]
We need to further convert the weight matrix for each gate when transferring from basiclstmcell to tf.keras.layers.cudnnlstmcell.
With square matrix: we can use transpose.
With matrix with size [row, col] (for example, input kernel)
we need to do the following:
[[0, 1, 2, 3],                [[0, 4, 1, 5],
[4, 5, 6, 7]] .     -->  [2, 6, 3, 7]]

@edchengg Thank you so much!

authors: edchengg gadagashwini qlzh727 zhuzilin


text:
ImportError: /opt/gnu/gcc/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOs 6.10 Linux: 2.6.32-754.2.1.el6.x86_64
TensorFlow installed from (source or binary): Conda install
TensorFlow version: 1.14.0
Python version: 3.7.5
Installed using virtualenv? pip? conda?: Conda
GCC/Compiler version (if compiling from source): c++ 4.9.2
CUDA/cuDNN version:
GPU model and memory:

Describe the problem
Import error ImportError: /opt/gnu/gcc/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /home/atanteck/anaconda3/lib/python3.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so). Indicated file is there at the path.
Provide the exact sequence of commands / steps that you executed before running into the problem
Import tensorflow
Any other info / logs

import tensorflow
Traceback (most recent call last):
File "/home/atanteck/anaconda3/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "/home/atanteck/anaconda3/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "/home/atanteck/anaconda3/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "/home/atanteck/anaconda3/lib/python3.7/imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "/home/atanteck/anaconda3/lib/python3.7/imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: /opt/gnu/gcc/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /home/atanteck/anaconda3/lib/python3.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
File "/home/atanteck/anaconda3/lib/python3.7/site-packages/tensorflow/init.py", line 34, in
from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
File "/home/atanteck/anaconda3/lib/python3.7/site-packages/tensorflow/python/init.py", line 49, in
from tensorflow.python import pywrap_tensorflow
File "/home/atanteck/anaconda3/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "/home/atanteck/anaconda3/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "/home/atanteck/anaconda3/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "/home/atanteck/anaconda3/lib/python3.7/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "/home/atanteck/anaconda3/lib/python3.7/imp.py", line 242, in load_module
return load_dynamic(name, filename, file)
File "/home/atanteck/anaconda3/lib/python3.7/imp.py", line 342, in load_dynamic
return _load(spec)
ImportError: /opt/gnu/gcc/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /home/atanteck/anaconda3/lib/python3.7/site-packages/tensorflow/python/_pywrap_tensorflow_internal.so)
Failed to load the native TensorFlow runtime.
See https://www.tensorflow.org/install/errors
for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.

This issue looks like a duplicate of https://github.com/tensorflow/tensorflow/issues/26826

I am closing this issue as it is duplicate of https://github.com/tensorflow/tensorflow/issues/26826 and workaround for the problem is provided there. We will followup with the issue there. Thanks!

I find this link very useful:
http://www.programmersought.com/article/3932478100/

I find this link very useful:
http://www.programmersought.com/article/3932478100/

this could help me, thanks.

I find this link very useful:
http://www.programmersought.com/article/3932478100/

thanks,it is useful for me

authors: AdrianTanTeckKeng RenzhiDaDa cache203 jvishnuvardhan kiseliu wdirons


text:
tf.contrib.image.transform in TF2.0

Please make sure that this is a feature request. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template
System information

TensorFlow version (you are using): 2.0
Are you willing to contribute it (Yes/No): No

Describe the feature and the current behavior/state.
In TF 1.X there were nice methods tf.contrib.image.transform() and tf.contrib.image.compose_transforms() that allowed to perform affine transforms with an image.
In TF 2.0 whole tf.contrib.* is dropped.
For instance, I was using those in my custom augmentation mechanisms to add rotation and shear transforms (along with all zooms, shifts, flips, etc). And in TF 2.0 I see no way to perform those.
I tried to implement such a functionality with numpy+opencv instead of TF. However when I tried injecting such a functionality to tf.dataset.map() pipeline tensors seemed not to have .nupmy() method yet.
I also found tf.keras.preprocessing.image.apply_affine_transform() which does perform affine transform on image, but according to docs it works with numpy array only.
Will this change the current api? How?
I guess that would be nice to port those functions form 1.14 to 2.X to: tf.image.compose_transforms() and tf.image.transform() or something. Should not affect existing API I guess.
Who will benefit with this feature?
Anybody who likes to perform affine transforms on image say in data augmentation step of model training.
Any Other info.
If I miss something and mentioned functionality can be implemented with existing API, please let me know.

tf.contrib.image.transform and tf.contrib.image.compose_transform are now part of the tensorflow_addons.image module which should implement the functionality you are looking for.

@chayka As mentioned in the previous comment, you can refer to tensorflow_addons to find the same functions. Closing this issue as it has been resolved.

Thank you guys a lot! Could be a nice idea to leave a note in tf.contrib.* branch docs that some functionality is not dropped, but could be found somewhere else. Spent quite some time looking for it and still missed it. But anyway, thanks for TF2.0, great job!

authors: chayka gowthamkpr lgeiger


text:
TF Keras fails to concatenate batches in predict when using distributed strategy

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): -
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): HPC Cluster
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary): via pip
TensorFlow version (use command below): 2.0.0
Python version: 3.7.1
Bazel version (if compiling from source): -
GCC/Compiler version (if compiling from source): -
CUDA/cuDNN version: cuDNN 7.6.4, CUDA 10.0.130
GPU model and memory: 8x GeForce GTX 1080 Ti with 10479 MB memory each

Describe the current behavior
When using the distributed mirrored strategy to train the network on multiple GPU's, the predict step fails with this error code:

File "/cluster/home/user/localization/base_model.py", line 76, in predict
prediction = self.model.predict(x, use_multiprocessing = False)
File "/cluster/home/cspreche/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 909, in predict
use_multiprocessing=use_multiprocessing)
File "/cluster/home/user/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 462, in predict
steps=steps, callbacks=callbacks, **kwargs)
File "/cluster/home/user/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 444, in _model_iteration
total_epochs=1)
File "/cluster/home/user/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 161, in run_one_epoch
batch_outs = _aggregate_predict_results(strategy, batch_outs, model)
File "/cluster/home/user/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 631, in _aggregate_predict_results
dist_utils.concat_along_batch_dimension(nest.flatten(nested_outs)))
File "/cluster/home/user/.local/lib/python3.7/site-packages/tensorflow_core/python/keras/distribute/distributed_training_utils.py", line 1205, in concat_along_batch_dimension
return np.concatenate(outputs)
ValueError: all the input array dimensions except for the concatenation axis must match exactly

When I print the dimensions of the elements in the concatenation, I get the following output:

(32, 320, 320, 1)
(32, 320, 320, 1)
(32, 320, 320, 1)
(32, 320, 320, 1)
(32, 320, 320, 1)
(32, 320, 320, 1)
(32, 320, 320, 1)
(32, 320, 320, 1)
(32, 320, 320, 1)
(32, 320, 320, 1)
(32, 320, 320, 1)
(32, 320, 320, 1)
(10, 320, 320, 1)
(0, 0, 0, 1)
(0, 0, 0, 1)
(0, 0, 0, 1)

It seems, that the empty sets do not inherit the dimensions of full ones and lead therefore to a crash with the numpy concatenate function.
I could solve the issue with replacing

return np.concatenate(outputs)

with

out = [z for z in outputs if z.shape[0] > 0]
return np.concatenate(out)

Code to reproduce the issue

ds_train = tfds.load(..)
ds_val = tfds.load(..)
strategy = tf.distribute.MirroredStrategy()
num_gpu = strategy.num_replicas_in_sync
ds_train = ds_train.batch(32 * num_gpu)
ds_val = ds_val.batch(32 * num_gpu)
with strategy.scope():
input_image = Input(...)
....
output = ....
model = tf.keras.Model(inputs=[input_image], outputs=[output])
model.compile(...)
model.fit(ds_train, validation_data = ds_val)
model.predict(ds_val, use_multiprocessing = True)

@christiansprecher Can you please provide a simple standalone code to reproduce the issue? Thanks!

Closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

authors: christiansprecher jvishnuvardhan


text:
tflite object detection predictions are always same irrespective of input

Hi have a tensorflow tflite model that i am trying to deploy on an android device, the issue is that the outputs of the model are always same. Irrespective of input.
Can anyone help ?
Here is the link to that model: https://esriis-my.sharepoint.com/:u:/g/personal/san10428_esri_com/EbO-1fGDczRKnW9E2-hDhbQB17aufX_se-VdMBZrhaid3A?e=UKRiyO
Here is the link to the android project: https://esriis-my.sharepoint.com/:u:/g/personal/san10428_esri_com/ES17ARjqhcdEnvgNlpq3O5MBaDOtHN1H7kk3Q75Y9M5VTA?e=IzemCE
tensorflow version: 2.0.0

In my gradle build settings i updated the version of tensorflow binary to 2.0.0 and it is working fine now, strange things. If something is wrong it should raise and error instead of predicting random values.


text:
Same input - different output

I have written a small code, and notice that I get a different result dependent on if I use Numpy or not.
This code should give the same result for both df_dy and df_dy_p, but it does not. It does not matter if I change the order of the calculations.
`from math import e
import numpy as np
def calc_sigma(y):
return 1 / (1 + e**-y)
def get_derivative(x, y):
with tf.GradientTape(persistent=True) as tape:
tape.watch(x)
sigma = cal_sigma(-y)
f = (x + sigma) / np.power((x - y), 2)
p = (x + sigma) / ((x -y)**2)
print(x, y, f, p)
df_dy = tape.gradient(f, x)
df_dy_p = tape.gradient(p, x)
print(df_dy)
print(df_dy_p)
x = tf.constant (1.)
y = tf.constant (-3.)
get_derivative(x, y)`
The output is:
tf.Tensor(1.0, shape=(), dtype=float32) tf.Tensor(-3.0, shape=(), dtype=float32) tf.Tensor(0.12203588, shape=(), dtype=float32) tf.Tensor(0.12203588, shape=(), dtype=float32)
tf.Tensor(0.0625, shape=(), dtype=float32)
tf.Tensor(0.0014820583, shape=(), dtype=float32)
As one can see, the input is the same, but the output differs.

@oeyvindds please provide with simple stand alone code with all dependencies from the code for us to replicate in our environment, i was unable to replicate the code shared by you due to dependencies in the code.
Also please highlight the tensorflow version you are facing th eissue in.

Hello.
Here is a complete code that runs fine on my machine (Anaconda Python) in an empty notebook:
`import tensorflow as tf
import numpy as np
from math import e
def calc_sigma(y):
return 1 / (1 + e**-y)
def get_derivative(x, y):
with tf.GradientTape(persistent=True) as tape:
tape.watch(x)
sigma = calc_sigma(-y)
f = (x + sigma) / np.power((x - y), 2)
p = (x + sigma) / ((x -y)**2)
print('Variable x: {}'.format(x))
print('Variable y: {}'.format(y))
print('Input from f: {}'.format(f))
print('Input from p: {}'.format(p))
df_dy_p = tape.gradient(p, x)
df_dy = tape.gradient(f, x)

print('Output from f: {}'.format(df_dy))
print('Output from p: {}'.format(df_dy_p))

x = tf.constant (1.)
y = tf.constant (-3.)
get_derivative(x, y)`
tf.version gives '2.1.0'
The code give the following output:
Variable x: 1.0
Variable y: -3.0
Input from f: 0.12203588336706161
Input from p: 0.12203588336706161
Output from f: 0.0625
Output from p: 0.0014820583164691925

@oeyvindds please provide us with indented code that is executable and has all dependencies, code shared is not indented , dependencies are not shared.
you could share a gist of your code using colab.

Here is the code indented:
from math import e
import numpy as np
import tensorflow as tf
print(tf.__version__)

def calc_sigma(y):
    return 1 / (1 + e**-y)

def get_derivative(x, y):
    with tf.GradientTape(persistent=True) as tape:
        tape.watch(x)
        sigma = calc_sigma(-y)
        f = (x + sigma) / np.power((x - y), 2)
        p = (x + sigma) / ((x -y)**2)

        print(x, y, f, p)
    df_dy = tape.gradient(f, x)
    df_dy_p = tape.gradient(p, x)
    print(df_dy)
    print(df_dy_p)

x = tf.constant (1.)
y = tf.constant (-3.)
get_derivative(x, y)

It gives this output:
2.1.0
tf.Tensor(1.0, shape=(), dtype=float32) tf.Tensor(-3.0, shape=(), dtype=float32) tf.Tensor(0.12203588, shape=(), dtype=float32) tf.Tensor(0.12203588, shape=(), dtype=float32)
tf.Tensor(0.0625, shape=(), dtype=float32)
tf.Tensor(0.0014820583, shape=(), dtype=float32)

I am able to replicate this issue, please find the gist here

@oeyvindds and @Saduf2019, The problem is because the f is evaluated by converting (x-y)^2 in numpy before computing gradient. So, in f, denominator np.power((x-y),2) is not a variable at the time of differentiation. Instead, it is a constant with value 16.
Whereas, in p, ((x-y)**2) is a tensor and it will be variable at the time of differentiation.
Look at below two codes which are tested and their outputs:
Code 1 :
from math import e
import numpy as np
import tensorflow as tf

def calc_sigma(y):
    return 1 / (1 + e**-y)

def get_derivative(x, y):
    with tf.GradientTape(persistent=True) as tape:
        tape.watch(x)
        sigma = calc_sigma(-y)
        f = (x + sigma) / np.power((x - y), 2)
        p = (x + sigma) / ((x-y)**2).numpy()

    df_dy = tape.gradient(f, x)
    df_dy_p = tape.gradient(p, x)
    print(df_dy)
    print(df_dy_p)

x = tf.constant (1.)
y = tf.constant (-3.)
get_derivative(x, y)

Output :
tf.Tensor(0.0625, shape=(), dtype=float32)
tf.Tensor(0.0625, shape=(), dtype=float32)

Code 2:
from math import e
import numpy as np
import tensorflow as tf

def calc_sigma(y):
    return 1 / (1 + e**-y)

def get_derivative(x, y):
    with tf.GradientTape(persistent=True) as tape:
        tape.watch(x)
        sigma = calc_sigma(-y)
        f = (x + sigma) / tf.math.square(x - y)
        p = (x + sigma) / ((x-y)**2)

    df_dy = tape.gradient(f, x)
    df_dy_p = tape.gradient(p, x)
    print(df_dy)
    print(df_dy_p)

x = tf.constant (1.)
y = tf.constant (-3.)
get_derivative(x, y)

Output :
tf.Tensor(0.0014820583, shape=(), dtype=float32)
tf.Tensor(0.0014820583, shape=(), dtype=float32)


text:
RaggedTensor are converted to Tensors inside exported signature.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: no
TensorFlow installed from (source or binary): no
TensorFlow version (use command below): 2.0.1
Python version: 3.7.3
Bazel version (if compiling from source): -
GCC/Compiler version (if compiling from source): -
CUDA/cuDNN version: -
GPU model and memory: -

Describe the current behavior
As properly documented, it is currently possible to pass a possibly nested sequence of tf.TensorSpec objects as a tf.function input_signature argument.
However, if one attempts to pass a tf.RaggedTensorSpec inside an input_signature sequence, everything still works, but the RaggedTensorSpec gets converted in its "dense" values + row_splits form.
Describe the expected behavior
I would expect that if one specifies a tf.RaggedTensorSpec element inside the input_signature sequence it should either be exported as a tf.RaggedTensorSpec, or raise an error.
Of course the firs option sounds more desiderable, since it's pretty common to have ragged input sequences. In this case tf.RaggedTensorSpec would be also lacking of a name argument, but this is easier to solve.
Standalone code to reproduce the issue
Colab Notebook
import tensorflow as tf

class SomeModule(tf.Module):

  @tf.function(input_signature=[
    tf.RaggedTensorSpec(shape=[None, None], dtype=tf.string)
  ])
  def return_ragged_inputs(self, inputs: tf.RaggedTensor):
    return inputs

ragged_inputs = tf.ragged.constant([["foo"], ["bar"], ["foo", "bar"]], dtype=tf.string)

some_module = SomeModule()
return_ragged_inputs_result = some_module.return_ragged_inputs(ragged_inputs)
print(f"ragged_inputs => {ragged_inputs}\n")
print("We can use the method with ragged inputs:")
print(f"return_ragged_inputs => {return_ragged_inputs_result}\n")
print("Saving some_module...")
tf.saved_model.save(some_module, export_dir="foobar")
some_module = tf.saved_model.load("foobar")
print(f"\nWe can pass and return ragged inputs even by saving and reloading the module, if we use it 'eagerly':")
print(some_module.return_ragged_inputs(ragged_inputs))
print("\nBut the graph signature tell us that our exported signature expects tf.Tensor values instead:")
exported_model_graph_inputs = some_module.signatures[tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY].inputs
exported_model_graph_outputs = some_module.signatures[tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY].outputs
print(f"Expected inputs  : {exported_model_graph_inputs}")
print(f"Expected outputs : {exported_model_graph_outputs}")
print("\nThis means we couldn't use the method as we would expect in (e.g.) TensorFlow Serving.")

Other info / logs
This issue follows up #37049 , as suggested by @gowthamkpr .

Was able to reproduce this error. Please find the gist here.

I believe this is due to RaggedTensor not being a first-class type inside GraphDef - internally, it is represented as a CompositeTensor, which in turn is lowered to a list of Tensor, by tf.nest.flatten(expand_composites=True) when the function is compiled.
When loading a saved model, that lowering becomes visible.
As a workaround, you should be able to build inputs compatible with the loaded model by flattening the ragged tensor: tf.nest.flatten(ragged_inputs, expand_composites=True).
I think we can eventually handle them correctly inside SavedModel, by placing the necessary type spec into the graph metadata and reconstructing it upon import, but I'll let @k-w-w and @edloper give more expert insights.
Nit: in the colab, did you mean the last two lines to be "exported", not "expected"?

So, @stefanondisponibile is expecting it to be a ragged tensor but its not the case as seen in the gist.

@gowthamkpr yes, for what I've seen so far, and as explained by @mdanatg this difference between Ragged and normal Tensors pops up when the function gets converted to a Graph. I can't do it in Colab, but if you tried to load that servable with tf serving you would see you would have to pass two different dense Tensors in place of the Ragged Tensor.
@mdanatg thanks for the explanation! I am currently working around this by actually passing two dense Tensors at inference time, one with the values and the other with the splits, and building them up inside the function itself via tf.RaggedTensor.from_row_splits. So those last two lines were as "expected", since I was expecting to pass RaggedTensors to the served model, but I end up passing dense ones, building the RaggedTensor inside the function.

Loaded model signatures currently flatten any structured input into a flat list of tensors.  This is true for RaggedTensor, but also for nested structure, such as a dictionary of tensors.  E.g., if we define a module whose function takes and returns a dict of tensors, then the model signature inputs & outputs are currently flat lists of tensors:
class SomeModule(tf.Module):

  @tf.function(input_signature=[{'foo': tf.TensorSpec(None, tf.int32),
                                 'bar': tf.TensorSpec(None, tf.string)}
  ])
  def return_dict(self, inputs):
    return inputs

some_module = SomeModule()
tf.saved_model.save(some_module, export_dir="foobar2")
some_module = tf.saved_model.load("foobar2")
print(some_module.signatures[tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY].inputs)
# prints: [<tf.Tensor 'inputs:0' shape=<unknown> dtype=string>, <tf.Tensor 'inputs_1:0' shape=<unknown> dtype=int32>]
print( some_module.signatures[tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY].outputs)
# prints: [<tf.Tensor 'Identity:0' shape=<unknown> dtype=string>, <tf.Tensor 'Identity_1:0' shape=<unknown> dtype=int32>]

Hopefully we can improve on this, but the problem is a general issue with any structured inputs & outputs, and is not specific to ragged tensors.

@stefanondisponibile
Please let us know if this is still an issue.

Hey @Saduf2019 , sorry for not having replied yet! I would say that yes, this still is an issue, but as @edloper kindly explained is not specific to ragged tensors, moreover there are workarounds/better-ways for tackling this, so I'll close this specific issue. Thanks for you help!

I'm having a similar issue, my model input is a RaggedTensor (1000, None, None) connected to an LSTM layer.
The model is working as expected and when calling predict I get the predictions.
input = tf.ragged.constant([[
                            [-0.9984272718429565, -0.9422321319580078, -0.27657580375671387, -3.185823678970337, -0.6360141634941101, -1.6579184532165527, -1.9000954627990723, -0.49169546365737915, -0.6758883595466614, -0.6677696704864502, -0.532067060470581],
                            [-0.9984272718429565, -0.9421600103378296, 2.2048349380493164, -1.273996114730835, -0.6360141634941101, -1.5917999744415283, 0.6147914528846741, -0.49169546365737915, -0.6673409938812256, -0.6583622694015503, -0.5273991227149963],
                            [-0.9984272718429565, -0.942145586013794, 2.48842453956604, -1.6836735010147095, -0.6360141634941101, -1.5785763263702393, -1.900200605392456, -0.49169546365737915, -0.6656315326690674, -0.6583622694015503, -0.5273991227149963]
]])
model.predict(input)

array([[0.5138151 , 0.3277698 , 0.26122513]], dtype=float32)

When saving the model the input is split into args_0and args_0_1.
MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['__saved_model_init_op']:
  The given SavedModel SignatureDef contains the following input(s):
  The given SavedModel SignatureDef contains the following output(s):
    outputs['__saved_model_init_op'] tensor_info:
        dtype: DT_INVALID
        shape: unknown_rank
        name: NoOp
  Method name is:

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['args_0'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 11)
        name: serving_default_args_0:0
    inputs['args_0_1'] tensor_info:
        dtype: DT_INT64
        shape: (-1)
        name: serving_default_args_0_1:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['dense_2'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 3)
        name: StatefulPartitionedCall:0
  Method name is: tensorflow/serving/predict
WARNING: Logging before flag parsing goes to stderr.
W0124 13:38:49.266864 139946102503296 deprecation.py:506] From /usr/local/lib/python2.7/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1786: calling __init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.

Defined Functions:
  Function Name: '__call__'
    Option #1
      Callable with:
        Argument #1
          DType: RaggedTensorSpec
          Value: RaggedTensorSpec(TensorShape([None, None, 11]), tf.float32, 1, tf.int64)
        Argument #2
          DType: bool
          Value: True
        Argument #3
          DType: NoneType
          Value: None
    Option #2
      Callable with:
        Argument #1
          DType: RaggedTensorSpec
          Value: RaggedTensorSpec(TensorShape([None, None, 11]), tf.float32, 1, tf.int64)
        Argument #2
          DType: bool
          Value: False
        Argument #3
          DType: NoneType
          Value: None

  Function Name: '_default_save_signature'
    Option #1
      Callable with:
        Argument #1
          DType: RaggedTensorSpec
          Value: RaggedTensorSpec(TensorShape([None, None, 11]), tf.float32, 1, tf.int64)

  Function Name: 'call_and_return_all_conditional_losses'
    Option #1
      Callable with:
        Argument #1
          DType: RaggedTensorSpec
          Value: RaggedTensorSpec(TensorShape([None, None, 11]), tf.float32, 1, tf.int64)
        Argument #2
          DType: bool
          Value: True
        Argument #3
          DType: NoneType
          Value: None
    Option #2
      Callable with:
        Argument #1
          DType: RaggedTensorSpec
          Value: RaggedTensorSpec(TensorShape([None, None, 11]), tf.float32, 1, tf.int64)
        Argument #2
          DType: bool
          Value: False
        Argument #3
          DType: NoneType
          Value: None

I tried to use tf.nest.flatten as follows:
input = tf.ragged.constant([[
                            [-0.9984272718429565, -0.9422321319580078, -0.27657580375671387, -3.185823678970337, -0.6360141634941101, -1.6579184532165527, -1.9000954627990723, -0.49169546365737915, -0.6758883595466614, -0.6677696704864502, -0.532067060470581],
                            [-0.9984272718429565, -0.9421600103378296, 2.2048349380493164, -1.273996114730835, -0.6360141634941101, -1.5917999744415283, 0.6147914528846741, -0.49169546365737915, -0.6673409938812256, -0.6583622694015503, -0.5273991227149963],
                            [-0.9984272718429565, -0.942145586013794, 2.48842453956604, -1.6836735010147095, -0.6360141634941101, -1.5785763263702393, -1.900200605392456, -0.49169546365737915, -0.6656315326690674, -0.6583622694015503, -0.5273991227149963]
]])

inputs = tf.nest.flatten(input, expand_composites=True)

and got a list of 3 tensors:
[<tf.Tensor: shape=(33,), dtype=float32, numpy=
 array([-0.9984273 , -0.94223213, -0.2765758 , -3.1858237 , -0.63601416,
        -1.6579185 , -1.9000955 , -0.49169546, -0.67588836, -0.6677697 ,
        -0.53206706, -0.9984273 , -0.94216   ,  2.204835  , -1.2739961 ,
        -0.63601416, -1.5918    ,  0.61479145, -0.49169546, -0.667341  ,
        -0.65836227, -0.5273991 , -0.9984273 , -0.9421456 ,  2.4884245 ,
        -1.6836735 , -0.63601416, -1.5785763 , -1.9002006 , -0.49169546,
        -0.66563153, -0.65836227, -0.5273991 ], dtype=float32)>,
 <tf.Tensor: shape=(2,), dtype=int64, numpy=array([0, 3])>,
 <tf.Tensor: shape=(4,), dtype=int64, numpy=array([ 0, 11, 22, 33])>]

I tried calling tensorflow serving like so:
import json
import requests
headers = {"content-type": "application/json"}
data = json.dumps({"instances":[
                                  {
                                      "args_0":np.array(inputs[0]).tolist(),
                                      "args_0_1":np.array(inputs[2]).tolist()
                                  }
                               ]
                   })

json_response = requests.post('http://localhost:8501/v1/models/fashion_model:predict', data=data, headers=headers)
predictions = json.loads(json_response.text)

But got the following error:
{'error': 'Matrix size-incompatible: In[0]: [0,33], In[1]: [11,64]\n\t [[{{node sequential_1/first_lstm/lstm_cell/MatMul}}]]'}
What am I missing?

I'm having a similar issue, my model input is a RaggedTensor (1000, None, None) connected to an LSTM layer.
The model is working as expected and when calling predict I get the predictions.
input = tf.ragged.constant([[
                            [-0.9984272718429565, -0.9422321319580078, -0.27657580375671387, -3.185823678970337, -0.6360141634941101, -1.6579184532165527, -1.9000954627990723, -0.49169546365737915, -0.6758883595466614, -0.6677696704864502, -0.532067060470581],
                            [-0.9984272718429565, -0.9421600103378296, 2.2048349380493164, -1.273996114730835, -0.6360141634941101, -1.5917999744415283, 0.6147914528846741, -0.49169546365737915, -0.6673409938812256, -0.6583622694015503, -0.5273991227149963],
                            [-0.9984272718429565, -0.942145586013794, 2.48842453956604, -1.6836735010147095, -0.6360141634941101, -1.5785763263702393, -1.900200605392456, -0.49169546365737915, -0.6656315326690674, -0.6583622694015503, -0.5273991227149963]
]])
model.predict(input)

array([[0.5138151 , 0.3277698 , 0.26122513]], dtype=float32)

When saving the model the input is split into args_0and args_0_1.
MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

signature_def['__saved_model_init_op']:
  The given SavedModel SignatureDef contains the following input(s):
  The given SavedModel SignatureDef contains the following output(s):
    outputs['__saved_model_init_op'] tensor_info:
        dtype: DT_INVALID
        shape: unknown_rank
        name: NoOp
  Method name is:

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['args_0'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 11)
        name: serving_default_args_0:0
    inputs['args_0_1'] tensor_info:
        dtype: DT_INT64
        shape: (-1)
        name: serving_default_args_0_1:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['dense_2'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 3)
        name: StatefulPartitionedCall:0
  Method name is: tensorflow/serving/predict
WARNING: Logging before flag parsing goes to stderr.
W0124 13:38:49.266864 139946102503296 deprecation.py:506] From /usr/local/lib/python2.7/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1786: calling __init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.

Defined Functions:
  Function Name: '__call__'
    Option #1
      Callable with:
        Argument #1
          DType: RaggedTensorSpec
          Value: RaggedTensorSpec(TensorShape([None, None, 11]), tf.float32, 1, tf.int64)
        Argument #2
          DType: bool
          Value: True
        Argument #3
          DType: NoneType
          Value: None
    Option #2
      Callable with:
        Argument #1
          DType: RaggedTensorSpec
          Value: RaggedTensorSpec(TensorShape([None, None, 11]), tf.float32, 1, tf.int64)
        Argument #2
          DType: bool
          Value: False
        Argument #3
          DType: NoneType
          Value: None

  Function Name: '_default_save_signature'
    Option #1
      Callable with:
        Argument #1
          DType: RaggedTensorSpec
          Value: RaggedTensorSpec(TensorShape([None, None, 11]), tf.float32, 1, tf.int64)

  Function Name: 'call_and_return_all_conditional_losses'
    Option #1
      Callable with:
        Argument #1
          DType: RaggedTensorSpec
          Value: RaggedTensorSpec(TensorShape([None, None, 11]), tf.float32, 1, tf.int64)
        Argument #2
          DType: bool
          Value: True
        Argument #3
          DType: NoneType
          Value: None
    Option #2
      Callable with:
        Argument #1
          DType: RaggedTensorSpec
          Value: RaggedTensorSpec(TensorShape([None, None, 11]), tf.float32, 1, tf.int64)
        Argument #2
          DType: bool
          Value: False
        Argument #3
          DType: NoneType
          Value: None

I tried to use tf.nest.flatten as follows:
input = tf.ragged.constant([[
                            [-0.9984272718429565, -0.9422321319580078, -0.27657580375671387, -3.185823678970337, -0.6360141634941101, -1.6579184532165527, -1.9000954627990723, -0.49169546365737915, -0.6758883595466614, -0.6677696704864502, -0.532067060470581],
                            [-0.9984272718429565, -0.9421600103378296, 2.2048349380493164, -1.273996114730835, -0.6360141634941101, -1.5917999744415283, 0.6147914528846741, -0.49169546365737915, -0.6673409938812256, -0.6583622694015503, -0.5273991227149963],
                            [-0.9984272718429565, -0.942145586013794, 2.48842453956604, -1.6836735010147095, -0.6360141634941101, -1.5785763263702393, -1.900200605392456, -0.49169546365737915, -0.6656315326690674, -0.6583622694015503, -0.5273991227149963]
]])

inputs = tf.nest.flatten(input, expand_composites=True)

and got a list of 3 tensors:
[<tf.Tensor: shape=(33,), dtype=float32, numpy=
 array([-0.9984273 , -0.94223213, -0.2765758 , -3.1858237 , -0.63601416,
        -1.6579185 , -1.9000955 , -0.49169546, -0.67588836, -0.6677697 ,
        -0.53206706, -0.9984273 , -0.94216   ,  2.204835  , -1.2739961 ,
        -0.63601416, -1.5918    ,  0.61479145, -0.49169546, -0.667341  ,
        -0.65836227, -0.5273991 , -0.9984273 , -0.9421456 ,  2.4884245 ,
        -1.6836735 , -0.63601416, -1.5785763 , -1.9002006 , -0.49169546,
        -0.66563153, -0.65836227, -0.5273991 ], dtype=float32)>,
 <tf.Tensor: shape=(2,), dtype=int64, numpy=array([0, 3])>,
 <tf.Tensor: shape=(4,), dtype=int64, numpy=array([ 0, 11, 22, 33])>]

I tried calling tensorflow serving like so:
import json
import requests
headers = {"content-type": "application/json"}
data = json.dumps({"instances":[
                                  {
                                      "args_0":np.array(inputs[0]).tolist(),
                                      "args_0_1":np.array(inputs[2]).tolist()
                                  }
                               ]
                   })

json_response = requests.post('http://localhost:8501/v1/models/fashion_model:predict', data=data, headers=headers)
predictions = json.loads(json_response.text)

But got the following error:
{'error': 'Matrix size-incompatible: In[0]: [0,33], In[1]: [11,64]\n\t [[{{node sequential_1/first_lstm/lstm_cell/MatMul}}]]'}
What am I missing?

authors: Saduf2019 edloper gowthamkpr mdanatg shlomi-schwartz stefanondisponibile


text:
Potential bug in Tensorflow 1 integration with Jupyter notebook

When running the attached jupyter notebook example from Manning book "Tensorflow in action" - I know there are some typos in the code but I correct them to make the code works - , I meet the following condition

When I run all steps for the first time, everything works fine

When I run the tf placeholder declaration twice

Code[
alpha = tf.constant(0.05)
curr_value = tf.placeholder(tf.float32)
prev_avg = tf.Variable(0.)
update_avg = alpha * curr_value + (1 - alpha) * prev_avg
]
the session crashes with an error :
Code[
init = tf.global_variables_initializer()
with tf.Session() as sess:
sess.run(init)
writer.add_graph(sess.graph)
for i in range(len(raw_data)):
summary_str, curr_avg = sess.run([merged, update_avg], feed_dict={curr_value: raw_data[i]})
sess.run(tf.assign(prev_avg, curr_avg))
print(raw_data[i], curr_avg)
writer.add_summary(summary_str, i)
]
Error message[
InvalidArgumentError                      Traceback (most recent call last)
 in ()
5     writer.add_graph(sess.graph)
6     for i in range(len(raw_data)):
----> 7         summary_str, curr_avg = sess.run([merged, update_avg], feed_dict={curr_value: raw_data[i]})
8         sess.run(tf.assign(prev_avg, curr_avg))
9         print(raw_data[i], curr_avg)
/usr/local/lib/python2.7/dist-packages/tensorflow_core/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata)
958     try:
959       result = self._run(None, fetches, feed_dict, options_ptr,
--> 960                          run_metadata_ptr)
961       if run_metadata:
962         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)
/usr/local/lib/python2.7/dist-packages/tensorflow_core/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata)
1181     if final_fetches or final_targets or (handle and feed_dict_tensor):
1182       results = self._do_run(handle, final_targets, final_fetches,
-> 1183                              feed_dict_tensor, options, run_metadata)
1184     else:
1185       results = []
...]
Workaround : I have to restart kernel and clear all outputs to run the session again.
Possible cause : either tensorflow or jupyter doesn't detect that placeholders have already been instantiated and instantiates them twice.
Environment : docker TF container tensorflow/tensorflow:latest-gpu-jupyter
ch2Tensorboard.ipynb.txt

@mdautrey, can you please share the full code or link of colab file to reproduce mentioned issue.

I think that the full code is attached to my initial post https://github.com/tensorflow/tensorflow/files/4362615/ch2Tensorboard.ipynb.txt

@mdautrey, I have checked this file by running on google colab and i found no issue. for your reference link of gist is here.

When you run the code twice, you get the attached error, even on google colab

when we run the code twice in colab i am able to reproduce the issue. Thanks!

Try adding tf.reset_default_graph() before creating graph. That should help. Thanks!
tf.reset_default_graph()
alpha = tf.constant(0.05)
...

It works! Thank you
Should the documentation be updated accordingly ? (https://www.tensorflow.org/api_docs/python/tf/compat/v1/summary/FileWriter )

authors: khimraj mdautrey ravikyram ymodak


text:
Model.add_loss is not documented

Issue type
Documentation Bug
Have you reproduced the bug with TensorFlow Nightly?
No
Source
binary
TensorFlow version
2.16.1
Custom code
Yes
OS platform and distribution
No response
Mobile device
No response
Python version
No response
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current behavior?
In the documentation of tf.keras.Model as available at https://www.tensorflow.org/api_docs/python/tf/keras/Model, the method Model.add_loss has been used in the example code for the method compute_loss. However, the method Model.add_loss itself has not been documented on this page.
The keras documentation provides some documentation for Layer.add_loss, but I couldn't find any documentation even there for Model.add_loss.
Standalone code to reproduce the issue
This is a missing documentation issue, so there is no code.

Relevant log output
No response

@piyushsrv The Model page doesn't explicitly mention add_loss, you can still use it on a Model object because it inherits from Layer. Model inherits from Layer in Keras. This means it gets all the methods and properties of the Layer by default. You can refer to the Layer.add_loss documentation for details on how it works: https://www.tensorflow.org/api_docs/python/tf/keras/losses.
Thank you!

authors: piyushsrv sushreebarsa


text:
Use asarray to avoid copy - ~2x speedup for feed_dict

This starts to address #2919

Jenkins, test this please.

@nanddalal can you sign the CLA?

Hi, sorry for the delay - I am still working on signing the CLA so I can contribute later on, but if you were already able to incorporate this change in a different PR, that would be great.

It turns out I fixed this independently in another commit (which is will appear in the next push from internal) - thanks for noticing the issue though!

authors: ebrevdo mrry nanddalal


text:
[ROCm] Adding ROCm support for the reduction ops

This PR adds ROCm support for the reduction ops
The changes in this PR are mostly trivial, please review and merge.
Notes:

ROCm does not have support for complex datatype for some BLAS operations. As a consequence you will see #ifdef GOOGLE_CUDA around code related to complex datatype
This PR merge (https://github.com/tensorflow/tensorflow/commit/41c36223dcd265603ca6916fec0a88016eb9abaa#diff-dfb288ce4a5f0ff51ccd12df2c86edac) seems to have accidentally undone ROCm support for the where op. The second commit in this PR, re-enables the ROCm support for the where op. It is included as a part of this PR, because ROCm support for where op is required for the --config=rocm build to complete, when ROCm support for reduction ops is also enabled.

@tatianashp @whchung @chsigg

@rthadur , I will push out a rebase shortly to resolve the merge conflict.

rebase done.
@chsigg , please re-approve...thanks

@rthadur
this is the 3rd of 4 PRs that seem to be stuck in the merge pipeline.
Please let me know if there is anything that needs to be done on my end.
thanks
deven

@rthadur
this is the 3rd of 4 PRs that seem to be stuck in the merge pipeline.
Please let me know if there is anything that needs to be done on my end.
thanks
deven

trying to pull this in again , thank you

@deven-amd there are some internal test failures happening for this change , @chsigg can you please help to resolve those ?

@rthadur @chsigg gentle ping. thanks.

@chsigg while we have your attention, can you please also propose next steps for this PR?

@chsigg gentle ping.

authors: dagamayank deven-amd rthadur


text:
[EXT-SYSLIB] Add absl_py logging submodule to build flow.

Enable proper build when absl_py is external LOCAL_LIBS.
ERROR: /home/cbalint/rpmbuild/BUILD/tensorflow/tensorflow/tensorflow/python/distribute/BUILD:1733:11: no such package '@absl_py//absl/logging': BUILD file not found in directory 'absl/logging' of external repository @absl_py. Add a BUILD file to a directory to mark it as a package. and referenced by '//tensorflow/python/distribute:multi_process_runner'

Cc @perfinion
Thank You !

@perfinion ping


text:
[MLIR][DISC] legalize disc to llvm

This PR provides conversion patterns to convert disc_ral.dispatch_op
and gpu.launch_op to its disc-compatible llvm format.

@joker-eph Hi, could you help to review the updated version? Thanks!

@joker-eph Hi, could you help to take a look? Thanks!

@joker-eph Hi，could you help to take a look? Thanks!

You may have missed my comment inline: https://github.com/tensorflow/tensorflow/pull/50434#discussion_r661107002 ?

authors: joker-eph wyzero