text:
Provide reachability info with describe task queue for the new versioning API (Backward incompatible)

What was changed

Provides an enhanced mode (by default) to temporal task-queue describe that provides task reachability information
for Build IDs. The goal is to facilitate retiring workers with old code that use the new worker versioning API.
The old behavior of describe is available using the legacy-mode flag.

Reachability could be an expensive operation and many time the callers may only be interested in pollers only. I think we should make it disabled by default and only report it when user sets a flag such as --report-reachability or similar.

Also similarly, one may not want poller info so ideally there should be a way to disable that such as --report-pollers=false.
Note that a new section will be added soon (BacklogInfo). That should also have a flag (maybe true by default).

Reachability could be an expensive operation and many time the callers may only be interested in pollers only. I think we should make it disabled by default and only report it when user sets a flag such as --report-reachability or similar.

I was thinking whether to export the flags or not. I thought it was a bit confusing not providing the info by default, and this is the cli, so I'm not expecting lots of calls using it, and create a real overhead problem. I'm assuming that most demanding users will use the SDK directly, and they can apply there the flags. Also, even with report off it provides info in json mode (e.g., similar to no pollers found), something that is confusing.
Maybe we can revisit after we get more experience with the pre-release?

:+1: Would like to know how expensive. It is a common use case for people to poll reachability I assume so you know when you can shut a worker down?

The expense comes from the fact that it needs to make up to 2 Visibility queries per build ID. Although those queries will not be needed for the "default" build ID (when no --select-* flag is used). We do caching of the Visibility queries in server, so maybe at the end of the day it is OK. We plan to do some more optimizations to reduce the visibility query cost.
I'm fine with keeping this as is and revisit after prerelease if we are OK adding the flags (and change default behavior) at a later stage if needed.

I'm fine with keeping this as is and revisit after prerelease if we are OK adding the flags (and change default behavior) at a later stage if needed.

Hrmm, I'm not sure we are ok w/ removing this from default and making a flag later. I noticed in the pending Go SDK update we require users to opt-in to this data as well. @antlai-temporal - if this data is not available by default in SDK or API and has an expense, can we put it behind a flag here too?

Hrmm, I'm not sure we are ok w/ removing this from default and making a flag later. I noticed in the pending Go SDK update >we require users to opt-in to this data as well. @antlai-temporal - if this data is not available by default in SDK or API and has >an expense, can we put it behind a flag here too?

There is caching and rate limiting built-in @ShahabT mentioned, and I'm not expecting cli users to be doing heavy stuff. More advanced users will use the SDK, where there is by default more control.
I really hate it when you issue a command in the cli, and nothing useful comes back, and you need to start digging with the options to get anything sensible... I prefer to get the info back by default, and if we need to save, have a disable flag. But instead of falling for premature optimization, I wanted to wait and see whether the flag was actually needed...

nothing useful comes back

But something useful is coming back by default (poller info), just not this. Many users don't need reachability anyways because they are not using versioning.

But something useful is coming back by default (poller info), just not this. Many users don't need reachability anyways because they are not using versioning.

In the SDK poller info is also gated by a flag, so if we do the same nothing would be returned by the cli by default. We could just add the enableReachability flag and get rid of the enablePollerInfo flag in the cli...

We could just add the enableReachability flag and get rid of the enablePollerInfo flag in the cli...

Yes, I vote this. Lets show the cheap valuable-to-everyone poller info by default and the more expensive valuable-to-versioning-users-only reachability info a flag is set.

Adding report-reachability flag. Note that there is a bug in the current server version that makes a new unit test fail. This will be fixed upgrading the server hash in the next "NOT FOR MERGE" commit in the versioning-2 branch.

authors: ShahabT antlai-temporal cretz

repo: temporalio/cli


text:
TestWorkflowEnvironment does not allow signalWithStart when workflow already exists

Expected Behavior
TestWorkflowEnvironment does not allow signalWithStart when workflow already exists - see exception below.
Actual Behavior
TestWorkflowEnvironment should work the same as standalone server in this matter.
Steps to Reproduce the Problem
Reproducer: SimpleWorkflowTest in https://github.com/rpost/temporal-test-bug - when test is run it fails with
Workflow execution {workflowId='my-workflow-2', runId='', workflowType='SampleWorkflow'} failed. Metadata: {closeEventType='EVENT_TYPE_WORKFLOW_EXECUTION_FAILED', retryState='RETRY_STATE_UNSPECIFIED', workflowTaskCompletedEventId=3'}
io.temporal.client.WorkflowFailedException: Workflow execution {workflowId='my-workflow-2', runId='', workflowType='SampleWorkflow'} failed. Metadata: {closeEventType='EVENT_TYPE_WORKFLOW_EXECUTION_FAILED', retryState='RETRY_STATE_UNSPECIFIED', workflowTaskCompletedEventId=3'}
	at io.temporal.internal.common.WorkflowExecutionUtils.getResultFromCloseEvent(WorkflowExecutionUtils.java:101)
	at io.temporal.internal.client.WorkflowClientLongPollHelper.getWorkflowExecutionResult(WorkflowClientLongPollHelper.java:70)
	at io.temporal.internal.client.RootWorkflowClientInvoker.getResult(RootWorkflowClientInvoker.java:353)
	at io.temporal.client.WorkflowStubImpl.getResult(WorkflowStubImpl.java:259)
	at io.temporal.client.WorkflowStubImpl.getResult(WorkflowStubImpl.java:240)
	at io.temporal.client.WorkflowStubImpl.getResult(WorkflowStubImpl.java:233)
	at io.temporal.testing.TimeLockingInterceptor$TimeLockingWorkflowStub.getResult(TimeLockingInterceptor.java:113)
	at rpost.SimpleWorkflowTest.shouldSignalWithStart(SimpleWorkflowTest.java:70)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:218)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:214)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:139)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
	at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:119)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:94)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:89)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:62)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
	at jdk.proxy1/jdk.proxy1.$Proxy2.stop(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:119)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:66)
	at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
	at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: io.temporal.failure.ActivityFailure: Activity with activityType='SignalWithStart' failed: 'Local Activity task failed'. scheduledEventId=0, startedEventId=0, activityId=a2731aa9-d29c-3699-b58f-73097f7cd4ea, identity='', retryState=RETRY_STATE_RETRY_POLICY_NOT_SET
	at io.temporal.failure.DefaultFailureConverter.failureToExceptionImpl(DefaultFailureConverter.java:163)
	at io.temporal.failure.DefaultFailureConverter.failureToException(DefaultFailureConverter.java:79)
	at io.temporal.common.converter.PayloadAndFailureDataConverter.failureToException(PayloadAndFailureDataConverter.java:143)
	... 88 more
Caused by: io.temporal.failure.ApplicationFailure: message='workflowId='mutexId', runId='37e667cc-b775-4a6a-a404-8798b7d217a3', workflowType='MutexWorkflow'', type='io.temporal.client.WorkflowExecutionAlreadyStarted', nonRetryable=false
	at io.temporal.client.WorkflowStubImpl.wrapStartException(WorkflowStubImpl.java:484)
	at io.temporal.client.WorkflowStubImpl.signalWithStartWithOptions(WorkflowStubImpl.java:190)
	at io.temporal.client.WorkflowStubImpl.signalWithStart(WorkflowStubImpl.java:217)
	at io.temporal.testing.TimeLockingInterceptor$TimeLockingWorkflowStub.signalWithStart(TimeLockingInterceptor.java:86)
	at rpost.SampleWorkflow$SignalWithStartActivityImpl.signalWithStart(SampleWorkflow.java:61)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at io.temporal.internal.activity.RootActivityInboundCallsInterceptor$POJOActivityInboundCallsInterceptor.executeActivity(RootActivityInboundCallsInterceptor.java:64)
	at io.temporal.internal.activity.RootActivityInboundCallsInterceptor.execute(RootActivityInboundCallsInterceptor.java:43)
	at io.temporal.internal.activity.ActivityTaskExecutors$BaseActivityTaskExecutor.execute(ActivityTaskExecutors.java:107)
	at io.temporal.internal.activity.ActivityTaskHandlerImpl.handle(ActivityTaskHandlerImpl.java:124)
	at io.temporal.internal.worker.LocalActivityWorker$AttemptTaskHandlerImpl.handle(LocalActivityWorker.java:472)
	at io.temporal.internal.worker.LocalActivityWorker$AttemptTaskHandlerImpl.handle(LocalActivityWorker.java:399)
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: io.temporal.failure.ApplicationFailure: message='ALREADY_EXISTS: WorkflowId: mutexId, RunId: 37e667cc-b775-4a6a-a404-8798b7d217a3', type='io.grpc.StatusRuntimeException', nonRetryable=false
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271)
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252)
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165)
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.signalWithStartWorkflowExecution(WorkflowServiceGrpc.java:4639)
	at io.temporal.internal.client.external.GenericWorkflowClientImpl.lambda$signalWithStart$3(GenericWorkflowClientImpl.java:113)
	at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:69)
	at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:60)
	at io.temporal.internal.client.external.GenericWorkflowClientImpl.signalWithStart(GenericWorkflowClientImpl.java:108)
	at io.temporal.internal.client.RootWorkflowClientInvoker.signalWithStart(RootWorkflowClientInvoker.java:164)
	at io.temporal.client.WorkflowStubImpl.signalWithStartWithOptions(WorkflowStubImpl.java:180)
	... 15 more

whereas when main method is run (with temporal server start-dev in background) it finishes fine.
Specifications

sdk version: 1.26.1
temporal server start-dev: CLI 1.1.0 (Server 1.25.0, UI 2.30.3)

Hm I tried to translate your reproduction into an SDK test and was not able to reproduce any bug. I will try to run your reproduction directly.

source: gharchive/issue

authors: Quinn-With-Two-Ns rpost

repo: temporalio/sdk-java

license_type: permissive

license_source: github-api


text:
Deserialization things

add to public API: consume_length and consume_prefix to have less code duplication in kms
remove explicit return statements: src/encoding.rs
rustfmt on amino-derive/src/lib.rs
authors

Looks great

Thanks for the review :-) You need to approve the changes, so that I can merge.

circleci seems stuck :-/ closing and reopening to trigger build

Ah OK. That's annoying: https://circleci.com/docs/2.0/workflows-waiting-status/

source: gharchive/pull-request


text:
tendermint-rs: Add AbciQuery to Method enum

Currently calling abci_query calls the abci_info rpc

D'oh. It'd be good to have an integration test for this (it's the only RPC endpoint that doesn't have one, afaik, because I didn't know how to write one).
That said, I'm going to merge this as-is. And FYI, this whole crate is moving soon to:
https://github.com/interchainio/tendermint-rs

How are you currently running all other integration tests?
Do I need a certain tendermint node running in the background?

The integration tests presently have the #[ignore] attribute, so you need to run them with:
$ cargo test --all-features -- --ignored

I'm presently testing them against gaiad hitting cosmoshub-2, although they're presently written in such a way they should work against any active Tendermint node which has synced to an outside network

authors: amanusk tarcieri

repo: tendermint/kms


text:
🛑 Cloud Storage API (minio-api.tcloud.dev) is down

In 33e390b, Cloud Storage API (minio-api.tcloud.dev) (https://minio-api.tcloud.dev/minio/health/live) was down:

HTTP code: 523
Response time: 3287 ms

Resolved: Cloud Storage API (minio-api.tcloud.dev) is back up in a346b66 after 37 minutes.

repo: tensegrity-labs-admin/uptime

license: MIT


text:
Deprecate MultiHeadAttention

Deprecate MultiHeadAttention since https://github.com/tensorflow/tensorflow/commit/f32c80b3ed0d64eb0363f4196171467de79390d1 is merged.
I'm quite confused what our roadmap is. Should we alias/wrap the functionality to core TF or just remove it? In gelu case, the default argument is changed, and for MultiHeadAttention, the input signature is totally different.

cc @tensorflow/sig-addons-maintainers

Yeah there are differences. IMO, we should remove it.

Just wondering, is there any policy on backward compatibility?

We have a no merged draft at https://github.com/tensorflow/community/pull/241

authors: AakashKumarNain WindQAQ bhack failure-to-thrive

repo: tensorflow/addons


text:
Linter throwing CS errors #1558

Issue https://github.com/tensorflow/magenta/issues/1558
Running "pytest"  before
6 failed, 738 passed, 318 skipped, 47 warnings in 170.86 seconds

Running "pytest" after
741 passed, 321 skipped, 47 warnings in 181.72 seconds

P.S. This is a new, cleaner, PR for the same issue, the old one got messy(after syncing with the master) and was closed, https://github.com/tensorflow/magenta/pull/1559

@adarob Probably there's a mismatch between the version python on CI and my local mac, which makes test to fail, python 3.5 vs python 3.7 and its safer to close the PR, and do not merge it at all.
Also, master is broken.

repo: tensorflow/magenta


text:
Tensorboard does not support multiple google cloud directory as logdir

I would like to launch tensorboard with
tensorboard --logdir gs://path1,gs://path2

or even better
 tensorboard --logdir model1:gs://path/1,model2:gs://path/2

but this is not currently supported.
It should be an easy addition and I can provide a patch if it make sense.

This works for me. Help me better understand why it doesn't work for you?

it only displays the first directory but not the second. Is useful you want to compare two different runs on Google cloud.

Chances are something's weird about the second one. Maybe it's slow to load. Maybe it's corrupted. Try flipping 1 and 2 and see what happens.

I'm closing this due to inactivity. Please feel free to re-open it in our new repository at https://github.com/tensorflow/tensorboard.

authors: Mistobaan jart wchargin

repo: tensorflow/tensorflow


text:
Infiniband with tensorflow

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Linux Ubuntu 14.04
TensorFlow installed from (source or binary):
source
TensorFlow version (use command below):
commit 3bee923c9
Bazel version (if compiling from source):
0.4.5
CUDA/cuDNN version:
cuda 8.0/cudnn 5.1.5
GPU model and memory:
Titan Xp
Exact command to reproduce:

Describe the problem
I tried to use Infiniband with tensorflow using  server = tf.train.Server(cluster, job_name=FLAGS.job_name, task_index=FLAGS.task_index, protocol='grpc+verbs').
But the session is not hanged.
I used S1, and S2, the cluster is configured with the Infiniband ip address. Is there something that I missed?
ibdev2netdev
S1
mlx5_0 port 1 ==> ib0 (Down)
mlx5_1 port 1 ==> ib1 (Up)
S2
mlx5_0 port 1 ==> ib0 (Up)
mlx5_1 port 1 ==> ib1 (Up)
Source code / logs

This a known issue that when the first IB port is down, the verbs server wouldn't run. You could manually change it to the second port here.
Or you could wait a little bit more time to try my GDR patch when it is merged :)

Sorry for only addressing this now, did you try changing the environment variable RDMA_DEVICE?


text:
Android - No OpKernel was registered to support Op 'Cos' with these attrs.  Registered devices: [CPU], Registered kernels:                                                                                [[Node: stft/hann_window/Cos = CosT=DT_FLOAT, _device="/device:CPU:0"]]

I've exported a model to Android that uses :
stfts = tf.contrib.signal.stft(transwav, frame_length=2048, frame_step=512,
fft_length=2048,window_fn=functools.partial(tf.contrib.signal.hann_window, periodic=False), pad_end=True)
the model works properly in pyhton
but when i load it in Android using the downloaded compiled tensorflow - compile 'org.tensorflow:tensorflow-android:+'
I get this error -
FATAL EXCEPTION: main
Process: org.tensorflow.demo, PID: 25836
java.lang.IllegalArgumentException: No OpKernel was registered to support Op 'Cos' with these attrs.  Registered devices: [CPU], Registered kernels:
 	 [[Node: stft/hann_window/Cos = CosT=DT_FLOAT, _device="/device:CPU:0"]]
which comes from the hann_window
any recommended work around?

This is a very common issue with Android development. You need to make sure all the kernels you need are in your build. Please check StackOverflow and search Google for similar issues.

Please note that C++ kernels use a registration pattern.

I have recompiled libtensorflow_inference.so with print_selective_registration_header
and i see in the ops_to_register.h file the Cos OP -
...
constexpr inline bool ShouldRegisterOp(const char op[]) {
return false
|| isequal(op, "Add")
|| isequal(op, "BatchToSpaceND")
|| isequal(op, "CTCBeamSearchDecoder")
|| isequal(op, "Cast")
|| isequal(op, "Complex")
|| isequal(op, "ComplexAbs")
|| isequal(op, "ConcatV2")
|| isequal(op, "Const")
|| isequal(op, "Conv2D")
|| isequal(op, "Cos")
|| isequal(op, "DecodeWav")
|| isequal(op, "Exp")
|| isequal(op, "ExpandDims")
|| isequal(op, "Fill")...
there for it should be compiled into the libtensorflow_inference.so
but i still get the same error message when trying to run the model in my android app

@eli99999 Please let me know if this help.

@bobeo thanks,
I looked into it, and it seems that there is  a problem with the bazel compilation
i changed the models a few time and then did the ops_to_register.h process,
each time the ops_to_register.h file changed but the libtensorflow_inference.so  does not.
I tried all sort of bazel clean and bazel dump, and not chaned the resulted  libtensorflow_inference.so
so i think that either there is some issue with the bazel caching or that the  bug is in the print_selective_registration_header process
i'm working on ubuntu 16.04
bazel 0.7.0
tf 1.4

I've answered a similar question on StackOverflow: https://stackoverflow.com/questions/49116833/no-opkernel-was-registered-to-support-op-cos-when-running-inference-on-android
Apparently, you might need to "hack" the BUILD file, according to one of the TensorFlow developers: https://github.com/tensorflow/tensorflow/issues/11804#issuecomment-318415228

authors: bobeo eli99999 jart naktinis


text:
Encountered a training error when using slim with multi-GPU connect by PIX type

Hi all, I encountered a training error when I training model in multi GPU.
The key condition is using slim to write model and training in multi GPU(GTX TITAN X) which connect by PIX type.
How to reproduce:

checkout office code in here

copy code in below(slim model code) and replace it to cifar10.py file. (Because only slim code can reproduce this issue)

start to training: CUDA_VISIBLE_DEVICES=1,0 python cifar10_multi_gpu_train.py --num_gpus=2

an incorrect loss will appear. (Also GPU usage is incorrect)

Note 1: Check GPU info in below. this issue only happend when I using GPU 0,1 or GPU 2,3 or GPU 4,5 or GPU 6,7(connect by PIX). In this case, everything is ok If I using other combinations
Note 2: I can't reproduce this issue in GTX 1080 ti
Thank you so much!
The training error log in below:
linux@172.25.52.02:~/models/tutorials/image/cifar10: CUDA_VISIBLE_DEVICES=1,0 python cifar10_multi_gpu_train.py --num_gpus=2
Filling queue with 20000 CIFAR images before starting to train. This will take a few minutes.
2017-11-09 18:13:02.264529: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are
available on your machine and could speed up CPU computations.
2017-11-09 18:13:06.359127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties:
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:05:00.0
Total memory: 11.92GiB
Free memory: 11.81GiB
2017-11-09 18:13:06.784690: W tensorflow/stream_executor/cuda/cuda_driver.cc:523] A non-primary context 0x5f00120 exists before initializing the StreamExecutor
. We haven't verified StreamExecutor works with that.
2017-11-09 18:13:06.786072: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 1 with properties:
name: GeForce GTX TITAN X
major: 5 minor: 2 memoryClockRate (GHz) 1.076
pciBusID 0000:04:00.0
Total memory: 11.92GiB
Free memory: 11.81GiB
2017-11-09 18:13:06.786507: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0 1
2017-11-09 18:13:06.786527: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0:   Y Y
2017-11-09 18:13:06.786592: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 1:   Y Y
2017-11-09 18:13:06.786623: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN X, pci bus id: 0000:05:00.0)
2017-11-09 18:13:06.786635: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX TITAN X, pci bus id: 0000:04:00.0)
2017-11-09 18:13:06.830077: I tensorflow/core/common_runtime/simple_placer.cc:697] Ignoring device specification /device:GPU:1 for node 'tower_1/fifo_queue_Dequeue' because the input edge from 'prefetch_queue/fifo_queue' is a reference connection and already has a device field set to /device:CPU:0
2017-11-09 18:13:06.830125: I tensorflow/core/common_runtime/simple_placer.cc:697] Ignoring device specification /device:GPU:0 for node 'tower_0/fifo_queue_Dequeue' because the input edge from 'prefetch_queue/fifo_queue' is a reference connection and already has a device field set to /device:CPU:0
2017-11-09 18:13:09.804877: step 0, loss = 2.30 (119.9 examples/sec; 1.067 sec/batch)
2017-11-09 18:13:17.096214: step 10, loss = 2.30 (421.3 examples/sec; 0.304 sec/batch)
2017-11-09 18:13:23.708459: step 20, loss = 2.30 (390.0 examples/sec; 0.328 sec/batch)
2017-11-09 18:13:31.362454: step 30, loss = 2.30 (362.5 examples/sec; 0.353 sec/batch)
2017-11-09 18:13:38.225150: step 40, loss = 2.30 (363.3 examples/sec; 0.352 sec/batch)
2017-11-09 18:13:44.695175: step 50, loss = 2.30 (389.8 examples/sec; 0.328 sec/batch)
2017-11-09 18:13:51.994245: step 60, loss = 2.30 (362.5 examples/sec; 0.353 sec/batch)
2017-11-09 18:13:58.623306: step 70, loss = 2.30 (421.2 examples/sec; 0.304 sec/batch)
2017-11-09 18:14:05.480061: step 80, loss = 2.30 (390.6 examples/sec; 0.328 sec/batch)
2017-11-09 18:14:11.952376: step 90, loss = 2.30 (389.2 examples/sec; 0.329 sec/batch)
2017-11-09 18:14:18.375831: step 100, loss = 293778.34 (362.7 examples/sec; 0.353 sec/batch)
Traceback (most recent call last):
  File "cifar10_multi_gpu_train.py", line 283, in <module>
    tf.app.run()
  File "/home/zhangjiguo/.local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "cifar10_multi_gpu_train.py", line 278, in main
    train()
  File "cifar10_multi_gpu_train.py", line 251, in train
    assert not np.isnan(loss_value), 'Model diverged with loss = NaN'
AssertionError: Model diverged with loss = NaN

My GPU list with command nvidia-smi -L
GPU 0: GeForce GTX TITAN X (UUID: GPU-2c9601fc-369e-fce7-d139-8c7144700def)
GPU 1: GeForce GTX TITAN X (UUID: GPU-eb4d8327-7058-39fe-e510-3437fe258745)
GPU 2: GeForce GTX TITAN X (UUID: GPU-62fbacf7-0120-5c6b-ceea-ad026fddc537)
GPU 3: GeForce GTX TITAN X (UUID: GPU-15be755d-9982-f4f4-4455-1d648d836b6e)
GPU 4: GeForce GTX TITAN X (UUID: GPU-e48fae89-debd-9769-3851-61ba44acd6e1)
GPU 5: GeForce GTX TITAN X (UUID: GPU-f796ad02-e104-04b5-d6c1-ff72d872654f)
GPU 6: GeForce GTX TITAN X (UUID: GPU-7e27fd1e-ba3b-2dd3-6d03-960b8c76a0be)
GPU 7: GeForce GTX TITAN X (UUID: GPU-8de522e7-f375-ab05-28ff-5ddfd9e325b0)

and GPU connection info with command nvidia-smi topo -m
       	GPU0   	GPU1   	GPU2   	GPU3   	GPU4   	GPU5   	GPU6   	GPU7   	mlx4_0 	CPU Affinity
GPU0   	 X     	PIX    	PHB    	PHB    	SOC    	SOC    	SOC    	SOC    	SOC    	0-13,28-41
GPU1   	PIX    	 X     	PHB    	PHB    	SOC    	SOC    	SOC    	SOC    	SOC    	0-13,28-41
GPU2   	PHB    	PHB    	 X     	PIX    	SOC    	SOC    	SOC    	SOC    	SOC    	0-13,28-41
GPU3   	PHB    	PHB    	PIX    	 X     	SOC    	SOC    	SOC    	SOC    	SOC    	0-13,28-41
GPU4   	SOC    	SOC    	SOC    	SOC    	 X     	PIX    	PHB    	PHB    	PHB    	14-27,42-55
GPU5   	SOC    	SOC    	SOC    	SOC    	PIX    	 X     	PHB    	PHB    	PHB    	14-27,42-55
GPU6   	SOC    	SOC    	SOC    	SOC    	PHB    	PHB    	 X     	PIX    	PHB    	14-27,42-55
GPU7   	SOC    	SOC    	SOC    	SOC    	PHB    	PHB    	PIX    	 X     	PHB    	14-27,42-55
mlx4_0 	SOC    	SOC    	SOC    	SOC    	PHB    	PHB    	PHB    	PHB    	 X

Legend:

  X   = Self
  SOC  = Connection traversing PCIe as well as the SMP link between CPU sockets(e.g. QPI)
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe switches (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing a single PCIe switch
  NV#  = Connection traversing a bonded set of # NVLinks

slim model code: (Please replace these code to cifar10.py)
def inference(images):
	# reimplement by slim
    slim = tf.contrib.slim
    conv1 = slim.conv2d(images, 64, [5, 5], scope='conv1',
                       weights_regularizer=slim.l2_regularizer(0.0),
                       weights_initializer=tf.truncated_normal_initializer(stddev=0.1),
                       activation_fn=tf.nn.relu)
    pool1 = slim.max_pool2d(conv1, [3, 3], 2, padding='SAME', scope='poll1')
    norm1 = tf.nn.lrn(pool1, 4, bias=1.0, alpha=0.001 / 9.0, beta=0.75, name='norm1')
    conv2 = slim.conv2d(norm1, 64, [5, 5], scope='conv2',
                       weights_regularizer=slim.l2_regularizer(0.0),
                       weights_initializer=tf.truncated_normal_initializer(stddev=5e-2),
                       biases_initializer=tf.constant_initializer(0.1),
                       activation_fn=tf.nn.relu)
    norm2 = tf.nn.lrn(conv2, 4, bias=1.0, alpha=0.001 / 9.0, beta=0.75, name='norm2')
    pool2 = slim.max_pool2d(norm2, [3, 3], 2, padding='SAME', scope='poll2')
    flatten2 = slim.flatten(pool2)
    local3 = slim.fully_connected(flatten2, 384, scope='local3',
                       weights_regularizer=slim.l2_regularizer(0.004),
                       weights_initializer=tf.truncated_normal_initializer(stddev=0.04),
                       biases_initializer=tf.constant_initializer(0.1),
                       activation_fn=tf.nn.relu)
    local4 = slim.fully_connected(local3, 192, scope='local4',
                       weights_regularizer=slim.l2_regularizer(0.004),
                       weights_initializer=tf.truncated_normal_initializer(stddev=0.04),
                       biases_initializer=tf.constant_initializer(0.1),
                       activation_fn=tf.nn.relu)
    softmax_linear = slim.fully_connected(local4, NUM_CLASSES, scope='softmax_linear',
                       weights_regularizer=slim.l2_regularizer(0.0),
                       weights_initializer=tf.truncated_normal_initializer(stddev=1/192.0),
                       biases_initializer=tf.constant_initializer(0.0),
                       activation_fn=None)
    return softmax_linear

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):CentOS Linux release 7.2.1511
TensorFlow installed from (source or binary):source
TensorFlow version (use command below):Tensorflow 1.2.0/ 1.0.1
Python version: 2.7
Bazel version (if compiling from source): 1.4.5
GCC/Compiler version (if compiling from source):4.8.5
CUDA/cuDNN version:CUDA 8.0 and cuDNN 5.0
GPU model and memory:TITAN X with 12GB
Exact command to reproduce:

This might be related to the customized model you defined, as the loss going to nan caused this training error. We can't debug or reproduce this since it seems it's hardware specific. Please debug further yourself and let us know if there is a bug on tensorflow side.

Hi there, can you open this. I'm facing the exact same issue with the thread owner, with GTX 1080ti


text:
Anyway to save the session as an object but not a file?

Hi,
I know that the tensorflow can save all variables into a file as a checkpoint
Is there anyway to save the tensor flow session into an object? like
a=Queue()
a.put( sess.status({%variables%}) )  #or maybe a.put( tf.train.status({%variables%}) )
So when I want to restore a session status. I just do a.get()
Is that possible?
Thank you.

I'm not sure what semantics you're suggesting for sess.status(...) or tf.train.status(...). If you're within the same Python process, you can treat a tf.Session like any other object.
A tf.Session is not serializable, but you can use the tf.MetaGraphDef protocol buffer, along with a checkpoint file, to represent an entire model.

I understand that tf.Session is as any other object. Maybe I did not explain my question clearly.
session_at_step1=tf.session().????
tf.session().run(...some feeds....)
session_at_step2=tf.session().????
In this case, the object will be changed. I mean, is there any way (a function at ????) to let session_at_step1 be different with session_at_step2?

I think the answer is "probably no," but I'm still unclear on what you want the behavior of ???? to be. If you describe a high-level use case, we would consider a feature request.

I've updated the question and made it clear. I mean, instead of having a handle(pointer) to the session, if there is anyway to copy the session.

Ah, ok. Right now there is no way to copy an entire session, and the best way to do what you're asking about is to use the tf.train.Saver to write out a MetaGraphDef proto, and import that into another session using tf.train.import_meta_graph(). This is still quite a new feature, but you can look at the relevant tests to see an example of how it's used. It does give you the ability to restore an entire graph and resume training etc.
If there is any improved tooling you'd like to see around this format, please let us know in a feature request!


text:
Fatal error while compiling Tensorflow with CUDA 9.1

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-104-generic x86_64)
TensorFlow installed from (source or binary):
Source
TensorFlow version (use command below):
unknown 1.4.0 (Source code is cloned from 798fa36d11119e6fdc13b90a14abfe1805e7de90)
Python version:
3.6.3
Bazel version (if compiling from source):
0.8.1
GCC/Compiler version (if compiling from source):
gcc version 5.4.0 20160609
CUDA/cuDNN version:
CUDA 9.1
cuDNN 7.0.5
GPU model and memory:
2 * Tesla V100-PCIE-16GB
Exact command to reproduce:
See description below.

Describe the problem
While trying to compile the latest TensorFlow(cloned from 798fa36d11119e6fdc13b90a14abfe1805e7de90), such error will be raised:
ERROR: /home/ubuntu/tensorflow/tensorflow/contrib/seq2seq/BUILD:64:1: error while parsing .d file: /home/ubuntu/.cache/bazel/_bazel_ubuntu/ad1e09741bb4109fbc70ef8216b59ee2/execroot/org_tensorflow/bazel-out/k8-py3-opt/bin/tensorflow/contrib/seq2seq/_objs/python/ops/_beam_search_ops_gpu/tensorflow/contrib/seq2seq/kernels/beam_search_ops_gpu.cu.pic.d (No such file or directory)
In file included from external/eigen_archive/unsupported/Eigen/CXX11/Tensor:14:0,
                 from ./third_party/eigen3/unsupported/Eigen/CXX11/Tensor:1,
                 from ./tensorflow/contrib/seq2seq/kernels/beam_search_ops.h:19,
                 from tensorflow/contrib/seq2seq/kernels/beam_search_ops_gpu.cu.cc:20:
external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/Core:59:34: fatal error: math_functions.hpp: No such file or directory

It turns out that in CUDA 9.1, math_functions.hpp is located at cuda/include/crt/math_functions.hpp, rather than cuda/include/math_functions.hpp (CUDA 9.0 does), which leads to this error.
ln -s /usr/local/cuda/include/crt/math_functions.hpp /usr/local/cuda/include/math_functions.hpp will fix this problem and complete the compiling process.
Reference
https://stackoverflow.com/a/47807106/2666624
Source code / logs
Traceback is available above.

I'll try to open a PR resolving this problem, after digging some source code.

I can confirm it's a bug of Eigen.
Such error will be raised while compiling Eigen (commit) with cmake .. -DEIGEN_TEST_CUDA=ON && make check -j4:
In file included from /home/ubuntu/eigen-eigen-5c43a259f0b5/Eigen/QR:11:0,
                 from /home/ubuntu/eigen-eigen-5c43a259f0b5/test/main.h:272,
                 from /home/ubuntu/eigen-eigen-5c43a259f0b5/test/cuda_basic.cu:23:
/home/ubuntu/eigen-eigen-5c43a259f0b5/Eigen/Core:59:34: fatal error: math_functions.hpp: No such file or directory

https://stackoverflow.com/questions/43113508/math-functions-hpp-not-found-when-using-cuda-with-eigen --> second answer seems to be related

@AlessioTonioni Yes, and I've already referenced this answer above.

Can you provide an exact command to help replicate this?

Also discussed in https://github.com/tensorflow/tensorflow/issues/15140

@angersson

Same on cuda 9.0

It's a known issue. I can reproduce it under Fedora 25 and CUDA 9.1.

/CC @nluehr can you fix this in Eigen?

Issued PR to Eigen which should resolve this issue.

Make a softlink from /usr/local/cuda-9.1/include/crt/math_functions.hpp to /usr/local/cuda-9.1/include/math_functions.hpp can solve the problem for me.

@JamesTheZ Which has already mentioned above, twice.

So, which version of eigen to use?

authors: AlessioTonioni JamesTheZ angersson hadaev8 icyblade keerthanpg nluehr reedwm sclarkson snnn


text:
Provide tensor/outer product method 'tf.outer'

System information
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux ubuntu 16.04
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 1.4.1 gpu
Python version: 3.5.4
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version: Cuda 8.0/Cudnn 6.0
GPU model and memory: Titan xp
Exact command to reproduce: no method for outer product
Describe the problem
Tensor (outer product) is the fundamental operation on tensors, but there appears to be no method tf.outer in tensorflow analogous to np.outer in numpy to compute the outer product of arbitrary tensors. A google search pulls up these implementation suggestions on stackoverflow: https://stackoverflow.com/questions/33858021/outer-product-in-tensorflow, but these require the dimensions of the tensors to be accessed/known beforehand.
I have the following hack. It would be nice to express this in terms of a pairwise (given associativity) operation tf.outer.
The tensor product for an arbitrary collection of tensors can be computed:
def tensor_product(*e):
    """ Tensor product of elements """
    if len(e) == 1:
        return e
    elif len(e) == 2:
        a, b = e
        r_a = len(a.get_shape().as_list())
        r_b = len(b.get_shape().as_list())
        s_a = tf.concat([tf.shape(a), tf.constant([1] * r_b)], axis=0)
        s_b = tf.concat([tf.constant([1] * r_a), tf.shape(b)], axis=0)
        a_reshaped = tf.reshape(a, s_a)
        b_reshaped = tf.reshape(b, s_b)
        return a_reshaped * b_reshaped
    prod = e[0]
    for tensor in e[1:]:
        prod = tensor_product(prod, tensor)
    return prod

The tensor product allows more elegant expression of rank-2 and greater tensors in a loss function.
For example here is a diagonal and elliptical (weighted terms) quadratic term:
def diagonal_M(batch_size, d):
    """ M_abij = delta_ab delta_ij """
    return tensor_product(tf.diag([1.] * batch_size), tf.diag([1.] * d))

def biased_diagonal_M(batch_size, d):
    """ M_abij = lambda_i delta_ab delta_ij """
    return tensor_product(tf.diag([1.] * batch_size), tf.diag([5.] + [1.] * (d-1)))

ref: https://github.com/4d55397500/quadratic-forms-tensorflow

@rmlarsen : Any thoughts/suggestions on this?

Nagging Assignee @rmlarsen: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 59 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 60 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 75 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 90 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 105 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 120 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 137 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 153 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 168 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 183 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 198 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Bump

Nagging Assignee @rmlarsen: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 29 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @rmlarsen: It has been 47 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Bump, no interest?

This functionality already exists in tf.tensordot, which works like the numpy.tensordot which does state the functionality in the documentation.
Code for outer product is the following:
tf.tensordot(tensor_a, tensor_b, axes=0)

(akin to np.tensordot(tensor_a, tensor_b, axes=0)
Documenting the outer product functionality of tensordot and/or creating an alias could fix this issue.

thanks @swierh, I appreciate it. Looks like a documentation problem. I may in fact suggest implementing a tf.outer operation analogous to np.outer. Closing this issue

It not often used, but used nonetheless.
I personally use it for some perceptron error propagation.
If it will be implemented as separate function maybe there is also some optimization possible in contrast of just using tf.tensordot(tensor_a, tensor_b, axes=0) ?
Also numpy and PyTorch have it, so it takes some time for me to find exactly how to do outer product in Tensorflow.

authors: 4d55397500 Nyanraltotlapun asimshankar redwrasse swierh tensorflowbutler


text:
Restored variables have unknown shape?

When restoring a saved graph, the variables all have <unknown> shape. I have searched a lot, and haven't seen this problem anywhere else. It is also stated in the docs that the size of variables is indeed saved, so it is confusing to me why it is not restored?
Relevant part of the code:
saved = tf.train.import_meta_graph('save.ckpt.meta')
saved.restore(sess, 'save.ckpt')
for x in tf.trainable_variables():
  print x.get_shape()

Output:
<unknown>
<unknown>
<unknown>
<unknown>
... etc

A fix for this has been checked in internally and will be available shortly.

Just wondering if there is any news on this, as I seem to experience this
issue with version 0.10
cheers
        sess, meta_graph_def = session_bundle.LoadSessionBundleFromPath("/tmp/sess/00000001")
        with sess.as_default():
            collection_def = meta_graph_def.collection_def
            signatures_any = collection_def[constants.SIGNATURES_KEY].any_list.value
            signatures = manifest_pb2.Signatures()
            signatures_any[0].Unpack(signatures)
            default_signature = signatures.default_signature
            input_name = default_signature.classification_signature.input.tensor_name
            output_name = default_signature.classification_signature.scores.tensor_name

            print("INP",sess.graph.get_tensor_by_name( input_name).get_shape())

gives: INP

authors: chrisruk rlrs sherrym


text:
Tensorflow mystique issue with MirroredStrategy freeze on particular local GPU combo

https://stackoverflow.com/questions/52425485/tensorflow-mirroredstrategy-freeze-on-particular-gpu-combo
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04 host Linux Ubuntu 16.04 inside Docker container
TensorFlow installed from (source or binary): source and binary
TensorFlow version (use command below): 1.10.1 - 1.11.rc1
Python version: 3.5 - 3.6
Bazel version (if compiling from source): 0.17.1
GCC/Compiler version (if compiling from source): 5
CUDA/cuDNN version: 9.0, 9,2/7.2
GPU model and memory:
(0) Nvidia Titan x 980 12gb
(1) Nvidia Titan x 980 12gb (same)
(2) Nvidia 1080ti 11gb
(3) Nvidia 1080ti 11gb (same)
Exact command to reproduce: example source below

Describe the problem
Four GPUs on the same desktop:
(0) Nvidia Titan x 980 12gb
(1) Nvidia Titan x 980 12gb (same)
(2) Nvidia 1080ti 11gb
(3) Nvidia 1080ti 11gb (same)
MirroredStrategy (say from the obvious example code below) goes ok in combinations of the above (1,2), (1,3) but freezes on (2,3), (1,2,3).
(0 - not used at distribution as causes another sort of CUDA_WAIT_TIMEOUT error due to guessed kernel timeouts for being used to render display)
Have no idea where to go further. Checked with original binary docker images both latest and nightly. Checked with compiled from source TensorFlow r1.10-r1.11 within original nvidia/cuda binary docker image. All the same behaviour.
Source code / logs
import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ['CUDA_VISIBLE_DEVICES']="2,3"

import tensorflow as tf

def model_fn(features, labels, mode):
  layer = tf.layers.Dense(1)
  logits = layer(features)

  if mode == tf.estimator.ModeKeys.PREDICT:
    predictions = {"logits": logits}
    return tf.estimator.EstimatorSpec(mode, predictions=predictions)

  loss = tf.losses.mean_squared_error(
               labels=labels, predictions=tf.reshape(logits, []))

  if mode == tf.estimator.ModeKeys.EVAL:
    return tf.estimator.EstimatorSpec(mode, loss=loss)

  if mode == tf.estimator.ModeKeys.TRAIN:
    train_op = tf.train.GradientDescentOptimizer(0.2).minimize(loss)
    return tf.estimator.EstimatorSpec(mode, loss=loss, train_op=train_op)

def input_fn():
  features = tf.data.Dataset.from_tensors([[1.]]).repeat(100)
  labels = tf.data.Dataset.from_tensors(1.).repeat(100)
  return tf.data.Dataset.zip((features, labels))

distribution = tf.contrib.distribute.MirroredStrategy()
config = tf.estimator.RunConfig(train_distribute=distribution)
classifier = tf.estimator.Estimator(model_fn=model_fn, config=config)

classifier.train(input_fn=input_fn)

classifier.evaluate(input_fn=input_fn)

freeze_output_r1.10.txt
freeze_output_r1.11.txt
normal_completion_output_r1.10.1.txt
normal_completion_output_r1.11.txt
tf_env_r1.10.1.txt
tf_env_r1.11.txt

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Mobile device

There was mentioned a lot ...:, ...:, ...:
ASUS plx chip and bios related issue.
Kernel parameter pci=nommconf has fixed everything.
iommu off - should as well but as a last measure.


text:
tf keras model.fit does not work with strings as labels

Please make sure that this is a bug. As per our [GitHub Policy]System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): tf 1.12, '2.0.0-dev20190503'
Python version: 3.6.7
CUDA/cuDNN version: not applicable
GPU model and memory: not applicable

Describe the current behavior
When using model.fit it throws an error:

tensorflow.python.framework.errors_impl.UnimplementedError: Cast string to float is not supported [Op:Cast] name: Cast/

Describe the expected behavior
This should work the same as when using a lower level implementation with a GradientTape (working fine, no error)
Code to reproduce the issue
    import tensorflow as tf
    import numpy as np

    images = np.random.rand(5, 108, 56, 3)
    y_pred = np.random.rand(5, 4)
    y_true = np.array(['aa', 'bb', 'cc', 'dd', 'ee'])

    dataset = tf.data.Dataset.from_tensor_slices((images, y_true))
    dataset = dataset.batch(5)
    dataset = dataset.repeat()

    model = tf.keras.Sequential([
        tf.keras.layers.Conv2D(16, [3,3], activation='relu'),
        tf.keras.layers.GlobalAveragePooling2D(),
        tf.keras.layers.Dense(4)
    ])

    def triplet_loss(y_true, y_pred):
        all_diffs = tf.expand_dims(y_pred, axis=1) - tf.expand_dims(y_pred, axis=0)
        distances = tf.sqrt(tf.reduce_sum(tf.square(all_diffs), axis=-1) + 1e-12)

        furthest_positive = tf.reduce_max(distances, axis=1)

        closest_negative = tf.map_fn(lambda x: tf.reduce_min(x),
                                     distances)

        diff = furthest_positive - closest_negative
        diff = tf.nn.softplus(diff)

        return diff

    optimizer = tf.optimizers.Adam(learning_rate=0.001)

    model.compile(loss=triplet_loss,
                  optimizer=optimizer)

    model.fit(dataset, steps_per_epoch=5, epochs=10, verbose=1)

When using a lower level implementation it works correctly:
for images, labels in dataset:
    with tf.GradientTape() as tape:
        y_pred = model(images, training=True)
        loss_value = triplet_loss(labels, y_pred)

        grads = tape.gradient(loss_value, model.trainable_variables)
        optimizer.apply_gradients(zip(grads, model.trainable_variables))
        print('iteration done')

Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

@dmus This question is better asked on StackOverflow since it is not a bug or feature request. There is also a larger community that reads questions there.If you think we've misinterpreted a bug, please comment again with a clear explanation, as well as all of the information requested in the issue template. Thanks!

I could reproduce the issue in TF2.0.0-alpha0. Gist is here. Thanks!

This is fixed TF 2.0 nightly '2.0.0-dev20190718'
Thanks!

I am still getting this error with the same code snippet
In [6]: tf.__version__
Out[6]: '2.2.0'

Error:
UnimplementedError:  Cast string to float is not supported
	 [[node Cast (defined at <ipython-input-5-b46e9e31129b>:1) ]] [Op:__inference_train_function_978]

I think am getting the same error while using labels as strings running tf 2.2.0.  Int labels seem to work just fine:
Epoch 1/5
UnimplementedError                        Traceback (most recent call last)
 in ()
1 model.fit(train_ds,
2           epochs=5,
----> 3           steps_per_epoch = 3600//320)
8 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py in _method_wrapper(self, *args, **kwargs)
64   def _method_wrapper(self, *args, **kwargs):
65     if not self._in_multi_worker_mode():  # pylint: disable=protected-access
---> 66       return method(self, *args, **kwargs)
67
68     # Running inside run_distribute_coordinator already.
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
846                 batch_size=batch_size):
847               callbacks.on_train_batch_begin(step)
--> 848               tmp_logs = train_function(iterator)
849               # Catch OutOfRangeError for Datasets of unknown size.
850               # This blocks until the batch has finished executing.
/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py in call(self, *args, **kwds)
578         xla_context.Exit()
579     else:
--> 580       result = self._call(*args, **kwds)
581
582     if tracing_count == self._get_tracing_count():
/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds)
609       # In this case we have created variables on the first call, so we run the
610       # defunned version which is guaranteed to never create variables.
--> 611       return self._stateless_fn(*args, **kwds)  # pylint: disable=not-callable
612     elif self._stateful_fn is not None:
613       # Release the lock early so that multiple threads can perform the call
/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in call(self, *args, **kwargs)
2418     with self._lock:
2419       graph_function, args, kwargs = self._maybe_define_function(args, kwargs)
-> 2420     return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
2421
2422   @property
/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _filtered_call(self, args, kwargs)
1663          if isinstance(t, (ops.Tensor,
1664                            resource_variable_ops.BaseResourceVariable))),
-> 1665         self.captured_inputs)
1666
1667   def _call_flat(self, args, captured_inputs, cancellation_manager=None):
/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in _call_flat(self, args, captured_inputs, cancellation_manager)
1744       # No tape is watching; skip to running the function.
1745       return self._build_call_outputs(self._inference_function.call(
-> 1746           ctx, args, cancellation_manager=cancellation_manager))
1747     forward_backward = self._select_forward_and_backward_functions(
1748         args,
/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py in call(self, ctx, args, cancellation_manager)
596               inputs=args,
597               attrs=attrs,
--> 598               ctx=ctx)
599         else:
600           outputs = execute.execute_with_cancellation(
/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
58     ctx.ensure_initialized()
59     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
---> 60                                         inputs, attrs, num_outputs)
61   except core._NotOkStatusException as e:
62     if name is not None:
UnimplementedError:  Cast string to float is not supported
[[node Cast (defined at :3) ]] [Op:__inference_train_function_1519]
Function call stack:
train_function

authors: dmus jvishnuvardhan muddham muditbac pastrop ymodak


text:
I have installed tensorflow-gpu 2.0.0 alpha,how can I upgrade it to tensorflow-gpu==2.0.0-beta1,but didn't influence my tensorlayer.

Please make sure that this is a build/installation issue. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:build_template
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary):
TensorFlow version:
Python version:
Installed using virtualenv? pip? conda?:
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:

Describe the problem
Provide the exact sequence of commands / steps that you executed before running into the problem
Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

@YonghuiXu, Try upgrading the tensorflow using  pip install tensorflow-gpu==2.0.0.beta1. Let us know if that helps. Thanks!

Yes, I have done it like what you said. It works well.
---Original---
From: "gadagashwini"notifications@github.com
Date: Fri, Aug 16, 2019 16:52 PM
To: "tensorflow/tensorflow"tensorflow@noreply.github.com;
Cc: "Mention"mention@noreply.github.com;"YonghuiXu"2259949930@qq.com;
Subject: Re: [tensorflow/tensorflow] I have installed tensorflow-gpu 2.0.0 alpha,how can I upgrade it to tensorflow-gpu==2.0.0-beta1,but didn't influence my tensorlayer. (#31615)
@YonghuiXu, Try upgrading the tensorflow using  pip install tensorflow-gpu==2.0.0.beta1. Let us know if that helps. Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@YonghuiXu Glad to know that it worked.
Closing the issue since its resolved. Thanks!

authors: YonghuiXu gadagashwini


text:
BatchNormalization virtual_batch_size does not work with None in input shape

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below):
v2.0.0-beta1-5101-gc75bb66 2.0.0-rc0
Python version: 3.6.8
Bazel version (if compiling from source): N/A
GCC/Compiler version (if compiling from source): N/A
CUDA/cuDNN version: CUDA 10.0, cuDNN 7.6.2.24-1
GPU model and memory: Nvidia RTX 2070 8 GB

Describe the current behavior
A constructor of a tf.keras Model that uses tf.keras.layers.BatchNormalization with virtual_batch_size set and unspecified input shape dimensions throws an exception.
Describe the expected behavior
Such a model should be usable.
Code to reproduce the issue
import tensorflow as tf

inp = tf.keras.layers.Input(shape=(None, None, 3))
net = tf.keras.layers.BatchNormalization(virtual_batch_size=8)(inp)

model = tf.keras.Model(inputs=inp, outputs=net)

Other info / logs
Traceback of the exception:
Traceback (most recent call last):
  File "/home/ikrets/tf2/lib/python3.6/site-packages/tensorflow_core/python/framework/tensor_util.py", line 541, in make_tensor_proto
    str_values = [compat.as_bytes(x) for x in proto_values]
  File "/home/ikrets/tf2/lib/python3.6/site-packages/tensorflow_core/python/framework/tensor_util.py", line 541, in <listcomp>
    str_values = [compat.as_bytes(x) for x in proto_values]
  File "/home/ikrets/tf2/lib/python3.6/site-packages/tensorflow_core/python/util/compat.py", line 71, in as_bytes
    (bytes_or_text,))
TypeError: Expected binary or unicode string, got 8

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ikrets/tf2/lib/python3.6/site-packages/tensorflow_core/python/framework/test_virtual_batch.py", line 6, in <module>
    net = tf.keras.layers.BatchNormalization(virtual_batch_size=8)(inp)
  File "/home/ikrets/tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/base_layer.py", line 802, in __call__
    outputs = call_fn(cast_inputs, *args, **kwargs)
  File "/home/ikrets/tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/layers/normalization.py", line 652, in call
    inputs = array_ops.reshape(inputs, expanded_shape)
  File "/home/ikrets/tf2/lib/python3.6/site-packages/tensorflow_core/python/ops/array_ops.py", line 131, in reshape
    result = gen_array_ops.reshape(tensor, shape, name)
  File "/home/ikrets/tf2/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_array_ops.py", line 8117, in reshape
    "Reshape", tensor=tensor, shape=shape, name=name)
  File "/home/ikrets/tf2/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 530, in _apply_op_helper
    raise err
  File "/home/ikrets/tf2/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 527, in _apply_op_helper
    preferred_dtype=default_dtype)
  File "/home/ikrets/tf2/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1296, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/home/ikrets/tf2/lib/python3.6/site-packages/tensorflow_core/python/framework/constant_op.py", line 286, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/home/ikrets/tf2/lib/python3.6/site-packages/tensorflow_core/python/framework/constant_op.py", line 227, in constant
    allow_broadcast=True)
  File "/home/ikrets/tf2/lib/python3.6/site-packages/tensorflow_core/python/framework/constant_op.py", line 265, in _constant_impl
    allow_broadcast=allow_broadcast))
  File "/home/ikrets/tf2/lib/python3.6/site-packages/tensorflow_core/python/framework/tensor_util.py", line 545, in make_tensor_proto
    "supported type." % (type(values), values))
TypeError: Failed to convert object of type <class 'list'> to Tensor. Contents: [8, -1, None, None, 3]. Consider casting elements to a supported type.

I am replicate the issue with TF 2.0.0.rc0. Please take a look at gist here. Thanks!

Added a PR #39131 for the fix.

authors: gadagashwini ikrets yongtang


text:
tf.losses.cosine_similarity is a negative quantity between -1 and 0

In tensorflow website, it describes tf.losses.cosine_simialrity as follows:
Note that it is a negative quantity between -1 and 0, where 0 indicates orthogonality and values closer to -1 indicate greater similarity.
In fact, the quantity is from 1. to -1., it just takes a negative from normal cosine_similarity.
The page is at
https://tensorflow.google.cn/api_docs/python/tf/keras/losses/cosine_similarity

I can find the code in github.
`
y_true = nn.l2_normalize(y_true, axis=axis)
y_pred = nn.l2_normalize(y_pred, axis=axis)
return -math_ops.reduce_sum(y_true * y_pred, axis=axis)
`

@songs18, Could you post complete standalone code snippet for us to analyze the issue better?

‘’‘Python
def cosine_similarity(y_true, y_pred, axis=-1):
"""Computes the cosine similarity between labels and predictions.
Note that it is a negative quantity between -1 and 0, where 0 indicates
orthogonality and values closer to -1 indicate greater similarity. This makes
it usable as a loss function in a setting where you try to maximize the
proximity between predictions and targets.
loss = -sum(y_true * y_pred)
Args:
y_true: Tensor of true targets.
y_pred: Tensor of predicted targets.
axis: Axis along which to determine similarity.
Returns:
Cosine similarity tensor.
"""
y_true = nn.l2_normalize(y_true, axis=axis)
y_pred = nn.l2_normalize(y_pred, axis=axis)
return -math_ops.reduce_sum(y_true * y_pred, axis=axis)
'''

@songs18 Thank you for bringing this up. We assume that prediction is a value between 0 and 1 and labels are binary ie. 0 or 1. With these assumptions cosine similarity will be in positive space - [0, 1] and the loss function will be in [-1, 0].

Closing this as the behavior is as expected. Please let us know if you have any questions. Thank you!

poor thing,. we cant use it for generic cosine similarity loss functions...

authors: gadagashwini gitgani pavithrasv songs18


text:
[RNN] [TFLiteConverter.] Input tensors containing unknown dimensions fails when coupled with LSTM

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Linux-4.19.104+-x86_64-with-Ubuntu-18.04-bionic (Google Colab's default environment)
TensorFlow installed from (source or binary):
pip install tf-nightly
TensorFlow version (or github SHA if from source):
2.2.0-dev20200421

Command used to run the converter or code if you’re using the Python API
If possible, please share a link to Colab/Jupyter/any notebook.
https://colab.research.google.com/drive/1-hjOM3qW5gY1PAqEZhFGHj7FFY5BaZsm
from tensorflow.lite.python import lite
from tensorflow.python import keras
import numpy as np

input_a = keras.layers.Input(shape=(3,3,), name='input_a')
interm_b = tf.keras.layers.LSTM(4, name='interm_1')(input_a)
output_c = keras.layers.Dense(1, name='dense_1')(interm_b)

model = tf.keras.models.Model(inputs=[input_a], outputs=[output_c])
model.compile(optimizer='sgd', loss='mean_squared_error')
model.summary()

batch_size = 10
sample_input = np.ones((batch_size,3,3),dtype=np.float32)

expected_value = model.predict(sample_input)

converter = lite.TFLiteConverterV2.from_keras_model(model = model)
converter.experimental_new_converter = True
with open("model.tflite", "wb") as f:
    f.write(converter.convert())

interpreter = lite.Interpreter(model_path="model.tflite")
print(interpreter.get_input_details())
interpreter.resize_tensor_input(0,[batch_size, 3,3])
interpreter.allocate_tensors()
interpreter.set_tensor(0, sample_input)
interpreter.invoke()
interpreter.get_tensor(interpreter.get_output_details()[0]["index"])

The output from the converter invocation
Model: "model_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
input_a (InputLayer)         [(None, 3, 3)]            0
_________________________________________________________________
interm_1 (LSTM)              (None, 4)                 128
_________________________________________________________________
dense_1 (Dense)              (None, 1)                 5
=================================================================
Total params: 133
Trainable params: 133
Non-trainable params: 0
_________________________________________________________________
[{'name': 'input_a', 'index': 0, 'shape': array([1, 3, 3], dtype=int32), 'shape_signature': array([-1,  3,  3], dtype=int32), 'dtype': <class 'numpy.float32'>, 'quantization': (0.0, 0), 'quantization_parameters': {'scales': array([], dtype=float32), 'zero_points': array([], dtype=int32), 'quantized_dimension': 0}, 'sparsity_parameters': {}}]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-492497848c68> in <module>()
     27 interpreter.allocate_tensors()
     28 interpreter.set_tensor(0, sample_input)
---> 29 interpreter.invoke()
     30 interpreter.get_tensor(interpreter.get_output_details()[0]["index"])

/usr/local/lib/python3.6/dist-packages/tensorflow/lite/python/interpreter.py in invoke(self)
    512     """
    513     self._ensure_safe()
--> 514     self._interpreter.Invoke()
    515
    516   def reset_all_variables(self):

RuntimeError: tensorflow/lite/kernels/concatenation.cc:74 t->dims->data[d] != t0->dims->data[d] (10 != 1)Node number 50 (CONCATENATION) failed to prepare.
Node number 10 (WHILE) failed to invoke.

Failure details
The conversion is successful, but the generated model cannot be resized to variable batch size.
Input tensors containing unknown dimensions fails when coupled with LSTM
Same script would work just fine if one removes creation of interm_b and pass input_a as the input to generate the output_c.

i am able to replicate this, please find the gist here as well as on nightly

Currently we need all dimensions are known.
Thanks.

Okay, thanks @renjie-liu.
Do I close the issue, or is it better to remain open as a feature request?

our fused ops will need the dimensions to be known because the states are maintained inside the kernel. so it is wai. :(

authors: Saduf2019 madarez renjie-liu


text:
Golang Tensorflow v2.2.0 fails install

System information

OS Platform and Distribution (Macos catalina 10.15.4):
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
TensorFlow installed from (source or binary):
TensorFlow version:
Python version:
Installed using virtualenv? pip? conda?:
Bazel version (if compiling from source):
GCC/Compiler version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:

Describe the problem
Provide the exact sequence of commands / steps that you executed before running into the problem
Following https://github.com/tensorflow/tensorflow/blob/master/tensorflow/go/README.mdhttps://github.com/tensorflow/tensorflow/blob/master/tensorflow/go/README.md
go generate github.com/tensorflow/tensorflow/tensorflow/go/op
fails with failed to process "api_def_VarHandleOp.pbtxt": Attribute allowed_devices not defined in base api for VarHandleOp exit status 1 go/src/github.com/tensorflow/tensorflow/tensorflow/go/op/generate.go:18: running "go": exit status 1
go test github.com/tensorflow/tensorflow/tensorflow/go passes
building or running go code using tensorflow then proceeds to fail with the following output
go: finding github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto latest go: finding github.com/tensorflow/tensorflow/tensorflow/go/core latest go: finding github.com/tensorflow/tensorflow/tensorflow/go latest go: finding github.com/tensorflow/tensorflow/tensorflow latest build command-line-arguments: cannot load github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto: cannot find module providing package github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto
the code run is the example that prints tf library version from https://www.tensorflow.org/install/lang_go
Any other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

FROM golang:1.13 AS builder WORKDIR /app COPY libtf22.tar.gz /app/libtf22.tar.gz RUN tar -C /usr/local -xzf libtf22.tar.gz RUN ldconfig
Running in Docker with the above Dockerfile, where libtf22.tar.gz is a libtensorflow build from from the v2.2.0 tag. (Which seems to work, if i use that tar.gz file and install tf 2.1 via go get, and run the tf_hello.go example, i get hello from version 2.2).
doing go get -d github.com/tensorflow/tensorflow/tensorflow/go
Produces
package github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto: cannot find package "github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto" in any of: 	/usr/local/go/src/github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto (from $GOROOT) 	/go/src/github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto (from $GOPATH)
the files are downloaded to /${GOPATH}/src/github.com/tensorflow though
Skipping Step 2 build the TensorFlow C Library
cd ${GOPATH}/src/github.com/tensorflow/tensorflow ./configure bazel build -c opt //tensorflow:libtensorflow.so
As it has already been build from, following: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/lib_package/README.md
go generate github.com/tensorflow/tensorflow/tensorflow/go/op in linux container still fails with:
failed to process "api_def_VarHandleOp.pbtxt": Attribute allowed_devices not defined in base api for VarHandleOp exit status 1 /go/src/github.com/tensorflow/tensorflow/tensorflow/go/op/generate.go:18: running "go": exit status 1

Okay managed to get go generate to work, by doing
cd ${GOPATH}/github.com/src/tensorflow/tensorflow
git checkout tags/v2.2.0
go generate github.com/tensorflow/tensorflow/tensorflow/go/op
This has now placed a lot of generated protobufs in tensorflow/go/vendor/github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto
Now doing
go mod tidy in my src directory seems to understand that there the core_protos for go exists.
Outputting the following:
go: finding github.com/tensorflow/tensorflow/tensorflow/go/core/framework/tensor_shape_go_proto latest go: finding github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto latest go: finding github.com/tensorflow/tensorflow/tensorflow/go/core/framework/types_go_proto latest go: finding github.com/tensorflow/tensorflow/tensorflow/go/core/framework latest go: finding github.com/tensorflow/tensorflow/tensorflow/go/core latest go: finding github.com/tensorflow/tensorflow/tensorflow/go latest go: finding github.com/tensorflow/tensorflow/tensorflow latest
But doing
go run main.go
or
go build main.go
Seems to try and download the module and fails with the following output
go: finding github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto latest go: finding github.com/tensorflow/tensorflow/tensorflow/go/core latest go: finding github.com/tensorflow/tensorflow/tensorflow/go latest go: finding github.com/tensorflow/tensorflow/tensorflow latest build command-line-arguments: cannot load github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto: cannot find module providing package github.com/tensorflow/tensorflow/tensorflow/go/core/core_protos_go_proto

@JacobSMoller Have you found a solution for this? I have the exact same issue.

@ConverJens Sorry didn't get a notification for you writing me.
Yes not a nice solution. But i did make it work...
Here is a public bucket with working libtensorflow 2.2 for unix and osx (If you haven't got them already)
https://console.cloud.google.com/storage/browser/vml-tf-lib?forceOnBucketsSortingFiltering=false&project=dev-vml-cm
The next step is the not so nice step...
`FROM golang:1.13 AS builder
ARG GO_MOD_TF_PATH=/go/pkg/mod/github.com/tensorflow/tensorflow@v2.2.0+incompatible/tensorflow/go
ARG GO_SRC_TF_PATH=/go/src/github.com/tensorflow/tensorflow/tensorflow/go/vendor/github.com/tensorflow/tensorflow/tensorflow/go/core/
WORKDIR /build-lib
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates autoconf automake libtool curl make g++ unzip curl
Install gcloud to get gsutil
RUN wget https://storage.googleapis.com/pub/gsutil.tar.gz  && tar xfz gsutil.tar.gz -C $HOME && rm gsutil.tar.gz
ENV PATH ${PATH}:/root/gsutil
Install libprotoc
RUN wget https://github.com/protocolbuffers/protobuf/releases/download/v3.11.4/protoc-3.11.4-linux-x86_64.zip
RUN unzip -o protoc-3.11.4-linux-x86_64.zip -d /usr/local bin/protoc
RUN unzip -o protoc-3.11.4-linux-x86_64.zip -d /usr/local 'include/*'
RUN rm protoc-3.11.4-linux-x86_64.zip
Install libtensorflow
RUN gsutil cp gs://vml-tf-lib/libtensorflow-unix.tar.gz .
RUN tar -C /usr/local -xzf libtensorflow-unix.tar.gz
RUN ldconfig
RUN rm libtensorflow-unix.tar.gz
WORKDIR /build-lib
Tensorflows repo is a little broken, so to continue ignore this download error.
RUN go get -d github.com/tensorflow/tensorflow/tensorflow/go/op; exit 0
WORKDIR /go/src/github.com/tensorflow/tensorflow/tensorflow
Checkout the 2.2.0 release in the downloaded source
RUN git checkout tags/v2.2.0
WORKDIR /build-lib
run the go generate script located in the downloaded source code, to generate go protos
RUN go generate github.com/tensorflow/tensorflow/tensorflow/go/op
WORKDIR /app
COPY go.sum /app/go.sum
COPY go.mod /app/go.mod
RUN go mod download
make the core folder in the go mod tensorflow path
RUN mkdir -p ${GO_MOD_TF_PATH}/core
RUN cp -r ${GO_SRC_TF_PATH}/core_protos_go_proto ${GO_MOD_TF_PATH}/core/core_protos_go_proto
RUN cp -r ${GO_SRC_TF_PATH}/framework ${GO_MOD_TF_PATH}/core/framework
Not needed for serving.
RUN rm ${GO_MOD_TF_PATH}/core/core_protos_go_proto/autotuning.pb.go
RUN rm ${GO_MOD_TF_PATH}/core/core_protos_go_proto/conv_autotuning.pb.go
COPY tftimer/tf.go /app/tf.go
RUN go build tf.go
Make runner as pr.
FROM debian:bullseye AS runner
WORKDIR /app
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates
COPY --from=builder /app/tf /app/tf
COPY --from=builder /app/doc /app/doc
COPY --from=builder /usr/local /usr/local
RUN ldconfig`
So to make it work i do the
go get -d as suggested in go documentation
Then do go generate
then I use go mod download to get the tensorflow package.
And finally copy the generated protos over from the downloaded source code.
Not nice in anyway. But haven't been able to find a better way.
And then copy

@JacobSMoller Thanks for the detailed answer! I will try it the week after next and let you know how it goes!

@JacobSMoller Thanks for the hint. I tried it, but failed with
Step 23/38 : COPY go.sum /app/go.sum
COPY failed: stat /var/lib/docker/tmp/docker-builder720470318/go.sum: no such file or directory
where do you get the go.sum and go.mod files from?

They are generated when using go modules.
https://blog.golang.org/using-go-modules

Fixed as part of:
https://github.com/tensorflow/tensorflow/commit/e5e495db7bee77cd0fd5dda3b06bd743cbcf1ef8
Note that it requires from master:
bazel build -c opt //tensorflow/tools/lib_package:libtensorflow
tar -xzvf the package in /usr/local
go generate github.com/tensorflow/tensorflow/tensorflow/go/op

@jhseu This as far as I can tell, still doesn't enable users to do go get github.com/tensorflow/tensorflow/tensorflow/go
and simply run things.
When using go modules I still need to generate the protobufs in $GOPATH/src/github.com/tensorflow/tensorflow And then move the generated protobufs in to $GOPATH/pkg/mod/github.com/tensorflow/...
Also doing the go generate command from master now returns a new error:
2020/07/27 13:07:50 failed to process "api_def_BatchFunction.pbtxt": Attribute enable_large_batch_splitting not defined in base api for BatchFunction exit status 1 go/src/github.com/tensorflow/tensorflow/tensorflow/go/op/generate.go:18: running "go": exit status

@JacobSMoller Yeah, go get doesn't work for now. I made a pull request to fix it, but there were some objections to supporting both the bazel and generated protobufs to make it work.
For that error: it occurs when there's a version mismatch between libtensorflow.so and the TF ops in the source code. See my command above for building libtensorflow.so.

@jhseu alright thanks :)
So no road map for having a go gettable Tensorflow isntallation :)?

I've spend a good day and a half truign to get this installed on my Ubuntu machine. Tried most things mentioned above, and have built everything from scratch. After installing Protobuf, it's not able to generate the protobufs

@silverark How are you installing protoc? It looks like you're missing the default protos from your installation. Typically they're in /usr/local/include after installation, like so:
jhseu0:~$ locate any.proto
/usr/include/google/protobuf/any.proto

Thanks @jhseu. Turns out I had a different version of proto hidden in my ~/local
Finally managed to get it working but only without module. If I use go mod in my project it downloads the version into pkg which is still broken. I'm currently at the stage @JacobSMoller is describing.

authors: ConverJens JacobSMoller NiBurhe jhseu silverark


text:
Optimizer + tf.reduce_prod() + GPU = Cannot assign a device to node 'gradients/Prod_grad/ListDiff'

Environment info
Operating System: Ubuntu 14.4.5 LTS
Installed version of CUDA and cuDNN:
CUDA 7.5.18 and cuDNN 4.0.7
-rw-r--r-- 1 root root 322936 Aug 15  2015 libcudadevrt.a
lrwxrwxrwx 1 root root     16 Jul 12 08:24 libcudart.so -> libcudart.so.7.5
lrwxrwxrwx 1 root root     19 Jul 12 08:24 libcudart.so.7.5 -> libcudart.so.7.5.18
-rwxr-xr-x 1 root root 383336 Aug 15  2015 libcudart.so.7.5.18
-rw-r--r-- 1 root root 720192 Aug 15  2015 libcudart_static.a

-rw-r--r-- 1 sauterme tumuser        0 Aug 22 18:29 libcudart.so.7.5
-rw-r--r-- 1 sauterme tumuser        0 Aug 22 18:29 libcudart.so.7.5.18
lrwxrwxrwx 1 sauterme tumuser       13 Feb  9  2016 libcudnn.so -> libcudnn.so.4
lrwxrwxrwx 1 sauterme tumuser       17 Feb  9  2016 libcudnn.so.4 -> libcudnn.so.4.0.7
-rwxr-xr-x 1 sauterme tumuser 61453024 Feb  8  2016 libcudnn.so.4.0.7
-rw-r--r-- 1 sauterme tumuser 62025862 Feb  8  2016 libcudnn_static.a

If installed from binary pip package, provide:

Which pip package you installed:
pip package version 0.10 for Python 2.7 with GPU support:
_https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.10.0rc0-cp27-none-linux_x86_64.whl_
The output from python -c "import tensorflow; print(tensorflow.__version__)".
0.10.0rc0

Steps to reproduce

Use a system with GPU

Use this code snippet
import tensorflow as tf

g = tf.Graph()
with g.as_default():
    with tf.Session(config=tf.ConfigProto(log_device_placement=True)) as sess:

        with tf.device('/gpu:0'):

            a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
            b = tf.Variable([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [7.0, 8.0, 9.0]], name='b')

            c = tf.matmul(a, b)
            res = tf.reduce_prod(c)

            #with tf.device('/cpu:0'): # <<< adding this solves the issue!
            train_op = tf.train.AdagradOptimizer(1e-3).minimize(res)

        sess.run(tf.initialize_all_variables())

        print sess.run(res)
        print sess.run(train_op)

You will get an InvalidArgumentError:
_InvalidArgumentError: Cannot assign a device to node 'gradients/Prod_grad/ListDiff': Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available.
[[Node: gradients/Prod_grad/ListDiff = ListDiff[T=DT_INT32, _device="/device:GPU:0"](gradients/Prod_grad/range_1, Const)]]
Caused by op u'gradients/Prod_grad/ListDiff', defined at:
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in run_code
exec code in run_globals

Cause of the issue: tf.reduce_prod() when running the train_op. So, the problem takes place while back-propagation through this op.

What have you tried?

Wrapping the tf.reduce_prod() with tf.device('/cpu:0'):
Result: still getting this error!

Exchangeing tf.reduce_prod() with other reduce_*() ops:
Result: no error!

Wrapping the Optimizer with  tf.device('/cpu:0'):
Result: Solves the error!

@ibab @girving Possibly related to #3351 ?

@vrv We were discussing a similar thing recently.  Is the right solution here just to tell people to leave the optimizer out of a location?  That seems heavy handed, but I don't think we can guarantee that gradients use no CPU-only ops.

Well, the answer that I personally don't like is to tell people to use soft_placement in the config options.  A better solution would be to implement the GPU versions of these ops when possible, and if not, to take the ops that are not implemented and wrap them in a with device(cpu:0) block.

(the latter approach is an implementation detail of the gradient that we can change).

@bsautermeister Does this commit fix your problem?  https://github.com/girving/tensorflow/commit/957fe73546f25c7f88232ed560ed285c1c97067d

@girving Indeed, it does! Thank you very much!
And (obviously) it's much faster than wrapping the Optimizer with tf.device('/cpu:0').

We shouldn't close it  until I actually submit the fix.

My model's performance degraded seriously because of this. tf version is 1.9, hoping the fix could be considered officially

authors: bsautermeister carusyte girving prb12 vrv


text:
File system scheme 's3' not implemented

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: No
TensorFlow installed from (source or binary): Binary
TensorFlow version (use command below): v2.2.0-rc4-8-g2b96f3662b 2.2.0
Python version: 3.6.5
Bazel version (if compiling from source): NA
GCC/Compiler version (if compiling from source): NA
CUDA/cuDNN version: NA
GPU model and memory: NA

Describe the current behavior
tensorflow.python.framework.errors_impl.UnimplementedError: File system scheme 's3' not implemented
Describe the expected behavior
Should be able to connect the s3 filesystem.
Standalone code to reproduce the issue
from tensorflow.python.lib.io import file_io
print file_io.stat('s3://bucketname/path/')
Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.

Can you try with the final 2.2 release please?

Oh. You are using Windows. Windows does not yet support S3 filesystems, but there is a GSoC project to enable that.

#39124 will be included in the upcoming 2.3 release.
Modular filesystems (tensorflow/community#101) will probably come in 2.4

Hi, I'm facing the exact same error with s3a://
My tf version is: 2.1.0
Please help

Also, I can read s3a:///wine-quality.csv using Spark.

Also, I can read s3a://path/to/wine-quality.csv using Spark.

@vaibhavsingh007 we only implement support for the s3:// URI scheme. Didn't know there is a s3a:// too. Do you know what are the differences?

Yes, s3a is the latest protocol and is more performant than its predecessors s3/s3n: https://stackoverflow.com/questions/33356041/technically-what-is-the-difference-between-s3n-s3a-and-s3
Would you know if or when we may expect TF to S3 functionality become available?

There is funcionality for basic s3:// scheme on Linux. Then there is work for tensorflow/community#101 to convert all these filesystem to a plugin based form so they can be used on all platforms, on demand.
Regarding s3a and s3n, can you file a new issue, feature request type, and assign it to me/mention me? just to keep this issue only for the s3 not implemented error.

Done. Seems the bot auto assigned and I can't change the assignee.

Thank you. Yes, the bot assigns oncall for triage.

So @mihaimaruseac I wanted to draw our attention back to the original issue of not being able to write to S3 from TF using s3://..
I get the error NotFoundError: Object s3:///wine-quality.csv does not exist (from the post above).

Hmm, what would be the output of tf.io.gfile.exists on that path?

o/p of above is 'False'.
Also, I was able to get some backend logs for the stat call, in case it helps:

And I'm setting these auth vars:
os.environ["AWS_ACCESS_KEY_ID"]="***"
os.environ["AWS_SECRET_ACCESS_KEY"]="***"
os.environ["AWS_REGION"]="us-east-1"

@mihaimaruseac any luck?

Sorry, didn't have time to look more into this. Can you manually issue the curl requests?

Well like I said can access the said files using s3a:// via Spark. But let me try curling as well.

@mihaimaruseac I couldn't set up curl, but successfully read file using aws api via boto3. Here is a snapshot:

Would you like to share a sample code to test read from private (secured) s3 repo using TF? I hope I am using the correct auth configuration (shared before).

And I'm setting these auth vars:
os.environ["AWS_ACCESS_KEY_ID"]="***"
os.environ["AWS_SECRET_ACCESS_KEY"]="***"
os.environ["AWS_REGION"]="us-east-1"

Current implementation uses curl to handle the requests to the S3 API. The modular filesystem approach that is currently in the works would use awscpp.
But until then, I think we need to differentiate between TF error and curl error.

Hi, I am not sure I understand what you mean by 'differentiate between TF error and curl error'. Is there something else you want me to try since we still cannot read/write to S3 using TF.

Can you intercept the requests TF makes during tf.io.gfile.file_exists on the path and then try playing them manually via curl? Looking at the provided logs it looks like there is a failure on the server / invalid credentials/cookies/arguments get sent.

Would you be able to direct me how I can do that?

Hmm, I was wrong, it seems we are already using AWS library not issuing curl requests manually
https://cs.opensource.google/tensorflow/tensorflow/+/master:tensorflow/core/platform/s3/s3_file_system.cc;l=707-766;drc=791f4f2b773208b21ece71e1d37204a91720346d
You could intercept the traffic using wireshark or if you can compile from source, I'd recommend adding some debug prints in the function I linked above.
Unfortunately I currently don't have access to an S3 bucket and won't be able to look much into this issue until mid of August. Apologies for the delay

Question: can you compile TF from source? Or parts of TF.
We have an experimental plugin for S3 support that might be easier to use than the existing non-modular approach. But at the moment it requires manual compile.

Hi @mihaimaruseac sorry for the delay.
I'm not sure we can do that but I may try. Can you share more details about the new API?

Hi.
We are planning to modularize filesystems in TF and only offer the local filesystem by default in the pip package, requiring every other filesystem to be loaded at runtime from a shared library. RFC is at https://github.com/tensorflow/community/blob/master/rfcs/20190506-filesystem-plugin-modular-tensorflow.md and currently the implementation is at https://github.com/tensorflow/tensorflow/tree/master/tensorflow/c/experimental/filesystem
I would suggest compiling the S3 plugin from the experimental filesystem and then using tf.load_library to load the shared object.

can you please describe steps to do for implementing s3 filesystem on winows platform or it still require loading shared library at runtime?

Compiling the plugin in https://cs.opensource.google/tensorflow/tensorflow/+/master:tensorflow/c/experimental/filesystem/plugins/s3/;drc=75a455a9416630e4a33d23b4dc27bbbdb9c75f41 gives you a shared object that you then load at runtime.
This should work on all operating systems.

The s3 filesystem is available on windows by install pip install tensorflow-io==0.17. Could you give it a try ? Note that in order to avoid conflicts with current s3 in TensorFlow, it is accessible by s3e uri, i.e s3e://bucket/key.
Example : https://github.com/tensorflow/io/blob/master/tests/test_s3_eager.py

@kumaradi08 @vaibhavsingh007  As per the mihai latest comment, it should work on all Operating systems.  Could you please confirm and if yes, Please go ahead and close the issue if you don't have any further queries.Thanks!

@saikumarchalla @kumaradi08 @vaibhavsingh007 Actually there is one more issue https://github.com/tensorflow/tensorflow/issues/49515 which will be fixed by #49520.

This is there in 2.6 version of tensorflow
"tensorflow/tensorflow:2.6.0"
File "/usr/local/lib/python3.6/dist-packages/keras/callbacks.py", line 2219, in set_model
    self._write_keras_model_summary()
  File "/usr/local/lib/python3.6/dist-packages/keras/callbacks.py", line 2258, in _write_keras_model_summary
    with self._train_writer.as_default():
  File "/usr/local/lib/python3.6/dist-packages/keras/callbacks.py", line 2228, in _train_writer
    self._train_dir)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/summary_ops_v2.py", line 554, in create_file_writer_v2
    create_fn=create_fn, init_op_fn=init_op_fn)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/summary_ops_v2.py", line 313, in __init__
    self._init_op = init_op_fn(self._resource)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_summary_ops.py", line 146, in create_summary_file_writer
    _ops.raise_from_not_ok_status(e, name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6941, in raise_from_not_ok_status
    six.raise_from(core._status_to_exception(e.code, message), None)
  File "<string>", line 3, in raise_from
tensorflow.python.framework.errors_impl.UnimplementedError: File system scheme 's3' not implemented (file: 's3://tfboard/logs/20210825-092239/train') [Op:CreateSummaryFileWriter]

I'm also getting the above error with Tensorflow 2.6
writer = tf.io.TFRecordWriter(tf_record_path)

Exception has occurred: UnimplementedError
File system scheme 's3' not implemented (file: 's3://ffda-poi/training_data/mauritania_by_areas/rural/od/large_tfrecords/train_mauritania_rural_od_large_training_data_001.tfrecords')
  File "/tfrecords/write_tfrecords_od.py", line 219, in write_tfrecord_file
    writer = tf.io.TFRecordWriter(tf_record_path)
  File "/tfrecords/write_tfrecords_od.py", line 280, in write_tfrecord_od
    write_tfrecord_file(train_df, tile_path, record_file_train, super_tile_size)
  File "/tfrecords/write_tfrecords_od.py", line 318, in main
    write_tfrecord_od(csv_file, output_dir, tile_path, super_tile_size, geojson_with_neg_path)
  File "/tfrecords/write_tfrecords_od.py", line 322, in <module>
    main()

I can confirm this error does not happen if I install tensorflow 2.3

The s3 filesystem is available on windows by install pip install tensorflow-io==0.17. Could you give it a try ? Note that in order to avoid conflicts with current s3 in TensorFlow, it is accessible by s3e uri, i.e s3e://bucket/key.
Example : https://github.com/tensorflow/io/blob/master/tests/test_s3_eager.py

This should be the solution here. For cloud filesystems you have to also install the SIG IO wheel.