text:
KeyOverlaidModifier except if other keys are being pressed

Is there a way to prevent ^F2  from being sent when there are other keys being currently pressed? (starting being pressed before option ⌥ )
<item>
      <name>Option_L to Option_L</name>
      <appendix>(+ When you type Option_L only, send Control+F2 (Move focus to the menu bar))</appendix>
      <identifier>remap.optionL2optionL_controlF2</identifier>
      <autogen>__KeyOverlaidModifier__ KeyCode::OPTION_L, KeyCode::OPTION_L, KeyCode::F2, ModifierFlag::CONTROL_L</autogen>
</item>

How about KeyDownUpToKey?
    <autogen>
      __KeyDownUpToKey__
      KeyCode::OPTION_L,
      KeyCode::F2, ModifierFlag::CONTROL_L,
    </autogen>

That doesn't seem to help.
^F2 should not to be sent in the following situation:
1. Press arrow key
2. Press alt key
3. Release alt key
4. Release arrow key

Thank you for feedback!
It's awesome idea.
I've added <pressingphysicalkeys_greaterthan> and <pressingphysicalkeys_lessthan> at v10.6.9.
Please upgrade to the latest beta version and use it like this:
    <pressingphysicalkeys_lessthan>2</pressingphysicalkeys_lessthan>
    <autogen>__KeyOverlaidModifier__ KeyCode::OPTION_L, KeyCode::OPTION_L, KeyCode::F2, ModifierFlag::CONTROL_L</autogen>

How to upgrade to the latest beta version:
https://pqrs.org/osx/karabiner/document.html#check-for-updates

Booom it works! Fantastic job! Thank you.

Thanks for the tip.
I use the following setting to use jj to enter Ubiquitous Vim Bindings
<modifierdef>VIRTUAL_J</modifierdef>
<item>
    <name>jj to Normal Mode</name>
    <appendix>Tap j : enter j</appendix>
    <appendix>Tap jj: enter normal mode</appendix>
    <identifier>private.j2normal</identifier>

    <not>TERMINAL, VI, PASSWORD, FIREFOX, EMACS, XCODE</not>
    <config_not>notsave.ubiq_vimnormal</config_not>

    <autogen>
      __KeyOverlaidModifier__
      KeyCode::J, ModifierFlag::VIRTUAL_J,

      @begin
      KeyCode::J,
      @end

      @begin
      KeyCode::DELETE,
      KeyCode::DELETE,
      KeyCode::VK_LOCK_ALL_FORCE_OFF,
      KeyCode::VK_CONFIG_FORCE_ON_notsave_ubiq_vimnormal,
      {{ UBIQUITOUS_VIM_BINDINGS_CANCEL_OPERATOR_PENDING }}
      @end
    </autogen>

    <autogen>
      __KeyOverlaidModifier__
      KeyCode::J,

      @begin
      KeyCode::J,
      KeyCode::VK_STICKY_VIRTUAL_J_FORCE_ON,
      @end
  </autogen>

</item>

authors: aristidesfl haifengkao tekezo

repo: tekezo/Karabiner

license: unlicense


text:
tektoncd-catalog org administration & test automation setup

Part of https://github.com/tektoncd/community/issues/889. This issue tracks the progress of the org-level administration setup using Prow:

[x] Label Sync: https://github.com/tektoncd/plumbing/pull/1318
[x] Peribolos: https://github.com/tektoncd/community/pull/933
[x] Prow: https://github.com/tektoncd/plumbing/pull/1324
[x] Org level shared files setup

https://github.com/tektoncd-catalog/.github/pull/1
https://github.com/tektoncd-catalog/.github/pull/2

[x] Test automation

https://github.com/tektoncd/plumbing/pull/1323
https://github.com/tektoncd/plumbing/pull/1334
https://github.com/tektoncd/plumbing/pull/1342
https://github.com/tektoncd/plumbing/pull/1343
https://github.com/tektoncd/plumbing/pull/1345
https://github.com/tektoncd/plumbing/pull/1350
https://github.com/tektoncd/plumbing/pull/1340

The tektoncd-catalog bootstrapping is done. The integration tests are pending https://github.com/tektoncd/plumbing/pull/1340 to be merged

/close

source: gharchive/issue

license_type: permissive

license_source: github-api


text:
Schedule Pods in resource-constrained environments

Expected Behavior
In a resource-constrained environment like a namespace with resource limits imposed (or just an insufficiently provisioned cluster), creating a TaskRun (Pod) that exceeds those limits should not fail the TaskRun, but should instead continually try to create the Pod until it either succeeds or times out.
Actual Behavior
Pods fail to start and the TaskRun is failed ~immediately.
Steps to Reproduce the Problem

Define a namespace with resource constraints (e.g., 10 CPU, 10 GB RAM)
Create 15 TaskRuns each requesting 1 CPU and 1 GB RAM, running hello world or something simple
~10 of those will be scheduled and will succeed, the rest will fail due to insufficient resources.

Additional Info
This is similar to how Jobs can handle Pod scheduling failures by retrying until they are successful.
It's unclear whether users would expect TaskRuns waiting for sufficient resources to queue in order of the time they were created, or whether they'd expect the Kubernetes scheduler to do whatever it needs to do to schedule the Pods. As an initial implementation it's probably fine to have Kubernetes schedule Pods, and not have to worry about enforcing FIFO.

/assign @sbwsg

Been working through some of the implementation details in a POC but want to drop current working notes here since I likely won't be able to work on it more until tomorrow.

Catching a pod failure is relatively straightforward; checking the error message produced by the createPod() func in pkg/reconciler/v1alpha1/taskrun/taskrun.go reveals the reason. From here it's quick to parse out the error message and look for e.g. "exceeded quota" in the string. This relies on a somewhat brittle contract though. I'll also need to check for the different messages generated both by LimitRanges as well as ResourceQuotas since it looks like they both enforce resource limits on a pod. I'm currently looking around to see if there's a less brittle approach to this error checking.

Once the resource constraint error is detected the pod then needs to be restarted. In my POC implementation this works by simply Enqueue()ing the TR to be re-assessed on the next reconcile loop. This results in many rapid reruns however when really it would be nicer to see an exponential backoff strategy similar to that used by k8s' job controller. The job controller uses a particular kind of workqueue to implement this ("ExponentialFailureRateLimiter") but TaskRun's controller Impl uses the "RateLimitedQueue", which is set up via knative's controller.NewImpl() func. So I'm looking at other alternatives to implement this.

I'm going to move this into a design doc. There are enough variables here to seed some discussion and it'd be good to get broader input before committing to one approach.

I've started the design doc here including use cases, a draft implementation, some open questions and possible alternative implementations that I'm still working through.

authors: ImJasonH sbwsg

repo: tektoncd/pipeline

license_source: bigquery


text:
Add script mode support to windows tasks

Changes
Steps and sidecars can contain a script field. In Linux tasks these
scripts are copied into files which are made executable and then
steps are added to the task to execute those files. This commit adds
comparable functionality if scripts are used in a task which will run
on a Windows node.
On a Windows node the mechanics are different, due to how Windows
handles executable files. The key difference is that Tekton needs to
know that a script will run on Windows, and how to run the file
(which interpreter to use). This is done through a ‘windows shebang’
line at the start of the script.
The line must begin with #!win. After that the user needs to provide
the interpreter to use, as well as any necessary arguments. The line
must be written such that the name of the file containing the script to
execute can be appended to the end.
For example, to run the script in the file test.ps1 with powershell,
the command would usually be powershell -File test.ps1 and so the
shebang line must be #!win powershell -File.
If no interpreter is provided (i.e. the shebang line is only ‘#!win’) then
the script contents will be stored in a .cmd file and executed.
Finally, since a pod cannot contain a mix of Windows and Linux
containers a windows shell image has been added to the Images
structure, which will be used in the place-scripts step when needed
on a windows node.
Related:
Issue #1826
TEP-0057
/kind feature
Submitter Checklist
As the author of this PR, please check off the items in this checklist:

[X] Docs included if any changes are user facing
[X] Tests included if any functionality added or changed
[X] Follows the commit message standard
[X] Meets the Tekton contributor standards (including
functionality, content, code)
[X] Release notes block below has been filled in or deleted (only if no user facing changes)

Release Notes
Add script mode support for windows Tasks/Pipelines

/assign @dibyom

/ok-to-test

Is there anything else I need to do here to get this moving forward?

After https://github.com/tektoncd/pipeline/pull/4139 is merged it would be great to have some e2e tests defined for this as well. We're still working on adding automation to those tests, though.

/retest

/retest

/lgtm

/retest

Thanks a lot for tackling those changes @DrWadsy . One more suggestion here - to maintain parity with other alpha features I think we should probably reject submitted TaskRuns / Tasks that use a windows script if the enable-api-fields feature flag is not alpha.  This gives users immediate feedback when they submit rather than waiting for a runtime error in the TaskRun's description.
Adding validation is super easy - we can do this in pkg/apis/pipeline/v1beta1/task_validation.go; here's existing code that performs a very similar check for the onError field of each Step: https://github.com/tektoncd/pipeline/blob/main/pkg/apis/pipeline/v1beta1/task_validation.go#L224-L233
I think the additional validation would look something like:
if s.Script != "" {
  cleaned := strings.TrimSpace(s.Script)
  if strings.HasPrefix(cleaned, "#!win") {
    errs = errs.Also(ValidateEnabledAPIFields(ctx, "windows script support", config.AlphaAPIFields).ViaField("script"))
  }
}

Adding a unit test to confirm this behaviour should then be as simple as adding an entry to the table in https://github.com/tektoncd/pipeline/blob/main/pkg/apis/pipeline/v1beta1/task_validation_test.go#L1200-L1202

Nice one, cheers!
/approve

Also just noticed - could you squash the commits down too? We typically squash down into 1 before merge to help keep the history a little more easily scannable. I normally do that with git rebase -i HEAD^^^^^^^^^^^^^ (one caret for each of the 13 commits) then you'd need to git push --force the branch. Doing so won't affect the approve I just added.

/retest

@imjasonh I made some code changes and squashed this all down into a single commit, and so the PR lost the lgtm you added. Could you please check this over one more time? :)

@imjasonh is out for another week I think, so I took a look.
/lgtm

source: gharchive/pull-request

authors: DrWadsy dlorenc imjasonh sbwsg vdemeester


text:
Remove experimental disable resolution controller flag

Changes
In commits 5681ce99232cffebc42b4a097514bc0a3e14a6e5 and
5fa4348bdbb7f28280bcfd518f6025c94670e420 we introduced a controller flag to
disable resolution of taskRefs and pipelineRefs. This was intended to support an
experimental controller that could externalize resolution of these refs. Tests
and documentation for this flag were added as well.
The experimental flag ultimately never realized its purpose and the
experimental controller was written to only support PipelineRuns started in
PipelineRunPending state. This commit removes the code, tests and docs
surrounding this flag and its behaviour.
Submitter Checklist

[x] Docs included if any changes are user facing
[x] Tests included if any functionality added or changed
[x] Follows the commit message standard
[x] Meets the Tekton contributor standards (including
functionality, content, code)
[x] Release notes block below has been filled in or deleted (only if no user facing changes)

Release Notes
An experimental controller flag was added in Pipelines v0.28.0 to disable resolution of taskrefs and pipelinerefs. This flag has now been removed.

/kind misc

/test pull-tekton-pipeline-alpha-integration-tests

/lgtm

still getting used to permissions on this repo 😅 anyway, looks good

thanks @sbwsg!
/lgtm

test/v1alpha1 TestPipelineTaskTimeout 1m5s
/test pull-tekton-pipeline-alpha-integration-tests

authors: jerop lbernick sbwsg


text:
🛑 Smart'Flore API is down

In dd6a777, Smart'Flore API (https://www.tela-botanica.org/smartflore-services/trails) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Smart'Flore API is back up in e932ad8.

repo: telabotanica/status

license: MIT


text:
Автоматически обновлять название канала

Слушать событие и отправлять в очередь

Все еще актуально

repo: telegram-bots/telegram-channels-feed

license: mit


text:
fix doc generation of init

see https://stackoverflow.com/questions/3757500/connect-sphinx-autodoc-skip-member-to-my-function

connected with #61

fixed

repo: telekom/HPOflow


text:
Wildcard (*) in excludedUrls is a catch all

Btw, kudos.
Describe the bug
I wanted {x,y,z}.foo.bar to be ignored, so I put *.foo.bar in excludedUrls. The result was Vimari not working anywhere. Btw, foo.bar seems to be enough for this purpose.
To Reproduce
Steps to reproduce the behavior:

Put something like *.facebook.com in excludedUrls.
Go to any page but *.facebook.com (example).
Nothing works.

Expected behavior
Key bindings to work.
Desktop (please complete the following information):

macOS version: 11.1
Safari version: 14.0.2
Vimari version: 2.1.0

Also, It'd be a good idea to make excludedUrls an array instead of CSV.

I have narrowed this down to here (L354): https://github.com/televator-apps/vimari/blob/768e7b93e8a36bb3882cdba043aa71f894f16864/Vimari Extension/js/injected.js#L352-L361; Then here (L374): https://github.com/televator-apps/vimari/blob/768e7b93e8a36bb3882cdba043aa71f894f16864/Vimari Extension/js/injected.js#L363-L380
What happens is: The js code tries to check whether it should activate in the first one. During this check, it tries to compile a regex from the provided strings in excludedUrls. But, something like *.foo.com is an invalid regex, and makes the code throw an exception, never reaching the activation, which is at L359-L360 (right?).
This can either be changed, so that *s are removed, etc.; Or, the documentation can be amended so that users are warned not to use * in their excludedUrls.
Either way (or any other), let me know what you think is the best approach for this, and I'll submit a PR.

repo: televator-apps/vimari


text:
Pipeline not triggered by webhook

Hiya!
Deeply considering the move to Concourse for a while now, and I'm currently constructing a copy of our existing CI setup for one of our projects to give it a spin.
Our flow today is as follows in (using GitHub + Azure Pipelines):

PR is opened
Since PR includes at least one commit, PR triggers CI pipeline
If any step fails, CI pushes a status to GitHub checks
If pipeline completed, CI pushes a status to GitHub checks

I've managed to replicate the last 3 steps in Concourse in full, quite fast and using the command line on a local Concourse setup which is super-duper-awesome (much, much faster the re-doing CI configurations in Azure).
Step 1, however, remains semi-unsolved. I've got two options to know that a PR has been opened:

poll - Use every_which in my resource declaration to poll the relevant resource every so-and-so seconds (where the resource's resource-type is github-pr-resource, based on this repo).
push - Wait for a webhook to come from GitHub whenever a new push or pull_request event have been created on the repo.

The poll version works fine, but has the obvious problem of waiting for the timeout to expire and of over-polling the GitHub API which may cause problems at the future.
The push version does not work, or at least not how I would expect it to work. I've looked at https://github.com/concourse/concourse/issues/3411 and it seems this is a similar issue to what I'm facing: I'd like the webhook to trigger the pipeline, no the check of the resource.
Some help would be much appreciated - I've set the webhook on the repo with the correct privileges, set the personal access token with the correct privileges as well, and can see that POSTing to the relevant webhook returns a 201 with the response I'd expect from concourse.
My config (based on the example from the documentation):
resource_types:
- name: pull-request-base-resource
  type: docker-image
  source:
    repository: teliaoss/github-pr-resource

resources:
- name: pull-request
  type: pull-request-base-resource
  check_every: 10s
  webhook_token: <WEBHOOK_TOKEN>
  source:
    repository: <MY_REPO>
    access_token: <MY_PERSONAL_ACCESS_TOKEN>
    base_branch: <MY_BRANCH>
jobs:
- name: test
  plan:
  - get: pull-request
    trigger: true
    version: latest # Tried with every as well...
  - put: pull-request
    params:
      path: pull-request
      status: pending
  - task: unit-test
    config:
      platform: linux
      image_resource:
        type: docker-image
        source: {repository: alpine/git, tag: "latest"}
      inputs:
        - name: pull-request
      run:
        path: /bin/sh
        args:
          - -xce
          - |
            cd pull-request
            git log --graph --all --color --pretty=format:"%x1b[31m%h%x09%x1b[32m%d%x1b[0m%x20%s" > log.txt
            cat log.txt
    on_failure:
      put: pull-request
      params:
        path: pull-request
        status: failure
  - put: pull-request
    params:
      path: pull-request
      status: success

@tomgs can you describe in a bit more detail what you mean by

The push version does not work, or at least not how I would expect it to work

? I'm assuming you're doing some kind of end-to-end manual test ("opening a PR and waiting for something to happen in Concourse") - what are the steps you are taking, what is the behaviour you would expect and what is the behaviour that you observe? Also it's awesome that you shared your pipeline; can you share the specific configuration you have on the GitHub side (just in case there might be a problem there - I'm sure the authors of this repo could advise)?
More specifically, can you validate that the webhook is actually being triggered? One way I could imagine to test, this based on the code here, is: you could try configuring the webhook on the github side with an invalid token - then you could create a PR and look in the logs for check-resource-webhook expecting to see invalid-token somewhere.
I'm asking this because I noticed this section in the README:

Note on webhooks:
This resource does not implement any caching, so it should work well with webhooks (should be subscribed to push and pull_request events).
One thing to keep in mind however, is that pull requests that are opened from a fork and commits to said fork will not generate notifications over the webhook. So if you have a repository with little traffic and expect pull requests from forks, you'll need to discover those versions with check_every: 1m for instance. check in this resource is not a costly operation,
so normally you should not have to worry about the rate limit.

Is it possible that, in the scenarios you are testing, your PR is coming from a fork?

Hey @pivotal-jamie-klassen, thanks for the swift response here and in the other issue!
I'll follow up here as well as on the other thread.
(Have I mentioned that Concourse is just freaking awesome already? Really hoping we can massage it to fix our use case, as it would make my day to day much more pleasant)
To answer your questions one by one:

I'm assuming you're doing some kind of end-to-end manual test ("opening a PR and waiting for something to happen in Concourse") - what are the steps you are taking, what is the behaviour you would expect and what is the behaviour that you observe?

Should have been clearer. Let's break this down:
What I want: A PR is opened targeting master in my repo / a commit is made to such a PR --> CI picks up on the PR being opened  immediately --> Pipeline is started immediately
What I have: A PR is opened targeting master on my repo / a commit is made to such a PR --> CI picks up on the PR being opened about 10 seconds after it has opened --> Pipeline is started 10-15 seconds or so after PR was opened
Besides making sense from a CI architecture perspective (if A happens than immediately B happens), the scenario I want buys us precious worker time back. It also eliminates a 10-20 second window where GitHub knows a PR has been opened, but CI has not caught up with the put step to change the check status to pending. In that window, a developer can "bypass" CI and just click merge (since no required CI check is blocking the merge from happening).
The steps I'm taking to check this:

Create a commit on a local copy of the PR's branch
Pushing the commit to GitHub
Observing Concourse + the GitHub PR checks page, noting the aforementioned behavior

If it matters, I'm spinning up Concourse locally using the (lovely) StarkAndWayne docker-compose setup fronm https://github.com/starkandwayne/concourse-tutorial/blob/master/docker-compose.yml. It's a bare-bones, two container setup that I use for evaluating the platform.

Also it's awesome that you shared your pipeline; can you share the specific configuration you have on the GitHub side (just in case there might be a problem there - I'm sure the authors of this repo could advise)?

Yeah, of course - while I can't share the actual repo for obvious reasons, the general idea is this:

Since I'm running locally, spin up an ngrok instance to tunnel into my localhost's port 8080 (where concourse lives).
In the repo I'm testing with, go to Settings --> Webhooks --> Add webhook
Following Concourse's resource webhook token docs, I've created a webhook URL of the following format:

https://MY_LOCAL_NGROK_ID.ngrok.io/api/v1/teams/TEAM_NAME/pipelines/PIPELINE_NAME/resources/RESOURCE_NAME/check/webhook?webhook_token=WEBHOOK_TOKEN

I entered said URL in the webhook configuration page:

Note that the WEBHOOK_TOKEN appended to the URL should be the same as the Secret I'm entering - made sure they match, and are a simple enough, short string so Concourse won't get confused (probably not needed, but just in case).
5. Created the webhook

More specifically, can you validate that the webhook is actually being triggered? One way I could imagine to test, this based on the code here, is: you could try configuring the webhook on the github side with an invalid token - then you could create a PR and look in the logs for check-resource-webhook expecting to see invalid-token somewhere.

Ah, that's an interesting point. I figured it was enough to receive a 2** response from Concourse to ensure the webhook's actually pinged. So what I did was:

curl -X POST https://MY_LOCAL_NGROK_ID.ngrok.io/api/v1/teams/TEAM_NAME/pipelines/PIPELINE_NAME/resources/RESOURCE_NAME/check/webhook?webhook_token=WEBHOOK_TOKEN
Get back nothing from curl
Observe that ngrok logs the request with a 2** response, which it does - I'm getting back a 201 created.

But let's dive deeper for a second. First, fly -t <MY_TARGET> check-resource  -r <MY_PIPELINE>/pull-request, seems to succeed:
id  name                        status     check_error                                                   │66125e75a5a1        concourse/concourse:5.6.0   "dumb-init /usr/loca…"   About a minute ago   Up 53 secon
4   pull-request-base-resource  succeeded                                                                │ds       0.0.0.0:8080->8080/tcp   spin-concourse_concourse_1
id  name          status     check_error                                                                 │7401a1d3d471        postgres                    "docker-entrypoint.s…"   About a minute ago   Up 53 secon
5   pull-request  succeeded

grepping the Concourse main container (not the db) logs for check-resource-webhook yields nothing. Same for webhook. This is bizarre, and I assume means the pull-request resource is not working as I expect it to. Could this be the source of the problem? I'm assuming that's out of your scope and down to the pull-request resource maintainers, I guess.

Is it possible that, in the scenarios you are testing, your PR is coming from a fork?

Nope, no forks for me, strictly PRs to the main repo.
Also taking a look at the other thread you mentioned. And thanks a lot, again!

Hi and thanks for the issue. With reference to the Concourse documentation:
https://concourse-ci.org/resources.html#resource-webhook-token

Note that the WEBHOOK_TOKEN appended to the URL should be the same as the Secret I'm entering in the window above. I made sure the two match to the same string, and that the string is a simple and short string so Concourse won't get confused (probably not needed, but just in case).

Just FYI; you don't need to set the Secret in Github. Concourse only cares about matching the WEBHOOK_TOKEN set in your resource definition with the path variable used when delivering the webhook (from Github).

I'd like the webhook to trigger the pipeline, no the check of the resource.

This is not possible. Concourse expects all webhooks to be delivered with a reference to the resource it should trigger, which is why the webhook_token is specified per resource and not per pipeline.

CI picks up on the PR being opened / the commit being made immediately --> Pipeline is started immediately

This is not possible because (as the docs referenced above states) when Concourse receives an incoming webhook, the payload is discarded and a regular check is scheduled instead, which takes time and makes Concourse less responsive; please see https://github.com/concourse/concourse/issues/2240 for more details.
Probably not the answers you were looking for, but I hope it helps anyhow 😅

@itsdalmo Thank you! This makes more sense now. I'll follow the other issue, and try to figure out another way forward without this.

What I have: A PR is opened targeting master on my repo / a commit is made to such a PR --> CI picks up on the PR being opened / the commit being made about 10 seconds after it has opened --> Pipeline is started 10-20 seconds or so after PR was opened

@tomgs I think I have a solution for your timing problem here (I hope 🤞).
tldr: Branch Protection Rules!
Recently in the concourse-chart repo we added some checks for PR's. After concourse reported the status of the check back to a PR in concourse-chart once, I was then able to set that check as required for all future PR's. This results in the 10-20 second gap never happening, therefore no one can hit merge without the check happening.
Under the Settings tab for the repo, when adding a branch protection rule you'll see this area:

Once you make the check required users won't be able to hit merge prematurely:

Hope this helps!

Closing this issue as I believe it has been answered (and also it is more related to Concourse than the resource itself). Feel free to reopen if I'm wrong 🙌

authors: itsdalmo pivotal-jamie-klassen taylorsilva tomgs

repo: telia-oss/github-pr-resource


text:
fix(docker): replace tini with dumb-init

Replace tini with dumb-init  as the use TINI_SUBREAPER  is causing issues where  PR_SET_CHILD_SUBREAPER is unavailable on some  platforms.

@abester Should we make this change across the other Dockerfiles we have?

@abester Do we have a write up of the underlying issue somewhere?  As @TelicentPaul notes we've used tini as the init process across pretty much all the platform containers so this will need fixing globally and it would be good to understand the problem so we aren't making the wrong fix and have to churn through another round of PRs

@abester Do we have a write up of the underlying issue somewhere? As @TelicentPaul notes we've used tini as the init process across pretty much all the platform containers so this will need fixing globally and it would be good to understand the problem so we aren't making the wrong fix and have to churn through another round of PRs

details of the problem: https://github.com/docker/for-mac/issues/6620#issuecomment-1694380189
its preventing the app from starting up when running in k8s on local desktop such as rancher-desktop , etc.

@abester Should we make this change across the other Dockerfiles we have?

I've done the PR for search, for now thats all that we need.

authors: TelicentPaul abester rvesse

repo: telicent-oss/smart-cache-graph


text:
Fetch and display data from multiple APIs in a GUI project

[x] Spotify- signed up already

Netflix- recommended
YouTube
Geographical databases
Instagram
OpenAI- signed up already
Pexels
Imdb
https://pypistats.org/api/
Plan:
Build on the success of using the Spotify API.

[x] Find the working function
[x] Connect it to a UI button.
[x] Combine it with the Qt tutorial templates.
[x] First mini goal: call the same function using a QPushButton
[x] Get the main window to launch

provide inputs for username and password before trying to make API request.

[x] 3. display something from the resulting HTTP response in the UI. Hardcode to Fire on the Cathedral for Now.
21 fields.

[x] Show them all.

[x] Show a definition of each field in the UI, at least on hover.

[ ] Add icons to each feature.

[x] instead of ms, convert to min and sec. S = ms / 1000. Minutes = s // 60. Seconds leftover = s % 60. Formatted song length = f”{min}:{s_leftover}”

[ ] There is a bug with the song duration: I saw 4:6 as the time. Needs to show as 4:06 maybe?

[x] Instead of an integer, display the actual key of the song, e.g. C#

[x] Need the ability to show text in a certain location.

[ ] For a consecutive search, make sure the UI clears before displaying the new results

[ ] A tree widget might be better for displaying tabular data like this, especially with alternating row colors

Discovered that I've been retrieving the track's audio features, whereas I can also make a GET request to retrieve a track object. This track object has more general information included.

[x] Use the Search endpoint to Get Spotify catalog information about albums, artists, playlists, tracks, shows, episodes or audiobooks that match a keyword string.
[x] Make it more general so that any track uri can be looked up- requires providing a text input to search for an artist.
[ ] Let the user hear the 30 second clip.
[x] Let the user see the album art.
[x] Let the user see the track name and artist (and album possibly)
[x] Allow clicking on the top result's track and artist to run and display the audio features for that track
[x] Turn the QLabel into a QPushButton
[x] Let the user search and get the track uri by typing the track name and/or artist

Use another API! At first just make it work for the second API as well. Later, Be able to use both without killing the app.

[x] Make it through all 6 steps of this Instagram user display tutorial. 1/3/2023
[x] https://developers.facebook.com/docs/instagram-basic-display-api/getting-started
[x] Get the ids and captions of all 3 of my posts through curl
[x] Used the returned image url to actually view the image
[ ] Convert the image to a format compatible with QPixmap and display it in GUI.
[ ] Convert the curl requests into python code
[ ] Convert the short-lived access token to a long-lived access token. Use https://developers.facebook.com/docs/instagram-basic-display-api/guides/long-lived-access-tokens

Integrate more functions
Clear the UI when the button for the second api is clicked, and vice versa- perhaps only add the labels when the "Get Audio Features Button is clicked" and remove them when a different button is clicked.

Maybe the top ten results should populate into a QComboBox, with a QPushButton attached
TODO: Connect to the song's audio
Even though the goal is to help listeners understand audio features of songs, letting them hear the song's audio is super pertinent
There is a bug with the song duration: I saw 4:6 as the time. Needs to show as 4:06 maybe?

Since we know we're searching for tracks, we can use the type=track filter, e.g. q=abacab&type=track.

Found the descriptions of the audio features! https://developer.spotify.com/documentation/web-api/reference/#/operations/get-audio-features

Notes about spotify's web ui:
When you search tracks in the spotify web player, you see the top four results
You get the track's icon which reveals a play button on hover, and a tool tip
You see the artist name small below the track; the artist's name has a URL for the artist's page
if you click on it.
You can click a heart icon to add it to your library
By default queries search all, but you can filter to song
If we filter to songs, we see a tabular view- 4 cols: result #, title, album, and duration
Title actually shows the artist and art as well


text:
🛑 Secret Site2 is down

In 9568801, Secret Site2 ($SECRET_SITE_2) was down:

HTTP code: 404
Response time: 96 ms

Resolved: Secret Site2 is back up in e898bb4 after 54 minutes.

repo: templain/mywatcher


text:
🛑 QRCODE VIEWER is down

In 418b56f, QRCODE VIEWER (https://qrcode-viewer.onrender.com/qrcode-viewer/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: QRCODE VIEWER is back up in 9d9bd72 after 2 hours, 13 minutes.


text:
added standard gitignore, and renamed folder to moneytransferapp

when installing and getting this running, the code doesn't compile because the folder is called moneytransfer, not moneytransferapp, so it was renamed.
Also, on my end, the gitignore didn't ignore some of the compiled files, so when I ran the code, git showed active changes

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

authors: CLAassistant GSmithApps

repo: temporalio/money-transfer-project-java


text:
Implement intern instruction (shallow)

Shallow implementation of the intern instruction.
The invoked function (symbol.c#rb_str_intern) actually may be tricky, as there's a conditional (USE_SYMBOL_GC) implementation.

@tenderlove do you have any preference, in general, about recycling (rewriting) the top of the temp stack, rather than popping it then repushing a value?

do you have any preference, in general, about recycling (rewriting) the top of the temp stack, rather than popping it then repushing a value?

I was going to say "no preference".  But I think we actually do want to pop / push when we know the type.  For example, in this case we know intern is going to push a Symbol on the stack and later instructions might want to use that information.  Since we don't pop / push the temp stack might indicate there's a String on top when actually it's a Symbol

Since we don't pop / push the temp stack might indicate there's a String on top when actually it's a Symbol

Oh that's a good point 🤔 I think I should go ahead and tweak the swap instruction to actually pop & push values, otherwise types may be confused

Updating to use push/pop, then merging in. The PR closure was accidental :grin:

authors: 64kramsystem nvasilevski tenderlove

repo: tenderlove/tenderjit


text:
🛑 Beeldoverleg Website is down

In 26e4c3d, Beeldoverleg Website (https://beeldoverleg.nl) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Beeldoverleg Website is back up in a6f5809 after 18 minutes.

repo: tendsgn/upptime


text:
Tokenizer class LLaMATokenizer does not exist or is not currently imported.

could you please help me ,I meet the follwing issue, thank you so much!
base) ubuntu@VM-48-6-ubuntu:~  modelz-llm -m decapoda-research/llama-7b-hf
Namespace(model='decapoda-research/llama-7b-hf', emb_model='sentence-transformers/all-MiniLM-L6-v2', dry_run=False, device='auto', port=8000, worker=1)
2023-10-18 23:55:43,181 - 5543 - INFO - model.py:32 - loading model and embedding: <class 'transformers.models.auto.modeling_auto.AutoModelForCausalLM'>(decapoda-research/llama-7b-hf) <class 'transformers.models.auto.tokenization_auto.AutoTokenizer'>(decapoda-research/llama-7b-hf)
Process SpawnProcess-1:
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/ubuntu/anaconda3/lib/python3.9/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/modelz_llm/uds.py", line 27, in init
self.func = cls(**kwargs)
File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/modelz_llm/model.py", line 39, in init
self.tokenizer = tokenizer_cls.from_pretrained(
File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/transformers/models/auto/tokenization_auto.py", line 748, in from_pretrained
raise ValueError(
ValueError: Tokenizer class LLaMATokenizer does not exist or is not currently imported.
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/bin/modelz-llm", line 8, in
sys.exit(main())
File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/modelz_llm/cli.py", line 62, in main
app = build_falcon_app(args)
File "/home/ubuntu/anaconda3/lib/python3.9/site-packages/modelz_llm/falcon_service.py", line 128, in build_falcon_app
raise RuntimeError("failed to start the service")
RuntimeError: failed to start the service
(base) ubuntu@VM-48-6-ubuntu:~$

Check the issue https://github.com/huggingface/transformers/issues/22222

authors: kemingy ken-xu-e

repo: tensorchord/modelz-llm


text:
Moved test out of run_in_graph_and_eager_mode in triplet_test

cc @gabrieldemarmiesse

@lc0
You are owner of some files modified in this pull request.
Would you kindly review the changes whenever you have the time to?
Thank you very much.

authors: autoih bot-of-gabrieldemarmiesse

repo: tensorflow/addons


text:
Fix LayerNormLSTMCell

Fix #321.
Bugs are caused by these lines
https://github.com/tensorflow/addons/blob/31750c99bc24620b2d6cba834a5c954e9c666f32/tensorflow_addons/rnn/cell.py#L324-L326
Both kernel_norm and recurrent_norm work on un-split tensors and will take tensors of shape [batch_size, self.units * 4] as input.
Have no idea why previous version could work though...

Ah, Thanks for fixing the issue. we changed the implementation for the normalization from normalize the individual gates to fused gates in some previous version. It might work previously if the layer was built twice with different shape.
The new code looks correct.


text:
terminate called after throwing an instance of 'std::out_of_range' error when call made to tf.contrib.tensor_forest.random_forest.TensorForestEstimator.predict()

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Ubuntu 16.04
TensorFlow installed from (source or binary):
Installed with pip install tensorflow-gpu
TensorFlow version (use command below):
('v1.2.0-rc2-21-g12f033d', '1.2.0')
Bazel version (if compiling from source):
n/a
CUDA/cuDNN version:
5.1.10 for CUDA 8.0
GPU model and memory:
name: GeForce GTX 1050 Ti
major: 6 minor: 1 memoryClockRate (GHz) 1.493
pciBusID 0000:01:00.0
Total memory: 3.94GiB
Free memory: 3.46GiB
Exact command to reproduce:
python RandomForestTrainer.py ?? I'd be happy to upload my code
You can collect some of this information using our environment capture script:
Collecting system information...
2017-06-26 02:27:23.087673: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-26 02:27:23.087713: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-26 02:27:23.087720: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-06-26 02:27:23.087727: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-06-26 02:27:23.394076: I tensorflow/core/common_runtime/gpu/gpu_device.cc:940] Found device 0 with properties:
name: GeForce GTX 1050 Ti
major: 6 minor: 1 memoryClockRate (GHz) 1.493
pciBusID 0000:01:00.0
Total memory: 3.94GiB
Free memory: 3.46GiB
2017-06-26 02:27:23.394119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:961] DMA: 0
2017-06-26 02:27:23.394126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0:   Y
2017-06-26 02:27:23.394144: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0)
Wrote environment to tf_env.txt. You can review the contents of that file.
and use it to populate the fields in the github issue template.

cat tf_env.txt
== cat /etc/issue ===============================================
Linux Desktop 4.4.0-79-generic #100-Ubuntu SMP Wed May 17 19:58:14 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
VERSION="16.04.2 LTS (Xenial Xerus)"
VERSION_ID="16.04"
VERSION_CODENAME=xenial
== are we in docker =============================================
No
== compiler =====================================================
c++ (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
== uname -a =====================================================
Linux Desktop 4.4.0-79-generic #100-Ubuntu SMP Wed May 17 19:58:14 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
== check pips ===================================================
numpy (1.12.1)
protobuf (3.3.0)
tensorflow-gpu (1.2.0)
== check for virtualenv =========================================
False
== tensorflow import ============================================
tf.VERSION = 1.2.0
tf.GIT_VERSION = v1.2.0-rc2-21-g12f033d
tf.COMPILER_VERSION = v1.2.0-rc2-21-g12f033d
Sanity check: array([1], dtype=int32)
== env ==========================================================
LD_LIBRARY_PATH /usr/local/cuda/lib64/
DYLD_LIBRARY_PATH is unset
== nvidia-smi ===================================================
Mon Jun 26 02:27:24 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.66                 Driver Version: 375.66                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 105...  Off  | 0000:01:00.0      On |                  N/A |
|  0%   45C    P0    36W / 120W |    448MiB /  4031MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      1164    G   /usr/lib/xorg/Xorg                             258MiB |
|    0     15045    G   ...el-token=7CE624E5F1863243374CB7B5F4C7B81C    72MiB |
|    0     20044    G   /usr/lib/xorg/Xorg                              41MiB |
|    0     25088    G   compiz                                          38MiB |
+-----------------------------------------------------------------------------+
== cuda libs  ===================================================
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudart_static.a
/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudart.so.8.0.61
/usr/local/cuda-8.0/doc/man/man7/libcudart.7
/usr/local/cuda-8.0/doc/man/man7/libcudart.so.7
Describe the problem
I created and trained a classifier with the tf.contrib.tensor_forest.random_forest.TensorForestEstimator class, but when I try to use the predict() method I get the following error:
2017-06-26 02:30:13.003812: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0)
terminate called after throwing an instance of 'std::out_of_range'
terminate called recursively
terminate called recursively
Aborted (core dumped)

I'd be happy to send the program that generated this if it helps as I don't see a way to post it here.  Sorry if I'm doing something wrong; I've never filed a bug like this before.

Can we close this or can you send a TF 2.x standalone minimal example or Colab to repdocue this?

@kessler9 ,
We see that you are using older version of tensorflow (1.x) which is not actively supported. We recommend that you upgrade to latest stable version of tensorflow 2.6.0 and let us know if the issue still persists in newer versions .Thanks!

authors: bhack kessler9 tilakrayal

repo: tensorflow/tensorflow


text:
tf.nn.separable_conv2d is slower than conv2d on GPU

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):Yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): TF 1.3
Python version: 3.6
Bazel version (if compiling from source):
CUDA/cuDNN version: CUDA8.0 /cuDNN6
GPU model and memory: GTX1080ti  11G

Describe the problem
In theory, separable_conv2d should be more efficient than conv2d, but when I test a simple model on Cifar10, the result shows that nn.separable_conv2d run slower on GPU, but is indeed faster on CPU.
Here is my test results on GPU:
training time for normal_conv after 2000 step: 8.18395892999979 sec
time for normal_conv after one forward step:  0.003980965999289765 sec
training time for separable_conv after 2000 step: 9.158266903999902 sec
time for separable_conv after one forward step:  0.0036441169995669043 sec

Source code / logs
Below is a fully self-contained example, I first define a model with two  conv2d , than I define another model with one conv2d followed by one  separable_conv2d, both model have 32 channels for each conv_layer and identical fc_layer.
import tensorflow as tf
import timeit
import numpy as np
from tensorflow.contrib.keras.python.keras.datasets.cifar10 import load_data

(x_train, y_train), (x_val, y_val) = load_data()
learning_rate = 0.001
num_steps = 1000
n_classes = 10
batch_size = 32

def reformat(labels):
    # Map 0 to [1.0, 0.0, 0.0 ...], 1 to [0.0, 1.0, 0.0 ...]
    labels = (np.arange(n_classes) == labels[:,None]).astype(np.float32)
    return  labels.reshape(labels.shape[0],10)
train_labels = reformat(y_train)
tf.reset_default_graph()
x = tf.placeholder(tf.float32, [None, 32, 32, 3])
y = tf.placeholder(tf.float32, [None, 10])
weights1 = {}
weights2 = {}
dtype = tf.float32
with tf.name_scope('INIT_OP'):
    conv_initializer =  tf.contrib.layers.xavier_initializer_conv2d(dtype=dtype)
    fc_initializer =  tf.contrib.layers.xavier_initializer(dtype=dtype)

k = 3
kernel = 16

# Define weights for normal ConvNet
with tf.name_scope('VARIABLES_1'):
    weights1['conv1'] = tf.get_variable('conv1', [k, k, 3, kernel], initializer=conv_initializer, dtype=dtype, trainable=True)
    weights1['b1'] = tf.get_variable('b1', initializer=tf.zeros([kernel]))
    weights1['conv2'] = tf.get_variable('conv2', [k, k, kernel, kernel], initializer=conv_initializer, dtype=dtype, trainable=True)
    weights1['b2'] = tf.get_variable('b2', initializer=tf.zeros([kernel]))

    weights1['wd1'] = tf.get_variable('wd1', [8*8*kernel, 512], initializer=fc_initializer, dtype=dtype, trainable=True)
    weights1['bd1'] = tf.get_variable('bd1',  initializer=tf.zeros([512]) )
    weights1['wd2'] = tf.get_variable('wd2', [512, 10], initializer=fc_initializer, dtype=dtype, trainable=True)
    weights1['bd2'] = tf.get_variable('bd2',  initializer=tf.zeros([10]) )

#Define weights for separable ConvNet
with tf.name_scope('VARIABLES_sep'):
    weights2['conv1'] = tf.get_variable('2_conv1', [k, k, 3, kernel], initializer=conv_initializer, dtype=dtype, trainable=True)
    weights2['conv_dw2'] = tf.get_variable('conv_dw2', [k, k, kernel, 1], initializer=conv_initializer, dtype=dtype, trainable=True)
    weights2['conv_pw2'] = tf.get_variable('conv_pw2', [1, 1, kernel, kernel], initializer=conv_initializer, dtype=dtype, trainable=True)

    weights2['b1'] = tf.get_variable('2_b1', initializer=tf.zeros([kernel]))
    weights2['b2'] = tf.get_variable('2_b2', initializer=tf.zeros([kernel]))

    weights2['wd1'] = tf.get_variable('2_wd1', [8*8*kernel, 512], initializer=fc_initializer, dtype=dtype, trainable=True)
    weights2['bd1'] = tf.get_variable('2_bd1',  initializer=tf.zeros([512]) )
    weights2['wd2'] = tf.get_variable('2_wd2', [512, 10], initializer=fc_initializer, dtype=dtype, trainable=True)
    weights2['bd2'] = tf.get_variable('2_bd2',  initializer=tf.zeros([10]) )

def forward_conv_sep( inp, weights):
    hidden = conv_block(inp, weights2['conv1'], weights2['b1'])
    hidden = maxpool2d(hidden)
    hidden = conv_block_dw(hidden, weights2['conv_dw2'], weights2['conv_pw2'], weights2['b2'])
    hidden = maxpool2d(hidden)
    hidden = tf.reshape( hidden, [-1, np.prod([int(dim) for dim in hidden.get_shape()[1:]])] )
    fc1 = tf.matmul(hidden, weights2['wd1']) + weights2['bd1']
    fc1 = tf.nn.relu(fc1)
    return tf.matmul(fc1, weights2['wd2']) + weights2['bd2']

def forward_conv( inp, weights):
    hidden = conv_block(inp, weights1['conv1'], weights1['b1'])
    hidden = maxpool2d(hidden)
    hidden = conv_block(hidden, weights1['conv2'], weights1['b2'])
    hidden = maxpool2d(hidden)
    hidden = tf.reshape( hidden, [-1, np.prod([int(dim) for dim in hidden.get_shape()[1:]])] )
    fc1 = tf.matmul(hidden, weights1['wd1']) + weights1['bd1']
    fc1 = tf.nn.relu(fc1)
    return tf.matmul(fc1, weights1['wd2']) + weights1['bd2']

def conv_block_dw(inp, cweight_w, cweight_p, bweight):
    no_stride =  [1,1,1,1]
    conv_output = tf.nn.separable_conv2d(inp, cweight_w, cweight_p, no_stride, 'SAME') + bweight
    return tf.nn.relu(conv_output)

def conv_block(inp, cweight, bweight, activation=tf.nn.relu):
    no_stride =  [1,1,1,1]
    conv_output = tf.nn.conv2d(inp, cweight, no_stride, 'SAME') + bweight
    return tf.nn.relu(conv_output)

def maxpool2d(inp, k=2):
    return tf.nn.max_pool(inp, ksize=[1, k, k, 1], strides=[1, k, k, 1],
                          padding='SAME')

#logits for normal ConvNet
with tf.name_scope("forward_conv"):
    pred1 = forward_conv(x, weights1)

#Cost for normal ConvNet
with tf.name_scope("cost1"):
    loss1 = tf.nn.softmax_cross_entropy_with_logits(logits=pred1, labels=y)
    cost1 = tf.reduce_mean(loss1)

#training op for normal ConvNet
with tf.name_scope('train_op1'):
    train_op1 = tf.train.RMSPropOptimizer(learning_rate, 0.9).minimize(cost1)

#logits for separable ConvNet
with tf.name_scope("forward_conv_sep"):
    pred2 = forward_conv_sep(x, weights2)

#Cost for separable ConvNet
with tf.name_scope("cost2"):
    loss2 = tf.nn.softmax_cross_entropy_with_logits(logits=pred2, labels=y)
    cost2 = tf.reduce_mean(loss2)

# training op for separable ConvNet
with tf.name_scope('train_op2'):
    train_op2 = tf.train.RMSPropOptimizer(learning_rate, 0.9).minimize(cost2)

with tf.name_scope('INIT'):
    init = tf.global_variables_initializer()

with tf.Session() as sess:

    sess.run(init)

    #train normal ConvNet for 2000 steps
    start = timeit.default_timer()
    for step in range(num_steps):
        r = np.random.choice(y_train.shape[0], batch_size, replace=False)
        batch_data = x_train[r]
        batch_labels = train_labels[r]

        feed_dict = {x : batch_data, y: batch_labels}
        _ , l = sess.run([train_op1,cost1], feed_dict=feed_dict)

    stop = timeit.default_timer()
    print ('training time for normal_conv after '+str(num_steps)+' step:',stop - start)

    start = timeit.default_timer()
    feed_dict = {x : batch_data, y: batch_labels}
    predictions1 = sess.run(pred1, feed_dict=feed_dict)
    stop = timeit.default_timer()
    print ('time for normal_conv after one forward step: ',stop - start)

    # train separable ConvNet for 2000 steps
    start = timeit.default_timer()
    for step in range(num_steps):
        r = np.random.choice(y_train.shape[0], batch_size, replace=False)
        batch_data = x_train[r]
        batch_labels = train_labels[r]

        feed_dict = {x : batch_data, y: batch_labels}
        _ , l = sess.run([train_op2,cost2], feed_dict=feed_dict)

    stop = timeit.default_timer()
    print ('training time for sep_conv after '+str(num_steps)+' step:',stop - start)

    start = timeit.default_timer()
    feed_dict = {x : batch_data, y: batch_labels}
    predictions = sess.run(pred2, feed_dict=feed_dict)
    stop = timeit.default_timer()
    print ('time for sep_conv after one forward step: ',stop - start)

@vrv my recollection is that there are some issues with backprop for separable convolutions on GPUs, which had been somewhat improved lately. Can you comment please on the state of the art?

I'm not aware of any specific issues, kernels get faster as important models need them!  In general, convolutions of different sizes will have different performance characteristics, and it's possible our separable convolution implementations may be slow for some combinations of shapes.  I'm not sure whether that's the case here, but it could be.  I also don't know whether theory matches practice here, since separable convolutions are less compute dense than normal convolutions.  I believe the benefits you get are that you get to use fewer parameters to express a larger capacity convolution.

At what kernel sizes will convolutions be computed via FFT instead of directly? Anyway, a speedup by doing a separable convolution is more noticeable for larger kernels, so for small kernels the overhead involved in doing two convolutions will be larger than the speedup, especially for what I assume is a highly-optimized convolution setting with the 3x3 kernel (Winograd).
Essentially, for a [m, n] kernel it would take m*n calculations for a convolution and m+n calculations for the separable convolution, if I'm not mistaken.

@carlthome @vrv  I do notice that when the number of filters get larger (128 or 192 etc. ), separable_conv2d is faster than conv2d, but in my case, I applied separable_conv2d on cifar10 with small number of filters, and it is actually slower than conv2d on my GPU, what could be the cause?

Same principle. For small kernels matrices the overhead involved in doing two convolutions might be larger than the speedup.

Understood, thanks for the explanation!

It has been 14 days with no activity and the awaiting tensorflower label was assigned. Please update the label and/or status accordingly.

Why is this closed?  I implemented UNet with separable conv2d and it was around 80% slower than using a standard conv2d.  Is there anything in the works to optimize this like using groups?

Same problem. @HouseOfFinwe Did you figure out to fix this?

@keunwoochoi I did not, please let me know if either of you managed.

@HouseOfFinwe Ok thanks. https://github.com/tensorlayer/tensorlayer/issues/416#issuecomment-374180396 says the speed issue was fixed in tf 1.5, but I'm having it with 1.13 now. Do you remember which versions have you tried?

Sadly I do not

@HouseOfFinwe Ok thanks. tensorlayer/tensorlayer#416 (comment) says the speed issue was fixed in tf 1.5, but I'm having it with 1.13 now. Do you remember which versions have you tried?

Still exited on 1.14.

why close ??? still happen with tf1.13

This should be fixed by #33836. It currently requires tf-nightly but will ship in the coming 2.2 release.

authors: HouseOfFinwe Niels-ns Uchiha-Jay byronyi carlthome cy89 imaxpayne keunwoochoi liuheng92 tensorflowbutler vrv


text:
ImportError: No module named 'tensorflowvisu'

Please go to Stack Overflow for help and support:
https://stackoverflow.com/questions/tagged/tensorflow
If you open a GitHub issue, here is our policy:

It must be a bug or a feature request.
The form below must be filled out.
It shouldn't be a TensorBoard issue. Those go here.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
TensorFlow installed from (source or binary):
TensorFlow version (use command below):
Python version:
Bazel version (if compiling from source):
CUDA/cuDNN version:
GPU model and memory:
Exact command to reproduce:

You can collect some of this information using our environment capture script:
https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh
You can obtain the TensorFlow version with
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
Describe the problem
Describe the problem clearly here. Be sure to convey here why it's a bug in TensorFlow or a feature request.
Source code / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem.

I apologize but I am having a hard time understanding what the problem is, where the problem is, and what version it affects. Please resubmit and pay attention to the issue template (https://github.com/tensorflow/tensorflow/issues/new) . Please provide all the information it asks. Thank you.

authors: Ahmeedtraabelsi asimshankar


text:
Bug?: reading from Google Cloud Storage appears to be accessing cached version

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes.
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): v1.4.0-rc1-11-g130a514 1.4.0
Python version: 3.6
Bazel version (if compiling from source): N/A
GCC/Compiler version (if compiling from source): N/A
CUDA/cuDNN version: observed on CPU & GPU.
GPU model and memory: observed on CPU & GPU.
Exact command to reproduce: See below.

Describe the problem
How this arose:
We are trying to set up a basic distributed TF version, where we have separate pods (on Kubernetes) doing validation and training (a simple version, with 1 of each).  GCS is used as the backend to store model output (checkpoints, etc.).
The validation pod periodically (via Experiment’s continuous_eval https://github.com/tensorflow/tensorflow/blob/f5f2f789ea395e585ddcbc43e088fa63d6b41d0e/tensorflow/contrib/learn/python/learn/experiment.py#L564) periodically polls for new checkpoints to evaluate (https://github.com/tensorflow/tensorflow/blob/f5f2f789ea395e585ddcbc43e088fa63d6b41d0e/tensorflow/contrib/learn/python/learn/experiment.py#L517).
If it doesn’t find a new checkpoint, it (per the underlying code) echoes out “No new checkpoint ready for evaluation” and continues to wait for a new one.
In practice, we found that, even as the training pod produces new checkpoints, the validation pod never picks up a new checkpoint, beyond the first one.  I.e., it collects an initial checkpoint, does evaluation, and then, in all future cycles, echoes out "No new checkpoint ready for evaluation".
In debugging, we found that the checkpoint file the saver tries to load up is always found to be some earlier iteration of the file (https://github.com/tensorflow/tensorflow/blob/f5f2f789ea395e585ddcbc43e088fa63d6b41d0e/tensorflow/python/training/saver.py#L1005) -- i.e., it seems like the GCS file loader reads the file once, and, from then on out, is continuous accessing a cached version of the data.
Digging into the code further, this appears to be an issue with how the file reader (file_io.read_file_to_string(...) and downstream methods) loads GCS files.  We were able to replicate this behavior separately, below.
Help appreciated!

Is this intended behavior?  Is there, e.g., some sort of GCS setting we have incorrectly set?
Assuming we're seeing something real, is there any suggested remediation here, with regards to our validation behavior?  Our next step is going to be to try monkey-patching some of the tf functions to just pull the GCS file local to disk and read it from there...although this is of course not preferred.

As a side note, Experiment does have a number of references to special handling around using GCS as the backend, although I don't have enough context to say if this is relevant to what we are seeing or not (https://github.com/tensorflow/tensorflow/blob/f5f2f789ea395e585ddcbc43e088fa63d6b41d0e/tensorflow/contrib/learn/python/learn/experiment.py#L94, https://github.com/tensorflow/tensorflow/blob/f5f2f789ea395e585ddcbc43e088fa63d6b41d0e/tensorflow/contrib/learn/python/learn/experiment.py#L269).
Source code / logs
Below is a pair of scripts that will replicate this issue.  Run the “Basic Reader” (using the same loading interface from get_checkpoint_state) and the “Basic Writer” simultaneously, and the reader will initially catch whatever is in the file, and then never update, even as the writer continues to write.
Other things we tried:

Changing the read/write path to local disk, instead of GCS.  This, unsurprisingly, worked.
A version of a reader with a context manager (below), to try to reset the reader each loop, and an explicit file.close() (not shown).  Both had the same behavior, i.e., new reads didn't provide the updated file.
Writing from the same file (process) that we read from: probably unsurprisingly, this does work; i.e., the writing activity either updates the local cache or otherwise convinces the reader to grab a fresh copy from GCS (we didn’t actually test which this might be).

Basic Reader:
#!/usr/bin/env python

from tensorflow.python.lib.io import file_io
import time
import os

file='gs://[MYPATH]'

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = '/usr/src/app/gcloud/keys/google-auth.json'

def read():
    counter=0
    while counter<15:
        print("reading...")
        print("Contents:")
        print(file_io.read_file_to_string(file))
        print("")
        print("Sleeping for a second...")
        time.sleep(3)
        print("")
        print("")
        print("")
        counter +=1

read()

Basic writer
#!/usr/bin/env python

from tensorflow.python.lib.io import file_io
import time
import os

file='gs://[MYPATH]'

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = '/usr/src/app/gcloud/keys/google-auth.json'

def write():
    counter=0
    while counter<15:
        with file_io.FileIO(file, mode='w') as f:
            f.write(str(counter))
        print("Wrote {}".format(counter))
        print("")
        print("Sleeping for a second...")
        time.sleep(3)
        print("")
        print("")
        print("")
        counter +=1

write()

Reader with context manager
#!/usr/bin/env python

from tensorflow.python.lib.io import file_io

import time
import os

file='gs://[MYPATH]'

os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = '/usr/src/app/gcloud/keys/google-auth.json'

def read():
    counter=0
    while counter<15:
        print("reading...")
        print("Contents:")
        with file_io.FileIO(file, mode='r') as f:
           print(f.read())
        print("")
        print("Sleeping for a second...")
        time.sleep(3)
        print("")
        print("")
        print("")
        counter +=1

read()

You may set env var GCS_READ_CACHE_MAX_SIZE_MB to zero, and try it again.

We tested this and this seems to work.  Is this the intended behavior of the GCS tf reader?  What are the (broader) consequences of setting GCS_READ_CACHE_MAX_SIZE_MB=0?  It isn't clear to me what the consequences are of embracing this approach (maybe nothing negative!).

Yes, it's by design.  Distributed filesystems like GCS are designed for write once, read multiple times.  You shouldn't rely on the content of the "checkpoint" file, because you need to read/write it multiple times.

Yes, it's by design. Distributed filesystems like GCS are designed for write once, read multiple times. You shouldn't rely on the content of the "checkpoint" file, because you need to read/write it multiple times.

With all due respect, this doesn't make much sense.  Tensorflow is designed to use GCS as a backend for model saving, and TF is built to use GCS as a back-end for model saving.  tf.Experiment is built with a continuous_eval mode which tries to load the checkpoint repeatedly, to discover state and use it appropriately.  Even if we try to chalk this up to tf.Experiment being in contrib (albeit maintained and built by Googlers), we still have the problem that GCS is advertised as a viable and supported backend storage for Tensorflow.  This is a pretty basic scenario for it to break.

This (TF's) behavior is also inconsistent with the behavior of the Google Python implementation (https://googlecloudplatform.github.io/google-cloud-python/latest/storage/blobs.html) for interacting with GCS.
E.g., performing the same read-write loop while using the provided "download_as_string()" has the expected result--i.e., it correctly pulls the updated file, each time this is invoked.

Nagging Awaiting TensorFlower: It has been 14 days with no activity and the awaiting tensorflower label was assigned. Please update the label and/or status accordingly.

Potentially related: I ran into the same issue when running standard estimator-based training on GCloud. In my case the trainer was accessing a GCS bucket in a different region (europe-west1 vs europe-west3, respectively). Setting GCS_READ_CACHE_MAX_SIZE_MB to 0 did not help. Placing both the worker and the bucket in a single region fixed the issue for me.

I'm not clear whether this bug persists into the current TF version (there
are some vague release notes as to improvements on GC storage), but it is
very disappointing not to get a more proactive response here from the TF
team--it shouldn't be extra hard to use Google products...with Google
products.
On Sat, Mar 10, 2018 at 1:46 PM, jonasz notifications@github.com wrote:

Potentially related: I ran into the same issue when running standard
estimator-based training on GCloud. In my case the trainer was accessing a
GCS bucket in a different region (europe-west1 vs europe-west3,
respectively). Setting GCS_READ_CACHE_MAX_SIZE_MB to 0 did not help.
Placing both the worker and the bucket in a single region fixed the issue
for me.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/issues/15530#issuecomment-372069548,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEc6EoC52pqzU9bg53ukrwMZuVA_l_Djks5tdEmxgaJpZM4RI6gT
.

Nagging TensorFlower: It has been 14 days with no activity and the awaiting tensorflower label was assigned. Please update the label and/or status accordingly.

Nagging Assignee @shivaniag: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @shivaniag: It has been 15 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

I had an issue of filling up the local RAM when loading many soundfiles from Google Cloud Bucket on Colaboratory using tf.data on Tensorflow 1.8.
Setting GCS_READ_CACHE_MAX_SIZE_MB=0 solved the issue for me.

Ran into a similar issue reading data from a GCS bucket with tf.read_file. Without setting GCS_READ_CACHE_MAX_SIZE_MB=0 all RAM was consumed by the cache. Not only was this useless (we were caching with tf Dataset anyways), it was also difficult to debug and not documented behaviour.

Nagging Assignee @shivaniag: It has been 15 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @shivaniag: It has been 30 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @shivaniag: It has been 44 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @shivaniag: It has been 61 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @shivaniag: It has been 76 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @shivaniag: It has been 91 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @shivaniag: It has been 106 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @shivaniag: It has been 121 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @shivaniag: It has been 136 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @shivaniag: It has been 166 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @shivaniag: It has been 180 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @shivaniag: It has been 196 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Hi @cbockman!
It seems you are using older versions(1.x versions) of Tensorflow. We recommend that you upgrade  your code base to 2.x  versions as many features and bug fixes has been done in newer versions and let us know if the issue still persists in newer versions. Thanks!

authors: agrinh amundv cbockman jonasz mohantym snnn tensorflowbutler


text:
AttributeError: module 'tensorflow.contrib' has no attribute 'distribute'

When I include mirroring strategy using following code -
Build the Estimator
distribution = tf.contrib.distribute.MirroredStrategy()
config = tf.estimator.RunConfig(train_distribute=distribution)
model = tf.estimator.Estimator(model_fn,config=config)
I get the error - AttributeError: module 'tensorflow.contrib' has no attribute 'distribute'
I tried this with both Tensorflow 1.7 and 1.8 - I get the same issue. I am using Python 3.6 https://colab.research.google.com/ .

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

Colab reasearch has v1.7. It has no module called distribute, so you should use that code with 1.8 and it works(tested).
https://github.com/tensorflow/tensorflow/tree/r1.7/tensorflow/contrib/distribute

Thank you for your comments! This was the issue happening only on Colab server. When I moved to my P2.8xlarge instance on AWS, it works fine there. Thank you for all your help and comments!


text:
Tensorflow Testing Problem

Hi,
I am working on Tensorflow with Python. I have a new idea about the pooling method. Currently there are two pooling methods  (max-pooling, avg-pooling). I want to do some experiment by some new idea about pooling method.
But for that i need to find the current pooling functions. But unfortunately i could not find that. Can anyone please help me to do this experiment and help me to find the exact flow of pooling because i am not so much fluent to Python.
Thanks.

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

@poxvoculi Hi
I am working on Tensorflow with Python. I have a new idea about the pooling method. Currently there are two pooling methods  (max-pooling, avg-pooling). I want to do some experiment by some new idea about pooling method.
But for that i need to find the current pooling functions. But unfortunately i could not find that. Can anyone please help me to do this experiment and help me to find the exact flow of pooling because i am not so much fluent to Python.
Thanks.

@tensorflowbutler ,  I don't know where to fill the fields you asked. So I am providing you information about these fields here:
Have I written custom code:    yes
OS Platform and Distribution:   Windows 10
TensorFlow installed from:      using pip
TensorFlow version:    1.7.0
Bazel version:   N/A
CUDA/cuDNN version:   8.0
GPU model and memory:     memory 24 GB
Exact command to reproduce  N/A

@tensorflowbutler
Please help to implement my idea about pooling.
Regards

@tensorflowbutler
Please help to implement my idea about pooling.
Regards

The pooling op source is in
pooling_ops_common.cc
and several similar files.  Look in core/kernels/* for anything with "pooling" in the name.
In the future, this sort of question is better asked on StackOverflow.  This forum is primarily for bug reports.

authors: mohsinali722 poxvoculi tensorflowbutler


text:
Add examples to documentation for CudnnRNN

Have I written custom code: N/A
OS Platform and Distribution: N/A
TensorFlow installed from: N/A
TensorFlow version: 1.8.0
Bazel version: N/A
CUDA/cuDNN version: 9.0/7.0
GPU model and memory: NVIDIA GTX 1080 Ti
Exact command to reproduce: N/A
It would be very helpful to add usage examples on how to use cudnn_rnn, particularly on cross-compatibility between non-CUDA and CUDA-supporting devices. Users should be able to figure out how to save/restore weights to run their models with, say, tf.nn.rnn_cell.LSTMCell or tf.contrib.cudnn_rnn.CudnnLSTM.
There are classes that seem to do this (e.g. tf.contrib.cudnn_rnn.CudnnLSTMSaveable) but there are no easily accessible code examples showing how they should be used.
A possible feature request: would it be possible to have a high-level wrapper that makes this choice automatically based on the availability of a CUDA device, saving and restoring weights accordingly? This should be possible with tf.test.is_gpu_available(cuda_only=True).

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

@danielwatson6 Could you point me if I'm correct?

While training, we can use just CudnnLSTM
When I want to freeze graph to pb, I should use CudnnCompatibleLSTM with MultiRNNCell and stack_bidirectional_dynamic_rnn (if  model was trained with such options)

Then, everything can be used in GPU and CPU.
If that pipeline is ok, it should be added to documentation?
Also question about CudnnLSTMSaveable, so what for this class was created?

I have manage to find a way to restore CudnnLSTM model on CPU.
Here are jupyters with example: https://gist.github.com/melgor/41e7d9367410b71dfddc33db34cba85f
The main problem was variable_scope which need to be set in proper way.

"A possible feature request: would it be possible to have a high-level wrapper that makes this choice automatically based on the availability of a CUDA device, saving and restoring weights accordingly? This should be possible with tf.test.is_gpu_available(cuda_only=True)"
We are actually trying to unify the API for this exact direction. Basically the new API will figure out the best choice implementation for user based on the available hardware, and user don't have to specify whether its CuDNN or CPU implementation in the code. Currently we are still in the design stage, and will post more results when the APIs are finalized. The new API will probably be available as part of the change in Tensorflow 2.0.

Nagging Assignee @robieta: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@melgor  which version of tensorflow you use in your scrip?

Nagging Assignee @robieta: It has been 16 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @robieta: It has been 31 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @robieta: It has been 61 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @robieta: It has been 76 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @robieta: It has been 92 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @robieta: It has been 106 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Closing as RNNs will now automatically select an implementation based on whether or not they are running on a GPU.

@robieta Which specific classes implement the automatic implementation selection you talk about? Are you suggesting that tensorflow.keras.layers.CuDNNGRU is now no longer needed because of those classes that implement automatic implementation selection?
As of TF 1.14, tensorflow.keras.layers.CuDNNGRU still behaves differently than tensorflow.keras.layers.GRU. In my experiment, when I switch between these two in the graph, the output I get during training is different; they simply calculate different things (whether it due to different formulas or different initial weights I cannot tell). The former also takes a shorter time to train each epoch, so there is still a good reason to prefer it for training.

The automatic CuDNN selection is only for the 2.x RNNs.

@robieta Which 2.0 RNN classes have the automatic CuDNN selection?

@jchia, LSTM and GRU layer has auto cudnn selection in the 2.x API.

@danielwatson6
Hi, I know this is a 2018 old issue but I am kind of stuck at those opaque kernel in tf 1.14,  so may I talk more details with you?
The situation is , I got a trained model with cudnnBLSTM in tf 1.14. What I want is to convert this tf models to MNN model format and then inference with that engine. I have a frozen graph of this model and I need to decode the weights and bias data from this frozen graph.
What I found is that the size of the opaque kernel is not equal to the size I calculate of this BLSTM. Specifically, say I got a 3 layers BLSTM with input dim 143 and state size 143, so the total size of parameter should be
(143 * 143 + 143 * 143 + 143) * 4 * 2 + 2 * (286 * 143 + 143 * 143 + 143) * 4 * 2 = 1312168

But when I try to read the size of opaque kernel, I got 1315600.
So there is no suprise when I read your comments about

The size and layout of the parameter buffers may change between
CUDA/CuDNN/GPU generations. Because of that, the opaque parameter variable
does not have a static shape and is not partitionable.

So my question are, is there anyway I can decode weights and bias from opaque kernel? How does the 'opaque kernel' organized? Is there any workaround consider my situation?
Thanks!

authors: Interfish chenbiaolong danielwatson6 jchia melgor qlzh727 robieta tensorflowbutler


text:
Performance decrease in TensorFlow 1.9 for large graphs

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): 1.9.0/1.8.0
Python version: 3.5
Bazel version (if compiling from source): n/a
GCC/Compiler version (if compiling from source): n/a
CUDA/cuDNN version: 9.0
GPU model and memory: GTX 980 Ti, 6GB
Exact command to reproduce:

import time

import tensorflow as tf

x = tf.zeros((1, 10))

for i in range(5000):
    x = tf.layers.dense(x, units=10)

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    print("a")
    start = time.time()
    sess.run(x)
    print(time.time() - start)

    print("b")
    start = time.time()
    sess.run(x)
    print(time.time() - start)

Describe the problem
On TensorFlow 1.9, this shows:
a
17.57940149307251
b
0.09075760841369629

On TensorFlow 1.8, this shows:
a
11.689541101455688
b
0.08776521682739258

So roughly a 50% increase in run time in TensorFlow 1.9.  Note that this only occurs on the first call to sess.run ("a" rather than "b"), so I suspect this is some change in the graph optimization step.  So, a couple questions:

What changed?  I can't find anything obviously related to this in the release notes.
Is this intended/expected (e.g., some new graph optimization steps were added that increase the initial time, but provide benefits in the long run)?  Or is it just a bug?
If it is intended, is there some way to optionally disable this new feature if we find that initial build time is dominating the run time?

I did a git bisect to figure out where the slowdowns occurred, and found 3 commits that contributed to the decreased performance:
https://github.com/tensorflow/tensorflow/commit/2c105ace934edce193669b55b13b64283caa24d7
Runs the graph optimizer pass twice (up from previous value of once), so makes sense that this would take longer.  Can be disabled via config.graph_options.rewrite_options.meta_optimizer_iterations=RewriterConfig.ONE.
https://github.com/tensorflow/tensorflow/commit/7e5d24de55572741f10ee7e17247ca195eaadfae
Turns on three new graph optimization steps: dead branch elimination, shape optimization, and remapping.  Shape optimization and remapping can be disabled via config.graph_options.rewrite_options.shape_optimization/remapping=RewriterConfig.OFF, but I don't think there is any way to selectively disable dead branch elimination (other than turning off all loop optimizations).
https://github.com/tensorflow/tensorflow/commit/2273b62a769aa477f8d2ef02ca7dee253b8ea7b0
This adds a lot of new logic to the SymbolicShapeRefiner, which is what is slowing things down I believe.  I don't think there is any way to disable this.
Note: all three commits are by @benoitsteiner, so he may be able to provide more detail.

Nagging Assignee @tatatodd: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

I'm experiencing a fairly large slowdown too (on Windows 7 and Python 3.6; CPU) with the release of TensorFlow 1.9 and 1.10. For example, in 1.8 and versions prior, each evaluation step for the CNN MNIST example model takes an average of 13 seconds while in 1.9 and 1.10, it trains for an average of 380 seconds. I hope this is still getting looked over.

I have the same issues after upgrading to tensorflow 1.9, 1.10 or nightly(one our ago) with docker images with Tesla V100 GPU. The global_step/sec slow down from 0.96 to 0.73 with the nasnet-large network trained with a customised version of tensorflow/models/research/slim

Nagging Assignee @tatatodd: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @tatatodd: It has been 29 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@drasmuss  Thanks for interesting investigation. Can you please test this for latest version of TensorFlow and post your findings?. We would like to know if this has been taken care of by latest version.

I had the hope that improvements of SymbolicShapeRefiner in commit 9f847cb in the latest nightly fix the issue, but nasnet-large in training is still slow (0.96 -> 0.72 global_step/sec).

Performance is the same (or slightly worse) on 1.11.0:
a
18.11762547492981
b
0.08776497840881348

CC @rmlarsen, in case Rasmus has more context about the Grappler changes in https://github.com/tensorflow/tensorflow/issues/20843#issuecomment-406702409

Nagging Assignees @tatatodd, @sherryy, @azaks2: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

I see following computation times for various TensorFlow versions and they look pretty descent to me:
TensorFlow Version 1.10 -
a
14.6700410843
b
0.0304279327393
TensorFlow Version 1.11 -
a
14.541062355041504
b
0.0331730842590332
TensorFlow Version 1.12.0 -
a
13.787318706512451
b
0.03347063064575195

The slowdown occurred in version 1.9, so you need to compare 1.9 to 1.8 if you want to see the difference (all the versions you are comparing are post-slowdown).

I talked with @rmlarsen and he mentioned that the regression in https://github.com/tensorflow/tensorflow/commit/2273b62a769aa477f8d2ef02ca7dee253b8ea7b0 has been fixed in later versions but later versions also introduced a few more optimizations.
I don't know details of your use-case but it is possible to disable all graph optimizations with the disable_meta_optimizer option and the time spent on optimizations can be restricted with the meta_optimizer_timeout_ms option.
Another approach could be to use a dummy "warm-up requests" as described in https://github.com/tensorflow/serving/issues/385#issuecomment-290443584.

I created an example script on gist with tensorflow models and the large nasnet. This script were run on the official tensorflow docker images.
On my Tesla V100 I got around for tensorflow 1.8-devel-gpu-py3
INFO:tensorflow:global_step/sec: 1.33334
With tensorflow 1.9-devel-gpu-py3 a maximum at
INFO:tensorflow:global_step/sec: 0.89946

@DavidWiesner That seems like a different issue to me and not related to the graph optimization overhead. If the regression is present in the latest release as well, consider filing a separate issue for that.

Nagging Assignees @sherryy, @rmlarsen: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Closing this issue as longer optimization time is expected with the presence of additional graph optimizations.

authors: DavidWiesner drasmuss mihi-r sherryy smit-hinsu tensorflowbutler ymodak


text:
Redundant summary in KMeansClustering

The estimator produces two summaries: loss and loss/raw. The former is due to the Estimator API, and the latter is due to this line. The summaries are identical, and I’m wondering what the reason for keeping the latter is. Is it that the Estimator API can, in some cases, augment the loss passed to it, or is it a leftover from before the API was introduced? Thanks!

@theweiho Can you please take a look? Thanks!

I suppose this one is no longer relevant, as contrib has been moved out.

authors: IvanUkhov ymodak


text:
Feature request: space_to_depth with overlapping blocks (like im2col)

Currently tf.space_to_depth only works for rearranging non-overlapping blocks into depth. It would be nice if also worked for overlapping blocks, with an extra parameter to set the stride:

A stride equal to the block size would be equivalent to the current non-overlapping behavior.
A stride equal to 1 would be equivalent to an im2col operation.

An im2col-like operation could be used to easily implement a locally connected layer ("convolution with unshared weights"), by combining it with an elementwise multiplication with the matrix that contains the local filters, followed by a summation along the channel dimension.

I tried to use tf.extract_image_patches to implement a locally layer and run into an error:
LookupError: No gradient defined for operation 'local/ExtractImagePatches' (op type: ExtractImagePatches)
I checked that  tf.space_to_depth has the gradient definition but  tf.extract_image_patches does not. And it seems not easy for me to implement the gradient. Any advice?

@jyshee did you get this working?

authors: bodokaiser dbbert jyshee


text:
[Sum/Average Loss Reduction Strategy] Why going with Sum as default ?

TF 1.14.0 introduced the following:

Set default loss reduction as AUTO for improving reliability of loss scaling with distribution strategy and custom training loops. AUTO indicates that the reduction option will be determined by the usage context. For almost all cases this defaults to SUM_OVER_BATCH_SIZE. When used in distribution strategy scope, outside of built-in training loops such as tf.keras compile and fit, we expect reduction value to be 'None' or 'SUM'. Using other values will raise an error.

Source: https://github.com/tensorflow/tensorflow/releases/tag/v1.14.0

Why SUM_OVER_BATCH_SIZE was made the default when the followings are standard practices:

averaging the loss over one batch
averaging gradients over multiple ranks

Could anyone give examples of use-cases where sum would give better results than average ? And why would this made default when averaging seems far more used in the state of the art ?

You can find it mentioned here https://www.tensorflow.org/beta/tutorials/distribute/training_loops
Normally, on a single machine with 1 GPU/CPU, loss is divided by the number of
examples in the batch of input. So, how should the loss be calculated when using
a tf.distribute.Strategy?For an example, let's say you have 4 GPU's and a batch
size of 64.One batch of input is distributed across the replicas (4 GPUs), each
replica getting an input of size 16.The model on each replica does a forward pass
with its respective input and calculates the loss. Now, instead of dividing the loss
by the number of examples in its respective input (BATCH_SIZE_PER_REPLICA = 16),
the loss should be divided by the GLOBAL_BATCH_SIZE (64).
Why do this?
This needs to be done because after the gradients are calculated on each replica,
they are synced across the replicas by summing them.
How to do this in TensorFlow? If you're writing a custom training loop, as in this
tutorial, you should sum the per example losses and divide the sum by the
GLOBAL_BATCH_SIZE: scale_loss = tf.reduce_sum(loss) * (1. / GLOBAL_BATCH_SIZE)
or you can use tf.nn.compute_average_loss which takes the per example loss, optional
sample weights, and GLOBAL_BATCH_SIZE as arguments and returns the scaled loss.

Automatically closing this out since I understand it to be resolved by @srihari-humbarwadi , but please let me know if I'm mistaken.Thanks!


text:
ImportError: No module named 'tensorflow_core.estimator'

Hi, today I updated tensorflow. While trying to run tensorboard, I get the following error:

ImportError: No module named 'tensorflow_core.estimator'

/home/niko/venv_tf2/bin/python /home/niko/workspace/pupil-detection-v2/tensorflow/diagnose_tensorboard.py
Diagnostics

Diagnostics output
--- check: autoidentify
INFO: diagnose_tensorboard.py version 4725c70c7ed724e2d1b9ba5618d7c30b957ee8a4

--- check: general
INFO: sys.version_info: sys.version_info(major=3, minor=5, micro=2, releaselevel='final', serial=0)
INFO: os.name: posix
INFO: os.uname(): posix.uname_result(sysname='Linux', nodename='helix', release='4.4.0-148-generic', version='#174-Ubuntu SMP Tue May 7 12:20:14 UTC 2019', machine='x86_64')
INFO: sys.getwindowsversion(): N/A

--- check: package_management
INFO: has conda-meta: False
INFO: $VIRTUAL_ENV: '/home/niko/venv_tf2'

--- check: installed_packages
INFO: installed: tensorboard==2.0.0
INFO: installed: tensorflow-gpu==2.0.0
INFO: installed: tensorflow-estimator==2.0.0

--- check: tensorboard_python_version
INFO: tensorboard.version.VERSION: '2.0.0'

--- check: tensorflow_python_version
INFO: tensorflow.__version__: '2.0.0'
INFO: tensorflow.__git_version__: 'v2.0.0-rc2-26-g64c3d38'

--- check: tensorboard_binary_path
INFO: which tensorboard: b'/home/niko/venv_tf2/bin/tensorboard\n'

--- check: readable_fqdn
INFO: socket.getfqdn(): 'helix'

--- check: stat_tensorboardinfo
INFO: directory: /tmp/.tensorboard-info
INFO: os.stat(...): os.stat_result(st_mode=16895, st_ino=2356525, st_dev=2066, st_nlink=2, st_uid=1000, st_gid=1000, st_size=4096, st_atime=1569914603, st_mtime=1569939157, st_ctime=1569939157)
INFO: mode: 0o40777

--- check: source_trees_without_genfiles
INFO: tensorboard_roots (1): ['/home/niko/venv_tf2/lib/python3.5/site-packages']; bad_roots (0): []

--- check: full_pip_freeze
INFO: pip freeze --all:
absl-py==0.8.0
astor==0.8.0
attrs==19.1.0
backcall==0.1.0
backports.weakref==1.0.post1
bleach==3.1.0
certifi==2019.6.16
chardet==3.0.4
colorama==0.4.1
cycler==0.10.0
decorator==4.4.0
defusedxml==0.6.0
entrypoints==0.3
enum34==1.1.6
gast==0.2.2
gitdb2==2.0.5
GitPython==3.0.2
google-pasta==0.1.7
grpcio==1.23.0
h5py==2.9.0
idna==2.8
ipykernel==5.1.2
ipython==7.7.0
ipython-genutils==0.2.0
jedi==0.15.1
Jinja2==2.10.1
json5==0.8.5
jsonschema==3.0.2
jupyter-client==5.3.1
jupyter-contrib-core==0.3.3
jupyter-core==4.5.0
jupyter-nbextensions-configurator==0.4.1
jupyterlab==1.0.9
jupyterlab-git==0.8.1
jupyterlab-server==1.0.6
Keras==2.2.5
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
keras-rectified-adam==0.10.0
kiwisolver==1.1.0
Markdown==3.1.1
MarkupSafe==1.1.1
matplotlib==3.0.3
mistune==0.8.4
nbconvert==5.6.0
nbdime==1.1.0
nbformat==4.4.0
notebook==6.0.1
numpy==1.17.1
opencv-python==4.1.0.25
opt-einsum==3.0.1
pandas==0.24.2
pandocfilters==1.4.2
parso==0.5.1
pexpect==4.7.0
pickleshare==0.7.5
Pillow==6.1.0
pip==19.2.3
pkg-resources==0.0.0
prometheus-client==0.7.1
prompt-toolkit==2.0.9
protobuf==3.9.1
ptyprocess==0.6.0
Pygments==2.4.2
pyparsing==2.4.2
pyrsistent==0.15.4
python-dateutil==2.8.0
pytz==2019.2
PyYAML==5.1.2
pyzmq==18.1.0
requests==2.22.0
scipy==1.3.1
Send2Trash==1.5.0
setuptools==41.2.0
six==1.12.0
smmap2==2.0.5
tensorboard==2.0.0
tensorflow-estimator==2.0.0
tensorflow-gpu==2.0.0
termcolor==1.1.0
terminado==0.8.2
testpath==0.4.2
tornado==6.0.3
traitlets==4.3.2
urllib3==1.25.3
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==0.15.5
wheel==0.33.6
wrapt==1.11.2

Next steps
No action items identified. Please copy ALL of the above output,
including the lines containing only backticks, into your GitHub issue
or comment. Be sure to redact any sensitive information.
Process finished with exit code 0

Can you post the versions of tensorflow, estimator and tensorboard you have?
Are you in a virtual env? what operating system? Can you please fill in the issue template?

I had the same issue, fixed it by uninstalling everything tensorflow related (also pip uninstall tensorflow-estimator tensorboard tb-nightly tf-estimator-nightly). A clean install afterwards fixed it.

@aaronpries , I uninstalled tensorboard, reinstalled estimator and tensorflow and it works fine now. I guess the issue can be closed.

@nikogamulin I came across the same error. What did you install and uninstall and reinstall can you clarify?

Thanks, that solved my problem!

authors: aaronpries mihaimaruseac ngamulin nikogamulin suminz9 ucalyptus


text:
how to convert NonMaxSuppressionV4 to older versions

Hi,
I am training a tensorflow object detection model and am converting it to tensorflow js. Although there is no NonMaxSuppressionV4 in the config.proto file (but has batch_non_max_suppression), I am getting the error
ValueError: Unsupported Ops in the model before optimization
NonMaxSuppressionV4
when trying to use tensorflowjs_converter to convert the trained model into tensorflow js. Is there a way I can convert my checkpoints into saved_model that doesn't convert nonmaxsuppression to v4 (I think v3 is supported in tensorflowjs)? I am using tensorflow 1.15.0, but back tracking to tensorflow 1.13.0 has not helped either.
I would appreciate it if someone explains what is happening, or how I can convert the model into supported tensorflow js operators (after creating tensorflowjs with --skip_op_check or before it at the time of model training or exporting it into saved or frozen model)?
Thank you in advance

@shnamin ,
This issue is related to Tensorflow.js repo, can you please open issue using this link? Also there are already issue's 2540, 2450 being reported for the same refer the same.Thanks!

authors: oanush shnamin