text:
Bulk Inserts and Collection Constraints

I don't think I'm using indexes correclty.
When merging 10,000+ records from a remote database into a local database, I have to use the id (not the $loki id) to make sure the document is not already there.  (if its there I want to merge instead of update).  To do this effeciently, I'm trying to use Loki indices:
let loki = new Loki('testdb');
myCollection = loki.addCollection('myCollection', {indices:['id']})

Then on each insert I do
localRecord = myCollection.findOne({id: remoteRecord.id})
if( localRecord ) {
  // Merge code
} else {
  myCollection.insert( remoteRecord );
}

When I disable the localRecord check and just insert, things are smoking fast!  Only slowed by data fetching/paging.

http://screencast.com/t/H3wsetchhQ

When I enable the findOne check, things slow down a bit:

http://screencast.com/t/gBJQ49yBJwW

When I hack an index to the side that is a big hash of ids mapped to $loki I get really fast again.

http://screencast.com/t/V5R0g7hPt

I think there must be something I don't understand about indexes.
Please advise :)

Interesting... what you are describing is not so much an index but a constraint.
Yea the problem being that on every insert you are invalidating the index and it needs to be rebuilt for the findOne.
Maybe @seriousme (who is exploring similar value hash lookups) can look into a unique constraint functionality using an approach similar to what you described.  This might be something you can define on the collection and let loki handle that for you or just a checkConstraint(indexName, value) type test function.
Our current indexes are designed for optimal search speeds when the ratio of insert/read is either above or below 50%. if you do a lot of inserts and one search it rebuilts index once.  if you do a single insert and a lot of finds it also rebuilts once.  insert/find/insert/find is worse case scenario we might alleviate with a check constraint functionality.

@jrhicks @obeliskos @seriousme this is a discussion we have been having through email and it's good to see that real-life scenarios are highlighting the necessity for optimization in the indexing system.
In the case of @jrhicks 's app, a unique index is ideal. As pointed out by @seriousme , even on sorted data, hashmap lookups will always be faster than binary search.
From this I draw two conclusions:

the need to support the "unique" type of index, much like any major DBMS around
the need to let the user choose the best index for their application, so to achieve the best performance.

A unique index is a relatively trivial piece of work to implement, my first and obvious idea is to create a hashmap whose keys are the values of the property to be indexed and the values are their position in the data array. I am open to suggestions!

Sounds good. Do we also keep $loki id or do we trust user would not have changed the constrained property?

i'd say we keep it. get(int) will work with $loki, and we could extend get to work like get(var, indexName) => gets a record by index name. Opinions?

Sounds good to me!

@techfort You rock! I was able to ditch my hack index, and just check-before-insert a little differently with much better results http://screencast.com/t/JLOAJLPTfj
Basically the only delay is the data fetching/paging!
So given a batch of 1000 data.  I first find all the ones that are already in my db
      for (let d of data) {
        localRec = db[collectionName].findOne({id: d.id})
        if (localRec) {
          localRecords[localRec.id] = localRec;
        }
      }

Then, when I loop over the inserts I just check the disposable localRecords.
        if (localRec[remoteRec.id]) {
          console.log("TODO: Merge");
        } else {
          let r = db[collectionName].insert(remoteRec);
          localRec[r.id] = r;
          loadJob.data.shift();
        }

@jrhicks woah that's fast :) so how do we go about making this a LokiJS facility for everybody?

@techfort I think your on to something with constraints!
In lieu of constraints, maybe a bulkInsert function that returns the records that failed the constraint???
notInsertedRecords = myCollection.bulkInsert(records, {uniqueConstraint: ['id']})

@techfort - I think your correct.  The best thing would be to be able to create a constraint, but also have a bulk insert routine that respected the constraint.  I guess its getting kinda late there - no reply necessary.

@jrhicks this sounds good to me. @obeliskos you good with that?

Sounds good

@techfort I'm timid to update core code.  Maybe after I've been using the product longer.  I do feel interested in helping to contribute to documentation though.  As time permits, I can start a conversation about that in a diferent thread.

@jrhicks no problem, I'll take a look at your implementation on the github project you referenced in the post and credit you in logs/commits/etc. This all looks great, the post /video is impressive!

As already discussed by mail there are various options to deal with indices.  (e.g. exact unique, exact non-unique, ordered (which is the current default) etc).
As shown by the initial posting its rather surprising to users that the first query after insert/update takes so long.  So I'd try to fix that as well by doing binary index updates instead of full index rebuild. The only place to do full index rebuild might be on bulk updates where it is explainable that updating the index  a record at the time might be suboptimal compared to updating the index only once per bulk update. (but even then.. ;-))
I think that the additional feature proposed by @jrhicks  to have:
notInsertedRecords = myCollection.bulkInsert(records, {uniqueConstraint: ['id']})

is a nice addition to that.
However as with traditional databases I'd rather define a uniqueIndex on the DB upfront and then reject the inserts that would break uniqueness.
e.g. something like:
myCollection = loki.addCollection('myCollection', {indices:[{ 'id': 'uniqueIndex' }]})
notInsertedRecords = myCollection.bulkInsert(records)

Where notInserted records would be a list of records with an indication why they were rejected.
(e.g. " 'id' not unique"  etc..)
When doing all this its important to keep in mind that all additional functionality comes at a price of adding complexity and code size. @jrhicks already showed that there is a workaround :-)
Its up to @techfort  to decide which direction he wants to take Loki :-)

I agree with @seriousme, define the constraint upfront and do not offer the option on the bulkInsert.  I think we all agree on this. @techfort

To be most backwards compatible, I think the default

insert

save

Should throw and error if a constrain was violated.  So we may need a pattern to have methods that return errors as apposed to throwing an error.  Ruby uses an exclamation point to distinguish between the two.  Maybe the soft (return errors on failure).  Might be???

tryInsert

trySave

That doesn't.  This communicates clearly that the action may fail and return an error.  Just an idea.

Yes, upfront index declaration is my preferred way too. Also the addition of a property in the options property is the most backward compatible.
As for errors, I'm all for returning results, let's not forget that Collection emits 'error' events, that's also an option to manage errors.
@seriousme @obeliskos @jrhicks index update (as @seriousme points out) is the way to go on binary indexes. Funnily enough, I implemented a (quite hidden) Key Value store class in Loki ( * ), and I used a binary search to establish the position at which a new element should slot into an existing sorted structure/array. What makes this Binary Search interesting is that it returns whether a value was found or not, and the index at which the value is or should be if it were to be inserted. So from there you would simply apply a splice at the specified position and off you go, no need to rebuild the index. You can see that BS function here: https://github.com/techfort/LokiJS/blob/master/src/lokijs.js#L3434
This should save us a bit of time when doing inserts/updates.
Bottom line we're good to go with the upfront declaration as far as I'm concerned.
( * ) the key-value store class is only there because keys can be complex structures such as objects or arrays. This is because I thought maybe at some stage you may want to use Loki objects as keys for values.

this is good. Strange though, this message didn't turn up on github..
On Wed, Mar 25, 2015 at 4:15 PM, Jeffrey Hicks notifications@github.com
wrote:

To be most backwards compatible, I think the default

insert
save
Should throw and error if a constrain was violated. So we may need a
pattern to have methods that return errors as apposed to throwing an error.
Ruby uses an exclamation point to distinguish between the two. Maybe the
soft (return errors on failure). Might be???

tryInsert
trySave
That doesn't. This communicates clearly that the action may fail and
return an error. Just an idea.
—
Reply to this email directly or view it on GitHub
https://github.com/techfort/LokiJS/issues/119#issuecomment-86101189.

great post @seriousme... i think you have great understanding of what can be done with this.
Yes @techfort's function sounds like it should do the trick.  The 'unique' part of this should make it simpler (single position).
I'm sure as this is being implemented we will be able to tell if providing a batch method could offer performance boost.

@jrhicks @seriousme @obeliskos I'm going ahead on the development of exact and unique constraints, stay tuned and happy Easter!

@techfort Awesome!  Happy Easter :)

@jrhicks @seriousme @obeliskos i have completed a very drafty implementation of unique indexes, with inserts and removes working as they should. Haven't benchmarked by() operations yet (by(field, value) is the method to obtain a record straight from the unique index rather than go through the query engine).
Before applying a pass in find() to check if the field is a unique index (and therefore returning the by() result instead of the query result), I would like to address updates, which at the moment are quite tricky (really only in the case of someone trying to update the property associated with a unique index).
just updating.

@techfort - I guess you would have to consume more space to map loki's id to unique id to see when the unique id was changed.  Is Loki id assured to be uneditable?  Could unique indexes be set to non-editable?  Maybe these wouldn't be indexes, they would be primary keys?  Just sorta brainstorming here

@techfort : although semantically the uniqueness is a constraint, technically its an index in which you try to insert a value, if the insertion fails its a constraint violation.
With regards to update: I would use the old value ( find record by I'd ) to determine what to remove from the index and then proceed in the same way as an add.  I would also abstract the index/constraint into classes so the collection could maintain a hashmap to map props to constraints/indices. That way the constraints/indices are cleanly isolated from the add, update, remove methods.

I look forward to this feature.... thanks!
I will try to look over implementation within a few days but i'm sure you and seriousme can make sure it's done right :)

@seriousme here's the problem I'm facing: cloning an object is - at the moment - an option. So imagine I have a collection with an element { username: 'joe', $loki: 1} then the index would look like { 'joe': <a reference to the original object>' }
This means that if i change username of the object to 'jack' the original object is changed, so now the index looks like { 'joe': <an-object-with-username-jack> }
I meant to follow exactly the pattern you illustrated, until i realized the current state of affairs makes that impossible, or convoluted and slow (so much for indexing!).
One quick solution would be to place a very obvious disclaimer in the docs, "if you declare ANY unique indexes then the clone option will activate by default. The objects you deal with are copies of the objects in the collection, and sync-ing is only possible with coll.update(obj)".
As for abstracting - yes absolutely. In fact you had already produced an example of that - I may just very blatantly grab off that - as it also included some of my work so I won't feel too guilty :D
If you or @obeliskos have any further ideas on this let me know.

@techfort: you might be able tot remove 'Joe' from the index, (idx['Joe'] = undefined) and then add Jack to the idx.
Feel free to take my (and your;-)) idx code and use it as you see fit.
In the current setup you either need to link to the array element (not its position in the array!) or store the records $Loki id.
If the idx points to an id then lookup will be a 2 step:

get id from idx hash
get record by id as already available with .get

This method takes a little more effort to resolve but makes serialization  a breeze :-)

So if our id index looks like { arrayIdx : $loki } and your new idx is on username, you have been adding an index for { 'joe' : obj }.  Seems we need an { $loki : 'joe' } index as well?  Either that or set up meta array (or force cloning).
So 2 index maps per unique constraint, or meta, or cloning? Are those options?

@seriousme yeah funny i had it working like that and then i thought it didn't really boost retrieval that much. So i reverted back to pointing directly to the object from the  key in the index. And yes - i also crashed 100mph face-first into linking to the array position and reverted knowing I was the only one to witness that code, but hey - life is about mistakes as much as it is about success so here's me candidly admitting to that blunder :D
@obeliskos I also thought of the two-maps index, should be pretty solid and would remedy force-cloning.
as a note: I think we should definitely return clones (without an option) in v2.

Just curious, have you tried refactoring idIndex from { id : $loki } to { $loki : object }  (yet?)
I just noticed our benchmarks no longer work under node?  I guess because of the refactoring to html benchmarks?   Should we keep two versions of that source, one for node and one for html.  I can see we will be needing to reference and extend benchmarking soon.

@obeliskos actually no i haven't. I'll create a separate branch for this as it's going to be too dangerous to play around with in master, then try the refactoring you suggested. I'll also look at those benchmarks.

Oh.. yea I guess file compatibility issues as well.  I would probably rename benchmark.js to benchmark_html.js and restore old benchmark.js.  I can do that later or let you do it if you prefer differently.

@obeliskos actually Dave if you don't mind doing that i'll stay on indexes so we don't trip each other. Thanks!

@obeliskos @jrhicks @seriousme i have added a first draft of the unique index abstracted into a class. First tests pass, I am yet to test updates and removes but it works nice for now.

Thats awesome @techfort :)  You rock.

I'm showing 1M+ ops/sec :)

I rest my case  ;-)

well technically techfort rested this case, you just kind of poked around it :)

actually - could you re-run those benchmarks? i "released the breaks" on
the by() method (as in, i'm not checking for the existence of a particular
unique index, if you try to call a by() on a non-existent index you get an
error)
should be a good bit 10% faster! my dev machine is a pentium, and got by()
running at 500k ops/s but it's a pretty crappy processor. Any decent
processor (like an i7) should hit 1.1M+ now.
On Thu, Apr 9, 2015 at 10:44 AM, obeliskos notifications@github.com wrote:

well technically techfort rested this case, you just kind of poked around
it :)
—
Reply to this email directly or view it on GitHub
https://github.com/techfort/LokiJS/issues/119#issuecomment-91176414.

Ok my yoga2pro (i5) gets 1.1M - 1.2M which is great.  Seeing this evolve has been very enlightening of the specifics of javascript indexing.  Good work @techfort (and @seriousme) :)

With the development of #144 , UniqueIndex and SortedIndex, this issue seems to have been split and taken care of in different features. Closing this, feel free to re-open or open separate issue to address whatever has not been addressed by the various indexes.

Hey guys where do I find this class / docs or commit on the proper way to do this?

Great thread. I'm also curious how bulk inserts worked out. It looks like maybe one way to optimize bulk inserts is to call .insertOne with the 2nd parameter set to true. Do I need to emit an event after that?

I also noticed that if Collection.prototype.update is passed an array, adaptiveBinaryIndexes are disabled during the duration of the updates, then ensureAllIndexes() is called to update the indexes. Pretty cool.
It doesn't look like that same optimization is applied to Collection.prototype.insert and I'm just curious if there's a specific reason for that.
Thanks for making this library public, it's freaking awesome! 🙏

authors: aguynamedben ericmuyser jrhicks obeliskos seriousme techfort

repo: techfort/LokiJS


text:
Cannot resolve module 'fs'

I'm not sure why but I've been getting it everytime when I install loki from npm install and initialise it with the following lines:
import Loki from 'lokijs'
const ldb = new Loki('example.db')

@najamkhn, just in case, are you trying to build a web-client app (= to be running in the browser) using LokiJS with the webpack / babel toolchain (or alike)?

Yes using it with webpack and building web app.

Okay, that's what I suspected based on the specific issue that you reported. But I needed to double-check my guess with you first. What you need to do is, to add the following configuration section and parameters into your webpack.config.js file:
node: { fs: 'empty' }

@najamkhn Also, depending on your specific configuration and other modules that you use in your web app, you may need to extend that section above to something like:
node: {
      fs: 'empty',
      net: 'empty',
      tls: 'empty',
      'crypto': 'empty'
},

@najamkhn Finally, in the case if you need a piece of Webpack's official documentation on that part, you can refer to: https://webpack.github.io/docs/configuration.html#node

will you extract it outside ?

@jetiny, extract what? I am not sure if I understand your question. Please, elaborate.

Thanks to all who answered, I believe this is resolved.  In future releases we may extract the (node) filesystem adapter out into its own module.

For what it's worth, I was having this problem with Angular 5 and Angular CLI 1.5.0 when running unit tests, but not when doing a regular build / serve.
While the webpack.config that angular cli was using already had  fs: 'empty' set, the problem was that karma was using different settings for webpack.  I fixed this issue by putting the following in my karma.config.js
    webpack: {
      node: {
        fs: 'empty'
      }
    }

Would someone give me some guidance as to how to fix the configuration in Rollup instead of Webpack? I also get the fs is not defined error. I tried adding
node: {
  fs: "empty",
  net: "empty",
  tls: "empty",
  crypto: "empty",
}

at the root of the default exported object of the rollup.config.js file with no success...

Thanks @davidsavoie1 Big time-saver, I was going to give up.
I wished LokiJs would be more browser-friendly out of the box, or have a lokijs-browser. Anyway, the functionalities look to be what I am looking for.

source: gharchive/issue

authors: VladimirTechMan davidsavoie1 jamesmikesell jeremychone jetiny najamkhn obeliskos

license: mit

license_type: permissive

license_source: bigquery


text:
no notification re-order point

[x] no notification passed for item quantity going to below of Re order level

[x] Even though multiple items’ quantity below re-order level, no-need multiple notifications.

[x] Notification generated mid night of selected time zone.

[x]  Show generated date and time with each notification.

issue fixed!

repo: techgebra/polosys-books

license: Unlicense

license_source: github-api


text:
`colorize': no implicit conversion of Hash into String

'rails generate tabs' creates tabulous.rb  but fails with a "colorize': no implicit conversion of Hash into String" error.  After editing tabulous.rb and inserting <%= tabs %> gives "undefined local variable or method tabs'" error.

tabulous depends on the colored gem.  It is likely that it is interfering with the colorize gem.  For example, see the discussion here: https://github.com/forward3d/cap-ec2/issues/39
If you cannot find an easy way around this, maybe tabulous needs to depend on the colorize gem (or not use colorized output at all, which is not strictly necessary).  Let me know if you can find an easy workaround or whether you think tabulous should be changed.

authors: c-ro techiferous

repo: techiferous/tabulous


text:
🍚 8월 1주차 식단 추가

8월 1주차 식단 추가합니다.

식단표는 식당 앞에 있습니다.

@all-contributors please add techinpark for doc

source: gharchive/pull-request

repo: techinpark/pangyo-smilecook

license: MIT


text:
account settings - searchable toggle shows off instead of on

works ok on mastodon.social but older builds (iosdev.space, universeodon.com) are not including indexable in Account all the time

could work around by making another verifyCredentials query in the settings editor

repo: technicat/fedicat


text:
feat: remove accuracy circle of geolocation

This PR removes the accuracy circle when geolocating yourself and the enableHighAccuracy option as an attempt to prevent the moving around of the location when the device is not moving. Needs to be tested, that's why this PR is only a draft.

To see the specific tasks where the Asana app for GitHub is being used, see below:

https://app.asana.com/0/0/1204430705532497

@ff6347 what needs to be done to make mapbox work in the preview? I added the preview url to the access token allowed urls but I still get at 403 forbidden from mapbox

repo: technologiestiftung/giessdenkiez-de


text:
How do you refresh the PDF file?

I'm building a screen in SwiftUI that lets me update parameters for the PDF report such as margins, orientation, row heights, etc. Basically putting a UI to many of the features of TPPDF into my app.
I've got the button, just like in the demo app to regenerate the PDF file. But the screen doesn't update when I re-run the generatePDF() function call. I've stepped through the debugger and I can see that my parameters are updated in the report building factory, but the screen just doesn't update. I looked around for a refresh or reload function, but I didn't find one.
Any suggestions?
Also I'm wondering if I really need to regenerate the whole report for some of these kinds of changes or can I just update the parameters on an existing PDFDocument / PDFTable? It could be much faster than looping through all of my model objects and rebuilding all the cells, etc.

The only way so far I've managed to refresh the report is to change the filename in the URL before generating the report.

Is there a better way?

A couple of things:

Make sure to create a new PDFGenerator every time. As it keeps a state inside the object, it might interfere if you call generate multiple times.
I do not know how you are displaying the PDF document in SwiftUI, but if you are reading it from the output URL, overwriting the same file content will most likely not trigger a SwiftUI view update, therefore not updating your screen. By writing it to a different file every time, you are also changing the SwiftUI state, which triggers the update
PDFDocument is a mutable object used as a description of the final PDF document, which is not modified during generation. This means you can use all the provided functions to modify it, and simply put the updated in a generator. If you need more advanced access to the data structure inside PDFDocument, I would recommend creating a fork, and adding the functionality you need. Feel free to open up a PR afterwards, and we can consider adding it to the main repo.

I am creating a new PDFGenerator every time I generate the report. I used your example SwiftUI implementation of your generatePDF() method as my starting point.
Since I'm using your sample SwiftUI code, I'm displaying the PDF document the exact same way you are in the sample app.
I'll look into the PDFDocument class to see what I'm able to modify there. But I guess I was thinking of things like changing page orientation or adjusting PDFTable cell alignments without having to regenerate the whole report by looping through all my data and rebuilding it.

For the refresh issue I just increment a counter now and append the counter to the end of the filename before the .pdf extension. Just before that I delete the previous report generation file so I don't clutter up the temp folder with all these PDF files.

Alright makes sense.
Another option would be creating a hash for the PDF file and store it in a @State, basically triggering a UI update only if the file content has changed.

I'm going to close this because now when I generate a new PDF file, I first delete the previous generated PDF file, then create the new one with an incrementing generation number. This causes the report to refresh.

authors: brendand philprime

repo: techprimate/TPPDF


text:
Fix footer height calculation

Fixes footer height calculation.
Without this fix footer height is equal to header height. That means that if you don't have a header but do have a footer your content will overlap the footer because header height is zero.

Hi, thanks for your contribution.
After a quick look I need you to provide an example to reproduce the issue, as your fix seems wrong:
The method you changed calculates the distance from the top edge, therefore it is paper_height - footer_height.
I could be wrong, that's why an example would be great

Hello @philprime,
Thanks for looking at this! The issue is reproducible when you have a table view that is rendered over a footer. To reproduce it you can paste this method to the HeaderFooterExampleFactory.swift:
func generateDocument() -> [PDFDocument] {
    let document = PDFDocument(format: .a4)

    // footer will be offset by the height that is equal to the header height
    document.add(.headerRight, textObject: PDFSimpleText(text: "Header right 1"))

    document.add(.footerLeft, textObject: PDFSimpleText(text: "Footer Left 1"))
    document.add(.footerLeft, textObject: PDFSimpleText(text: "Footer Left 2"))
    document.add(.footerLeft, textObject: PDFSimpleText(text: "Footer Left 3"))
    document.add(.footerLeft, textObject: PDFSimpleText(text: "Footer Left 4"))

    var items = [[String?]]()

    for _ in 0..<200 {
        items.append(["Lorem Ipsum is simply dummy text of the printing and typesetting industry."])
    }

    let table = PDFTable(rows: items.count, columns: 1)

    table.content = items
    table.rows.allCellsStyle = PDFTableCellStyle(colors: (fill: .clear, text: .black))
    table.widths = [1]

    table.padding = 10.0

    document.add(table: table)

    return [document]
}

Here's how it looks without the fix:

And this is with the fix:

Thanks a lot for the example! Helps massively.
I will look into it a little further as I am still confused where my thought is wrong.

I took a closer look, and you were right. The calculation itself doesn't make any sense and seems to be a copy paste error. Anyways I added a suggestion so the method includes all paddings, spacings etc.
If you approve my recommendation we can merge it into develop (I will change the branch)

Can you please apply my suggested change and update the PR? I will merge it afterwards and release an update

Relates to #249

Amazing, thanks for reviewing this PR!

Don't know why the code analysis is failing. Codacy doesn't provide much context.

@philprime are you planning to release #205 too?

Don't worrry about Codacy, looks like some project linking issue. They changed their GitHub integration a little while ago so it might be broken.
Yes I am planning to release #223 too

authors: lpeancovschi philprime


text:
🛑 zdrofit.pl is down

In 3f16c45, zdrofit.pl (https://www.zdrofit.pl) was down:

HTTP code: 500
Response time: 4249 ms

Resolved: zdrofit.pl is back up in 2967b5c after 17 minutes.

repo: techsupp/uptime


text:
Separators replacement

Hi! Could be possible to setup which separators to use?
For example if I'd like to use something like: myapp.exe cmd1 -par=var1... Where "-" replaces the default "/" and "=" replaces the default ":"
Do you think it could be added as a feature?

Other command patterns are planned. I'll publish a new version in the coming days.

Ok! Thank you very much!

I just released a new package. To use myapp.exe cmd1 -par=var1 style command setup your console as following:
NCommands commands = new NCommands();
commands.Context.Configuration.Notation = ParserNotation.Unix;
commands.RunConsole(args);

Thank you very much!
I will test it in the next days!

authors: gandaldf tectil

repo: tectil/NCommand


text:
Help Enabling virtual scroll wheel on Sierra.

I have a Logitech Marble Mouse, and I love it too much to switch to anything else, but having a scroll wheel is incredibly useful on a mac. Currently, I'm on El Capitan because Karabiner has a virtual scroll wheel feature where I can hold down CTL + CMD and then my trackball turns into a scroll wheel. It's the only thing preventing me from upgrading to Sierra.
I am a capable programmer, so I'd like to help development by enabling this one feature on Sierra, but I'd need some guidance. Can anyone help me enable virtual scroll wheel on Sierra?

I believe "Hammerspoon" can do it, just google "hammerspoon scroll wheel". Many people switch to hammerspoon after Sierra due to Karabiner is not supported in Sierra.

Thanks!

Following up on this:
I have been able to successfully reproduce the Karabiner Virtual Scroll using Hammerspoon. For those who are interested, here is my hammerspoon code:
-- HANDLE SCROLLING
local oldmousepos = {}
local scrollmult = -4	-- negative multiplier makes mouse work like traditional scrollwheel

mousetap = hs.eventtap.new({5}, function(e)
	oldmousepos = hs.mouse.getAbsolutePosition()
	local mods = hs.eventtap.checkKeyboardModifiers()
	if mods['ctrl'] and mods['cmd'] then
		-- print ("will scroll")
		local dx = e:getProperty(hs.eventtap.event.properties['mouseEventDeltaX'])
		local dy = e:getProperty(hs.eventtap.event.properties['mouseEventDeltaY'])
		local scroll = hs.eventtap.event.newScrollEvent({dx * scrollmult, dy * scrollmult},{},'pixel')
		scroll:post()

		-- put the mouse back
		hs.mouse.setAbsolutePosition(oldmousepos)

		-- return true, {scroll}
		return true
	else
		return false, {}
	end
	-- print ("Mouse moved!")
	-- print (dx)
	-- print (dy)
end)
mousetap:start()

If anyone is interested:
I have adopted the above answer, which allows me to scroll vertically/horizontally using my Logitech M570 (Trackball), while holding a mouse button (instead of ctrl + cmd)
-- HANDLE SCROLLING
local oldmousepos = {}
-- positive multiplier (== natural scrolling) makes mouse work like traditional scrollwheel
local scrollmult = 4

-- The were all events logged, when using `{"all"}`
mousetap = hs.eventtap.new({0,3,5,14,25,26,27}, function(e)
	oldmousepos = hs.mouse.getAbsolutePosition()
	local mods = hs.eventtap.checkKeyboardModifiers()
    local pressedMouseButton = e:getProperty(hs.eventtap.event.properties['mouseEventButtonNumber'])

    -- If OSX button 4 is pressed, allow scrolling
    local shouldScroll = 3 == pressedMouseButton
    if shouldScroll then
		local dx = e:getProperty(hs.eventtap.event.properties['mouseEventDeltaX'])
		local dy = e:getProperty(hs.eventtap.event.properties['mouseEventDeltaY'])
		local scroll = hs.eventtap.event.newScrollEvent({dx * scrollmult, dy * scrollmult},{},'pixel')
		scroll:post()

		-- put the mouse back
		hs.mouse.setAbsolutePosition(oldmousepos)

		return true, {scroll}
	else
		return false, {}
	end
	-- print ("Mouse moved!")
	-- print (dx)
	-- print (dy)
end)
mousetap:start()

I had the same problem, but I prefer to use right mouse button to init scrolling, so I used your code and modified it to perform scroll during right mouse drag. There was some problems with deferring menuOpen on rightMouseDown, but I ended with this script, which works great.
Note: this will make right mouse dragging impossible, but who needs it anyway?
-- HANDLE SCROLLING

local deferred = false

overrideRightMouseDown = hs.eventtap.new({ hs.eventtap.event.types.rightMouseDown }, function(e)
    --print("down"))
    deferred = true
    return true
end)

overrideRightMouseUp = hs.eventtap.new({ hs.eventtap.event.types.rightMouseUp }, function(e)
    -- print("up"))
    if (deferred) then
        overrideRightMouseDown:stop()
        overrideRightMouseUp:stop()
        hs.eventtap.rightClick(e:location())
        overrideRightMouseDown:start()
        overrideRightMouseUp:start()
        return true
    end

    return false
end)

local oldmousepos = {}
local scrollmult = -4	-- negative multiplier makes mouse work like traditional scrollwheel
dragRightToScroll = hs.eventtap.new({ hs.eventtap.event.types.rightMouseDragged }, function(e)
    -- print("scroll");

    deferred = false

    oldmousepos = hs.mouse.getAbsolutePosition()

    local dx = e:getProperty(hs.eventtap.event.properties['mouseEventDeltaX'])
    local dy = e:getProperty(hs.eventtap.event.properties['mouseEventDeltaY'])
    local scroll = hs.eventtap.event.newScrollEvent({dx * scrollmult, dy * scrollmult},{},'pixel')

    -- put the mouse back
    hs.mouse.setAbsolutePosition(oldmousepos)

    return true, {scroll}
end)

overrideRightMouseDown:start()
overrideRightMouseUp:start()
dragRightToScroll:start()

Thanks again! Worked perfectly!
On Wed, Jan 31, 2018 at 2:43 PM, Joel O'Neill joelbassist@gmail.com wrote:

Thanks I'll try this!...
On Jan 31, 2018 7:42 AM, "Randy Dod" notifications@github.com wrote:

greazyfingaz, the last time I checked (Jan 11, 2018), Karabiner Elements
cannot do this. My code is for Hammerspoon. Use jeffmikels' code above (his
2/21/17 post) and replace his "if mods['ctrl'] and mods['cmd'] then" line
with this line instead:
if mods['shift'] then
As far as I can tell, Hammerspoon does not have the ability to
distinguish between the left and right shift keys, so the code will make
scrolling work with either shift key.
http://www.hammerspoon.org/go can help you get started using it.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tekezo/Karabiner/issues/814#issuecomment-361971600,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AiU9U0IrG3d2Sqoo6fjF8gbP7W-0E3GHks5tQImsgaJpZM4L9g_5
.

One more question for you:
Is there some kind of command line I could add to this so that if Im
holding down shift + another modifier or key it will NOT scroll? (Pro Tools)
On Jan 31, 2018 4:16 PM, "Joel O'Neill" joelbassist@gmail.com wrote:

Thanks again! Worked perfectly!
On Wed, Jan 31, 2018 at 2:43 PM, Joel O'Neill joelbassist@gmail.com
wrote:

Thanks I'll try this!...
On Jan 31, 2018 7:42 AM, "Randy Dod" notifications@github.com wrote:

greazyfingaz, the last time I checked (Jan 11, 2018), Karabiner Elements
cannot do this. My code is for Hammerspoon. Use jeffmikels' code above (his
2/21/17 post) and replace his "if mods['ctrl'] and mods['cmd'] then" line
with this line instead:
if mods['shift'] then
As far as I can tell, Hammerspoon does not have the ability to
distinguish between the left and right shift keys, so the code will make
scrolling work with either shift key.
http://www.hammerspoon.org/go can help you get started using it.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/tekezo/Karabiner/issues/814#issuecomment-361971600,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AiU9U0IrG3d2Sqoo6fjF8gbP7W-0E3GHks5tQImsgaJpZM4L9g_5
.

I encourage you to dig through Hammerspoon's documentation, as that's what I would have to do to figure this out. In other words, I don't know how. A brief reading of their examples reveals that apparently it cannot discern between the left and right shift keys or other dual keys. It can interface w/ Karabiner (see http://www.hammerspoon.org/go/#karabinerurl), which does not work in Sierra, but it does not mention Karabiner Elements.
Looking through their docs, http://www.hammerspoon.org/docs/, these functions look the most promising. hs.eventtap, hs.eventtap.event, hs.expose, hs.keycodes
You might post your question by itself to see if someone else here would know how to do this.

@greazyfingaz Yes. Look through the code above. You want to add an "if" statement that checks if specific modifier keys are down, and if they are down, sets "shouldScroll" to false.
Are you familiar with how to write code?

@randydod-teamunify thanks for the hammerspoon modification!
For future users, I found out, through experimentation, that the left small mouse button is actually index "2" on my trackman marble.

getting error
2018-07-10 10:16:44: 10:16:44 ERROR:   LuaSkin: hs.shutdownCallback: attempt to call a nil value stack traceback:
with this code https://github.com/tekezo/Karabiner/issues/814#issuecomment-281422447
hammerspoon - Version 0.9.66 (0.9.66)

Leaving my script for scrolling with mouse wheel pressed. Mouse wheel button in my case is 2. The button is configurable on the second line of code.
Natural scrolling on both axes:
-- HANDLE SCROLLING WITH MOUSE BUTTON PRESSED
local scrollMouseButton = 2
local deferred = false
local oldmousepos = {}
local scrollmult = -4	-- negative multiplier makes mouse work like traditional scrollwheel

dragMidToScroll = hs.eventtap.new({ hs.eventtap.event.types.otherMouseDragged }, function(e)
    local pressedMouseButton = e:getProperty(hs.eventtap.event.properties['mouseEventButtonNumber'])
    -- print ("pressed mouse " .. pressedMouseButton)
    if scrollMouseButton == pressedMouseButton
        then
            -- print("scroll");
            deferred = false
            oldmousepos = hs.mouse.getAbsolutePosition()
            local dx = e:getProperty(hs.eventtap.event.properties['mouseEventDeltaX'])
            local dy = e:getProperty(hs.eventtap.event.properties['mouseEventDeltaY'])
            local scroll = hs.eventtap.event.newScrollEvent({-dx * scrollmult, dy * scrollmult},{},'pixel')
            -- put the mouse back
            hs.mouse.setAbsolutePosition(oldmousepos)
            return true, {scroll}
        else
            return false, {}
        end
end)

dragMidToScroll:start()

Thanks, I updated the script to fix that

Yes, the scripts above indicate how to send a scroll event. You need to
wrap it in a hs.timer.doEvery function call , and you need to set a hotkey
to start and stop that timer.
On Thu, Nov 8, 2018, 12:21 AM Madd0g notifications@github.com wrote:

I see a lot of hammerspoon masters here. Does anyone know how to replicate
the firefox auto-scroll feature with HS? I want to press a button and have
the screen scroll down until I press it again, without holding anything, is
it possible?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/tekezo/Karabiner/issues/814#issuecomment-436876883,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADV_ZEIk2myx0v6scuC0tmS3nWjNxGqUks5us79ugaJpZM4L9g_5
.

@jeffmikels - thanks, I'll try, I was a bit worried about jerkiness from using a timer, but I guess I'll tweak the timer/scroll values until it looks ok. Thanks

For anyone who finds this thread via Google that is a first timer at hammerspoon:

Click "open config" in hammerspoon menu
Paste one of the above scripts into init.lua
In hammerspoon, click "show console"
Click "reload config"
If you get hs.eventtap:start() Unable to create eventtap. Is Accessibility enabled? follow the accessibility directions in the faq http://www.hammerspoon.org/faq/

Thanks for posting this code everybody!

FYI, on the Logitech Trackman, using the mini left button, I used @randydod-teamunify's code with these settings:

Logitech Control Center: set mini left button to Advanced Click, button no. 5
In the code: local shouldScroll = 4 == pressedMouseButton, i.e. button no. 4

No idea why/how those are mixed up. I suppose it's off-by-one somewhere ;-)

Huge thanks to everyone, this thread makes a normal mouse/trackball useable on a mac.
I wrote another version to help scrolling through big files and added a dead-zone for clicking (in other versions if you moved the mouse even 1 pixel between pressing and releasing the button it wouldn't fire the click):

when you press and release the wheel it acts like  normal click wheel (middle mouse button in my case),
when you press and drag it keeps scrolling in the direction of the cursor until you release the button. The further away you drag while pressed the faster it will scroll:

------------------------------------------------------------------------------------------
-- AUTOSCROLL WITH MOUSE WHEEL BUTTON
-- timginter @ GitHub
------------------------------------------------------------------------------------------

-- id of mouse wheel button
local mouseScrollButtonId = 2

-- scroll speed and direction config
local scrollSpeedMultiplier = 0.1
local reverseScrollDirection = false
local mouseScrollTimerDelay = 0.01

-- circle config
local mouseScrollCircleRad = 15
local mouseScrollCircleDeadZone = 5

------------------------------------------------------------------------------------------

local mouseScrollCircle = nil
local mouseScrollTimer = nil
local mouseScrollStartPos = 0
local mouseScrollDragPosX = nil
local mouseScrollDragPosY = nil

overrideScrollMouseDown = hs.eventtap.new({ hs.eventtap.event.types.otherMouseDown }, function(e)
    -- uncomment line below to see the ID of pressed button
    --print(e:getProperty(hs.eventtap.event.properties['mouseEventButtonNumber']))

    if e:getProperty(hs.eventtap.event.properties['mouseEventButtonNumber']) == mouseScrollButtonId then
        -- remove circle if exists
        if mouseScrollCircle then
            mouseScrollCircle:delete()
            mouseScrollCircle = nil
        end

        -- stop timer if running
        if mouseScrollTimer then
            mouseScrollTimer:stop()
            mouseScrollTimer = nil
        end

        -- save mouse coordinates
        mouseScrollStartPos = hs.mouse.getAbsolutePosition()
        mouseScrollDragPosX = mouseScrollStartPos.x
        mouseScrollDragPosY = mouseScrollStartPos.y

        -- start scroll timer
        mouseScrollTimer = hs.timer.doAfter(mouseScrollTimerDelay, mouseScrollTimerFunction)

        -- don't send scroll button down event
        return true
    end
end)

overrideScrollMouseUp = hs.eventtap.new({ hs.eventtap.event.types.otherMouseUp }, function(e)
    if e:getProperty(hs.eventtap.event.properties['mouseEventButtonNumber']) == mouseScrollButtonId then
        -- send original button up event if released within 'mouseScrollCircleDeadZone' pixels of original position and scroll circle doesn't exist
        mouseScrollPos = hs.mouse.getAbsolutePosition()
        xDiff = math.abs(mouseScrollPos.x - mouseScrollStartPos.x)
        yDiff = math.abs(mouseScrollPos.y - mouseScrollStartPos.y)
        if (xDiff < mouseScrollCircleDeadZone and yDiff < mouseScrollCircleDeadZone) and not mouseScrollCircle then
            -- disable scroll mouse override
            overrideScrollMouseDown:stop()
            overrideScrollMouseUp:stop()

            -- send scroll mouse click
            hs.eventtap.otherClick(e:location(), mouseScrollButtonId)

            -- re-enable scroll mouse override
            overrideScrollMouseDown:start()
            overrideScrollMouseUp:start()
        end

        -- remove circle if exists
        if mouseScrollCircle then
            mouseScrollCircle:delete()
            mouseScrollCircle = nil
        end

        -- stop timer if running
        if mouseScrollTimer then
            mouseScrollTimer:stop()
            mouseScrollTimer = nil
        end

        -- don't send scroll button up event
        return true
    end
end)

overrideScrollMouseDrag = hs.eventtap.new({ hs.eventtap.event.types.otherMouseDragged }, function(e)
    -- sanity check
    if mouseScrollDragPosX == nil or mouseScrollDragPosY == nil then
        return true
    end

    -- update mouse coordinates
    mouseScrollDragPosX = mouseScrollDragPosX + e:getProperty(hs.eventtap.event.properties['mouseEventDeltaX'])
    mouseScrollDragPosY = mouseScrollDragPosY + e:getProperty(hs.eventtap.event.properties['mouseEventDeltaY'])

    -- don't send scroll button drag event
    return true
end)

function mouseScrollTimerFunction()
    -- sanity check
    if mouseScrollDragPosX ~= nil and mouseScrollDragPosY ~= nil then
        -- get cursor position difference from original click
        xDiff = math.abs(mouseScrollDragPosX - mouseScrollStartPos.x)
        yDiff = math.abs(mouseScrollDragPosY - mouseScrollStartPos.y)

        -- draw circle if not yet drawn and cursor moved more than 'mouseScrollCircleDeadZone' pixels
        if mouseScrollCircle == nil and (xDiff > mouseScrollCircleDeadZone or yDiff > mouseScrollCircleDeadZone) then
            mouseScrollCircle = hs.drawing.circle(hs.geometry.rect(mouseScrollStartPos.x - mouseScrollCircleRad, mouseScrollStartPos.y - mouseScrollCircleRad, mouseScrollCircleRad * 2, mouseScrollCircleRad * 2))
            mouseScrollCircle:setStrokeColor({["red"]=0.3, ["green"]=0.3, ["blue"]=0.3, ["alpha"]=1})
            mouseScrollCircle:setFill(false)
            mouseScrollCircle:setStrokeWidth(1)
            mouseScrollCircle:show()
        end

        -- send scroll event if cursor moved more than circle's radius
        if xDiff > mouseScrollCircleRad or yDiff > mouseScrollCircleRad then
            -- get real xDiff and yDiff
            deltaX = mouseScrollDragPosX - mouseScrollStartPos.x
            deltaY = mouseScrollDragPosY - mouseScrollStartPos.y

            -- use 'scrollSpeedMultiplier' and math.floor - scroll event accepts integers only
            deltaX = math.floor(deltaX * scrollSpeedMultiplier)
            deltaY = math.floor(deltaY * scrollSpeedMultiplier)

            -- reverse scroll if 'reverseScrollDirection' set to true
            if reverseScrollDirection then
                deltaX = deltaX * -1
                deltaX = deltaY * -1
            end

            -- send scroll event
            hs.eventtap.event.newScrollEvent({-deltaX, deltaY}, {}, 'pixel'):post()
        end
    end

    -- restart timer
    mouseScrollTimer = hs.timer.doAfter(mouseScrollTimerDelay, mouseScrollTimerFunction)
end

-- start override functions
overrideScrollMouseDown:start()
overrideScrollMouseUp:start()
overrideScrollMouseDrag:start()

------------------------------------------------------------------------------------------
-- END OF AUTOSCROLL WITH MOUSE WHEEL BUTTON
------------------------------------------------------------------------------------------

keywords: mac mouse autoscroll hammerspoon wheel pressed

did anyone figure out how to have the scrollwheel scrolling feel more natural (scroll bigger distances, maybe accelerate a little bit?) I just can't shake the muscle memory of using the scrollwheel and it's a pretty horrible experience.

Thank you all for this.
Could someone explain how to deactivate scrolling when a specific application is active?

kek, the best answer (right mouse scroll) author deleted his account. It is the only way I found to do it on macosx without paying money for 3d party software.

You can change one of the above variants and make them use right mouse button for scroll. The button code is declared as variable.

authors: Burdu Madd0g Marviel MorphoV banderlog dunkarooftop greazyfingaz ianjennings jeffmikels mansoff randydod-teamunify reinder42 timginter vclav weitzj

repo: tekezo/Karabiner

license: unlicense


text:
🛑 Tekhnee Cloud is down

In 4e060bc, Tekhnee Cloud ($CLOUD_BENCHMARK_LINK) was down:

HTTP code: 500
Response time: 766 ms

Resolved: Tekhnee Cloud is back up in 490914a.

repo: tekhnee/status


text:
Review

test
https://github.com/tekonfo/deepequal2cmp/blob/master/deepequal2cmp_test.go#L15
*testing.T を引数にとってあげるとエラーが起きたときにtesting.T.Fatal 関数でテストを終えられるので活用したいです。
更に testing.T.Helper 関数でこの関数はヘルパー関数だと言うのを明示すると、テストでコケたときに見やすくなります。

@knsh14
返信が遅くなり申し訳ありません、レビューありがとうございます！！！
ご指摘いただいた点は、改善していこうと思います！！！


text:
Variable Substitution Not Working in Tasks

Expected Behavior
Pipeline parameters substituting when using $(params.<name>) syntax as specified here.
Task File:
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: hello
spec:
  params:
    - name: person
      description: Name of person to greet
      type: string
  steps:
    - name: say-hello
      image: docker.io/library/ubuntu
      command:
        - /bin/bash
      args: ['-c', 'echo Hello $(params.person)']

CLI command: tkn task start --showlog -p person=World hello
Expected Output:
[say-hello] Hello World
Actual Behavior
Pipeline parameters not substituting.
Actual Output:
PS > tkn task start --showlog -p person=World hello
TaskRun started: hello-run-qf6qn
Waiting for logs to be available...
[say-hello] {"level":"info","ts":1614028051.8906758,"logger":"fallback-logger","caller":"logging/config.go:69","msg":"Fetch GitHub commit ID from kodata failed: \"KO_DATA_PATH\" does not exist or is empty"}
[say-hello] /bin/bash: params.person: command not found

Steps to Reproduce the Problem

Create new task containing params property
Run task via tkn CLI tkn task start --showlog -p person=World hello

Note: The issue occurs regardless task is invoked via tkn CLI or via taskRun YAML file.
In addition to the task file above, I have also attempted to map the variable into the shell script like this:
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: ubuntu-test
  namespace: tekton-pipelines
spec:
  params:
    - name: var
      type: string
  steps:
    - env:
        - name: VAR
          value: $(params.var)
      image: ubuntu
      name: ''
      resources: {}
      script: |
        #!/usr/bin/env bash
        echo "Hello from Bash $VAR!"

When I run this task, I see the following ouput:
[unnamed-0] Hello from Bash $(params.var)!
Additional Info

Kubernetes version: v1.19.0
Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0+1833054", GitCommit:"183305424c91636d683dfb4a379204c313976be2", GitTreeState:"clean", BuildDate:"2021-01-22T14:47:21Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

Tekton Pipeline version:

Client version: 0.16.0
Pipeline version: v0.10.1
Triggers version: v0.11.2
Dashboard version: v0.14.0

Pipeline version: v0.10.1

I don't think v1beta1 resources were supported this far back - I think they were added in 0.11.0.  I'm kinda surprised a Task with tekton.dev/v1beta1 was accepted by the api server!
Here's the output running the same task on a later version of Pipelines (current HEAD of main branch):
$ tkn version
Client version: 0.15.0
Pipeline version: devel

$ tkn task start --showlog -p person=World hello
TaskRun started: hello-run-w656d
Waiting for logs to be available...
[say-hello] Hello World

So several options:

Upgrade to later pipelines version (0.21.0 was just released)
or
Change your variables from $(params.person) to $(inputs.params.person).  This was the param variable format in 0.10.1.

Thank you, this was it! My mistake was following an outdated tutorial for the Tekton installation using
https://storage.googleapis.com/tekton-releases/latest/release.yaml. It turns out this "latest" release is indeed 0.10.1. After installing the latest pipelines release, the task is working as expected.

@rjackowens Interesting info, thanks a lot!  Which tutorial did you follow?

@rjackowens Interesting info, thanks a lot! Which tutorial did you follow?

No problem! This is the tutorial I followed. It was published in mid-2019 which should have raised a red flag... I think the "latest" tag on the CDN link gave me a false sense of assurance.

authors: rjackowens sbwsg

repo: tektoncd/pipeline


text:
clean up reconcile unit test to use YAML parser

Changes

/kind cleanup
Submitter Checklist
As the author of this PR, please check off the items in this checklist:

[ ] Docs included if any changes are user facing
[x] Tests included if any functionality added or changed
[x] Follows the commit message standard
[x] Meets the Tekton contributor standards (including
functionality, content, code)
[x] Release notes block below has been filled in or deleted (only if no user facing changes)

Release Notes

NONE

refactor TestReconcileOnCancelledRunFinallyPipelineRunWithFinalTask
Pipeline object to use MustParsePipeline
Continued work on https://github.com/tektoncd/pipeline/issues/4610
Signed-off-by: jbpratt jbpratt78@gmail.com

thank you @jbpratt 👍
/ok-to-test

/lgtm

weird error, reported here ...
/test pull-tekton-pipeline-alpha-integration-tests

/retest

/test pull-tekton-pipeline-integration-tests

/test pull-tekton-pipeline-integration-tests

authors: jbpratt pritidesai


text:
Add configmap for tracing config

Changes

This PR moves the configuration of tracing from environment variables into a configmap.
Submitter Checklist
As the author of this PR, please check off the items in this checklist:

[x] Has Docs if any changes are user facing, including updates to minimum requirements e.g. Kubernetes version bumps
[x] Has Tests included if any functionality added or changed
[x] Follows the commit message standard
[x] Meets the Tekton contributor standards (including functionality, content, code)
[ ] Has a kind label. You can add one by adding a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
[ ] Release notes block below has been updated with any user facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings). See some examples of good release notes.
[ ] Release notes contains the string "action required" if the change requires additional action from users switching to the new release

Release Notes
NONE

/kind misc

thanks @kmjayadeep! would you mind adding some more context to your commit message/PR description on the reason for this change? Also, this likely needs a release note.

I have updated the description and release notes. I think it is clear enough. Feel free to update it directly as required.

No problem with merging
/approve

authors: kmjayadeep lbernick


text:
Refactoring Token

Fixes #280
Description
This is just a proposal for better implementation of the Token concept, currently represented as a couple of interfaces with repeated concepts in different attributes like address-contract, amount-balance, decimals-precision or logo-logoURI
This PR implements a simple class hierarchy merging all source formats into one unified class, which can be enhanced with as many utility functions as we need in the process of developing this new platform.
This is a diagram of the new classes to represent tokens and balances.

Checklist:

[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have cleaned up the code in the areas my change touches
[x] My changes generate no new warnings
[x] Any dependent changes have been merged and published in downstream modules
[x] I have checked my code and corrected any misspellings
[x] I have removed any unnecessary console messages
[x] I have included all english text to the translation file
[x] I have created a new issue with the required translations for the currently supported languages
[ ] I have added appropriate test coverage

This PR is actually blocked because it was developed on top of the last not-merged-yet @EJSG's PR.
It would be a good thing to wait until that PR gets approved and merged.

Questions on the graph, where is the dimension of the price kept so what fiat is it, USD, ETH, PESO whatever? Shouldn't there be many prices for each token, one in each possible currency (not only fiat but every other currency out there eth, btc ...)?

What a nice question!!! because I can extract a hidden requirement for the following function:
TokenPrice.getAmountInThisToken(amount, targetToken): BN

and with this new function, I can answer your question:
All Tokens have a unique price and that price can be expressed in any other currency/token.
In this implementation, the information about the token price is hidden behind the TokenMarketData.info so it can be anything (currently covers the indexer response). Once we have that info for the token, the token.price.isAvailable variable becomes true and then you would be able to ask the following:
const otherToken = await this.getToken('ETH');
const converted = balance.token.price.getAmountInThisToken(
   balance.amount,
   otherToken,
);
// converted is a BigNumber containing the amount of ETH the balance is worth

I will add this function right now !!

I'm closing this PR in favor of this other new PR because it was easy that rebase with the current develop branch
https://github.com/telosnetwork/telos-wallet/pull/362

repo: telosnetwork/telos-wallet


text:
🛑 Secret Site2 is down

In a85f03c, Secret Site2 ($SECRET_SITE_2) was down:

HTTP code: 404
Response time: 86 ms

Resolved: Secret Site2 is back up in 0f1e392 after 60 days, 6 hours, 24 minutes.

repo: templain/mywatcher


text:
🛑 Secret Site2 is down

In 6814cbd, Secret Site2 ($SECRET_SITE_2) was down:

HTTP code: 404
Response time: 26 ms

Resolved: Secret Site2 is back up in 6ff0c68 after 19 minutes.


text:
merge from template-kronos-app

package.json

chore(deps): add ^13.0.0 remove ^12.3.1 (devDependencies.npm-pkgbuild)

:tada: This PR is included in version 1.11.0 :tada:
The release is available on:

npm package (@latest dist-tag)
v1.11.0
GitHub release

Your semantic-release bot :package::rocket:

repo: template-tools/template-sync-hook


text:
[Bug] Explanatory text on ### headings should be  included in --help output

I think the explanatory text on the ### headings should make it into the user-facing interface, but currently I don't think it is. E.g.
...
### temporal workflow: Start, list, and operate on Workflows
### temporal workflow cancel: Send cancellation to Workflow Execution
### temporal workflow count: Number of Workflow Executions
### temporal workflow delete: Remove Workflow Execution...
...

$ temporal workflow --help | head
Workflow commands perform operations on Workflow Executions:

This should become moot with #620

This does not show up in the --help output for a specific command by design. The explanatory text is used as the one-line summary for each command that's output when you ask for a list of sub-commands, for example:
❯ temporal workflow
Workflow commands perform operations on Workflow Executions:

temporal workflow [command] [options]

For example:

temporal workflow list

Usage:
  temporal workflow [command]

Available Commands:
  cancel           Send cancellation to Workflow Execution
  count            Number of Workflow Executions
  delete           Remove Workflow Execution
  describe         Show Workflow Execution info
  execute          Start new Workflow Execution
  fix-history-json Updates an event history JSON file
  list             Show Workflow Executions
  query            Retrieve Workflow Execution state
  reset            Move Workflow Execution history point
  show             Display Event History
  signal           Send a message to a Workflow Execution
  stack            Trace a Workflow Execution
  start            Initiate a Workflow Execution
  terminate        Forcefully end a Workflow Execution
  trace            Workflow Execution live progress
  update           Synchronously run a Workflow update handler

Ah, thanks, I see. I think it would make sense for it to be the first sentence displayed by --help, with a newline separating it from the rest of the text. This would help ensure that the short sentence is a good summary of the subcommand, and help make the two not repeat each other and generally work well together (since the short must work well as the intro to the longer explanation).
This would be reminiscent of how command line tools use short help and long help pages. E.g. from Rust's clap library:
short help with -h
--blame-code-style <STYLE>                                    Style string for the code section of a git blame line

long help with --help
--blame-code-style <STYLE>
    Style string for the code section of a git blame line.

    By default the code will be syntax-highlighted with the same background color as the

Yeah, I'm not super wild about the help output generally. We can probably track this under #604. Are you okay if I mark this as a dup of that issue?

Are you okay if I mark this as a dup of that issue?

Sure! I'll do that now.
Closing as dup of #604

authors: cretz dandavison josh-berry

repo: temporalio/cli


text:
Use Secret instead of ConfigMap for config_template.yaml

config_template.yaml has secrets in it and should be stored as a secret so that any kubernetes or etcd encryption may be applied and access by pods/users can be limited

took note for @tsurdilo

authors: sw-yx underrun

repo: temporalio/helm-charts


text:
Fix improper casts during clone

What was changed
Fix bad casting during clone and write test that confirms clone works with all of our hand-written options classes
Checklist

Closes #274

Heh, that's a pretty fun bit of reflection

And one that many runtimes can't do. Getting all classes on all JARs in Java is rough and JVM/Python/Node/etc are often based on lazy loading so some things may not be there. But .NET assemblies give you this information up front which is nice.

repo: temporalio/sdk-dotnet


text:
Double Update Accepted Event Due to Replay

Expected Behavior
An update should be accepted (or rejected) exactly once
Actual Behavior
If a workflow replays due to something like a worker crash while there is an update in-flight, we don't re-run the validator for the update but we do call Accept() on the update state machine. That leads to an update.Acceptance message being sent on the WFTCompleted which is duly made durable into the history by the server.
Note that one might consider the fact that the server doesn't catch the mistake to also be a bug. That will be covered with a separate issue.
Steps to Reproduce the Problem
There will be a test in temporalio/features for this (it's how the problem was found in the first place). The general flow is:

Start a workflow with separate activity and workflow workers
Start an update that itself launches an activity
Block in the activity
Stop or otherwise kill the workflow worker
Unblock the blocked activity
Start the workflow worker again
Observe that the workflow replays to rebuild state but that the incorrect extra UpdateAccepted event is added to the history. This history will not replay correctly.

Specifications

Version: v1.21
Platform: all

Closed by #1059

repo: temporalio/sdk-go


text:
Customize buckets per metric

Is your feature request related to a problem? Please describe.
I would like to customize the buckets per metric. The only way supported right now is with Tally, which will customize the buckets for all the metrics.
Please refer to the discussion on this thread for more context.
Describe the solution you'd like
Say, I'm using a metric temporal_activity_execution_latency_bucket and would like more buckets than the default number of buckets. Right now, it's not possible to increase the number of buckets only for this metric. The solution linked in the thread referenced in the previous section will increase the number of buckets for all the metrics. I would like to change the buckets only for the particular metric.
Describe alternatives you've considered
The only alternative will change the buckets for all metrics which isn't preferable.
Additional context
NA

temporal_activity_execution_latency_bucket  is a Timer,  in Tally I believe the only way to configure Timer histogram is to use Reporter.RegisterTimer

Closing as this configuration is part of the scope

authors: Quinn-With-Two-Ns Shubhamurkade


text:
Scheduling API

What was changed
Schedule API:

Added ScheduleClient
Added Schedule DTOs

Marshaller update:

Better Marshaller toe detection. The type will be detected automatically if there specified only name property in Marshal attribute.
Added converter from Proto to DTO with mapping for common proto Messages. It is used internally for client requests only. We might open it as feature later.
Added support for OneOf rule for Marshaller.
Added support for multiple Marshal attributes. It allows to unmarshal value from multiple fields.

Why?

Checklist

Closes #228
How was this tested: manually; unit tests.

TODO

[x] Read API
[x] Write API
[x] Resolve TODOs
[x] Increase DX for Scheduling DTOs (fix immutability issues about setters and readonly props)
[ ] Tests
[ ] Docs

It seems I forgot to create a DTO for the patch() method 🤦

repo: temporalio/sdk-php


text:
[Bug] mTLS doesn't support non-EC private keys

What are you really trying to do?
Using a PEM file beginning with -----BEGIN EC PRIVATE KEY-----:
Traceback
RuntimeError: Failed client connect: Server connection error: tonic::transport::Error(Transport, PrivateKeyParseError)

Temporary remedy
Convert your pkey to pkcs8:
openssl pkcs8 -topk8 -nocrypt -in key.pem -out key-pkcs8.pem

Describe the bug
python-sdk is pointing to an older commit in sdk-core which depends on an older version of tonic.
The key issue has been fixed in this tonic PR.
Minimal Reproduction
def setup_tls_config() -> TLSConfig:
    """Helper func to build TLSConfig struct."""
    with open(os.getenv("TEMPORAL_CLI_TLS_CERT"), "rb") as f:
        client_cert = f.read()
    with open(os.getenv("TEMPORAL_CLI_TLS_KEY"), "rb") as f:
        client_private_key = f.read()
    with open(os.getenv("TEMPORAL_CLI_TLS_CA"), "rb") as f:
        server_root_ca_cert = f.read()
    return TLSConfig(
        client_cert=client_cert,
        client_private_key=client_private_key,
        domain=os.getenv("TEMPORAL_CLI_TLS_SERVER_NAME"),
        server_root_ca_cert=server_root_ca_cert,
    )

Environment/Versions

OS and processor: Linux i7
Temporal Version: temporalio==1.3.0
Are you using Docker or Kubernetes or building Temporal from source? Kubernetes

Additional context
If updating to a more recent core SDK version is fairly straightforward, feel free to throw me some pointers, and I will put together a PR.

:+1: We just overcame a hurdle to going past 0.8 Tonic (OTel libs), so we will be upgrading Tonic and all of these libraries before next release.

Awesome, good to know!

authors: cretz dgjustice

repo: temporalio/sdk-python


text:
Nullable as a supported argument

terrascan version: v1.15.0
Operating System: centos7

Description
A variable Nullable setting doesn't seem to be recognized as a valid setting in terraform by terrascan.
What I Did
When trying to run terrascan scan against my terraform codebase I continually get Unsupported argument; An argument named "nullable" is not expected here.
Nullable is a valid variable setting in terraform, can terrascan either support this option to make this warning go away, or is there some way I can completely ignore it throwing a fit about a nullable setting?

I can reproduce the issue with v1.15.2

I am also seeing this (still version v1.15.2) at this point in time.

See this issue when running terrascan on a MacOS (12.6)
$ terrascan version
version: v1.16.0

variables.tf:5,3-11: Unsupported argument; An argument named "nullable" is not expected here.

extract variables.tf
variable "slack_url" {
  type        = string
  sensitive   = true
  description = "Slack Web Hook URL"
  nullable    = false
}

Sadly seeing this error as well.
`david:$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"
david:$ terrascan version
version: v1.17.1
david:$ terrascan scan --iac-type terraform --non-recursive
2023-01-03T15:44:21.480+0100    warn    commons/load-dir.go:312 failed to build unified config. errors:
/tmp/jbmd8j/moved.tf:7,1-6: Unsupported block type; Blocks of type "moved" are not expected here.
/tmp/jbmd8j/moved.tf:11,1-6: Unsupported block type; Blocks of type "moved" are not expected here.
/tmp/jbmd8j/moved.tf:15,1-6: Unsupported block type; Blocks of type "moved" are not expected here.
/tmp/jbmd8j/moved.tf:19,1-6: Unsupported block type; Blocks of type "moved" are not expected here.
/tmp/jbmd8j/moved.tf:23,1-6: Unsupported block type; Blocks of type "moved" are not expected here.
/tmp/jbmd8j/variables.tf:8,3-11: Unsupported argument; An argument named "nullable" is not expected here.
/tmp/jbmd8j/variables.tf:15,3-11: Unsupported argument; An argument named "nullable" is not expected here.
/tmp/jbmd8j/variables.tf:26,3-11: Unsupported argument; An argument named "nullable" is not expected here.
/tmp/jbmd8j/variables.tf:33,3-11: Unsupported argument; An argument named "nullable" is not expected here.
/tmp/jbmd8j/variables.tf:44,3-11: Unsupported argument; An argument named "nullable" is not expected here.
/tmp/jbmd8j/variables.tf:51,3-11: Unsupported argument; An argument named "nullable" is not expected here.
/tmp/jbmd8j/variables.tf:58,3-11: Unsupported argument; An argument named "nullable" is not expected here.
/tmp/jbmd8j/variables.tf:65,3-11: Unsupported argument; An argument named "nullable" is not expected here.
/tmp/jbmd8j/variables.tf:76,3-11: Unsupported argument; An argument named "nullable" is not expected here.
/tmp/jbmd8j/variables.tf:83,3-11: Unsupported argument; An argument named "nullable" is not expected here.
/tmp/jbmd8j/variables.tf:90,3-11: Unsupported argument; An argument named "nullable" is not expected here.
/tmp/jbmd8j/variables.tf:97,3-11: Unsupported argument; An argument named "nullable" is not expected here.
/tmp/jbmd8j/variables.tf:104,3-11: Unsupported argument; An argument named "nullable" is not expected here.
/tmp/jbmd8j/variables.tf:111,3-11: Unsupported argument; An argument named "nullable" is not expected here.
/tmp/jbmd8j/variables.tf:118,3-11: Unsupported argument; An argument named "nullable" is not expected here.
/tmp/jbmd8j/variables.tf:125,3-11: Unsupported argument; An argument named "nullable" is not expected here.
/tmp/jbmd8j/variables.tf:132,3-11: Unsupported argument; An argument named "nullable" is not expected here.
/tmp/jbmd8j/variables.tf:139,3-11: Unsupported argument; An argument named "nullable" is not expected here.
/tmp/jbmd8j/variables.tf:146,3-11: Unsupported argument; An argument named "nullable" is not expected here.
/tmp/jbmd8j/variables.tf:153,3-11: Unsupported argument; An argument named "nullable" is not expected here.
/tmp/jbmd8j/variables.tf:178,3-11: Unsupported argument; An argument named "nullable" is not expected here.
2023-01-03T15:44:21.562+0100    error   cli/run.go:138  scan run failed{error 26 0  1 error occurred:
* failed to build terraform allResourcesConfig
}
`

Issue continues on MacOS using terraform 1.6 and  terrascan 1.18.

+1 this issue is a blocker. Duplicate: https://github.com/tenable/terrascan/issues/1547

+1, seeing this on macOS + terrascan 1.19.9.
Is anyone maintaining this as it looks that there is no reaction for 2 and half years?

authors: b4nst davidjeddy mjnowen nbjohnson pcanham turboBasic whitelancer

repo: tenable/terrascan


text:
🛑 Beeldoverleg Website is down

In 3daba83, Beeldoverleg Website (https://beeldoverleg.nl) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Beeldoverleg Website is back up in 13fc107 after 6 minutes.

repo: tendsgn/upptime


text:
BasicDecoder: TypeError: call() got an unexpected keyword argument 'training'

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Google Colaboratory
TensorFlow version and how it was installed (source or binary): 2.4.1
TensorFlow-Addons version and how it was installed (source or binary): 0.8.3
Python version: 3.6
Is GPU used? (yes/no): yes

Problem
I am trying to migrate a spell checker model that uses Seq2Seq Encoder-Decoder from Tensorflow 1.1 to Tensorflow 2.4.
In TF version 1.1 Training Decoding Layer looks like this:
def training_decoding_layer(dec_embed_input, targets_length, dec_cell, initial_state, output_layer,
                            vocab_size, max_target_length):
    with tf.name_scope("Training_Decoder"):
        training_helper = tf.contrib.seq2seq.TrainingHelper(inputs=dec_embed_input,
                                                            sequence_length=targets_length,
                                                            time_major=False)

        training_decoder = tf.contrib.seq2seq.BasicDecoder(dec_cell,
                                                           training_helper,
                                                           initial_state,
                                                           output_layer)

        training_logits, _ = tf.contrib.seq2seq.dynamic_decode(training_decoder,
                                                               output_time_major=False,
                                                               impute_finished=True,
                                                               maximum_iterations=max_target_length)
        return training_logits

I migrated code to Tensorflow 2.4 with the help of the Seq2Seq Migration Guide. But I got errors with my different approaches to solve the problem.
Migration Guide a Like Approach
def training_decoding_layer(dec_embed_input, targets_length, dec_cell, initial_state, output_layer,
                            vocab_size, max_target_length):

    with tf.name_scope("Training_Decoder"):
        training_helper = tfa.seq2seq.TrainingSampler(time_major=False)

        training_decoder = tfa.seq2seq.BasicDecoder(dec_cell, training_helper,
                                                    output_layer)

        output, _, _ = training_decoder(dec_embed_input,
                                        sequence_length=targets_length,
                                        initial_state=initial_state,
                                        training=True,
                                        )

        return output

I got his error:

TypeError                                 Traceback (most recent call last)
 in ()
8                                                     threshold)
9             model = build_graph(keep_probability, rnn_size, num_layers, batch_size,
---> 10                                 learning_rate, embedding_size, direction)
11             train(model, epochs, log_string)
5 frames
 in build_graph(keep_prob, rnn_size, num_layers, batch_size, learning_rate, embedding_size, direction)
19                                                       batch_size,
20                                                       embedding_size,
---> 21                                                       direction)
22
23     # Create tensors for the training logits and inference logits
 in seq2seq_model(inputs, targets, keep_prob, inputs_length, targets_length, max_target_length, vocab_size, rnn_size, num_layers, vocab_to_int, batch_size, embedding_size, direction)
25                                                         batch_size,
26                                                         num_layers,
---> 27                                                         direction)
28
29     return training_logits, inference_logits
 in decoding_layer(dec_embed_input, embeddings, enc_output, enc_state, vocab_size, inputs_length, targets_length, max_target_length, rnn_size, vocab_to_int, keep_prob, batch_size, num_layers, direction)
36                                                   output_layer,
37                                                   vocab_size,
---> 38                                                   max_target_length)
39     with tf.compat.v1.variable_scope("decode", reuse=True):
40         inference_logits = inference_decoding_layer(embeddings,
 in training_decoding_layer(dec_embed_input, targets_length, dec_cell, initial_state, output_layer, vocab_size, max_target_length)
11                                         sequence_length=targets_length,
12                                         initial_state=initial_state,
---> 13                                         training=True,
14                                         )
15
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py in call(self, *args, **kwargs)
784               with autocast_variable.enable_auto_cast_variables(
785                   self._compute_dtype_object):
--> 786                 outputs = call_fn(cast_inputs, *args, **kwargs)
787
788             except errors.OperatorNotAllowedInGraphError as e:
/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py in wrapper(*args, **kwargs)
668       except Exception as e:  # pylint:disable=broad-except
669         if hasattr(e, 'ag_error_metadata'):
--> 670           raise e.ag_error_metadata.to_exception(e)
671         else:
672           raise
TypeError: in user code:
/usr/local/lib/python3.6/dist-packages/tensorflow_addons/seq2seq/decoder.py:163 call  *
    self,
/usr/local/lib/python3.6/dist-packages/tensorflow_addons/seq2seq/decoder.py:405 body  *
    (next_outputs, decoder_state, next_inputs, decoder_finished) = decoder.step(
/usr/local/lib/python3.6/dist-packages/tensorflow_addons/seq2seq/basic_decoder.py:134 step  *
    cell_outputs, cell_state = self.cell(inputs, state, training=training)
/usr/local/lib/python3.6/dist-packages/tensorflow_addons/seq2seq/attention_wrapper.py:2035 call  *
    cell_output, next_cell_state = self._cell(cell_inputs, cell_state, **kwargs)

TypeError: __call__() got an unexpected keyword argument 'training'

Second Approach: Delete training parameter
So I deleted the training argument:
def training_decoding_layer(dec_embed_input, targets_length, dec_cell, initial_state, output_layer,
                            vocab_size, max_target_length):

    with tf.name_scope("Training_Decoder"):
        training_helper = tfa.seq2seq.TrainingSampler(time_major=False)

        training_decoder = tfa.seq2seq.BasicDecoder(dec_cell, training_helper,
                                                    output_layer)

        output, _, _ = training_decoder(dec_embed_input,
                                        sequence_length=targets_length,
                                        initial_state=initial_state
                                        )

        return output

Then I got the same error:

TypeError                                 Traceback (most recent call last)
 in ()
8                                                     threshold)
9             model = build_graph(keep_probability, rnn_size, num_layers, batch_size,
---> 10                                 learning_rate, embedding_size, direction)
11             train(model, epochs, log_string)
5 frames
 in build_graph(keep_prob, rnn_size, num_layers, batch_size, learning_rate, embedding_size, direction)
19                                                       batch_size,
20                                                       embedding_size,
---> 21                                                       direction)
22
23     # Create tensors for the training logits and inference logits
 in seq2seq_model(inputs, targets, keep_prob, inputs_length, targets_length, max_target_length, vocab_size, rnn_size, num_layers, vocab_to_int, batch_size, embedding_size, direction)
25                                                         batch_size,
26                                                         num_layers,
---> 27                                                         direction)
28
29     return training_logits, inference_logits
 in decoding_layer(dec_embed_input, embeddings, enc_output, enc_state, vocab_size, inputs_length, targets_length, max_target_length, rnn_size, vocab_to_int, keep_prob, batch_size, num_layers, direction)
36                                                   output_layer,
37                                                   vocab_size,
---> 38                                                   max_target_length)
39     with tf.compat.v1.variable_scope("decode", reuse=True):
40         inference_logits = inference_decoding_layer(embeddings,
 in training_decoding_layer(dec_embed_input, targets_length, dec_cell, initial_state, output_layer, vocab_size, max_target_length)
10         output, _, _ = training_decoder(dec_embed_input,
11                                         sequence_length=targets_length,
---> 12                                         initial_state=initial_state
13                                         )
14
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer_v1.py in call(self, *args, **kwargs)
784               with autocast_variable.enable_auto_cast_variables(
785                   self._compute_dtype_object):
--> 786                 outputs = call_fn(cast_inputs, *args, **kwargs)
787
788             except errors.OperatorNotAllowedInGraphError as e:
/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py in wrapper(*args, **kwargs)
668       except Exception as e:  # pylint:disable=broad-except
669         if hasattr(e, 'ag_error_metadata'):
--> 670           raise e.ag_error_metadata.to_exception(e)
671         else:
672           raise
TypeError: in user code:
/usr/local/lib/python3.6/dist-packages/tensorflow_addons/seq2seq/decoder.py:163 call  *
    self,
/usr/local/lib/python3.6/dist-packages/tensorflow_addons/seq2seq/decoder.py:405 body  *
    (next_outputs, decoder_state, next_inputs, decoder_finished) = decoder.step(
/usr/local/lib/python3.6/dist-packages/tensorflow_addons/seq2seq/basic_decoder.py:134 step  *
    cell_outputs, cell_state = self.cell(inputs, state, training=training)
/usr/local/lib/python3.6/dist-packages/tensorflow_addons/seq2seq/attention_wrapper.py:2035 call  *
    cell_output, next_cell_state = self._cell(cell_inputs, cell_state, **kwargs)

TypeError: __call__() got an unexpected keyword argument 'training'

I really need help.

How is dec_cell defined in this example?

In Tf 1.1, it defined like this:
 with tf.name_scope("Attention_Wrapper"):
        dec_cell = tf.contrib.seq2seq.DynamicAttentionWrapper(dec_cell,
                                                              attn_mech,
                                                              rnn_size)

    initial_state = tf.contrib.seq2seq.DynamicAttentionWrapperState(enc_state,
                                                                    _zero_state_tensors(rnn_size,
                                                                                batch_size,
                                                                                tf.float32))

I converted to version 2.4 like this:
    with tf.name_scope("Attention_Wrapper"):
        dec_cell = tfa.seq2seq.AttentionWrapper(dec_cell, attn_mech, rnn_size,
                                                {'state_size': rnn_size})

    zero_state = dec_cell.get_initial_state(batch_size=batch_size, dtype=tf.float32)
    initial_state = zero_state.clone(cell_state=enc_state)

I disabled eager execution because this code uses placeholders etc. Could it be a problem for this situation?

What is the RNN cell class that you are passing to AttentionWrapper?

Sorry, I did not see the dec_cell parameter.
version 2.4
with tf.name_scope("RNN_Decoder_Cell"):
    for layer in range(num_layers):
        with tf.compat.v1.variable_scope('decoder_{}'.format(layer)):
            lstm = tf.compat.v1.nn.rnn_cell.LSTMCell(rnn_size)
            dec_cell = tf.compat.v1.nn.rnn_cell.DropoutWrapper(lstm,
                                                     input_keep_prob = keep_prob)

See this comment: https://github.com/tensorflow/addons/issues/647#issuecomment-547838754

Thanks for your help 👍

authors: fzehracetin guillaumekln

repo: tensorflow/addons


text:
Update all tutorials to use cleverhans.train.train rather than cleverhans.utils_tf.train

Addresses Issue https://github.com/tensorflow/cleverhans/issues/549

I don't understand why the tests are failing. The issues pointed out by the pep8 tracker seem to be files that haven't even been edited. New to pep8 linters, so any help would be appreciated.

I've reviewed this and it looks good. I'll push the fix for the linter.

https://github.com/tensorflow/cleverhans/pull/727

repo: tensorflow/cleverhans


text:
[it] tutorials/customization/basics

Migrate https://github.com/tensorflow/docs/pull/1379 by @mcaci
CLA verified in that PR

I have reviewed @codeadmin-peritiae comments and I agree with them, I don't know @lamberta if you can make these changes or if I should open a new PR with them (thanks already for the migration of the PR :smile: )

I created PR #192 to introduce the page and the needed changes

authors: lamberta mcaci


text:
Fix issues with tf_job_gpu test

The setup cluster step should wait for the TfJob operator deployment to
be ready.

Ensure that all exceptions result in a failure message being reported to Gubernator.

This change is

Coverage increased (+0.1%) to 37.915% when pulling 2f4efd088c543934c1363b3198746ce6b50eb447 on jlewi:gpu_test_fix into e108d556309a6b8921f33fe575d2931fe6ec1e39 on tensorflow:master.

Coverage remained the same at 37.782% when pulling f2644abaac30c1a977309f1c79b2aad7b18eabd0 on jlewi:gpu_test_fix into e108d556309a6b8921f33fe575d2931fe6ec1e39 on tensorflow:master.

Coverage remained the same at 37.782% when pulling 657adf421f49af46b8f7a14d6c4843e3e3e9fac7 on jlewi:gpu_test_fix into e108d556309a6b8921f33fe575d2931fe6ec1e39 on tensorflow:master.

/test all

Coverage remained the same at 37.782% when pulling 1df6e8704f91d4790bf64b61ae0c70cd11a8b1e0 on jlewi:gpu_test_fix into e108d556309a6b8921f33fe575d2931fe6ec1e39 on tensorflow:master.

/test all

authors: coveralls jlewi


text:
Uninitialized tensors as buffer handles?

Is it possible to tell TFRT not to allocate memory for uninitialized tensors in MLIR but keep their attributes (dimensions, total size, unique std::hash-able instance, etc.)?
The device I`m writing a backend for does not work off the CPU memory: all constants need to be sent to it through DMA, and in case of uninitialized buffers it`s useless to DMA zeroes back and forth.
Furthermore, some networks the device is intended for can really clog the CPU memory if all unique uninitialized buffers it requires get (needlessly) allocated.
Can this be avoided — and if yes, how exactly?

There is a kludge to get around this problem — namely, allocating a 'fake' linear DHT whose size is equal to the intended rank and the elements equal the intended dimensions.
But still, I`m certain there is a more elegant solution I`m not aware of.

repo: tensorflow/runtime


text:
Fix for bef thunk custom call test

Thanks for your contribution! Unfortunately, tensorflow/runtime is currently not
accepting contributions. Please see the
Contribution Guidelines for
more information.
/cc @chsigg

This is wrong so closing.


text:
TFRecordDatasetSampler does not seen to gaurantee example_per_class

TensorFlow: 2.7.0
TensorFlow Similarity 0.14.10
I have large dataset in the format of TFRecords, so I am trying out this sampler. My image classification task has 5 classes, and i like to have 6 instance of each class per batch. I put each class into their own separate TFRecord files (like train_class_0.tfrecords, train_class_1.tfrecords, etc), so they are definitely contiguous all through. (I admittedly am confused on this contiguous requirement as stated in the code comment, but this is the best i can come up with).
sampler = TFRecordDatasetSampler(
    shard_path='.',
    deserialization_fn=lambda x: decode_func(parse_tf_records_fn(x)),
    example_per_class=6,
    batch_size=30,
    shard_suffix='train_*.tfrecords'
)

And I try to examine the labels for each batch:
x, y = next(iter(sampler))
y

And got:
<tf.Tensor: shape=(30,), dtype=int64, numpy=
array([2, 2, 2, 2, 2, 2, 2, 2, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 3, 3,
3, 3, 3, 3, 4, 4, 4, 4])>
After iterating a few times, It seems to always over-sample the 1st label and under sample the last. I tried to isolate and test the relevant code that construct the ds, it seems to me if you set deterministic=True, then each class will correctly have 6 instances. I understand from comment that non-determinism is needed for creating random batch, so I am speculating something about tf dataset interleave may have change?? Or I have some misunderstanding somewhere.
I further try to read the tf interleave documentation. And I run with this simple test case (modified slightly from that doc):
dataset = tf.data.Dataset.range(1, 6)  # ==> [ 1, 2, 3, 4, 5 ]
dataset = dataset.interleave(
    lambda x: tf.data.Dataset.from_tensors(x).repeat(-1),
    cycle_length=5, block_length=6, num_parallel_calls=AUTO, deterministic=False)

dataset = dataset.map(lambda x: x, num_parallel_calls=AUTO)
dataset = dataset.repeat(count=-1)
dataset = dataset.batch(30)
dataset = dataset.prefetch(10)
# np.array(list(dataset.as_numpy_iterator()))

x = next(iter(dataset))
x

<tf.Tensor: shape=(30,), dtype=int64, numpy=
array([1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 4, 4, 4, 4, 4,
4, 5, 5, 5, 5, 5, 5, 1])>
This also fails to yield 6 instances of each class. (If you set deterministic to True, then you will).
It seems to me that it will generate on average, 6/class. But if you look at only one batch, there's no guarantee.
Currently, it may still be ok for example_per_class=6, since i haven't seen any class with <2 instances in the batch, although I can't be sure. It may be a problem if example_per_class is only 2. If the similarity loss can "gracefully" handle the occasional "lone class", it may still work.
Alternatively, I wonder if you can utilize tf.data.Dataset.sample_from_datasets. Although it doesn't gaurantee exactly this number of example/class (only probabilistically through the weights).

I've run into similar issues with interleave and have also recently come to feel that deterministic needs to be set to True in order to ensure the correct number of classes per batch. I discussed some of the issues here as well #171.  The original idea was to create a bunch of TFRecords that had contiguous blocks of a class, likely of size block_length. Then each time the interleave would load a record and consume it one block_length at a time. However, it seems that when deterministic is False, the interleave is not guaranteed to consume the entire block_length from an iterator before switching over to another one.  Additionally, TFRecords can only be consumed sequentially, which means we will always have a fixed order within the block_length, but the hope was that each individual file would be loaded randomly.
My current thought is that we can write many small TFRecords for a each class, with each file of size block_length. Then the file list can be shuffled before the interleave to introduce the random sampling. This will allow us to set deterministic as False and guarantee the order.
Regarding the use of tf.data.Dataset.sample_from_datasets, it seems to be very very slow, ... at least in the tests I ran a while back. I can take another look at it though.

I haven't tested this yet, but I think one potential solution could be something like the following:
NUM_SHARDS_PER_CLASS = # num_images_in_class // num_examples_per_class

def create_records(...):
    # Here we materialize a sharded TFRecord per class.
    # Each TFRecord contains NUM_SHARDS_PER_CLASS = num_images_in_class // num_examples_per_class
    pass

def tfrecord_sampler(class_id):
    random_shard_id = tf.random.uniform(
        shape=(),
        maxval=NUM_SHARDS_PER_CLASS-1,
        dtype=tf.int32,
    )
    shard_path = f"path/to/shard/class_{class_id}_{random_shard_id}"

    return tf.data.TFRecordDataset(filenames = [shard_path])

class_ids = list(range(200))

ds = tf.data.Dataset.from_tensor_slices(class_ids)
ds = ds.shuffle(len(class_ids))
ds = ds.interleave(
    lambda x: tfrecord_sampler,
    cycle_length=num_classes_per_batch,
    block_length=num_examples_per_class,
    num_parallel_calls=tf.data.AUTOTUNE,
    deterministic=True,
)
ds = ds.map(lambda x: x, num_parallel_calls=tf.data.AUTOTUNE)
ds = ds.repeat(count=-1)
ds = ds.batch(num_classes_per_batch*num_examples_per_class)
ds = ds.prefetch(tf.data.AUTOTUNE)

@owenvallis  I will likely give your code/approach a try when i get a chance, and if i understood this correctly.
So your idea is to break up your single class tfrecord into many small tfrecords (still of the same class)? I am still a little bit confused over variable names you used. ‘Cos it appears each shard (or the little tfrecord) should contain num_examples_per_class examples (however, u said Each TFRecord contains NUM_SHARDS_PER_CLASS….).
which should be a very small number? It will help a lot to provide an extremely simple but concrete example, including the specific # of examples and their class within each broken down tfrecord.
I actually went ahead and use the current sampler even knowing this imperfection. It seems to train ok (maybe lucky not to have any lone class within a batch).
Aside:
The primary reason i m interesting in a tfrecord based samplers is to eventually run training on TPU. In the past, I always run into issues on first trial and many issues have to do with unimplemented op. If some of TPU “bug” still not fixed, i anticipate your notebook will fail on TPU due to use of data augmentation layers within the model. I sort of always workaround that by doing data augmentation within the tf dataset pipeline (done on CPU). If i really do hit that later, i will probably just document it as an issue.

Hi @kechan, I'll try and setup a working example later today, and apologies for any confusing var names above. To clarify, I'm hoping the above approach will enable use to:

Use shuffle to load a random set of class ids into interleave().
Use the class id to load a random shard from that class.
Each shard should be of size num_example_per_class.

This should make the interleave() function consume an entire shard and then load another random class id. While the examples within each shard are fixed, we should still get a good mix within the batches as the collection of shards will be fairly random.
And thanks for bringing the TPU issue to our attention. I'll see if I can run some tests on TPUs as well.

@owenvallis Thanks a lot.
One concern I just have is this approach will generate a large # of tfrecord files? I never try more than 50 before. Do you anticipate other performance bottleneck? For TPU, it seems to have very fast link to cloud storage, so i hope there’s no surprise there.

@owenvallis
Update for tf.data.Dataset.sample_from_datasets
I went ahead and attempted a solution using tf.data.Dataset.sample_from_datasets(...) and ran it on TPU, it is quite fast at ~125ms/step, in the range i expected for my dataset. I suspect the slowness you saw, could be due to the prior bug I raised related to tf.convert_to_tensor(...)

authors: kechan owenvallis

repo: tensorflow/similarity