text:
Add ES6 modules to whitelist

Babel support the ES6 module syntax and will per default transpile it to commonjs syntax: https://babeljs.io/docs/usage/modules/
This syntax is a lot cleaner than the current way of requiring and destructing. These is no reason not to allow this :-)
Example
import React, { Component, View, Text } from 'react-native'

class Foobar extends Component {
  render() {
    return (
      <View><Text>Foo</Text></View>
    )
  }
}

export default Foobar

The reason for the extra React import is that the transpiled JSX is React.createElement(...)

please rerun tests 1 errored out, not failed.

I'm not sure why we aren't using this internally - @vjeux?

fyi #1940 should also go in if this PR is merged.

@sahrens @vjeux I don't understand why you are using a whitelist at all after transitioning to BabelJS?

@ide yes please - annoying when you use max 80 character lines :-)

@hkjorgensen: I don't understand why you are using a whitelist at all after transitioning to BabelJS?

Tooling
Babel is only one piece of the infrastructure, we also need to make sure that flow, lint, packager... knows about it. The react-native packager implements support for ES6 imports but it has never really been tested in practice. Our web packager most certainly doesn't support it right now, so this means that any module we write for react-native cannot be used on our web stack if we wanted to. I'm not sure if flow supports it yet, but if it doesn't, then this means that adding a single file with this import is going to break flow for the entire project.
Spec Evolution
Also, the new js specs evolve really quickly and we want to make sure that if we allow developers to use it internally, we are able to maintain it and keep up with the specification. For example, we were probably the first big user of ES6 classes and at some point it was decided that super call was required /before/ you can assign anything to this. Our transpiler at the time didn't enforce this rule (as it didn't exist), but now in order to bring back our codebase closer to the spec, we need to update thousands of js files.
It has a real cost to adopt early stage software and maintaining it, we want to make sure that we are ready to support every feature we whitelist. Otherwise we're just going to whitelist everything and be oncall for the next 3 months trying to stop all the fires.
Developer Education
Having two ways to do something really sucks for developers. Now they've got to learn this new thing that does essentially the same thing as the old one. But they see both in the codebase so it's confusing which one should be used when.
If we are to allow internal developers to use es6 imports, what we are probably going to do is to spend the time to convert all the require to es6 imports. This is going to be painful and a lot of work, but at the end of the day we'll only have one way to do things.
Conclusion
If you are just hacking on React Native on your own, feel free to whitelist all the features that you want. Facebook right now is driving React Native and we don't want the open source version to diverge too far from the internal version because we're not going to be able to do a good job at maintaining something that we do not use.
I know it's a bit frustrating to be held off by Facebook internal adoption of features but I believe that this is the right move right now given that the project is still in a very early stage. Note that we're fully committed to moving in the direction of es6/7 and you will see all those features being rolled out in the future.

@vjeux thank you for a very good answer. In general I agree, but I think your current implementation can be improved and still fit your internal requirements.
If I - as a consumer of ReactNative - see "Babel" and then hit the wall because it's only "partial" implemented - I think it's you who made a bad implementation and a flawed product. It's a bad developer experience.
Instead of restricting and locking settings down, you should enable the consumer to take control without hacking the sourcecode directly.
A simple example could be support for a .babelrc file in the root folder.
Or encourage the user the build their own transpiler layer - which is kind of stupid, when you already have all the best tools setup and ready to go :-)

A simple example could be support for a .babelrc file in the root folder.

I am using .babelrc and it's mostly awesome, especially the ES7 transforms that make writing classes a lot better. Can use everything except the runtime features.

Or encourage the user the build their own transpiler layer

You could also run the packager with --transform /path/to/custom-transformer.js
This all said I think it's a hackathon's worth of work for Facebook to mostly move over to ES6 modules (see https://github.com/facebook/jscodeshift). ES6 is finalized AFAIK so it shouldn't be a moving target -- the hard part is identifying which require() calls can't be convert to import statements.

One other thought to add is that we're also trying to figure out the best way to do dynamic requires where chunks of JavaScript aren't loaded until needed, and this functionality might influence the way we change our import semantics.

On Jul 16, 2015, at 6:26 AM, James Ide notifications@github.com wrote:
A simple example could be support for a .babelrc file in the root folder.
I am using .babelrc and it's mostly awesome, especially the ES7 transforms that make writing classes a lot better. Can use everything except the runtime features.
Or encourage the user the build their own transpiler layer
You could also run the packager with --transform /path/to/custom-transformer.js
This all said I think it's a hackathon's worth of work for Facebook to mostly move over to ES6 modules (see https://github.com/facebook/jscodeshift). ES6 is finalized AFAIK so it shouldn't be a moving target -- the hard part is identifying which require() calls can't be convert to import statements.
—
Reply to this email directly or view it on GitHub.

Hey just another voice from the outside mentioning that it would be awesome to allow the ES6 import/export syntax. I understand that having more than one way to do things can be confusing, and that it makes sense to build RN to match the style Facebook uses internally. However, as @ide mentioned ES6 is a finished standard, and gaining mindshare rapidly in the larger JS community. It's also especially popular among users of React, probably largely thanks to Babel's stable support for it. Lots of examples/tutorials online use it by default, and not being able to transfer that syntax over to React Native may be confusing for users not familiar with commonjs.
And when considered from the perspective of a new user, ES6 modules are definitely a simpler and more  intuitive syntax than their commonjs equivalents, and more powerful to boot (you get a default export plus optional named ones instead of having to choose between a single function vs an object with a bunch of properties).
Babel's default functionality of transforming import/export into the equivalent commonjs statements also means that a project can be converted from the old syntax to the new one in a piecemeal fashion without breaking functionality.

Okay, I just invoked the JS council inside of Facebook and here's the status. We are happy with ES6 Modules and we are going to move to it in the future. Right now it's not super high pri but it's going to happen.
As you figured out, it already works in React Native and flow ( https://github.com/facebook/flow/issues/560 ) and jest should also support it. The remaining tooling where it doesn't work is fb internal.
So, we're good to enable the flag. This way people in open source can use it. We're not going to use it yet in react-native itself until we move the entire fb codebase to it.
@hkjorgensen can you rebase your pull request and update the JS Environment docs? https://github.com/facebook/react-native/blob/42eb5464fd8a65ed84b799de5d4dc225349449be/docs/JavaScriptEnvironment.md Thanks!

@hkjorgensen are you able to get to this? Otherwise I'll put up a PR in a couple days.

This is effectively in master: https://github.com/facebook/react-native/blob/ce2bb0fe162d904ed31bd8d7f91a8648858dea3f/packager/transformer.js#L43

Sorry for being late at the party! Nice to see this functionality is now in master :+1:

authors: corbt hkjorgensen ide sahrens vjeux


text:
Bump Android NDK to r17b

This PR bumps Android NDK version to r17b (latest). Cleaned up redundant LOCAL_EXPORT_CPPFLAGS rules in .mk files
Test Plan:
Everything compiles and runs as usual
Release Notes:
[ANDROID] [ENHANCEMENT] [NDK] - Bump to r17b version

Thanks for the job, nicely done. Let's wait the ci fixed on master and then merge this.

I did some cleanup and will push once master CI fixed.

Done and CI is green 😄. Had to bump buck because it was trying to use removed ABI from NDK, and buck v2018.07.09.01 fixed it with https://github.com/facebook/buck/commit/4ef8e96bae72988f4bd77994a0228a595aa42ac6.

@hramos please review and merge

I would like to note that RNTester app works as usual. And I found that LOCAL_EXPORT_CPPFLAGS exports CFLAGS to including module, which resulted in polluted and mixed CFLAGS.

source: gharchive/pull-request

authors: dulmandakh gengjiawen

repo: facebook/react-native

license: MIT

license_type: permissive

license_source: github-api


text:
Fix: Add displayName to ActivityIndicator

Summary:
Similar to #21950, this adds displayName to ActivityIndicator, so it displays the correct component name in snapshots instead of Component.
Fixes #21937 (a little bit more than it was already fixed).
Test Plan:
yarn test - ActivityIndicator's - ActivityIndicator-test.js.snap should Show
Changelog/Release Notes:
[General] [Fixed] - Add displayName to ActivityIndicator for snapshots

Thanks for the fix!
As you mentioned there aren't any tests that caught this. Would you like to try adding a simple jest test for ActivityIndicator that uses a snapshot test that would keep this problem from happening again?

@TheSavior I added a basic Snapshot for ActivityIndicator that should catch regressions on this!

Awesome, thanks!

authors: TheSavior miblanchard


text:
Missing comma

I find missing commas
Please confirm

Trailing commas are optional, and there doesn't seem to be consistency across the repo. I wonder what the preferred style is?

The preferred style is to have a trailing comma, however this PR has languished for quite sometime and because of the current sync process it would be much easier if someone could just hunt down all missing cases and fix them in a single PR, thanks though!

authors: Depapepe arthuralee brentvatne


text:
iOS: Enable views to be nested within

Previously, only Text and Image could be nested within Text. Now, any
view can be nested within Text. One restriction of this feature is
that developers must give inline views a width and a height via
the style prop.
Previously, inline Images were supported by using iOS's built-in support
for rendering images with an NSAttributedString via NSTextAttachment.
However, NSAttributedString doesn't support rendering arbitrary views.
This change adds support for nesting views within Text by creating one
NSTextAttachment per inline view. The NSTextAttachments act as placeholders.
They are set to be the size of the corresponding view. After the text is
laid out, we query the text system to find out where it has positioned each
NSTextAttachment. We then position the views to be at those locations.
This commit also contains a change in RCTShadowText.m
_setParagraphStyleOnAttributedString:heightOfTallestSubview:. It now only sets
lineHeight, textAlign, and writingDirection when they've actually changed. This
avoids unnecessarily dirtying the shadow tree which is important in some inline
view scenarios.
Here's a screen shot demoing an inline view and an inline image:

Adam Comella
Microsoft Corp.

@rigdern updated the pull request.

@rigdern updated the pull request.

@facebook-github-bot import

Thanks for importing. If you are an FB employee go to Phabricator to review.

Thanks for importing. If you are an FB employee go to Phabricator to review.

This looks great!
I'm not sure the way that you're handling vertical alignment is correct though. It seems like you're basically aligning the views to the top of the line rect, but I would expect them to be aligned with the baseline of the text by default, which is how iOS normally positions text attachments, and how inline images were aligned on iOS previously.
Baseline alignment isn't always desirable, and we should probably consider ways to allow the developer to adjust the offset, as well as ensuring consistency with Android, but for now I'd like to ensure we remain consistent with how this worked on iOS previously.

@rigdern updated the pull request.

@nicklockwood I updated the PR to use 0.0 for the y coordinate like it did before my change. Hopefully the PR is good to go now.
Here are some questions/comments about future work:
If I wanted to support customizing the vertical alignment of inline views, what would you recommend? A new style on View that only works on inline views and is a no-op on other views (e.g. inlineVerticalAlignmentwith values like "bottom" and "baseline")?
We're currently working on adding a similar feature to the Android implementation.
Adam Comella
Microsoft Corp.

This looks good. I'll need to do some testing with our internal apps just to check that dropping support for automatically detecting image size hasn't broken anything.

If I wanted to support customizing the vertical alignment of inline views, what would you recommend? A new style on View that only works on inline views and is a no-op on other views (e.g. inlineVerticalAlignmentwith values like "bottom" and "baseline")?

That's a good question. A simple solution would just be to respect the existing "top" and/or "marginTop" styles to adjust the view by a relative offset. That should work pretty-much out of the box on iOS since NSTextAttachment respects the bounds.origin and treats it as a relative offset (although we might want to flip the y value since it follows the Mac convention of y pointing up the screen instead of down IIRC).
Having a verticalAlignment property that follows the CSS spec would be nice as well, although personally I've never found it very useful in practice because visually aligning images with text usually involves pixel-level adjustment.
Either way, the key thing is that the behavior should match between iOS and Android so we don't need to fork the styles every time we use inline views.

@facebook-github-bot import

cc @dmmiller for thoughts on alignment API

Thanks for importing. If you are an FB employee go to Phabricator to review.

@rigdern the alignment is still wrong. It seems like the problem is that [layoutManager boundingRectForGlyphRange:range inTextContainer:textContainer] doesn't return the actual frame of the NSTextAttachment, but a frame whose height matches the height of the row.
You've compensated for this by replacing the width/height with the actual width/height of the view (line 141 or RCTShadowText), but that doesn't fix the origin, which is still incorrect as it places the view at the top of the row frame instead of aligning it with the baseline.
I was able to improve this slightly by changing the logic to
  CGRect childFrame = [layoutManager boundingRectForGlyphRange:range inTextContainer:textContainer];
  childFrame = (CGRect){{
    RCTRoundPixelValue(childFrame.origin.x),
    RCTRoundPixelValue(childFrame.origin.y + childFrame.size.height - height)
  }, {
    RCTRoundPixelValue(width),
    RCTRoundPixelValue(height)
  }};

So that the view is aligned with the bottom of the frame instead of the top, but that's still not the same as the baseline, which is what it should be. There's presumably some way to get the actual NSTextAttachment frame instead of the frame for the placeholder glyph, but I couldn't find it after a few minutes of searching.

@rigdern updated the pull request.

@nicklockwood Sorry about that. The vertical alignment should be good now. I tested it with short and tall images and compared the results with the original implementation.
Here's what the UIExplorer example now looks like:

I did this as a separate commit so that you can see exactly what I changed. Let me know if you need me to squash it. In general, does your team prefer pull request revisions to be done as separate commits or squashed into the original commit?

@nicklockwood Is there anything else blocking this change from being merged?

@rigdern sorry for the delay. This looks good, thanks!
@facebook-github-bot import

Thanks for importing. If you are an FB employee go to Phabricator to review.

@facebook-github-bot shipit

Thanks for importing. If you are an FB employee go to Phabricator to review.

This regressed one of our application startup metrics by 5%. Any clue where this could be coming from? Otherwise, I'd prefer to revert this for now, until we get this metrics-neutral.

@javache Without doing some analysis, I can't tell which part of this change slowed down your startup metrics. Just reverting this change fixes your startup metrics? Does your app use a lot of inline images?
@nicklockwood, do you have any thoughts on this?

@rigdern unfortunately I haven't been able to work out what's causing the regression. I'm going to revert this for now and try landing it again next week after our branch cut.
I did find a few obvious inefficiencies, such as not stripping the views associated with RawText (these are regular UIViews, so your check for RCTText views doesn't strip them), but I tried fixing that and it had no measurable effect.
I'm guessing that the problem must be related to the extra logic added to the layout, but I don't have any ideas beyond that right now.

@nicklockwood Can you provide any details about the scenario such as how many inline images there are and how the perf measurements are being taken (e.g. are you using some markers built into RN or does the app do the instrumentation itself? If the app is doing its own instrumentation, what specific timeframe is being measured and what is being used to take the measurements?)

@nicklockwood I see that this got checked in again. Were you able to fix the performance problem? If so, what was the cause?

@rigdern yes. I made some changes in RCTText which seemed to eliminate the regression, I'll annotate them for you in the commit.

@nicklockwood Thanks for the explanations!


text:
else-if formatting pretty bad

See #79. Bumping that one as high pri in the list of pretty print improvements. I wish we could bump/pin issues.

I just removed the lowpro of task 79. I think this task can be closed? I think commenting on the task can bump it, and if not, you can make a tag for priorities. Feel free to clean up/close/reprioritize everything via tags.

source: gharchive/issue

authors: chenglou jordwalke

repo: facebook/reason


text:
Test suite fails to run with Python 3.2

$ PYTHONPATH="." python3.2 tornado/test/runtests.py
Traceback (most recent call last):
  File "tornado/test/runtests.py", line 33, in <module>
    tornado.testing.main()
  File "/tmp/tornado-2.2/build/lib/tornado/testing.py", line 347, in main
    argv = [sys.argv[0]] + parse_command_line(sys.argv)
  File "/tmp/tornado-2.2/build/lib/tornado/options.py", line 147, in parse_command_line
    enable_pretty_logging()
  File "/tmp/tornado-2.2/build/lib/tornado/options.py", line 347, in enable_pretty_logging
    channel.setFormatter(_LogFormatter(color=color))
  File "/tmp/tornado-2.2/build/lib/tornado/options.py", line 364, in __init__
    logging.DEBUG: str(curses.tparm(fg_color, 4), # Blue
TypeError: 'str' does not support the buffer interface
$ python3.2 -c 'import sys; print(sys.version)'
3.2.2+ (3.2:77188bc37c74+, Jan 29 2012, 21:00:23)
[GCC 4.5.3]

With Python 3.1.4:
$ PYTHONPATH="." python3.1 tornado/test/runtests.py
...
Ran 183 tests in 0.715s

FAILED (failures=2, errors=84)

Python 3.2 is supported; 3.1 is not.  What directory are you running this from?  For python 3, you must use setup.py install so that 2to3 gets run; it won't work to just add the original source to PYTHONPATH.

$ cd /tmp
$ wget https://github.com/downloads/facebook/tornado/tornado-2.2.tar.gz &> /dev/null
$ tar -xzf tornado-2.2.tar.gz
$ cd tornado-2.2
$ python3.2 setup.py build
running build
running build_py
creating build
creating build/lib
creating build/lib/tornado
copying tornado/autoreload.py -> build/lib/tornado
copying tornado/locale.py -> build/lib/tornado
copying tornado/process.py -> build/lib/tornado
copying tornado/httpclient.py -> build/lib/tornado
copying tornado/stack_context.py -> build/lib/tornado
copying tornado/__init__.py -> build/lib/tornado
copying tornado/util.py -> build/lib/tornado
copying tornado/wsgi.py -> build/lib/tornado
copying tornado/websocket.py -> build/lib/tornado
copying tornado/netutil.py -> build/lib/tornado
copying tornado/ioloop.py -> build/lib/tornado
copying tornado/database.py -> build/lib/tornado
copying tornado/testing.py -> build/lib/tornado
copying tornado/httputil.py -> build/lib/tornado
copying tornado/curl_httpclient.py -> build/lib/tornado
copying tornado/escape.py -> build/lib/tornado
copying tornado/iostream.py -> build/lib/tornado
copying tornado/gen.py -> build/lib/tornado
copying tornado/options.py -> build/lib/tornado
copying tornado/simple_httpclient.py -> build/lib/tornado
copying tornado/auth.py -> build/lib/tornado
copying tornado/web.py -> build/lib/tornado
copying tornado/httpserver.py -> build/lib/tornado
copying tornado/template.py -> build/lib/tornado
creating build/lib/tornado/test
copying tornado/test/escape_test.py -> build/lib/tornado/test
copying tornado/test/template_test.py -> build/lib/tornado/test
copying tornado/test/testing_test.py -> build/lib/tornado/test
copying tornado/test/twisted_test.py -> build/lib/tornado/test
copying tornado/test/import_test.py -> build/lib/tornado/test
copying tornado/test/httpclient_test.py -> build/lib/tornado/test
copying tornado/test/auth_test.py -> build/lib/tornado/test
copying tornado/test/stack_context_test.py -> build/lib/tornado/test
copying tornado/test/web_test.py -> build/lib/tornado/test
copying tornado/test/__init__.py -> build/lib/tornado/test
copying tornado/test/curl_httpclient_test.py -> build/lib/tornado/test
copying tornado/test/process_test.py -> build/lib/tornado/test
copying tornado/test/runtests.py -> build/lib/tornado/test
copying tornado/test/httpserver_test.py -> build/lib/tornado/test
copying tornado/test/httputil_test.py -> build/lib/tornado/test
copying tornado/test/iostream_test.py -> build/lib/tornado/test
copying tornado/test/ioloop_test.py -> build/lib/tornado/test
copying tornado/test/simple_httpclient_test.py -> build/lib/tornado/test
copying tornado/test/gen_test.py -> build/lib/tornado/test
copying tornado/test/run_pyversion_tests.py -> build/lib/tornado/test
copying tornado/test/wsgi_test.py -> build/lib/tornado/test
creating build/lib/tornado/platform
copying tornado/platform/__init__.py -> build/lib/tornado/platform
copying tornado/platform/posix.py -> build/lib/tornado/platform
copying tornado/platform/auto.py -> build/lib/tornado/platform
copying tornado/platform/twisted.py -> build/lib/tornado/platform
copying tornado/platform/interface.py -> build/lib/tornado/platform
copying tornado/platform/windows.py -> build/lib/tornado/platform
copying tornado/ca-certificates.crt -> build/lib/tornado
copying tornado/test/README -> build/lib/tornado/test
copying tornado/test/test.crt -> build/lib/tornado/test
copying tornado/test/test.key -> build/lib/tornado/test
creating build/lib/tornado/test/static
copying tornado/test/static/robots.txt -> build/lib/tornado/test/static
Fixing build/lib/tornado/autoreload.py build/lib/tornado/locale.py build/lib/tornado/process.py build/lib/tornado/httpclient.py build/lib/tornado/stack_context.py build/lib/tornado/__init__.py build/lib/tornado/util.py build/lib/tornado/wsgi.py build/lib/tornado/websocket.py build/lib/tornado/netutil.py build/lib/tornado/ioloop.py build/lib/tornado/database.py build/lib/tornado/testing.py build/lib/tornado/httputil.py build/lib/tornado/curl_httpclient.py build/lib/tornado/escape.py build/lib/tornado/iostream.py build/lib/tornado/gen.py build/lib/tornado/options.py build/lib/tornado/simple_httpclient.py build/lib/tornado/auth.py build/lib/tornado/web.py build/lib/tornado/httpserver.py build/lib/tornado/template.py build/lib/tornado/test/escape_test.py build/lib/tornado/test/template_test.py build/lib/tornado/test/testing_test.py build/lib/tornado/test/twisted_test.py build/lib/tornado/test/import_test.py build/lib/tornado/test/httpclient_test.py build/lib/tornado/test/auth_test.py build/lib/tornado/test/stack_context_test.py build/lib/tornado/test/web_test.py build/lib/tornado/test/__init__.py build/lib/tornado/test/curl_httpclient_test.py build/lib/tornado/test/process_test.py build/lib/tornado/test/runtests.py build/lib/tornado/test/httpserver_test.py build/lib/tornado/test/httputil_test.py build/lib/tornado/test/iostream_test.py build/lib/tornado/test/ioloop_test.py build/lib/tornado/test/simple_httpclient_test.py build/lib/tornado/test/gen_test.py build/lib/tornado/test/run_pyversion_tests.py build/lib/tornado/test/wsgi_test.py build/lib/tornado/platform/__init__.py build/lib/tornado/platform/posix.py build/lib/tornado/platform/auto.py build/lib/tornado/platform/twisted.py build/lib/tornado/platform/interface.py build/lib/tornado/platform/windows.py
Skipping implicit fixer: buffer
Skipping implicit fixer: idioms
Skipping implicit fixer: set_literal
Skipping implicit fixer: ws_comma
Fixing build/lib/tornado/autoreload.py build/lib/tornado/locale.py build/lib/tornado/process.py build/lib/tornado/httpclient.py build/lib/tornado/stack_context.py build/lib/tornado/__init__.py build/lib/tornado/util.py build/lib/tornado/wsgi.py build/lib/tornado/websocket.py build/lib/tornado/netutil.py build/lib/tornado/ioloop.py build/lib/tornado/database.py build/lib/tornado/testing.py build/lib/tornado/httputil.py build/lib/tornado/curl_httpclient.py build/lib/tornado/escape.py build/lib/tornado/iostream.py build/lib/tornado/gen.py build/lib/tornado/options.py build/lib/tornado/simple_httpclient.py build/lib/tornado/auth.py build/lib/tornado/web.py build/lib/tornado/httpserver.py build/lib/tornado/template.py build/lib/tornado/test/escape_test.py build/lib/tornado/test/template_test.py build/lib/tornado/test/testing_test.py build/lib/tornado/test/twisted_test.py build/lib/tornado/test/import_test.py build/lib/tornado/test/httpclient_test.py build/lib/tornado/test/auth_test.py build/lib/tornado/test/stack_context_test.py build/lib/tornado/test/web_test.py build/lib/tornado/test/__init__.py build/lib/tornado/test/curl_httpclient_test.py build/lib/tornado/test/process_test.py build/lib/tornado/test/runtests.py build/lib/tornado/test/httpserver_test.py build/lib/tornado/test/httputil_test.py build/lib/tornado/test/iostream_test.py build/lib/tornado/test/ioloop_test.py build/lib/tornado/test/simple_httpclient_test.py build/lib/tornado/test/gen_test.py build/lib/tornado/test/run_pyversion_tests.py build/lib/tornado/test/wsgi_test.py build/lib/tornado/platform/__init__.py build/lib/tornado/platform/posix.py build/lib/tornado/platform/auto.py build/lib/tornado/platform/twisted.py build/lib/tornado/platform/interface.py build/lib/tornado/platform/windows.py
Skipping implicit fixer: buffer
Skipping implicit fixer: idioms
Skipping implicit fixer: set_literal
Skipping implicit fixer: ws_comma
$ cd build/lib
$ PYTHONPATH="." python3.2 tornado/test/runtests.py
Traceback (most recent call last):
  File "tornado/test/runtests.py", line 33, in <module>
    tornado.testing.main()
  File "/tmp/tornado-2.2/build/lib/tornado/testing.py", line 347, in main
    argv = [sys.argv[0]] + parse_command_line(sys.argv)
  File "/tmp/tornado-2.2/build/lib/tornado/options.py", line 147, in parse_command_line
    enable_pretty_logging()
  File "/tmp/tornado-2.2/build/lib/tornado/options.py", line 347, in enable_pretty_logging
    channel.setFormatter(_LogFormatter(color=color))
  File "/tmp/tornado-2.2/build/lib/tornado/options.py", line 364, in __init__
    logging.DEBUG: str(curses.tparm(fg_color, 4), # Blue
TypeError: 'str' does not support the buffer interface
$

Hmm.  It works on my build of python 3.2.2, but you're running "3.2.2+ (3.2:77188bc37c74+, Jan 29 2012, 21:00:23)", which looks like an unreleased build from hg.    Looks like this change: http://bugs.python.org/issue10570.  I'm glad they're fixing this correctly, although it's now a pain to support both versions.  I'll probably have to have an explicit version check for <= 3.2.2.  In the meantime you can work around the problem by passing --logging=none to skip this code.

authors: Arfrever bdarnell

repo: facebook/tornado


text:
Convergence problem (Does "DECOMP.RSSD" have to converge?)

NRMSE converged, but DECOMP.RSSD NOT converged.

DECOMP.RSSD NOT converged: sd@qt.20 0.084 > 0.025 & |med@qt.20| 0.17 <= 0.36
NRMSE converged: sd@qt.20 0.01 <= 0.022 & |med@qt.20| 0.048 <= 0.12
(R2 train:0.9114, NRMSE=0.048, DECOMP.RSSD=0.0902)

1.If the spending on marketing was very far from ideal, DECOMP.RSSD may not converge.So it is not essential for DECOMP.RSSD to converge.
Also, I use 12 years data ,because I have to use monthly data. In 12 years, media environment and plannnninng  has changed a lot, so spending distribution may not make sense in the first place.
2.DECOMP.RSSD NOT converged, but DECOMP.RSSD = 0.0902 in my model.That seems low enough to me, but isn't it?
3.Looking at "Multi-objective Evolutionary Performance", there is also a darker colored dot in the upper right corner. (iterations is close to 2000, but both NRMSE and DECOMP.RSSD are large)
As for the convergence result, NRMSE has converged, but is this convergence process problematic? (DECOMP.RSSD not converged.)
#Robyn 3.7.1
#R version 4.2.1

This issue is already answered in https://www.facebook.com/groups/robynmmm/permalink/1358750738226389/

repo: facebookexperimental/Robyn


text:
Cannot run optimizer

Trying to run budget allocator function and getting this error:
Error in names(channel_constr_low) <- set_mediaVarName :
'names' attribute [6] must be the same length as the vector [1]
I am giving the function the following:
f.budgetAllocator(modID = "2_16_3",
,optim_algo = "MMA_AUGLAG"
,expected_spend = NULL
,expected_spend_days = NULL
,channel_constr_low = 0.5
,channel_constr_up = 2
,scenario = "max_historical_response"
,maxeval = 100000
,constr_mode = "eq")
if channel_constr_low is an integer, why is the function applying names() to it ?
Thanks!
Federico

Hey Fede,
Hope you are well and thanks for trying our code. Let me show you a best practice example so that you copy it on your end:
model_output_collect$allSolutions
optim_result <- f.budgetAllocator(modID = "2_16_3" # input one of the model IDs in model_output_collect$allSolutions to get optimisation result
                                  ,scenario = "max_historical_response" # c(max_historical_response, max_response_expected_spend)
                                  #,expected_spend = 100000 # specify future spend volume. only applies when scenario = "max_response_expected_spend"
                                  #,expected_spend_days = 90 # specify period for the future spend volumne in days. only applies when scenario = "max_response_expected_spend"
                                  ,channel_constr_low = c(0.25, 0.25, 0.25, 0.25, 0.25, 0.25) # must be between 0.01-1 and has same length and order as set_mediaVarName
                                  ,channel_constr_up = c(3, 3, 3, 3, 3, 3) # not recommended to 'exaggerate' upper bounds. 1.5 means channel budget can increase to 150% of current level
)

Please note to just comment the lines for expected_spend and expected_spend_days as well as, putting unoe channel_constr_low and one channel_constr_up per each channel involved. In the example above we have 6 channels to assess.
Names is used basically to see if the number of channels listed in the optimization actually match the ones that are used in the budget allocator.
Let me know if that works,
Hope it helps!
Thanks,
Leo

It worked, makes sense that channel_constr_low should be a vector.
Thanks Leo!


text:
Country Request: Australia

Requesting RapID for Australia. Especially our more rural and remote areas

Thanks for the request @tastrax
I've added Australia to the list

Australia is now available.
Happy mapping!

Thanks Folks - when will it be available in the downloads? https://github.com/facebookmicrosites/Open-Mapping-At-Facebook/wiki/Available-Countries

authors: jeffdefacto tastrax

repo: facebookincubator/RapiD

license: ISC


text:
fbshipit-source-id: d9eebb25a00223865a8f7aeb898bc63acfe4d4fa

Resync files between internal repo and Github; this is needed after we changed of config recently.

@jrwats: Is there a some other kind of branch protection in Github that's configurable only by you?
I'm seeing this error message:

Merging is blocked
The base branch restricts merging to authorized users. Learn more about protected branches.

This seems to have merged just fine?

authors: jrwats kayhadrin

repo: facebookincubator/fbt


text:
max_data_size_for_stats gives different results for Presto/Velox

Description
I am not sure it is intentional for this function to give different results in presto/velox , however I see some runs fail due to this (It also seems to be part of custom verification).
Error Reproduction
No response
Relevant logs
I20240124 23:51:12.548703 20604910 AggregationFuzzer.cpp:357] ==============================> Started iteration 39 (seed: 3677319442)
I20240124 23:51:13.565384 20604910 AggregationFuzzerBase.cpp:364] Executing query plan:
-- Aggregation[SINGLE [g0, g1, g2] a0 := max_data_size_for_stats(ROW["c0"]) ORDER BY c0 ASC NULLS LAST] -> g0:ARRAY<VARBINARY>, g1:ARRAY<INTEGER>, g2:ROW<f0:INTEGER,f1:SMALLINT,f2:SMALLINT,f3:TIMESTAMP,f4:SMALLINT,f5:INTEGER>, a0:BIGINT
  -- Values[1000 rows in 10 vectors] -> c0:VARBINARY, g0:ARRAY<VARBINARY>, g1:ARRAY<INTEGER>, g2:ROW<f0:INTEGER,f1:SMALLINT,f2:SMALLINT,f3:TIMESTAMP,f4:SMALLINT,f5:INTEGER>
I20240124 23:51:13.625216 20617727 Task.cpp:1082] All drivers (1) finished for task test_cursor 1111 after running for 59 ms.
I20240124 23:51:13.625252 20617727 Task.cpp:1766] Terminating task test_cursor 1111 with state Finished after running for 59 ms.
I20240124 23:51:14.798066 20604910 Writer.cpp:546] Stripe 0: Flush overhead = 326976, data length = 8519100, pre flush mem = 9368512, post flush mem = 9626752. Closing = true
I20240124 23:51:14.804138 20604910 FileSink.cpp:123] closing file: /Users/kpai/presto_install/presto-server-0.284/data/tpch/tmp/fuzzer.dwrf,  total size: 8.13MB
/Users/kpai/src/velox/velox/exec/tests/utils/QueryAssertions.cpp:1128: Failure
Failed
Expected 961, got 961
819 extra rows, 819 missing rows
10 of extra rows:
	null | null | null | 3839
	null | null | [null,1925,21975,"289242895-12-27T03:11:16.933000000",3846,567563705] | 147
	null | null | [null,10791,28838,"117499795-04-24T14:48:01.902000000",11064,2040928900] | 3749
	null | null | [241561893,25491,21975,"140732592-06-13T22:55:40.511000000",null,1797721265] | 3659
	null | null | [477367585,24585,9635,"131083602-06-22T10:23:25.626000000",null,1404281493] | 310
	null | null | [541074125,null,null,"122607092-06-11T19:17:25.058000000",null,754233925] | 3867
	null | null | [603795230,10791,28838,"54206544-09-01T10:22:10.131000000",15171,null] | 3749
	null | null | [708862417,null,6630,null,19975,null] | 2594
	null | null | [768146278,19622,5114,null,11508,1630278026] | 3040
	null | null | [848299657,10791,28838,"242697729-11-24T08:23:54.506000000",22580,null] | 3749
10 of missing rows:
	null | null | null | 3835
	null | null | [null,1925,21975,"289242895-12-27T03:11:16.933000000",3846,567563705] | 143
	null | null | [null,10791,28838,"117499795-04-24T14:48:01.902000000",11064,2040928900] | 3745
	null | null | [241561893,25491,21975,"140732592-06-13T22:55:40.511000000",null,1797721265] | 3655
	null | null | [477367585,24585,9635,"131083602-06-22T10:23:25.626000000",null,1404281493] | 306
	null | null | [541074125,null,null,"122607092-06-11T19:17:25.058000000",null,754233925] | 3863
	null | null | [603795230,10791,28838,"54206544-09-01T10:22:10.131000000",15171,null] | 3745
	null | null | [708862417,null,6630,null,19975,null] | 2590
	null | null | [768146278,19622,5114,null,11508,1630278026] | 3036
	null | null | [848299657,10791,28838,"242697729-11-24T08:23:54.506000000",22580,null] | 3745

I think this is the same issue as https://github.com/facebookincubator/velox/issues/6330.

We had the same finding in https://github.com/prestodb/presto/issues/20909#issuecomment-1735963832 https://github.com/prestodb/presto/issues/20909

authors: aditi-pandit kagamiori kgpai

repo: facebookincubator/velox


text:
Preserve constant encodings in Presto serializer

When appending constants, we record the constant value and count of repeats. We keep a vector that holds a copy of each constant. if it is all the same constant, we produce a constant. If there are several, we produce a dictionary vector. We revert to flat vector if flat vectors are appended or if nulls are appended from null-introducing dictionaries. We will later consider preserving dictionaries and/or dictionarizing low cardinality flat vectors.
The test encodedConcatenation goes over appending different constants, flats and dictionaries into a serializer and reads the data back. The data is read back from separately encoded pieces to test combining encodings in deserialization. At the same time, it appends the encoded pieces into a single serializer to test combining encodings at send time.

This looks like it's at least partially a duplicate of https://github.com/facebookincubator/velox/pull/8249
At least it looks like you're both trying to solve the same problem.

This looks like it's at least partially a duplicate of #8249
At least it looks like you're both trying to solve the same problem.

Thanks for bringing this up. We've chatted about this yesterday. These are related but not quite the same, though this code will likely have to be refactored at some point to provide a more intuitive API.
The focus of #8249 is to make sure that if you only want to serialize one rowVector, we preserve the original encoding as much as we can. This PR focuses on finding the best encoding as you successfully add rowVectors to the serialized buffer.

authors: kevinwilfong oerling pedroerp


text:
load pretrained model, there are errors. RuntimeError: Error(s) in loading state_dict for ConvNeXt: 	Missing key(s) in state_dict

RuntimeError: Error(s) in loading state_dict for ConvNeXt:
Missing key(s) in state_dict: "stem.0.weight", "stem.0.bias", "stem.1.weight", "stem.1.bias", "stages.0.blocks.0.gamma", "stages.0.blocks.0.conv_dw.weight", "stages.0.blocks.0.conv_dw.bias", "stages.0.blocks.0.norm.weight", "stages.0.blocks.0.norm.bias", "stages.0.blocks.0.mlp.fc1.weight", "stages.0.blocks.0.mlp.fc1.bias", "stages.0.blocks.0.mlp.fc2.weight", "stages.0.blocks.0.mlp.fc2.bias", "stages.0.blocks.1.gamma", "stages.0.blocks.1.conv_dw.weight", "stages.0.blocks.1.conv_dw.bias", "stages.0.blocks.1.norm.weight", "stages.0.blocks.1.norm.bias", "stages.0.blocks.1.mlp.fc1.weight", "stages.0.blocks.1.mlp.fc1.bias", "stages.0.blocks.1.mlp.fc2.weight", "stages.0.blocks.1.mlp.fc2.bias", "stages.0.blocks.2.gamma", "stages.0.blocks.2.conv_dw.weight", "stages.0.blocks.2.conv_dw.bias", "stages.0.blocks.2.norm.weight", "stages.0.blocks.2.norm.bias", "stages.0.blocks.2.mlp.fc1.weight", "stages.0.blocks.2.mlp.fc1.bias", "stages.0.blocks.2.mlp.fc2.weight", "stages.0.blocks.2.mlp.fc2.bias", "stages.1.downsample.0.weight", "stages.1.downsample.0.bias", "stages.1.downsample.1.weight", "stages.1.downsample.1.bias", "stages.1.blocks.0.gamma", "stages.1.blocks.0.conv_dw.weight", "stages.1.blocks.0.conv_dw.bias", "stages.1.blocks.0.norm.weight", "stages.1.blocks.0.norm.bias", "stages.1.blocks.0.mlp.fc1.weight", "stages.1.blocks.0.mlp.fc1.bias", "stages.1.blocks.0.mlp.fc2.weight", "stages.1.blocks.0.mlp.fc2.bias", "stages.1.blocks.1.gamma", "stages.1.blocks.1.conv_dw.weight", "stages.1.blocks.1.conv_dw.bias", "stages.1.blocks.1.norm.weight", "stages.1.blocks.1.norm.bias", "stages.1.blocks.1.mlp.fc1.weight", "stages.1.blocks.1.mlp.fc1.bias", "stages.1.blocks.1.mlp.fc2.weight", "stages.1.blocks.1.mlp.fc2.bias", "stages.1.blocks.2.gamma", "stages.1.blocks.2.conv_dw.weight", "stages.1.blocks.2.conv_dw.bias", "stages.1.blocks.2.norm.weight", "stages.1.blocks.2.norm.bias", "stages.1.blocks.2.mlp.fc1.weight", "stages.1.blocks.2.mlp.fc1.bias", "stages.1.blocks.2.mlp.fc2.weight", "stages.1.blocks.2.mlp.fc2.bias", "stages.2.downsample.0.weight", "stages.2.downsample.0.bias", "stages.2.downsample.1.weight", "stages.2.downsample.1.bias", "stages.2.blocks.0.gamma", "stages.2.blocks.0.conv_dw.weight", "stages.2.blocks.0.conv_dw.bias", "stages.2.blocks.0.norm.weight", "stages.2.blocks.0.norm.bias", "stages.2.blocks.0.mlp.fc1.weight", "stages.2.blocks.0.mlp.fc1.bias", "stages.2.blocks.0.mlp.fc2.weight", "stages.2.blocks.0.mlp.fc2.bias", "stages.2.blocks.1.gamma", "stages.2.blocks.1.conv_dw.weight", "stages.2.blocks.1.conv_dw.bias", "stages.2.blocks.1.norm.weight", "stages.2.blocks.1.norm.bias", "stages.2.blocks.1.mlp.fc1.weight", "stages.2.blocks.1.mlp.fc1.bias", "stages.2.blocks.1.mlp.fc2.weight", "stages.2.blocks.1.mlp.fc2.bias", "stages.2.blocks.2.gamma", "stages.2.blocks.2.conv_dw.weight", "stages.2.blocks.2.conv_dw.bias", "stages.2.blocks.2.norm.weight", "stages.2.blocks.2.norm.bias", "stages.2.blocks.2.mlp.fc1.weight", "stages.2.blocks.2.mlp.fc1.bias", "stages.2.blocks.2.mlp.fc2.weight", "stages.2.blocks.2.mlp.fc2.bias", "stages.2.blocks.3.gamma", "stages.2.blocks.3.conv_dw.weight", "stages.2.blocks.3.conv_dw.bias", "stages.2.blocks.3.norm.weight", "stages.2.blocks.3.norm.bias", "stages.2.blocks.3.mlp.fc1.weight", "stages.2.blocks.3.mlp.fc1.bias", "stages.2.blocks.3.mlp.fc2.weight", "stages.2.blocks.3.mlp.fc2.bias", "stages.2.blocks.4.gamma", "stages.2.blocks.4.conv_dw.weight", "stages.2.blocks.4.conv_dw.bias", "stages.2.blocks.4.norm.weight", "stages.2.blocks.4.norm.bias", "stages.2.blocks.4.mlp.fc1.weight", "stages.2.blocks.4.mlp.fc1.bias", "stages.2.blocks.4.mlp.fc2.weight", "stages.2.blocks.4.mlp.fc2.bias", "stages.2.blocks.5.gamma", "stages.2.blocks.5.conv_dw.weight", "stages.2.blocks.5.conv_dw.bias", "stages.2.blocks.5.norm.weight", "stages.2.blocks.5.norm.bias", "stages.2.blocks.5.mlp.fc1.weight", "stages.2.blocks.5.mlp.fc1.bias", "stages.2.blocks.5.mlp.fc2.weight", "stages.2.blocks.5.mlp.fc2.bias", "stages.2.blocks.6.gamma", "stages.2.blocks.6.conv_dw.weight", "stages.2.blocks.6.conv_dw.bias", "stages.2.blocks.6.norm.weight", "stages.2.blocks.6.norm.bias", "stages.2.blocks.6.mlp.fc1.weight", "stages.2.blocks.6.mlp.fc1.bias", "stages.2.blocks.6.mlp.fc2.weight", "stages.2.blocks.6.mlp.fc2.bias", "stages.2.blocks.7.gamma", "stages.2.blocks.7.conv_dw.weight", "stages.2.blocks.7.conv_dw.bias", "stages.2.blocks.7.norm.weight", "stages.2.blocks.7.norm.bias", "stages.2.blocks.7.mlp.fc1.weight", "stages.2.blocks.7.mlp.fc1.bias", "stages.2.blocks.7.mlp.fc2.weight", "stages.2.blocks.7.mlp.fc2.bias", "stages.2.blocks.8.gamma", "stages.2.blocks.8.conv_dw.weight", "stages.2.blocks.8.conv_dw.bias", "stages.2.blocks.8.norm.weight", "stages.2.blocks.8.norm.bias", "stages.2.blocks.8.mlp.fc1.weight", "stages.2.blocks.8.mlp.fc1.bias", "stages.2.blocks.8.mlp.fc2.weight", "stages.2.blocks.8.mlp.fc2.bias", "stages.2.blocks.9.gamma", "stages.2.blocks.9.conv_dw.weight", "stages.2.blocks.9.conv_dw.bias", "stages.2.blocks.9.norm.weight", "stages.2.blocks.9.norm.bias", "stages.2.blocks.9.mlp.fc1.weight", "stages.2.blocks.9.mlp.fc1.bias", "stages.2.blocks.9.mlp.fc2.weight", "stages.2.blocks.9.mlp.fc2.bias", "stages.2.blocks.10.gamma", "stages.2.blocks.10.conv_dw.weight", "stages.2.blocks.10.conv_dw.bias", "stages.2.blocks.10.norm.weight", "stages.2.blocks.10.norm.bias", "stages.2.blocks.10.mlp.fc1.weight", "stages.2.blocks.10.mlp.fc1.bias", "stages.2.blocks.10.mlp.fc2.weight", "stages.2.blocks.10.mlp.fc2.bias", "stages.2.blocks.11.gamma", "stages.2.blocks.11.conv_dw.weight", "stages.2.blocks.11.conv_dw.bias", "stages.2.blocks.11.norm.weight", "stages.2.blocks.11.norm.bias", "stages.2.blocks.11.mlp.fc1.weight", "stages.2.blocks.11.mlp.fc1.bias", "stages.2.blocks.11.mlp.fc2.weight", "stages.2.blocks.11.mlp.fc2.bias", "stages.2.blocks.12.gamma", "stages.2.blocks.12.conv_dw.weight", "stages.2.blocks.12.conv_dw.bias", "stages.2.blocks.12.norm.weight", "stages.2.blocks.12.norm.bias", "stages.2.blocks.12.mlp.fc1.weight", "stages.2.blocks.12.mlp.fc1.bias", "stages.2.blocks.12.mlp.fc2.weight", "stages.2.blocks.12.mlp.fc2.bias", "stages.2.blocks.13.gamma", "stages.2.blocks.13.conv_dw.weight", "stages.2.blocks.13.conv_dw.bias", "stages.2.blocks.13.norm.weight", "stages.2.blocks.13.norm.bias", "stages.2.blocks.13.mlp.fc1.weight", "stages.2.blocks.13.mlp.fc1.bias", "stages.2.blocks.13.mlp.fc2.weight", "stages.2.blocks.13.mlp.fc2.bias", "stages.2.blocks.14.gamma", "stages.2.blocks.14.conv_dw.weight", "stages.2.blocks.14.conv_dw.bias", "stages.2.blocks.14.norm.weight", "stages.2.blocks.14.norm.bias", "stages.2.blocks.14.mlp.fc1.weight", "stages.2.blocks.14.mlp.fc1.bias", "stages.2.blocks.14.mlp.fc2.weight", "stages.2.blocks.14.mlp.fc2.bias", "stages.2.blocks.15.gamma", "stages.2.blocks.15.conv_dw.weight", "stages.2.blocks.15.conv_dw.bias", "stages.2.blocks.15.norm.weight", "stages.2.blocks.15.norm.bias", "stages.2.blocks.15.mlp.fc1.weight", "stages.2.blocks.15.mlp.fc1.bias", "stages.2.blocks.15.mlp.fc2.weight", "stages.2.blocks.15.mlp.fc2.bias", "stages.2.blocks.16.gamma", "stages.2.blocks.16.conv_dw.weight", "stages.2.blocks.16.conv_dw.bias", "stages.2.blocks.16.norm.weight", "stages.2.blocks.16.norm.bias", "stages.2.blocks.16.mlp.fc1.weight", "stages.2.blocks.16.mlp.fc1.bias", "stages.2.blocks.16.mlp.fc2.weight", "stages.2.blocks.16.mlp.fc2.bias", "stages.2.blocks.17.gamma", "stages.2.blocks.17.conv_dw.weight", "stages.2.blocks.17.conv_dw.bias", "stages.2.blocks.17.norm.weight", "stages.2.blocks.17.norm.bias", "stages.2.blocks.17.mlp.fc1.weight", "stages.2.blocks.17.mlp.fc1.bias", "stages.2.blocks.17.mlp.fc2.weight", "stages.2.blocks.17.mlp.fc2.bias", "stages.2.blocks.18.gamma", "stages.2.blocks.18.conv_dw.weight", "stages.2.blocks.18.conv_dw.bias", "stages.2.blocks.18.norm.weight", "stages.2.blocks.18.norm.bias", "stages.2.blocks.18.mlp.fc1.weight", "stages.2.blocks.18.mlp.fc1.bias", "stages.2.blocks.18.mlp.fc2.weight", "stages.2.blocks.18.mlp.fc2.bias", "stages.2.blocks.19.gamma", "stages.2.blocks.19.conv_dw.weight", "stages.2.blocks.19.conv_dw.bias", "stages.2.blocks.19.norm.weight", "stages.2.blocks.19.norm.bias", "stages.2.blocks.19.mlp.fc1.weight", "stages.2.blocks.19.mlp.fc1.bias", "stages.2.blocks.19.mlp.fc2.weight", "stages.2.blocks.19.mlp.fc2.bias", "stages.2.blocks.20.gamma", "stages.2.blocks.20.conv_dw.weight", "stages.2.blocks.20.conv_dw.bias", "stages.2.blocks.20.norm.weight", "stages.2.blocks.20.norm.bias", "stages.2.blocks.20.mlp.fc1.weight", "stages.2.blocks.20.mlp.fc1.bias", "stages.2.blocks.20.mlp.fc2.weight", "stages.2.blocks.20.mlp.fc2.bias", "stages.2.blocks.21.gamma", "stages.2.blocks.21.conv_dw.weight", "stages.2.blocks.21.conv_dw.bias", "stages.2.blocks.21.norm.weight", "stages.2.blocks.21.norm.bias", "stages.2.blocks.21.mlp.fc1.weight", "stages.2.blocks.21.mlp.fc1.bias", "stages.2.blocks.21.mlp.fc2.weight", "stages.2.blocks.21.mlp.fc2.bias", "stages.2.blocks.22.gamma", "stages.2.blocks.22.conv_dw.weight", "stages.2.blocks.22.conv_dw.bias", "stages.2.blocks.22.norm.weight", "stages.2.blocks.22.norm.bias", "stages.2.blocks.22.mlp.fc1.weight", "stages.2.blocks.22.mlp.fc1.bias", "stages.2.blocks.22.mlp.fc2.weight", "stages.2.blocks.22.mlp.fc2.bias", "stages.2.blocks.23.gamma", "stages.2.blocks.23.conv_dw.weight", "stages.2.blocks.23.conv_dw.bias", "stages.2.blocks.23.norm.weight", "stages.2.blocks.23.norm.bias", "stages.2.blocks.23.mlp.fc1.weight", "stages.2.blocks.23.mlp.fc1.bias", "stages.2.blocks.23.mlp.fc2.weight", "stages.2.blocks.23.mlp.fc2.bias", "stages.2.blocks.24.gamma", "stages.2.blocks.24.conv_dw.weight", "stages.2.blocks.24.conv_dw.bias", "stages.2.blocks.24.norm.weight", "stages.2.blocks.24.norm.bias", "stages.2.blocks.24.mlp.fc1.weight", "stages.2.blocks.24.mlp.fc1.bias", "stages.2.blocks.24.mlp.fc2.weight", "stages.2.blocks.24.mlp.fc2.bias", "stages.2.blocks.25.gamma", "stages.2.blocks.25.conv_dw.weight", "stages.2.blocks.25.conv_dw.bias", "stages.2.blocks.25.norm.weight", "stages.2.blocks.25.norm.bias", "stages.2.blocks.25.mlp.fc1.weight", "stages.2.blocks.25.mlp.fc1.bias", "stages.2.blocks.25.mlp.fc2.weight", "stages.2.blocks.25.mlp.fc2.bias", "stages.2.blocks.26.gamma", "stages.2.blocks.26.conv_dw.weight", "stages.2.blocks.26.conv_dw.bias", "stages.2.blocks.26.norm.weight", "stages.2.blocks.26.norm.bias", "stages.2.blocks.26.mlp.fc1.weight", "stages.2.blocks.26.mlp.fc1.bias", "stages.2.blocks.26.mlp.fc2.weight", "stages.2.blocks.26.mlp.fc2.bias", "stages.3.downsample.0.weight", "stages.3.downsample.0.bias", "stages.3.downsample.1.weight", "stages.3.downsample.1.bias", "stages.3.blocks.0.gamma", "stages.3.blocks.0.conv_dw.weight", "stages.3.blocks.0.conv_dw.bias", "stages.3.blocks.0.norm.weight", "stages.3.blocks.0.norm.bias", "stages.3.blocks.0.mlp.fc1.weight", "stages.3.blocks.0.mlp.fc1.bias", "stages.3.blocks.0.mlp.fc2.weight", "stages.3.blocks.0.mlp.fc2.bias", "stages.3.blocks.1.gamma", "stages.3.blocks.1.conv_dw.weight", "stages.3.blocks.1.conv_dw.bias", "stages.3.blocks.1.norm.weight", "stages.3.blocks.1.norm.bias", "stages.3.blocks.1.mlp.fc1.weight", "stages.3.blocks.1.mlp.fc1.bias", "stages.3.blocks.1.mlp.fc2.weight", "stages.3.blocks.1.mlp.fc2.bias", "stages.3.blocks.2.gamma", "stages.3.blocks.2.conv_dw.weight", "stages.3.blocks.2.conv_dw.bias", "stages.3.blocks.2.norm.weight", "stages.3.blocks.2.norm.bias", "stages.3.blocks.2.mlp.fc1.weight", "stages.3.blocks.2.mlp.fc1.bias", "stages.3.blocks.2.mlp.fc2.weight", "stages.3.blocks.2.mlp.fc2.bias", "head.norm.weight", "head.norm.bias", "head.fc.weight", "head.fc.bias".
Unexpected key(s) in state_dict: "downsample_layers.0.0.weight", "downsample_layers.0.0.bias", "downsample_layers.0.1.weight", "downsample_layers.0.1.bias", "downsample_layers.1.0.weight", "downsample_layers.1.0.bias", "downsample_layers.1.1.weight", "downsample_layers.1.1.bias", "downsample_layers.2.0.weight", "downsample_layers.2.0.bias", "downsample_layers.2.1.weight", "downsample_layers.2.1.bias", "downsample_layers.3.0.weight", "downsample_layers.3.0.bias", "downsample_layers.3.1.weight", "downsample_layers.3.1.bias", "norm.weight", "norm.bias", "stages.0.0.gamma", "stages.0.0.dwconv.weight", "stages.0.0.dwconv.bias", "stages.0.0.norm.weight", "stages.0.0.norm.bias", "stages.0.0.pwconv1.weight", "stages.0.0.pwconv1.bias", "stages.0.0.pwconv2.weight", "stages.0.0.pwconv2.bias", "stages.0.1.gamma", "stages.0.1.dwconv.weight", "stages.0.1.dwconv.bias", "stages.0.1.norm.weight", "stages.0.1.norm.bias", "stages.0.1.pwconv1.weight", "stages.0.1.pwconv1.bias", "stages.0.1.pwconv2.weight", "stages.0.1.pwconv2.bias", "stages.0.2.gamma", "stages.0.2.dwconv.weight", "stages.0.2.dwconv.bias", "stages.0.2.norm.weight", "stages.0.2.norm.bias", "stages.0.2.pwconv1.weight", "stages.0.2.pwconv1.bias", "stages.0.2.pwconv2.weight", "stages.0.2.pwconv2.bias", "stages.1.0.gamma", "stages.1.0.dwconv.weight", "stages.1.0.dwconv.bias", "stages.1.0.norm.weight", "stages.1.0.norm.bias", "stages.1.0.pwconv1.weight", "stages.1.0.pwconv1.bias", "stages.1.0.pwconv2.weight", "stages.1.0.pwconv2.bias", "stages.1.1.gamma", "stages.1.1.dwconv.weight", "stages.1.1.dwconv.bias", "stages.1.1.norm.weight", "stages.1.1.norm.bias", "stages.1.1.pwconv1.weight", "stages.1.1.pwconv1.bias", "stages.1.1.pwconv2.weight", "stages.1.1.pwconv2.bias", "stages.1.2.gamma", "stages.1.2.dwconv.weight", "stages.1.2.dwconv.bias", "stages.1.2.norm.weight", "stages.1.2.norm.bias", "stages.1.2.pwconv1.weight", "stages.1.2.pwconv1.bias", "stages.1.2.pwconv2.weight", "stages.1.2.pwconv2.bias", "stages.2.0.gamma", "stages.2.0.dwconv.weight", "stages.2.0.dwconv.bias", "stages.2.0.norm.weight", "stages.2.0.norm.bias", "stages.2.0.pwconv1.weight", "stages.2.0.pwconv1.bias", "stages.2.0.pwconv2.weight", "stages.2.0.pwconv2.bias", "stages.2.1.gamma", "stages.2.1.dwconv.weight", "stages.2.1.dwconv.bias", "stages.2.1.norm.weight", "stages.2.1.norm.bias", "stages.2.1.pwconv1.weight", "stages.2.1.pwconv1.bias", "stages.2.1.pwconv2.weight", "stages.2.1.pwconv2.bias", "stages.2.2.gamma", "stages.2.2.dwconv.weight", "stages.2.2.dwconv.bias", "stages.2.2.norm.weight", "stages.2.2.norm.bias", "stages.2.2.pwconv1.weight", "stages.2.2.pwconv1.bias", "stages.2.2.pwconv2.weight", "stages.2.2.pwconv2.bias", "stages.2.3.gamma", "stages.2.3.dwconv.weight", "stages.2.3.dwconv.bias", "stages.2.3.norm.weight", "stages.2.3.norm.bias", "stages.2.3.pwconv1.weight", "stages.2.3.pwconv1.bias", "stages.2.3.pwconv2.weight", "stages.2.3.pwconv2.bias", "stages.2.4.gamma", "stages.2.4.dwconv.weight", "stages.2.4.dwconv.bias", "stages.2.4.norm.weight", "stages.2.4.norm.bias", "stages.2.4.pwconv1.weight", "stages.2.4.pwconv1.bias", "stages.2.4.pwconv2.weight", "stages.2.4.pwconv2.bias", "stages.2.5.gamma", "stages.2.5.dwconv.weight", "stages.2.5.dwconv.bias", "stages.2.5.norm.weight", "stages.2.5.norm.bias", "stages.2.5.pwconv1.weight", "stages.2.5.pwconv1.bias", "stages.2.5.pwconv2.weight", "stages.2.5.pwconv2.bias", "stages.2.6.gamma", "stages.2.6.dwconv.weight", "stages.2.6.dwconv.bias", "stages.2.6.norm.weight", "stages.2.6.norm.bias", "stages.2.6.pwconv1.weight", "stages.2.6.pwconv1.bias", "stages.2.6.pwconv2.weight", "stages.2.6.pwconv2.bias", "stages.2.7.gamma", "stages.2.7.dwconv.weight", "stages.2.7.dwconv.bias", "stages.2.7.norm.weight", "stages.2.7.norm.bias", "stages.2.7.pwconv1.weight", "stages.2.7.pwconv1.bias", "stages.2.7.pwconv2.weight", "stages.2.7.pwconv2.bias", "stages.2.8.gamma", "stages.2.8.dwconv.weight", "stages.2.8.dwconv.bias", "stages.2.8.norm.weight", "stages.2.8.norm.bias", "stages.2.8.pwconv1.weight", "stages.2.8.pwconv1.bias", "stages.2.8.pwconv2.weight", "stages.2.8.pwconv2.bias", "stages.2.9.gamma", "stages.2.9.dwconv.weight", "stages.2.9.dwconv.bias", "stages.2.9.norm.weight", "stages.2.9.norm.bias", "stages.2.9.pwconv1.weight", "stages.2.9.pwconv1.bias", "stages.2.9.pwconv2.weight", "stages.2.9.pwconv2.bias", "stages.2.10.gamma", "stages.2.10.dwconv.weight", "stages.2.10.dwconv.bias", "stages.2.10.norm.weight", "stages.2.10.norm.bias", "stages.2.10.pwconv1.weight", "stages.2.10.pwconv1.bias", "stages.2.10.pwconv2.weight", "stages.2.10.pwconv2.bias", "stages.2.11.gamma", "stages.2.11.dwconv.weight", "stages.2.11.dwconv.bias", "stages.2.11.norm.weight", "stages.2.11.norm.bias", "stages.2.11.pwconv1.weight", "stages.2.11.pwconv1.bias", "stages.2.11.pwconv2.weight", "stages.2.11.pwconv2.bias", "stages.2.12.gamma", "stages.2.12.dwconv.weight", "stages.2.12.dwconv.bias", "stages.2.12.norm.weight", "stages.2.12.norm.bias", "stages.2.12.pwconv1.weight", "stages.2.12.pwconv1.bias", "stages.2.12.pwconv2.weight", "stages.2.12.pwconv2.bias", "stages.2.13.gamma", "stages.2.13.dwconv.weight", "stages.2.13.dwconv.bias", "stages.2.13.norm.weight", "stages.2.13.norm.bias", "stages.2.13.pwconv1.weight", "stages.2.13.pwconv1.bias", "stages.2.13.pwconv2.weight", "stages.2.13.pwconv2.bias", "stages.2.14.gamma", "stages.2.14.dwconv.weight", "stages.2.14.dwconv.bias", "stages.2.14.norm.weight", "stages.2.14.norm.bias", "stages.2.14.pwconv1.weight", "stages.2.14.pwconv1.bias", "stages.2.14.pwconv2.weight", "stages.2.14.pwconv2.bias", "stages.2.15.gamma", "stages.2.15.dwconv.weight", "stages.2.15.dwconv.bias", "stages.2.15.norm.weight", "stages.2.15.norm.bias", "stages.2.15.pwconv1.weight", "stages.2.15.pwconv1.bias", "stages.2.15.pwconv2.weight", "stages.2.15.pwconv2.bias", "stages.2.16.gamma", "stages.2.16.dwconv.weight", "stages.2.16.dwconv.bias", "stages.2.16.norm.weight", "stages.2.16.norm.bias", "stages.2.16.pwconv1.weight", "stages.2.16.pwconv1.bias", "stages.2.16.pwconv2.weight", "stages.2.16.pwconv2.bias", "stages.2.17.gamma", "stages.2.17.dwconv.weight", "stages.2.17.dwconv.bias", "stages.2.17.norm.weight", "stages.2.17.norm.bias", "stages.2.17.pwconv1.weight", "stages.2.17.pwconv1.bias", "stages.2.17.pwconv2.weight", "stages.2.17.pwconv2.bias", "stages.2.18.gamma", "stages.2.18.dwconv.weight", "stages.2.18.dwconv.bias", "stages.2.18.norm.weight", "stages.2.18.norm.bias", "stages.2.18.pwconv1.weight", "stages.2.18.pwconv1.bias", "stages.2.18.pwconv2.weight", "stages.2.18.pwconv2.bias", "stages.2.19.gamma", "stages.2.19.dwconv.weight", "stages.2.19.dwconv.bias", "stages.2.19.norm.weight", "stages.2.19.norm.bias", "stages.2.19.pwconv1.weight", "stages.2.19.pwconv1.bias", "stages.2.19.pwconv2.weight", "stages.2.19.pwconv2.bias", "stages.2.20.gamma", "stages.2.20.dwconv.weight", "stages.2.20.dwconv.bias", "stages.2.20.norm.weight", "stages.2.20.norm.bias", "stages.2.20.pwconv1.weight", "stages.2.20.pwconv1.bias", "stages.2.20.pwconv2.weight", "stages.2.20.pwconv2.bias", "stages.2.21.gamma", "stages.2.21.dwconv.weight", "stages.2.21.dwconv.bias", "stages.2.21.norm.weight", "stages.2.21.norm.bias", "stages.2.21.pwconv1.weight", "stages.2.21.pwconv1.bias", "stages.2.21.pwconv2.weight", "stages.2.21.pwconv2.bias", "stages.2.22.gamma", "stages.2.22.dwconv.weight", "stages.2.22.dwconv.bias", "stages.2.22.norm.weight", "stages.2.22.norm.bias", "stages.2.22.pwconv1.weight", "stages.2.22.pwconv1.bias", "stages.2.22.pwconv2.weight", "stages.2.22.pwconv2.bias", "stages.2.23.gamma", "stages.2.23.dwconv.weight", "stages.2.23.dwconv.bias", "stages.2.23.norm.weight", "stages.2.23.norm.bias", "stages.2.23.pwconv1.weight", "stages.2.23.pwconv1.bias", "stages.2.23.pwconv2.weight", "stages.2.23.pwconv2.bias", "stages.2.24.gamma", "stages.2.24.dwconv.weight", "stages.2.24.dwconv.bias", "stages.2.24.norm.weight", "stages.2.24.norm.bias", "stages.2.24.pwconv1.weight", "stages.2.24.pwconv1.bias", "stages.2.24.pwconv2.weight", "stages.2.24.pwconv2.bias", "stages.2.25.gamma", "stages.2.25.dwconv.weight", "stages.2.25.dwconv.bias", "stages.2.25.norm.weight", "stages.2.25.norm.bias", "stages.2.25.pwconv1.weight", "stages.2.25.pwconv1.bias", "stages.2.25.pwconv2.weight", "stages.2.25.pwconv2.bias", "stages.2.26.gamma", "stages.2.26.dwconv.weight", "stages.2.26.dwconv.bias", "stages.2.26.norm.weight", "stages.2.26.norm.bias", "stages.2.26.pwconv1.weight", "stages.2.26.pwconv1.bias", "stages.2.26.pwconv2.weight", "stages.2.26.pwconv2.bias", "stages.3.0.gamma", "stages.3.0.dwconv.weight", "stages.3.0.dwconv.bias", "stages.3.0.norm.weight", "stages.3.0.norm.bias", "stages.3.0.pwconv1.weight", "stages.3.0.pwconv1.bias", "stages.3.0.pwconv2.weight", "stages.3.0.pwconv2.bias", "stages.3.1.gamma", "stages.3.1.dwconv.weight", "stages.3.1.dwconv.bias", "stages.3.1.norm.weight", "stages.3.1.norm.bias", "stages.3.1.pwconv1.weight", "stages.3.1.pwconv1.bias", "stages.3.1.pwconv2.weight", "stages.3.1.pwconv2.bias", "stages.3.2.gamma", "stages.3.2.dwconv.weight", "stages.3.2.dwconv.bias", "stages.3.2.norm.weight", "stages.3.2.norm.bias", "stages.3.2.pwconv1.weight", "stages.3.2.pwconv1.bias", "stages.3.2.pwconv2.weight", "stages.3.2.pwconv2.bias", "head.weight", "head.bias".

The names of the layers in your architecture must match the names of the original model in order to load weights

authors: xiaopanchen xraptorgg

repo: facebookresearch/ConvNeXt


text:
DINO fine-tune hyper-parameters

Hi @mathildecaron31 and @TouvronHugo,
thank you for sharing the code to your inspirational work. I am trying out fine-tuning experiments using DINO (teacher) on following downstream datasets: CIFAR10, CIFAR100, INat18, INat19, Flowers, Cars, ImageNet and am unable to replicate the results.
I have seen issue 81 and issue 144 both of which seem related to CIFAR 10. What about other datasets? I tried the mentioned recipe for CIFAR 10 on INat18 and INat19 and was not able to replicate the results (iNat 18: got 71.02 vs reported 72.6 and iNat19: 76.1 vs 78.6).

I trained this using latest DINO code and pytorch 1.10.0+cu113.
Can you please resolve the mystery of downstream datasets fine-tuning hyper-parameters?

Hi @srikar2097,
Thanks for your question.
It is normal the finetuning configuration for CIFAR is different from the one used for iNaturalist.
The hyper-parameters for iNaturalist are:
--batch-size 128 (for each gpu)
--lr 5e-5 (lr before scaling)
--epochs 300
--weight-decay 0.05
--sched cosine
--input-size 224
--repeated-aug
--smoothing 0.1
--warmup-epochs 5
--aa rand-m9-mstd0.5-inc1
--mixup .8
--cutmix 1.0
--remode pixel
--reprob 0.25
--drop-path 0.1
--opt adamw
--warmup-lr 1e-6
This config is for 8 gpus.
It should be taken into account that the version of the librairy used can have a slight impact on the performance and that there is a more important std on iNat than on ImageNet.
Best,
Hugo

Hi @TouvronHugo,
much appreciated for sharing the recipe for iNat. Assuming this is used for both INat18, INat19. Two follow up questions:

When you day "there is a more important std on iNat than on ImageNet" you mean multiple runs on val data gives numbers with large variations?
What fine-tuning recipe's were used for other datasets? CIFAR100, Flowers, Cars, ImageNet (we covered CIFAR10 and INat18, INat19)

thank you again!

repo: facebookresearch/dino


text:
'Connection' object has no attribute 'getstate'

To make it simple, I didn't modify the code in this repo and I was running pointnav task using ppo_pointnav.yaml. The conda environment was newly created. All modules were installed according to Readme.md. However, the following error occurred while initializing the environment:

'Connection' object has no attribute '__getstate__'
  File "/home/m/habitat-lab/habitat/utils/pickle5_multiprocessing.py", line 73, in __getattr__ (Current frame)
    return getattr(self.conn, name)
  File "/home/m/habitat-lab/habitat/core/vector_env.py", line 329, in _spawn_workers
    ps.start()
  File "/home/m/habitat-lab/habitat/core/vector_env.py", line 186, in __init__
    workers_ignore_signals=workers_ignore_signals,
  File "/home/m/habitat-lab/habitat_baselines/utils/env_utils.py", line 105, in construct_envs
    workers_ignore_signals=workers_ignore_signals,
  File "/home/m/habitat-lab/habitat_baselines/rl/ppo/ppo_trainer.py", line 197, in _init_envs
    workers_ignore_signals=is_slurm_batch_job(),
  File "/home/m/habitat-lab/habitat_baselines/rl/ppo/ppo_trainer.py", line 250, in _init_train
    self._init_envs()
  File "/home/m/habitat-lab/habitat_baselines/rl/ppo/ppo_trainer.py", line 712, in train
    self._init_train()
  File "/home/m/habitat-lab/habitat_baselines/run.py", line 60, in execute_exp
    trainer.train()
  File "/home/m/habitat-lab/habitat_baselines/run.py", line 77, in run_exp
    execute_exp(config, run_type)
  File "/home/m/habitat-lab/habitat_baselines/run.py", line 40, in main
    run_exp(**vars(args))
  File "/home/m/habitat-lab/habitat_baselines/run.py", line 81, in <module>
    main()

I haven't seen that before, my guess is that it's something related to VSCode's debugger

authors: erikwijmans zhengzaiyi

repo: facebookresearch/habitat-lab


text:
ETA for next release?

Thanks for the library!
There have been many bug fixes and improvements since that latest release.
Lastest release of pytorchvideo package is from Jan 20, 2022: https://pypi.org/project/pytorchvideo/
Lastest release of pytorchvideo-nightly package is from Feb 9, 2022: https://pypi.org/project/pytorchvideo-nightly/
When will you publish the next release on PYPI?

@kalyanvasudev Any updates on this or the state of the project as a whole?

authors: RafayAK fcakyon

repo: facebookresearch/pytorchvideo


text:
How to add val_dataset

Thanks for your great work for model and training framework both!
We want to add val_dataset and it seems that the framework has implemented it.
We add following changes in sam2.1_hiera_b+_MOSE_finetune.yaml:

Change mode to 'train'.
Copy trainer.data.train to trainer.data.val
change training: false
change file_list_txt

# @package _global_

scratch:
  resolution: 1024
  train_batch_size: 1
  num_train_workers: 10
  num_frames: 8
  max_num_objects: 3
  base_lr: 5.0e-6
  vision_lr: 3.0e-06
  phases_per_epoch: 1
  num_epochs: 40

dataset:
  # PATHS to Dataset
  img_folder: /fsx-onevision/shared/data/academic_vos_data/MOSE/train/JPEGImages # PATH to MOSE JPEGImages folder
  gt_folder: /fsx-onevision/shared/data/academic_vos_data/MOSE/train/Annotations/  # PATH to MOSE Annotations folder
  file_list_txt: training/assets/MOSE_sample_train_list.txt # Optional PATH to filelist containing a subset of videos to be used for training
  multiplier: 2

# Video transforms
vos:
  train_transforms:
    - _target_: training.dataset.transforms.ComposeAPI
      transforms:
        - _target_: training.dataset.transforms.RandomHorizontalFlip
          consistent_transform: True
        - _target_: training.dataset.transforms.RandomAffine
          degrees: 25
          shear: 20
          image_interpolation: bilinear
          consistent_transform: True
        - _target_: training.dataset.transforms.RandomResizeAPI
          sizes: ${scratch.resolution}
          square: true
          consistent_transform: True
        - _target_: training.dataset.transforms.ColorJitter
          consistent_transform: True
          brightness: 0.1
          contrast: 0.03
          saturation: 0.03
          hue: null
        - _target_: training.dataset.transforms.RandomGrayscale
          p: 0.05
          consistent_transform: True
        - _target_: training.dataset.transforms.ColorJitter
          consistent_transform: False
          brightness: 0.1
          contrast: 0.05
          saturation: 0.05
          hue: null
        - _target_: training.dataset.transforms.ToTensorAPI
        - _target_: training.dataset.transforms.NormalizeAPI
          mean: [0.485, 0.456, 0.406]
          std: [0.229, 0.224, 0.225]

trainer:
  _target_: training.trainer.Trainer
  mode: train_only
  max_epochs: ${times:${scratch.num_epochs},${scratch.phases_per_epoch}}
  accelerator: cuda
  seed_value: 123

  model:
    _target_: training.model.sam2.SAM2Train
    image_encoder:
      _target_: sam2.modeling.backbones.image_encoder.ImageEncoder
      scalp: 1
      trunk:
        _target_: sam2.modeling.backbones.hieradet.Hiera
        embed_dim: 112
        num_heads: 2
        drop_path_rate: 0.1
      neck:
        _target_: sam2.modeling.backbones.image_encoder.FpnNeck
        position_encoding:
          _target_: sam2.modeling.position_encoding.PositionEmbeddingSine
          num_pos_feats: 256
          normalize: true
          scale: null
          temperature: 10000
        d_model: 256
        backbone_channel_list: [896, 448, 224, 112]
        fpn_top_down_levels: [2, 3]  # output level 0 and 1 directly use the backbone features
        fpn_interp_model: nearest

    memory_attention:
      _target_: sam2.modeling.memory_attention.MemoryAttention
      d_model: 256
      pos_enc_at_input: true
      layer:
        _target_: sam2.modeling.memory_attention.MemoryAttentionLayer
        activation: relu
        dim_feedforward: 2048
        dropout: 0.1
        pos_enc_at_attn: false
        self_attention:
          _target_: sam2.modeling.sam.transformer.RoPEAttention
          rope_theta: 10000.0
          feat_sizes: [32, 32]
          embedding_dim: 256
          num_heads: 1
          downsample_rate: 1
          dropout: 0.1
        d_model: 256
        pos_enc_at_cross_attn_keys: true
        pos_enc_at_cross_attn_queries: false
        cross_attention:
          _target_: sam2.modeling.sam.transformer.RoPEAttention
          rope_theta: 10000.0
          feat_sizes: [32, 32]
          rope_k_repeat: True
          embedding_dim: 256
          num_heads: 1
          downsample_rate: 1
          dropout: 0.1
          kv_in_dim: 64
      num_layers: 4

    memory_encoder:
        _target_: sam2.modeling.memory_encoder.MemoryEncoder
        out_dim: 64
        position_encoding:
          _target_: sam2.modeling.position_encoding.PositionEmbeddingSine
          num_pos_feats: 64
          normalize: true
          scale: null
          temperature: 10000
        mask_downsampler:
          _target_: sam2.modeling.memory_encoder.MaskDownSampler
          kernel_size: 3
          stride: 2
          padding: 1
        fuser:
          _target_: sam2.modeling.memory_encoder.Fuser
          layer:
            _target_: sam2.modeling.memory_encoder.CXBlock
            dim: 256
            kernel_size: 7
            padding: 3
            layer_scale_init_value: 1e-6
            use_dwconv: True  # depth-wise convs
          num_layers: 2

    num_maskmem: 7
    image_size: ${scratch.resolution}
    # apply scaled sigmoid on mask logits for memory encoder, and directly feed input mask as output mask
    sigmoid_scale_for_mem_enc: 20.0
    sigmoid_bias_for_mem_enc: -10.0
    use_mask_input_as_output_without_sam: true
    # Memory
    directly_add_no_mem_embed: true
    no_obj_embed_spatial: true
    # use high-resolution feature map in the SAM mask decoder
    use_high_res_features_in_sam: true
    # output 3 masks on the first click on initial conditioning frames
    multimask_output_in_sam: true
    # SAM heads
    iou_prediction_use_sigmoid: True
    # cross-attend to object pointers from other frames (based on SAM output tokens) in the encoder
    use_obj_ptrs_in_encoder: true
    add_tpos_enc_to_obj_ptrs: true
    proj_tpos_enc_in_obj_ptrs: true
    use_signed_tpos_enc_to_obj_ptrs: true
    only_obj_ptrs_in_the_past_for_eval: true
    # object occlusion prediction
    pred_obj_scores: true
    pred_obj_scores_mlp: true
    fixed_no_obj_ptr: true
    # multimask tracking settings
    multimask_output_for_tracking: true
    use_multimask_token_for_obj_ptr: true
    multimask_min_pt_num: 0
    multimask_max_pt_num: 1
    use_mlp_for_obj_ptr_proj: true
    # Compilation flag
    # compile_image_encoder: False

    ####### Training specific params #######
    # box/point input and corrections
    prob_to_use_pt_input_for_train: 0.5
    prob_to_use_pt_input_for_eval: 0.0
    prob_to_use_box_input_for_train: 0.5  # 0.5*0.5 = 0.25 prob to use box instead of points
    prob_to_use_box_input_for_eval: 0.0
    prob_to_sample_from_gt_for_train: 0.1  # with a small prob, sampling correction points from GT mask instead of prediction errors
    num_frames_to_correct_for_train: 2  # iteratively sample on random 1~2 frames (always include the first frame)
    num_frames_to_correct_for_eval: 1  # only iteratively sample on first frame
    rand_frames_to_correct_for_train: True  # random #init-cond-frame ~ 2
    add_all_frames_to_correct_as_cond: True  # when a frame receives a correction click, it becomes a conditioning frame (even if it's not initially a conditioning frame)
    # maximum 2 initial conditioning frames
    num_init_cond_frames_for_train: 2
    rand_init_cond_frames_for_train: True  # random 1~2
    num_correction_pt_per_frame: 7
    use_act_ckpt_iterative_pt_sampling: false

    num_init_cond_frames_for_eval: 1  # only mask on the first frame
    forward_backbone_per_frame_for_eval: True

  data:
    train:
      _target_: training.dataset.sam2_datasets.TorchTrainMixedDataset
      phases_per_epoch: ${scratch.phases_per_epoch}
      batch_sizes:
        - ${scratch.train_batch_size}

      datasets:
        - _target_: training.dataset.utils.RepeatFactorWrapper
          dataset:
            _target_: training.dataset.utils.ConcatDataset
            datasets:
            - _target_: training.dataset.vos_dataset.VOSDataset
              transforms: ${vos.train_transforms}
              training: true
              video_dataset:
                _target_: training.dataset.vos_raw_dataset.PNGRawDataset
                img_folder: ${dataset.img_folder}
                gt_folder: ${dataset.gt_folder}
                file_list_txt: training/assets/MOSE_sample_train_list.txt
              sampler:
                _target_: training.dataset.vos_sampler.RandomUniformSampler
                num_frames: ${scratch.num_frames}
                max_num_objects: ${scratch.max_num_objects}
              multiplier: ${dataset.multiplier}
      shuffle: True
      num_workers: ${scratch.num_train_workers}
      pin_memory: True
      drop_last: True
      collate_fn:
        _target_: training.utils.data_utils.collate_fn
        _partial_: true
        dict_key: all
    val:
      _target_: training.dataset.sam2_datasets.TorchTrainMixedDataset
      phases_per_epoch: ${scratch.phases_per_epoch}
      batch_sizes:
        - ${scratch.train_batch_size}

      datasets:
        - _target_: training.dataset.utils.RepeatFactorWrapper
          dataset:
            _target_: training.dataset.utils.ConcatDataset
            datasets:
              - _target_: training.dataset.vos_dataset.VOSDataset
                transforms: ${vos.train_transforms}
                training: false
                video_dataset:
                  _target_: training.dataset.vos_raw_dataset.PNGRawDataset
                  img_folder: ${dataset.img_folder}
                  gt_folder: ${dataset.gt_folder}
                  file_list_txt: training/assets/MOSE_sample_val_list.txt
                sampler:
                  _target_: training.dataset.vos_sampler.RandomUniformSampler
                  num_frames: ${scratch.num_frames}
                  max_num_objects: ${scratch.max_num_objects}
                multiplier: ${dataset.multiplier}
      shuffle: True
      num_workers: ${scratch.num_train_workers}
      pin_memory: True
      drop_last: True
      collate_fn:
        _target_: training.utils.data_utils.collate_fn
        _partial_: true
        dict_key: all

  optim:
    amp:
      enabled: True
      amp_dtype: bfloat16

    optimizer:
      _target_: torch.optim.AdamW

    gradient_clip:
      _target_: training.optimizer.GradientClipper
      max_norm: 0.1
      norm_type: 2

    param_group_modifiers:
      - _target_: training.optimizer.layer_decay_param_modifier
        _partial_: True
        layer_decay_value: 0.9
        apply_to: 'image_encoder.trunk'
        overrides:
          - pattern: '*pos_embed*'
            value: 1.0

    options:
      lr:
        - scheduler:
            _target_: fvcore.common.param_scheduler.CosineParamScheduler
            start_value: ${scratch.base_lr}
            end_value: ${divide:${scratch.base_lr},10}
        - scheduler:
            _target_: fvcore.common.param_scheduler.CosineParamScheduler
            start_value: ${scratch.vision_lr}
            end_value: ${divide:${scratch.vision_lr},10}
          param_names:
            - 'image_encoder.*'
      weight_decay:
        - scheduler:
            _target_: fvcore.common.param_scheduler.ConstantParamScheduler
            value: 0.1
        - scheduler:
            _target_: fvcore.common.param_scheduler.ConstantParamScheduler
            value: 0.0
          param_names:
            - '*bias*'
          module_cls_names: ['torch.nn.LayerNorm']

  loss:
    all:
      _target_: training.loss_fns.MultiStepMultiMasksAndIous
      weight_dict:
        loss_mask: 20
        loss_dice: 1
        loss_iou: 1
        loss_class: 1
      supervise_all_iou: true
      iou_use_l1_loss: true
      pred_obj_scores: true
      focal_gamma_obj_score: 0.0
      focal_alpha_obj_score: -1.0

  distributed:
    backend: nccl
    find_unused_parameters: True

  logging:
    tensorboard_writer:
      _target_: training.utils.logger.make_tensorboard_logger
      log_dir:  ${launcher.experiment_log_dir}/tensorboard
      flush_secs: 120
      should_log: True
    log_dir: ${launcher.experiment_log_dir}/logs
    log_freq: 10

  # initialize from a SAM 2 checkpoint
  checkpoint:
    save_dir: ${launcher.experiment_log_dir}/checkpoints
    save_freq: 0 # 0 only last checkpoint is saved.
    model_weight_initializer:
      _partial_: True
      _target_: training.utils.checkpoint_utils.load_state_dict_into_model
      strict: True
      ignore_unexpected_keys: null
      ignore_missing_keys: null

      state_dict:
        _target_: training.utils.checkpoint_utils.load_checkpoint_and_apply_kernels
        checkpoint_path: ./checkpoints/sam2.1_hiera_base_plus.pt # PATH to SAM 2.1 checkpoint
        ckpt_state_dict_keys: ['model']

launcher:
  num_nodes: 1
  gpus_per_node: 8
  experiment_log_dir: null # Path to log directory, defaults to ./sam2_logs/${config_name}

# SLURM args if running on a cluster
submitit:
  partition: null
  account: null
  qos: null
  cpus_per_task: 10
  use_cluster: false
  timeout_hour: 24
  name: null
  port_range: [10000, 65000]

However, it mistakes when assert in https://github.com/z-jiaming/sam2_plus/blob/aa9b8722d0585b661ded4b3dff1bd103540554ae/training/trainer.py#L973
What should I do to fix it, such as fix trainer.loss?

Hi @z-jiaming, we don't have support for val_dataset  in this release (our internal version of validation depends on Meta internal infrastructure). The vos_inference.py at https://github.com/facebookresearch/sam2/tree/main/tools is the adapted public version that could run in open-source environments.
For validation, you can use vos_inference.py to save the predictions on VOS datasets (and possibly splitting the dataset video list into per-GPU chunks to save PNG outputs using multiple GPUs), and then use the eval toolkits on each dataset to get the results.

Thank you for your quick response.
I saw the val_datasetand assumed that the current framework had implemented it.
Looking forward to your subsequent implementation!

authors: ronghanghu z-jiaming


text:
fix: symbolic links to sam2.1

This fixes the symbolic links for the model configuration as mentioned here.
This will fix the symbolic links for SAM2.1 when installing SAM2 using pip.

Hi @pevers, these symbolic links are meant to point to older SAM 2 checkpoints and were added to not break backward compatibility with older versions of the repo. The SAM 2.1 configs can be found in sam2/configs/sam2.1

Hi @haithamkhedr , thanks makes sense. The configuration is not included in the setup.py. But of course can be loaded separately if needed.

They're not included as a separate package, but should be part of sam2 package.

They're not included as a separate package, but should be part of sam2 package.

Are you sure? If I look at this https://github.com/facebookresearch/sam2/blob/52198ead0eb13ae8270bea6ca768ef175f5bf167/setup.py#L168
It seems that it only includes the root folder .YAML files, which are the SAM2.0 configurations. In the site-packages/sam2_configs folder I only find these.

Hi @pevers, to reference SAM 2.1 configs, please refer to their path relative to the sam2 package (which is included in the PATHs after your installation). An example would be:
from sam2.build_sam import build_sam2
model = build_sam2("configs/sam2.1/sam2.1_hiera_l", ckpt_path="./checkpoints/sam2.1_hiera_large.pt", device ="cpu")

Please let me know if this doesn't work on your side.

I see, in that case I misunderstood your original question then. Will submit a fix for the non-editable scenario. Thanks for pointing it out!

authors: haithamkhedr pevers


text:
console.log in code

Hi there!
Thank you for your job!
I'm just wondering about console.log (for ex: https://github.com/facing-dev/vue-facing-decorator/blob/7567cb0bf961ababcb36391840b5f1e6c2da52a5/src/option/ref.ts#L22)
It's just mistake that you put it console.log in published lib ?

Sorry for that, I will check and clear the package.

v2.1.5 released

repo: facing-dev/vue-facing-decorator


text:
🛑 biofabrica.misiones.gob.ar is down

In 4713345, biofabrica.misiones.gob.ar (https://biofabrica.misiones.gob.ar/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: biofabrica.misiones.gob.ar is back up in 99fd5fd.


text:
Validation of Pod, filename, and directory names

Hi there,
I have a question regarding the naming conventions for pods, directories, and files within the project. I am wondering what attributes should be considered when determining the name, maximum size, and allowed characters for these items. Additionally, I am curious about the maximum nesting of directories and files.
As far as I understand, it is important that these naming conventions be compatible with popular operating systems. Is there a standard that covers these requirements?
Any thoughts or suggestions would be greatly appreciated. Thank you!
@nugaon @tfius @molekilla @crtahlin @asabya

@asabya any kind of limits stem from FUSE compatilibity, to work on all OSes?

basically we are fucked... and we should take the best of them all, most versatile, break things and make them as they should be

Simply put, file or folder names can have these character sets
Uppercase A to Z
Lowercase a to z
Numbers 0 to 9
Period (.)
Underscore (_)
Hyphen (-)

taken from here.
We follow / on POSIX for path separator.

authors: IgorShadurin asabya crtahlin tfius

repo: fairDataSociety/fdp-storage


text:
Using Falco with Elasticsearch

/area documentation
What would you like to be added:
On the Falco slack channel, @tembleking shared a very nice deck he is using to present how to send Falco events to Elasticsearch. I think we want to have that same content in the form of a documentation page that our users can follow when installing Falco.
Presentation here: https://docs.google.com/presentation/d/1vC5PIHMjh_x8BsWRou-g7pGHe6f6lOK2aeSbIAVh7B4/edit?usp=sharing
Why is this needed:
Because many users want a place to aggregate their events, and this can be a good opportunity to simplify their lives by providing a tutorial.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle rotten

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://github.com/falcosecurity/community.
/close

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://github.com/falcosecurity/community.
/close

@poiana: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://github.com/falcosecurity/community.
/close

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@poiana: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://github.com/falcosecurity/community.
/close

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

authors: fntlnz poiana

repo: falcosecurity/falco-website


text:
Add KafkaRest as new output for Falco Sidekick

Signed-off-by: Engin Diri engin.diri@mail.schwarz

What type of PR is this?
Fixes #262

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind feature

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area build

/area config

/area outputs

/area tests

What this PR does / why we need it:
The Kafka REST Proxy provides a RESTful interface to a Apache Kafka cluster, making it easy to produce and consume messages, view the state of the cluster, and perform administrative actions without using the native Kafka protocol or clients.
Which issue(s) this PR fixes:  #262

Fixes #262
Special notes for your reviewer:

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: dirien
To complete the pull request process, please assign nibalizer after the PR has been reviewed.
You can assign the PR to them by writing /assign @nibalizer in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Welcome @dirien! It looks like this is your first PR to falcosecurity/falcosidekick 🎉

Thanks for your pull request. Before we can look at it, you'll need to add a 'DCO signoff' to your commits.
:memo: Please follow instructions in the contributing guide to update your commits with the DCO
Full details of the Developer Certificate of Origin can be found at developercertificate.org.
The list of commits missing DCO signoff:

5a0e7ce Add KafkaRest as new output for Falco Sidekick

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Thanks for your pull request. Before we can look at it, you'll need to add a 'DCO signoff' to your commits.
:memo: Please follow instructions in the contributing guide to update your commits with the DCO
Full details of the Developer Certificate of Origin can be found at developercertificate.org.
The list of commits missing DCO signoff:

e4bb41f Add KafkaRest as new output for Falco Sidekick

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

Thank you very much, this is a very valuable PR, with a really nice description. I'll review asap but a first look tells me it's already really good.
👍

@dirien: Adding label do-not-merge/contains-merge-commits because PR contains merge commits, which are not allowed in this repository.
Use git rebase to reapply your commits on top of the target branch. Detailed instructions for doing so can be found here.

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Thanks for your pull request. Before we can look at it, you'll need to add a 'DCO signoff' to your commits.
:memo: Please follow instructions in the contributing guide to update your commits with the DCO
Full details of the Developer Certificate of Origin can be found at developercertificate.org.
The list of commits missing DCO signoff:

016781b Merge branch 'falcosecurity:master' into kafkarest

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by: dirien
To complete the pull request process, please ask for approval from issif after the PR has been reviewed.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@Issif, Thanks for the feedback. I changed everything according to your suggestions. looking forward. 😄

@Issif, done :) during the  Yandex rebase it must slipped through 😓

LGTM label has been added.  Git tree hash: c9f7536a6b043e7ccb6bc9a8b98d10a4f984ab63

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: dirien, Issif
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [Issif]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

authors: Issif dirien poiana

repo: falcosecurity/falcosidekick


text:
[4/n] new(libscap): platform abstraction support: clean up savefiles a little bit

What type of PR is this?
/kind cleanup
/kind design
Any specific area of the project related to this PR?
/area libscap
What this PR does / why we need it:
This is a "silence before the storm" episode in the platform saga: a few cleanups in savefile code. One cherry on top is that we can finally remove m_mode from the scap handle (there's nothing that depends on it any more, since everything should be properly abstracted via engines and platforms by now)
Special notes for your reviewer:
This builds on top of #1042 so looks huge right now. BTW I expect n>=6 at least but I hope I won't go into double digits :)
Does this PR introduce a user-facing change?:
NONE

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: gnosek
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [gnosek]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: gnosek, incertum
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [gnosek,incertum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

LGTM label has been added.  Git tree hash: b8d1c5a0a789f8dea23d77f0bd7812919416e06b

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP, gnosek, incertum
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP,gnosek,incertum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

authors: gnosek poiana

repo: falcosecurity/libs


text:
refactor(cri): restructure CRI API (improve robustness, clarity and maintainability)

What type of PR is this?

Uncomment one (or more) /kind <> lines:

/kind bug

/kind cleanup

/kind design

/kind documentation

/kind failing-test

/kind feature

Any specific area of the project related to this PR?

Uncomment one (or more) /area <> lines:

/area API-version

/area build

/area CI

/area driver-kmod

/area driver-bpf

/area driver-modern-bpf

/area libscap-engine-bpf

/area libscap-engine-gvisor

/area libscap-engine-kmod

/area libscap-engine-modern-bpf

/area libscap-engine-nodriver

/area libscap-engine-noop

/area libscap-engine-source-plugin

/area libscap-engine-savefile

/area libscap

/area libpman

/area libsinsp

/area tests

/area proposals

Does this PR require a change in the driver versions?

/version driver-API-version-major

/version driver-API-version-minor

/version driver-API-version-patch

/version driver-SCHEMA-version-major

/version driver-SCHEMA-version-minor

/version driver-SCHEMA-version-patch

What this PR does / why we need it:
I have added very detailed comments to the code diff re the implemented changes.
At a high level the refactor achieves the following:

Add robustness wrt to metadata retrieved from the PodSandboxStatusResponse
Improve technical clarity and maintainability (flow and steps are now easy to follow and understand)
Remove some redundancies
Expand unit tests (further improving technical clarity and maintainability)
This PR does NOT add any extra API calls that weren't done prior to this refactor

Which issue(s) this PR fixes:
https://github.com/falcosecurity/libs/issues/1589
Why this PR matters:

Currently, we are not handling or enriching pod sandbox container processes correctly. This means adopters can falsely report that a container image is missing, when in fact it may be legitimate. Once this PR is merged, adopters will have the option to compare container and pod sandbox IDs. If they are equal, they will know that the container images will never exist.
Additionally, I would rate this code cleanup as highly crucial for slightly optimizing the CRI API lookups. More importantly, it will make them unit-testable and improve transparency and maintainability in the long run.

Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?:

NONE

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: incertum
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [incertum]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@gnosek may I ask for a first-pass review from you? Thanks a bunch in advance?

If ok with everyone
/milestone 0.15.0

@falcosecurity/libs-maintainers kindly checking in on the status of this PR. Thank you.

Please note that this PR has been open for 2 months now without receiving any review whatsoever. During this time, it appears that our focus has been primarily directed towards prioritizing and integrating similarly complex code alterations.
Why this PR matters:

Currently, we are not handling or enriching pod sandbox container processes correctly. This means adopters can falsely report that a container image is missing, when in fact it may be legitimate. Once this PR is merged, adopters will have the option to compare container and pod sandbox IDs. If they are equal, they will know that the container images will never exist.
Additionally, I would rate this code cleanup as highly crucial for slightly optimizing the CRI API lookups. More importantly, it will make them unit-testable and improve transparency and maintainability in the long run.

PR LGTM, but i am in no way an expert on this part of the code :)
I really like the refactor though, that is the most important thing.
/cc @gnosek :pray:

On my side, I will prioritize this once https://github.com/falcosecurity/libs/pull/1595 gets merged (hopefully soon).
/assign
I still want to confirm that this is targetted for libs 0.15

@leogr this PR should be the next one. Rebased and added another commit on top in an attempt to improve comments and naming and code organization even more.
If it help, happy to schedule a call to more interactively review this PR, just let me know, thanks a bunch!

@leogr this PR should be the next one. Rebased and added another commit on top in an attempt to improve comments and naming and code organization even more.
If it help, happy to schedule a call to more interactively review this PR, just let me know, thanks a bunch!

I'm working on it. @therealbobo already did some tests, and everything worked properly (@therealbobo do you confirm?)/
cc @falcosecurity/libs-maintainers for visibility

I'm working on it. @therealbobo already did some tests, and everything worked properly (@therealbobo do you confirm?)/

Thank you! While it's a terrible diff, I believe this PR is less risky, plus at least it's backed up my unit tests (acknowledging some limitations w/ mock CRI response unit tests, but still better than before when we had nothing).
Perhaps, double check and pay close attention to all conditional checks etc.
Plus plz let me know if the filterchecks fallback mechanisms are ok with you:

We now consistently first try to get all pod sandbox related fields from the container
Falling back to fetching the separate pod sandbox container from the cache (which is now also fully populated, it was not the case before). The cniresult string was chosen as check for it as it should always be there in Kubernetes, while labels could legitimately be empty who knows. perhaps down the road there could be another improvement PR< but I wouldn't aim for absolutely perfect right now either.
The fallback could be important if for example the pod sandbox CRI call failed after a successful container CRI calls. Probably not very common, but it could happen since we do not mark the overall lookups within parse as unsuccessful when the container lookups succeeded but the subsequent additional pod sandbox CRI call failed.

@leogr I did! I also see new commits. Let me check! 👀

@leogr I did! I also see new commits. Let me check! 👀

Only the last commit is new, else conflict-free rebase. The last commit is merely one more renaming and adding more comments pass etc, no changes really.

Since we are doing this, let's do it right: I pushed one more commit to really be super consistent also in the order of calls in the CRI code base (plus few more comments). Ok to call me crazy on those details, but as said hopefully we don't need to touch this again for a bit.

Since we are doing this, let's do it right: I pushed one more commit to really be super consistent also in the order of calls in the CRI code base (plus few more comments). Ok to call me crazy on those details, but as said hopefully we don't need to touch this again for a bit.

:+1: Testing this again (in progress).

LGTM label has been added.  Git tree hash: 0087a2cc73d9fc24d2e741c5bb350dc12e4c7e33

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: incertum, leogr
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [incertum,leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: FedeDP, incertum, leogr
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [FedeDP,incertum,leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

authors: FedeDP incertum leogr poiana therealbobo