text:
React-native bundle command generated old main.jsbundle file

This issue is driving me insane. Every time I run the bundle command it bundles an old version that I previously bundled. I'm not sure where this old code is being cached and loaded from. Running latest version. Anyone?

Check the time stamp on main.jsbundle after you bundle. Does it change?

Similar issue: https://github.com/facebook/react-native/issues/4268

I have the same issue.. It's stuck on the old one..

I've been facing this issue as well. Anybody found any solution for this?

I found the problem
Run the command in a terminal: ps ax | grep react-native
Kill all processes and run bundle script again.

@regeda didn't work for me, even rebooting the computer

I found a workaround. I renamed the bundle the main2.jsbundle and changed the reference in appdelegate. This seems to work. For some reason it seems to only cache main.jsbundle

Ran into the same issue. Cleaning in Xcode before doing every build worked for us.

@bellicose100xp - Could you try @jjshammas suggestion and report back to us if it did the trick?

You can also try clearing the packager cache and see if it helps rm -rf $TMPDIR/react-*

@christopherdro @jjshammas suggestion didn't work but @slavik0329 suggestion did. I changed the name of the bundle to main2.jsbundle and changed the reference in appDelegate.m file

so i changed the name back to main.jsbundle and builds are working fine without doing anything else.. I don't really know what fixed it but I've tried all of the suggestions listed in this thread..

@bellicose100xp This might sound silly, but have you tried restarting your computer?
Almost every time I've ran into issues that kept me scratching my head, a simple restart did the trick.

@regeda's trick seemed to work for me!

In my case, I had stupidly added a manually generated main.jsbundle to my existing project via XCode (since it did not have main.jsbundle like newly created projects) which automatically added it to the Copy Bundle Resources step in Build Phases (I had no idea that it would do that)
I removed main.jsbundle from the Copy Bundle Resources step, removed it from my project and cleared the packager's cache (rm -rf $TMPDIR/react-*)
Now everything works right.
It's a pretty complicated build system.

@christopherdro 's solution of restarting my computer allows you to make a change to the JS and bundle once. After more than one time, it keeps reusing the already generated main.js bundle. So if I want to make anymore changes after bundling once, I need to restart my computer again.
Has there been any progress on this?

rm -rf $TMPDIR/react-* did the trick for me. thanks @satya164

Having the same issue on 0.15.0, resolved with @satya164's suggestion as well.
rm -rf $TMPDIR/react-*

me too, have to do the above

Running with option --resetCache true works for me.

Is seems that there are workarounds were identified, but the underlying issue was not fixes. Unless there is a reason I am missing to reuse the cached main.jsbundle? That seems very odd since the xcode project takes care of generating the bundle and that is hidden from the user so there is no way for developers to know that a cache version is being used. It seems like either this is still a bug or --resetCache true should be added to react-native/packager/react-native-xcode.sh.
The linked issue #4268 was closed in favor of this bug. Should this be reactivated or is there another bug tacking a fix?

I'm having the same issue with latets react-native, i think this one should be re-opened

Running rm -rf $TMPDIR/react-* did the trick which shows it's a react-native issue and should be fixed accordingly. On a previous app I had accidently submitted an old version of my app because of that.

God, I'm glad I ran across this thread. I've been pulling my hair out doing cleans, build cleans, touch, etc. and nothing worked before except literally generating an all-new XCode project.
@satya164's suggestion worked like a charm. Also on 0.15.0.

Been there. --reset-cache really needs to be added to the build script. Can this be reopened to the original point that the build command generates an outdated bundle?

I'm not sure why this ticket is closed, the issue happens to me on Mac OSX on 0.17.
My method of getting around is to make my 'Bundle React Native code and images' Build Phase look like this:
if [ "${CONFIGURATION}" = "Release" ]; then
# https://github.com/facebook/react-native/issues/4289
echo "DELETING CACHED DATA FOR RELEASE"
rm -rf $TMPDIR/react-*
fi

../node_modules/react-native/packager/react-native-xcode.sh

Hacky, but so far it works.

Can not work at all. Every time I change & save a file - I get old results like I changed nothing. Working with ios emulator.

I vote to reopen this issue.

I too am having this issue after upgrading to RN 0.23.1 on Windows/Android.  I find I have to uninstall the App on Genymotion and rebuild in order for it take in my changes.
It's weird though, some changes are reflected but some others are not (even though I make changes in the SAME file).

I was also facing this problem on IOS and Windows 10. Below are the commands i use to create signed apk.
react-native bundle --platform android --dev false --entry-file index.android.js   --bundle-output android/app/src/main/assets/index.android.bundle   --assets-dest android/app/src/main/res/
Mac
watchman watch-del-all && rm -rf node_modules/ && npm cache clean && npm install && npm start -- --reset-cache
Windows
Permanent delete node_modules
npm cache clean –force
npm install
npm start -- --reset-cache
Open SDK -> Click on rebuild -> Fixed gradle build issue -> click on rebuild again -> generate signed apk.

Still having the same bug with the following config:

Environment:
OS: macOS Sierra 10.12.4
Node: 8.1.2
Yarn: 0.24.6
npm: 5.5.1
Watchman: 4.9.0
Xcode: Xcode 8.3.3 Build version 8E3004b
Android Studio: Not Found
Packages: (wanted => installed)
react: 16.2.0 => 16.2.0
react-native: 0.52.2 => 0.52.2

I tried to run this:
$: react-native start --reset-cache

But it seems the option has no effect.

authors: BilalBudhani Intellicode Nyte7 VansonLeung ajwhite akaiworld bellicose100xp benvium cglacet christopherdro ericvera genki ggarg2 idibidiart j-wang jaygarcia jjshammas normanrz olivierlesnicki pangers radiovisual regeda satya164 slavik0329


text:
[BREAKING] Add blob implementation with WebSocket integration

This is the first PR from a series of PRs @grabbou and me will make to add blob support to React Native. The next PR will include blob support for XMLHttpRequest.
I'd like to get this merged with minimal changes to preserve the attribution. My next PR can contain bigger changes.
Motivation
Blobs are used to transfer binary data between server and client. Currently React Native lacks a way to deal with binary data. The only thing that comes close is uploading files through a URI.
Current workarounds to transfer binary data includes encoding and decoding them to base64 and and transferring them as string, which is not ideal, since it increases the payload size and the whole payload needs to be sent via the bridge every time changes are made.
The PR adds a way to deal with blobs via a new native module. The blob is constructed on the native side and the data never needs to pass through the bridge. Currently the only way to create a blob is to receive a blob from the server via websocket.
The PR is largely a direct port of https://github.com/silklabs/silk/tree/master/react-native-blobs by @philikon into RN (with changes to integrate with RN), and attributed as such.

Note: This is a breaking change for all people running iOS without CocoaPods. You will have to manually add RCTBlob.xcodeproj to your Libraries and then, add it to Build Phases. Just follow the process of manual linking. We'll also need to document this process in the release notes.

Related discussion - https://github.com/facebook/react-native/issues/11103
Known issues

Image can't show image when URL.createObjectURL is used with large images on Android

Test plan
The websocket integration can be tested via a simple server,
const fs = require('fs');
const http = require('http');

const WebSocketServer = require('ws').Server;

const wss = new WebSocketServer({
  server: http.createServer().listen(7232),
});

wss.on('connection', (ws) => {
  ws.on('message', (d) => {
    console.log(d);
  });

  ws.send(fs.readFileSync('./some-file'));
});

Then on the client,
const ws = this.ws = new WebSocket('ws://localhost:7232');

ws.binaryType = 'blob';

ws.onerror = (error) => {
  console.error(error);
};

ws.onmessage = (e) => {
  ws.send(e.data);
};

cc @brentvatne @ide

Also if you want to test it, here's demo: https://github.com/callstack-io/react-native-blob-test

cc @bestander need some help on writing tests for this :D

I'll update the example app layout later to fix the failing iOS tests

Also note that some of the changes to .xcodeproj are because of other changes made outside of this pull request. I've only added RCTBlob and libRCTBlob.a. Looks like someone changed native structure in master but forgot to open .xcodeproj.

@bestander any idea why tests are failing?

Test environment broke in Travis but was fixed earlier today.
Circle has e2e tests failing, we are investigating why but can't identify the reason yet.
Help is appreciated :)

@grabbou iOS tests are failing for some reason. looks like the same error I got when building the example app.

I think we should have unit or integration tests that cover this, too - it's a really big chunk of functionality so it seems quite testworthy.

@lacker I'll look into how to write tests for this. Will likely need help :D

If you go ahead and import https://github.com/facebook/react-native/pull/11433 , you'll have a basic WebSocket test infrastructure that you can add blob tests to.

iOS tests fixed - the issue started to happening after the rebase, that includes the breaking change of imports on the native side. Fixed.

Me and @Kureev were talking about this yesterday about the current behaviour of close. Currently it'll release the blob for all sliced blobs in addition to the parent blob. It's not probably what we want and likely not spec-compliant.
I'll fix this behaviour in my next PR. But there is one problem with this. Libraries like Firebase slice the blobs internally. There's no way to close those blob slices since we don't have a reference to them. The only way I can think of is to have something like BlobManager.closeSlices(blob) to achieve this. Any suggestions will be nice.

iOS and tvOS should be passing now on Travis.

I propose to use BlobManager.close(blob). It can be any kind of blob: a sliced or an original one.

What does the spec say about close?  It doesn't look like any browsers support it yet (source: "Browser compatibility table in MDN docs".

@brentvatne unfortunately I couldn't find much documentation on the behaviour of close. and since browsers haven't implemented it, I couldn't test it either. But it's likely that closing the parent blob when sliced blob is closed and vice-versa is non-spec compliant, since it doesn't sound very intuitive.

I would expect slices to provide the illusion that memory is copied (whether or not that actually happens is up to the implementation -- if you make a small slice you might want to not copy at first but then make a copy if the large parent is later freed -- this is all below the level of abstraction). This matches my reading of the spec https://www.w3.org/TR/FileAPI/#dfn-close
Does the browser free Blobs when they are GC'd? If that's the case we might want to build a synchronous implementation that interacts with the native JSC API and frees native memory when JS GCs it. This would also pave the way towards moving the implementation to the TypedArray API in iOS 10 for even more efficient Blobs -- this is what we do in our OpenGL implementation.
My main concern is that we're aiming to implement an existing API so that people can use existing code but we might not be providing existing semantics around freeing memory. Or if I'm wrong about this that solves the problem too =P

@ide

I would expect slices to provide the illusion that memory is copied (whether or not that actually happens is up to the implementation -- if you make a small slice you might want to not copy at first but then make a copy if the large parent is later freed -- this is all below the level of abstraction).

Yeah, I'm planning to implement this behaviour in my next PR

Does the browser free Blobs when they are GC'd? If that's the case we might want to build a synchronous implementation that interacts with the native JSC API and frees native memory when JS GCs it.

I think so. I'm not very familiar with JSC and C++ to be able to implement this unfortunately. Can someone help?

My main concern is that we're aiming to implement an existing API so that people can use existing code but we might not be providing existing semantics around freeing memory.

Yeah, we should totally provide existing semantics.

@evv only breaking change is that iOS users have to manually link this for the app to compile. The behaviour stays the same.

I see, so that should be covered by react-native-git-upgrade or react-native upgrade.

I have applied some of the tweaks, mainly related to the xcodeproj. Will continue with the rest now.

Any updates on this? Blob upload via XMLHttpRequest would be super helpful.

@SpecialDOS I'm waiting for someone from the core team to review the Java code. css @lacker

Hi @satya164, I started looking at the Java changes and will finish reviewing today.

Thanks for the review Martin. I probably forgot some stuff because the initial code was not written by me and it's been quite some time, but will try to get it upto speed soon.
This will be a follow up to this PR https://github.com/facebook/react-native/pull/11573

❤  @mkonicek

This is broken right now after rebase. Will get back to it very soon™.

hi all, is there any eta on adding / merging this PR ? we REALLY need Blob support for our project that is close to beta (govt. funded RN app for schools in aust.) and very much would like to see this in Expo too. Thanks for your help and hard work!

There's no ETA, but I'll be working on it this week.

:(

any update on when this PR might land?

I need to rebase and wrap up iOS part - I've been super busy because of
many conferences recently, but it's on my list and trying to get to it as
soon as possible.
Will update the thread as soon as there's progress.
On Tue, 23 May 2017 at 17:06 Lee Wilson notifications@github.com wrote:

any update on when this PR might land?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/facebook/react-native/pull/11417#issuecomment-303427718,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ACWcxh0_2jdU0dSc8p9FXKYHGUlb-40Iks5r8vXkgaJpZM4LKWpp
.

iOS part is rebased. I've tried to rebase Android but pretty sure forgot to factor in recent okhttp3 upgrade which likely caused some imports to fail. I've let @satya164 know of the issues so we should get it green pretty soon.

The PR has been updated and ready for review.

@sahrens will check asap!

Hey @sahrens. Fixed what I could. The tests that are still failing are failing on master too :(

Woohoo, this finally landed (sorry for the delay!) I changed the approach slightly so this is all opt-in now. Please let me know if you have any feedback.

source: gharchive/pull-request

authors: Kureev SpecialDOS ajonno bestander brentvatne dlowder-salesforce ericvicenti grabbou ide indefinitelee javache lacker mkonicek satya164 sonaye

repo: facebook/react-native

license: MIT

license_type: permissive

license_source: github-api


text:
add asdf-vm support in react-native-xcode.sh

About asdf-vm: https://asdf-vm.com/
Summary
This PR add support for asdf-vm during calling scripts/react-native-xcode.sh in Bundle React Native code and images build phrase
Changelog
[iOS] [Added] - add asdf-vm support in react-native-xcode.sh
Test Plan
Xcode is able to complete Bundle React Native code and images build phrase without any errors when node is installed by asdf-vm

I guess this and https://github.com/facebook/react-native/pull/31317 can go together. I see some tiny differences, -x, -s, -f not used the same way.

also, why is this script for xcode not using the find-node script??!

HI @pastleo, can we move this change to find-node.sh alongside the other environment manager code?

also, why is this script for xcode not using the find-node script??! The change there should be enough for all the things within RN to find the right node without duplicating code.

because this PR was created (2020/10) before scripts/find-node.sh (2021/1, if the commit history is correct)

HI @pastleo, can we move this change to find-node.sh alongside the other environment manager code?

sure, please do, and maybe I should close this PR?

@pastleo, sorry, I meant do you mind making the change? :)

@pastleo, sorry, I meant do you mind making the change? :)

OK, but I wasn't developing RN for quite a while now, I will make the change when I have time (and setup a project to test as well)
Should I open new PR? or just modify this one?

@charlesbdudley I just moved the change to find-node.sh, also created a new RN project to test and it is working

authors: charlesbdudley pastleo pvinis


text:
Use NDK 23 only for Windows users.

Summary:
Bumping the NDK to 23 to prevent build failures due to the NDK
using longer paths.
Changelog:
[Android] [Fixed] - Use NDK 23 only for Windows users.
Differential Revision: D35547459

I can help in the tests, basically just change the file?

@cortinico I can help in the tests, basically just change the file?

You can find simple instructions on how to test it here: https://gist.github.com/cortinico/6a297ac4c6a2a4633f58f1e4b375dd88

@cortinico I can help in the tests, basically just change the file?

Thanks for that @RalissonMattias
You can find simple instructions on how to test it here: https://gist.github.com/cortinico/6a297ac4c6a2a4633f58f1e4b375dd88
Please publish the gradle scans once you're done with it.

I tested here and it didn't work

After applying the new patch, generate new error

After applying the new patch, generate new error

Thanks for the feedback. I ended up finding a windows machine and finding a (hopefully) working solution. It works locally for me but I'd like someone else to give it another try.
I've updated the .patch file in gist (here https://gist.github.com/cortinico/6a297ac4c6a2a4633f58f1e4b375dd88) with a new one (fix33611-v3.patch). Could you give it another try (make sure to delete the project and re-create it).

Here still no success.
First git applyt throws errors:

After manually inserting only the fix33611-v3.patch:

After manually combining and entering patches v1 and v3:

@cortinico I gave it a go.
-- ❌  First build failed https://gradle.com/s/qzgdhi6kbxlbc
-- ❌  Second build still failed, using v3 of patch from gist https://gradle.com/s/uly2zz74hljzo
-- ironically, when I run git apply fix33611-v3.patch as downloaded via "raw" view mode + save file from browser, in powershell I get "Corrupt file at line 40" which seems to be some sort of unix vs dos patch format thing. Using unix2dos did not result in success but using notepad fix33611-v3.patch and just adding a newline at end of file + save + close resulted in patch that successfully applied 🤷

  make: *** Deleting file 'C:\Users\micro\work\react-random\RN068\.cxx/local/armeabi-v7a/librrc_slider.a'
  C:/Users/micro/AppData/Local/Android/Sdk/ndk/21.4.7075529/build//../toolchains/llvm/prebuilt/windows-x86_64/bin/arm-linux-androideabi-ar: C:\Users\micro\work\react-random\RN068\.cxx/local/armeabi-v7a/objs-debug/rrc_slider/C_/Users/micro/work/react-random/RN068/node_modules/react-native/ReactAndroid/__/ReactCommon/react/renderer/components/slider/platform/android/react/renderer/components/slider/SliderMeasurementsManager.o: No such file or directory
  make: *** [C:/Users/micro/AppData/Local/Android/Sdk/ndk/21.4.7075529/build//../build/core/build-binary.mk:600: C:\Users\micro\work\react-random\RN068\.cxx/local/armeabi-v7a/librrc_slider.a] Error 1
  make: *** Waiting for unfinished jobs....
  make: *** Deleting file 'C:\Users\micro\work\react-random\RN068\.cxx/local/armeabi-v7a/librrc_progressbar.a'
  C:/Users/micro/AppData/Local/Android/Sdk/ndk/21.4.7075529/build//../toolchains/llvm/prebuilt/windows-x86_64/bin/arm-linux-androideabi-ar: C:\Users\micro\work\react-random\RN068\.cxx/local/armeabi-v7a/objs-debug/rrc_progressbar/C_/Users/micro/work/react-random/RN068/node_modules/react-native/ReactAndroid/__/ReactCommon/react/renderer/components/progressbar/android/react/renderer/components/progressbar/AndroidProgressBarShadowNode.o: No such file or directory
  make: *** [C:/Users/micro/AppData/Local/Android/Sdk/ndk/21.4.7075529/build//../build/core/build-binary.mk:600: C:\Users\micro\work\react-random\RN068\.cxx/local/armeabi-v7a/librrc_progressbar.a] Error 1
  make: *** Deleting file 'C:\Users\micro\work\react-random\RN068\.cxx/local/armeabi-v7a/librrc_switch.a'
  C:/Users/micro/AppData/Local/Android/Sdk/ndk/21.4.7075529/build//../toolchains/llvm/prebuilt/windows-x86_64/bin/arm-linux-androideabi-ar: C:\Users\micro\work\react-random\RN068\.cxx/local/armeabi-v7a/objs-debug/rrc_switch/C_/Users/micro/work/react-random/RN068/node_modules/react-native/ReactAndroid/__/ReactCommon/react/renderer/components/switch/androidswitch/react/renderer/components/androidswitch/AndroidSwitchShadowNode.o: No such file or directory
  make: *** [C:/Users/micro/AppData/Local/Android/Sdk/ndk/21.4.7075529/build//../build/core/build-binary.mk:600: C:\Users\micro\work\react-random\RN068\.cxx/local/armeabi-v7a/librrc_switch.a] Error 1

-- ✔  Advised to move project from C:\Users\micro\work\react-random\RN068 to root of C: and re-try, so, a recompile while it is in C:\RN068 worked https://gradle.com/s/2yhpn6nkfxjro

-- ✔ Advised to move project from C:\Users\micro\work\react-random\RN068 to root of C: and re-try, so, a recompile while it is in C:\RN068 worked gradle.com/s/2yhpn6nkfxjro

Awesome thanks for giving it a try. I'll close this PR as we'll merge this instead:

https://github.com/facebook/react-native/pull/33582

Which contains the v3 patch I've added in the gist.
I hope that with RN 0.69, as we moved to CMake, those kind of problems won't appear anymore.

I know you already closed PR but I wanted to report that following the steps of @mikehardy worked correctly here too. Good job, see you later

Thanks for your feedback @Ralisson-Mattias 🙏 Yup I've closed this one as we're going to merge the other PR: https://github.com/facebook/react-native/pull/33582
Thanks again for testing this on your windows machine.

On my trials, ndk23 is able to handle long paths, but the build fails because of long commmand lines (>8192 characters) .. I've created another PR with the ndk bump too ..

authors: Ralisson-Mattias RalissonMattias cortinico mganandraj mikehardy


text:
Reduce visibility of ReactPackageHelper

Summary:
In an attempt to reduce footprint of React Native Android public APIs we are reducing visibility of classes and interfaces that are not meant to be used publicly OR are public but have no usages.
As part of our analysis, which involved looking for usages inside the Meta codebase and code search in OSS, we've detected that this class/interface is public but it's not used from other packages.
If you are using this class or interface please comment in this PR and we will restate the public access.
changelog: [Android][Changed] Reducing visibility of ReactPackageHelper
Reviewed By: RSNara
Differential Revision: D49752142

iOS 17


text:
Add queueMicrotask method to JSI

Summary:
Changelog: [internal]
Context
Microtasks are an important aspect of JavaScript and they will become increasingly important in the hosts where we're currently using JSI.
For example, React Native is going to adopt an event loop processing model similar to the one on the Web, which means it would need the ability to schedule and execute microtasks in every iteration of the loop. See https://github.com/react-native-community/discussions-and-proposals/pull/744 for details.
JSI already has a method to execute all pending microtasks (drainMicrotasks) but without a method to schedule microtasks this is incomplete.
We're currently testing microtasks with Hermes using an internal method to schedule microtasks (HermesInternal.enqueueJob) but we need a method in JSI so this also works in other runtimes like JSC and V8.
Changes
This adds the queueMicrotask to the Runtime API in JSI so we have symmetric API for microtasks and we can implement the necessary functionality.
The expectation for JSI implementations is to queue microtasks from this method and from built-ins like Promises and async functions in the same queue, and not drain that queue until explicitly done via drainMicrotasks in JSI.
This also modifies Hermes and JSC to provide stubs for those methods, and the actual implementation will be done in following diffs.
Differential Revision: D54302536

@rubennorte this opens the road for this?

https://github.com/facebook/react-native/issues/42742

@rubennorte this opens the road for this?
https://github.com/facebook/react-native/issues/42742

@efstathiosntonas yes! The plan is to enable the new event loop in the new RN architecture soon, which will unblock WeakRefs in Hermes.

@rubennorte thanks for the info, so this is targeting exclusively on new arch?

@rubennorte thanks for the update, so this is targeting exclusively on new arch?

Yes, that's right.

This was committed in https://github.com/facebook/react-native/commit/036f47ebc4cb372e13788c2498dcbf3c88379a1d

authors: efstathiosntonas rubennorte


text:
Adding regex handling for negative signs

Context
Add parsing for signage in string length unit values:

Adding parsing for (+/-) signs
Multiplying parsed float by -1 if sign = "-"
Updated tests and snapshots

Testing
Modifying tests to account for positive and negative signs
Ran npm run jest /packages/react-strict-dom/tests/css-test.native.js

Thanks for working on the fix! Please leave merging to maintainers next time

authors: mellyeliu necolas

repo: facebook/react-strict-dom


text:
Add side_effect_modules to config

Some imports have known side-effects, and should be consdiered as block separators.  Would be nice if these could be set in configuration.  I don't think there are any in stdlib, but django.settings as an example may import your settings module that changes sys.path.

This was done in #39

source: gharchive/issue

repo: facebookexperimental/usort


text:
Error when installing packages

Bug description
When trying to install the new prerequisite packages for running GeoLift I am getting the following errors:  ERROR: configuration failed for package ‘units’ & ERROR: dependency ‘CommutingZones’ is not available for package ‘GeoLift’ This is causing the install to fail which is then giving the further error  Error in library(GeoLift) : there is no package called ‘GeoLift’ when trying to import the GeoLift package.
Session information
Please paste the output after running sessionInfo() in your R session.
Reproduction steps
install.packages("remotes", repos='http://cran.us.r-project.org')
remotes::install_github("facebookincubator/CommutingZones")
remotes::install_github("ebenmichael/augsynth")
library(GeoLift)

I've just removed and reinstalled using the following code. It works.
remotes::install_github("facebookincubator/CommutingZones")
remotes::install_github("ebenmichael/augsynth")
remotes::install_github("facebookincubator/GeoLift")

You can't call a library using library() if it's not yet installed.

^ just tried this and it doesn't work for me, getting ERROR: dependency ‘CommutingZones’ is not available for package ‘GeoLift’

What error do you get when installing commutingzones?

I get these three errors
ERROR: configuration failed for package ‘units’
ERROR: dependency ‘units’ is not available for package ‘sf’
ERROR: dependency ‘sf’ is not available for package ‘CommutingZones’

@mdb374 FYI can you please check this? Thanks!

@gufengzhou @mdb374 are there any updates on this?

Hi, I work with @ffionwells and had a look at this and think I've identified the issue. They are trying to install on to a databricks notebook and the requirement for CommutingZones has introduced a requirement for units - this package requires the linux package libudunits2-dev which wasn't installed on the cluster (See here)
Is it worth updating the documentation to specify you may need to install libudunits2-dev first (i.e. run something like sudo apt install libudunits2-dev)?

Hi both, thanks for the insights. We've failed to reproduce this error and could only google the error message. Glad that it's working now. We'll update the documentation accordingly
FYI @JussanN @raphaeltamaki @mdb374

Hi, I'm closing this issue as it was already solved. FYI it's no longer required to install CommutingZones when installing GeoLift so the first remains optional. You can see more details on the GeoLift website. Thank you. https://facebookincubator.github.io/GeoLift/docs/GettingStarted/InstallingR

authors: JussanN ffionwells gufengzhou tellis92

repo: facebookincubator/GeoLift


text:
build error - undefined reference to `facebook::strings::ByteSinkBuffer::flush()'

Hi all,
Building the dwrf tests throws error because it cannot find the definition of the ByteSinkBuffer::flush function. I don't find the definition in the code base. Does anyone know why this happens and how to fix?
Here is an example error log:

[795/1208] Linking CXX executable velox/dwio/dwrf/test/velox_dwrf_writer_extended_test
FAILED: velox/dwio/dwrf/test/velox_dwrf_writer_extended_test
: && /home/kang.1002/app/gcc-12.2.0/bin/c++ -I/home/kang.1002/app/re2/usr/local/include -L/home/kang.1002/app/re2/usr/local/lib -lre2 -I/home/kang.1002/app/protobuf-3.21.3/include -L/home/kang.1002/app/protobuf-3.21.3/lib -lprotobuf -lprotoc -mavx2 -mfma -mavx -mf16c -mlzcnt -std=c++17 -mbmi2 -D USE_VELOX_COMMON_BASE -D HAS_UNCAUGHT_EXCEPTIONS -Wall -Wextra -Wno-unused        -Wno-unused-parameter        -Wno-sign-compare        -Wno-ignored-qualifiers        -Wno-implicit-fallthrough          -Wno-empty-body          -Wno-class-memaccess          -Wno-comment          -Wno-int-in-bool-context          -Wno-redundant-move          -Wno-array-bounds          -Wno-maybe-uninitialized          -Wno-unused-result          -Wno-format-overflow          -Wno-strict-aliasing          -Wno-type-limits -O3 -DNDEBUG -Wl,-export-dynamic velox/dwio/dwrf/test/CMakeFiles/velox_dwrf_writer_extended_test.dir/WriterExtendedTests.cpp.o -o velox/dwio/dwrf/test/velox_dwrf_writer_extended_test  -Wl,-rpath,/home/kang.1002/app/libevent-2.1.12/lib:/home/kang.1002/app/lz4-1.9.4/lib:/home/kang.1002/app/zstd-1.5.2/lib:/home/kang.1002/app/glog/lib:/home/kang.1002/app/boost_1_75_0/lib  velox/dwio/dwrf/test/utils/libvelox_dwrf_test_utils.a  velox/common/caching/libvelox_caching.a  velox/dwio/common/libvelox_dwio_common.a  velox/dwio/common/exception/libvelox_dwio_common_exception.a  velox/dwio/type/fbhive/libvelox_dwio_type_fbhive.a  velox/common/encode/libvelox_encode.a  velox/common/base/libvelox_exception.a  velox/common/memory/libvelox_memory.a  velox/common/process/libvelox_process.a  velox/common/serialization/libvelox_serialization.a  velox/type/libvelox_type.a  velox/vector/libvelox_vector.a  _deps/folly-build/libfolly.a  /home/kang.1002/app/double_conversion/usr/local/lib/libdouble-conversion.a  /home/kang.1002/app/libevent-2.1.12/lib/libevent.so  /home/kang.1002/app/snappy-1.1.8/lib/libsnappy.a  -ldl  /home/kang.1002/app/fmt-8.0.0/lib/libfmt.a  /home/kang.1002/app/fmt-8.0.0/lib/libfmt.a  _deps/folly-build/libfolly.a  /home/kang.1002/app/double_conversion/usr/local/lib/libdouble-conversion.a  /home/kang.1002/app/libevent-2.1.12/lib/libevent.so  /home/kang.1002/app/snappy-1.1.8/lib/libsnappy.a  -ldl  /home/kang.1002/app/fmt-8.0.0/lib/libfmt.a  /home/kang.1002/app/fmt-8.0.0/lib/libfmt.a  /home/kang.1002/app/lz4-1.9.4/lib/liblz4.so  /home/kang.1002/app/lzo-2.10/lib/liblzo2.a  /home/kang.1002/app/zstd-1.5.2/lib/libzstd.so  /usr/lib/x86_64-linux-gnu/libz.so  velox/dwio/common/tests/utils/libvelox_dwio_common_test_utils.a  velox/dwio/dwrf/proto/libvelox_dwio_dwrf_proto.a  velox/dwio/dwrf/reader/libvelox_dwio_dwrf_reader.a  velox/dwio/dwrf/writer/libvelox_dwio_dwrf_writer.a  lib/libgtest.a  lib/libgtest_main.a  lib/libgmock.a  /home/kang.1002/app/glog/lib/libglog.so  -lstdc++fs  /home/kang.1002/app/lzo-2.10/lib/liblzo2.a  velox/dwio/dwrf/common/libvelox_dwio_dwrf_common.a  velox/dwio/dwrf/utils/libvelox_dwio_dwrf_utils.a  velox/dwio/dwrf/proto/libvelox_dwio_dwrf_proto.a  /home/kang.1002/app/protobuf-3.21.3/lib/libprotobuf.so  velox/dwio/common/libvelox_dwio_common.a  velox/common/caching/libvelox_caching.a  velox/common/file/libvelox_file.a  velox/dwio/common/compression/libvelox_dwio_common_compression.a  velox/dwio/common/encryption/libvelox_dwio_common_encryption.a  velox/dwio/common/exception/libvelox_dwio_common_exception.a  velox/vector/libvelox_vector.a  velox/type/libvelox_type.a  velox/common/encode/libvelox_encode.a  velox/common/serialization/libvelox_serialization.a  velox/type/tz/libvelox_type_tz.a  velox/external/date/libvelox_external_date.a  velox/common/time/libvelox_time.a  velox/buffer/libvelox_buffer.a  velox/common/memory/libvelox_memory.a  velox/common/base/libvelox_common_base.a  velox/common/base/libvelox_exception.a  velox/common/process/libvelox_process.a  /home/kang.1002/app/glog/lib/libglog.so  /usr/lib/x86_64-linux-gnu/libgflags.so.2.2.1  _deps/folly-build/libfolly.a  /usr/lib/x86_64-linux-gnu/libgflags.so.2.2.1  /home/kang.1002/app/fmt-8.0.0/lib/libfmt.a  /home/kang.1002/app/glog/lib/libglog.so  /usr/lib/x86_64-linux-gnu/libz.so  /usr/lib/x86_64-linux-gnu/libssl.so  /usr/lib/x86_64-linux-gnu/libcrypto.so  /home/kang.1002/app/lz4-1.9.4/lib/liblz4.so  /home/kang.1002/app/zstd-1.5.2/lib/libzstd.so  /home/kang.1002/app/boost_1_75_0/lib/libboost_program_options.so.1.75.0  /home/kang.1002/app/boost_1_75_0/lib/libboost_context.so.1.75.0  /home/kang.1002/app/boost_1_75_0/lib/libboost_filesystem.so.1.75.0  /home/kang.1002/app/boost_1_75_0/lib/libboost_regex.so.1.75.0  -licudata  -licui18n  -licuuc  /home/kang.1002/app/boost_1_75_0/lib/libboost_thread.so.1.75.0  /home/kang.1002/app/boost_1_75_0/lib/libboost_system.so.1.75.0  /home/kang.1002/app/boost_1_75_0/lib/libboost_date_time.so.1.75.0  /home/kang.1002/app/boost_1_75_0/lib/libboost_atomic.so.1.75.0  /home/kang.1002/app/double_conversion/usr/local/lib/libdouble-conversion.a  /home/kang.1002/app/libevent-2.1.12/lib/libevent.so  /home/kang.1002/app/snappy-1.1.8/lib/libsnappy.a  -ldl  /home/kang.1002/app/fmt-8.0.0/lib/libfmt.a  lib/libgtest.a  -lpthread && :
velox/dwio/dwrf/reader/libvelox_dwio_dwrf_reader.a(SelectiveTimestampColumnReader.cpp.o): In function facebook::strings::ByteSinkBuffer::sync()': SelectiveTimestampColumnReader.cpp:(.text._ZN8facebook7strings14ByteSinkBuffer4syncEv[_ZN8facebook7strings14ByteSinkBuffer4syncEv]+0x5): undefined reference to facebook::strings::ByteSinkBuffer::flush()'
collect2: error: ld returned 1 exit status

@donghekang Thank you for reporting this issue. Which platform do you use for building?

I build it on Ubuntu 18.04.6 LTS (Bionic Beaver) . I don't have the root privilege so that I installed all dependence by myself. Here is how I compile it:

EXTRA_CMAKE_FLAGS="-DCMAKE_PREFIX_PATH="/home/kang.1002/app/boost_1_75_0/;/home/kang.1002/app/glog;/home/kang.1002/app/re2/usr/local/;/home/kang.1002/app/double_conversion/usr/local;/home/kang.1002/app/libevent-2.1.12;/home/kang.1002/app/protobuf-3.21.3;/home/kang.1002/app/fmt-8.0.0;/home/kang.1002/app/lz4-1.9.4;/home/kang.1002/app/lzo-2.10;/home/kang.1002/app/snappy-1.1.8;/home/kang.1002/app/zstd-1.5.2" -DCMAKE_CXX_FLAGS="-I/home/kang.1002/app/re2/usr/local/include -L/home/kang.1002/app/re2/usr/local/lib -lre2 -I/home/kang.1002/app/protobuf-3.21.3/include -L/home/kang.1002/app/protobuf-3.21.3/lib -lprotobuf -lprotoc"" TREAT_WARNINGS_AS_ERRORS=0 make

Hello, I ran into a similar issue. I was able to solve the issue by downgrading to GCC 10.

Closed as stale. Feel free to reopen if the same error persists or open a new issue if there are additional issues.

authors: assignUser dillon-cullinan donghekang mbasmanova

repo: facebookincubator/velox


text:
VeloxRuntimeError: Expression: source->size() >= sourceIndex + count Function: copyValuesAndNulls

Description
Fuzzer test failed in my PR: https://github.com/facebookincubator/velox/pull/4834
I think we need reset the sizes even for newElements->size() == input->elements()->size(), I verified this locally and passed.
Error Reproduction
_build/debug/velox/expression/tests/velox_expression_fuzzer_test
--seed 4154417068
--enable_variadic_signatures
--velox_fuzzer_enable_complex_types
--lazy_vector_generation_ratio 0.2
--velox_fuzzer_enable_column_reuse
--velox_fuzzer_enable_expression_reuse
--retry_with_try
--duration_sec 60
--logtostderr=1
--minloglevel=0
--repro_persist_path=/tmp/fuzzer_repro
Relevant logs
I20230505 04:12:17.847862 302208 ExpressionVerifier.cpp:31] 1 vectors as input:
I20230505 04:12:17.847870 302208 ExpressionVerifier.cpp:33] 	[DICTIONARY ARRAY<DATE>: 100 elements, 10 nulls], [DICTIONARY ARRAY<DATE>: 100 elements, 15 nulls], [DICTIONARY ARRAY<DATE>: 100 elements, 9 nulls], [ARRAY ARRAY<DATE>: 100 elements, no nulls]
E20230505 04:12:17.848726 302208 Exceptions.h:68] Line: ../.././velox/vector/FlatVector-inl.h:231, Function:copyValuesAndNulls, Expression: source->size() >= sourceIndex + count (969 vs. 980), Source: RUNTIME, ErrorCode: INVALID_STATE
E20230505 04:12:17.849282 302208 ExpressionVerifier.cpp:157] Common eval: Exceptions other than VeloxUserError are not allowed.
I20230505 04:12:17.849977 302208 ExpressionVerifier.cpp:316] Persisted input: --fuzzer_repro_path /tmp/fuzzer_repro/velox_expressionVerifier_vd1OY3 --input_path /tmp/fuzzer_repro/velox_expressionVerifier_vd1OY3/input_vector --result_path /tmp/fuzzer_repro/velox_expressionVerifier_vd1OY3/result_vector --sql_path /tmp/fuzzer_repro/velox_expressionVerifier_vd1OY3/sql
terminate called after throwing an instance of 'facebook::velox::VeloxRuntimeError'
  what():  Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: (969 vs. 980)
Retriable: False
Expression: source->size() >= sourceIndex + count
Context: cast((array_sort(c0)) as ARRAY<VARCHAR>)
Top-Level Context: array_max(try(cast((array_sort(c0)) as ARRAY<VARCHAR>)))
Function: copyValuesAndNulls
File: ../.././velox/vector/FlatVector-inl.h
Line: 231
Stack trace:
# 0  std::shared_ptr<facebook::velox::VeloxException::State const> facebook::velox::VeloxException::State::make<facebook::velox::VeloxException::make(char const*, unsigned long, char const*, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, bool, facebook::velox::VeloxException::Type, std::basic_string_view<char, std::char_traits<char> >)::{lambda(auto:1&)#1}>(facebook::velox::VeloxException::Type, facebook::velox::VeloxException::make(char const*, unsigned long, char const*, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, bool, facebook::velox::VeloxException::Type, std::basic_string_view<char, std::char_traits<char> >)::{lambda(auto:1&)#1})
# 1  facebook::velox::VeloxException::VeloxException(char const*, unsigned long, char const*, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, bool, facebook::velox::VeloxException::Type, std::basic_string_view<char, std::char_traits<char> >)
# 2  facebook::velox::VeloxRuntimeError::VeloxRuntimeError(char const*, unsigned long, char const*, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, bool, std::basic_string_view<char, std::char_traits<char> >)
# 3  void facebook::velox::detail::veloxCheckFail<facebook::velox::VeloxRuntimeError, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(facebook::velox::detail::VeloxCheckFailArgs const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
# 4  facebook::velox::FlatVector<facebook::velox::StringView>::copyValuesAndNulls(facebook::velox::BaseVector const*, int, int, int)
# 5  facebook::velox::FlatVector<facebook::velox::StringView>::copyRanges(facebook::velox::BaseVector const*, folly::Range<facebook::velox::BaseVector::CopyRange const*> const&)
# 6  facebook::velox::BaseVector::copy(facebook::velox::BaseVector const*, int, int, int)
# 7  facebook::velox::FlatVector<facebook::velox::StringView>::copy(facebook::velox::BaseVector const*, int, int, int)
# 8  facebook::velox::ArrayVectorBase::copyRangesImpl(facebook::velox::BaseVector const*, folly::Range<facebook::velox::BaseVector::CopyRange const*> const&, std::shared_ptr<facebook::velox::BaseVector>*, facebook::velox::BaseVector const*, std::shared_ptr<facebook::velox::BaseVector>*, facebook::velox::BaseVector const*)
# 9  facebook::velox::ArrayVector::copyRanges(facebook::velox::BaseVector const*, folly::Range<facebook::velox::BaseVector::CopyRange const*> const&)
# 10 facebook::velox::BaseVector::copy(facebook::velox::BaseVector const*, int, int, int)
# 11 facebook::velox::BaseVector::copy(facebook::velox::BaseVector const*, facebook::velox::SelectivityVector const&, int const*)::{lambda(int)#1}::operator()(int) const
# 12 void facebook::velox::SelectivityVector::applyToSelected<facebook::velox::BaseVector::copy(facebook::velox::BaseVector const*, facebook::velox::SelectivityVector const&, int const*)::{lambda(int)#1}>(facebook::velox::BaseVector::copy(facebook::velox::BaseVector const*, facebook::velox::SelectivityVector const&, int const*)::{lambda(int)#1}) const
# 13 facebook::velox::BaseVector::copy(facebook::velox::BaseVector const*, facebook::velox::SelectivityVector const&, int const*)
# 14 facebook::velox::BaseVector::ensureWritable(facebook::velox::SelectivityVector const&, std::shared_ptr<facebook::velox::Type const> const&, facebook::velox::memory::MemoryPool*, std::shared_ptr<facebook::velox::BaseVector>&, facebook::velox::VectorPool*)
# 15 facebook::velox::exec::Expr::addNulls(facebook::velox::SelectivityVector const&, unsigned long const*, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::Type const> const&, std::shared_ptr<facebook::velox::BaseVector>&)
# 16 facebook::velox::exec::CastExpr::apply(facebook::velox::SelectivityVector const&, std::shared_ptr<facebook::velox::BaseVector> const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::Type const> const&, std::shared_ptr<facebook::velox::Type const> const&, std::shared_ptr<facebook::velox::BaseVector>&)
# 17 facebook::velox::exec::CastExpr::evalSpecialForm(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&)
# 18 facebook::velox::exec::Expr::evalSpecialFormWithStats(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&)
# 19 facebook::velox::exec::Expr::evalAllImpl(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&)
# 20 facebook::velox::exec::Expr::evalAll(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&)
# 21 facebook::velox::exec::Expr::evalWithNulls(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&)
# 22 facebook::velox::exec::Expr::evalEncodings(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&)
# 23 facebook::velox::exec::Expr::eval(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&, bool)
# 24 facebook::velox::exec::TryExpr::evalSpecialForm(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&)
# 25 facebook::velox::exec::Expr::evalSpecialFormWithStats(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&)
# 26 facebook::velox::exec::Expr::evalAllImpl(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&)
# 27 facebook::velox::exec::Expr::evalAll(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&)
# 28 facebook::velox::exec::Expr::evalWithNulls(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&)
# 29 facebook::velox::exec::Expr::evalEncodings(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&)
# 30 facebook::velox::exec::Expr::eval(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&, bool)
# 31 auto facebook::velox::exec::Expr::evalAllImpl(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&)::{lambda(auto:1)#1}::operator()<int>(int) const
# 32 bool facebook::velox::exec::Expr::evalArgsDefaultNulls<facebook::velox::exec::Expr::evalAllImpl(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&)::{lambda(auto:1)#1}>(facebook::velox::exec::MutableRemainingRows&, facebook::velox::exec::Expr::evalAllImpl(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&)::{lambda(auto:1)#1}, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&)
# 33 facebook::velox::exec::Expr::evalAllImpl(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&)
# 34 facebook::velox::exec::Expr::evalAll(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&)
# 35 facebook::velox::exec::Expr::evalWithNulls(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&)
# 36 facebook::velox::exec::Expr::evalWithMemo(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&)
# 37 facebook::velox::exec::Expr::evalEncodings(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&)
# 38 facebook::velox::exec::Expr::eval(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::shared_ptr<facebook::velox::BaseVector>&, bool)
# 39 facebook::velox::exec::ExprSet::eval(int, int, bool, facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::vector<std::shared_ptr<facebook::velox::BaseVector>, std::allocator<std::shared_ptr<facebook::velox::BaseVector> > >&)
# 40 facebook::velox::exec::ExprSet::eval(facebook::velox::SelectivityVector const&, facebook::velox::exec::EvalCtx&, std::vector<std::shared_ptr<facebook::velox::BaseVector>, std::allocator<std::shared_ptr<facebook::velox::BaseVector> > >&)
# 41 facebook::velox::test::ExpressionVerifier::verify(std::shared_ptr<facebook::velox::core::ITypedExpr const> const&, std::shared_ptr<facebook::velox::RowVector> const&, std::shared_ptr<facebook::velox::BaseVector>&&, bool, std::vector<unsigned int, std::allocator<unsigned int> >)
# 42 facebook::velox::test::ExpressionFuzzer::go()
# 43 facebook::velox::test::expressionFuzzer(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<facebook::velox::exec::FunctionSignature const*, std::allocator<facebook::velox::exec::FunctionSignature const*> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::vector<facebook::velox::exec::FunctionSignature const*, std::allocator<facebook::velox::exec::FunctionSignature const*> > > > >, unsigned long)
# 44 FuzzerRunner::run(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
# 45 main
# 46 __libc_start_main
# 47 _start

@laithsakka and @kagamiori, I saw you discussed related changes in this PR: https://github.com/facebookincubator/velox/pull/4446

This is duplicate of #4754


text:
Support deserialize without resultOffset parameter in  Exchange::getOutput for backward-compatibility

Fix #8240.
Support deserialize without resultOffset parameter in  Exchange::getOutput for backward-compatibility.

Closing per https://github.com/facebookincubator/velox/issues/8240#issuecomment-1877201694

authors: icejoywoo mbasmanova


text:
Fix flaky MultiThreadedHashJoinTest.outOfJoinKeyColumnOrder

The test failed because there were not enough keys on
the HashBuild side to bump the spill level to the max.
Verified by stress running the originally flaky test 300 times.

@xiaoxmeng Hi Xuan, could you please help review this PR? Thanks.


text:
Result for semantic segmentation？

May I ask if you are using the annotation of the Panoptic Segmentation, but using mIoU as the evaluation? If so, this is not semantic segmentation. Your annotation information is different, and the Panoptic segmentation uses more annotation signals.

@jianlong-yuan Hi，have you try to train the net on windows ? could you give me some informations? Please guide me.

For semantic segmentation benchmarks, we only use semantic segmentation annotations.

authors: Buling-Knight bowenc0221 jianlong-yuan


text:
nits

Stack from ghstack (oldest at bottom):

#1173
#1172
#1171
#1170
-> #1169
#1168
#1167
#1166
#1165
#1164
#1163
#1162

Codecov Report

Merging #1169 (26a06b3) into gh/anuragprat1k/21/base (090f503) will decrease coverage by 0.02%.
The diff coverage is n/a.

:exclamation: Current head 26a06b3 differs from pull request most recent head f672403. Consider uploading reports for the commit f672403 to get more accurate results

@@                     Coverage Diff                     @@
##           gh/anuragprat1k/21/base    #1169      +/-   ##
===========================================================
- Coverage                    34.00%   33.98%   -0.03%
===========================================================
  Files                          345      345
  Lines                        34174    34174
===========================================================
- Hits                         11622    11615       -7
- Misses                       22552    22559       +7

Impacted Files
Coverage Δ

lowlevel/minecraft/pyworld/physical_interfaces.py
88.18% <0.00%> (-2.37%)
:arrow_down:

lowlevel/minecraft/pyworld/fake_mobs.py
87.28% <0.00%> (-1.70%)
:arrow_down:

interpreter/craftassist/tasks.py
76.12% <0.00%> (-0.36%)
:arrow_down:

repo: facebookresearch/fairo


text:
visualization issue

Thanks for your excellent work!
How can I get the same visualization as Fig. 7?

Have you been able to achieve full-room visualizations? If so could you please share your setup, results, and expertise on open issue #19?

authors: BillyMazotti chenziyi-zzz

repo: facebookresearch/iSDF


text:
70B Fine-tuning, using 'accelerate' on a single GPU (e.g. A10 with 24GB)

System Info
I’m trying to fine-tune the 70B Llama 2 model using the llama-recipes/examples/quickstart.ipynb file, on my sing 4090 GPU server with 24GB VRAM (which is an online rented one from HostKey for this purpose).
The quickstart.ipynb file says “This notebook shows how to train a Llama 2 model on a single GPU (e.g. A10 with 24GB) using int8 quantization and LoRA”. However, I could only train the 7B and 13B model with it. (For training the 13B modle, I even had to change 'per_device_train_batch_size': 1 ).
However, for training the 70B, I keep running into “Out Of Memory Error: CUDA out of memory”. Therefore, I decided to use 'accelerate'.
For example I updated the first code cell with 'accelerate' as below, and it no longer gives the Out Of Memory Error because 'accelerate' is a library designed to simplify the usage of hardware accelerators for training and deploying deep learning models.
import torch
from transformers import LlamaForCausalLM, LlamaTokenizer
from accelerate import init_empty_weights, load_checkpoint_and_dispatch
from accelerate import Accelerator

# Initialize tokenizer from pretrained model
model_id = "../Llama-2-70b-hf"

# Initialize Accelerator
accelerator = Accelerator()

#
tokenizer = LlamaTokenizer.from_pretrained(model_id)

# Initialize an empty skeleton of the model
with init_empty_weights():
    model = LlamaForCausalLM.from_pretrained(model_id)

# Specify the folder to offload model parts to disk
offload_folder = "../offload_folder"

# Load checkpoint and dispatch
checkpoint_file = "../Llama-2-70b-hf/pytorch_model.bin.index.json"
model = load_checkpoint_and_dispatch(
    model,
    checkpoint=checkpoint_file,
    device_map="auto",
    offload_folder=offload_folder  # Include the offload_folder here
)

# Use lower precision
model = model.to(dtype=torch.float16)

model, tokenizer = accelerator.prepare(model, tokenizer)

However, later on, I run into a problem on the Fine-tune cell block since I’m using the latest version of 'accelerate' (not the old accelerate 0.15.0 & transformers 4.28.1, I run into a problem which says: AttributeError: 'AcceleratorState' object has no attribute 'distributed_type'  when I run the below code.
from transformers import default_data_collator, Trainer, TrainingArguments

# Define training args
training_args = TrainingArguments(
    output_dir=output_dir,
    overwrite_output_dir=True,
    bf16=True,  # Use BF16 if available
    push_to_hub=False,  # make sure to include this for Accelerate compatibility
    # logging strategies
    logging_dir=f"{output_dir}/logs",
    logging_strategy="steps",
    logging_steps=10,
    save_strategy="no",
    optim="adamw_torch_fused",
    max_steps=total_steps if enable_profiler else -1,
    **{k:v for k,v in config.items() if k != 'lora_config'}
)

# Remember to prepare your data_collator using accelerator if you've defined any
data_collator = default_data_collator
data_collator = accelerator.prepare(data_collator)

# Also, modify the Trainer call to include the accelerator's device
trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_dataset,
 #   eval_dataset=eval_dataset,
    tokenizer=tokenizer,
 #   devices=accelerator.device_count,
    callbacks=[profiler_callback] if enable_profiler else None, # Include the profiler callback if profiling is enabled
 #   compute_metrics=compute_metrics, # Include this if you have defined a metrics function
)

train_results = trainer.train()

Here is the full error message I get for the above code cell:
AttributeError                            Traceback (most recent call last)
Cell In[31], line 21
      4 training_args = TrainingArguments(
      5     output_dir=output_dir,
      6     overwrite_output_dir=True,
   (...)
     16     **{k:v for k,v in config.items() if k != 'lora_config'}
     17 )
     19 # Remember to prepare your data_collator using accelerator if you've defined any
     20 #data_collator = default_data_collator
---> 21 data_collator = accelerator.prepare(data_collator)
     23 # Also, modify the Trainer call to include the accelerator's device
     24 trainer = Trainer(
     25     model=model,
     26     args=training_args,
   (...)
     33  #   compute_metrics=compute_metrics, # Include this if you have defined a metrics function
     34 )

File ~/Desktop/llama-2_for_70B/llama/70B_env/lib/python3.10/site-packages/accelerate/accelerator.py:1142, in Accelerator.prepare(self, device_placement, *args)
   1137 elif len(device_placement) != len(args):
   1138     raise ValueError(
   1139         f"`device_placement` should be a list with {len(args)} elements (the number of objects passed)."
   1140     )
-> 1142 if self.distributed_type == DistributedType.FSDP:
   1143     model_count = 0
   1144     optimizer_present = False

File ~/Desktop/llama-2_for_70B/llama/70B_env/lib/python3.10/site-packages/accelerate/accelerator.py:468, in Accelerator.distributed_type(self)
    466 @property
    467 def distributed_type(self):
--> 468     return self.state.distributed_type

AttributeError: 'AcceleratorState' object has no attribute 'distributed_type'

I was also getting Out Of Memory Error with the prepare_model_for_int8_training function from peft, and I couldn’t use 'accelerate'? What could be the solution? Can I comment out the below prepare_model_for_int8_training part, and would the fine-tuning still work?
model = prepare_model_for_int8_training(model)

In summary, please make a version of the quickstart.ipynb file the uses 'accelerate' or something else, so that we can fine-tune the 70B Llama 2 model on a single GPU (e.g. A10 with 24GB).

I don't care how long it takes as I can live with it being slow to fine-tune/train, as long as it works.
Information

[X] The official example scripts
[X] My own modified scripts

🐛 Describe the bug
However, later on, I run into a problem on the Fine-tune cell block since I’m using the latest version of 'accelerate' (not the old accelerate 0.15.0 & transformers 4.28.1, I run into a problem which says: AttributeError: 'AcceleratorState' object has no attribute 'distributed_type'  when I run the below code.
from transformers import default_data_collator, Trainer, TrainingArguments

# Define training args
training_args = TrainingArguments(
    output_dir=output_dir,
    overwrite_output_dir=True,
    bf16=True,  # Use BF16 if available
    push_to_hub=False,  # make sure to include this for Accelerate compatibility
    # logging strategies
    logging_dir=f"{output_dir}/logs",
    logging_strategy="steps",
    logging_steps=10,
    save_strategy="no",
    optim="adamw_torch_fused",
    max_steps=total_steps if enable_profiler else -1,
    **{k:v for k,v in config.items() if k != 'lora_config'}
)

# Remember to prepare your data_collator using accelerator if you've defined any
data_collator = default_data_collator
data_collator = accelerator.prepare(data_collator)

# Also, modify the Trainer call to include the accelerator's device
trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_dataset,
 #   eval_dataset=eval_dataset,
    tokenizer=tokenizer,
 #   devices=accelerator.device_count,
    callbacks=[profiler_callback] if enable_profiler else None, # Include the profiler callback if profiling is enabled
 #   compute_metrics=compute_metrics, # Include this if you have defined a metrics function
)

train_results = trainer.train()

Error logs
AttributeError                            Traceback (most recent call last)
Cell In[31], line 21
      4 training_args = TrainingArguments(
      5     output_dir=output_dir,
      6     overwrite_output_dir=True,
   (...)
     16     **{k:v for k,v in config.items() if k != 'lora_config'}
     17 )
     19 # Remember to prepare your data_collator using accelerator if you've defined any
     20 #data_collator = default_data_collator
---> 21 data_collator = accelerator.prepare(data_collator)
     23 # Also, modify the Trainer call to include the accelerator's device
     24 trainer = Trainer(
     25     model=model,
     26     args=training_args,
   (...)
     33  #   compute_metrics=compute_metrics, # Include this if you have defined a metrics function
     34 )

File ~/Desktop/llama-2_for_70B/llama/70B_env/lib/python3.10/site-packages/accelerate/accelerator.py:1142, in Accelerator.prepare(self, device_placement, *args)
   1137 elif len(device_placement) != len(args):
   1138     raise ValueError(
   1139         f"`device_placement` should be a list with {len(args)} elements (the number of objects passed)."
   1140     )
-> 1142 if self.distributed_type == DistributedType.FSDP:
   1143     model_count = 0
   1144     optimizer_present = False

File ~/Desktop/llama-2_for_70B/llama/70B_env/lib/python3.10/site-packages/accelerate/accelerator.py:468, in Accelerator.distributed_type(self)
    466 @property
    467 def distributed_type(self):
--> 468     return self.state.distributed_type

AttributeError: 'AcceleratorState' object has no attribute 'distributed_type'

Expected behavior
In summary, please make a version of the quickstart.ipynb file the uses 'accelerate' or something else, so that we can fine-tune the 70B Llama 2 model on a single GPU (e.g. A10 with 24GB).
I don't care how long it takes as I can live with it being slow to fine-tune/train, as long as it works.

@BrookMakF thanks for trying out the recipes, with current features fine-tuning 70B on a single GPU specially A10 is not feasible, even loading the model to GPU int8 will take 70GB, maybe disk offload of accelerate could be helpful. I will close this issue for now, but pls feel free to send a PR/ we keep the docs updated if could make it work.

authors: BrookMakF HamidShojanazeri

repo: facebookresearch/llama-recipes


text:
Keep only final optimizer in portfolio

Types of changes

[ ] Docs change / refactoring / dependency upgrade
[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Motivation and Context / Related issue

How Has This Been Tested (if it applies)

Checklist

[x] The documentation is up-to-date with the changes I made.
[x] I have read the CONTRIBUTING document and completed the CLA (see CLA).
[x] All tests passed, and additional code has been covered with new tests.

Tested and approved.
Thanks a lot.

authors: jrapin teytaud

repo: facebookresearch/nevergrad


text:
unable to load video via LabeledVideoDataset

Here is the dame frame, containing path
	group     label	path
0	17	        90	../../splitvideos/17/0/0.avi
1	17		90	../../splitvideos/17/0/1.avi
2	17		91	../../splitvideos/17/0/2.avi
3	17		90	../../splitvideos/17/1/0.avi
4	17		90	../../splitvideos/17/1/1.avi

I passed it as follow
train_dataset=LabeledVideoDataset(labeled_video_paths=(list(train_df.path),list(train_df.pulse)),
                   clip_sampler=make_clip_sampler("random", 2),decode_audio=False)
train_loader=DataLoader(train_dataset,batch_size=2)

But when i do
batch=next(iter(train_loader))
it throws error StopIteration
EDIT:
I changed the code to
train_dataset=LabeledVideoDataset(labeled_video_paths=list(zip(train_df.path,train_df.pulse)),

but now i am getting error when i run batch=next(iter(train_loader))
TypeError: 'int' object is not a mapping

This issue is because of the following line, which in my case return a label.
https://github.com/facebookresearch/pytorchvideo/blob/6744240a239b615bdc3ab3e2b795977a4db38848/pytorchvideo/data/labeled_video_dataset.py#L213

I believe this line
https://github.com/facebookresearch/pytorchvideo/blob/6744240a239b615bdc3ab3e2b795977a4db38848/pytorchvideo/data/labeled_video_dataset.py#L213
should be replaced with
"info_dict":info_dict,

@talhaanwarch, you are not using LabeledVideoDataset correctly. labeled_video_paths should a list of dictionaries i.e, each element in the list is a dictionary. And each dictionary contains lablels and other information associated with a single clip.
Please refer to our documentation for more details.
An example of labeled_video_paths can be found at - https://github.com/facebookresearch/pytorchvideo/blob/main/pytorchvideo/data/labeled_video_paths.py#L13
Closing the issue now. Feel free to re-open it if you have more questions.

@kalyanvasudev can you please provide a sample dictionary

authors: kalyanvasudev talhaanwarch

repo: facebookresearch/pytorchvideo


text:
Add support for SRI like in Webpack-Html-Plugin

It would be nice when the plugin would be able to use the integrity information added to assets by the SRI plugin: https://www.npmjs.com/package/webpack-subresource-integrity.
This section might be helpful for implementation:
https://www.npmjs.com/package/webpack-subresource-integrity#without-htmlwebpackplugin
With this is place it would be safe to push all these bundles onto a CDN as the browser checks whether the SRI checksums are matching and the correct files are being delivered.

That doesn't seem too hard. I don't use it. It's probably best in userland. It seems if you just have an array of chunkNames (which I can return in the final return as a minor change), you can use just the returned stylesheets and scripts and construct the <script /> and <link /> strings yourself by using each chunk name to get the integrity value from stats.compilation.assets[mainAssetName].integrity.
I.e. to create something like this:
<link
     rel="stylesheet"
     href="<%= htmlWebpackPlugin.files.css[index] %>"
     integrity="<%= htmlWebpackPlugin.files.cssIntegrity[index] %>"
     crossorigin="<%= webpackConfig.output.crossOriginLoading %>"
  />

And of course the crossOriginLoading it seems you will already know.
If you're using Webpack "magic comments" and React Universal Component you will in fact already have the chunk names for all dynamic chunks, and you should know the chunk names for named entry points.
Perhaps that solves your problem for now. You can use the Low-level API to achieve this.

I'm just gonna close this cuz I'm cleaning house and likely won't be able to prioritize it.

authors: faceyspacey swernerx

repo: faceyspacey/webpack-flush-chunks


text:
🛑 POST to /search is down

In 41c8f34, POST to /search (https://dev.factiverse.no/v1/search) was down:

HTTP code: 503
Response time: 93 ms

Resolved: POST to /search is back up in f0a1c16.

repo: factiverse/upptime-dev


text:
0.16 Recipes Support

Support 0.16 recipe set / data, and expose version picker.

https://www.reddit.com/r/factorio/comments/gje3t1/factorio_lab/fqls6tl?utm_source=share&utm_medium=web2x

repo: factoriolab/factorio-lab


text:
fix: "npm ci" → "npm install" message on project setup

Following the instructions on https://featurevisor.com/docs/quick-start/, at some point the CLI let me run npm ci on a project that doesn't have the package-lock.json file just yet (documentation says to run npm install actually`).

thanks so much, @maxgallo! <3
will be made available in new semver via this job: https://github.com/fahad19/featurevisor/actions/runs/5562703074/jobs/10161170467


text:
your site not worked

I want to see your site and your result, but the site is 404

His sight is off for now.


text:
patch the max_consumers parameter of the Falco kernel-module driver

Co-authored-by: Lorenzo Fontana lo@linux.com
Signed-off-by: Leonardo Di Donato leodidonato@gmail.com
What type of PR is this?
/kind bug
Any specific area of the project related to this PR?
/area build
What this PR does / why we need it:
Which issue(s) this PR fixes:
NONE
Special notes for your reviewer:
This bug happens when many Falco instances start because the check of how many driver consumers can work at the same time is not correct.
Does this PR introduce a user-facing change?:
fix: a bug that prevents Falco driver to be consumed by many Falco instances in some circumstances

[APPROVALNOTIFIER] This PR is NOT APPROVED
This pull-request has been approved by:
To complete the pull request process, please assign leodido after the PR has been reviewed.
You can assign the PR to them by writing /assign @leodido in a comment when ready.
The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

/cc @fntlnz

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: leogr
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

LGTM label has been added.  Git tree hash: 7a2672688b5a55482e903474bca0f98b19f627b9

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: fntlnz, leogr
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [fntlnz,leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

authors: leodido poiana

repo: falcosecurity/falco


text:
ci: remove arm64 jobs from GHA

What type of PR is this?
/kind cleanup
Any specific area of the project related to this PR?
/area CI
Does this PR require a change in the driver versions?
No
What this PR does / why we need it:
No we have arm64 jobs on circleCI so we don't need anymore these QEMU jobs
Which issue(s) this PR fixes:
Special notes for your reviewer:
Does this PR introduce a user-facing change?:
NONE

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [Andreagit97]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97, FedeDP
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [Andreagit97,FedeDP]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

LGTM label has been added.  Git tree hash: 416003d08669d09f0bfe6ee36707fe626306b61a

[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: Andreagit97, FedeDP, jasondellaluce
The full list of commands accepted by this bot can be found here.
The pull request process is described here

Needs approval from an approver in each of these files:

OWNERS [Andreagit97,FedeDP,jasondellaluce]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

repo: falcosecurity/libs


text:
add an prebuilt SDK-provided implementation of event source instances

Motivation
In order to develop plugins with event sourcing capability, developers need to provide a custom implementation of the falcosecurity::event_sourcer::instance interface, usually with the only purpose of overriding its next() method.
We had a similar situation with the Go SDK (see: https://github.com/falcosecurity/plugin-sdk-go/pull/50).
Feature
Having a prebuilt and SDK-provided implementation of falcosecurity::event_sourcer::instance that makes use of more high-level constructs and make life easier to plugin developers. A good candidate could be a "functional" pattern like the pull model of the Go SDK.

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale

/remove-lifecycle stale

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale

/remove-lifecycle stale

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale

authors: jasondellaluce poiana

repo: falcosecurity/plugin-sdk-cpp


text:
🛑 Tesscut is down

In 4f6e07e, Tesscut (https://mast.stsci.edu/tesscut) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Tesscut is back up in 77a08c9 after 7 minutes.

repo: falkben/mast-status


text:
Correct tests documentation

Since #461 documentation in tests/ is not truly accurate anymore. This needs a fix and improvement.

This got fixed with latest work on CI and tests

repo: farcaster-project/farcaster-node