text:
DDC doesn't work on MS Edge

Hy! I'm opening this issue based on this one: dart-lang/site-webdev#1940

Dart SDK Version: 2.5.0-dev.0.0
I'm using Windows, the issue is tested on two PC-s.
I'm using Edge.

The issue is:
webdev serve fails on Microsoft Edge. It's stuck on the "Loading..." which sits between the  tags.
longer explanation:
I tried to run the official quickstart repository, and it ran flawlessly on Chrome and Firefox, however, it produced the above mentioned error on Edge. I tried it with multiple repos on multiple computers with freshly cleaned Edge browsers, the issue stayed consistently.  I even tried to change the versions on various dependencies, no luck there.
Adding the "--release" tag to the webdev serve command solved the issue, so in the previous issue we came to the conclusion, that it must be something DDC releated, since switching the compiler to dart2js instantly solved the issue, on the same codebase where DDC failed.

Note, we don't official support Edge in debug mode (we do in release).  Informally, Edge / debug appears to work in dev Edge releases (which are Chromium based).

@vsmenon I thought we wanted to support Edge in debug mode, so that DartPad with Flutter would run in more browsers. (I can't find a bug about that, though.)

Since the older Chakra-based Edge is no longer distributed or supported, I'm going to close this as stale.
If anyone has issues with the newer Chromium-based Edge, please open a new issue. Thanks!
Thanks again for the report :)

authors: dewsy kwalrath parlough vsmenon


text:
Is it possible to use signal handler to detect stack overflow and remove aot compiler's "CheckStackOverflow" instructions, just like Android ART VM?

As title, Android ART VM's aot compiler use a signal handler chain to detect Java's StackOverflowException and NullPointerException:
NullPointerException is this commit:

https://android.googlesource.com/platform/art/+/b373e09

I think with this, we can reduce the instructions binary size after gen_snapshot and also may speed up the runtime performance.
@mraleph How do you think?
Recently I made a experiment, removed these CheckStackOverflow instructions in runtime/vm/compiler/frontend/base_flow_graph_builder.cc:
diff --git a/runtime/vm/compiler/frontend/base_flow_graph_builder.cc b/runtime/vm/compiler/frontend/base_flow_graph_builder.cc
index f510129c0ac..8832fe9d3b5 100644
--- a/runtime/vm/compiler/frontend/base_flow_graph_builder.cc
+++ b/runtime/vm/compiler/frontend/base_flow_graph_builder.cc
@@ -229,20 +229,22 @@ Fragment BaseFlowGraphBuilder::Return(TokenPosition position,
 Fragment BaseFlowGraphBuilder::CheckStackOverflow(TokenPosition position,
                                                   intptr_t stack_depth,
                                                   intptr_t loop_depth) {
-  return Fragment(new (Z) CheckStackOverflowInstr(
-      InstructionSource(position), stack_depth, loop_depth, GetNextDeoptId(),
-      CheckStackOverflowInstr::kOsrAndPreemption));
+  return Fragment();
 }

 Fragment BaseFlowGraphBuilder::CheckStackOverflowInPrologue(
     TokenPosition position) {
-  if (IsInlining()) {
-    // If we are inlining don't actually attach the stack check.  We must still
-    // create the stack check in order to allocate a deopt id.
-    CheckStackOverflow(position, 0, 0);
-    return Fragment();
-  }
-  return CheckStackOverflow(position, 0, 0);
+  return Fragment();
 }

And compile a local flutter engine, build an Android flutter-gallery's release apk(https://github.com/flutter/gallery), before and after size(only armv7's app.so):

From 23428712 bytes to 23069264 bytes (reduced 1.5%, 359448 bytes).

It's is possible, though you can't completely remove all them because they are used for more than just overflow checking - they also serve as a safepointing mechanism, so 1.5% saving is an overestimation.
@mkustermann has previously (long ago) done this experiment - see https://codereview.chromium.org/2334243002/, we did not go forward with this implementation because we were worried about embeddability / portability of the VM. Signal based solution is also much more complicated and finicky - though does have better code size (and potentially performance).
I have similarly prototyped a patch which turned null object into an actual 0 (or rather 1) and used signal handler to implement implicit null-checks. It worked but again it was too complicated - so I iced it as well.
As of today we don't have plans to move to signal handlers - because our concerns about portability and complexity remain. This might change in the future at some point.

I see, thank you! @mraleph  😄

source: gharchive/issue

authors: eggfly mraleph

repo: dart-lang/sdk

license_type: permissive

license_source: github-api


text:
Detect erroneous multi-line strings.

It is a (relatively) common error to forgot commas between two items of a string list:
var a = ["foo",
         "bar"  // <===
         "gee"];
In this case "bar" and "gee" will be concatenated without any warning.
It would be nice if the editor offered some kind of warning for this.
In general this is difficult, but the following heuristics could potentially work:

wrong number of arguments to a function call.
in lists: if the two strings easily fit onto one line. Here, for example, the user could just as well write "bargee" if she wanted one string. Having concatenated strings is thus probably an error.

I don't think the analyzer is going to do either of the suggestions:

We already warn about wrong number of arguments to a function.
I don't think the analyzer is going to start looking at whether the two strings fit on one line together... does it do any positional/formatting analysis like this today?

A request for this same functionality was just added to the linter project. It appears that it is a fairly common mistake, and we ought to provide support.

Ah. dart-lang/linter#225. The list literal case. I think we have to be careful in what we're fixing, to avoid generating noise warnings.
Should there just be a hint in List literals to never use "space"-concatenated String literals? I think the request in dart-lang/linter#225 is based on the idea that trailing plus signs are much more visible than trailing (or missing) commas.

Should there just be a hint in List literals to never use "space"-concatenated String literals?

That seems reasonable to me, although I'd make it a lint rather than a hint.

This is now implemented as a lint: no_adjacent_strings_in_list, and the formatter also indents such that the error is noticeable.

authors: bwilkerson floitschG srawlins


text:
Type is not cast

Same without <dynamic,dynamic>

But in that case it starts working

Dart SDK version: 2.17.6 (stable)
Mac OS arm64

You're not showing us the original declaration of origin.
If the third example allows promotion to work, most likely the origin variable is a local variable declared as something like Provider<T, dynamic>, some type where FactoryProvider<T, dynamic> is a subtype, but FactoryProvider<dynamic, dynamic> (or the equivalent FactoryProvider) is not, because the dynamic type argument is not a subtype of T.
Dart promotion requires the new type to be a subtype of the current type, otherwise it wouldn't be promotion.
Now, it would be nice if we could infer type arguments for is FactoryProvider which allows promotion, but we currently don't.

Great explanation. Thanks

authors: AlexanderFarkas lrhn


text:
wasm module does not work after binaryen

I have tried compiling one of our protobuf benchmarks to Wasm GC out of benchmarking curiosity using instructions provided by @osa1
$ git clone https://github.com/google/protobuf.dart.git
$ cd protobuf.dart/benchmarks
$ dart pub get
$ tool/compile_protos.sh
$ $DART_SDK_SRC/sdk/bin/dart2wasm bin/deep_copy.dart out/deep_copy.wasm

Resulting module runs fine on V8 (10.9.48):
$ ~/.jsvu/v8 --experimental-wasm-gc --wasm-gc-js-interop --experimental-wasm-stack-switching --experimental-wasm-type-reflection $DART_SDK_SRC/pkg/dart2wasm/bin/run_wasm.js -- out/deep_copy.wasm

But if I try TOT wasm-opt on it then any optimization level above -O1 breaks it:
$ ~/src/third_party/binaryen/bin/wasm-opt -all --nominal -O2 -o out/deep_copy.opt.wasm out/deep_copy.wasm
$ ~/.jsvu/v8 --experimental-wasm-gc --wasm-gc-js-interop --experimental-wasm-stack-switching --experimental-wasm-type-reflection $HOME/src/dart/sdk/pkg/dart2wasm/bin/run
_wasm.js -- out/deep_copy.opt.wasm
/usr/local/google/home/vegorov/src/dart/sdk/pkg/dart2wasm/bin/run_wasm.js:23: CompileError: WebAssembly.Module(): Compiling function #333 failed: type error in branch[0] (expected (ref any), got anyref) @+176222
        return new WebAssembly.Module(bytes);
               ^
CompileError: WebAssembly.Module(): Compiling function #333 failed: type error in branch[0] (expected (ref any), got anyref) @+176222
    at compile (/usr/local/google/home/vegorov/src/dart/sdk/pkg/dart2wasm/bin/run_wasm.js:23:16)
    at main (/usr/local/google/home/vegorov/src/dart/sdk/pkg/dart2wasm/bin/run_wasm.js:41:68)

1 pending unhandled Promise rejection(s) detected.

It's unclear to me where the bug is exactly. I am uploading deep_copy.wasm for convenience.
deep_copy.wasm.zip
/cc @askeksa-google @osa1 @joshualitt @eyebrowsoffire
/cc @kripken

This seems like a binaryen bug because it fails to validate the module it generates.
Original module runs fine:
d8 --experimental-wasm-gc --wasm-gc-js-interop --experimental-wasm-stack-switching --experimental-wasm-type-reflection ~/dart/sdk/sdk/pkg/dart2wasm/bin/run_wasm.js -- query_encode_binary.wasm
protobuf_query_encode_binary(RunTimeRaw): 13840 us.

wasm-opt validates the original module:
wasm-opt -all --nominal --name-types -S query_encode_binary.wasm -o query_encode_binary_opt.wat

Optimize the original module with wasm-opt:
wasm-opt -all --nominal --name-types query_encode_binary.wasm -o query_encode_binary_opt.wasm -O

wasm-opt fails to validate the resulting module:
wasm-opt -all --nominal --name-types -S query_encode_binary_opt.wasm -o query_encode_binary_opt.wat
[wasm-validator error in function 290] break type must be a subtype of the target block type, on
(block $label$62 (result (ref any))
 (br $label$61
  (ref.test_static ${mut:i32}
   (br_on_non_data $label$62
    (extern.internalize
     (local.get $0)
    )
   )
  )
 )
)
Fatal: error validating input

I can confirm this is hitting a binaryen bug. I'll investigate, thanks for the report!

There are at least two issues here, see the above links.
First, the wasm spec has an apparent bug. Until that is fixed and V8 is updated to that, the only way to work in V8 is to disable the gto pass. That is run by default in -O2 and above. It can be manually disabled by removing the relevant line. You can also use -O1 which does not run it (but also does not run a lot of other useful stuff like inlining and devirtualization).
Second, the Binaryen optimizer had a bug with the recently introduced internalize/externalize instructions, see the PR for a fix.
With gto disabled and that PR included, I can optimize deep_copy.wasm at various opt levels without errors in Binaryen or V8.
@osa1 I can't reproduce the error you refer to on deep_copy.wasm. Where can I get that query_encode_binary.wasm file that you mention?

@kripken I see that the spec is actually correct - which means V8 does not implement it properly? Is there a monorail bug for this or should I file one?

@kripken query_encode_binary.wasm attached, though it seems like after your PR in binaryen, binaryen now validates the optimized version of it.
query_encode_binary.zip

@mraleph I'm not aware if there's a V8 bug filed or not - @jakobkummerow ?
@osa1 Thanks, I can reproduce the "break type" error with that file. I'll look into it.

@osa1 Ok, good, I am seeing the same as you - that was the same issue as fixed by that PR, and things work properly after it landed.

Then I think we are all good! Thanks @kripken @jakobkummerow!
I can run benchmarks now. They are rather slow (12x/6x slower than native or 2x/6x slower than JS, depending on the benchmark), but that's just more work for dart2wasm team then :)

Let me know if you see anything we can optimize on the binaryen side!

authors: kripken mraleph osa1


text:
Web compiler crash

Dart SDK 3.1.1 (shipped with flutter 3.13.3)
OS: Windows 11

Log
Compiling lib/main.dart for the Web...
Target dart2js failed: ProcessException: Process exited abnormally:
../../../../../../../AppData/Local/Pub/Cache/hosted/pub.dev/go_router-10.1.2/lib/src/information_provider.dart:125:34:
Warning: Operand of null-aware operation '!' has type 'String' which excludes null.
location: routeInformation.location!,
^
../../../../../../../AppData/Local/Pub/Cache/hosted/pub.dev/go_router-10.1.2/lib/src/parser.dart:87:50:
Warning: Operand of null-aware operation '!' has type 'String' which excludes null.
configuration.findMatch(routeInformation.location!, extra: state.extra);
^
file:///C:/Users/remy/Documents/dev/workspace/veolys/nouvelatrium/apps/nouvelatrium/.dart_tool/flutter_build/387c38ab88f96dc7786f99cb83c14955/main.dart:
Internal Error: The compiler crashed when compiling this element.

Stack trace
The compiler crashed: Invalid argument(s): Type parameter TypeParameter(T) is not indexed
#0      TypeParameterIndexer.[] (package:kernel/binary/ast_to_binary.dart:3371:8)
#1      BinaryPrinter.visitTypeParameterType (package:kernel/binary/ast_to_binary.dart:2528:38)
#2      TypeParameterType.accept (package:kernel/ast.dart:12528:42)
#3      BinaryPrinter.writeNode (package:kernel/binary/ast_to_binary.dart:439:10)
#4      BinaryPrinter.writeNodeList (package:kernel/binary/ast_to_binary.dart:349:7)
#5      BinaryPrinter.visitInterfaceType (package:kernel/binary/ast_to_binary.dart:2435:7)
#6      InterfaceType.accept (package:kernel/ast.dart:11385:42)
#7      BinaryPrinter.writeNode (package:kernel/binary/ast_to_binary.dart:439:10)
#8      BinaryPrinter.visitAsExpression (package:kernel/binary/ast_to_binary.dart:1960:5)
#9      AsExpression.accept (package:kernel/ast.dart:7636:44)
#10     BinaryPrinter.writeNode (package:kernel/binary/ast_to_binary.dart:439:10)
#11     BinaryPrinter.writeOptionalNode (package:kernel/binary/ast_to_binary.dart:531:7)
#12     BinaryPrinter.visitReturnStatement (package:kernel/binary/ast_to_binary.dart:2315:5)
#13     ReturnStatement.accept (package:kernel/ast.dart:10127:43)
#14     BinaryPrinter.writeNode (package:kernel/binary/ast_to_binary.dart:439:10)
#15     BinaryPrinter.writeOptionalNode (package:kernel/binary/ast_to_binary.dart:531:7)
#16     BinaryPrinter.visitFunctionNode (package:kernel/binary/ast_to_binary.dart:1529:5)
#17     FunctionNode.accept (package:kernel/ast.dart:3908:38)
#18     BinaryPrinter.writeFunctionNode (package:kernel/binary/ast_to_binary.dart:446:10)
#19     BinaryPrinter.visitProcedure (package:kernel/binary/ast_to_binary.dart:1357:5)
#20     Procedure.accept (package:kernel/ast.dart:3311:40)
#21     BinaryPrinter.writeProcedureNode (package:kernel/binary/ast_to_binary.dart:467:10)
#22     BinaryPrinter.writeProcedureNodeList (package:kernel/binary/ast_to_binary.dart:358:7)
#23     BinaryPrinter.visitClass (package:kernel/binary/ast_to_binary.dart:1260:5)
#24     Class.accept (package:kernel/ast.dart:1422:38)
#25     BinaryPrinter.writeClassNode (package:kernel/binary/ast_to_binary.dart:481:10)
#26     BinaryPrinter.writeClassNodeList (package:kernel/binary/ast_to_binary.dart:376:7)
#27     BinaryPrinter.visitLibrary (package:kernel/binary/ast_to_binary.dart:1096:5)
#28     Library.accept (package:kernel/ast.dart:577:38)
#29     BinaryPrinter.writeLibraryNode (package:kernel/binary/ast_to_binary.dart:460:10)
#30     BinaryPrinter.writeLibraries (package:kernel/binary/ast_to_binary.dart:766:9)
#31     BinaryPrinter.writeComponentFile. (package:kernel/binary/ast_to_binary.dart:605:7)
#32     Timeline.timeSync (dart:developer/timeline.dart:171:22)
#33     BinaryPrinter.writeComponentFile (package:kernel/binary/ast_to_binary.dart:586:14)
#34     SerializationTask.serializeComponent. (package:compiler/src/serialization/task.dart:77:15)
#35     CompilerTask.measureSubtask (package:compiler/src/common/tasks.dart:181:35)
#36     SerializationTask.serializeComponent (package:compiler/src/serialization/task.dart:67:5)
#37     Compiler.produceKernel (package:compiler/src/compiler.dart:422:29)

#38     Compiler.runSequentialPhases (package:compiler/src/compiler.dart:711:20)

#39     Compiler.runInternal. (package:compiler/src/compiler.dart:311:7)

#40     Compiler.runInternal (package:compiler/src/compiler.dart:310:5)

#41     Compiler.run. (package:compiler/src/compiler.dart:231:11)

#42     main (package:compiler/src/dart2js.dart:1241:3)

Command: C:\Users\remy\Documents\dev\flutter\bin\cache\dart-sdk\bin\dart.exe --disable-dart-dev C:\Users\remy\Documents\dev\flutter\bin\cache\dart-sdk\bin\snapshots\dart2js.dart.snapshot --platform-binaries=C:\Users\remy\Documents\dev\flutter\bin\cache\flutter_web_sdk\kernel --invoker=flutter_tool -Ddart.vm.product=true -Denv=sit -DFLUTTER_WEB_AUTO_DETECT=true -DFLUTTER_WEB_CANVASKIT_URL=https://www.gstatic.com/flutter-canvaskit/b8d35810e91ab8fc39ba5e7a41bff6f697e8e3a8/ --native-null-assertions -o C:\Users\remy\Documents\dev\workspace\veolys\nouvelatrium\apps\nouvelatrium.dart_tool\flutter_build\387c38ab88f96dc7786f99cb83c14955\app.dill --packages=.dart_tool/package_config.json --cfe-only C:\Users\remy\Documents\dev\workspace\veolys\nouvelatrium\apps\nouvelatrium.dart_tool\flutter_build\387c38ab88f96dc7786f99cb83c14955\main.dart
#0      RunResult.throwException (package:flutter_tools/src/base/process.dart:125:5)
#1      _DefaultProcessUtils.run (package:flutter_tools/src/base/process.dart:278:19)

#2      Dart2JSTarget.build (package:flutter_tools/src/build_system/targets/web.dart:189:5)

#3      _BuildInstance._invokeInternal (package:flutter_tools/src/build_system/build_system.dart:853:9)

#4      Future.wait. (dart:async/future.dart:525:21)

#5      _BuildInstance.invokeTarget (package:flutter_tools/src/build_system/build_system.dart:791:32)

#6      Future.wait. (dart:async/future.dart:525:21)

#7      _BuildInstance.invokeTarget (package:flutter_tools/src/build_system/build_system.dart:791:32)

#8      FlutterBuildSystem.build (package:flutter_tools/src/build_system/build_system.dart:620:16)

#9      WebBuilder.buildWeb (package:flutter_tools/src/web/compile.dart:89:34)

#10     BuildWebCommand.runCommand (package:flutter_tools/src/commands/build_web.dart:198:5)

#11     FlutterCommand.run. (package:flutter_tools/src/runner/flutter_command.dart:1297:27)

#12     AppContext.run. (package:flutter_tools/src/base/context.dart:150:19)

#13     CommandRunner.runCommand (package:args/command_runner.dart:212:13)

#14     FlutterCommandRunner.runCommand. (package:flutter_tools/src/runner/flutter_command_runner.dart:339:9)

#15     AppContext.run. (package:flutter_tools/src/base/context.dart:150:19)

#16     FlutterCommandRunner.runCommand (package:flutter_tools/src/runner/flutter_command_runner.dart:285:5)

#17     run.. (package:flutter_tools/runner.dart:115:9)

#18     AppContext.run. (package:flutter_tools/src/base/context.dart:150:19)

#19     main (package:flutter_tools/executable.dart:90:3)

@srujzs This looks like https://github.com/dart-lang/sdk/issues/53291

Indeed it does. @llfbandit, can you share a repro here?

@johnniwinther is right. We're in the exact same situation as the other issue. I use also the record package in my app.
Simply adding go_router seems to be not enough to reproduce the crash. Adding record package makes the issue reproducible.
Funny enough, I'm the author of the record package. Is there any action to take on my side to workaround or lower the risk for such issue for the future?

The issue arises from the intersection of @staticInterop, type parameters, and factories. I don't really think you can work around using a factory unless you start using a static method that forwards to a js_util call or something, and getting rid of type parameters is probably going to be painful for your users.
I think a cherry-pick for this change makes sense so it can be available before 3.2.

authors: johnniwinther llfbandit srujzs


text:
Unexpected non-zero index in CalculateElementAddressInstr

import 'dart:typed_data';

Uint8List var7 = Uint8List(37);
Int32List? var16 = Int32List(2);

@pragma("vm:entry-point")
foo(int par3) {
  switch (par3) {
    case 308366437:
      var16 = Int32List(2).sublist(par3, 8);
      break;
  }
}

main() {
  try {
    foo(var7[-9223372032559808513]);
  } catch (e, st) {
    print('X2() throws');
  }
}

DART_CONFIGURATION='DebugX64C' DART_VM_FLAGS='--enable-asserts' pkg/vm/tool/precompiler2 --optimization_level=3 fuzz.dart snapshot

../../runtime/vm/compiler/backend/il_x64.cc: 382: error: expected "0" = 0, actual "Smi::Cast(index_loc.constant()).Value()" = 308366437

import 'dart:typed_data';
Uint8List var7 = Uint8List(37);
Int32List? var16 = Int32List(2);
@pragma("vm:entry-point")
foo(int par3) {
switch (par3) {
case 308366437:
if (par3 >= 0 && par3 < 2) {
var16 = Int32List(2).sublist(par3, 2);
} else {
print('Invalid range for sublist');
}
break;
default:
print('Unhandled case');
}
}
main() {
try {
int index = 0;
if (index >= 0 && index < var7.length) {
foo(var7[index]);
} else {
print('Index out of bounds');
}
} catch (e, st) {
print('X2() throws');
print(e);
print(st);
}
}

/cc @sstrickl

authors: AkashKeote alexmarkov rmacnak-google


text:
Make Float32x4 const constructible

As part of adding wide gamut color support to Flutter I'm looking to migrate Color to hold floating point values.
If I use double color components this will quadruple the size of Color.  It would be nice to use Float32x4 but Float32x4 is not const constructible despite being immutable.  Using Float32x4 would make Color not const constructible which would be a huge breaking change.
There are performance benefits to storing things in Float32x4 too for things like colorspace conversions.
This isn't required for the the wide gamut work but it would be nice to have.

This will require special support in compilers and back-ends.  A Float32x4 constructor requires doing conversion from double to 32-bit float, which Dart has no direct method for, and definitely no constant operation for.
The memory layout of a Float32x4 is not a normal Dart object, it's 128 bits of storage that's interpreted specially when accessed, and initializing that special storage at compile-time is a new feature that doesn't exist today.
Being immutable is not the same as being compatible with being constant. If the operations performed by the constructor aren't trivial, it's not just a normal const constructor.

Mraleph framed it well in the Dart discord.  The gist is that Float32x4 is conceptually no different than something like a double or a string and the mechanism that is used to make those const constructible should be used for the SIMD types too.

Somewhat related https://dart-review.googlesource.com/c/sdk/+/357440

authors: gaaclarke lrhn rmacnak-google


text:
[vm] Bug fix in dedup optimization when build deferred components.

Dedup optimization will remove duplicated Code object and keep one instance.
When building deferred components, the only Code object (e.x. as callee B) maybe
exist in different loading unit from the caller A. This will cause a rellocation
bug if A->B was a relative call.
Add a flag to indicate building with deferred components.
When working with dedup and deferred components, assignUnit first and
Code object from different unit is unequal.
Issues: https://github.com/dart-lang/sdk/issues/49372
Signed-off-by: fangzheyuan fangzheyuan@bytedance.com

This Pr fix one bug in building deferred component. I first found this error in https://github.com/dart-lang/sdk/issues/49372.
At first, i thought this bug only existed in older versions. But when i found out the real cause of the issue, i tnink this bug still exist in current master branch.
I will explain this bug.
___TabsNonScrollableDemoState&State&SingleTickerProviderStateMixin&RestorationMixin._updateProperty) from Code([Optimized] __HomePageState&State&SingleTickerProviderStateMixin&RestorationMixin.registerForRestoration.listener)
This error log shows than xxx.registerForRestoration.listener(caller) called xxx._updateProperty(callee) but callee could be
found in relocation phase.
__HomePageState is in unit4 and ___TabsNonScrollableDemoState is in another unit and this call entry's kind is relative call, that means caller and callee shouid in the same unit. But they are in the different units, so crash.
The cause is the dedup optimization. Actually 2.16.2's dedup. In 2.16.2, the dedup try to merge Code Object instead of instruciton.
Before dedup, this call entry is from __HomePageState.xxx.registerForRestoration.listener to __HomePageState.xxx._updateProperty, relative call is ok.
After dedup, __HomePageState.xxx._updateProperty is replaced with ___TabsNonScrollableDemoState.xxx and ___TabsNonScrollableDemoState is not exist in unit4.

Thanks for the contribution! Great catch.
Initial code review comments are here: https://dart-review.googlesource.com/c/sdk/+/250491

Change to  https://dart-review.googlesource.com/c/sdk/+/250720/

source: gharchive/pull-request

authors: lawangle mraleph


text:
Install a Flutter SDK in the publish workflow

Supposed to fix https://github.com/dart-lang/setup-dart/issues/68
Not tested yet.

Reopening in my own fork

repo: dart-lang/setup-dart

license_source: bigquery


text:
How to generate genesis state and header for new parachain from an existing substrate chain?

If parachain deployment does not support hot-swap in fly, then for an existing substrate chain being able to connect to relay chain, it will need to prepare genesis state data for collators and prepare genesis header for parachain auction.
We need to have solutions to generate this genesis state and genesis header for parachain.
And btw, how to stop an existing chain by upgrade runtime/client. (A tricky but simple way to stop produce/finalize new blocks). Since not all the clients will upgrade(they might do not agree with the parachain plan), an onchain governance  through runtime upgrade is necessary.

We can start draft a summary document about the solution:
Using fork-off-substrate, scraping the on-chain data, and customize according to the parachain requirement.

The hot-swap requirements are also from the bridge network connecting to relay chains, if we want to use Crab/Darwinia directly as parachain:
https://user-images.githubusercontent.com/1070122/121144387-851b8e00-c870-11eb-80f2-2711b45a0dd3.png

One approach will be using bridge messages to migrate the state of solo chain to the parachain, instead of current approach of hot-swap, how generate the genesis state from solo-chain(Have to stop the solo-chain which is unacceptable).

repo: darwinia-network/research

license: MIT


text:
Sign ProRegTx collaterals with a string message instead of payload hash

Required for HW wallet support. See https://github.com/dashpay/dips/pull/28

Hmm I'm currently not happy with including the magic string "DarkCoin Signed Message:\n" in the message. Should we maybe change CMessageSigner to allow signing without the magic str or with a specific magic str for the use case?

@codablock, I missed what @UdjinM6 noticed: protx_register function doesn't accept parameter for the payloadSig field of ProRegTx transaction.
Here is the assumption, that the collateral key is controlled by the the local node's wallet, but we also should use the payloadSig if provided as an argument.

@Bertrand256 @UdjinM6 My expectation was that the external tool which does the signature would also have to fully implement serialization of the payload and TX. Otherwise it wouldn't be able to calculate the payload hash to actually create the message to sign, as it also includes the payload hash. If the external tool creates the whole transaction, then it can also pass the fully signed TX to the network.

My expectation was that the external tool which does the signature would also have to fully implement serialization of the payload and TX.

Yep, this was smth I assumed as well but it seems that HW wallets won't be able to sign this "weird" raw tx because there is no way to how to deserialize/hash/sign txes with data they don't recognize. For trezor the flow is https://wiki.trezor.io/Developers_guide:Message_Workflows#SignTx. Note that there is TXEXTRADATA for zcash but it's not working the way we need, it's a per-input data as far as I was able to understand. This means that the only way to sign such tx (without modifying HW wallet firmware) is Dash Core (or whatever wallet/tool will be able to sign special txes). So the workflow I see atm would be smth like this:

create non-signed tx with empty payloadSig via proregtx (proregtx create?) in Dash Core, outputs rawtx + payload "text" to sign;
sign the payload "text" via SignMessage in some external tool (could be DMT or even official wallet app/page);
sign and submit proregtx by providing rawtx from step 1 and sig from step 2 (proregtx submit?).

The process is far from being user friendly unfortunately but I don't see how to simplify it atm.

@UdjinM6 Why do we need step 1? Is that non-signed tx to be used along with the payload data to create a signature in step 2 and thus act as a protection against replay or smth else?

Added 2 commits:

Rename fund_register to register_fund (better grouping of commands)
Implemented register_prepare and register_submit. register_prepare expects the same args as the other 2 register commands but returns an unsigned ProRegTx and the message to sign. You can then use signmessage (or a HW wallet) with the collateral address and the message to sign that message. register_submit is then used to finalize and submit the TX.

Also rebased on develop.

@Bertrand256 we need step 1 to construct the payload data

repo: dashpay/dash

license: mit


text:
[ENH] Support a "coalesce" repartitioning type hint

In SQL, it's common to work w/ large data and aggregate or filter it down to few enough rows that it could be merged into a single partition in memory.
Today you can achieve this with something like:
import pandas as pd
import dask.dataframe as dd

pdf = pd.DataFrame({'val': [0, 1, 2, 3, 4, 5]})
ddf = dd.from_pandas(pdf, npartitions=2)

c.create_table('test', ddf)
print(c.sql("select * from test").npartitions)
# 1
coalesced_ddf = c.sql("select * from test").repartition(npartitions=1)
c.create_table("test_coalesced", coalesced_ddf)
print(c.sql("select * from test_coalesced").npartitions)
# 2

It would be nice to support something like:
c.sql("""
SELECT /*+ COALESCE(1) */ * FROM test
""").npartitions
# 1

As a motivator, Dask DataFrames can use a broadcast or "map-side" join if one of the DataFrames consists of a single partition. Allowing users to specify partition coalescing hints will allow finer control over performance of Dask-SQL join performance.

I can handle this one

repo: dask-contrib/dask-sql


text:
Cluster keeps appending "interface" flag to job script

When I run the following code snippet:
from dask_jobqueue import SLURMCluster

cluster = SLURMCluster(memory="100GB", cores=40, interface="ib0")

for _ in range(20):
    _ = cluster.job_script()

print(cluster.job_script())

I get the following output:
#!/usr/bin/env bash

#SBATCH -J dask-worker
#SBATCH -n 1
#SBATCH --cpus-per-task=40
#SBATCH --mem=94G
#SBATCH -t 00:30:00

/path/to/python -m distributed.cli.dask_worker tcp://xx.xx.xx.xx:pppp --nthreads 5 --nworkers 8 --memory-limit 11.64GiB --name dummy-name --nanny --death-timeout 60 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0 --interface ib0

It seems everytime the job script method is called, the interface flag is appended to it. I am running dask-jobqueue==0.8.0  off the conda-forge channel. (Python 3.10 if relevant)
You can replicate this with PBSCluster as well. The issue seems to be in how worker_extra_args are handled in the __init__() method of dask_jobqueue.core.Job

Hi @theXYZT,
Good catch!
It seems that when we retrieve the worker-extra-args from dask.config (https://github.com/dask/dask-jobqueue/blob/main/dask_jobqueue/core.py#L226), as we don't make a copy, the object is kept in memory in dask.config, and so we retrieve the same object each time we construct a Job.
I'm not sure why yet, but this issue was not present in previous release (0.7.4) before we renamed extra in worker-extra-args.
I'll propose some fix, but I'd like to understand which change we made that lead to this result. cc @jolange if you have some idea.
@theXYZT does is causes errors on your system or is it just a visual problem?

I am not sure if this causes any problems currently. Although I use a few more worker-extra-args and all of them get duplicated quite a bit in the job script over time.

I did not notice this funny feature so far, thanks! From a quick look at the renaming diffs, I don't see what might have caused that, but if you are sure that it did not happen in the previous release, I will have a closer look later!

From a quick look at the renaming diffs, I don't see what might have caused that

Neither do I, but I tested the problem on 0.7.4 and it did not behave that way. I admit I have no idea if this comes from the renaming or something else. It's ok if you don't have time, I just wanted to know if something strikes you.

Ok, this is a subtle combination of two things:

The one you noted: The config is assigned to worker_extra_args:
https://github.com/dask/dask-jobqueue/blob/f79f9136542abd86566e0a36f7370c144052ee9d/dask_jobqueue/core.py#L226-L228
In the renaming, I changed from X = X + NEW to X += NEW:

         if interface:
-            extra = extra + ["--interface", interface]
+            worker_extra_args += ["--interface", interface]

This overwrites the config, whereas before worker_extra_args (or extra then) was newly assigned. I will revert that!

Nice catch and many thanks @jolange.

authors: guillaumeeb jolange theXYZT

repo: dask/dask-jobqueue


text:
Statuts update on Incremental and Grid Search

Quick status update: I wanted to explore a workflow that would use scikit-learn
as much as possible. We'd use scikit-learn for all the hyper-parameter
optimization and the actual training. Dask would just provide the large
arrays.
The end goal is something as close as possible to GridSearchCV(SGDClassifier()) trained on a larger-than-memory dataset.
X, y = load_dask_arrays()

clf = sklearn.linear_model.SGDClassifier()
gs = sklearn.model_selection.GridSearchCV(clf, param_grid)
gs.fit(X, y)

First, we have to avoid passing a large Dask array to SGDClassifier.fit, as it
would be converted to an ndarray. So we wrap it with Incremental, which passes
blocks to the SGDClassifier.partial_fit:
import dask_ml.wrappers

X, y = load_dask_arrays()

clf = sklearn.linear_model.SGDClassifier()
inc = dsak_ml.wrappers.Incremental(clf)
gs = sklearn.model_selection.GridSearchCV(clf, param_grid)
gs.fit(X, y)

At this point, inc.fit(X, y) works fine (I'm currently debugging unexpectedly
high memory usage, but let's ignore that for now), but there's a subtle issue with
inc.score(X, y), which surfaces in GridSeachCV. By default, a pass-through
scorer is used, which is SGDClassifier.score. This ends up using
sklearn.metrics.accuracy_score, which end up converting the test Dask arrays
into large ndarrays on a single worker. This is captured in #200.
A workaround to the scoring issue is to manually pass a scorer that is able to
work well with Dask arrays. We've implemented a few in Dask-ML:
import dask_ml.wrappers
import dask_ml.metrics
from sklearn.metrics import make_scorer

X, y = load_dask_arrays()
scorer = make_scorer(dask_ml.metrics.accuracy_score)

clf  = sklearn.linear_model.SGDClassifier()
inc = dask_ml.wrappers.Incremental(clf, scoring=scorer)
gs = sklearn.model_selection.GridSearchCV(clf, param_grid)
gs.fit(X, y)

This gets us serial hyper-parameter optimization on larger-than-memory Dask
arrays. To do things in parallel, we can use the distributed backend.
import dask_ml.joblib
from sklearn.externals import joblib

with joblib.parallel_backend("dask"):
    gs.fit(X, y)

I'll post benchmarks later when I've run them.

cc @stsievert

Here's a gist showing out of core SGD https://gist.github.com/245d492f95f2f4d6519dffdc3bd72ed9 (no hyperparameter optimization yet).
Here's the scheduler UI on a 24 GB dataset (my laptop has 16GB of RAM): https://streamable.com/1goff
We go through all of the zarr reading (including some re-writing to disk since the dataset is too large), before moving onto the first fit.
Ideally, we would start fitting the first block, and releasing it from memory, as soon as it's available. I haven't dug into why we don't do this yet.

This is the graph for a smaller problem.

generated on
make_data(n_samples=50_000, n_features=500, chunks=5_000, random_state=0)
from dask_ml._partial import fit

val = fit(clf, X, y, classes=[0, 1], compute=False)
val.visualize(rankdir='LR', color='order');

Interesting.  It's preferring the outer nodes first instead of the inner ones.  It would be good to investigate why this is.

If you want to dive into dask.order that is

Yes, I would like to debug this, though I may ping you if I get stuck :)

The tests for ordering are probably a good place to start.  The test
docstrings might be informative about the kinds of cases we consider.
On Fri, Jun 8, 2018 at 4:03 PM, Tom Augspurger notifications@github.com
wrote:

Yes, I would like to debug this, though I may ping you if I get stuck :)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask-ml/issues/206#issuecomment-395874785, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszKcAszQ0glmI4ElDYhNlgNejBSF3ks5t6tipgaJpZM4UgiQo
.

Interestingly, we take the right path for a totally random dataset from da.random.normal
from dask_ml._partial import fit

X = da.random.uniform(size=(50_000, 500), chunks=5_000)
y = da.random.uniform(size=(50_000,), chunks=5_000)
val = fit(clf, X, y, classes=[0, 1], compute=False)

val.visualize('random-uniform', rankdir='LR', color='order');

Some notes comparing those two, da.random.uniform (good) and zarr (bad).
Identical up on the initial pass through stack up until the items
(da.random.uniform, 0, 0) / (X.zarr, 0, 0). They have different dependents.

random uniform has no deps
(X.zarr, 0, 0) has one: {'zarr-X.zarr'}

So we diverge here. The da.random.uniform version correctly goes (X, 0, 0), (y, 0), fit-0, ... at this point.
The zarr version throws zarr-X.zarr on the stack and then starts working through (X, 9, 0), (X, 8, 0), ... (y, 9), (y, 8), ... fit-0, fit-1, ..., fit-9)

Here's the some trimmed total_dependents:
# random version
{('da.random.uniform-1fbca3c0347dfde415f84ec4a950bd61', 0, 0): 11,
# ...
 ('da.random.uniform-1fbca3c0347dfde415f84ec4a950bd61', 9, 0): 2,
 ('da.random.uniform-fce7cf97942af8ebd4353a56e37d6783', 0): 11,
# ...
 ('da.random.uniform-fce7cf97942af8ebd4353a56e37d6783', 9): 2,
 ('fit-630d7694014736f7c87d2604689f65d9', -1): 11,
 ('fit-630d7694014736f7c87d2604689f65d9', 0): 10,
# ...
 ('fit-630d7694014736f7c87d2604689f65d9', 9): 1}

# Zarr version
{'zarr-X.zarr': 66,
 'zarr-y.zarr': 66,
 ('fit-e1fcb35a27c9528708d6eb0ab04b0f47', -1): 11,
 ('fit-e1fcb35a27c9528708d6eb0ab04b0f47', 0): 10,
...
 ('fit-e1fcb35a27c9528708d6eb0ab04b0f47', 9): 1,
 ('zarr-X.zarr', 0, 0): 11,
...
 ('zarr-X.zarr', 9, 0): 2,
 ('zarr-y.zarr', 0): 11,
...
 ('zarr-y.zarr', 9): 2}

I wonder if it's as simple as Zarr-X.zarr has lots of dependents, so lets go through that first...

I suspect that in practice data would be coming from a single source and then split apart rather than two separate sources.  Task ordering might behave better in this situation.

Interesting. It's preferring the outer nodes first instead of the inner ones. It would be good to investigate why this is.

This is looking reminiscent of the issues we have seen with ordering in map_overlap. ( https://github.com/dask/dask/issues/3554 ) Is it possible they share a common cause? Don't know too much about how ordering works currently. So apologize if this is a naive question.

I suspect that in practice data would be coming from a single source and then split apart rather than two separate sources. Task ordering might behave better in this situation.

Not for this simple example unfortunately (X and y in the same Zarr file, read in together, and then split).

def make_data(n_samples=1_000_000, n_features=1_000, chunks=5_000, random_state=None):
    X, y = make_classification(n_samples=n_samples, n_features=n_features, chunks=chunks,
                               random_state=random_state)
    y = y.reshape(-1, 1)
    Xy = da.concatenate([X, y], axis=1)
    print("Generating random X-y pair")
    print(f"{X.nbytes/1e9} GB")
    print(f"{X.numblocks} blocks")
    da.to_zarr(Xy, "Xy.zarr", overwrite=True)

def load_data():
    Xy = da.from_zarr("Xy.zarr")
    y = Xy[:, -1]
    X = Xy[:, :-1]
    return X, y

X, y = load_data()
val = fit(clf, X, y, classes=[0, 1], compute=False)
val.visualize('xy-zarr', rankdir='LR', color='order');

That seems like an excellent fail case.  My next step here would be to make an minimal abstract test, similar to what is found in dask/tests/test_order.py and then tweak order.py to see what is going on.

Ok, I think this is a fairly minimal example:
def f(*args):
    pass

dsk1 = {
    'X0': 0,
    'y0': 0,
    'b0': 0,
    'b1': (f, 'b0', 'X0', 'y0'),
    'X1': 1,
    'y1': 1,
    'b2': (f, 'b1', 'X1', 'y1')
}

dsk2 = {
    'Xy': 0,
    'X0': (f, 'Xy', 0, 0),
    'y0': (f, 'Xy', 0, 1),
    'b0': 0,
    'b1': (f, 'b0', 'X0', 'y0'),
    'X1': (f, 'Xy', 1, 0),
    'y1': (f, 'Xy', 1, 1),
    'b2': (f, 'b1', 'X1', 'y1')
}

dsk1 This is handled well. The blocks of X don't have a common ancestor. This is the "shallow" case, and we make the right choice.

dsk2 is not handled well. The blocks of X (and y) share a common ancestor. We choose to go down the outer edge first

I'll start playing with the ordering heuristics.

make an minimal abstract test

(That was a useful exercise)

Have fun!  This will undoubtedly introduce you to the many other graph motifs like this that we try to handle optimally.  My experience is that in trying to touch any part of this problem you undoubtedly end up having to load the entire thing into your brain.  This isn't necessarily bad, it's an interesting and reasonably sized problem.

https://github.com/dask/dask/pull/3588 solves the ordering issue.

authors: TomAugspurger jakirkham mrocklin

repo: dask/dask-ml


text:
DataFrame merge: clarify that merging on column puts all matching values into one partition

I naively tried to do dd.merge(a, b, on="column_with_ten_values"), where a and b were both large DataFrames with thousands of partitions each.
Eventually the compute failed with:
[File /opt/conda/envs/coiled/lib/python3.9/site-packages/dask/dataframe/multi.py:275, in merge_chunk()

File /opt/conda/envs/coiled/lib/python3.9/site-packages/pandas/core/frame.py:9329, in merge()

File /opt/conda/envs/coiled/lib/python3.9/site-packages/pandas/core/reshape/merge.py:122, in merge()

File /opt/conda/envs/coiled/lib/python3.9/site-packages/pandas/core/reshape/merge.py:716, in get_result()

File /opt/conda/envs/coiled/lib/python3.9/site-packages/pandas/core/reshape/merge.py:967, in _get_join_info()

File /opt/conda/envs/coiled/lib/python3.9/site-packages/pandas/core/reshape/merge.py:941, in _get_join_indexers()

File /opt/conda/envs/coiled/lib/python3.9/site-packages/pandas/core/reshape/merge.py:1509, in get_join_indexers()

File ~/Library/Caches/pypoetry/virtualenvs/workshop-one-3dbJnD_1-py3.9/lib/python3.9/site-packages/pandas/_libs/join.pyx:48, in pandas._libs.join.inner_join()
     ]()[46](file:///~/Library/Caches/pypoetry/virtualenvs/workshop-one-3dbJnD_1-py3.9/lib/python3.9/site-packages/pandas/_libs/join.pyx?line=45)[             count += lc * rc
     ]()[47](file:///~/Library/Caches/pypoetry/virtualenvs/workshop-one-3dbJnD_1-py3.9/lib/python3.9/site-packages/pandas/_libs/join.pyx?line=46)[
---> ]()[48](file:///~/Library/Caches/pypoetry/virtualenvs/workshop-one-3dbJnD_1-py3.9/lib/python3.9/site-packages/pandas/_libs/join.pyx?line=47)[ left_indexer = np.empty(count, dtype=np.intp)
     ]()[49](file:///~/Library/Caches/pypoetry/virtualenvs/workshop-one-3dbJnD_1-py3.9/lib/python3.9/site-packages/pandas/_libs/join.pyx?line=48)[ right_indexer = np.empty(count, dtype=np.intp)
     ]()[50](file:///~/Library/Caches/pypoetry/virtualenvs/workshop-one-3dbJnD_1-py3.9/lib/python3.9/site-packages/pandas/_libs/join.pyx?line=49)[

MemoryError: Unable to allocate 224. TiB for an array with shape (30824647463820,) and data type int64]()

I was confused why a 224 TiB array needed to be allocated. As a user, this was a strange and concerning error to see. Eventually I realized that my on column had only a few unique values, so (I think) it was trying to concatenate all those values across all partitions into one during the shuffle stage of the hash join. Because the shuffle ensures that:

After this operation all elements that have the same index will be in the same partition.

It would be good to:

Document clearly that when merging on a column, every row with the same value needs to be able to fit into one partition. If there are only a few unique values for the column, but many rows, you'll likely encounter an error.
Catch/preemptively check for this error at runtime, and reraise a more informative error to users. Not sure what alternative we'd suggest though. The fact that divisions, and by extension shuffling, can't handle repeated values (xref https://github.com/dask/dask/issues/8437) I think makes this fundamentally unsupported right now?

cc @jsignell

The documentation task seems like a good-first-issue :)

repo: dask/dask


text:
Add (lazy) shape property to dataframe and series

[x] Tests added / passed
[x] Passes flake8 dask
[x] Fully documented, including docs/source/changelog.rst for all changes
and one of the docs/source/*-api.rst files for new API

Since dask provides (lazy) size, and in order to get close to Pandas API, I propose an implementation of shape property, both for Dataframes and Series.
Tests Example
Imports

import dask
import numpy as np
import pandas as pd
import dask.dataframe as dd

Generation Function

#generate dummy df
def generate_df(size):
    d = {
        'dates':
        pd.date_range('1980-01-01', periods=size, freq='1T'),
        'signal1':
        np.random.normal(0, 0.3, size=size).cumsum() + 50,
        'signal2':
        np.random.normal(0, 0.3, size=size).cumsum() + 50,
        'signal3':
        np.random.normal(0, 0.3, size=size).cumsum() + 50,
        'fake_categorical1':
        np.random.choice(['A', 'B', 'C'], size=size),
        'fake_categorical2':
        np.random.choice(['D', 'E', 'F'], size=size)
    }

    return pd.DataFrame(d)

Generate df and convert to Dask

df = generate_df(15000)
ddf = dd.from_pandas(df, npartitions=5)
ddf.divisions

(0, 3000, 6000, 9000, 12000, 14999)

Using Shape on DataFrame

_shp = ddf.shape
print('Using len: ',len(ddf))
print('Using Dask shape: ',_shp.compute())
print('Using Pandas shape: ',df.shape)

Using len:  15000
Using Dask shape:  (15000, 6)
Using Pandas shape:  (15000, 6)

Get only row lenght
_shp = ddf.shape[0]
print('Dask Row lenght: ',_shp.compute())
print('Pandas Row lenght: ',df.shape[0])

Dask Row lenght:  15000
Pandas Row lenght:  15000

Using Shape on Series

_shp = ddf.signal1.shape
print('Using len: ',len(ddf.signal1))
print('Using Dask shape: ',_shp.compute())
print('Using Pandas shape: ',df.signal1.shape)

Using len:  15000
Using Dask shape:  (15000,)
Using Pandas shape:  (15000,)

@mrocklin Please tell me if I need to change more stuff. Thanks

It looks like this causes other tests to fail.  I recommend clicking on the "details" link to the right of the travis-ci failure at the bottom of this PR (look for a red X).  It looks like other parts of code are checking for the .shape property and calling == on it.

@mrocklin There is some method to check if a variable is delayed or not? I cannot find it
The problem seems to be in dask/utils.py on the is_arraylike function. This function uses shape method to check if the variable is a numpy array or something similar. Then, when x.shape is called it can't convert the result into a boolean because the result is a delayed

I'm still in favor of using dask.compute and a tuple of delayed/int objects.
On Mon, Feb 26, 2018 at 9:13 PM, Stephan Hoyer notifications@github.com
wrote:

@shoyer commented on this pull request.
In dask/utils.py
https://github.com/dask/dask/pull/3212#discussion_r170797560:

@@ -893,6 +893,10 @@ def is_arraylike(x):

is_arraylike('cat')
False
"""

return (hasattr(x, 'array') and

import dask.dataframe as dd
if isinstance(x, (dd.DataFrame, dd.Series)):

   return False

else:

   return (hasattr(x, '__array__') and

The top-level dask.compute() function is probably the best way to
evaluate these.
I suppose the other option is to return a 1D dask array instead of a tuple.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/pull/3212#discussion_r170797560, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszE4XitHEGH1Io-yHdhBh2ooC3jnwks5tY2TYgaJpZM4STFvY
.

Ok, I agree. This will take a bit more time to me to dig and understand the code under the hood. I just started using Dask last week, so I'm newbie on this but I will try anyway

We appreciate your effort on this @henriqueribeiro .  Please let us know if you have further questions and how we can help.

Hey @mrocklin! Finally I found some time to work on this again. On this commits, and following your suggestion, I created a Tuple class for dask. Now it's possible to have a tuple of dask objects and then call compute. In the end it turned out to be a very simple implementation but I'm not 100% sure that I'm doing it correctly.
Also, I saw that some tests are failing but I believe it's not because of the shape method or the tuple class. Can you please have a look on it? Thanks a lot

I think that we should avoid creating a tuple class in Dask.  While I agree that this might be useful sometimes it leads us down a path of recreating an entire language in Dask, which more than I think we want to accomplish.
Instead I recommend that we just return a normal Python tuple that contains a computable object.  Perhaps something like the following?
class Series(...):
    def shape(self):
        return (self.size(),)

class DataFrame(...):
    def shape(self):
        return (self.size(), len(self.columns))

This won't have the nice df.shape.compute() semantics that your current solution has, but has the nice advantage of not increasing the scope of the project.

I understand your point, but the problem with that approach is that then you need to iterate over each element and compute it. Still, and despite the increase of the project's scope, this class is very very simple, only adding compute and persist method to original tuple class.

Still, and despite the increase of the project's scope, this class is very very simple, only adding compute and persist method to original tuple class

But now people can ask for dict, set, list and other similar classes.  I am -1 on adding a tuple class.  I strongly believe that it will create a non-trivial amount of maintenance burden in the future.  You're quite welcome to create a Dask Tuple class, but I would prefer that it not be defined or used within the main codebase.

but the problem with that approach is that then you need to iterate over each element and compute it

Yes, I agree that this is suboptimal.  People would have to use the dask.compute function instead.  I believe that the usability benefit for dataframe.shape here does not balance against the maintenance costs.

You, better than me, have a much better overview of the whole project and so, I trust on your decision. I will change to the implementation that you proposed.
Off topic: there is a list of these small things/methods that should be implemented in order to have, approximately, the same API as Pandas? I would like to continue contributing to the project and maybe these methods are the best methods to get in.

FWIW one can do da.compute(*my_lazy_shape) to get the computed tuple. Maybe not as nice as the method, but not much more complicated.

Do you think is it ok now?

In principle I agree with the changes here.  However, it looks like the tests are failing.  You can see the logs by clicking on "Details" in the travis-ci section above, and then click on any of the four failing builds:

It looks like tests around numpy ufuncs have started to fail?  I wonder if, now that we have a .shape dask dataframes/series look enough like numpy arrays to be treated differently by some other part of the ecosystem?  The next thing I would do would be to try out some of the failing tests locally and investigate why they are failing:
py.test dask/dataframe/tests/test_ufunc.py --pdb --verbose

@mrocklin you are right. The tests are failing because now the dataframe/series have the shape method and, whereas in the past the value returned was -1 for the "shape", now it returns a delayed value.
With this, how do you think is the best way to handle it? To compute the shape and proceed or replace the delayed values and proceed?

Apologies if this has been discussed already, but have we considered returning a tuple of (delayed_n_rows, concrete_n_columns)? We know the number of columns, so it needn't be delayed.
A common use of DataFrame.shape is getting the length of the second axis in a pandas & numpy generic way. If we have the number of columns be delayed, users will still need to check whether they have a dataframe or array.

I feel like this has been discussed somewhere, but I can't find the issue.
Returning a tuple of (delayed, int) makes sense to me
On Mon, Jul 23, 2018 at 7:31 AM, Tom Augspurger notifications@github.com
wrote:

Apologies if this has been discussed already, but have we considered
returning a tuple of (delayed_n_rows, concrete_n_columns)? We know the
number of columns, so it needn't be delayed.
A common use of DataFrame.shape is getting the length of the second axis
in a pandas & numpy generic way. If we have the number of columns be
delayed, users will still need to check whether they have a dataframe or
array.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/pull/3212#issuecomment-407026452, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszLgeoRo2nxyA2dhtPZd7pbuf-oNFks5uJbP5gaJpZM4STFvY
.

Thanks. @henriqueribeiro do you have time to implement this soon? I have a short-term need for it.
I think something roughly like
@property
def shape(self):
    return (self.reduction(len, np.sum, token='len', meta=int, split_every=False),
            len(self.columns))

docs and tests should do the trick.

Hey @TomAugspurger and @mrocklin
Yes, we had that discusssion some time ago but in the end, and in the current implmentation, the shape returns a tuple with a delayed object and a scalar, exactly as you suggested.
The only problem that I see is that some methods use the absence of shape method to check if it's dask or not (at least is my understanding). How can we overcome this?

@TomAugspurger exactly. Tell me if you need something from me

We should get some input from @mrocklin

Yeah, CI is passing now, but I'm not happy with the _unknown_shape hack.
I'm still thinking through how to avoid that.
On Fri, Jul 27, 2018 at 7:36 AM henriqueribeiro notifications@github.com
wrote:

We should get some input from @mrocklin https://github.com/mrocklin
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/pull/3212#issuecomment-408406257, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABQHIjbA24HMXEvc9z0hmRDj1_x6v2XAks5uKwllgaJpZM4STFvY
.

Just pushed a commit removing the _unknow_shape hack, at the cost of having to check is_dask_collection(x)on every element ofarr.shape` in a few places. This is a bit more expensive, but I'm personally OK with that.
Do you have a preference @mrocklin or @shoyer?

This looks reasonable to me.

Thanks. Merging tomorrow if there are no objections.

Just pushed a commit removing the _unknow_shape hack, at the cost of having to check is_dask_collection(x) on every element of arr.shape in a few places.

Didn't actually know about that hack. Anyways sounds like an improvement to me.

authors: TomAugspurger henriqueribeiro jakirkham mrocklin shoyer


text:
[WIP] [Parquet] Handle new (dtype) pandas metadata with pyarrow

Closes #6085
Uses ignore_metadata=False during arrow -> pandas conversion in ArrowEngine parquet reader.  Before these changes, ignore_metadata=True was used to simplify the setting of the index.  However, the pandas metadata is needed to get certain dtype conversions correct.

[ ] Tests added / passed
[ ] Passes black dask / flake8 dask

@martindurant @msebbah

Looks like the test failures here are unrelated to the changes (recently-merged #6091 also had the same failures)

Thanks for the review @TomAugspurger! Any other thoughts here @martindurant ?

Thanks @rjzamora!

authors: TomAugspurger rjzamora


text:
Import graphviz.Diagraph from top of module, not from graphviz.dot

[x] Closes #8362
[x] Tests added / passed
[x] Passes pre-commit run --all-files

I'll merge this when it passes CI

There are still some issues with windows 3.7 but I'm going to merge this to unblock CI
_______________________________ test_visualize ________________________________
[gw3] win32 -- Python 3.7.12 C:\Miniconda3\envs\test-environment\python.exe

tmpdir = local('C:\\Users\\runneradmin\\AppData\\Local\\Temp\\pytest-of-runneradmin\\pytest-0\\popen-gw3\\test_visualize0')

    def test_visualize(tmpdir):
        pytest.importorskip("graphviz")
        da = pytest.importorskip("dask.array")
        fn = str(tmpdir)
        a = da.ones(10, chunks=(5,))
        b = a + 1
        c = a + 2
        d = b + c
>       d.dask.visualize(fn)

dask\tests\test_highgraph.py:21:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
dask\highlevelgraph.py:897: in visualize
    graphviz_to_file(g, filename, format)
dask\dot.py:288: in graphviz_to_file
    data = g.pipe(format=format)
C:\Miniconda3\envs\test-environment\lib\site-packages\graphviz\piping.py:116: in pipe
    return self._pipe_lines(*args, input_encoding=self._encoding, **kwargs)
C:\Miniconda3\envs\test-environment\lib\site-packages\graphviz\backend\piping.py:149: in pipe_lines
    proc = execute.run_check(cmd, capture_output=True, quiet=quiet, **kwargs)
C:\Miniconda3\envs\test-environment\lib\site-packages\graphviz\backend\execute.py:80: in run_check
    popen = subprocess.Popen(cmd, stdin=subprocess.PIPE, **kwargs)
C:\Miniconda3\envs\test-environment\lib\subprocess.py:800: in __init__
    restore_signals, start_new_session)
C:\Miniconda3\envs\test-environment\lib\subprocess.py:1148: in _execute_child
    args = list2cmdline(args)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

seq = [WindowsPath('dot'), '-Kdot', '-Tpng']

    def list2cmdline(seq):
        """
        Translate a sequence of arguments into a command line
        string, using the same rules as the MS C runtime:

        1) Arguments are delimited by white space, which is either a
           space or a tab.

        2) A string surrounded by double quotation marks is
           interpreted as a single argument, regardless of white space
           contained within.  A quoted string can be embedded in an
           argument.

        3) A double quotation mark preceded by a backslash is
           interpreted as a literal double quotation mark.

        4) Backslashes are interpreted literally, unless they
           immediately precede a double quotation mark.

        5) If backslashes immediately precede a double quotation mark,
           every pair of backslashes is interpreted as a literal
           backslash.  If the number of backslashes is odd, the last
           backslash escapes the next double quotation mark as
           described in rule 3.
        """

        # See
        # http://msdn.microsoft.com/en-us/library/17w5ykft.aspx
        # or search http://msdn.microsoft.com for
        # "Parsing C++ Command-Line Arguments"
        result = []
        needquote = False
        for arg in seq:
            bs_buf = []

            # Add a space to separate this argument from the others
            if result:
                result.append(' ')

>           needquote = (" " in arg) or ("\t" in arg) or not arg
E           TypeError: argument of type 'WindowsPath' is not iterable

C:\Miniconda3\envs\test-environment\lib\subprocess.py:555: TypeError

---------- coverage: platform win32, python 3.7.12-final-0 -----------


text:
Allow split_out to be None, which then defaults to 1 in groupby().aggregate()

[x] Closes #9490
[ ] Tests added / passed
[x] Passes pre-commit run --all-files

Also, should we deprecate allowing None here?

Thanks @ian-r-rose! Would you mind adding a small test (e.g. similar to the example in #9490)?

I was a bit uncomfortable adding a test since I wasn't sure if we actually wanted to support None here. I'd be curious to hear from @wence- if there are any things that would become hugely onerous if we dropped that support.

Agreed, it'd be good to get thoughts from @wence- on his particular use case. That said, do we advertise anywhere that split_out can be None? From a brief skim of the API docs, it looks like we're consistent in saying it's an int

I only see None documented in one place (sort of), which is the docstring for groupby.aggregate: https://docs.dask.org/en/latest/generated/dask.dataframe.groupby.DataFrameGroupBy.aggregate.html#dask.dataframe.groupby.DataFrameGroupBy.aggregate
Even then, it's saying it's an optional arg, which is a bit ambiguous.

The implementation for apply_concat_apply clearly anticipates a None value, but I might consider that an internal API.

The implementation for apply_concat_apply clearly anticipates a None value, but I might consider that an internal API.

That also has a bug :( (sort=True, split_out=None => TypeError)

Thanks @ian-r-rose! Would you mind adding a small test (e.g. similar to the example in #9490)?

I was a bit uncomfortable adding a test since I wasn't sure if we actually wanted to support None here. I'd be curious to hear from @wence- if there are any things that would become hugely onerous if we dropped that support.

I don't think it would be problematic. I noticed because in dask_cudf we explicitly test groupby().agg with split_out=None. But that is in tests so I can remove it.
The maintenance issue here is somehow that there are lots of public APIs that call into the few "semi-private" APIs. So there are lots of places for the defaults to go out of whack.
One way to partially solve this is to not provide default values for arguments to internal APIs, and promote the default handling to the public callers. I generally think that internal APIs should not do argument defaulting, since all that business logic handling is not really germane to the usage of the function.
It does make calling these functions slightly more verbose (especially given that they all have many arguments), though one can utilise my favourite python argument list approach of required keyword-only args:
def aca(*, split_out, split_every, ...)

though one can utilise my favourite python argument list approach of required keyword-only args:

If I ever have a week of nothing to do, I'd love to make 90% of the APIs here make use of keyword-only args.

Also, should we deprecate allowing None here?

I am inclined to say yes. I will change things on the dask_cudf side.

My concern with dropping support for split_out=None is that we may find a reasonable way to automatically infer a good value for split_out in the future.  For example, perhaps the Groupby class itself should optionally infer group cardinality up-front.  In that case, we would probably not want to override split_out if it was already set to a specific integer value.
This hypothetical concern may not be realistic.

My concern with dropping support for split_out=None is that we may find a reasonable way to automatically infer a good value for split_out in the future. For example, perhaps the Groupby class itself should optionally infer group cardinality up-front. In that case, we would probably not want to override split_out if it was already set to a specific integer value.
This hypothetical concern may not be realistic.

I think it's a realistic hypothetical, but I guess that in that case I would advocate for expanding the accepted types to include None, or a sentinel value like the pandas NoDefault when it actually comes up. That change shouldn't be API-breaking for, except to the extent that we might want to make inference default.

authors: ian-r-rose jrbourbeau rjzamora wence-


text:
413 Payload Too Large when recursive remove a large directory

When calling rm(path, recursive=True), the request fails without an actual error response:
_call non-retriable exception:
Traceback (most recent call last):
  File "/home/paul/anaconda3/envs/antenna/lib/python3.7/site-packages/gcsfs/core.py", line 487, in _call
    validate_response(r, path)
  File "/home/paul/anaconda3/envs/antenna/lib/python3.7/site-packages/gcsfs/core.py", line 132, in validate_response
    raise HttpError({"code": r.status_code})
gcsfs.utils.HttpError

When digging deeper:
<Response [413]>
This happens basically on large directories.

rm includes a batchsize= parameter, default 20. Can you find a smaller value that works?
Perhaps the code logic is wrong, and you can find out by debugging a little how many files to delete were actually included in the call that failed.
I would appreciate if, while digging, you also helped improve the code to dig out the actual error from the general HttpError case. Note that I have deleted large numbers of files in a single call before.

I just checked, and the batchsize parameter is only available in the latest version. Unfortunately the latest version is unusable due to https://github.com/dask/gcsfs/issues/279
Closing this issue though as it has been resolved in the latest version (I think however that 20 is quiet low, this code has ran before with a large amount of files, but I assume it has crossed the threshold now).

Let's get that issue fixed, then. Did you try #280?

I think however that 20 is quite low

Since we can now send many requests concurrently, there is less of a need to pack them into large batches. Indeed, this very issue shows the problem. I don't know what an optimal batch-size would be...

I didn't know the new version automatically did many calls at once, that's a great improvement (as well as asyncio support!). I will try out the PR, didn't see that last week.

authors: martindurant pvanderlinden

repo: dask/gcsfs


text:
Describe what extensions are and how to use them

Note that an alternative idea discussed earlier was a separate function get_extension(), but IIRC that was not considered a good idea. Now that we have module-level __getattr__'s, it should not be a problem for any library to use a specified name like linalg or fft.
Whether the functions in the linalg extensions should all be required to exist or not was discussed in gh-403 (no clear conclusion there). One option discussed there to deal with hard to implement or more niche APIs is to create a separate status/label or some other way to track desired signatures (details to be worked out if we want to go that way).
Quoting @shoyer: My two cents is that it would be valuable if the standard specified the interface for these functions, even if not every library is going to implement them. It's really not a big deal to need to look up a compatibility table for advanced linear algebra functionality.
There is something to say for that as well (once we have a single easy to use compatibility table in a central place - see gh-402 and gh-462); I see both pros and cons. The pro is easy:

having a signature & semantics defined for a larger set of functions is, over time, going to lead to fewer differences between libraries.

Concerns I have are:

Scope management: until now it was relatively easy to draw a line about what included in the standard yes or no; this would make the scope significant wider (and that also implies more work I suspect),
Spending a lot of time on existing mismatches for fairly niche functionality, e.g. see this comment for the matrix exponential function(s).
We also do not do this for other functionality (e.g., most libraries have a median function, but we left it out on purpose because it's not great for distributed libraries; the median interface is not available anywhere in this repo).

There were no more comments on this. What is here should be pretty straightforward. It also closes issue gh-501, which was opened after this PR.
A separate list of signatures for fft/linalg functions that are not included in the standard can always be worked on separately.
Let's get this in.

repo: data-apis/array-api


text:
Added grid to stacked bar chart

@SejoB great work!

authors: SejoB atalyaalon


text:
[ISSUE] workspaceClient.Users.ListAll hangs if no StartIndex & TotalResults are returned

Observed in 0.26.1 that if I'm using the following fixture in test:
	userListIdUsernameFixture = qa.HTTPFixture{
		Method:   "GET",
		Resource: "/api/2.0/preview/scim/v2/Users?attributes=id%2CuserName&count=100&startIndex=1",
		Response: iam.ListUsersResponse{
			Resources: []iam.User{
				{
					Id:       "id",
					UserName: "user@domain.com",
				},
			},
		},
		ReuseRequest: true,
	}

then workspaceClient.Users.ListAll hangs
It was found in the context of https://github.com/databricks/terraform-provider-databricks/pull/2980

FYI: in the TF repo, I added https://github.com/databricks/terraform-provider-databricks/blob/master/qa/scim_fixtures.go to help set up fixtures more easily for TF.
Separately, in https://github.com/databricks/databricks-sdk-go/commit/b3cdc973d681e848158d2712a0dd18f3933f2e6c, we added support for higher-level mocking of the WorkspaceClient. Now you can just mock the SDK calls rather than the underlying requests on the network, which should make testing with the WorkspaceClient easier.

authors: alexott mgyucht

repo: databricks/databricks-sdk-go


text:
databricks.sql.connect hangs in a long retrying loop when an invalid access token is used

Steps to reproduce
Supply any invalid access token to databricks.sql.connect, e.g.
sql.connect(
     server_hostname=my_workspace_url,
     http_path=http_path_to_my_warehouse,
     access_token="this-is-not-a-valid-access-token"
)

Current behavior
The call gets stuck in a long retrying loop, eventually failing with a RequestError, which does not indicate an authentication error.
Expected behavior
The function immediately raises an exception indicating an authentication error.

Related
https://github.com/databricks/databricks-sql-python/issues/346#issuecomment-1967984742

The fix for this has merged.

authors: mbelak-dtml susodapop

repo: databricks/databricks-sql-python


text:
_LOAD_BALANCING_LOSS returns empty list sometimes

Hello, I am using Eleuther AI's gpt-neox implementation with megablocks, but I get 2 errors related to the _LOAD_BALANCING_LOSS.

the tokens_per_expert gives me this error at this line.  ValueError: Expected 14 token_per_experts but found 7.  Here's the stack trace.

  File "/home/etnguyen/test/savanna/train.py", line 10, in <module>                                                                                [53/1963]
    pretrain(global_config=global_config)
  File "/home/etnguyen/test/savanna/savanna/training.py", line 228, in pretrain
    iteration = train(
                ^^^^^^
  File "/home/etnguyen/test/savanna/savanna/training.py", line 1004, in train
    loss_dict, skipped_iter = train_step(
                              ^^^^^^^^^^^
  File "/home/etnguyen/test/savanna/savanna/training.py", line 919, in train_step
    loss = forward_step(
           ^^^^^^^^^^^^^
  File "/home/etnguyen/test/savanna/savanna/training.py", line 515, in forward_step
    moe_loss = mb_moe_loss_func(global_config, loss_mask, outputs)[0]
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/etnguyen/test/savanna/savanna/training.py", line 464, in mb_moe_loss_func
    lbl = moe.batched_load_balancing_loss(megablocks_args)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/etnguyen/.local/lib/python3.11/site-packages/megablocks/layers/moe.py", line 43, in batched_load_balancing_loss
    raise ValueError(
ValueError: Expected 14 token_per_experts but found 7.
num_layers = 14
pipeline_model_parallel_size = 1
num_layers_per_virtual_pipeline_stage = None

I get this error when the expert_interval=2, ie when the default value is used, and so the number of experts is actually half the number of layers (14 layers, 7 Megablocks layers used). This error gets fixed when I set the expert_interval=1 so that there are 14 Megablocks, and 14 layers.  But I don't know the root cause of this discrepancy, especially if I want to change the expert_interval and number of Megablocks to every other layer.

The second issue is, let's say I do use expert_interval=1 to get around the issue above, so every layer uses a Megablock, then my next error I get is that the return value of get_load_balancing_loss occasionally returns an empty list, which then errors out, meaning the _LOAD_BALANCING_LOSS is an empty list.  Critically, this happens part way through training, like 30 secs in, so some batches it's fine and returns the expected losses.

Does this sound familiar to anybody?  I'd very much appreciate any insights, thank you!

Can you double check num_layers is appropriately passed in Arguments to dmoe/moe when expert_interval=2? It should be equal to 7.
Hm... I'm a little less sure since I'm not as familiar with Eleuther's harness. You can check out an example with LLMFoundry here if helpful. I'm not sure what's going on here... it should save every time you do a forward pass 🤔. Could you ensure it is in train mode? If the model is in eval mode, it won't save LBL loss, and then if you try to backprop it will error

repo: databricks/megablocks


text:
Is this compatible with Spark >2.0?

Hi, does this work with newer versions of Spark?

Hi Ramon, it's possible that some minor changes will be needed for compatibility with newer spark versions.

Up to what version has it been tested? How do you install it?
Thanks!

Also, have you tested by any chance against standalone state-of-the-art LP solvers like CPLEX?

Hi Ramon, the sbt build file specifies spark 1.4.1. Generally you can include a git repo in your project using sbt, but if you are making modifications you may want to clone and then include your local version using sbt. Contributing back any changes you make is of course encouraged!
We have not compared with other solvers. It may be helpful to read up on the capabilities of TFOCS, of which TFOCS for Spark is a partial clone (http://cvxr.com/tfocs/).

Would be good to include some installation instructions. =)

authors: convexset huevosabio staple

repo: databricks/spark-tfocs


text:
xinference部署的embedding模型请求报错！404

现象：知识库导入文件解析失败！

查看xinference日志，post请求失败！！

repo: dataelement/bisheng


text:
Correction for raw affiliation Edysan (UMR CNRS 7058) Université de Picardie Jules Verne  Amiens Cedex France

Correction needed for raw affiliation Edysan (UMR CNRS 7058) Université de Picardie Jules Verne  Amiens Cedex France
raw_affiliation_name: Edysan (UMR CNRS 7058) Université de Picardie Jules Verne  Amiens Cedex France
new_rors: 01gyxrk03;02feahw73;02bhp7a56
previous_rors: 01gyxrk03;02feahw73
works_examples: W4382283503
contact: 50f4dccf79ad35ecd5ea1499875247f4:17d8b6f32fab064b80dee5650fb99f87 @ u-picardie.fr

This issue was accepted and ingested by the OpenAlex team on 2024-10-10. The new affiliations should be visible within the next 7 days.

repo: dataesr/openalex-affiliations


text:
Correction for raw affiliation UMR IATE, Université Montpellier 2, Place Eugène Bataillon, 34095 Montpellier Cedex, France

Correction needed for raw affiliation UMR IATE, Université Montpellier 2, Place Eugène Bataillon, 34095 Montpellier Cedex, France
raw_affiliation_name: UMR IATE, Université Montpellier 2, Place Eugène Bataillon, 34095 Montpellier Cedex, France
new_rors: 051escj72;0000n5x09
previous_rors: 051escj72
works_examples: W2322484979
contact: 8e3ff03b9b67d347d918706fd7fc013e:7ffab562e9c2dab4bee468d9e1d5540553e6 @ inrae.fr

This issue was accepted and ingested by the OpenAlex team on 2024-10-09. The new affiliations should be visible within the next 7 days.


text:
Correction for raw affiliation SIRIS Academic, Barcelona, Spain

Correction needed for raw affiliation SIRIS Academic, Barcelona, Spain
raw_affiliation_name: SIRIS Academic, Barcelona, Spain
new_rors: 02vzx8197
previous_rors:
works_examples: W2888737347
contact: e1fb71d75ecb8a40c05e8cf07b7d9f48:a6126f95 @ ourresearch.org

This issue was accepted and ingested by the OpenAlex team on 2024-11-01. The new affiliations should be visible within the next 7 days.


text:
Missing "live" version of Business Data Google My Business Info

This client is missing the Google My Business Info Live types and endpoints.
It has endpoints for Task POST, Task GET, and Tasks Ready - but not the Live version.
Therefore we cannot do something like this, as per the docs of this client in the README:
import * as client from "dataforseo-client";

const businessDataApi = new client.BusinessDataApi(
  "https://api.dataforseo.com",
  {
    fetch: authFetch,
  },
);

let task = new client.BusinessDataGoogleMyBusinessInfoLiveTaskInfo(); // < --- THIS IS NOT FOUND
task.location_name = "New York,New York,United States";
task.language_code = "en";
task.keyword = "RustyBrick, Inc.";

const resp = await businessDataApi.googleMyBusinessInfoLive(task);  // < --- THIS IS NOT FOUND

thanks for noticing the issue. that endpoint added in new latest version (1.0.19)
code example:
`
const authFetch = createAuthenticatedFetch(username, password);
let serpApi = new client.BusinessDataApi("https://api.dataforseo.com", { fetch: authFetch });
let task = new client.BusinessDataTaskRequestInfo();
task.location_name = "New York,New York,United States";
task.language_code = "en";
task.keyword = "RustyBrick, Inc.";

let resp = await serpApi.googleMyBusinessInfoLive([task]);

`

authors: BenJackGill lVics

repo: dataforseo/TypeScriptClient


text:
Climatisation

Je suppose qu'une proportion non négligeable des foyers français est en train de s'équiper de clim. Question soulevée dans #158

Je ferme, car c'est forcément compté dans la facture d'électricité. Sauf les fuites de gaz très réchauffants, mais c'est plutôt dans l'empreinte de construction #854

repo: datagir/nosgestesclimat


text:
Inform the user that he has been logged out

Depends on https://github.com/datagovsg/beeline-server/issues/201

https://github.com/datagovsg/beeline-server-archive/issues/201
https://github.com/datagovsg/beeline-server-archive/pull/374

Is this still alive?

The PR was reverted in https://github.com/datagovsg/beeline-server-archive/pull/375 so we are not ready to do this yet

authors: chuasweechin joelchoo xkjyeah

repo: datagovsg/beeline-frontend


text:
API Documentation (GET&POST) + CURL instructions on showcase page

Original issue in datahub-qa
As a programmer I want to have proper docs re using datahub API so I could use it in my project
tasks
Write an article[s] in hackmd

[x] document Web API for getting data and datasets

[x] /r/ links ...
[x] datapackage.json

[x] Web API for creating datasets and pushing data
[x] curl and data get instructions on the showcase page

Get a review

[ ] review from @zelima

Update the docs:

[x] docs section on datahub.io
[x] getting data section on the dataset's showcase page

Acceptance criteria
Review from Irakly or Anu

Adding more fixes: https://github.com/datahq/datahub-qa/issues/157#issuecomment-373139268

FIXED

authors: AcckiyGerman anuveyatsu

repo: datahq/frontend


text:
Enable AWS ES connection

We need to be able to connect to AWS elastic search as well.
Acceptance Criteria

[ ] Able to connect to AWS elastic search if provided appropriate credentials

Task

[ ] refactor code to use requests_aws4auth

WONTFIX as irrelevant

repo: datahq/metastore


text:
fix(ingest/bigquery): Support cross project usage using FileBackedDict

May be easier to review by commit. Adding some more tests, but ready for review.
A lot of the log warnings should be report warnings... not sure if I should make the change here or in a separate PR where we go over all log / report warnings.
Comparing performance of in-memory implementation vs this one, with 10_000 tables, 7 days, 100_000 selects, 20_000 operations, 10 users, 10_000 average query length:
FILE BACKED
-------------------------------- live log call ---------------------------------
WARNING  datahub.ingestion.source.state.stateful_ingestion_base:stateful_ingestion_base.py:151 Stateful ingestion is disabled, disabling store_last_usage_extraction_timestamp config option as well
INFO     datahub.ingestion.source.bigquery_v2.bigquery_report:bigquery_report.py:85 Time spent in stage <All: Seed Data Generation at 2023-04-06 12:46:41.307718>: 0.23 seconds
Events generated: 420223
Data structures size: 1.63 GB
INFO     datahub.ingestion.source.bigquery_v2.bigquery_report:bigquery_report.py:85 Time spent in stage <All: Event Generation at 2023-04-06 12:46:41.536087>: 39.68 seconds
INFO     datahub.ingestion.source.bigquery_v2.usage:usage.py:377 Total number of events aggregated = 420223.
INFO     datahub.ingestion.source.bigquery_v2.bigquery_report:bigquery_report.py:85 Time spent in stage <All: Event Ingestion at 2023-04-06 12:47:21.216895>: 180.94 seconds
INFO     datahub.ingestion.source.bigquery_v2.bigquery_report:bigquery_report.py:85 Time spent in stage <All: Usage Extraction Operational Stats at 2023-04-06 12:50:22.216642>: 76.62 seconds
INFO     datahub.ingestion.source.bigquery_v2.bigquery_report:bigquery_report.py:85 Time spent in stage <All: Usage Extraction Usage Aggregation at 2023-04-06 12:51:38.835335>: 198.30 seconds
Workunits Generated: 89711
Seconds Elapsed: 455.86 seconds
Memory Used: 335.65 MB
PASSED

IN MEMORY
-------------------------------- live log call ---------------------------------
WARNING  datahub.ingestion.source.state.stateful_ingestion_base:stateful_ingestion_base.py:151 Stateful ingestion is disabled, disabling store_last_usage_extraction_timestamp config option as well
INFO     datahub.ingestion.source.bigquery_v2.bigquery_report:bigquery_report.py:87 Time spent in stage <All: Seed Data Generation at 2023-04-06 12:57:58.831374>: 0.22 seconds
Events generated: 421554
Data structures size: 1.57 GB
INFO     datahub.ingestion.source.bigquery_v2.bigquery_report:bigquery_report.py:87 Time spent in stage <All: Event Generation at 2023-04-06 12:57:59.052493>: 33.88 seconds
INFO     datahub.ingestion.source.bigquery_v2.usage:usage.py:732 Number of read events joined with query events: 301554
INFO     datahub.ingestion.source.bigquery_v2.usage:usage.py:220 Total number of events aggregated = 301554.
INFO     datahub.ingestion.source.bigquery_v2.bigquery_report:bigquery_report.py:87 Time spent in stage <All: Event Ingestion at 2023-04-06 12:58:32.932223>: 57.13 seconds
Workunits Generated: 86974
Seconds Elapsed: 57.13 seconds
Memory Used: 9.62 GB
PASSED

Checklist

[ ] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
[ ] Links to related issues (if applicable)
[ ] Tests for the changes have been added/updated (if applicable)
[ ] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
[ ] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

Codecov Report
Patch coverage: 59.03% and project coverage change: -11.01 :warning:

Comparison is base (c7d35ff) 74.90% compared to head (ae37304) 63.89%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
Additional details and impacted files
@@             Coverage Diff             @@
##           master    #7663       +/-   ##
===========================================
- Coverage   74.90%   63.89%   -11.01%
===========================================
  Files         353      353
  Lines       35395    35444       +49
===========================================
- Hits        26511    22646     -3865
- Misses       8884    12798     +3914

Flag
Coverage Δ

pytest-testIntegration
?

pytest-testIntegrationBatch1
?

pytest-testQuick
63.89% <59.03%> (+0.30%)
:arrow_up:

pytest-testSlowIntegration
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

...c/datahub/ingestion/source/bigquery_v2/bigquery.py
34.72% <0.00%> (+0.43%)
:arrow_up:

...ub/ingestion/source/bigquery_v2/bigquery_schema.py
66.44% <ø> (ø)

...rc/datahub/ingestion/source/bigquery_v2/lineage.py
26.22% <0.00%> (+0.62%)
:arrow_up:

...n/src/datahub/utilities/file_backed_collections.py
100.00% <ø> (ø)

.../src/datahub/ingestion/source/bigquery_v2/usage.py
52.03% <55.93%> (+29.77%)
:arrow_up:

...hub/ingestion/source/bigquery_v2/bigquery_audit.py
54.88% <81.81%> (+0.83%)
:arrow_up:

...ub/ingestion/source/bigquery_v2/bigquery_config.py
94.44% <100.00%> (+0.06%)
:arrow_up:

...ub/ingestion/source/bigquery_v2/bigquery_report.py
98.48% <100.00%> (+3.24%)
:arrow_up:

...src/datahub/ingestion/source/usage/usage_common.py
100.00% <100.00%> (ø)

... and 98 files with indirect coverage changes
Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

@asikowitz looks like there are conflicts, can you please resolve them ?

repo: datahub-project/datahub


text:
refactor(all): Prevent merge conflicts in the future

Can add more in the future
Checklist

[ ] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
[ ] Links to related issues (if applicable)
[ ] Tests for the changes have been added/updated (if applicable)
[ ] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
[ ] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

Closing in favor of split up PRs


text:
Add to  bootcamp 2023 s1

unable to label, join-bootcamp-2023s1 @sparsh-ai

From the right side of the screen, it can be added. I just added it for you.

invited you to this repo, once we have that, you will be able to see project board. I will also be sharing the project board link, just in case.

repo: datalaker/de-labs


text:
Matching financial transactions?

Hi, Just come across your repo here and am wondering if matching financial transactions would be a good fit it..?
I'm look for ways to parse finance transactions for matches where - for example - an invoice exists and there are many payments against this invoice but no key field to tie them together. They could have matching texts, vendor name or something but its largely unknown.
I'm thinking I could use dedupe to find matches, then go through each matched group of transactions and sum the dollars - if its zero we can clear those transactions... and so on...
So just wondering if you think dedupe could be used in this scenario?

Give it a shot.

repo: datamade/dedupe


text:
🛑 DataMeet is down

In 22dd99d, DataMeet (http://datameet.org/) was down:

HTTP code: 0
Response time: 0 ms

Resolved: DataMeet is back up in 53641fa.

repo: datameet/upptime


text:
🛑 Passport User Registration Page is down

In f3beb65, Passport User Registration Page (https://portal2.passportindia.gov.in/AppOnlineProject/user/RegistrationBaseAction?request_locale=en) was down:

HTTP code: 400
Response time: 1617 ms

Resolved: Passport User Registration Page is back up in c49f457.


text:
DDL queries triggering "invalid statement type" error

DDL queries are not idempotent so it's an easy case that should have
been handled explicitly. It was not incorrect before, but it caused
unnecessary logging.

Tested manually by deploying Temporal on Astra DB:  https://github.com/datastax/cql-proxy/issues/84#issuecomment-1083119080

parser/lexer.go is generated code, don't be fooled by the diff size!

repo: datastax/cql-proxy


text:
[HW] Praveen Ogirala

Name: Venkata Naga Praveen Ogirala
Email: nagapraveen.ogirala@gmail.com
Linkedin Profile: https://www.linkedin.com/in/praveenogirala/
Attach the homework step I screenshots below:

Attach the homework step II screenshots below:

Looks good Venkata, thank you. Here's your badge -> https://api.badgr.io/public/assertions/mcPrMn2KThWbXiK9-6JFLA?identity__email=nagapraveen.ogirala%40gmail.com

authors: SonicDMG praveenogirala

repo: datastaxdevs/workshop-astra-tik-tok


text:
[#3072] fix(spark-connector):  fix local run SparkIcebergCatalogIT failed for non-utc timezone

What changes were proposed in this pull request?
configurate spark time zone to UTC
Why are the changes needed?
if not specifying timezone, Iceberg partition path varies for different time zone in local machine.
Fix: #3072
Does this PR introduce any user-facing change?
no
How was this patch tested?
test in local machine

@FANNG1 can you please change the PR title to be more specific to this fix?

@jerryshao @caican00 please help to review agian, thanks

@FANNG1 please address the comment.

LGTM

repo: datastrato/gravitino


text:
refactor: generalize some code of vis library

Generalises the vis library

@dvzacharycutler I tested this PR in Ordino Enterprise and get the following compile error:
webpack compiled with 2 warnings
ERROR in ./src/store/workbenchReducer.ts:79:51
TS2339: Property 'numColumnsSelected' does not exist on type 'Partial<IVisConfig>'.
  Property 'numColumnsSelected' does not exist on type 'Partial<ISankeyConfig>'.
    77 |
    78 |     if (state.workbenches[workbenchIndex].visConfig) {
  > 79 |       state.workbenches[workbenchIndex].visConfig.numColumnsSelected = [];
       |                                                   ^^^^^^^^^^^^^^^^^^
    80 |       if (state.workbenches[workbenchIndex].visConfig.type === ESupportedPlotlyVis.BAR) {
    81 |         (<IBarConfig>state.workbenches[workbenchIndex].visConfig).catColumnSelected = null;
    82 |       } else if (state.workbenches[workbenchIndex].visConfig.type === ESupportedPlotlyVis.HEXBIN) {

Found 1 error in 16917 ms.

I used the package.json resolutions to install this branch:
  "resolutions": {
    "@types/react": "~18.2.0",
    "@types/react-dom": "~18.2.0",
    "react": "~18.2.0",
    "react-dom": "~18.2.0",
    "visyn_core": "git+ssh://git@github.com/datavisyn/visyn_core.git#mh/sankey"
  },

Could you please have a look what needs to be changed in Ordino Enterprise and raise another PR there if necessary.

Yes I will take a look at it

I cannot approve this PR but I reviewed the changes made by zach and they are ok

Should be closed after https://github.com/datavisyn/visyn_core/pull/76 is merged

Closed in favor of https://github.com/datavisyn/visyn_core/pull/76.

authors: dvmoritzschoefl thinkh


text:
CVE-2021-14343 (PyYAML)

Vulnerability CVE-2021-14343 shows up against datawire/ambassador:1.12.3 (was also present against 1.12.2) when scanning image with Anchore scanner version 0.9.3.
This vulnerability is fixed in version 5.4 of PyYAML, however because PyYAML is a distutils installed project it is not able to be fixed by upgrading PyYAML using pip3.
Suggested fix: a version of datawire/ambassador with PyYAML v5.4 installed instead of PyYAML-5.3.1.

Hey Erik, thanks for the report!
Since the beginning, Ambassador has always used the PyYAML SafeLoader and only the SafeLoader, because of exactly this sort of thing. CVE-2021-14343 is only present in the FullLoader class, which we don't use, so Ambassador isn't vulnerable to it. (We actually dug into the source on this one to make sure that we couldn't be somehow bitten through the SafeLoader class; all looks safe.)
We're going to go ahead and get to PyYAML 5.4 anyway (tracked in https://github.com/datawire/ambassador/issues/3349), but the CVE itself shouldn't be a problem. Thanks!

klfynn,
As always thank you so much for the quick reply and great information.
Thanks,
Erik

authors: dfl-erik kflynn

repo: datawire/ambassador


text:
Allow configuring idle_timeout Envoy parameter on HTTP Connection Manager

Description
We currently have Ambassador deployed on-premise behind a firewall and fronted by a CDN. We encountered an issue where on low-traffic services, our clients were receiving some 503 first byte timeouts.
The issue is present because when the firewall chose to remove idle tcp sessions from its session table, it would not send a FIN or RST to the CDN. On the next request, the CDN thinks that it still has a tcp connection open and attempts to re-use the now-closed TCP connection, resulting in the request getting 'black-holed', and leads to an HTTP 503.
Because Envoy doesn't ever attempt to close idle tcp connections by default, we're left with Envoy thinking that sessions are open, but our firewall has closed them after a 30 minute timeout (without notifying the CDN that the connection is closed), and the CDN still thinks that the connection is open.
Our "options" for remediating this scenario were:

Configure tcp idle timeout on the CDN side to control connection idle time

This turned out to not be an option that we can readily control on our CDN. They have 'dynamic' connection times.

Make the firewall send a FIN or RST to the CDN when it removes a session from its session table

We investigated this as well, but on our firewall this is not an option that is configurable.

Have Envoy close idle tcp connections gracefully after a configurable timeout (in our case, some timer that is lower than our firewall idle session timeout)

We believe that this is the most appropriate way to solve the issue, and gives us control when tuning idle timeouts across each network device in the path, so this PR addresses making the idle_timeout Envoy setting configurable on the Envoy http listener.

Related Issues
#2155
#1738
#2126
Testing

Manual testing via netcat with ambassador running locally in docker-for-desktop Kubernetes cluster.
Will run in our staging environment once we upgrade our stack to Ambassador v1.0

Manual testing:
Default configuration - observe that the TCP connection can remain open for (at least) 2 mins and 30 seconds. I sent a request, waited a few mins, then was able to send an additional request over the same connection.
$ time nc localhost 80
GET / HTTP/1.1
Host: localhost
HTTP/1.1 200 OK
content-length: 2002
content-disposition: inline; filename="index.html"
accept-ranges: bytes
etag: "334cde9c1e9a17acbc4b7fc05810191319a69eec"
content-type: text/html; charset=utf-8
vary: Accept-Encoding
date: Thu, 09 Jan 2020 20:15:36 GMT
x-envoy-upstream-service-time: 2
server: envoy
<!doctype html>....
GET / HTTP/1.1
Host: localhost
HTTP/1.1 200 OK
content-length: 2002
content-disposition: inline; filename="index.html"
accept-ranges: bytes
etag: "334cde9c1e9a17acbc4b7fc05810191319a69eec"
content-type: text/html; charset=utf-8
vary: Accept-Encoding
date: Thu, 09 Jan 2020 20:18:03 GMT
x-envoy-upstream-service-time: 4
server: envoy
<!doctype html>....

nc localhost 80  0.00s user 0.01s system 0% cpu 2:31.62 total

Now with idle_timeout: 30s setting configured - observe that the TCP connection is closed by envoy after 30 seconds of being idle. I send one request, then it gets gracefully closed ~30s later because there is no further activity.
$ time nc localhost 80
GET / HTTP/1.1
Host: localhost
HTTP/1.1 200 OK
content-length: 2002
content-disposition: inline; filename="index.html"
accept-ranges: bytes
etag: "334cde9c1e9a17acbc4b7fc05810191319a69eec"
content-type: text/html; charset=utf-8
vary: Accept-Encoding
date: Thu, 09 Jan 2020 20:21:05 GMT
x-envoy-upstream-service-time: 1
server: envoy
<!doctype html>....
nc localhost 80  0.00s user 0.01s system 0% cpu 37.362 total

Todos

[x] Tests
[x] Documentation

Other

If this is a documentation change, please open the pull request at https://github.com/datawire/ambassador-docs instead.
Code changes should occur against master.

@kflynn  Please review when you are able


text:
Prompt to press enter to search

Display small text prompt, like "press enter to search for a game" after X seconds when stopped typing.

Added PR to fix this issue. #5

authors: IvanSanchezSalvador datguysheepy