text:
Intl is not working for multiple packages project

I created some packages and one flutter application , these projects use Intl for localizations.
Packages and application:

a package named member_end, pure dart package, contains business logic codes, it uses Intl and intl_translation for localizations, it has a custom MemberLocalizations class that defined localization message getters and has a load method.
member_end_flutter, Flutter package, contains common widgets and implements for Flutter, it depends on member_end, it uses intl and intl_utils for localizations, the localizations class is named MemberFlutterLocalizations.
member_end_app, Flutter application, it depends on member_end and member_end_flutter, it uses intl and intl_utils for localizations, the localizations class is default S.

These projects supports en and zh Locales.
Files:

member_end
member_end
|---lib
|---|---l10n
|---|---|---intl_en.arb
|---|---|---intl_zh.arb
|---|---src
|---|---|---intl
|---|---|---|---messages_all.dart
|---|---|---|---messages_en.dart
|---|---|---|---messages_zh.dart
member_end_flutter
member_end_flutter
|---lib
|---|---l10n
|---|---|---intl_en.arb
|---|---|---intl_zh.arb
|---|---generated
|---|---|---l10n.dart
|---|---|---intl
|---|---|---|---messages_all.dart
|---|---|---|---messages_en.dart
|---|---|---|---messages_zh.dart
member_end_app
member_end_app
|---lib
|---|---l10n
|---|---|---intl_en.arb
|---|---|---intl_zh.arb
|---|---generated
|---|---|---l10n.dart
|---|---|---intl
|---|---|---|---messages_all.dart
|---|---|---|---messages_en.dart
|---|---|---|---messages_zh.dart

Let's say the current locale is zh, the Localizations classes are loaded in order

MemberLocalizations
MemberFlutterLocalizations
S

The problem is only the first MemberLocalizations will load its member_end/lib/src/intl/messages_zh.dart, this cause member_end_flutter and member_end_app can't get the correct locale messages.
In Localizations classes static Future<S> load(Locale locale) method, it use Future<bool> initializeMessages(String localeName) method to init and load messages,  Future<bool> initializeMessages(String localeName) use CompositeMessageLookup to add locale messages, let's check CompositeMessageLookup.addLocale method:
  /// If we do not already have a locale for [localeName] then
  /// [findLocale] will be called and the result stored as the lookup
  /// mechanism for that locale.
  @override
  void addLocale(String localeName, Function findLocale) {
    if (localeExists(localeName)) return;
    var canonical = Intl.canonicalizedLocale(localeName);
    var newLocale = findLocale(canonical);
    if (newLocale != null) {
      availableMessages[localeName] = newLocale;
      availableMessages[canonical] = newLocale;
      // If there was already a failed lookup for [newLocale], null the cache.
      if (_lastLocale == newLocale) {
        _lastLocale = null;
        _lastLookup = null;
      }
    }
  }

When the first MemberLocalizations load, the locale zh is not exists, so localeExists(localeName) returns false, and then the member_end package's zh locale message will load. MemberFlutterLocalizations will be loaded by next in the order, when it runs into CompositeMessageLookup.addLocale, localeExists(localeName) returns true, because locale zh MessageLookupByLibrary is already added by MemberLocalizations in member_end package, S will be the same when it's loading.
To solve this issue, I have few ways to do:

Write hardcode local messages in sub Localizations class, like Flutter framework do. But this is not the way to use intl.
Create a subclass of CompositeMessageLookup named CustomCompositeMessageLookup and override method addLocale, check if locale exists and then merge the new MessageLookupByLibrary into the old MessageLookupByLibrary, if the locale message name is already exists then overwrite with the new value that provided by the new MessageLookupByLibrary, then call void initializeInternalMessageLookup(()=>CustomCompositeMessageLookup()) method in the main method to init global field MessageLookup messageLookup. But initializeInternalMessageLookup is not a public API.
As a feature request, maybe you guys can do this awesome work, make intl works in multiple projects.

If there is other better way to solve this, please tell me :)

I have the same problem, can you share with me the solution 2 you did?

@Douglas-Pontes
Solution 2:
class MultiCompositeMessageLookup extends CompositeMessageLookup {
  @override
  void addLocale(String localeName, Function findLocale) {
    final canonical = Intl.canonicalizedLocale(localeName);
    final newLocale = findLocale(canonical);
    if (newLocale != null) {
      final oldLocale = availableMessages[localeName];
      if (oldLocale != null && newLocale != oldLocale) {
        if (newLocale is! MessageLookupByLibrary) {
          throw Exception('Merge locale messages failed, type ${newLocale.runtimeType} is not supported.');
        }
        // solve issue https://github.com/dart-lang/i18n/issues/798 if you are using intl_translate and intl_util both.
        if (oldLocale.messages is Map<String, Function> && newLocale.messages is! Map<String, Function>) {
          final newMessages = newLocale.messages.map((key, value) => MapEntry(key, value as Function));
          oldLocale.messages.addAll(newMessages);
        } else {
          oldLocale.messages.addAll(newLocale.messages);
        }
        return;
      }
      super.addLocale(localeName, findLocale);
    }
  }
}

Then call initializeInternalMessageLookup(() => MultiCompositeMessageLookup()); before any localizations class load method.

I have the same problem. Almost all examples I found use a very simple one-package setup. How do people use this in larger projects?

@stwarwas Just call initializeInternalMessageLookup(() => MultiCompositeMessageLookup()); at the first line in your main method.

simple solution is - https://github.com/Luvti/i18n
dependency_overrides:
  intl: #0.19.0
    git:
      url: https://github.com/Luvti/i18n
      path: pkgs/intl

authors: Douglas-Pontes Luvti codelovercc stwarwas


source: gharchive/issue

repo: dart-lang/native

license_type: permissive

license_source: github-api


text:
dartanalyzer is too slow.

$ time dartanalyzer <redacted>.dart
Analyzing <redacted>.dart...
<redacted>
1 error and 1 warning found.
real	0m4.150s
user	0m10.053s
sys	0m0.248s
This is on a current-model ThinkPad, analyzing a 71 line source file.
IMO, an acceptable run time for this class of tool is <500ms.

I use dartanalyser with vim plugin syntastic and it's blocking vim when I run :SyntasticCheck about 2~3 sec. There is my time result:
$ time dartanalyzer foo.dart
Analyzing [foo.dart]...
No issues found

real	0m2.469s
user	0m2.404s
sys	0m0.156s

$ dart --version
Dart VM version: 1.12.1 (Tue Sep  8 11:14:08 2015) on "linux_x64"

authors: jbdeboer uralbash

repo: dart-lang/sdk


text:
Please review new void used as type message

I have looked at these error messages:
foo.dart:1:1: expected identifier, but got 'void'
void x;
^^^^
foo.dart:1:5: Error: Type "void" is only allowed in a return type.
foo(void x) {}
    ^^^^
It is changed to:
foo.dart:1:1: Error: Type 'void' can't be used here because it isn't a return type.
Try removing 'void' keyword or replace it with 'var', 'final', or a type.
void x;
^^^^
foo.dart:1:5: Error: Type 'void' can't be used here because it isn't a return type.
Try removing 'void' keyword or replace it with 'var', 'final', or a type.
foo(void x) {}
    ^^^^
Let me know what you think.

I'm guessing this has been resolved and can be closed, right @peter-ahe-google?

authors: bkonyi peter-ahe-google


text:
Extend OAuth2 package to support OpenID connect

This issue was originally filed by warren.strange...@gmail.com

After thinking this through, I realize this is a fairly major feature request.
The current OAuth2 package (http://pub.dartlang.org/packages/oauth2)
does not support OpenID connect. For example, Openid connect returns a id_token as part of the authorization flow.
A small enhancement would be to extend Credentials.dart to provide the raw value of id_token if it is present.
Ideally, support would be provided for JSON web tokens, signature verification, etc.

This issue has been moved to dart-lang/oauth2#8.


text:
dart:io WebSocket needs OnBadCertificate() callback for wss:// socket connections

This issue was originally filed by j.m.sloc...@gmail.com

Using Dart VM version: 1.1.1 (Wed Jan 15 04:11:49 2014) on "linux_x64"
The following dart program starts a secure HTTP server and waits for a websocket connection
import 'dart:io';
void main(List<String> args){
  String password = new File('pwdfile').readAsStringSync().trim();
  SecureSocket.initialize(database: "./",
                          password: password);
  HttpServer.bindSecure(InternetAddress.ANY_IP_V4, 4443,
      certificateName: "CN=devcert")
    .then((HttpServer server) {
      print("Secure server listening on 4443...");
      server.serverHeader = "Secure WebSocket server";
      server.listen((HttpRequest request) {
        if (request.headers.value(HttpHeaders.UPGRADE) == "websocket"){
          WebSocketTransformer.upgrade(request).then(handleWebSocket);
        }
        else {
          request.response.statusCode = HttpStatus.FORBIDDEN;
          request.response.reasonPhrase = "WebSocket connections only";
          request.response.close();
        }
      });
    });
}
void handleWebSocket(WebSocket socket){
  print("Secure client connected!");
  socket.listen((String s) {
    print('Client sent: $s');
    socket.add('echo: $s');
  },
  onDone: () {
    print('Client disconnected');
  });
}
The following program is a client that can connect to websockets.
import 'dart:io';
WebSocket ws;
void main(List<String> args){
  if (args.length < 1){
    print('Please specify a server URI. ex ws://example.org');
    exit(1);
  }
  String server = args[0];
  //Open the websocket and attach the callbacks
  WebSocket.connect(server).then((WebSocket socket) {
    ws = socket;
    ws.listen(onMessage, onDone: connectionClosed);
  });
  //Attach to stdin to read from the keyboard
  stdin.listen(onInput);
}
void onMessage(String message){
  print(message);
}
void connectionClosed() {
  print('Connection to server closed');
}
void onInput(List<int> input){
  String message = new String.fromCharCodes(input).trim();
  //Exit gracefully if the user types 'quit'
  if (message == 'quit'){
    ws.close();
    exit(0);
  }
  ws.add(message);
}
What is the expected output? What do you see instead?
When I run this server using a self signed cert and try to connect with a client I get the following exception
$ dart secureWebSocketClient.dart wss://localhost:4443
Uncaught Error: HandshakeException: Handshake error in client (OS Error: Issuer certificate is invalid., errno = -8156)
Unhandled exception:
HandshakeException: Handshake error in client (OS Error: Issuer certificate is invalid., errno = -8156)
0      _rootHandleUncaughtError.<anonymous closure>.<anonymous closure> (dart:async/zone.dart:677)
1      _asyncRunCallback (dart:async/schedule_microtask.dart:18)
2      _asyncRunCallback (dart:async/schedule_microtask.dart:21)
3      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:119)
However there is no way of indicating to the WebSocket class to ignore certificate errors.
The server works if I use a "plain" HTTP server, but not a secure server. It would appear that the WebSocket class should have an onBadCertificate(X509Certificate) callback like the SecureSocket classes.

Has this been addressed?  I am having an issue where I need to accept a self signed cert and this is holding me up.

after 5 years, this is still open. A solution needed.

At least for the dart:io version of websockets, an onBadCertificate() would be great!

authors: DartBot EPNW linuxjet neaplus


text:
Comparison of String and dynamic (function) fails with uncaught error in core_ajax_dart.dart

This issue was originally filed by frable9...@gmail.com

What steps will reproduce the problem?

Fire a CoreAjax request with go()

What is the expected output? What do you see instead?
I get an uncaught error from the core_ajax_dart.dart.
The essential code snippet from core_ajax_dart.dart is:
if (!hasContentType && this.contentType) {
  headers['Content-Type'] = this.contentType;
}
whereby hasContentType is a function with a boolean as return value and this.contentType is a String.
Exception: Uncaught Error: type 'String' is not a subtype of type 'bool' of 'boolean expression'.
Stack Trace:
#0      CoreAjax.go (package:core_elements/core_ajax_dart.dart:285:33)
#1      AelAjax.go (http://localhost:8080/components/ael-ajax/ael-ajax.dart:39:17)
#2      AelCtrl.getUserProfile (http://localhost:8080/app.dart:157:26)
#3      AelCtrl.AelCtrl (http://localhost:8080/app.dart:77:19)
#4      main.<anonymous closure>.<anonymous closure> (http://localhost:8080/app.dart:27:18)
#5      _RootZone.runUnary (dart:async/zone.dart:1082)
#6      _Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:488)
#7      _Future._propagateToListeners (dart:async/future_impl.dart:571)
#8      _Future._completeWithValue (dart:async/future_impl.dart:331)
#9      _Future._asyncComplete.<anonymous closure> (dart:async/future_impl.dart:393)
#10     _asyncRunCallbackLoop (dart:async/schedule_microtask.dart:41)
#11     _asyncRunCallback (dart:async/schedule_microtask.dart:48)
#12     _handleMutation (dart:html:39006)
What version of the product are you using?
core_elements 0.2.1+1
Dart 1.6.0
On what operating system?
Windows 7 64 bit
What browser (if applicable)?
Dartium 37.0.2062.76
Please provide any additional information below.

This issue has been moved to dart-lang/polymer-dart#301.


text:
Analyzer: When multiple imports provide the same symbol, one import should be marked as "unused"

e.g. In Angular, the package:angular/angular.dart file exports package:di/di.dart.
In the following file, the Module symbol is coming from di.dart, but also exported through angular.
import package:angular/angular.dart
import package:di/di.dart

class MyModule extends Module { ... }

Currently, the analyzer does not give any hints about unused imports.
However, I would expect angular.dart to be flagged as "unused". angular.dart is not used since Module is also available through di.dart.

Even a subset of this, examining just shown names would be useful. I found some code with:
import 'package:a/a.dart';
import 'package:a/src/foo.dart' show foo;

because at one point, a.dart did not export foo. But now it does, so the second import is unnecessary. Not sure if one is easier to implement or faster to run than the other...

I'll close this in favor of the issue I've been referencing when landing changes. https://github.com/dart-lang/sdk/issues/44569

authors: jbdeboer srawlins


text:
dartanalyzer crashes when .packages target is empty

.packages:
foo:

stderr:
Bad state: No element
#0      List.last (dart:core-patch/growable_array.dart:212)
#1      startsWith (package:analyzer/src/generated/utilities_dart.dart:47:20)
#2      SourceFactory._getPackageMapping.<anonymous closure> (package:analyzer/src/generated/source.dart:786:13)
#3      _HashVMBase&MapMixin&&_LinkedHashMapMixin.forEach (dart:collection-patch/compact_hash.dart:340)
#4      MapView.forEach (dart:collection/maps.dart:194)
#5      SourceFactory._getPackageMapping (package:analyzer/src/generated/source.dart:784:23)
#6      SourceFactory.restoreUri (package:analyzer/src/generated/source.dart:762:32)
#7      Driver._computeLibrarySource (package:analyzer_cli/src/driver.dart:341:38)
#8      Driver._analyzeAll (package:analyzer_cli/src/driver.dart:137:23)
#9      Driver.start.<anonymous closure> (package:analyzer_cli/src/driver.dart:99:16)
#10     _BatchRunner.runAsBatch.<anonymous closure> (package:analyzer_cli/src/driver.dart:536:39)
#11     _RootZone.runUnaryGuarded (dart:async/zone.dart:1103)
#12     _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:341)
#13     _BufferingStreamSubscription._add (dart:async/stream_impl.dart:270)
#14     _SinkTransformerStreamSubscription._add (dart:async/stream_transformers.dart:67)
#15     _EventSinkWrapper.add (dart:async/stream_transformers.dart:14)
#16     _StringAdapterSink.add (dart:convert/string_conversion.dart:256)
#17     _LineSplitterSink._addLines (dart:convert/line_splitter.dart:127)
#18     _LineSplitterSink.addSlice (dart:convert/line_splitter.dart:102)
#19     StringConversionSinkMixin.add (dart:convert/string_conversion.dart:180)
#20     _ConverterStreamEventSink.add (dart:convert/chunked_conversion.dart:80)
#21     _SinkTransformerStreamSubscription._handleData (dart:async/stream_transformers.dart:119)
#22     _RootZone.runUnaryGuarded (dart:async/zone.dart:1103)
#23     _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:341)
#24     _BufferingStreamSubscription._add (dart:async/stream_impl.dart:270)
#25     _SinkTransformerStreamSubscription._add (dart:async/stream_transformers.dart:67)
#26     _EventSinkWrapper.add (dart:async/stream_transformers.dart:14)
#27     _StringAdapterSink.add (dart:convert/string_conversion.dart:256)
#28     _StringAdapterSink.addSlice (dart:convert/string_conversion.dart:260)
#29     _Utf8ConversionSink.addSlice (dart:convert/string_conversion.dart:336)
#30     _Utf8ConversionSink.add (dart:convert/string_conversion.dart:329)
#31     _ConverterStreamEventSink.add (dart:convert/chunked_conversion.dart:80)
#32     _SinkTransformerStreamSubscription._handleData (dart:async/stream_transformers.dart:119)
#33     _RootZone.runUnaryGuarded (dart:async/zone.dart:1103)
#34     _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:341)
#35     _BufferingStreamSubscription._add (dart:async/stream_impl.dart:270)
#36     _StreamController&&_SyncStreamControllerDispatch._sendData (dart:async/stream_controller.dart:744)
#37     _StreamController._add (dart:async/stream_controller.dart:616)
#38     _StreamController.add (dart:async/stream_controller.dart:562)
#39     _Socket._onData (dart:io-patch/socket_patch.dart:1793)
#40     _RootZone.runUnaryGuarded (dart:async/zone.dart:1103)
#41     _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:341)
#42     _BufferingStreamSubscription._add (dart:async/stream_impl.dart:270)
#43     _StreamController&&_SyncStreamControllerDispatch._sendData (dart:async/stream_controller.dart:744)
#44     _StreamController._add (dart:async/stream_controller.dart:616)
#45     _StreamController.add (dart:async/stream_controller.dart:562)
#46     _RawSocket._RawSocket.<anonymous closure> (dart:io-patch/socket_patch.dart:1344)
#47     _NativeSocket.issueReadEvent.issue (dart:io-patch/socket_patch.dart:728)
#48     _microtaskLoop (dart:async/schedule_microtask.dart:43)
#49     _microtaskLoopEntry (dart:async/schedule_microtask.dart:52)
#50     _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:96)
#51     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:149)

This looks like an invalid .packages file that should have been caught by the code that reads the .packages file.

We are asking Lasse what the appropriate thing to do is in this case:
https://codereview.chromium.org/1298323002/
On Tue, Aug 18, 2015, 4:50 PM Brian Wilkerson notifications@github.com
wrote:

This looks like an invalid .packages file that should have been caught by
the code that reads the .packages file.
—
Reply to this email directly or view it on GitHub
https://github.com/dart-lang/sdk/issues/24126#issuecomment-132393040.

We are asking Lasse what the appropriate thing to do is in this case:
https://codereview.chromium.org/1298323002/

Awesome.  Please update this issue when we know where we're headed.  It'd be easy enough to guard against on our end but probably better handled in package_config once and for all rather than in all the clients.

This crash looks like a "real" crash, and thus a candidate for fixing for 1.12.

This crash looks like a "real" crash, and thus a candidate for fixing for 1.12.

Agreed.
We should fix it in package_config.  Feel free to open a bug there and assign it to me and I'll happily take a look.

We should fix it in package_config. Feel free to open a bug there and assign it to me and I'll happily take a look.

Actually, I'm less sure now.  I'm looking into it.

https://codereview.chromium.org/1298393004/

Fixed with e11ce8ba87952ee2efeb7ed8211801f6cb6d9c9d.
Request to merge to dev filed here: https://github.com/dart-lang/sdk/issues/24138.

authors: bwilkerson hterkelsen pq sethladd


text:
FSE in Analyzer

This may or may not be a bug. The occurance count looks a little high. The elided path name is available on request, it was in Angular.

2016-02-04T11:51:32.918: Common stack: Count: 392
Session: 1454002852255.0693
Caused by FileSystemException:: Cannot open file, path = '...' (OS Error:: No such file or directory, errno = 2)
#0      _File.throwIfError (dart::io/file_impl.dart::562)
#1      _File.openSync (dart::io/file_impl.dart::413)
#2      _File.readAsBytesSync (dart::io/file_impl.dart::473)
#3      _File.readAsStringSync (dart::io/file_impl.dart::507)
#4      JavaFile.readAsStringSync (package::analyzer/src/generated/java_io.dart::100)
#5      FileBasedSource.contentsFromFile (package::analyzer/src/generated/source_io.dart::180)
#6      FileBasedSource.contents.<anonymous closure> (package::analyzer/src/generated/source_io.dart::163)
#7      _PerformanceTagImpl.makeCurrentWhile (package::analyzer/src/generated/utilities_general.dart::170)
#8      FileBasedSource.contents (package::analyzer/src/generated/source_io.dart::162)
#9      AnalysisContextImpl.getContents (package::analyzer/src/context/context.dart::777)
#10     AnalysisContextImpl.parseCompilationUnit (package::analyzer/src/context/context.dart::1070)
#11     DartCompletionManager.computeFast.<anonymous closure> (package::analysis_server/src/services/completion/dart_completion_manager.dart::148)
#12     CompletionPerformance.logElapseTime (package::analysis_server/src/services/completion/completion_manager.dart::165)
#13     DartCompletionManager.computeFast (package::analysis_server/src/services/completion/dart_completion_manager.dart::147)
#14     DartCompletionManager.computeSuggestions.<anonymous closure> (package::analysis_server/src/services/completion/dart_completion_manager.dart::243)
#15     CompletionPerformance.logElapseTime (package::analysis_server/src/services/completion/completion_manager.dart::165)
#16     DartCompletionManager.computeSuggestions (package::analysis_server/src/services/completion/dart_completion_manager.dart::242)
#17     CompletionManager.results.<anonymous closure>.<anonymous closure> (package::analysis_server/src/services/completion/completion_manager.dart::108)
#18     _rootRun (dart::async/zone.dart::903)
#19     _CustomZone.run (dart::async/zone.dart::802)
#20     _CustomZone.runGuarded (dart::async/zone.dart::708)
#21     _CustomZone.bindCallback.<anonymous closure> (dart::async/zone.dart::733)
#22     _rootRun (dart::async/zone.dart::907)
#23     _CustomZone.run (dart::async/zone.dart::802)
#24     _CustomZone.runGuarded (dart::async/zone.dart::708)
#25     _CustomZone.bindCallback.<anonymous closure> (dart::async/zone.dart::733)
#26     _microtaskLoop (dart::async/schedule_microtask.dart::43)
#27     _microtaskLoopEntry (dart::async/schedule_microtask.dart::52)
#28     _runPendingImmediateCallback (dart::isolate-patch/isolate_patch.dart::96)
#29     _RawReceivePortImpl._handleMessage (dart::isolate-patch/isolate_patch.dart::151)

","stackTrace"::"#0      AnalysisContextImpl.parseCompilationUnit (package::analyzer/src/context/context.dart::1072)
#1      DartCompletionManager.computeFast.<anonymous closure> (package::analysis_server/src/services/completion/dart_completion_manager.dart::148)
#2      CompletionPerformance.logElapseTime (package::analysis_server/src/services/completion/completion_manager.dart::165)
#3      DartCompletionManager.computeFast (package::analysis_server/src/services/completion/dart_completion_manager.dart::147)
#4      DartCompletionManager.computeSuggestions.<anonymous closure> (package::analysis_server/src/services/completion/dart_completion_manager.dart::243)
#5      CompletionPerformance.logElapseTime (package::analysis_server/src/services/completion/completion_manager.dart::165)
#6      DartCompletionManager.computeSuggestions (package::analysis_server/src/services/completion/dart_completion_manager.dart::242)
#7      CompletionManager.results.<anonymous closure>.<anonymous closure> (package::analysis_server/src/services/completion/completion_manager.dart::108)
#8      _rootRun (dart::async/zone.dart::903)
#9      _CustomZone.run (dart::async/zone.dart::802)
#10     _CustomZone.runGuarded (dart::async/zone.dart::708)
#11     _CustomZone.bindCallback.<anonymous closure> (dart::async/zone.dart::733)
#12     _rootRun (dart::async/zone.dart::907)
#13     _CustomZone.run (dart::async/zone.dart::802)
#14     _CustomZone.runGuarded (dart::async/zone.dart::708)
#15     _CustomZone.bindCallback.<anonymous closure> (dart::async/zone.dart::733)
#16     _microtaskLoop (dart::async/schedule_microtask.dart::43)
#17     _microtaskLoopEntry (dart::async/schedule_microtask.dart::52)
#18     _runPendingImmediateCallback (dart::isolate-patch/isolate_patch.dart::96)
#19     _RawReceivePortImpl._handleMessage (dart::isolate-patch/isolate_patch.dart
Original message:
{"event"::"server.error","params"::{"isFatal"::false,"message"::"Failed to handle completion domain request:: {clientRequestTime:: 1449509388893, params:: {file:: ..., offset:: 214}, method:: completion.getSuggestions, id:: 384}
AnalysisException:: Could not get contents of ...
Caused by FileSystemException:: Cannot open file, path = '...' (OS Error:: No such file or directory, errno = 2)
#0      _File.throwIfError (dart::io/file_impl.dart::562)
#1      _File.openSync (dart::io/file_impl.dart::413)
#2      _File.readAsBytesSync (dart::io/file_impl.dart::473)
#3      _File.readAsStringSync (dart::io/file_impl.dart::507)
#4      JavaFile.readAsStringSync (package::analyzer/src/generated/java_io.dart::100)
#5      FileBasedSource.contentsFromFile (package::analyzer/src/generated/source_io.dart::180)
#6      FileBasedSource.contents.<anonymous closure> (package::analyzer/src/generated/source_io.dart::163)
#7      _PerformanceTagImpl.makeCurrentWhile (package::analyzer/src/generated/utilities_general.dart::170)
#8      FileBasedSource.contents (package::analyzer/src/generated/source_io.dart::162)
#9      AnalysisContextImpl.getContents (package::analyzer/src/context/context.dart::777)
#10     AnalysisContextImpl.parseCompilationUnit (package::analyzer/src/context/context.dart::1070)
#11     DartCompletionManager.computeFast.<anonymous closure> (package::analysis_server/src/services/completion/dart_completion_manager.dart::148)
#12     CompletionPerformance.logElapseTime (package::analysis_server/src/services/completion/completion_manager.dart::165)
#13     DartCompletionManager.computeFast (package::analysis_server/src/services/completion/dart_completion_manager.dart::147)
#14     DartCompletionManager.computeSuggestions.<anonymous closure> (package::analysis_server/src/services/completion/dart_completion_manager.dart::243)
#15     CompletionPerformance.logElapseTime (package::analysis_server/src/services/completion/completion_manager.dart::165)
#16     DartCompletionManager.computeSuggestions (package::analysis_server/src/services/completion/dart_completion_manager.dart::242)
#17     CompletionManager.results.<anonymous closure>.<anonymous closure> (package::analysis_server/src/services/completion/completion_manager.dart::108)
#18     _rootRun (dart::async/zone.dart::903)
#19     _CustomZone.run (dart::async/zone.dart::802)
#20     _CustomZone.runGuarded (dart::async/zone.dart::708)
#21     _CustomZone.bindCallback.<anonymous closure> (dart::async/zone.dart::733)
#22     _rootRun (dart::async/zone.dart::907)
#23     _CustomZone.run (dart::async/zone.dart::802)
#24     _CustomZone.runGuarded (dart::async/zone.dart::708)
#25     _CustomZone.bindCallback.<anonymous closure> (dart::async/zone.dart::733)
#26     _microtaskLoop (dart::async/schedule_microtask.dart::43)
#27     _microtaskLoopEntry (dart::async/schedule_microtask.dart::52)
#28     _runPendingImmediateCallback (dart::isolate-patch/isolate_patch.dart::96)
#29     _RawReceivePortImpl._handleMessage (dart::isolate-patch/isolate_patch.dart::151)

","stackTrace"::"#0      AnalysisContextImpl.parseCompilationUnit (package::analyzer/src/context/context.dart::1072)
#1      DartCompletionManager.computeFast.<anonymous closure> (package::analysis_server/src/services/completion/dart_completion_manager.dart::148)
#2      CompletionPerformance.logElapseTime (package::analysis_server/src/services/completion/completion_manager.dart::165)
#3      DartCompletionManager.computeFast (package::analysis_server/src/services/completion/dart_completion_manager.dart::147)
#4      DartCompletionManager.computeSuggestions.<anonymous closure> (package::analysis_server/src/services/completion/dart_completion_manager.dart::243)
#5      CompletionPerformance.logElapseTime (package::analysis_server/src/services/completion/completion_manager.dart::165)
#6      DartCompletionManager.computeSuggestions (package::analysis_server/src/services/completion/dart_completion_manager.dart::242)
#7      CompletionManager.results.<anonymous closure>.<anonymous closure> (package::analysis_server/src/services/completion/completion_manager.dart::108)
#8      _rootRun (dart::async/zone.dart::903)
#9      _CustomZone.run (dart::async/zone.dart::802)
#10     _CustomZone.runGuarded (dart::async/zone.dart::708)
#11     _CustomZone.bindCallback.<anonymous closure> (dart::async/zone.dart::733)
#12     _rootRun (dart::async/zone.dart::907)
#13     _CustomZone.run (dart::async/zone.dart::802)
#14     _CustomZone.runGuarded (dart::async/zone.dart::708)
#15     _CustomZone.bindCallback.<anonymous closure> (dart::async/zone.dart::733)
#16     _microtaskLoop (dart::async/schedule_microtask.dart::43)
#17     _microtaskLoopEntry (dart::async/schedule_microtask.dart::52)
#18     _runPendingImmediateCallback (dart::isolate-patch/isolate_patch.dart::96)
#19     _RawReceivePortImpl._handleMessage (dart::isolate-patch/isolate_patch.dart::151)
"}}

v 1.14.0-dev.7.2

Confirmed this is ocuring in v 1.14.0-dev.7.2

@danrubel The method DartCompletionManager.computeFast (from line 11 of the stack trace) no longer exists, nor does that file contain any invocations of AnalysisContextImpl.parseCompilationUnit (line 10). I'm guessing this can be closed as stale.

Yes, this is stale.

authors: bwilkerson danrubel lukechurch


text:
BitTestImmediate crashes on Windows 32-bit in Dart 1 mode

This tests started failing after landing 2beb05b8. There is a corresponding line in vm.status. Marking as P2 because it only occurs in Dart 1 mode.

https://dart-review.googlesource.com/c/sdk/+/78861 should fix this.

authors: a-siva sjindel-google


text:
error: expected: !is_optimized() || (pc_offset == EntryPoint() - PayloadStart())

Isolate (/b/s/w/itHvzd2Z/dart_fuzzHFHIBB) NO-FFI NO-FP : AOT-ReleaseX64 - KBC-INT-SRC-DebugIA32: !DIVERGENCE! 1.37:1253594753 (0 vs -6)

fail2:
-6

../../runtime/vm/object.cc: 15284: error: expected: !is_optimized() || (pc_offset == EntryPoint() - PayloadStart())
version=2.6.0-edge.2246f0a8a36fbad0cd9c87aacc22ba29086d1c74 (Fri Sep 6 03:29:28 2019 +0000) on "linux_ia32"
thread=15485, isolate=main(0x4910700)
  pc 0x01cc698c fp 0xf607ce08 dart::Profiler::DumpStackTrace(void*)
  pc 0x021cc6b4 fp 0xf607ce28 Dart_DumpNativeStackTrace
  pc 0x0193de8b fp 0xf607ce58 dart::Assert::Fail(char const*, ...)
  pc 0x01c6b895 fp 0xf607cea8 dart::Code::GetStackMap(unsigned int, dart::Array*, dart::StackMap*) const
  pc 0x01d5a7bd fp 0xf607cf28 dart::StackFrame::VisitObjectPointers(dart::ObjectPointerVisitor*)
  pc 0x01d6ec0a fp 0xf607cfd8 dart::Thread::VisitObjectPointers(dart::ObjectPointerVisitor*, dart::ValidationPolicy)
  pc 0x01bc1f9d fp 0xf607d018 dart::Isolate::VisitObjectPointers(dart::ObjectPointerVisitor*, dart::ValidationPolicy)
  pc 0x01f564a6 fp 0xf607d098 dart::Scavenger::IterateRoots(dart::Isolate*, dart::ScavengerVisitor*)
  pc 0x01f57adc fp 0xf607d1b8 dart::Scavenger::Scavenge()
  pc 0x01f446a0 fp 0xf607d238 dart::Heap::CollectNewSpaceGarbage(dart::Thread*, dart::Heap::GCReason)
  pc 0x01f426bd fp 0xf607d278 dart::Heap::AllocateNew(int)
  pc 0x01bff03b fp 0xf607d2a8 /b/s/w/ir/out/DebugIA32/dart+0x17ff03b
  pc 0x01c02dee fp 0xf607d308 dart::Object::Allocate(int, int, dart::Heap::Space)
  pc 0x01c804b9 fp 0xf607d358 dart::OneByteString::New(int, dart::Heap::Space)
  pc 0x01c80262 fp 0xf607d3b8 dart::String::FromUTF8(unsigned char const*, int, dart::Heap::Space)
  pc 0x01efa0f2 fp 0xf607d418 dart::kernel::TranslationHelper::DartString(dart::kernel::StringIndex, dart::Heap::Space)
  pc 0x01f01dd7 fp 0xf607d468 dart::kernel::KernelReaderHelper::ReadNameAsMethodName()
  pc 0x01edbfef fp 0xf607d598 dart::kernel::StreamingFlowGraphBuilder::BuildMethodInvocation(dart::TokenPosition*)
  pc 0x01ed46a7 fp 0xf607d5e8 dart::kernel::StreamingFlowGraphBuilder::BuildExpression(dart::TokenPosition*)
  pc 0x01ed6257 fp 0xf607d618 dart::kernel::StreamingFlowGraphBuilder::BuildStatement()
  pc 0x01edfcda fp 0xf607d668 dart::kernel::StreamingFlowGraphBuilder::BuildBlockExpression()
  pc 0x01ed46b6 fp 0xf607d6b8 dart::kernel::StreamingFlowGraphBuilder::BuildExpression(dart::TokenPosition*)
  pc 0x01ed4373 fp 0xf607d738 dart::kernel::StreamingFlowGraphBuilder::BuildGraphOfFieldInitializer()
  pc 0x01ed8e8b fp 0xf607d858 dart::kernel::StreamingFlowGraphBuilder::BuildGraph()
  pc 0x01eeddc4 fp 0xf607d9b8 dart::kernel::FlowGraphBuilder::BuildGraph()
  pc 0x01f1814c fp 0xf607dac8 dart::DartCompilationPipeline::BuildFlowGraph(dart::Zone*, dart::ParsedFunction*, dart::ZoneGrowableArray<dart::ICData const*>*, int, bool)
  pc 0x01f199b0 fp 0xf607de48 dart::CompileParsedFunctionHelper::Compile(dart::CompilationPipeline*)
  pc 0x01f1ab55 fp 0xf607dfd8 /b/s/w/ir/out/DebugIA32/dart+0x1b1ab55
  pc 0x01f1b6bf fp 0xf607e058 dart::Compiler::CompileOptimizedFunction(dart::Thread*, dart::Function const&, int)
  pc 0x01f1c70d fp 0xf607e0d8 dart::BackgroundCompiler::Run()
  pc 0x01f1d7a7 fp 0xf607e0f8 /b/s/w/ir/out/DebugIA32/dart+0x1b1d7a7
  pc 0x01d7372f fp 0xf607e148 dart::ThreadPool::Worker::Loop()
  pc 0x01d732ca fp 0xf607e198 dart::ThreadPool::Worker::Main(unsigned int)
  pc 0x01cc0bbf fp 0xf607e2e8 /b/s/w/ir/out/DebugIA32/dart+0x18c0bbf
  pc 0xf7ce6295 fp 0xf607e3a8 /lib/i386-linux-gnu/libpthread.so.0+0x6295
-- End of DumpStackTrace

To reproduce:
dart fuzz.dart.txt

fuzz.dart.txt

Fixes pending at:
#38231
#38248

authors: aartbik feli-citas


text:
Dart Analyzer Error

Analyzer Feedback from IntelliJ
Version information

IDEA AI-191.8026.42.35.5977832
2.5.0-edge.1ef83b86ae637ffe7359173804cbc6d3fa25e6db
AI-191.8026.42.35.5977832, JRE 1.8.0_202-release-1483-b03x64 JetBrains s.r.o, OS Windows 10(amd64) v10.0 , screens 1536x864, 2560x1440

Exception
Dart analysis server, SDK version 2.5.0-edge.1ef83b86ae637ffe7359173804cbc6d3fa25e6db, server version 1.27.2, error: Captured exception
RangeError: Value not in range: 1
#0      _StringBase.substring (dart:core-patch/string_patch.dart:392:7)
#1      unescapeString (package:front_end/src/fasta/quote.dart:140:14)
#2      AstBuilder.endLiteralString (package:analyzer/src/fasta/ast_builder.dart:1415:22)
#3      Parser.parseSingleLiteralString (package:front_end/src/fasta/parser/parser.dart:4892:14)
#4      Parser.parseLiteralString (package:front_end/src/fasta/parser/parser.dart:4817:13)
#5      Parser.parsePrimary (package:front_end/src/fasta/parser/parser.dart:4251:14)
#6      Parser.parseUnaryExpression (package:front_end/src/fasta/parser/parser.dart:4184:12)
#7      Parser.parsePrecedenceExpression (package:front_end/src/fasta/parser/parser.dart:3972:13)
#8      Parser.parseExpression (package:front_end/src/fasta/parser/parser.dart:3944:13)
#9      Parser.parseLiteralListSuffix (package:front_end/src/fasta/parser/parser.dart:4437:19)
#10     Parser.parsePrimary (package:front_end/src/fasta/parser/parser.dart:4290:14)
#11     Parser.parseUnaryExpression (package:front_end/src/fasta/parser/parser.dart:4184:12)
#12     Parser.parsePrecedenceExpression (package:front_end/src/fasta/parser/parser.dart:3972:13)
#13     Parser.parseExpression (package:front_end/src/fasta/parser/parser.dart:3944:13)
#14     Parser.parseVariableInitializerOpt (package:front_end/src/fasta/parser/parser.dart:2513:15)
#15     Parser.parseOptionallyInitializedIdentifier (package:front_end/src/fasta/parser/parser.dart:5389:13)
#16     Parser.parseVariablesDeclarationRest (package:front_end/src/fasta/parser/parser.dart:5370:13)
#17     Parser.parseExpressionStatementOrDeclarationAfterModifiers (package:front_end/src/fasta/parser/parser.dart:5363:15)
#18     Parser.parseStatementX (package:front_end/src/fasta/parser/parser.dart:3738:16)
#19     Parser.parseStatement (package:front_end/src/fasta/parser/parser.dart:3716:20)
#20     Parser.parseFunctionBody (package:front_end/src/fasta/parser/parser.dart:3621:15)
#21     Parser.parseTopLevelMethod (package:front_end/src/fasta/parser/parser.dart:2460:13)
#22     Parser.parseTopLevelMemberImpl (package:front_end/src/fasta/parser/parser.dart:2340:14)
#23     Parser.parseTopLevelDeclarationImpl (package:front_end/src/fasta/parser/parser.dart:493:14)
#24     Parser.parseUnit (package:front_end/src/fasta/parser/parser.dart:350:15)
#25     ParserAdapter.parseCompilationUnit2 (package:analyzer/src/generated/parser_fasta.dart:157:32)
#26     ParserAdapter.parseCompilationUnit (package:analyzer/src/generated/parser_fasta.dart:152:12)
#27     _File._parse (package:analyzer/src/services/available_declarations.dart:1839:23)
#28     _File.refresh (package:analyzer/src/services/available_declarations.dart:1160:30)
#29     DeclarationsTracker._performChangeFile (package:analyzer/src/services/available_declarations.dart:701:10)
#30     DeclarationsTracker.doWork (package:analyzer/src/services/available_declarations.dart:567:7)
#31     CompletionLibrariesWorker.performWork (package:analysis_server/src/domains/completion/available_suggestions.dart:283:13)
<asynchronous suspension>
#32     AnalysisDriverScheduler._run (package:analyzer/src/dart/analysis/driver.dart:1901:35)
<asynchronous suspension>
#33     AnalysisDriverScheduler.start (package:analyzer/src/dart/analysis/driver.dart:1855:5)
#34     new AnalysisServer (package:analysis_server/src/analysis_server.dart:212:29)
#35     SocketServer.createAnalysisServer (package:analysis_server/src/socket_server.dart:86:26)
#36     StdioAnalysisServer.serveStdio (package:analysis_server/src/server/stdio_server.dart:37:18)
#37     Driver.startAnalysisServer.<anonymous closure> (package:analysis_server/src/server/driver.dart:572:21)
#38     _rootRun (dart:async/zone.dart:1124:13)
#39     _CustomZone.run (dart:async/zone.dart:1021:19)
#40     _runZoned (dart:async/zone.dart:1516:10)
#41     runZoned (dart:async/zone.dart:1463:12)
#42     Driver._captureExceptions (package:analysis_server/src/server/driver.dart:689:12)
#43     Driver.startAnalysisServer (package:analysis_server/src/server/driver.dart:570:7)
#44     Driver.start.<anonymous closure> (package:analysis_server/src/server/driver.dart:474:9)
#45     _AsyncAwaitCompleter.start (dart:async-patch/async_patch.dart:43:6)
#46     Driver.start.<anonymous closure> (package:analysis_server/src/server/driver.dart:469:43)
#47     CompilerContext.runInContext.<anonymous closure>.<anonymous closure> (package:front_end/src/fasta/compiler_context.dart:122:46)
#48     new Future.sync (dart:async/future.dart:224:31)
#49     CompilerContext.runInContext.<anonymous closure> (package:front_end/src/fasta/compiler_context.dart:122:19)
#50     _rootRun (dart:async/zone.dart:1124:13)
#51     _CustomZone.run (dart:async/zone.dart:1021:19)
#52     _runZoned (dart:async/zone.dart:1516:10)
#53     runZoned (dart:async/zone.dart:1463:12)
#54     CompilerContext.runInContext (package:front_end/src/fasta/compiler_context.dart:121:12)
#55     CompilerConte...

For additional log information, please append the contents of
file://C:\Users\phoenix\AppData\Local\Temp\report.txt.

Thanks for the report! This was previously reported in #39092 and has been fixed. Please upgrade to the latest version, reopen if you still see the failure.

Duplicate of #39092

authors: double-headed-eagle srawlins


text:
documentation for strong-mode rules

Describe the issue
there seems to be no documentatioin for the analyzer.strong-mode options. though the implicit-casts option is mentioned briefly here, i can't seem to find any documentation for the other options.
i think there are 2 more (implicit-dynamic and declaration-casts)

raised #48651

@srawlins

The strong-mode rules are soft deprecated; soon to be for reals deprecated, so we will not be writing documentation for them.

I think this can be closed since the strong-mode rules were removed in Dart 3 and the strict language modes are documented in https://dart.dev/guides/language/analysis-options#enabling-additional-type-checks :D

I'm going to close this as the replacement strict language modes are documented at https://dart.dev/tools/analysis#enabling-additional-type-checks and https://github.com/dart-lang/sdk/issues/50679 is tracking removing the old strong-mode options.
Please open an issue on site-www if you'd like to see any further improvements to the docs. Thanks!

authors: DetachHead parlough scheglov srawlins


text:
gzip decodes only first line of the file

GZipCodec can't decode attached file. It decodes only the first line of the file. I could decode it with gzip shell comand. I have also tried other libraries from pub.dev but with no luck.
void main(List<String> arguments) async {
  final file =  File('test.csv.gz');
  final bytes = file.readAsBytesSync();
  print(bytes.length); // prints 3796
  print(gzip.decode(bytes).length); // prints 65
}

test.csv.gz

I suspect the archive itself is corrupt or somehow non-standard. On macos, I see:

Closing as I think the issue is with the archive and not the GZipCodec class. If on further investigation of the archive you believe its well-formed / something GZipCodec should parse, please re-open.

@devoncarew I updated the file. Now it opens with the standard macos archive utility. Could you check it again?

//cc @brianquinlan

I can repro. Python is able to decompress this file:
>>> s = open('test.csv.gz', 'rb').read()
>>> import gzip
>>> t = gzip.decompress(s)
>>> len(t)
68541

The bytes that Dart actually decodes are:
>>> x = [105, 100, 83, 117, 98, 67, 97, 109, 112, 97, 105, 103, 110, 84, 105, 116, 108, 101, 44, 105, 100, 83, 117, 98, 65, 100, 83, 101, 116, 84, 105, 116, 108, 101, 44, 105, 100, 83, 117, 98, 67, 97, 109, 112, 97, 105, 103, 110, 44, 105, 100, 67, 97, 109, 112, 97, 105, 103, 110, 84, 105, 116, 108, 101, 10]
>>> bytes(x)
b'idSubCampaignTitle,idSubAdSetTitle,idSubCampaign,idCampaignTitle\n'

Which is the first line of the file. If I understand correctly, the GZIP file format consists of concatenated compressed data sets. So maybe we are only decoding the first data set?

I get the same output as Dart when use the zpipe example after changing:
-    ret = inflateInit(&strm);
+    ret = inflateInit2(&strm, 32 + 15);

The Python implementation looks very similar to ours.

If I extract the file and recompress it with gzip, both Dart and zpipe and decompress the file. How did you generate this archive?

@brianquinlan This archive is from raw data export API response of https://docs.tracker.my.com/api/export-api/raw/about

Seems related to https://github.com/dart-lang/sdk/issues/47244

Yep. OK, I missed that Python deals with gzip data starting in Python code:
https://github.com/python/cpython/blob/fc21c7f7a731d64f7e4f0e82469f78fa9c104bbd/Lib/gzip.py#L622
I also found an example on how to handle concatenated gzip streams in C from Mark Adler himself:
https://stackoverflow.com/questions/17820664/is-this-a-bug-in-this-gzip-inflate-method/17822217#17822217

I have a straightforward fix for this but it will take a while for me to convince myself that it always works.

authors: a-siva brianquinlan devoncarew lrhn meowofficial


text:
issue 4433: updated web.dart & create_test.dart

added --webdev serve to add a small caveat

Spam.

source: gharchive/pull-request

authors: anujcontractor mraleph


text:
Setting up dart on a self hosted runner using tool cache not working as expected

We are in the process of migrating from GitHub's macos-latest runners to self hosted runners running on Mac Minis. When we use the dart-lang/setup-dart action now, the first run is fine, but from the second and onwards runs we run into the following issue.
Installing Dart SDK version "2.13.3" from the stable channel on macos-x64
12Downloading https://storage.googleapis.com/dart-archive/channels/stable/release/2.13.3/sdk/dartsdk-macos-x64-release.zip...
13 % Total % Received % Xferd Average Speed Time Time Time Current
14 Dload Upload Total Spent Left Speed
15
16 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
17 0 183M 0 20786 0 0 74501 0 0:42:56 --:--:-- 0:42:56 74235
18 26 183M 26 48.2M 0 0 34.8M 0 0:00:05 0:00:01 0:00:04 34.8M
19 64 183M 64 118M 0 0 52.9M 0 0:00:03 0:00:02 0:00:01 52.8M
20100 183M 100 183M 0 0 58.6M 0 0:00:03 0:00:03 --:--:-- 58.6M
21replace /Users/runner/actions-runner/_work/_tool/dart-sdk/bin/dart? [y]es, [n]o, [A]ll, [N]one, [r]ename: NULL
22(EOF or read error, treating as "[N]one" ...)
23Error: Download failed! Please check passed arguments.
24Error: Process completed with exit code 1.

TL;DR How do we use the dart-lang/setup-dart properly, in combination with the tool cache and caching multiple dart versions. Maybe as a workaround we can just answer yes instead of NULL on the replace /Users/runner/actions-runner/_work/_tool/dart-sdk/bin/dart? question?
More detail
I see this in setup.sh
Unzipping dartsdk.zip into the RUNNER_TOOL_CACHE directory
# Download installation zip.
curl --connect-timeout 15 --retry 5 "$URL" > "${HOME}/dartsdk.zip"
unzip "${HOME}/dartsdk.zip" -d "${RUNNER_TOOL_CACHE}" > /dev/null

Then appending to the GITHUB_PATH
# Update paths.
echo "${HOME}/.pub-cache/bin" >> $GITHUB_PATH
echo "${RUNNER_TOOL_CACHE}/dart-sdk/bin" >> $GITHUB_PATH

So this action is not doing anything with versioning or checking if the requested version is already installed, like we see done in flutter for example
/Users/runner/actions-runner/_work/_tool runner$ ls -l flutter/
total 0
drwxr-xr-x  4 runner  staff   128B Jun 14 12:52 ./
drwxr-xr-x  6 runner  staff   192B Jun 14 14:26 ../
drwxr-xr-x  4 runner  staff   128B Jun 14 12:52 2.0.3-stable/
drwxr-xr-x  4 runner  staff   128B Jun 14 09:54 2.2.1-stable/

/Users/runner/actions-runner/_work/_tool runner$ ls -l dart-sdk
total 40
drwx------  10 runner  staff   320B Jun  9 13:02 ./
drwxr-xr-x   6 runner  staff   192B Jun 14 14:26 ../
-rw-r--r--   1 runner  staff   1.5K Jun  7 13:14 LICENSE
-rw-r--r--   1 runner  staff   981B Jun  7 13:14 README
drwx------  14 runner  staff   448B Jun 10 10:05 bin/
-rw-r--r--   1 runner  staff   189B Jun  9 13:02 dartdoc_options.yaml
drwxr-xr-x   9 runner  staff   288B Jun  9 13:02 include/
drwxr-xr-x  28 runner  staff   896B Jun  9 13:19 lib/
-rw-r--r--   1 runner  staff    41B Jun  9 13:02 revision
-rw-r--r--   1 runner  staff     7B Jun  9 13:02 version

Anybody else running into this?
Update: The https://github.com/cedx/setup-dart action doesn't have this issue, so reverting to that for now.

The issue seems to be related to the unzip command in this line:
https://github.com/dart-lang/setup-dart/blob/ade92c2f32c026078e6297a030ec6b7933f71950/setup.sh#L80
A possible solution would be to pass -o to disable user input and force overriding of files like described in the man page here: https://linux.die.net/man/1/unzip .

@hauketoenjes did you confirm that the -o option fixes the issue? If so, are you interested in sending a PR for that?

authors: hauketoenjes jpelgrim mit-mit

repo: dart-lang/setup-dart

license_source: bigquery


text:
Wallet node stop syncing

Hello,
After 22.06.2019 wallet node stop syncing.
Dashd version: v0.14.0.1 (from releases binary)
Machine specs:

OS: Ubuntu 18.04.2 LTS (Bionic Beaver)
CPU: Intel(R) Celeron(R) CPU J3355 @ 2.00GHz
RAM: 8Gb
Disk size: 500Gb
Disk Type (HD/SDD): HDD

debug.log attached: debug.log

Possible duplicate of #2995 , pls try the solution mentioned there i.e. reconsiderblock 00000000000000112e41e4b3afda8b233b8cc07c532d2eac5de097b68358c43e

Thank you, seems that helped (I was need to wait about a 15-60 minutes to node started syncing)

repo: dashpay/dash

license: mit


text:
feat: stage getitem calls

@lgray , this doesn't actually work, but I thought you would appreciate a glimpse of what I had in mind.

Yeah this is more or less what I did for the histograms in the end, so that makes sense. I guess I just don't see how to pull it to the end in the case of getitems.

don't see how to pull it to the end

What do you mean?
I am thinking, that the with_field case is essntially identical, and instead of queueing a specific set of things (items to get) like this, we can have a small structure of stuff to do, where there can be a couple of specific sorts, and for each a single method says how to execute. Execution happens as soon as we see an operation that doesn't map into the queue (or ._dask,._meta get accessed).

Oh - as in - starting from that entry point I don't see how to get it to a functioning implementation because my brain is occupied with other tasks. :-)

I'm sure I could see the whole way through in a more quiet moment. The initial direction makes a lot of sense though.

A couple of failures here to wrap my head around, perhaps because of mutation somewhere; but here are the timings
Post
In [1]: import dask_awkward as dak
In [2]: arr = dak.from_lists([{"a": {"b": [1, 2, 3]}}]*5)
In [3]: arr2 = arr.a.b
In [4]: %timeit arr2 = arr.a.b
85.9 µs ± 280 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)

Pre
In [1]: import dask_awkward as dak
In [2]: arr = dak.from_lists([{"a": {"b": [1, 2, 3]}}]*5)
In [3]: arr2 = arr.a.b
In [4]: %timeit arr2 = arr.a.b
215 µs ± 3.12 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

Notes:

a typetracer does not have a deterministic token in cache; am using meta.typestr
small optimization in .fields that I'm pretty sure is harmless.

Yeah I went ahead and tried it - definitely a noticeable improvement!
It leaves fancy indexing as the last thing that's taking significant time.

However, this PR has some annoying rebasing issues with #477 so I can't compose it all together pleasantly. Can't quite yet see the full picture of what is left.

@lgray , merge done, back to the same three failures as before

OK, so the problem is, that the cache also contains the output divisions, which depend on the input divisions at the time of first call. If those divisions become known, the result would be different. Interestingly, one of the couple of failing tests has this at the start:
def test_single_int(daa: dak.Array, caa: ak.Array) -> None:
    daa = dak.copy(daa)
    daa.eager_compute_divisions()

because it wants known divisions, but doesn't want to mutate the object held by the fixture.

Shouldn't be too hard to keep track of the state of divisions somehow as well?

I should have translated: I know what's wrong, I can fix it.

Ah, notes rather than discussion, gotcha. No problem, and cool!

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 93.16%. Comparing base (8cb8994) to head (9e39611).
Report is 29 commits behind head on main.

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #475      +/-   ##
==========================================
+ Coverage   93.06%   93.16%   +0.09%
==========================================
  Files          23       23
  Lines        3290     3322      +32
==========================================
+ Hits         3062     3095      +33
+ Misses        228      227       -1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

OK, that fixes it. @lgray , maybe a speed test would be nice - I got around the former problem by simply not caching the divisions, as I don't think this part was contributing a lot compared to making meta and layers. I could instead make the cache more complex, if necessary.
I perhaps ought to write a test looking at the contents of the cache? I'm not sure we need that if all passes and times are clearly faster.

Yes with the awkward _new patch + multifill + this we're getting all the speedup we've seen thus far.

one nitpick here:
tokenize(fn, *args, meta is not None and meta.typestr, **kwargs)
appears to be noticeable only due to the meta.typestr call (ok it's half a second but that's not small when we are down to 4 seconds). Particular str(self.type) over in awkward that this calls is costly when spammed.
Would be good savings if we can get around that.

The remaining place that may give us some time back after all these improvements appears to be:

I thought meta.typestr was faster than str(meta), which is what the previous version would do. It sounds like it doesn't matter. So question: should map_partitions be expected to produce the identical result whether or not meta is provided? If yes, it doesn't need to be in this tokenize call at all. If no, then it does, and I don't know of a faster way to get a unique identifier of it.
Also, I reckon output_divisions should probably have been in the tokenize, since that does change the nature of the layer produced.

For building the layers themselves it doesn't matter. But I'd like to ruminate on it for a bit.

Yeah I think my position is as follows:

the _meta only alters the outcome of evaluating typetracers, not graph structure
the from-uproot io layer, as an example, does not change its output keys when its columns are projected/optimized

this applies to any AwkwardInputLayer

likewise when mocking / optimizing we don't change the keys of layers based on the meta

but we do generate the key based on the meta which is inconsistent with typical meaning

similarly, in dask.array.Array the meta not tokenized after checking in a few expensive algorithms, as well as dask.array.Array

Therefore I agree with not tokenizing the meta.

Done. This is final once green, unless we can think of some test that might help.

Furthermore if a user is trying to manually overwrite keys they'll probably have found the cache in the first place and can manipulate it as they need to.

Agreed, I think the mapping of collection name to graph/meta is natural. I don't even think there's any particular documentation that should go with this, except that maybe the cache size should be configurable? That doesn't need to happen yet.

I'd motion for going ahead and merging this today and getting a release out, then a bunch of wheels can turn on the coffea side of things.

@martindurant can I go ahead and merge/release? You tend to do the honors on these PRs, but I'm happy to turn the cranks.

Go ahead

authors: codecov-commenter lgray martindurant

repo: dask-contrib/dask-awkward


text:
[BUG][GPU Logic Bug] "SELECT ()||(<column(decimal)>) FROM " brings Error

What happened:
"SELECT (<string>)||(<column(decimal)>) FROM <table>" brings different results, when using CPU and GPU.
What you expected to happen:
It is the same result, when using CPU and GPU.
Minimal Complete Verifiable Example:
import pandas as pd
import dask.dataframe as dd
from dask_sql import Context

c = Context()

df = pd.DataFrame({
    'c0': [0.5113391810437729]
})
t1 = dd.from_pandas(df, npartitions=1)

c.create_table('t1', t1, gpu=False)
c.create_table('t1_gpu', t1, gpu=True)

print('CPU Result:')
result1= c.sql("SELECT ('A')||(t1.c0) FROM t1").compute()
print(result1)

print('GPU Result:')
result2= c.sql("SELECT ('A')||(t1_gpu.c0) FROM t1_gpu").compute()
print(result2)

Result:
CPU Result:
    Utf8("A") || t1.c0
0  A0.5113391810437729
GPU Result:
  Utf8("A") || t1_gpu.c0
0           A0.511339181

Anything else we need to know?:
Environment:

dask-sql version: 2023.6.0
Python version: Python 3.10.11
Operating System: Ubuntu22.04
Install method (conda, pip, source): Docker deploy by https://hub.docker.com/layers/rapidsai/rapidsai-dev/23.06-cuda11.8-devel-ubuntu22.04-py3.10/images/sha256-cfbb61fdf7227b090a435a2e758114f3f1c31872ed8dbd96e5e564bb5fd184a7?context=explore

Trying out your reproducer with latest main gives me an error 😕 looks like at some point between now and 2023.6.0 our logical plan has changed such that we skip the casting of the non-string column:
# 2023.6.0
Projection: Utf8("A") || CAST(t1.c0 AS Utf8)
  TableScan: t1 projection=[c0]

# main
Projection: Utf8("A") || t1.c0
  TableScan: t1 projection=[c0]

Leading to errors in the binary operation; cc @jdye64 if you have any capacity to look into this. As for the original issue, it seems like that generally comes down to difference in the behavior of cast operations on CPU/GPU, as the following shows the same issue:
print('CPU Result:')
result1= c.sql("SELECT CAST(c0 AS STRING) FROM t1").compute()
print(result1)

print('GPU Result:')
result2= c.sql("SELECT CAST(c0 AS STRING) FROM t1_gpu").compute()
print(result2)

Can look into that, would you mind modifying your issue description / title to reflect this?

Dask-sql version 2024.3.0 has fixed it.

authors: charlesbluca griffith-maker qwebug

repo: dask-contrib/dask-sql

license: MIT


text:
Add a 'decision_function()' method to the 'LogisticRegression' class.

I noticed that the dask_ml.linear_model.LogisticRegression class lacked a decision_function() method like is implemented in the corresponding scikit-learn API.
This PR adds a decision_function() method and updates one corresponding test.

Thanks. The CI failures are known and unrelated.

authors: TomAugspurger wfondrie

repo: dask/dask-ml


text:
dask-mpi not working

I have this script dask_mpi_test.py:
from dask_mpi import initialize
initialize()

from distributed import Client
import dask
client = Client()

df = dask.datasets.timeseries()

print(df.groupby(['time', 'name']).mean().compute())
print(client)

When I try to run this script with:
mpirun -np 4 python dask_mpi_test.py

I get these errors:
~/workdir $ mpirun -np 4 python dask_mpi_test.py
distributed.scheduler - INFO - Clear task state
distributed.scheduler - INFO -   Scheduler at: tcp://xxxxxx:8786
distributed.scheduler - INFO -       bokeh at:                     :8787
distributed.worker - INFO -       Start worker at: tcp://xxxxx:44712
/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/bokeh/core.py:57: UserWarning:
Port 8789 is already in use.
Perhaps you already have a cluster running?
Hosting the diagnostics dashboard on a random port instead.
  warnings.warn('\n' + msg)
distributed.worker - INFO -       Start worker at: tcp://xxxxxx:36782
distributed.worker - INFO -          Listening to:               tcp://:44712
distributed.worker - INFO -              bokeh at:                      :8789
distributed.worker - INFO -          Listening to:               tcp://:36782
distributed.worker - INFO - Waiting to connect to: tcp://xxxxxx:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -              bokeh at:                     :43876
distributed.worker - INFO - Waiting to connect to: tcp://xxxxx:8786
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -               Threads:                          1
distributed.worker - INFO -                Memory:                    3.76 GB
distributed.worker - INFO -                Memory:                    3.76 GB
distributed.worker - INFO -       Local Directory: /gpfs/fs1/scratch/abanihi/worker-uoz0vtci
distributed.worker - INFO -       Local Directory: /gpfs/fs1/scratch/abanihi/worker-bb0u_737
distributed.worker - INFO - -------------------------------------------------
distributed.worker - INFO - -------------------------------------------------
Traceback (most recent call last):
  File "dask_mpi_test.py", line 6, in <module>
    client = Client()
  File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/client.py", line 640, in __init__
    self.start(timeout=timeout)
  File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/client.py", line 763, in start
    sync(self.loop, self._start, **kwargs)
  File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/utils.py", line 321, in sync
    six.reraise(*error[0])
  File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/six.py", line 693, in reraise
    raise value
  File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/utils.py", line 306, in f
    result[0] = yield future
  File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/client.py", line 851, in _start
    yield self._ensure_connected(timeout=timeout)
  File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/tornado/gen.py", line 1133, in run
    value = future.result()
  File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/tornado/gen.py", line 1141, in run
    yielded = self.gen.throw(*exc_info)
  File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/distributed/client.py", line 892, in _ensure_connected
    self._update_scheduler_info())
  File "/glade/work/abanihi/softwares/miniconda3/envs/analysis/lib/python3.7/site-packages/tornado/gen.py", line 1133, in run
    value = future.result()
tornado.util.Timeout

$ conda list dask
# packages in environment at /glade/work/abanihi/softwares/miniconda3/envs/analysis:
#
# Name                    Version                   Build  Channel
dask                      1.2.0                      py_0    conda-forge
dask-core                 1.2.0                      py_0    conda-forge
dask-jobqueue             0.4.1+28.g5826abe          pypi_0    pypi
dask-labextension         0.3.3                    pypi_0    pypi
dask-mpi                  1.0.2                    py37_0    conda-forge

$ conda list tornado
# packages in environment at /glade/work/abanihi/softwares/miniconda3/envs/analysis:
#
# Name                    Version                   Build  Channel
tornado                   5.1.1           py37h14c3975_1000    conda-forge
$ conda list distributed
# packages in environment at /glade/work/abanihi/softwares/miniconda3/envs/analysis:
#
# Name                    Version                   Build  Channel
distributed               1.27.0                   py37_0    conda-forge

Is anyone aware of anything that must have happened in an update to dask or distributed to cause dask-mpi to break?
Ccing @kmpaul

The last CircleCI tests ran with dask=1.1.0 and distributed=1.25.2.  However, I've tried to reproduce the same environment as was run in the last CircleCI test, and it fails on my laptop.  ...Yet, rerunning the CircleCI test worked fine.

I can reproduce this with the following environment on macOS.
I am running
mpirun dask-mpi --scheduler-file my_scheduler.json --nthreads 1
python -c "from distributed import Client; c = Client(scheduler_file='my_scheduler.json')"
I see this issue with:
dask                      1.2.0                      py_0    conda-forge
dask-core                 1.2.0                      py_0    conda-forge
dask-mpi                  1.0.2                    py36_0    conda-forge
distributed               1.28.1                   py36_0    conda-forge
tornado                   6.0.2            py36h01d97ff_0    conda-forge

and (downgraded dask)
dask                      1.1.5                      py_0    conda-forge
dask-core                 1.1.5                      py_0    conda-forge
dask-mpi                  1.0.2                    py36_0    conda-forge
distributed               1.28.1                   py36_0    conda-forge
tornado                   6.0.2            py36h01d97ff_0    conda-forge

and (downgraded distributed to 1.27.1)
dask                      1.2.0                      py_0    conda-forge
dask-core                 1.2.0                      py_0    conda-forge
dask-mpi                  1.0.2                    py36_0    conda-forge
distributed               1.27.1                   py36_0    conda-forge
tornado                   6.0.2            py36h01d97ff_0    conda-forge

and
dask                      1.1.5                      py_0    conda-forge
dask-core                 1.1.5                      py_0    conda-forge
dask-mpi                  1.0.2                    py36_0    conda-forge
distributed               1.26.1                   py36_0    conda-forge
tornado                   6.0.2            py36h01d97ff_0    conda-forge

and
dask                      1.1.1                      py_0    conda-forge
dask-core                 1.1.1                      py_0    conda-forge
dask-mpi                  1.0.2                    py36_0    conda-forge
distributed               1.25.3                   py36_0    conda-forge
tornado                   6.0.2            py36h01d97ff_0    conda-forge

however, the following works!
dask                      0.20.2                     py_0    conda-forge
dask-core                 0.20.2                     py_0    conda-forge
dask-mpi                  1.0.2                    py36_0    conda-forge
distributed               1.24.2                py36_1000    conda-forge
tornado                   6.0.2            py36h01d97ff_0    conda-forge

Downgrading distributed below 1.25 to 1.24 and dask to 0.20 (below 1.0) seems to work. Since they are coupled, I'm not sure where the issue is, but it's clearly upstream of dask-mpi.

I had the same timeout problem.
I was able to run my job while using dask-scheduler instead of dask-mpi to create the scheduler.
After some search it appears that the main difference of the dask-scheduler cli is that it's using the current tornado IoLoop : https://github.com/dask/distributed/blob/1.28.1/distributed/cli/dask_scheduler.py#L197
Using current instead of a new instance here : https://github.com/dask/dask-mpi/blob/master/dask_mpi/cli.py#L52 and it's running.

@Timshel, @bocklund, thank you for chiming in. I am going to take a stab at a fix.
Moving forward, we may need to extend our testing environment to test different combinations of dask and distributed versions (or at least make sure that everything works with the latest versions).

I am getting the same problem with the latest versions of dask and distributed and running the example from the docs.
This is blocking https://github.com/basnijholt/adaptive-scheduler/pull/11.

Fixed with #33.  Thank you @Timshel for the tip.

authors: Timshel andersy005 basnijholt bocklund kmpaul

repo: dask/dask-mpi


text:
Accelerate intra-node IPC with shared memory

When implementing e.g. a data loading pipeline for machine learning with Dask, I can choose either:

threaded scheduler: Only fast, when GIL is released
forking scheduler: Only fast, when the data calcuation is very CPU intense compared to the result size.

I often face the issue that the threaded scheduler effectively uses only 150% CPU, no matter how many cores it gets, because of python code that does not parallelize.
The forking scheduler sometimes works better but only if the data loading is very CPU-intense.
Recently, I tried Ray and it could speed up some of my prediction models by 5-fold due to some reason.
I'm not 100% up to date with the latest development in Dask, but AFAIK Dask serializes all data when sending it between workers. That's why I assume the huge speed difference is due to the shared-memory object store Plasma that allows zero-copy transfers of Arrow arrays from the worker to Tensorflow.
=> I'd like to share two ideas how Plasma or Ray could be helpful for Dask:

Have a shared object cache between all threads/forks in dask/cachey
Shared memory communication:
Allow producer to calculate data and consumer to read it without (de)serialization or copying

Related issues:

Investigate using plasma
Investigate UNIX domain sockets

What is the workload?
Have you tried the dask.distributed scheduler?  You can set up a system with sensible defaults by running the following:
from dask.distributed import Client

client = Client()

#  then run your normal Dask code

https://docs.dask.org/en/latest/scheduling.html#dask-distributed-local

In general a system like Plasma will be useful when you want to do a lot of random access changes to a large data structure and you have to use many processes for some reason.
In my experience, the number of cases where this is true is very low.  Unless you're doing something like a deep learning parameter server on one machine and can't use threads for some reason there is almost always a simpler solution.

When implementing e.g. a data loading pipeline for machine learning with Dask, I can choose either:

A data loading pipeline shouldn't really require any communication, and certainly not high speed random access modifications to a large data structure.  It sounds like you just want a bunch of processes (because you have code that holds the GIL) and want to minimize data movement between those processes.  The dask.distributed scheduler should have you covered there, you might want to add the threads_per_worker=1 (or 2) if you have a high core machine.

In addition to what Matt said, we have tended to keep Dask's dependencies pretty lightweight when possible. My guess is if we were to implement shared memory it would either involve multiprocessing.shared_memory (added in Python 3.8 with a backport package) or using UNIX domain sockets ( https://github.com/dask/distributed/issues/3630 ) (as noted above).
That said, if serialization is really a bottleneck for you, would suggest you take a closer look at what is being serialized. If it's not something that Dask serializes efficiently (like NumPy arrays), then it might just be you need to implement Dask serialization. If you have some simple Python classes consisting of things Dask already knows how to serialize efficiently, you might be able to just register those classes with Dask. It will then recurse through them and serialize them efficiently.
Additionally if you are Python with pickle protocol 5 support and a recent version of Dask, you can get efficient serialization with plain pickle thanks to out-of-band pickling ( https://github.com/dask/distributed/pull/3784 ). Though you would have to check and make sure you are meeting those requirements. This may also require some work on your end to ensure your objects use things that can be handled out-of-band by either wrapping them in PickleBuffers (like in the docs) or using NumPy arrays, which have builtin support.

plasma might be ideally suited for e.g. shuffling operations, https://github.com/dask/dask/issues/6164

Maybe.  We're not really bound by bandwidth there yet.  Even if we were,
the people who are concerned about performance for dataframe shuffle
operations are only really concerned when we start talking about very large
datasets, for which single-node systems wouldn't be appropriate.
On Thu, Jun 11, 2020 at 5:11 PM Dave Hirschfeld notifications@github.com
wrote:

plasma might be ideally suited for e.g. shuffling operations, #6164
https://github.com/dask/dask/issues/6164
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/6267#issuecomment-642991348, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AACKZTGABCAUEKJ6C2SYJ23RWFXCZANCNFSM4NO46HZA
.

plasma might be ideally suited for e.g. shuffling operations, #6164

Though if you have thoughts on how plasma would help in that issue, please feel free to suggest over there. I'm sure people would be interested to hear 😉

In the context of distributed you could have a plasma store per node and instead of having workers communicating data directly, have them send the data to the plasma store on the receiving node and only send the guid / unique reference directly to the worker. All workers on that node would then have access to that data (by passing around the guid) without having to copy or deserialize the data.
I think that could have pretty big performance benefits for a number of workloads. IIUC that's basically what ray does.

To illustrate the benefits of Plasma, we demonstrate an 11x speedup (on a machine with 20 physical cores) for sorting a large pandas DataFrame (one billion entries). The baseline is the built-in pandas sort function, which sorts the DataFrame in 477 seconds. To leverage multiple cores, we implement the following standard distributed sorting scheme...

Anyway, it would would be very big piece of work so, not something I could invest time in.I thought I'd mention it as an option if people are considering big changes to improve performance.

Yeah, I think that having some sort of shuffling service makes sense (this
is also what Spark does).  I'm not sure that we need all of the machinery
that comes along with Plasma though, which is a bit of a bear.  My guess is
that a system that just stores data in normal vanilla RAM on each process
would do the trick.
On Thu, Jun 11, 2020 at 5:32 PM Dave Hirschfeld notifications@github.com
wrote:

In the context of distributed you could have a plasma store per node and
instead of having workers communicating data directly, have them send the
data to the plasma store on the receiving node and only send the guid /
unique reference directly to the worker. All workers on that node would
then have access to that data (by passing around the guid) without having
to copy or deserialize the data.
I think that could have pretty big performance benefits for a number of
workloads. IIUC that's basically what ray
https://ray-project.github.io/2017/08/08/plasma-in-memory-object-store.html
does.
To illustrate the benefits of Plasma, we demonstrate an 11x speedup (on a
machine with 20 physical cores) for sorting a large pandas DataFrame (one
billion entries). The baseline is the built-in pandas sort function, which
sorts the DataFrame in 477 seconds. To leverage multiple cores, we
implement the following standard distributed sorting scheme...
Anyway, it would would be very big piece of work so, not something I could
invest time in.I thought I'd mention it as an option if people are
considering big changes to improve performance.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/6267#issuecomment-643000455, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AACKZTE7ESNRUJGQDG7DQ73RWFZSDANCNFSM4NO46HZA
.

I could totally be wrong though.  It would be great if people wanted to run
experiments here and report back.
On Thu, Jun 11, 2020 at 6:58 PM Matthew Rocklin mrocklin@gmail.com wrote:

Yeah, I think that having some sort of shuffling service makes sense (this
is also what Spark does).  I'm not sure that we need all of the machinery
that comes along with Plasma though, which is a bit of a bear.  My guess is
that a system that just stores data in normal vanilla RAM on each process
would do the trick.
On Thu, Jun 11, 2020 at 5:32 PM Dave Hirschfeld notifications@github.com
wrote:

In the context of distributed you could have a plasma store per node and
instead of having workers communicating data directly, have them send the
data to the plasma store on the receiving node and only send the guid /
unique reference directly to the worker. All workers on that node would
then have access to that data (by passing around the guid) without having
to copy or deserialize the data.
I think that could have pretty big performance benefits for a number of
workloads. IIUC that's basically what ray
https://ray-project.github.io/2017/08/08/plasma-in-memory-object-store.html
does.
To illustrate the benefits of Plasma, we demonstrate an 11x speedup (on a
machine with 20 physical cores) for sorting a large pandas DataFrame (one
billion entries). The baseline is the built-in pandas sort function, which
sorts the DataFrame in 477 seconds. To leverage multiple cores, we
implement the following standard distributed sorting scheme...
Anyway, it would would be very big piece of work so, not something I
could invest time in.I thought I'd mention it as an option if people are
considering big changes to improve performance.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/6267#issuecomment-643000455, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AACKZTE7ESNRUJGQDG7DQ73RWFZSDANCNFSM4NO46HZA
.

cc @rjzamora @madsbk (in case this is of interest)

Has there been any further discussion on the multiprocessing shared memory implementation? I also run dask on single machines with high core counts and have read-only datastructures that I want shared.

@alexis-intellegens the ray depelopers created a Dask scheduler for this called dask-on-ray.
I'd recommend you to try this one, it magically dropped my memory usage by an order of magnitue.
Note that you may need to use sth like this:
# don't do this:
dask.compute(dask_fn(large_object))
# instead do this:
large_object_ref = ray.put(large_object)
dask.compute(dask_fn(large_object_ref))

ray will automatically de-reference the object for you.

Very interesting! I'll give it a go. Thanks @Hoeze

Out of curiosity, what were to happen if I made a shared memory object (via Python 3.8 multiprocessing) and tried to access it in dask workers? I'll try it later today.

Out of curiosity, what were to happen if I made a shared memory object (via Python 3.8 multiprocessing) and tried to access it in dask workers? I'll try it later today.

That should work, they'd pickle as references to the shared memory buffer and be remapped in the receiving process (provided all your workers are running on the same machine, otherwise you'd get an error). In general I think we're unlikely to add direct shared memory support in dask itself, but users are free to make use of it in custom workloads using e.g. dask.delayed. So if you have an object you want to share between workers, you can explicitly build this into your dask computations yourself (using either multiprocessing shared_memory or  something more complicated like plasma).
As stated above, shared memory would make the most sense if you have objects that can be mapped to shared memory without copying (meaning they contain large buffers, like a numpy array) but also still hold the GIL. In practice this is rare - if you're using large buffers you also probably are doing something numeric (like numpy) in which case you release the GIL and threads work fine.
Closing.

authors: Hoeze alexis-intellegens dhirschfeld jakirkham jcrist mrocklin

repo: dask/dask


text:
Some methods of the DataFrame API Documentation are not in the summary table

What happened: In the summary table of the Dataframe API, https://docs.dask.org/en/latest/dataframe-api.html some methods are not present.
What you expected to happen: I will expect to have the following methods in the summary table,

[ ] DataFrame.abs
[ ] DataFrame.align
[ ] DataFrame.all
[ ] DataFrame.any
[ ] DataFrame.applymap
[ ] DataFrame.bfill
[ ] DataFrame.copy
[ ] DataFrame.diff
[ ] DataFrame.divide
[ ] DataFrame.eq
[ ] DataFrame.eval
[ ] DataFrame.ffill
[ ] DataFrame.first
[ ] DataFrame.ge
[ ] DataFrame.gt
[ ] DataFrame.idxmax
[ ] DataFrame.idxmin
[ ] DataFrame.info
[ ] DataFrame.isin
[ ] DataFrame.items
[ ] DataFrame.iteritems
[ ] DataFrame.last
[ ] DataFrame.le
[ ] DataFrame.lt
[ ] DataFrame.melt
[ ] DataFrame.mode
[ ] DataFrame.ne
[ ] DataFrame.nsmallest
[ ] DataFrame.pivot_table
[ ] DataFrame.resample
[ ] DataFrame.round
[ ] DataFrame.select_dtypes
[ ] DataFrame.sem
[ ] DataFrame.size
[ ] DataFrame.squeeze
[ ] DataFrame.to_html
[ ] DataFrame.to_string
[ ] DataFrame.to_timestamp

Minimal Complete Verifiable Example: For example, DataFrame.abs is not present in the summary table,

Anything else we need to know?: If I receive instructions of how can I help and add this methods in the documentation I will like to open the PR :)

Thank you @steff456 for the report!  I think you can find those name can be added to the RST file here:
https://github.com/dask/dask/blob/master/docs/source/dataframe-api.rst
A PR would be most welcome!

Thanks for the quick response! I'll create the PR shortly 👍


text:
Dask repartition / trouble with keeping bounds as returned by divisions

Hello,
Trying out and completing this example provided in the doc works.
import pandas as pd
from dask import dataframe as dd
df = pd.DataFrame(dict(a=list('aabbcc'), b=list(range(6))),index = pd.date_range(start='20100101', periods=6))
ddf = dd.from_pandas(df, npartitions=3)
ddf.divisions
ddf = ddf.repartition(partition_size="10MB")
ddf.divisions

First divisions returns
(Timestamp('2010-01-01 00:00:00', freq='D'),
 Timestamp('2010-01-03 00:00:00', freq='D'),
 Timestamp('2010-01-05 00:00:00', freq='D'),
 Timestamp('2010-01-06 00:00:00', freq='D'))

Second one returns
(Timestamp('2010-01-01 00:00:00', freq='D'),
 Timestamp('2010-01-06 00:00:00', freq='D'))

Now, trying on another example, the 2nd divisions fails this time.
from dask import dataframe as dd
import pandas as pd
import numpy as np
dti = pd.date_range(start='1/1/2018', end='1/08/2018', periods=100000)
df = pd.DataFrame(np.random.randint(100,size=(100000, 20)),columns=['A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T'], index=dti)
ddf = dd.from_pandas(df, npartitions=1)
ddf.divisions
ddf = ddf.repartition(partition_size="10MB")
ddf.divisions

First divisions returns
(Timestamp('2018-01-01 00:00:00'), Timestamp('2018-01-08 00:00:00'))

Second one returns
(None, None, None)

Please, why is that so? Is there a bug somewhere?
Only displaying ddf shows me the index appears to have been lost after the repartition.
Before.
ddf
Dask DataFrame Structure:
                   A      B      C      D      E      F      G      H      I      J      K      L      M      N      O      P      Q      R      S      T
npartitions=1
2018-01-01     int64  int64  int64  int64  int64  int64  int64  int64  int64  int64  int64  int64  int64  int64  int64  int64  int64  int64  int64  int64
2018-01-08       ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...
Dask Name: from_pandas, 1 tasks

After
Dask DataFrame Structure:
                   A      B      C      D      E      F      G      H      I      J      K      L      M      N      O      P      Q      R      S      T
npartitions=2
               int64  int64  int64  int64  int64  int64  int64  int64  int64  int64  int64  int64  int64  int64  int64  int64  int64  int64  int64  int64
   ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...
   ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...    ...
Dask Name: repartition, 6 tasks

Thanks for your help and support,
Bests,

Further comment.
Using npartitions instead of partition_size produces expected results.
from dask import dataframe as dd
import pandas as pd
import numpy as np

n_per=20*5000
dti = pd.date_range(start='1/1/2018', end='2/08/2019', periods=n_per)
col = ['A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T']
df = pd.DataFrame(np.random.randint(100,size=(n_per, len(col))),columns=col, index=dti)
ddf = dd.from_pandas(df, npartitions=1)
#ddf = ddf.repartition(partition_size="10MB")
ddf = ddf.repartition(npartitions=2)
ddf.divisions

Produces
(Timestamp('2018-01-01 00:00:00'),
 Timestamp('2018-07-21 12:00:00'),
 Timestamp('2019-02-08 00:00:00'))

Seems related to issue #6362

I think https://github.com/dask/dask/issues/6362#issuecomment-652507357 describes the behavior that you are seeing. Note that divisions does not have to be set. It is perfectly fine to have unknown divisions. If you would like them to be set you can use ddf.reset_index().set_index("index")

I think https://github.com/dask/dask/issues/6362#issuecomment-652507357 describes the behavior that you are seeing. Note that divisions does not have to be set. It is perfectly fine to have unknown divisions. If you would like them to be set you can use ddf.reset_index().set_index("index")

authors: jsignell yohplala


text:
Check and compare shapes in assert_eq

Add some checks for shapes in assert_eq. Particularly compare shapes before computing results and compare shapes between dask arrays and computed results.

Merging this soon if there are no further comments

LGTM

authors: jakirkham mrocklin


text:
Add token kwarg to da.map_blocks

Add the token kwarg to map_blocks, mirroring the token kwarg of atop. If provided, this is the prefix of the output key, but not the key itself.
Fixes #2380.

We may want to rethink these keyword names at some point. It'd be a bit of a pain to deprecate since this is public api, but the current keywords aren't the clearest (existing for historical reasons).
If I was to redo them I'd probably have key_name be for specifying the full key (name currently), and key_prefix for just the prefix (token currently). If we were to change them we'd probably want to mirror this convention in dask.dataframe and dask.bag as well.

LGTM. Thanks @jcrist.

authors: jakirkham jcrist


text:
COMPAT: Pandas 0.22.0 astype for categorical dtypes

Change in https://github.com/pandas-dev/pandas/pull/18710 caused a dask failure
when reading CSV files, as our .astype relied on the old (broken) behavior.
Closes https://github.com/dask/dask/issues/2996

All green. Since master is currently failing on this I'll merge this later today, but if I'd appreciate it if someone could take a look.

Whoops, thanks.

Thanks @TomAugspurger.

authors: TomAugspurger jakirkham


text:
Added compute method to raise error on use

[ ] Closes #8695
[ ] Tests added / passed
[ ] Passes pre-commit run --all-files

First (very) naive implementation of compute method

Can one of the admins verify this patch?

ok to test

It would also be great if you could add a test for this. Probably you can just add a few lines to this test: https://github.com/dask/dask/blob/2ed45454bde5a3406a0df9f492bf2917e3d15b37/dask/dataframe/tests/test_groupby.py#L103-L123

Thanks for taking this on @Dranaxel! I think this will really help people :)

authors: Dranaxel GPUtester jsignell


text:
Do not ravel array when slicing with same shape

I'm not very familiar with the array API so I might be missing something obvious. However, I encountered a troubling complex graph when doing an operation like array[array > 100]
import dask.array as da
arr = da.random.random((200,200,200), chunks=(20, 20, 20))
arr[arr > 1].dask

This is under the hood raveling the arrays before indexing. Raveling is effectively a rechunking operation which is relatively expensive, specifically for an operation that should be elemwise.
I'm wondering if I'm missing anything here or why there is a reason for this complexity

Three test cases failed. I assume I'm missing something. I'd be happy to be educated about the topic more. Maybe there is a way to get this working with minor modifications

I ran into this a bit ago, and I think I tried something like what you're doing here, but also found it didn't work.
I vaguely recall it had to do with the order of the elements not matching with NumPy if you just do it blockwise on multidimensional arrays. Kind of like this warning mentions: https://github.com/dask/dask/blob/834a19eaeb6a5d756ca4ea90b56ca9ac943cb051/dask/array/slicing.py#L1149-L1152
Because x[x > 100] produces a 1D array when x is N-D, if you do the operation elemwise, each chunk will be flattened. But if you just concatenate all those 1D arrays, the elements will not be overall row-major order like you'd get from NumPy. If the chunks are squares, say, chunk 0 will contain elements from multiple rows, then chunk 1 will contain elements from multiple rows. You'd expect all the elements from row 0 to come before elements in row 1.
So it kind of makes sense that rechunking is involved, since there isn't a 1:1 mapping between chunks in the input and chunks in the output.


text:
Implementing Gofer suite of components as a third-party

Hi all,
I am a graduate student at RPI who is evaluating the gofer suite of auto-grading components as a possible architecture for implementing grading of Jupyter notebooks within courses at RPI.
I see this is a new project that hasn't been matured for use by third parties and I wanted to reach out to see if you can help me get this working as a third-party and in return I am willing to help mature the project (documentation, refactoring for generalization, etc.)
In looking through the code I have been able to update the gofer_nb.py script to work on my system, but I hit a wall when trying to figure out how to construct the docker container invoked by grade_lab.py.
Is there any existing documentation on how this docker image is created and how it can be customized for different courses and or labs?
Thanks,
Stephan

@yuvipanda

you probably want to ask @vipasu, who has been doing most of the work on the gofer grader + service!

you probably want to ask @vipasu, who has been doing most of the work on the gofer grader + service!

@zednis I created a sample directory with a sample Dockerfile and 3 notebooks with various levels of correctness. Check it out and maybe it will be able to clarify some things.  The binder directory has the docker file.  https://github.com/RPI-DATA/submitty

@zednis I created a sample directory with a sample Dockerfile and 3 notebooks with various levels of correctness. Check it out and maybe it will be able to clarify some things.  The binder directory has the docker file.  https://github.com/RPI-DATA/submitty

Hi! Sorry for joining the party late, here. We actually have an public dockerfile here: https://github.com/data-8/materials-x18/blob/master/Dockerfile
As you can see, it's quite minimal. Apart from listing your packages, there is also a line that copies the tests/etc. for the course (contained in the repo) into the docker image. Because we have all of the assignments in the directory, we only need a single image rather than one per assignment (though this would also be a reasonable approach). Building currently happens manually since it shouldn't need to be rebuilt, but if assignments are changing frequently, then it might be worth automating the rebuild procedure.
Let me know if this helps and if you have additional questions!

Hi @zednis, hi everyone,
Did you get to implement the service for your courses at RPI?
We at Leuphana are trying to achieve something in the same direction. I found gofer_service and gofer_submit and thought they sound perfect for integrating them into our JupyterHub deployment.

... and in return I am willing to help mature the project (documentation, refactoring for generalization, etc.)

I would also be happy to contribute in this regard if there are intentions to further develop this extension

authors: choldgraf franasa jkuruzovich vipasu zednis


text:
Downloader nodes

Updates
Downloader nodes
It’s now possible to create nodes which will download resulting data in the browser.
Implementation details
DownloaderNode
Simple class that extends default Node type adding one extra property — downloadData, which is supposed to be filled by Node on run execution.
DownloadData
Class that holds:

data
mimeType of that data
fileName
fileExtension

This class have pretty generic download method that can be reused among most common cases, so all required work for every specific down-loader node is to specify right data, mimeType and fileName, everything about the downloading will be handled by DownloadData.
Class also supports generics so it’s possible to specify type of the Data downloaded, for now ti just any
this.downloadData = new DownloadData<any>({
    data: [],
    mimeType: 'application/json',
    fileName: fileName,
    fileExtension: 'json',
});

Diagram run
To check what nodes are supposed to run with Diagram run method, we are simply doing a couple of checks
if (
    // Check of whether node is downloader
    node instanceof DownloaderNode &&
    // Check of whether code runs in browser
    // environment
    (isBrowser || isJsDom)
) {
    await node.downloadData.download();
}

DownloadJSON node
Can be ran with dot-notated paths, so if we for example specify title as an attribute to download, we will get only those attributes from all features. Filename will be in such format
[node_name] [date_downloaded].json

Example of usage

Things need to be added/fixed/considered
Tests
Find a good way of testing downloader nodes. Possibly it might be testing of right DownloadData creation for the nodes in the core, and e2e tests for the download functionality in the gui
Downloading in headless mode
For now downloading functionality works only when code runs in the browser environment, it may be good feature to add separate downloading cases for both browser and node environment. So in node environment download will save data to some cross-platform data folder like dataDir/data-story/filename_date.json

Possibly, method DownloadData.download can be implemented in gui as a callback or similar instead since it is so heavily involved with browser (document, createElement etc) ? I see we have a guard clause to see if it is running in browser environment, but anyways might be a good separation to make

That's how I started implementing it at first, but that approach will require extra actions looping through the node list and applying callback on the gui side, so I decided to do all things in one place with just environment checking.

gui/"file-saver": "^2.0.5", - this can be removed. Nice to implement this without a package

Yes, current solution works pretty nice, though it may be a good idea to take a look on StreamSaver, which can handle bigger files creation asynchronously using streams directly to the file-system

Add pretty print option on json

I think that this must be an option in form of select parameter in DownloadJSON node, so user can choose whether he want's to format downloaded data or not.

When handling dot notation, we can make use of Obj.get helper

Yeah, it handles just what I've done manually, feels like nice possibility to decrease code verbosity

Might be another reason to move the actual downloading part to gui?

The problem here is by moving download function to the gui we'll still have to create the same testing workflow as with download isolated at the DataStory class in the core. So still e2e tests of downloading functionality in gui and  tests of right data creation in core.

Updates
It's now possible to specify multiple attributes which then will be downloaded, so for example if we had such a config for a DownloadJSON node

We would get all attributes we specified downloaded

That's how I started implementing it at first, but that approach will require extra actions looping through the node list and applying callback on the gui side, so I decided to do all things in one place with just environment checking.

Ok that makes sense, lets keep it in core 👍

authors: Lenivaya ajthinking

repo: data-story-org/core


text:
ModuleNotFoundError: No module named 'databricks' when using apply_batch or apply_transform

Hi,
I'm using Spark in client mode and I've gotten Koalas working, but the apply_batch method seems to indicate that koalas is missing from the executor nodes. It it really so that koalas must be explicitly installed to worker nodes? Or is it another issue / something simple I'm missing? Spark version: 2.4.3, Koalas version: 1.2.0.
Example:
  kdf = ks.DataFrame({'a': range(0, 20000), 'i': range(0, 20000)}).set_index('i')
  # --> works
  kdf.head(10)
  # --> also works

  def test_apply(df):
    return df
  kdf.koalas.apply_batch(test_apply)
  # --> fails, see error below

Error:
...
  File "/var/lib/mesos/slaves/ad6bc800-ab3b-486e-bfa2-cf24ca7aebae-S1/frameworks/7461c35c-4cf7-47a5-ae69-3ba9362cee61-71216/executors/1/runs/71cf1309-75b9-4ac2-b14e-3abc04506810/spark-2.4.3-bin-datalake-hadoop-2.9.2-1/python/lib/pyspark.zip/pyspark/serializers.py", line 580, in loads
    return pickle.loads(obj, encoding=encoding)
ModuleNotFoundError: No module named 'databricks'

	at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.handlePythonException(PythonRunner.scala:452)
	at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:172)
	at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:122)
	at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:406)
	at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
	at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:255)
	at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:247)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$24.apply(RDD.scala:836)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:121)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Hi @maxpagels, yes, Koalas must be explicitly installed to worker nodes, as well as pandas, PyArrow, and numpy.

authors: maxpagels ueshin

repo: databricks/koalas