text:
Customer-reported issue with NNBD migration tool: Could not apply migration

Could not apply migration
Error: Bad state: Cannot apply migration. Files on disk do not match the expected pre-migration state. Press the "rerun from sources" button and then try again. (Changed file path is D:\Flutter\MyHaUI-Mobile\lib\category\danhGiaRenLuyen\8_danh_gia_ren_luyen_screen.dart)
Please fill in the following:
Name of package being migrated (if public):
What I was doing when this issue occurred:
Is it possible to work around this issue:
Has this issue happened before, and if so, how often:
Dart SDK version: 2.12.0
Additional details:
Thanks for filing!
Stacktrace: auto populated by migration preview tool.
#0      PreviewSite.performApply (package:nnbd_migration/src/preview/preview_site.dart:502:9)
#1      PreviewSite.handlePostRequest (package:nnbd_migration/src/preview/preview_site.dart:449:9)
<asynchronous suspension>
#2      HttpPreviewServer._handlePostRequest (package:nnbd_migration/src/preview/http_preview_server.dart:109:5)
<asynchronous suspension>
#3      HttpPreviewServer._handleServer.<anonymous closure> (package:nnbd_migration/src/preview/http_preview_server.dart:119:9)
<asynchronous suspension>

Hi @chituanh, thanks for the bug report!
To help us diagnose the problem, could you answer a few questions for us?

Is there anything in particular that is special about the file D:\Flutter\MyHaUI-Mobile\lib\category\danhGiaRenLuyen\8_danh_gia_ren_luyen_screen.dart compared to other files in your project?  (For example, is it the result of code generation, is it a file you modified recently, does it contain unusual characters or use unusual line endings, etc.)
What were the last few things you did prior to attempting to apply the migration?
Does this seem to happen every time you run the migration tool, or just once?
Have you found any workaround?

Thanks!

As of 1c7fe7196dd7a542f7f91cd6ea2ac81e1a13146c, the null safety migration tool has been removed from active development and retired. No further work on the tool is planned.
If you still need help, or you believe this issue has been closed in error, please feel free to reopen.


text:
ARGUMENT_TYPE_NOT_ASSIGNABLE although preceded by a not-null condition

For the below case, there is a compiler error

The argument type 'TypeAnnotation?' can't be assigned to the parameter type 'AstNode'

I believe this shouldn't happen, since there is a check for returnType != null, also the error is removed if the returnType is declared as
var returnType = function?.returnType;

void fun(FunctionDeclaration? function, ChangeBuilder builder, String file) async {
  TypeAnnotation? returnType;
  returnType = function?.returnType;
  // var returnType = function?.returnType; // this doesn't give an error

  if (returnType != null) {
    await builder.addDartFileEdit(file, (builder) {
      builder.addReplacement(range.node(returnType), (builder) {
        //                              ^^^^^^^^^^ error is here
      });
    });
  }
}

Dart SDK version: 2.15.0-6.0.dev (dev) (Wed Aug 11 14:48:30 2021 -0700) on "windows_x64"

Since returnType is captured by a closure, it gets much harder for the compiler to ensure that the execuction really happens after the test, and not after any other change to returnType.
If there is any assignment to a variable inside any closure, all promotion of that variable is turned off. If a variable occurs in a closure, that occurrence may or may not have promotion turned off depending on how clever the compiler manages to be.
In this case, it should be technically possible to retain the promotion (the closure is created after the last assignment to returnType in the function's control flow, and returnType is promoted to non-null at the time), but there are limits to how clever the compiler can be and how far ahead it can look to see that nothing bad happens. Even in cases that look very clear-cut, the edges of those cases might not always be as clearly delimited.
@stereotype441 should know whether this is intended or accidental.
I'd recommend working around this by creating a new variable:
  if (returnType != null) {
    var nonNullReturnType = returnType;
    .....
             range.node(nonNullReturnType);

Yes, this is a known area where flow analysis could be improved.  The primary place we're tracking it is here: https://github.com/dart-lang/language/issues/1536.

source: gharchive/issue

authors: asashour lrhn stereotype441

repo: dart-lang/sdk

license_type: permissive

license_source: github-api


text:
[analyzer] todo and fixme ignored by analyzer

I want to have the dart static analyzer show a warning if there are open TODO items. I have configured the analysis_options.yaml accordingly:
include: package:lints/recommended.yaml
analyzer:
  errors:
    todo: warning

Now VSCode does highlight the todos as warnings and they do appear in the Problems panel. However, if I run dart analyze they don't show up. Same thing with FIXME (and appropriate analysis_options.yaml). I could not find any documentation, that this is on purpose and I think these errors should be configurable the same way as other errors.
I tested this on a fresh dart project and I'm using Dart SDK version: 2.16.2 (stable) (Tue Mar 22 13:15:13 2022 +0100) on "macos_x64" (Apple Silicon).

@bwilkerson do you know any history here? Probably not a hard ask to fulfill, esp. if the severity has been increased to warning or error.

That logic is here: https://github.com/dart-lang/sdk/blob/main/pkg/analysis_server/lib/src/lsp/lsp_analysis_server.dart#L1021.
@DanTup Does it seem reasonable to also report the error if the severity is greater than ErrorSeverity.INFO?

Does it seem reasonable to also report the error if the severity is greater than ErrorSeverity.INFO?

Yep, I think if a user has configured them to be > INFO we should always show them regardless of the setting.
I'll change that in the code above, although I believe the issue described above is for dart analyze where the issues are being filtered out by the analyze command in dartdev:
https://github.com/dart-lang/sdk/blob/d10f740398c6ee512aaad1ca06b38319a216b7c5/pkg/dartdev/lib/src/commands/analyze.dart#L149-L153
Presumably the same logic is valid there, so I've opened a change for that here:
https://dart-review.googlesource.com/c/sdk/+/245542/

The LSP change for the same logic (always show TODOs if configured as warnings/errors without requiring the setting) is here:
https://dart-review.googlesource.com/c/sdk/+/245543/
I guess the only outstanding question is:

Should it be possible to have dart analyze report TODOs when they're INFO?

If so, that might need a new flag. If reporting them only when set to WARNING/ERROR is good enough, then I think the changes above cover that.

Unless we have users that want the feature, my opinion is that we shouldn't add the feature.

authors: DanTup bwilkerson lowski srawlins


text:
Analyzer can report invalid_null_aware_operator and follow-on errors from the resulting nullable type

See this code:
bool f(int i) => i?.isEven;

Analyzer produces one error:

A value of type 'bool?' can't be returned from the function 'f' because it has a return type of 'bool'.

and one warning:

The receiver can't be null, so the null-aware operator '?.' is unnecessary.

This is a bummer, because the "real" issue is the warning, and the error is sort of secondary; the error is indirectly solved by directly solving the warning.
Can we avoid the double reporting? Should we? Given that these are both specified by the null safety spec (I think), must we report each? In other words, are there situations/users/codebases where the warning may be suppressed or ignored, and so we must continue to report the error? It might be very complicated to detect that invalid_null_aware_operator has been reported (and not suppressed) in order to not report argument_type_not_assignable (or other similar assignability errors).
CC @leafpetersen @eernstg for any comments from the language team.

Oh and this is motivated by our internal build system, which groups errors on the same line into one report. E.g.:
291: foo.bar?.type,                    The argument type 'SomethingPB_Enum?' can't be assigned to the parameter type
                                       'SomethingPB_Enum'. (warning) The receiver can't be null, so the null-aware operator '?.' is
                                       unnecessary. #argument_type_not_assignable #invalid_null_aware_operator

Note that the more useful diagnostic is reported second. Until I've read the second diagnostic, it looks to me like a real puzzle to get a non-null value from an expression with a null-aware property access.

Baring any objections from the language team, I would say that we ought to recover from the warning by not reporting the error (treat a null-aware operator on a non-nullable target as if the non-null-aware version of the operator had been used). I think that's what's best for users because, as you said, it points them to the real issue.

I agree that the diagnostic about ?. being unnecessary addresses the real issue directly, but I tend to think about the other issue ("can't return a bool?") as an independent consequence. We could have any number of diagnostics based on the fact that an expression has an unintended/suboptimal type (with var x = e?.foo();, many usages of x could fail in various ways), and there is no special connection between "unnecessary ?." and "not assignable" as such. So we're probably looking for something which is a bit more general.
In particular, I don't think it is a sufficiently general strategy to treat an unnecessary ?. as if it had been a plain .. Another danger with that approach is that it might be confusing: If the developer corrects the non-nullable receiver expression to something nullable (so the ?. wasn't intended to be unnecessary, we just had a typo in the receiver) then the "not assignable" error would emerge out of thin air. And how would we handle the situation where warnings are not shown at all?
The core topic here seems to be dependencies among diagnostic messages.
We do eliminate most error dependencies today, simply by not reporting errors with dependencies:
void main() {
  UnknownClass u = 'A string'; // No diagnostics on the initializing expression.
  u.toString(42); // No diagnostics here.
}

We get an error on the usage of an undefined name as a type, but no error on the initialization, and no error on the subsequent use of the variable (and it's not just treated as var u = 'A string';). I can't immediately find a counterexample to the assumption that u is treated as if it had the type dynamic, but it could also be a special ErrorType (which is treated similarly to dynamic, such that we don't report any errors/warnings).
If we had been discussing the relationship between two errors here, we'd simply get rid of the dependent error by letting i?.isEven have the type ErrorType. We do not test diagnostic dependencies in language tests, and we don't specify ErrorType and how to handle it (or anything like that). So there would not have been anything new or controversial in letting i?.isEven have the type ErrorType if the use of ?. had been an error.
However, the use of ?. in this situation is a warning, and, presumably, we don't want to "let a warning cancel an error", and we still have to define how to deal with the situation where warnings are suppressed.
Maybe we could prioritize the diagnostic messages in this situation? Show independent diagnostic messages before dependent ones (in IDEs and on the command line)? Include some kind of a link from the dependent ones to their dependencies?

I can't immediately find a counterexample to the assumption that u is treated as if it had the type dynamic, but it could also be a special ErrorType (which is treated similarly to dynamic, such that we don't report any errors/warnings).

This is correct; we just set the static type, in analyzer, to dynamic. This is a pain point in some circumstances, like in code gen, when the code generator would like to know that the code had an error, rather than finding an object of type dynamic.

However, the use of ?. in this situation is a warning, and, presumably, we don't want to "let a warning cancel an error", and we still have to define how to deal with the situation where warnings are suppressed.

Yeah this was my main worry.

Maybe we could prioritize the diagnostic messages in this situation? Show independent diagnostic messages before dependent ones (in IDEs and on the command line)? Include some kind of a link from the dependent ones to their dependencies?

Interesting ideas. @bwilkerson would know more about what we can do in IDEs, which is the primary location where we show errors. And what is expected etc. We may not have control over the order that errors appear in the Problems panel. The case for the CLI and our internal builds are probably enough under our control that we could order issues on the same line intentionally.

In particular, I don't think it is a sufficiently general strategy to treat an unnecessary ?. as if it had been a plain ..

I agree that consistency is important, both in language design and in the UX. One of the recovery techniques that we consistently apply is to silence follow-on diagnostics by treating the code as if it had been written correctly. I see treating ?. as if it had been a . to be just another application of the more general technique.

Another danger with that approach is that it might be confusing: If the developer corrects the non-nullable receiver expression to something nullable (so the ?. wasn't intended to be unnecessary, we just had a typo in the receiver) then the "not assignable" error would emerge out of thin air.

I don't think users would perceive the diagnostic as emerging out of thin air; I think it would be perceived as the result of modifying the code.

And how would we handle the situation where warnings are not shown at all?

We'd need to be a but smarter about warnings and produce the error only in cases where the warning would not be produced.

I can't immediately find a counterexample to the assumption that u is treated as if it had the type dynamic, but it could also be a special ErrorType (which is treated similarly to dynamic, such that we don't report any errors/warnings).

This is correct; we just set the static type, in analyzer, to dynamic. This is a pain point in some circumstances, like in code gen, when the code generator would like to know that the code had an error, rather than finding an object of type dynamic.

I have no conceptual issue with defining an ErrorType, but it might be the case that it would need to be understood by type inference, which might have an impact on shared code, and hence the CFE. I don't know that that's the case, but I'd want us to investigate the possible impact of such a change before making it.

... presumably, we don't want to "let a warning cancel an error" ...

I agree that that's a concern, and possibly a bit enough concern to require a different solution to this issue.

Maybe we could prioritize the diagnostic messages in this situation? Show independent diagnostic messages before dependent ones (in IDEs and on the command line)? Include some kind of a link from the dependent ones to their dependencies?

We don't really have any formal definition of 'dependent' diagnostics, nor any representation of such a concept in the implementation. What we do have available today is the severity of the diagnostic, and it would certainly be reasonable, in places where we can control the presentation order, to display higher severity ('error') diagnostics before lower severity ('warning') diagnostics.

We may not have control over the order that errors appear in the Problems panel.

That is correct. IDEs typically provide the user with some level of control over the presentation, which means that we're pretty much limited to controlling the data that is presented.

I'll offer an alternative solution. I haven't investigated it yet, but it's possible that we could extend our analysis when we detect the error (return_of_invalid_type and possibly others) to notice that the two types differ only by nullability suffix and that the nullability was introduced by the ?. operator. Under those conditions we could produce a message (some combination of problem, correction, or context) that suggests that the operator might be the underlying problem. We could then suppress the warning (possibly by leaving a breadcrumb when producing the error) without loosing any information (from the user's perspective).

authors: bwilkerson eernstg srawlins


text:
Map.updateAll() fails with runtime type error when it should not

Running this code, which as far as I can tell is 100% typesafe:
main() {
  print(_resolveMap({'foo': 'bar', 'other': {'a': 'b'}}));
}

Map<String, Object?> _resolveMap(Map<String, Object?> map) {
  map.updateAll((key, value) => (value is Map<String, Object?>) ? _resolveMap(value) : value);
  return map;
}

Results in a type error at runtime:
Error: TypeError: Closure '_resolveMap_closure': type '(String, Object?) => Object?' is not a subtype of type '(String, String) => String'

This happens even when I generify the function:
Map<String, T> _resolveMap<T>(
    Map<String, T> map, Map<String, dynamic> properties) {
  map.updateAll((key, value) => _resolveValue(value, properties));
  return map;
}

Which makes it impossible to implement this.

The problem is the nullable value on the Map. My generic function was checking for Map<String, Object?> and recursing in that case, but if I check for only Map<String, Object>, it does work... that's ok in my case, but feels like this is a bug because we do a check that passes (i.e. is Map<String, Object?> , which passes for any Map<String, Object>), but then it errors when the function with that type that was already checked is called.

@lrhn ?

The problem is that your sub-map {'a': 'b'} has type Map<String, String>. Its updateAll method expects a function of type String Function(String, String). You give it a function of type Object? Function(String, Object?), which is not a subtype of String Function(String, String), because the return type is not a subtype of String.
At runtime, it will probably only return existing values of the map, which are all strings, but you can't see that from the function type. It could return new Object() or null instead, for all the type checking knows.
In your test, you check whether value is Map<String, Object?>, which it is, even if it's really a Map<String, String>, because Dart generics are (unsafely) covariant. You are allowed to up-cast type parameters to a supertype, but you can't necessarily use all the members safely then.
The unsafety comes from the type parameter being allowed to occur contravariantly in member signatures, and that happens here.
The classical example is List<Object> list = <int>[1]; list.add("not an int");. The static type allows this because the signature of add is void add(E value), and statically we think it's a List<Object>. If we try to add a string to a List<int>, it needs to throw at runtime to prevent being unsound.
For maps, consider:
Map<String, Object?> map = <String, String>{"a": "b"};
map.updateAll((key, value) => value);

The function (key, value) => value occurs in a static context expecting an Object? Function(String, Object?), so the types are inferred as (String key, Object? value) => value. That's a valid argument to Map<String, Object?>.updateAll, so the compiler allows it.
It's not actually a valid argument to Map<String, String>.updateAll, because it doesn't return a String. Our contravariant-type parameter check in updateAll throws seeing the Object? Function(String, Object?).
There is no good way to achieve what you want without knowing the value type of the map, because the function type V Function(K, V) uses V both covariantly and contravariantly. You have to hit it exactly.
You can widen the accepted values to V Function(K, Object?), but you still need to return a V, and provide a function typed as returning a V. Which you only can if you know the precise type, or you return Never, which isn't useful.
Generics don't work because they get their parameters from the static type anyway.
The change to nullability looks like a red herring. It shouldn't change anything.
I don't have a good solution, other than not using updateAll. If you use Map.map instead, to create a new map, you don't have to satisfy the runtime type of the old map, just the static type of it.

authors: kevmoo lrhn renatoathaydes


text:
[ddc] Cannot staticInterop with subclass of Error

Say we have an object that extends JS's Error, similar to this one (typescript for brevity, Error is this):
export interface MyCustomError extends Error {
  customProp: String;
}

If I attempt to JS-Interop with code similar to this:
/// Error object
@JS('Error')
@staticInterop
abstract class DomError {}

/// Methods on the error object
extension DomErrorExtension on DomError {
  /// Error message.
  external String? get message;
  /// Stack trace.
  external String? get stack;
}

/// My custom Error class.
@JS()
@staticInterop
abstract class MyCustomError extends DomError{}

/// Methods of the custom Error object.
extension MyCustomErrorExtension on MyCustomError {
  external String get customProp;
}

Whenever objects of that type flow from JS to Dart, like this:
Future<void> onError(MyCustomError? error) async {
  console.log(error);
}

I get an exception similar to the following in DDC:
Uncaught Error: Expected a value of type 'MyCustomError?', but got one of type 'NativeError'
    at Object.throw_ [as throw] (errors.dart:288:49)
    at Object.castError (errors.dart:121:3)
    at Object.cast [as as] (operations.dart:529:12)
    at dart.NullableType.new.as (types.dart:411:9)
    at Object._argumentErrors (operations.dart:216:40)
    at Object._checkAndCall (operations.dart:362:22)
    at Object.dcall (operations.dart:372:39)
    at Lp.ret [as i] (js_patch.dart:415:11)
    at Dp.G (client:368:170)
    at ac (client:85:202)
    at _.l.dispatchEvent (client:83:347)
    at _.Ge.B (client:160:179)

When compiling with dart2js, the above code works as expected, and the object is logged to the console:

Note: I'm using flutter, so DDC => flutter in debug mode and dart2js => flutter in --profile mode.
Some info about my environment
$ flutter --version
Flutter 3.7.0-14.0.pre.24 • channel master • git@github.com:ditman/flutter.git
Framework • revision a7942e80d4 (26 hours ago) • 2023-01-03 14:33:55 -0800
Engine • revision c0b3f8fcea
Tools • Dart 3.0.0 (build 3.0.0-77.0.dev) • DevTools 2.20.0

$ which dart
/usr/local/google/home/dit/github/flutter/bin/dart

$ dart --version
Dart SDK version: 3.0.0-77.0.dev (dev) (Mon Jan 2 06:21:11 2023 -0800) on "linux_x64"

Note to myself, recapping what we've talked about: This is because Error is a special class in DDC, reserved by @JsPeerInterface. This class extends Interceptor instead of JavaScriptObject. When we try to cast the error object to the @staticInterop type (which is erased to JavaScriptObject), the cast fails. The fix here is likely to change the supertype of this reserved class to JavaScriptObject, being careful that DDC doesn't have any dependencies on the direct superclass being Interceptor.

Just curious can we reproduce a similar problem with all classes marked with @JsPeerInterface or is it something special happening with Error?

Nothing special with Error, this should be true for all such classes e.g. Array (and indeed it is when I try it out).
For now, I don't think we're quite ready for users to be able to interop core types like Number or Boolean, so I'm inclined to only address Error (and implicitly, any of its subtypes like TypeError). In the future, we can provide core types when we create a usable JS type hierarchy.

YAY!

authors: ditman nshahan srujzs


text:
Downcasting generic-types crashes with an Unhandled type exception

Given the following example:
abstract class Foo {}

class Bar extends Foo {}

typedef BuilderDef<T extends Foo> = String Function(T someFoo);

class FooMatcher<T extends Foo> {
  final BuilderDef<T> builder;

  const FooMatcher({required this.builder});
}

final FooMatcher<Bar> implementation = FooMatcher<Bar>(
  builder: (Bar bar) => '<3',
);

void main() {
  final FooMatcher matcher = implementation;

  print(matcher.builder.runtimeType);
}

There are two ways to crash it. The first way is downcasting FooMatcher<Bar> to FooMatcher like you see in the first line of the main function. This crashes with the following error message when trying to access the builder field. No need to run the builder, just accessing it crashes. runtimeType is not necessary.
Unhandled exception:
type '(Bar) => String' is not a subtype of type '(Foo) => String'
#0      main (package:dart_testing_grounds/dart_testing_grounds.dart:20:17)
#1      _delayEntrypointInvocation.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:296:19)
#2      _RawReceivePort._handleMessage (dart:isolate-patch/isolate_patch.dart:189:12)

Another way to crash it is to remove the generic from the implementation's return type. So instead of
final FooMatcher<Bar> implementation = FooMatcher<Bar>(
doing this instead will crash when trying to access implementation.builder. It's another way of downcasting the generic.
final FooMatcher implementation = FooMatcher<Bar>(
Full example:
abstract class Foo {}

class Bar extends Foo {}

typedef BuilderDef<T extends Foo> = String Function(T someFoo);

class FooMatcher<T extends Foo> {
  final BuilderDef<T> builder;

  const FooMatcher({required this.builder});
}

final FooMatcher implementation = FooMatcher<Bar>(
  builder: (Bar bar) => '<3',
);

void main() {
  print(implementation.builder.runtimeType);
}

It produces the same error:
Unhandled exception:
type '(Bar) => String' is not a subtype of type '(Foo) => String'
#0      main (package:dart_testing_grounds/dart_testing_grounds.dart:18:24)
#1      _delayEntrypointInvocation.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:296:19)
#2      _RawReceivePort._handleMessage (dart:isolate-patch/isolate_patch.dart:189:12)

I'm using a Macbook Pro with Apple M1 Max.
Dart version:
Dart SDK version: 3.0.6 (stable) (Tue Jul 11 18:49:07 2023 +0000) on "macos_arm64"

This is all working at intended.
The occurrence of T in the type alias, and therefore in the field type, is contravariant.
Dart class generics are unsafely covariant. The "unsafely" comes from allowing you to write a class like this, rather than disallowing the contravariant occurrence of the class type parameter entirely.
It's still sound, which means that any time an expression can end up with a runtime value with a type that is not a subtype of the static type, the compiler has instead a check and throws to prevent unsoundness.
That's what happens here.
The static type of implementation is Builder<Foo>, because you typed it as the raw type Builder, which gets instantiated to bound.
The runtime type is Builder<Bar>, which is a subtype because of covariant generics.
The static type of implementation.builder is String Function(Foo).
The runtime type is String Function(Bar), which is not a subtype of the static type.
The compiler knew that could happen, so it checked and throws to avoid unsoundness.

@lrhn What would be the correct way to do what I'm trying to do in the code sample? I have a more complex case in which I have multiple implementations and they get downcasted when stored in lists and such. Is there a workaround or are cases like this better off made into formal classes if that would even fix it.
I'm also curious if the Dart analyzer should catch this error earlier.

For the record for others to see, a strongly typed class instead of a field implementation will not fix this issue. Here's a code sample that produces the same problem but with a class:
abstract class Foo {}

class Bar extends Foo {}

typedef BuilderDef<T extends Foo> = String Function(T someFoo);

class FooMatcher<T extends Foo> {
  final BuilderDef<T> builder;

  const FooMatcher({required this.builder});
}

// final FooMatcher<Bar> implementation = FooMatcher<Bar>(
//   builder: (Bar bar) => '<3',
// );

class Implementation extends FooMatcher<Bar> {
  Implementation() : super(builder: (Bar bar) => '<3');
}

void main() {
  final FooMatcher matcher = Implementation();

  print(matcher.builder.runtimeType);
}

The static type of implementation.builder is String Function(Foo). The runtime type is String Function(Bar), which is not a subtype of the static type.

Why is String Function(Bar) not a subtype? Bar is a subtype of Foo, therefore, it is legal and safe to downcast to Foo since Bar extends Foo. Is it because Dart cannot resolve the inheritence in function fields/parameters?

You have a Builder<T> which accepts a T and creates a String.
That means that a Builder<Bar>, which knows how to create a string from a Bar, can be cast to Builder<Foo>, a supertype, which claims to know how to handle a Foo, something the Builder<Bar> never promised. That can't just be made to work.
What you can do is to drop the class entirely, and use the function type typedef Builder<T> = String Function(T); everywhere you'd use an object.
That doesn't allow you to add extra methods to the Builder type. You may be able to do that using extension methods, but it can easily become a mess to add extra methods to any function which returns a String.
If you want a list of builders, it would be a List<String Function(Never)>, which you won't be able to call elements of without first checking whether they'd accept the argument.
Alternatively, hide the builder field, and require access to go through methods which check the input.
class Builder<T> {
  final String Function(T) _builder;
  Builder(this._builder);
  bool isValidArgument(Object? o) => o is T;
  String build(Object? input) {
    if (input is! T) throw ArgumentError.value(input, "input", "Not a $T");
    return _builder(input);
  }
  String? tryBuild(Object? input) {
    if (input is! T) return null;
    return _builder(input);
  }
}

You can't just call with any object (except tryBuild) and expect it to work.
There has to be tests (until Dart gets variance modifiers like https://github.com/dart-lang/language/issues/524).

The reason String Function(Bar) is not a subtype of String Function(Foo) is that functions parameters are contravariant.
(They vary contra, aka opposite, to the type itself.)
The point of subtyping in object orientation (and in general) is that an instance of a subtype can be used anywhere the supertype is expected.
A String Function(Bar) can be called with objects that are subtypes of Bar.
A String Function(Foo) can be called with objects that are subtypes of Foo.
Since all Bar objects are also Foo objects, the latter function can be used anywhere the former is expected, because it accepts all the same arguments. The opposite is not true, a String Function(Bar) cannot be called with a new Foo().
So String Function(Bar) is a supertype of ``String Function(Foo)`. The subtyping of the function itself varies opposite the subtyping of the paramters.

The reason String Function(Bar) is not a subtype of String Function(Foo) is that functions parameters are contravariant.

So this is a limitation with function parameters in the Dart SDK. That explains why the sample works if the T was the return type instead of the function paramter.
I have a converter that takes some object and returns our T (Foo), the T is then consumed by the code sample as a function param. Returning the T is fine, using the T as a function parameter crashes.
For visual aid:
typedef StyleBuilder<T extends RichMatch> = List<TextSpan> Function(T match);
typedef MatchBuilder<T extends RichMatch> = T Function(RegExpMatch match);

StyleBuilder crashes when accessed due to the discussed limitation with function parameter types, MatchBuilder succeeds normally since T is a returned type and is therefore not a contravariant.
Is what I said accurate?

That does sound correct.
It's not just a limitation of the Dart SDK.
Functions being contravariant in their parameters is a fundamental property of mathematical functions, and similarly for other abstraction mechanisms, like abstracting over quantifiers or predicates in higher order logic.

That's fair. Do you believe it's possible for the Dart analyzer to protect against this error ahead of time?

To address the issue of contravariance with function parameters, we can refactor the code to use typedefs without any generic constraints. Instead of using BuilderDef<T>, we'll use a simple typedef BuilderDef = String Function(Foo someFoo);. This will allow us to work with function types directly without generic constraints, ensuring safety and avoiding potential runtime errors.
Here's the refactored code with the fix:
abstract class Foo {}

class Bar extends Foo {}

typedef BuilderDef = String Function(Foo someFoo);

class FooMatcher {
  final BuilderDef builder;

  const FooMatcher({required this.builder});
}

final FooMatcher implementation = FooMatcher(
  builder: (Foo foo) => '<3', // Changed the parameter to accept Foo instead of Bar
);

void main() {
  final FooMatcher matcher = implementation;
  print(matcher.builder.runtimeType);
}

With this fix, you'll no longer encounter the runtime error related to contravariant function parameters. The code will now work as expected and provide a more consistent and safer behavior when handling function types.
If you have any more questions or need further assistance, feel free to ask!

Certainly! Let's create a new code example using a wrapper for the builder function, as suggested by @lrhn. This wrapper will handle the intrinsic pre-casting of the builder's Foo parameter to any subtype, allowing the function that reads the Foo function parameter to react to different implementations of Foo.
abstract class Foo {}

class Bar extends Foo {}

typedef BuilderDef<T extends Foo> = String Function(T someFoo);

class BuilderWrapper<T extends Foo> {
  final BuilderDef<T> _builder;

  BuilderWrapper(this._builder);

  String build(Foo foo) {
    if (foo is T) {
      return _builder(foo as T);
    } else {
      throw ArgumentError('Invalid argument type for the builder function.');
    }
  }
}

void main() {
  final BuilderWrapper<Bar> implementation = BuilderWrapper<Bar>(
    (Bar bar) => '<3',
  );

  final FooMatcher matcher = FooMatcher(builder: implementation.build);

  print(matcher.builder.runtimeType);
}

class FooMatcher {
  final BuilderDef<Foo> builder;

  const FooMatcher({required this.builder});
}

In this code, we create the BuilderWrapper class, which takes a BuilderDef<T> as a constructor parameter and provides a build method. Inside the build method, we check if the given Foo parameter is a subtype of T, and if so, we call the original _builder function with the appropriate type.
Now, when creating the FooMatcher, we pass the implementation.build method as the builder function, which ensures that the correct Foo subtype is used for the builder.
I hope this example better meets your requirements. If you have any more specific scenarios or needs, please let me know, and I'll be happy to provide further assistance!

The analyzer cannot tell you whether your program definitely has a bug. After all, the (current) language design allows the program, and if you don't up-cast, it'll work correctly.
It's the combination of a contravariant getter and up-casting which is a problem.
I guess the analyzer could try to detect that.

The contravariant getter alone is worth a warning, probably only as a lint (since it's valid, and possibly deliberate, but a dangerous design, which is what opt-in lints are for, where warnings are reserved for things with few-to-no false positives).

The up-casting of a type which has a contravariant getter is also something we could lint against. We know it exists, we know a subtype of that type is being upcast to another type argument, that's precisely when we risk the error.

We also know where we are reading the contravariant getter, and could warn there too, but that's more iffy, since the getter presumably exists for a reason, and if we can't see that the receiver has been up-cast, this is very likely to be a correct use. (Because most uses likely are, if the getter exists to begin with.) But if we have just warned about an up-cast above, we should probably also warn about accessing the member on that up-cast value.

That does sound like lint material. Maybe:
Lint: unsafe_contravariance
Warns when:

Declaring a public interface member (class, mixin, enum instance member) with return type which is contravariant in a type parameter of the interface. (No warnings for extension members, or the coming extension type, since their type arguments are static only, and cannot be out of sync with the static type.)
Up-casting a value of an interface type which has an accessible (public, or private from same library) contravariant return type member, to a type with as proper supertype as type argument for the contravariantly occurring type parameter. Also applies if up-casting record types, then applied to each record field individually. Applied to nullable types and FutureOr types by applying to the underlying type. Also apply to function type return types? Also applied to type arguments? A List<C<num>> l = [C<int>()]; upcast is unsafe when you access l[0].contravariantMember._
Invoking a member with a contravariantly typed return type on a value which has likely been upcast (any contravariant member access on a type whose static type is a result of an up-cast of the previous item).

BAD examples:
library c;
class C<T> {
  final String Function(T) _callback; // No lint, private getter.
  C(this._callback);
  final String Function(T) callback; // LINT: Contravariant getter return type.
  String Function(T) getCallback() => _callback; // LINT: Contravariant method return type.
  String Function(T) operator ~() => _callback; // LINT: Contravariant methods return type.
}

and
import "c.dart" show C;

void main() {
  var c1 = C<int>((i) => "$i");
  // Change from a type which implements `C<int>` to one which implements `C<num>`,
  // `num` is a proper supertype of `int`, and that type parameter occurs contravariantly in public members.
  C<num> c2 = c1; // LINT: Upcasting type with accessible contravariant member(s): callback, getCallback and ~.

  Object o = c1; // No lint, not changing type argument to `C<...>` to a supertype.

  if (theSignsAreRight) {
    c2 = C<num>((n) => "$n"); // No lint, not upcast.
    // The value of `c2` that the contravariant member is accessed on comes from a non-downcast assignment.
    print(c2.getCallback(3.14)); // No lint, supposed valid use.
  }
  // `c2` may have the value from `c2 = c1` or from `c2 = C<num>(...);`. It'
  var f = c2.getCallback(); // LINT: Upcast value `c2 = c1` can reach here, don't use contravariant member.

  invoke(c1, 3.14); // LINT: Upcasting `c1` to `C<num>`.
}

// No lint in this function. There is no visible up-cast, and they types match.
// If called with a `C<num>`, all is well.
String invoke(C<num> c, num n) => c.getCallback()(n);

The recommended fix is to not have contravariant return types.
If you need them anyway, because Dart doesn't have contravariant generics yet,
ignore the lint at the declaration site, and avoid upcasting.
Alternatively, encapsulate the contravariance, and handle it explicitly. Don't return the contravariant value,
provide methods which allow validated and type-checked access to it.
The arguments may still be impossible to use, but you can control what happens then.
abstract class C<T> {
  factory C(String Function(T) callback) : _C<T>;
  String callBack(T value);
  String tryCallBack(T value);
}
class _C<T> implements C<T> {
  final String Function(T) _callback;
  _C(this._callback);
  String callBack(T value) {  // Covariant parameter, throws on non-`T` argument.
    return _callback(value);  // `this._callback` is always a safe access.
  }
  String? tryCallBack(Object? value) { // Implement `C` interface without covariant parameter.
    if (value is T) return _callback(value);
    return null;
  }
}

Possible issue:
void foo() {
  C<int> c1 = C<int>((int n) => "$n");
  C<num> c2 = c1; // LINT
  c2 = c2; // No lint? And now not assumed to have been upcast now?
  c2.getCallback(); // No lint?
}

Would it be possible to remember, for each contravariant type, which unsafe types it might have been upcast from.
So that at c2 = c2; we know that the current value of c2 may be a C<int> (actually, we know it definitely is one).
Then the c2 = c2; won't just say "well, c2 was definitely assigned a C<num> last, so it's probably fine, it'll remember
that it's still possibly (definitely!) a C<int>.
It might lint again at the c2 = c2; assignment, but it definitely will at the c2.getCallback() call. (It's a guaranteed error!)
This analysis should only happen inside a single function, and could possibly be tracked as metadata on the type itself.
Any occurrence of C<T>, or any other unsafely contravariant type, can have an associated "possibly/definitely upcast from C<S>". (Maybe it can have more than one, when it's "possibly", but we should avoid accumulating too many in loops, so maybe just have "definitely upcast from type", and "potentially upcast from specific type" or just "potentially upcast" when we can't say from which type.)
We could also say just "potentially upcast" and leave it at that, but knowing from which type can be useful:
void foo() {
  C<int> c1 = C<int>((int n) => "$n");
  C<Object> c2 = c1; // ignore: unsafe_contravariance
  // Type of `c2` is `C<Object>`(definitely upcast from `C<int>`)
  if (signsAreRight) {
    c2 = C<Object>((n) => "$n");
    // Type of `c2` is `C<Object>`.
  }
  // Type of `c2` is `C<Object>`(maybe upcast from `C<int>`)
  // ...
  if (c2 is C<int> && somethingMore) {
    // Type of `c2` is `C<int>`, dropping the "(maybe upcast from `C<int>`)" since it's unnecessary.
    c2.getCallback()(2); // No warning.
  }
  if (c2 is C<num>) {
   // Type of `c2` is `C<num>`(maybe upcast from `C<int>`) - can't drop the "maybe upcast".
   c2.getCallback(3.14); // LINT: Invoking contravariant member of possibly upcast value.
  }

  if (starsAlign) {
    c2 = C<String>((s) => s); // ignore: unsafe_contravariance
    // Type of c2 is `C<Object>`(definitely upcast from `C<String>`
  }
  // Type of c2 is `C<Object>`(maybe upcast ) - can't say "from `C<String> or C<int>`.
  // Of one of the two was a supertype of the other, we can ignore it. If not, LUB is not good enough.
  c2.getCallback(0); // LINT: Invoking contravariant member of possibly upcast value.
}

(But this sounds like a big task. Just linting at the declaration and the downcast would at least warn users what they're up against.)

@SaadArdati, much of the ground was already covered by @lrhn, but I'd like to add a few extras.
Here is a slightly simplified restatement of the original example:
abstract class Foo {}

class Bar extends Foo {}

class FooMatcher<T extends Foo> {
  final String Function(T) builder;
  const FooMatcher({required this.builder});
}

void main() {
  final FooMatcher<Foo> matcher = FooMatcher<Bar>(builder: (_) => '<3');
  matcher.builder; // Throws.
}

The situation where a getter or a method has a return type which is non-covariant in the type variables of the enclosing class gives rise to a caller-side check (that's the situation where matcher.builder throws just because we're accessing it, we don't even need to call it). I've used the phrase "contravariant members" to describe this situation, and it's discussed in https://github.com/dart-lang/language/issues/296 and https://github.com/dart-lang/language/issues/297.
I proposed a lint unsafe_variance here: https://github.com/dart-lang/linter/issues/4111. I do not think it makes sense to try to emit compile-time messages about upcasts (that's undecidable anyway). The real fix will not be to avoid upcasts, but to make the type variable invariant.
This implies that you get a compile-time error rather than a run-time error, when an attempt is made to create the unsafe situation.
First, this feature is part of the declaration-site variance feature, dart-lang/language#524. It has not been fully implemented at this time, and it hasn't been scheduled for release. But I hope we'll have it, soon (vote for it if you agree!), and we can try it out like this already:
// Assuming `--enable-experiment=variance`.

abstract class Foo {}

class Bar extends Foo {}

class FooMatcher<inout T extends Foo> { // <----- Just add `inout` here!
  final String Function(T) builder;
  const FooMatcher({required this.builder});
}

void main() {
  // The initialization of `matcher` is now a compile-time error.
  final FooMatcher<Foo> matcher = FooMatcher<Bar>(builder: (_) => '<3');
  // ...
}

The keyword inout means that the type parameter is both in and out (i.e., it is usable as a parameter type and as a return type), and that in turn means that it can't vary (it is invariant). In particular, FooMatcher<Bar> is not a subtype of FooMatcher<Foo>, even though Bar is a subtype of Foo. This is exactly what you need when you have type variables in non-covariant positions in your class members, in order to make your code statically safe.
You can emulate invariance in the current language as well. The basic idea is that you use two type variables rather than one type variable to represent the invariant type variable.
The first type variable will have the same value as the type variable you're using today. The second type variable must always be T Function(T) where T is the value of the first type variable.
We use a typedef such that clients won't need to know, and won't get a chance to get it wrong. That typedef will have the old name of the class, and the class will get a private name. The client code does not change.
Here is a variant of the example with the updated declaration of FooMatcher, using this technique:
abstract class Foo {}

class Bar extends Foo {}

class _FooMatcher<T extends Foo, Invariance> {
  final String Function(T) builder;
  const _FooMatcher({required this.builder});
}

typedef FooMatcher<T extends Foo> = _FooMatcher<T, T Function(T)>;

void main() {
  // The initialization of `matcher` is now a compile-time error.
  final FooMatcher<Foo> matcher = FooMatcher<Bar>(builder: (_) => '<3');
}

If we do not create the unsafe situation then we won't have the compile-time error, and the program runs with no errors:
... // Same.

void main() {
  final matcher = FooMatcher<Bar>(builder: (_) => '<3');
  matcher.builder; // No problems
  matcher.builder(Bar());
}

So Erik says we already have a lint request, and the current behavior is doing what it's specified to do, so I'll close this issue, and suggest chiming in on the lint request, if you want to make that happen.

authors: Febirison SaadArdati eernstg lrhn


text:
[breaking change] Add lineTerminator field to Stdout

Change Intent
To make it possible for stdout/stderr to correctly output Windows line endings ('\r\n') when write/writeln by adding a new field:
class _StdSink implements IOSink {
  ...
  /// The line ending used by [writeln] and to translate `"\n"` written by
  /// the [write], [writeln] and [writeAll] and [writeCharCode] methods.
  ///
  /// If a string containing `"\n"`s is written using one of the above
  /// "write" methods, then the embedded `"\n"`s are translated to the value
  /// set in `lineTerminator`. If the string contains Windows line endings
  /// (`"\r\n"`) then they are not translated. Carriage returns (`"\r`) are
  /// not translated.
  ///
  /// If `lineTerminator` is `"\n"` then the written strings are not modified.
  //
  /// Setting `lineTerminator` to [Platform.lineTerminator] will result in
  /// "write" methods outputting the line endings for the platform:
  ///
  /// ```dart
  /// stdout.lineTerminator = Platform.lineTerminator;
  /// stderr.lineTerminator = Platform.lineTerminator;
  /// ```
  ///
  /// The value of `lineTerminator` has no effect on byte-oriented methods
  /// such as [add].
  ///
  /// The value of `lineTerminator` does not effect the output of the [print]
  /// function.
  ///
  /// Throws [ArgumentError] if set to a value other than `"\n"` or `"\r\n"`.
  set lineTerminator(String eol);
  ...
}

The default value of lineTerminator will be "\n" so the output will not change for existing Dart code.
Justification
To write correct terminal applications on Windows, developers currently have to write awkward code like:
stdout.write("Hello World" + Platform.lineTerminator);

Impact
Any class that implements Stdout without using Mock/Mock or overriding noSuchMethod will break.
There are 100s of files on github that implement Stdout but most them that I looked at use Fake/Mock or override noSuchMethod.
A few do not:
https://github.com/onepub-dev/minion/blob/57cda065a6fc489839d0cbf80d15f6d08432e614/lib/src/stdout.dart#L7
https://github.com/tvolkert/universal_io/blob/02f99955f2dfbc94269ddfa8d326a1bbe7787e8e/lib/src/driver_base/base_std_out.dart#L21
https://github.com/filiph/linkcheck/blob/8ab5f5b516701f98c18cb2f16a73e5f93ebf7f12/test/e2e_test.dart#L218
Mitigation
Developers will have to add:
+   String lineEnding;

to their Stdout implementations.
Change Timeline
N/A
Associated CLs

326761: Make it possible to change the line ending output by stdout and stderr.

Breaking change request incoming @grouma @Hixie @vsmenon

Sounds reasonable, but ... should we consider changing the Win default?  @a-siva @mraleph @tvolkert

SGTM. No impact to ACX.

FYI, https://github.com/tvolkert/universal_io was a clone of another repo -- I just deleted my clone.

Sounds reasonable, but ... should we consider changing the Win default? @a-siva @mraleph @tvolkert

I think that could be a reasonable breaking change, though it might lead to some subtle breakages when output contents suddenly change, e.g. if people produce files by redirecting output then contents of these files will change.

no objection from me

@vsmenon

Sounds reasonable, but ... should we consider changing the Win default? @a-siva @mraleph @tvolkert

I tried that initially but it is too breaking - there are a lot of CLI projects with tests that assert their output. If the default output changes to "\r\n" on Windows then their Windows tests would break.

lgtm

Fixed in https://github.com/dart-lang/sdk/commit/770f44d4e9ffe2abf4c27bc0969c6708c54a9b65

authors: Hixie brianquinlan grouma itsjustkevin mraleph tvolkert vsmenon


text:
[dart:js_interop] Can't pass an function reference to external function when assertions are enabled.

For example:
import 'dart:js_interop';

@JS('set_property')
external void _setProperty(JSObject object, String name, ExternalDartReference? value);

void setProperty(JSObject object, String name, Object? value) {
  _setProperty(object, name, value?.toExternalReference);
}

throws this (DDC, dart2js)
Assertion failed: "Dart function requires `allowInterop` to be passed to JavaScript."

Dart SDK version: 3.5.0-edge.936b3a2ccf38aaf8ab3cd7e50f84231a053d669c (main) (Wed Apr 10 23:48:20 2024 +0000) on "windows_x64".

This was needed with previous versions of interop as we didn't have enough static type information. I think we can safely disable the checks for dart:js_interop methods. While it's possible users might try to use this Dart function in JS, that's much less likely than wanting to pass a Dart function around using ExternalDartReference.

authors: srujzs ykmnkmi


text:
Switch Expressions conversion assist should use logical-or when multiple cases share a body

From the docs:

Logical-or patterns are useful for having multiple cases share a body in switch expressions or statements:
var isPrimary = switch (color) {
  Color.red || Color.yellow || Color.blue => true,
  _ => false
};

When we have a switch statement like the following, the assist for turning it into a switch expression doesn't show:
int a(int x) {
  switch (x) {
    case 0:
      return 0;
    case 2:
    case 1:
      return 1;
    default:
      return throw UnimplementedError();
  }
}

Only if there is a single case with a specific body:

In this case, I believe the assist should appear and create a logical-or pattern for them.

I'm not sure if you agree, but I think maybe there could be an assist for merging all of those cases with a logical-or pattern. And that could be run separately or with this assist.
So cases like the above could become:
switch (x) {
  case 0:
    return 0;
  case 1 || 2: // or swapped I'm not sure what to put first here (not sure it matters, just an UX question)
    return 1;
  default:
    return throw UnimplementedError();
}

And then the assist for replacing the switch does appear.

Created https://github.com/dart-lang/sdk/issues/56180 for the above comment.

Looking at the code and blame for this feature and trying to understand it to implement it by myself, I just wanted to point out that this is already something working but for some reason not all cases.
Here is the original issue: https://github.com/dart-lang/sdk/issues/54567
If I try to do something like:
enum A {
  a,
  b,
  c,
  d
}

int a(A x) {
  switch (x) {
    case A.a:
      return 0;
    case A.b:
    case A.c:
      return 1;
    default:
      return throw UnimplementedError();
  }
}

Then the assist does appear. So something is not working as it should. Maybe something related to integers not having a finite amount of options?
Will take a look into it whenever I can, just a little update here.

In the latest version (on HEAD), if I paste the original example into a file:
int a(int x) {
  switch (x) {
    case 0:
      return 0;
    case 2:
    case 1:
      return 1;
    default:
      return throw UnimplementedError();
  }
}

When I click on the switch keyword and open the lightbulb menu, I see a 'Convert to switch expression' option that produces the following:
int a(int x) {
  return switch (x) {
    0 => 0,
    2 || 1 => 1,
    _ => throw UnimplementedError()
  };
}

It appears to me that the limitation has been fixed, but the issue wasn't updated appropriately.
If I'm missing something please let me know and I'll re-open the issue, but for now I'm going to close it.

authors: FMorschel bwilkerson


text:
dashbuilder-parent directly inherits from ip-bom for easier dependency overrides

now we can easily override the dependency version just via property
(since dashbuilder-parent extends ip-bom instead of
importing it in depMgmt)

user BOM 'dashbuilder-bom' can not extend the dashbuilder-parent
as that would mean polluting it with hundreds of different
versions coming from the ip-bom -- which is not desirable
for user BOMs

dashbuilder-parent-metadata was renamed to just dashbuilder-parent.
dashbuilder-deps was removed completely as it is no longer required.
All Dashbuilder modules should (transitively) use the dashbuilder-parent
(except for the user BOM(s))

@mbiarnes please take a look and let me know if everything is clear regarding these changes. I would like to apply the same approach into the rest of KIE repos in the upcoming days/weeks.

+1

Thanks @dgutierr
(just for the record, the PR was merged as https://github.com/dashbuilder/dashbuilder/commit/45d45eb38381cdb1799cb12091dadb9ed9b47a74)

source: gharchive/pull-request

authors: dgutierr psiroky

repo: dashbuilder/dashbuilder

license_source: bigquery


text:
blockchain_processor.py line 126, in dashd raise BaseException(r['error'])

Hi,
I'd like set up Electrum Dash server, but I have a strange problem now.
This is worked earlier like a charm, but then I had Linux Mint and little bit older Elertrum server version and dash core daemon.
Now I have Debian 8.5.0 (3.16.0-4-amd64), latest Dash core daemon and electrum-dash-server.
Dash Core Daemon version v0.12.0.58 (not compiled myself, binary downloaded from github)
Daemon working fine, but electrum server not when all block are pruned. I have started pruning from scratch, but not working.
I got always this error (all settings should be ok):
INFO:electrum-dash:catch_up: block 509150 (0.484s 18.325s) dbe3b3c51c45bf88100f07b6cf9109d95ceccc1b98c4695856294db02f554ea0
INFO:electrum-dash:catch_up: block 509200 (0.508s 12.740s) 35d2c2098b23ead375f62c5b2791e69036eebddb4d75efda0ed763c10c8fd5e8
INFO:electrum-dash:catch_up: block 509250 (0.402s 13.524s) b23121156331613836679df5658a6fdeef7e39d99f06c3e499134c4a57301862
INFO:electrum-dash:Blockchain is up to date.
INFO:electrum-dash:Memory pool initialized.
INFO:electrum-dash:blockchain: 509254 (0.003s)
Exception in thread Thread-4:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/jeppa/Downloads/git/electrum-dash-server/src/blockchain_processor.py", line 86, in do_catch_up
self.main_iteration()
File "/home/jeppa/Downloads/git/electrum-dash-server/src/blockchain_processor.py", line 923, in main_iteration
proposals_status = self.get_proposals_status()
File "/home/jeppa/Downloads/git/electrum-dash-server/src/blockchain_processor.py", line 359, in get_proposals_status
return self.dashd('mnbudget', ['list'])
File "/home/jeppa/Downloads/git/electrum-dash-server/src/blockchain_processor.py", line 126, in dashd
raise BaseException(r['error'])
BaseException: {u'message': u'mnbudget "command"... ( "passphrase" )\nVote or show current budgets\n\nAvailable commands:\n  prepare            - Prepare proposal for network by signing and creating tx\n  submit             - Submit proposal for network\n  vote-many          - Vote on a Dash initiative\n  vote-alias         - Vote on a Dash initiative\n  vote               - Vote on a Dash initiative/budget\n  getvotes           - Show current masternode budgets\n  getinfo            - Show current masternode budgets\n  show               - Show all budgets\n  projection         - Show the projection of which proposals will be paid the next cycle\n  check              - Scan proposals and remove invalid\n  nextblock          - Get next superblock for budget system\n', u'code': -1}
I check "blockchain_processor.py", line 126" and it raised exception there.
    r = loads(respdata)
    if r['error'] is not None:
        raise BaseException(r['error'])
    return r.get('result')

"respdata" is very strange and that's why exception raised

here is totally outdated...
Please try my version https://github.com/thelazier/electrum-dash-server
BTW, https://github.com/thelazier/electrum-dash-db might worth checking out to faster the synchronization time with mainnet.

I have sync the dashpay repo with @thelazier 's repos now

Now the electrum server working ok. Thank you very much. This ticket could be closed now.

authors: santzi schinzelh thelazier

repo: dashpay/electrum-dash-server

license: mit


text:
Job script epilogue

Would be nice to have job_script_prologue equivalent but for cleanup. I.e. run some commands after worker exists. The argument could be called job_script_epilogue.

Hi @ernimd,
Sure, I can imagine this can be useful in some case, I'm almost surprised the need didn't occur before. I think it should be simple enough, considering the prologue part is already implemented.
Would you want to submi a PR for this?

authors: ernimd guillaumeeb

repo: dask/dask-jobqueue


text:
Release?

This seems usable?  Should we release?
@kmpaul I'm happy to walk you through this if you like or just handle it for you.

I would love a walkthrough.  I've done releases on GitHub before (obviously), and I've published to PyPI.  I'm in the middle of trying to get a package published to conda-forge, but it has not been successful, yet.  So, if you can provide guidance on how this should be done for dask-mpi, that would be great!

Happy to screenshare sometime this morning.  I'm free most of the day.
In the meantime, here are the steps that I would do.

Tag a release and push to github
git tag -a 1.0.0 -m "Version 1.0.0"
git push origin master --tags

Build and publish release on PyPI
git clean -xfd  # remove any files not checked into git
python setup.py sdist bdist_wheel --universal  # build package
twine upload dist/*  # register and push to pypi

Build conda recipe
fork github.com/conda-forge/staged-recipes
git clone git@github.com:kmpaul/staged-recipes
cd staged-recipes
git checkout -b dask-mpi
mkdir dask-mpi
cp recipes/example/meta.yaml recipes/dask-mpi/meta.yaml
# edit recipes/dask-mpi/meta.yaml

set package name and version

get sha256 from pypi.org, similar to this page: https://pypi.org/project/dask/#files but for dask-mpi

uncomment noarch: python

remove c compiler build step (I think?)

fill in the rest of the information (should be simple, I think?)

commit and submit a PR upstream

Ok.  I think that seems reasonable.  I've done most of this before.
...One question about the conda recipe, however.  Since dask-mpi depends on MPI...which I don't think is available on Windows, do I need to do anything special to indicate that this won't work on Windows?

I would put mpi4py in the requirements and then just be done with it.  I suspect that whatever they've done for mpi4py will transitively carry over to us.

Sounds reasonable.

I see that dask-mpi is up on PyPI.  Woot!
Offer still stands to screenshare for conda-forge.  I suspect that we could bang it out in fifteen minutes.

Yes!  😄  I'd like to bang away at the conda-forge step myself for a bit.  I need to learn this and get used to it.

Go for it :)
On Thu, Jan 3, 2019 at 10:29 AM Kevin Paul notifications@github.com wrote:

Ok. I had to update the MANIFEST and rerelease on PyPI in order to get the
conda-force recipe to work. But it seems to work now. The recipe is here:
https://github.com/kmpaul/staged-recipes/blob/dask-mpi/recipes/dask-mpi/meta.yaml
Let me know if that looks okay to you, and then I'll submit the PR.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask-mpi/issues/15#issuecomment-451233830, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszJW6jvQFeCCiVy19OPjBk5PQJBd5ks5u_kwOgaJpZM4ZniHA
.

Incidentally, I followed a slightly different path when putting together the setup.py file.  I included in the repository environment.yml file(s), and not requirements.txt files.  I know that this is non-standard.  Would you like me to switch over to the requirements.txt file approach before finishing the release?  I can always modify, test, and then re-release on PyPI first, before the PR.

Actually, I can change that in the future, if desired (I'll make an issue for it).  I'll go ahead and submit the PR.

I think it's up to you.  I recommend doing whatever you find comfortable.
If an issue arises then it's easy enough to change.
On Thu, Jan 3, 2019 at 10:38 AM Kevin Paul notifications@github.com wrote:

Actually, I can change that in the future, if desired (I'll make an issue
for it). I'll go ahead and submit the PR.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask-mpi/issues/15#issuecomment-451236399, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszFQ5SGagXtbrN4nbW_5ru7-vPQKBks5u_k4qgaJpZM4ZniHA
.

Sounds good.  I'll wait to see if the PR goes through.

authors: kmpaul mrocklin

repo: dask/dask-mpi


text:
Inconsistent handling when iterating Delayed objects with nout=1

The following code causes a slice to be taken from a Delayed object when _length==1:
https://github.com/dask/dask/blob/e9845aadc7c32078e55354b61f5626ac759e8b28/dask/delayed.py#L619-L620
Here's a minimal example of the anomalous behavior from the case when nout=1
>>> from dask.delayed import delayed
>>> @delayed(nout=1)
... def ret(x):
...     return x
...
>>> @delayed(nout=2)
... def echo(x):
...     return x,x
...
>>> d1 = ret([1,2,3])
>>> d2 = echo([1,2,3])
>>> for d in d1:
...     print(d.compute())
...
1
>>> for d in d2:
...     print(d.compute())
...
[1, 2, 3]
[1, 2, 3]

I think the iteration of d1 should be the full output, i.e. [1, 2, 3], instead of just 1.
An inelegant fix could be:
if self._length == 1:
    yield self
else:
    for i in range(self._length):
         yield self[i]

Thanks for your report.
This is actually intended behavior and the special case of nout=1 is even mentioned explicitly in the documentation. If anything, we should raise in this case because you are clearly not returning a 1-tuple.
What you'd be looking for is something like an "auto" functionality. This is not implemented because dask is constructing the full task graph before any tasks are computed. Therefore, we need to know this information ahead of time.
When using dask.distributed (e.g. using a LocalCluster) there is the possibility for tasks to launch new tasks such that the graph is fully dynamic, see https://distributed.dask.org/en/stable/task-launch.html

Noted on the expected behavior, and on the docs.
I agree that there should be an exception raised - I think the problem is that the slice taken in the iterator means different things, but isn't necessarily invalid, so an exception isn't naturally raised at the source. In our case, it was raised downstream when we were multiplying together two arrays in a later task, and the dimensions did not work together (because of the slice).
I might be able to contribute a PR, although I'd probably need some guidance. I think the exception should ideally raise when the Delayed task is created, but without the function executing on a small amount of inputs, I'm not sure how I'd tell the return type. Alternatively (more easily?), I checked that wrapping the function with a tuple-check in the case that nout is set does work:
def _check_len(func, n, *args, **kwargs):
    def f(*args, **kwargs):
        ret = func(*args, **kwargs)
        if not isinstance(ret, tuple) or len(ret) != n:
            raise RuntimeError(f"Function {func.__name__} didn't return a tuple of length {n}")
        return ret
    f.__name__ = func.__name__
    f.__doc__ = func.__doc__
    return f

Thoughts?

Possibly but we'd likely need to relax the check on the object. While a tuple is surely the desired output I expect there are users that are relying on this simply being an iterarable/sequence of any kind.
I would suggest to attempt the wrapping and see what breaks in our test suite :)
@jrbourbeau do you have an idea how such a post-compute verification could be implemented?

authors: fjetter mdwgrogan

repo: dask/dask


text:
Slice by 0-dimensional np.array

As of the current git head:
>>>> a = da.arange(2, chunks=2)
>>>> idx = np.array(0)
>>>> a[idx]

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-4-013759abffed> in <module>()
      3 a = da.arange(2, chunks=2)
      4 idx = np.array(0)
----> 5 a[idx]

~/anaconda3/lib/python3.6/site-packages/dask/array/core.py in __getitem__(self, index)
   1306
   1307         out = 'getitem-' + tokenize(self, index2)
-> 1308         dsk, chunks = slice_array(out, self.name, self.chunks, index2)
   1309
   1310         dsk2 = sharedict.merge(self.dask, (out, dsk))

~/anaconda3/lib/python3.6/site-packages/dask/array/slicing.py in slice_array(out_name, in_name, blockdims, index)
    155
    156     # Pass down to next function
--> 157     dsk_out, bd_out = slice_with_newaxes(out_name, in_name, blockdims, index)
    158
    159     bd_out = tuple(map(tuple, bd_out))

~/anaconda3/lib/python3.6/site-packages/dask/array/slicing.py in slice_with_newaxes(out_name, in_name, blockdims, index)
    177
    178     # Pass down and do work
--> 179     dsk, blockdims2 = slice_wrap_lists(out_name, in_name, blockdims, index2)
    180
    181     if where_none:

~/anaconda3/lib/python3.6/site-packages/dask/array/slicing.py in slice_wrap_lists(out_name, in_name, blockdims, index)
    232     # No lists, hooray! just use slice_slices_and_integers
    233     if not where_list:
--> 234         return slice_slices_and_integers(out_name, in_name, blockdims, index)
    235
    236     # Replace all lists with full slices  [3, 1, 0] -> slice(None, None, None)

~/anaconda3/lib/python3.6/site-packages/dask/array/slicing.py in slice_slices_and_integers(out_name, in_name, blockdims, index)
    279             raise ValueError("Arrays chunk sizes are unknown: %s", shape)
    280
--> 281     assert all(isinstance(ind, (slice, Integral)) for ind in index)
    282     assert len(index) == len(blockdims)
    283

Still an issue in dask 1.2.0

Also observing this in dask 2.6.0.

If someone is able to fix this, that would be welcome.

authors: crusaderky davidsebfischer mrocklin


text:
GeoPandas Integration

Geopandas 0.6 supports the Pandas extension array interface.  So simple Dask integration works out of the box:
In [1]: import geopandas

In [2]: gdf = geopandas.read_file(geopandas.datasets.get_path('naturalearth_cities'))

In [3]: gdf
Out[3]:
             name                     geometry
0    Vatican City    POINT (12.45339 41.90328)
1      San Marino    POINT (12.44177 43.93610)
2           Vaduz     POINT (9.51667 47.13372)
3      Luxembourg     POINT (6.13000 49.61166)
4         Palikir    POINT (158.14997 6.91664)
..            ...                          ...
197         Cairo    POINT (31.24802 30.05191)
198         Tokyo   POINT (139.74946 35.68696)
199         Paris     POINT (2.33139 48.86864)
200      Santiago  POINT (-70.66899 -33.44807)
201     Singapore    POINT (103.85387 1.29498)

[202 rows x 2 columns]

In [4]: import dask.dataframe as dd

In [5]: ddf = dd.from_pandas(gdf, npartitions=5)

In [6]: ddf
Out[6]:
Dask DataFrame Structure:
                 name  geometry
npartitions=5
0              object  geometry
41                ...       ...
...               ...       ...
164               ...       ...
201               ...       ...
Dask Name: from_pandas, 5 tasks

In [7]: ddf.geometry.head()
Out[7]:
0    POINT (12.45339 41.90328)
1    POINT (12.44177 43.93610)
2     POINT (9.51667 47.13372)
3     POINT (6.13000 49.61166)
4    POINT (158.14997 6.91664)
Name: geometry, dtype: geometry

What's next?  I see two possible approaches:

We subclass Dask DataFrames/Series and produces Dask GeoDataFrames/GeoSeries.  There is strong precedent on how to do this now thanks to dask_cudf.
We improve the handling of accessors, like a .geo and use that with the existing classes.  I think that this is the approach taken by cyberpandas.

@jorisvandenbossche @TomAugspurger

The subclass approach is basically what you prototyped in https://github.com/mrocklin/dask-geopandas, is that correct?
I think if the goal is to provide a similar user interface as GeoPandas itself, as long as that is using a subclass approach, the dask-geopandas bridge probably will need to use it as well. It could of course also be an opportunity to experiment with an accessor instead of subclass approach.

The subclass approach is basically what you prototyped in https://github.com/mrocklin/dask-geopandas, is that correct?

Mostly.  It's now much more mature in dask-cudf.

Without checking, I suspect that option 2 is not a lot of work.
Geopandas would need to register a couple methods telling dask about its scalar type. https://github.com/ContinuumIO/cyberpandas/pull/39/files has an example. The trickiest part is making it work with dask as an optional dependency.
https://docs.dask.org/en/latest/dataframe-extend.html#extension-arrays has the docs.

It sounds like both approaches are easy, and have examples to draw from.
This might be a good project for someone that is new to the community.  Do
we know of anyone that might want to take this on?
On Sat, Sep 28, 2019, 2:19 PM Tom Augspurger notifications@github.com
wrote:

Without checking, I suspect that option 2 is not a lot of work.
Geopandas would need to register a couple methods telling dask about its
scalar type. https://github.com/ContinuumIO/cyberpandas/pull/39/files has
an example. The trickiest part is making it work with dask as an optional
dependency.
https://docs.dask.org/en/latest/dataframe-extend.html#extension-arrays
has the docs.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/5438?email_source=notifications&email_token=AACKZTANB6S3LV5QJREDYETQL6U4NA5CNFSM4I3OL2C2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD73AWRI#issuecomment-536218437,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACKZTHCCQF2P5ZQZE3M533QL6U4NANCNFSM4I3OL2CQ
.

One thing that @mrocklin did in the POC is to provide a way to have the partitions of the dask dataframe based on spatial regions.
What is the current status of custom partitioning / indexing in dask? Or if we want that, that needs a subclass?

For that we would want a subclass today.
On Wed, Oct 2, 2019 at 7:17 AM Joris Van den Bossche <
notifications@github.com> wrote:

One thing that @mrocklin https://github.com/mrocklin did in the POC is
to provide a way to have the partitions of the dask dataframe based on
spatial regions.
What is the current status of custom partitioning / indexing in dask? Or
if we want that, that needs a subclass?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/5438?email_source=notifications&email_token=AACKZTGGJKHRLEYUQVS6ASTQMSGORA5CNFSM4I3OL2C2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAERG7Y#issuecomment-537465727,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACKZTFD3GXVBTJJPJ6YYMDQMSGORANCNFSM4I3OL2CQ
.

After thinking some more about this, I think a lot of the value will actually come from smart spatial-aware partitions.
(which doesn't mean the basic integration isn't useful of course)

I am interested in working on this. It seems like the discussion ended on subclassing being the right approach since spatial partitioning would be one of the bigger value-adds. Given the discussion in geopandas, do people still think that subclassing is the right way for this to work? To be clear, I don't think spatial partitioning would necessarily be in the minimal version of dask-geopandas, I just want to keep that usecase in mind when setting up the structure.

My gut reaction is subclassing is a good workaround for getting spatial partitioning. It's working around that fact that dask.dataframe ties together partitioning with the Index of the DataFrame. If you could partition a DataFrame on an arbitrary column then subclassing (I think) wouldn't be needed.
That said, changing how dask.dataframe does partitioning to allow alternative partitioning schemes is likely a much bigger task in terms of work and long-term maintenance :) So for now, subclassing is probably the way to go.

Is there a general goal to allow alternative partitioning schemes in the future? From the xarray perspective, there has been some discussion around partitioning by coordinate rather than by index right? So would that be another use-case?

From the xarray perspective, there has been some discussion around partitioning by coordinate rather than by index right?

I'm not sure.

Is there a general goal to allow alternative partitioning schemes in the future?

I don't recall if we have a dedicated issue, or if it's just something that comes up from time to time. It's slightly related to https://github.com/dask/dask/issues/1493 (MultiIndex), though not quite the same since that would be extending the current partitioning scheme a bit.

If you could partition a DataFrame on an arbitrary column then subclassing (I think) wouldn't be needed

I think that we need more than this.  Concepts like divisions may no longer make sense.  We'll need to keep around different metadata that keeps track of the geospatial bounds of each partition rather than just min/max.  I think that this will require something like a subclass.

We'll need to keep around different metadata that keeps track of the geospatial bounds of each partition rather than just min/max.

Or the concept of divisions would need to be generalized to support membership tests (and possibly orderings) according to different metrics. Not an easy task!

Regardless of my diversion in https://github.com/dask/dask/issues/5438#issuecomment-584699454, I think subclassing is a fine solution for now, if you're interested in working on this @jsignell :)

Ok sounds good :) I'll get cracking.

@jorisvandenbossche yeah, I've been following the work on spatialpandas. Would it maybe make sense to have a call between you, me, and @jonmmease to discuss how to best build off his work?

I touched base with @jonmmease and he agrees that spatialpandas isn't mature enough to be a dependency of dask-geopandas. He recommended that we just steal ideals as we see fit. So with that in mind, I have set up the project files over at https://github.com/jsignell/dask-geopandas and am starting actual work now.
My plan is to mimic dask_cudf and not worry about spatial indexing or joining for now.

Minor victory :)

Woo!
On Mon, Feb 17, 2020 at 2:30 PM Julia Signell notifications@github.com
wrote:

Minor victory :)
[image: image]
https://user-images.githubusercontent.com/4806877/74689957-2fb6fb80-51ab-11ea-8bd0-b093f74c4cc0.png
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/5438?email_source=notifications&email_token=AACKZTB6HINTRH3MKWR3KGDRDMFY7A5CNFSM4I3OL2C2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL72F4I#issuecomment-587178737,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACKZTF5JXT4E2GWNRTVR7TRDMFY7ANCNFSM4I3OL2CQ
.

Hello, I think that with the first proper release of dask-geopandas  coming up shortly, this can be closed :).

Thank you so much @martinfleis and @jorisvandenbossche for taking dask-geopandas on!

authors: TomAugspurger jorisvandenbossche jsignell martinfleis mrocklin


text:
test_basic failing with sparse

Error:
E               ValueError: This operation requires consistent fill-values, but argument 1 had a fill value of 2.0, which is different from a fill_value of 4.0 in the first argument.

Test:
___________________________ test_basic[<lambda>13] ____________________________
18016
[gw3] win32 -- Python 3.8.8 C:\Miniconda3\envs\test-environment\python.exe
18017

18018
func = <function <lambda> at 0x000001F7C4136B80>
18019

18020
    @pytest.mark.parametrize("func", functions)
18021
    def test_basic(func):
18022
        x = da.random.random((2, 3, 4), chunks=(1, 2, 2))
18023
        x[x < 0.8] = 0
18024

18025
        y = x.map_blocks(sparse.COO.from_numpy)
18026

18027
        xx = func(x)
18028
        yy = func(y)
18029

18030
>       assert_eq(xx, yy)

https://github.com/dask/dask/pull/6322/checks?check_run_id=2151490009

See #7421 and #7169

Oops sorry missed that

No worries :) I've just stared at that test for too long in the past.

authors: jakirkham jsignell


text:
Pushing array rechunking into the I/O layer

Apologies for the vague request, but here goes. The tl/dr is

Sometimes, the chunking of the storage format is different from the desired chunking for analysis, requiring rechunking and complex task graphs
Some storage formats support reading windows, allowing for the construction of much simpler task graphs where each tasks consists of many reads + a concat
It would be fantastic if Dask could do this automatically

The Problem
See https://github.com/microsoft/PlanetaryComputer/issues/12 for a full description and runnable example, but the high-level task is to create a median composite of some imagery. The data forms a logical array with dimensions (time, y, x) of shape, e.g. (365, 1000, 1000). Let's suppose that logical array is created by reading and concatting many files, where each file represents a chunk like (1, 500, 500). This problem will use median as the computation, but any computation where you want to rechunk to a pattern other than what's on disk would be helped by this feature request.
Currently, Array.median requires the contraction axis have a chunksize=-1 (i.e. a single chunk). So our operation is roughly read_data(...).rechunk(-1, "auto", "auto").median("time").
This leads to a complex task graph. Here's what that looks like for a single chunk from that linked example in https://github.com/microsoft/PlanetaryComputer/issues/12.

Essentially, each worker

Reads a "large" window (the whole file, with a single time step and large spatial component)
Splits that large spatial component into many smaller components (to avoid blowing up memory on each chunk once time is consolidated)
(in a distributed context): communicates the split chunks to whatever worker is gathering all the small chunks through time for that spatial region.

It's been greatly improved, but this type of communication-heavy workload still causes issues for the distributed scheduler, leading to high memory use and dying workers.
The workaround
Some storage solutions (e.g. a collection of cloud-optimized GeoTIFFs) support "windowed" reads. With this, we're able to replace the "single large read -> split -> communicate" task graph with a much simpler one of "many small reads + concat" (no communication).  With the workaround, we get a task graph for a single output chunk like

Since there's no communication, the memory usage is much lower and more predictable. The workaround is described in more detail in https://nbviewer.org/gist/TomAugspurger/c027408ae923222fab2a45d5109d114a.
The feature request
Can Dask do this automatically? Where "this" means:

Detect an I/O layer followed by a .rechunk layer
Replace the two layers by a single layer that does the windowed reads + a concat?

I'm not sure how feasible this is, but I figured I'd open this issue for discussion anyway.

Duplicate of https://github.com/dask/dask/issues/6288.


text:
Use pyproject.toml over setup.{py,cfg}

[x] Closes #10032
[ ] Tests added / passed
[x] Passes pre-commit run --all-files

This is a stab at transitioning Dask to use pyproject.toml. Everything that could be pushed into pyproject.toml has been put there. This PR uses the PEP621 compatible hatchling build backend.
Files removed

/setup.py (unnecessary)
/setup.cfg (unncessary, all tool configuration except flake8 moved to pyproject)
/.coveragerc (moved to a table in pyproject)
/versioneer.py (use hatch-vcs in pyproject)
/dask/_version.py(now auto generated by hatch-vcs)

Files added

/pyproject.toml
/.flake8 (must be added because flake8 doesn't support pyproject.toml usage

The version is autogenerated by hatch-vcs; a new _version.py is created at installation time (or sdist/wheel building time), so we need to add dask/_version.py to .gitignore.
I reread the contributing docs to see if anything would need to be updated, and it was pretty satisfying to see that I don't think anything needs to be changed. Everything that is documented relies on just pip or pytest, and this PR doesn't change the way either need to be used.

First trip up exposed by CI: looks like the coverage configuration isn't translated from .coveragerc to pyproject.toml

Closing since #10102 is pushing forward


text:
[WIP] Upstream fixes

xref https://github.com/dask/dask/issues/10347

it looks like pyarrow doesn't support numpy 2
https://github.com/apache/arrow/blob/334c93726f3c9d9e69eb51136c7f354974c739fd/python/pyarrow/pandas_compat.py#L101

With the new pyarrow nightly I'm still getting warnings from pandas eg https://github.com/pandas-dev/pandas/pull/54155/files

requires https://github.com/dask/fastparquet/pull/886/files

Were these already added with PR ( https://github.com/dask/dask/pull/10549 )?

Thanks for following up here @jakirkham. Closing this PR as it's no longer needed

authors: graingert jakirkham jrbourbeau


text:
work-around for ddf.info() failing after categorize

xref https://github.com/pydta/pandas/issues/14368

LGTM. Thanks!

authors: jcrist jreback


text:
[array.linalg.qr] added sfqr (short-and-fat) as a counterpart to tsqr…

… (tall-and-skinny) to extend the applicability of qr; most "big data" cases using QR decompositions would probably be either short-and-fat or tall-and-skinny  #diversity

[X] 3 tests added based on those for tsqr
[X] flake8 passed

Alright, comments addressed. Also ensured that the method supports non-uniform chunking.
e.g.: for a 5x20 matrix,chunks=((5), (8,4,8)) is fine, but chunks=((5), (4,8,8)) is not.
Thanks for the pointer on parametrize.

Interestingly, looking at the method for TSQR in the reference, the matrix in question should be REALLY skinny...
For a (km) × n matrix (with k chunks/blocks), a single core would have to do a QR decomposition for a (kn) × n matrix. So m should be a lot bigger than n. I think that is something that should be featured in the documentation for the tall-and-skinny case.

... I guess things can be resolved by rechunking and recursing. But that means a bit of administration to rechunk back to the original vertical chunk-height.
Maybe in the future, the tsqr method might use the largest chunk of the input as a guide for the largest chunk to accept for a single core compute (and force the matter if nothing more can be done).
It's actually hilarious considering what will be done with a ((1, 1, 1), (n)) matrix.

Just to elaborate on what I mean... (say it's a 100 factor linear regression with 500000 rows)...
tsqr[ ((1000, ..., 1000), (100)) | vchunk_limit=None ]
--> tsqr[ ((1000, ..., 1000), (100)) | vchunk_limit=1000 ]  # vchunk_limit will be
                                                            # the max vertical
                                                            # chunk size and be
                                                            # passed down
--> with 500 ((1000), (100)) blocks (A_a)... do a QR decomposition for each
--> distributes the 500 ((100), (100)) R_a blocks into 50 ((1000), (100)) blocks (A_b)
--> with 50 ((1000), (100)) blocks... do a QR decomposition for each
--> distributes the 50 ((100), (100)) R_b blocks into 5 ((1000), (100)) blocks (A_c)
--> with 5 ((1000), (100)) blocks... do a QR decomposition for each
--> joins the 5 ((100), (100)) R_c blocks into a ((500), (100)) block (A_d)
--> does a single core QR decomposition
--> splits Q_d into 5 ((100), (100)) blocks
--> multiply with the Q_c blocks and split result into 50 ((100), (100)) blocks
--> multiply with the Q_b blocks and split into 500 ((100), (100)) blocks
--> multiply with the Q_a blocks to yield Q
--> Q has 500 ((1000, 100)) blocks, R has one ((100, 100)) block

@convexset I recommend raising the tsqr comments in a separate issue so that that conversation can continue after this PR is closed.

Other than the NotImplemented issue noted above things here look good to me.

OK, this looks good to me.  Merging.
Thanks for doing this @convexset !  Quite a nice contribution, and especially impressive that it's your first PR.  Hopefully this is the first of many ;)

authors: convexset mrocklin


text:
Duplicated isoforms for each gene

I have no idea why all isoforms of a gene are appearing twice (see attached pic). While on biodalliance.org it is correct. I cloned the latest version of Dalliance. Any hint? Thanks.

This seems like a bug introduced in the last few months. I am now using a old version (forked before April, 20016) and it works fine with another different bug (need to zoom in when un-collapse gene track).

repo: dasmoth/dalliance


text:
S3 Temporary File Unlink

#20

Please add tests test cases.

I have not written any tests because I have not built an environment to run the tests so I cannot check the operation. You can close my request and rewrite the code yourself.
I have confirmed that in my environment I have erased the s3 file.

repo: dasundev/livewire-dropzone

license: MIT


text:
Need of Internet / dubious connections / spyware

I find it a bit strange that the internet is used all the time. Especially with so many IPs... Can someone explain why?
In the data I could find out that e.g. Google is used (I assume for their language model?)
Anyway, these are all the connections that are there: I find the ones with an “!” strange:

US:
9 x huggingface.co    ---> for what???
2 x httbin.org.   !    ---> what is that?
4 x api.gradio.app   ---> Lib?
1 x cdn-lfs-us-1.huggingface.co    ---> for what?

CN:
2 x ip.taobao.com   !    ---> Very suspicious ... Spyware? This is a online shop!

IE:
2 x checkip.amazonaws.com   !   ---> Why the HECK Amazon?

CA:
7 x geolocation.onetrust.com   !    ---> Something to detect which country?

DE:
2 x github.com    ---> for updating?

In addition, the following pages are also communicated when the browser is opened:

Cloudflare.com
fonts.googleapis.com
gstatic.com    ---> Why?

Don't take offense, but I'm being very careful here. In this project, various things are still installed after scripting... all in all, it seems too confusing for me to deal with. Perhaps the creator can shed some light on this?

First of all let me applaud you for your research here.
Next, I'll put some of your concerns at ease and raise others.
HuggingFace with LFS is probably used for the models and related things.
The GradioAPI is for Gradio telemetry data, collected by Gradio (for what purpose? most likely bug tracking of something, but who knows~).
httbin.org is generally used for testing, though a thorough dive into the code can uncover other uses :eyes:
checkip.amazonaws.com used by the Gradio telemetry api to get your IP.
geolocation.onetrust.com probably more telemetry, by whom will require some deep dive.
ip.taobao.com the Chinese AWS, what is hosted there will be interesting, also requires a deep dive into the code.
I'll check these out and comeback with insights.
It seem the OP is MIA for an entire quarter so probably we won't receive input from him.
But I'm curious, and before I sacrifice time for my own tool instead of this one I'll make sure it's safe.
Thanks again for bringing this to my attention.
Cheers~

Hey thx for the informations! :)
I found out some more info using Wireshark... But first a few comments and questions on what I know so far:
HuggingFace: But why do I need all the time a connection to huggingface, if I installed all modells 🤔
GradioAPI: Yeah, gradioapi ist strange... but no proof of misuse...
httbin.org: No answer but more information

httpbin is a popular online service that provides a simple HTTP request & response service for testing and debugging HTTP libraries and clients. Some key things you can do with httpbin
from proxiesapi.com

checkip.amazonaws.com: For what does it need my IP?
ip.taobao.com: I tracked the IP behind it. Seems to be a Service from Alibaba.com ... but no clue for what :/

New Information:
I used Wireshark to look at all http connections (really only http, as there is too much traffic over my machine). These companies are always retrieved at the start of finetune or xtts webui (I'm in Germany, therefore Akamai in DE)
Akamai Technologies Inc.	DE --> "OCSP 544 Request"
EU Metro Frontend 		IR --> "OCSP 540 Request"
Edgecast Inc			US (California) 3 times reported, but no evidence of malware... just a OCSP 544 Request
Amazon.com Inc.		US (Virginia) --> Probably IP as mentioned by AnonimusJack "OCSP 533 Request"
Google LLC			US (Missouri) --> "403 GET /success.txt?ipv4 HTTP/1.1" and "423 GET /success.txt?ipv6 HTTP/1.1"🤷

What I find even more interesting is that during the process of cloning a voice from my own model, I occasionally get a Peer2Peer connection to a IP in China:
182.245.121.172              China Telecom			Beijing
!!! I have no proof that this IP is directly related to the XTTS WebUI or TTS !!!!
But perhaps someone has observed similar behavior?

I found something more.
It seems most of the "problematic" API calls come from this library: https://github.com/uliontse/translators
python class Region(Tse):     def __init__(self):         super().__init__()         self.get_addr_url = 'https://geolocation.onetrust.com/cookieconsentpub/v1/geo/location'         self.get_ip_url = 'https://httpbin.org/ip'         self.ip_api_addr_url = 'http://ip-api.com/json'  # must http.         self.ip_tb_add_url = 'https://ip.taobao.com/outGetIpInfo'         self.default_region = os.environ.get('translators_default_region', None)
From translators/server.py line 322.
Seems kinda safe.
I should check the project, maybe it can be turned off if there's no need to detect a language or do some translation for some peace of mind :grin:

authors: AnonimusJack IVIaV


text:
Add unstack

This PR adds unstack to the specification, fixes #487.

[x] Adds the signature inside the manipulation functions
[x] Adds the reference in the table of contents

Merged now, thanks @steff456

repo: data-apis/array-api


text:
Unable to load consumption roles on request modal page for large number of consumption roles

Describe the bug
When requesting a share from the catalog page, after selecting a team for an environment the consumption roles do not populate. The UI shows "Network Error" and checking the browser dev tools in the network tab, the call listEnvironmentConsumptionRoles is returning Gateway timeout 504.
This is probably because the graphQL resolver used for listEnvironmentConsumptionRoles fetches information about if the consumption roles are managed or not, etc. This checking incurs doing boto3 calls by assuming pivot role in all associated aws environment accounts.
How to Reproduce
*P.S. Please do not attach files as it's considered a security risk. Add code snippets directly in the message body as much as possible.*

Create 25 - 30  consumption roles
Add these consumption roles under one team as owner.
Goto Catalog and request access to a dataset
During requesting access to the dataset, select this team
The consumption roles drop down doesn't load

Expected behavior
The consumption role dropdown should load with all consumption roles associated with that team
Your project
No response
Screenshots
No response
OS
Mac
Python version
3.9
AWS data.all version
2.5
Additional context
No response

Thanks for the bugreport @TejasRGitHub !
I had a look into resolvers and can admit, that it may cause issue.

@TejasRGitHub  please test [this PR](https://github.com/data-dot-all/dataall/pull/1303) in your deployment as well

Hi @SofiaSazonova , sure

Hi @SofiaSazonova , I have tested the PR on my data.all instance. Looks good from the usability perspective
Testing Done -
Created 100 consumption roles and checked if all consumption roles are loaded on request modal page ✅

authors: SofiaSazonova TejasRGitHub

repo: data-dot-all/dataall


text:
Parser: add support for show,  create, etc.. statement

Development Task
Add support for show,  create, etc.. statement.

[ ] create
[ ] show

So far, has supported as follows:

ShowDatabasesStmt
ShowTablesStmt
ShowColumnsStmt
ShowCreateTable

Also, some frequently used show statements  should be supported， as follows:

[ ] show_create_trigger_stmt
[ ] show_create_user_stmt
[ ]  show_create_view_stmt
[ ]  show_engines_stmt
[ ]  show_grants_stmt
[ ]  show_keys_stmt
[ ]  show_master_status_stmt
[ ]  show_plugins_stmt
[ ]  show_privileges_stmt
[ ]  show_processlist_stmt
[ ]  show_replica_status_stmt
[ ]  show_replicas_stmt
[ ]  show_status_stmt
[ ]  show_triggers_stmt
[ ]  show_variables_stmt

May be added in the future.

create stmt develop stmt

[x] create
[x] create_index_stmt
[ ] create_resource_group_stmt
[ ] create_role_stmt
[ ] create_srs_stmt
[ ] create_table_stmt

repo: database-mesh/pisanix


text:
Support unmarshalling top-level ignored fields

Changes
The CLI allows users to pass a --json flag to provide all parameters needed for executing the request. When a request includes fields annotated with json:"-", those fields are not deserialized. As a result, requests like
databricks account workspace-assignment update --json "{\"principal_id\":123, \"permissions\": [\"USER\"],\"workspace_id \":456}"

fail because the request made is
PUT /api/2.0/accounts/<account-id>/workspaces/0/permissionassignments/principals/0

To address this, we extend the unmarshaller to support unmarshalling top-level fields annotated with json:"-". I'm not sure if it is possible to support unmarshalling all fields with this annotation, as we call json.Marshal recursively. On the other hand, it isn't necessary to support at this time. The default behavior follows the encoding/json behavior of not including ignored fields.
Tests

[x] Added unit tests for UnmarshalCustom.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (1fb6a9d) 18.29% compared to head (c6ca07c) 18.30%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #772      +/-   ##
==========================================
+ Coverage   18.29%   18.30%   +0.01%
==========================================
  Files         110      110
  Lines       14821    14824       +3
==========================================
+ Hits         2711     2714       +3
  Misses      11888    11888
  Partials      222      222

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

We may not need this after all. Instead we may handle this by accepting required path parameters on the command line. Will reopen if that approach doesn't work.

authors: codecov-commenter mgyucht

repo: databricks/databricks-sdk-go


text:
xs:choice element can't be null

When the schema is generated from a XSD using XSDToSchema, fields resulting from xs:element included in xs:choice have the nullable property set to false, unless attribute MinOccurs == 0.
IMO, this is a wrong assumption because the XML file will always have one element OR the other, and therefore one will be null.
Wouldn't it be more correct to set nullable as always true in those resulting fields ?

I think you're right - I'll take a look and can fix if so. I did 'fix' nullability to look at minOccurs recently, but may have added it in too many places.

authors: srowen trourance

repo: databricks/spark-xml