text:
fix: removal of old tasks

Fixes #138
Checklist

[x] Ensure PR title matches the conventional commits format.

CodiumAI Agent Commands
As required run the following CodiumAI Agent commands:

Request PR review using @CodiumAI-Agent /review (and if required, request an incremental update with @CodiumAI-Agent /review -i)
Request PR improvement suggestions with @CodiumAI-Agent /improve --extended
Request PR update using @CodiumAI-Agent /describe

@CodiumAI-Agent /review

PR Analysis

🎯 Main theme: The PR is about fixing an issue where old tasks were not being removed properly.
📝 PR summary: The PR modifies the logging statement in the 'sync_project' function to include the item type and title. This is done to improve the clarity of the logs when a task is removed due to not being updated within a certain threshold.
📌 Type of PR: Bug fix
🧪 Relevant tests added: No
⏱ Estimated effort to review [1-5]: 1
The PR is small and straightforward, only changing a logging statement. It should be quick and easy to review.
🔒 Security concerns: No

PR Feedback

💡 General suggestions: The PR seems to be straightforward and doesn't introduce any major changes. However, it would be beneficial to have tests that ensure the correct tasks are being removed.

 🤖 Code feedback:

relevant file: ado_asana_sync/sync/sync.py
suggestion: Consider using f-strings for formatting your strings as they are more readable and efficient. [medium]
relevant line: "+                    "%s: %s:Task has not been updated in %s days, removing mapping","

How to use

Tag me in a comment '@CodiumAI-Agent' and add one of the following commands:
/review [-i]: Request a review of your Pull Request. For an incremental review, which only considers changes since the last review, include the '-i' option.
/describe: Modify the PR title and description based on the contents of the PR.
/improve [--extended]: Suggest improvements to the code in the PR. Extended mode employs several calls, and provides a more thorough feedback.
/ask <QUESTION>: Pose a question about the PR.
/update_changelog: Update the changelog based on the PR's contents.

To edit any configuration parameter from configuration.toml, add --config_path=new_value
For example: /review --pr_reviewer.extra_instructions="focus on the file: ..."
To list the possible configuration parameters, use the /config command.

How to use PR-Agent

Tag me in a comment '@CodiumAI-Agent' and add one of the following commands:
/review [-i]: Request a review of your Pull Request. For an incremental review, which only considers changes since the last review, include the '-i' option.
/describe: Modify the PR title and description based on the contents of the PR.
/improve [--extended]: Suggest improvements to the code in the PR. Extended mode employs several calls, and provides a more thorough feedback.
/ask <QUESTION>: Pose a question about the PR.
/update_changelog: Update the changelog based on the PR's contents.

To edit any configuration parameter from configuration.toml, add --config_path=new_value
For example: /review --pr_reviewer.extra_instructions="focus on the file: ..."
To list the possible configuration parameters, use the /config command.

authors: CodiumAI-Agent danstis

repo: danstis/ado-asana-sync


text:
Implement _check

DRY by implementing _check and all of test passed

Thank you very much for accepted and merged my code

source: gharchive/pull-request

repo: danthareja/contribute-to-open-source

license: MIT

license_type: permissive

license_source: github-api


text:
Multiple included headers with space aren't available in input.request.headers

According to the docs,  one should be allowed to use multiple headers in the "inlcudedHeaders" metadata input.  For example:
`

name: includedHeaders
value: "x-my-custom-header, x-jwt-header"
`

When providing multiple headers I was only able to get the first header if there was a space between the header names.  However, if a space was not between the end of a header name and the comma it provided the second header in the input.request.headers.  So "x-my-custom-header, x-jwt-header" doesn't provide x-jwt-header but ""x-my-custom-header,x-jwt-header" does.
Actual Behavior
I expect to be able to follow the documentation and get as many headers as I need with a space between the comma and the next header name.
Steps to Reproduce the Problem
Follow the instructions for more than one header and try to get the value of the second header if there is a space between the comma and the next header name.

#1610 fixed by this pr

source: gharchive/issue

authors: addjuarez ben-kotvis

repo: dapr/components-contrib


text:
Hashicorp Vault Secret Store - To support cert. rotation and renewals

Enhance Hashicorp Vault Secret store to support password/ cert. renewals and publish the notification/ event when change takes place.

Hi,
Please consider a scenario when a microservice is connected to Hashicorp Vault using Dapr and a database. Hashicorp vault is managing the database credentials (Static role for this scenario) and Microservice is getting the credentials from Dapr (by Vault) and Creating a connection pool to connect to database using the credentials and performs the operation.
Suppose there is a password rotation happens by vault and database password got changed and vault also rotates its credentials.
This scenario won't affect Microservice for the new connection pool creation as it will fetch the credentials from Vault. However the Old pool will still in the active state with old credentials so it will try to create new connection and get the exception as credentials are old and not updated.
There is a need of some mechanism to inform the microservice to nullify the pool / pool credentials on the password rotation at database through Vault. i.e. If there is an updation of the password / Lease at Vault then Dapr should trigger some event back to microservice and inform as there is an update of password / lease happened at Vault & database so Microservice can also refresh / nullify the current connection pool.

Tagged for 1.10

Tagged for 1.10

We do not tie certification efforts to a particular milestone. Also, certifying this component for GA does not require certificate renewals. These are two separate things.

/assignee

👋

The Certification of Hashicorp to stable is being handled by other issue, this will be targeted to enable the Hashicorp vault supporting cert. / password renewals and publishing an event when renewal takes place.

The Certification of Hashicorp to stable is being handled by other issue, this will be targeted to enable the Hashicorp vault supporting cert. / password renewals and publishing an event when renewal takes place.
@mukundansundar - can you help update the issue description to - Feature request to support password/ cert. renewals and publish the notification/ event when change takes place in Hashicorp vault secret store.

Done

//comment

//comment

authors: Radhikabollineni mukundansundar rabollin vivekthakur-ai yaron2


text:
kafka pubsub always use app-max-concurrency = 1

/area runtime
What version of Dapr?
1.9.5
Hi all!
Could you please tell me if kafka can be used for parallel event processing?
We did not set the dapr.io/app-max-concurrency: "1" annotation, however, faced the fact that the application does not start processing a new event while it does not complete the processing of the previous one.
Expected Behavior
Application should start processing a new event even if it does not complete the processing of the previous one.
Actual Behavior
Application does not start processing a new event while it does not complete the processing of the previous one.
Steps to Reproduce the Problem
Our pubsub yaml file:
apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: donum-pubsub
spec:
  type: pubsub.kafka
  version: v1
  metadata:
    - name: brokers
      value: "kafka:9092"
    - name: authType
      value: none
    - name: initialOffset
      value: "oldest"

https://discord.com/channels/778680217417809931/901141467840524289/1075982502986002582

The next person that finds this issue while googling won't need to join the discord to the find the answer. :)

authors: VuiDJi mheguy

repo: dapr/dapr


text:
Appcallback versioning

Description
Add Appcallback versioning for HTTP APIs
We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.
Please reference the issue this PR will close: #1678
Checklist
Please make sure you've  completed the relevant tasks for this PR, out of the following list:

[ *] Code compiles correctly
[ *] Created/updated tests
[ ] Unit tests passing
[ *] End-to-end tests passing
[ ] Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
[ *] Specification has been updated / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
[ ] Provided sample for the feature / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

/ok-to-test

app callback versioning is not needed for topics, binding and method invocation.
Also we need clarity that when 2.0 is implemented, what will be called at startup (app is still implementing c1.0):
v1.0/dapr/config or v2.0/dapr/config
v1.0/dapr/subscribe or v2.0/dapr/subscribe
Also for actors callback, once v2.0 is implemeted (app is still implementing c1.0):
How would runtime determine which version to call?
Could you please set up sometime to discuss it with @yaron2 @youngbupark and me

We decided to not implement appcallback versioning for HTTP. Changes no longer needed

Ack.

authors: amanbha artursouza vinayada1


text:
Utilize NPM @dapr

Describe the proposal
We have now been granted access to the @dapr organization on NPM.
I propose to start using it under @dapr/dapr instead of dapr-client
Since this is a critical change, I propose on maintaining both in the release pipeline where we in the build.yml process setup an extra step that modifies package.json and swaps the name for @dapr/dapr and releases with the new NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN_DAPR }} env variable

@shubham1172 @paulyuk @amulyavarote

Should we use npm-deprecate and stop releasing new dapr-client packages (immediately or after a few releases)? Reference https://stackoverflow.com/questions/28371669/renaming-a-published-npm-module
Till then we can keep the new one, i.e. @dapr/dapr in package.json and write a helper function to publish simultaneously to dapr-client.

I like the idea of npm-deprecate for smooth transition although since this access was granted very recently, if we plan on doing this quickly we may not need to go npm-deprecate route. But overall I think using dapr/dapr instead of dapr-client would feel more in sync with rest of our stuff.

@tanvigour but then the existing projects that use dapr-client will break, right? Unless we tell them in advance that we are moving to another name completely from XYZ date onwards, they won't have time to change their code.

it depends on the rate of adoption. If we already have more than handful users, yes we do not need go npm-deprecate route

I would follow @shubham1172 his idea, allowing people to adopt gradually as NPM packages like to remain on a version.
Personally, I would maintain the dapr-client for around 6 months or until we are able to measure 0 download rate on it

@XavierGeerinck I can think of two options for maintaining dapr-client, once it is deprecated and before we unpublish it:

Create a new branch from the next release for dapr-client and only push security updates/bug fixes for the next 6 months.
Keep updating dapr-client with the same stuff as @dapr/dapr

You can never unpublish a package :) but I agree on those points

I also remember the same, but I stumbled upon this https://docs.npmjs.com/unpublishing-packages-from-the-registry
Looks like it is possible, given the following conditions:

@paulyuk can you confirm on this plan and what dates will be ideal for deprecation and unpublishing?

@paulyuk can you please confirm on this plan and what dates will be ideal for deprecation and unpublishing?

Hi there.  This sounds like a breaking change.  The way I would do that is provide a new version and allow users time to adopt it.  I agree with Xavier we should use data/telemetry to measure people moving off of the current version.  We need to make this change very clear and visible too IMO.
Now in terms of how to do this best in NPM this will be my first time.  I would weigh what Xavier recommends heavily.  We can also consult with the experts in VS Code too.

Proposed next steps:

Release the new package and announce deprecation plans (say after 6 months)

Both the packages will receive the same updates
For announcements, add to README, pin a GH issue, mention on Discord, Twitter
Add a warning message in old package's build to announce intention/date/plan of action for consumers

On deprecation date, use npm-deprecate to announce archival of the old package (say after 12 months)

Till archival date, the deprecated package will only receive major updates

Monitor the downloads and spread more words if we still see active downloads for the old package. Finally stop pushing any updates to it and archive the package (change the npm-deprecate message to reflect the same).

Thoughts on this? Also cc @yaron2 for ideas.

Fixed by #256

authors: XavierGeerinck paulyuk shubham1172 tanvigour

repo: dapr/js-sdk


text:
Added a new optional section for VS Code specific debugging

Description
Have added new steps to show users how to use VSCode launch configurations for debugging and added changes to the .vscode folder to allow a clone and run experience with the hello-world project.
Issue reference
https://github.com/dapr/docs/issues/1657
Checklist
Please make sure you've completed the relevant tasks for this PR, out of the following list:

[x] The quickstart code compiles correctly
[x] You've tested new builds of the quickstart if you changed quickstart code
[x] You've updated the quickstart's README if necessary

Auto Validation failed since the current link added in the readme by me isn't available in the Dapr Docs yet. Its in this pull request: https://github.com/dapr/docs/pull/1673

@orizohar should we have this as its own quickstart? Does this even belong in the quickstarts? We should probably be IDE agnostic in the quickstarts?

@orizohar should we have this as its own quickstart? Does this even belong in the quickstarts? We should probably be IDE agnostic in the quickstarts?
Love the content, just want to make sure this is the best place for it.

I think it's okay as an optional section at the end of the quickstart especially since VS Code is a very popular IDE. I wouldn't make it its own quickstart. However, @jasonviviano if you are writing a whole page in the docs that runs through this we don't need the detailed steps here, we can just refer them there. I would point out the .vscode files are included in the quickstart and send users to the docs for details

@wcs1only @orizohar  Let me know if the section change aligns more with your idea of keeping it agnostic. Also added the .vscode section I created for a plug and go experience for users. BTW i tested it on my Windows PC and everything works just like i did on my Mac however I ended up encountering a bug that's effecting the newest version of Mac's with M1 here https://github.com/microsoft/vscode/issues/129214 incase any of you want to try it and have a Mac.

Are the .png files used anymore? They are still in the commit, but I don't see them referenced anywhere.

Are the .png files used anymore? They are still in the commit, but I don't see them referenced anywhere.

Yeah just deleted them they were left by mistake.

authors: jasonviviano orizohar wcs1only

repo: dapr/quickstarts


text:
Receive timeout when using module in Microsoft Azure

Hi,
I've not found a lot online about using this within Azure, however I've uploaded the module into Azure, and am trying to connect to an SSH enabled server, but keeping receiving New-SSHSession : Session operation has timed out.
The code is very basic at the moment, and works fine on my local machine
$secpwd = ConvertTo-SecureString "mypassword" -AsPlainText -Force
$mycreds = New-Object System.Management.Automation.PSCredential ("myusername", $secpwd)
New-SSHSession -ComputerName "myservername" -Credential $mycreds -ConnectionTimeout 30 -AcceptKey -Verbose
Invoke-SSHCommand -Index 0 -Command "pwd"
Remove-SSHSession -SessionId 0
Any help appreciated.
Thanks,
Steve

If you try to connect with plink or putty from the same host to the target does it work?
Sent from my iPhone

On Apr 23, 2015, at 11:21 AM, smneal notifications@github.com wrote:
Hi,
I've not found a lot online about using this within Azure, however I've uploaded the module into Azure, and am trying to connect to an SSH enabled server, but keeping receiving New-SSHSession : Session operation has timed out.
The code is very basic at the moment, and works fine on my local machine
$secpwd = ConvertTo-SecureString "mypassword" -AsPlainText -Force
$mycreds = New-Object System.Management.Automation.PSCredential ("myusername", $secpwd)
New-SSHSession -ComputerName "myservername" -Credential $mycreds -ConnectionTimeout 30 -AcceptKey -Verbose
Invoke-SSHCommand -Index 0 -Command "pwd"
Remove-SSHSession -SessionId 0
Any help appreciated.
Thanks,
Steve
—
Reply to this email directly or view it on GitHub.

Hi, Thanks for your response. I haven't tried this, but what I have done is use the error output from this in a try catch clause to determine if the VM is shutdown or not. It seems that shutting down a Linux VM from bash in Azure doesn't communicate back to the Azure management interfaces that it is shutdown.
Calling this command generates either a System.Net.Sockets.SocketException or a Renci.SshNet.Common.SshOperationTimeoutException depending on whether the vm is shutdown or running respectively.
Not the neatest, but it works.
Thanks

authors: darkoperator smneal

repo: darkoperator/Posh-SSH


text:
Fallback to a different cache system (local => distributed)

Let's say we want to use Redis or a similar distributed system as a backend.
We could have a chain of cache providers like using say Badger+Redis, this way Badger could keep the local copy in memory for hot items, but check Redis on a miss. This would allow to optimize network traffic and memory usage across multiple Souin nodes.
Say we have two Caddy+Souin nodes, S1 and S2, each with say 2GB of dedicated RAM locally. We also have a Redis cluster with say 32GB.
How it would work is both S1 and S2 would have local copies of hot items which they can serve very quickly, but on a miss they'd query Redis before passing the request. It would minimize both duplication, memory and network use and involved latency (since the nodes would be self-sufficient in many cases, assuming 80/20 rule) but would still take advantage of a big memory pool and not pass too often because their own cache pool is small. It would work almost as if both nodes have 34GB of RAM each.
The only drawback would be invalidation where you'd need to invalidate all instances in the Souin pool when you purge, but that's already an issue if you're using two local Souin instances AFAIK?
Sidenote: is Redis supported as a backend with Caddy? I see it should be supported with Traefik, but there's nothing in the docs. Nevermind, it's in the cache plugin docs, but not here.

@darkweak ah, great to hear we're on the same page. I think it should be a nice feature, one which is not provided by common caching solutions that I know of / have used. :+1:

Hey @dkarlovi this issue will be in the next release (no ETA yet) 🙂

authors: darkweak dkarlovi

repo: darkweak/souin


text:
Disable FOV dependency?

Hi! Solid mod you've made here! My friends and I really love what you got going on here, only issue is with the FOV dependency. From what we understand, the reason why the FOV mod is listed as a dependency is that there is only the one command that adjusts the FOV value. That is only 1 command out of the 37 commands your mod adds, so it isn't necessarily essential for this mod to work. Would it be possible for you to remove it from the dependency in the manifest?
The reason I ask is because two of our friends have this strange issue where their PCs crash when they have the FOV added to our current mod list.  And with their slow primitive internet (LOL), using a mod manager is the only way they can get access to our mods in a timely fashion. But with mod managers, it forces users to download every dependency listed, and there's no way to bypass it from what we know.
Thank you for reading!

Yeah odd enough you're not the first person to request this. I'll make it a soft dependency in the next update 👍

also I had no idea there was 37 commands now 👀

2.2.3 update removes hard dependency of Fov_Adjust 👍

authors: darmuh jvresol

repo: darmuh/TerminalStuff


text:
Documentation Iterating

Hi maintainers!
I am working on a bot, to integrate it with dart ecosystem (pub, dart documentations, dart services - based on dartpad service) I need apis for accessing them online.
I read the source code but I think this service just serves static files and there are no apis, do you have any plan to create json api urls for some stuff that related to documentations? (Like searching, actually I have no problem with pub and dart-services because they provide public apis and generic services)

As I understand currently the only way is to request the https://api.dart.dev/<channel>/<version>/index.json URL and the dart team themselves use this way in the search box. As you are aware for achieving the latest API doc versions we should wait until redirection but it is ok for my usage =))))

license_source: bigquery


text:
Error formatting needs some work

Error compiling to JavaScript:
DetailedApiRequestError(status: 400, message: Compilation of 0c234b396d2dae804ff7557f97573fead01e37f6 failed with errors: [error, line 12] Expected an expression, but got '}'.)
There are lots of things wrong here, @kasperpeulen do you fancy working some magic?
Things to do:
-> The 'DetailedApiRequestError' is internal technical data from how we did the API request. It shouldn't be exposed to the user.
-> The status number should only be exposed if it is a 5xx series, which indicates that something went wrong with the server, and request the user to file an issue with the contents of their source.
-> The '0c234b396d2dae804ff7557f97573fead01e37f6' is a random name generated by the compilation process and shouldn't be shown.
The only bit that needs to be explained to the user is [error, line 12] Expected an expression, but got '}'.
Reformat this so the bit after the ], but before the . is in white to make it stand out a bit more.
I think that will be a decent set of improvements to start with. Thanks!

@lukechurch It feels a bit to me that those javascript erros are allmost never useful. Most of the time the errors from the analyzer seems to be better, and if the analyser doesn't catch an error, the  it doesn't seem to me that the javascript error makes it any better.
For example, today I tried this code at dartpad:
import 'dart:html';

main() async {
  var url = 'https://developer.mozilla.org/en-US/docs/Web/API/HTMLTextAreaElement/autofocus';
  var response = await HttpRequest.getString(url);
}

The error I got was:
Uncaught Uncaught Error: Instance of 'Event'

The dartium error would be:
GET https://developer.mozilla.org/en-US/docs/Web/API/HTMLTextAreaElement/autofocus 404 (NOT FOUND)
Exception: Uncaught Error: Instance of '_XMLHttpRequestProgressEvent'

Sometimes I wonder if it would be somehow possible to show dartium errors instead of javascript errors. I think we give the people wrong impression what dart debugging feels like if we let people debug javascript errors.

I totally agree that the errors from the compiled JS code are usually not helpful, and will probably scare off new Dart developers.
We could host Dartium in the cloud, of course :)
See also https://github.com/dart-lang/dart-pad/issues/186

We could host Dartium in the cloud, of course :)

I have no idea how that would work, but if that is somehow possible, then I think we should go for that. So maybe something like, if there are javacscript errors, then load the file in dartium instead, and show the console errors that dartium produces.
Something else I was thinking about, maybe consider a version of dartpad that can run in dartium and doesn't need to compile to JS to show the result. I think this would be a nice option for larger files that otherwise take an extremely long time to load in dartpad.

I think the sustainable approach is to work on better errors from the compiled JS code, so that our system works great no matter what browser you are using.

I agree with the long term solution of having better errors from our compiled JS. Hosting Dartium would be a lot of work - completely disproportionate for the benefit, and as @sethladd suggests doesn't solve the problem for users of Firefox.
@kasperpeulen If something takes a long time to load in Dart Pad, that's a bug. Please file it.
What I'd like to do here is to do a small amount of reformatting work to help the user separate whether it is a problem with their code (err 4xx) or a problem with the server (err 5xx).

I agree with the long term solution of having better errors from our compiled JS. Hosting Dartium would be a lot of work - completely disproportionate for the benefit, and as @sethladd suggests doesn't solve the problem for users of Firefox.

Huh? If you host dartium on the server side, and ask for the console log of dartium on the server side, it doesn't matter which browser you use on the client side, right ?
But if it such a huge work, we probably shouldn't do it, but I do think that having quality errors is a huge benefit. If there wouldn't exist something like dartium, I don't think I would be attracted to dart. If you are going to debug javascript, it feels like you are better off learning (and writing) javascript in the first place.
Now when I use dartpad, and I get unreadable javascript errors, I go back to webstorm and load the file in dartium. I think this is a step that I shouldn't have to do.

What I'd like to do here is to do a small amount of reformatting work to help the user separate whether it is a problem with their code (err 4xx) or a problem with the server (err 5xx).

In the example I posted:
import 'dart:html';

main() async {
  var url = 'https://developer.mozilla.org/en-US/docs/Web/API/HTMLTextAreaElement/autofocus';
  var response = await HttpRequest.getString(url);
}

The error I get is, not err4xx, or err5xx, but just:
Uncaught Uncaught Error: Instance of 'Event'
nothing more.

Right now uncaught exceptions bubble out to an JS exception handler in our iframe:
window.onerror = function(message, url, lineNumber) {
  parent.postMessage(
    {'sender': 'frame', 'type': 'stderr', 'message': message.toString()}, '*');
};

It's possible - I'm not sure - that the error text could get better if we run the user's app in a zone, install an exception handler for that zone, and pipe those errors back to dartpad proper. It would require post-processing the user's dart source before we compile it, which could be error-prone.

Errors are better now -

authors: devoncarew kasperpeulen lukechurch sethladd

repo: dart-lang/dart-pad


text:
Change Future to void for initializeDateFormatting method in da…

Change Future to void for initializeDateFormatting method in date_symbol_data_local.dart.
Inside this method, exclusively synchronous functions are called and there is no need to return the future

[ ] I’ve reviewed the contributor guide and applied the relevant portions to this PR.

Contribution guidelines:

See our contributor guide for general expectations for PRs.
Larger or significant changes should be discussed in an issue before creating a PR.
Contributions to our repos should follow the Dart style guide and use dart format.
Most changes should add an entry to the changelog and may need to rev the pubspec package version.
Changes to packages require corresponding tests.

Note that many Dart repos have a weekly cadence for reviewing PRs - please allow for some latency before initial review feedback.

I am inclined to not return a future, to align with the other initializeDateFormatting implementations and make switching seamless.

Closing as there is no activity, and IMO no need to apply the change.

authors: EKoretsky mosuem


text:
"Missing entry point: tool/status.dart" but it's there

pub serve prints

[Info from Reflectable on polymer_elements_demos|ReflectableTransformed]:
Missing entry point: tool/status.dart

When I load a page from the polymer_elements_demos project (for example web/index.html
- reflectable:
    entry_points:
    - web/index.dart
    - tool/status.dart

Maybe related to the directory tool instead of web
This seems not to cause any harm, all seems working fine, still wondering if this needs some fix.

This is really a bug on the reflectable transformer. It expects to be able to see all files but that isn't always the case with pub serve. I don't think the tool directory for instance gets served with just pub serve.

see https://github.com/dart-lang/reflectable/issues/17

I don't think the tool directory for instance gets served with just pub serve.

Right, didn't think of that because despite the warning the app works fine.  I guess pub serve just serves the files in tool untranasformed.

interesting, I would have just expected it to not work at all

I was thinking more about it. pub serve is started by WebStorm and I guess it knows that it has to start pub serve with the tool directory. Then we are back to square one, why does it print this message ;-)

Recall that pub serve normally serves a single directory, by default it is just web, but you can make it serve test or other directories by calling pub serve dir. Could it be that WebStorm is starting multiple pub-serve servers? What URL can you use to fetch the code under tool when WebStorm launches pub-serve in web?

@alexander-doroshko might be able to shed some light. If WebStorm starts pub serve for just one directory, the this would explain the message.

WebStorm starts a separate pub serve process for each root folder requested by browser. So if files both from tool and web are queried then WebStorm starts 2 instances of pub serve on different ports. You can see full information in Pub Serve tool window.
Note that normally in WebStorm you do not access pub serve directly, so you never have its url and port in browser page address. Instead all requests are made to WebStorm internal web server (default port is 63342) that simply behaves as a proxy for pub serve. For example in Pub Serve tool window you may see that there are 2 instances started:
pub serve web --port=65119
pub serve tool --port=56875
In browser you have URLs like this:
http://localhost:63342/ProjectName/web/foo.html
http://localhost:63342/ProjectName/tool/bar.html
WebStorm proxies first request to one instance of pub and the second one - to another. This is transparent to user, user doesn't have to know details about pub serve processes, their ports, etc.

That explains the message.
Thanks @alexander-doroshko.

Hello all,
the reflectable transformer produces that Info message because it gets invoked with a barback settings object (which tells a transformer how pub was invoked) where an entry point is specified (e.g., 'web/index.dart'), but that entry point is not among the assets offered by pub to the transformer. So we are explicitly asked to work on that file, but it doesn't exist (as seen from the transformer). As Alexander explained, this could be because there are several pub processes, or simply because you have some entry points in 'web', but right now you are running pub build test so pub doesn't give us access to any of the entry points in 'web'.
We have kept the Info message in place because it seems dangerous to ignore requests for doing something specific ("please transform 'web/index.dart'") if it is not possible to do so. For instance, the problem could be caused by a misspelled entry point name, and that should not be ignored silently. However, we wish to run an alternate test: I we can simply check for the existence of the specified entry point (as a file on disk, not as an asset delivered by pub) then we can suppress the Info message.
For that, we need support from barback (currently we cannot obtain information about the root directory of the package currently being transformed, so we cannot check for the existence of a file given as a relative path from there). That's what https://github.com/dart-lang/reflectable/issues/17 is about.

:+1: thanks for the detailed explanation.

Hello again,
CL https://codereview.chromium.org/1448333004/ is a new iteration on this issue: It eliminates as many 'Missing entry point' messages as possible (namely the ones where we can verify that the file exists, it just wasn't offered by pub {build,serve} .. to the transformer). It also allows for specifying 'suppressWarnings: missingEntryPoint' to eliminate all messages of that kind, and similarly for other warnings. The CL is included in commit https://github.com/dart-lang/reflectable/commit/2dc5e46ef6aa17daab9240a73f79fd7364c76193.

great :smiley:

authors: alexander-doroshko eernstg jakemac53 sigmundch zoechi

repo: dart-lang/polymer-dart


text:
Created package, delete itself and code

This issue was originally filed by zux...@gmail.com

What steps will reproduce the problem?

Right click "packages [package:] and click "New folder"
Give the folder a name.
Move some code into it or create a new file.
Now if you build your application or debug (or randomly) the code and package disappears, and the code is deleted.

What is the expected output? What do you see instead?
I wanted to see my own package stay there and allow me to create a package like on any other language. Instead the package and code was delted
What version of the product are you using? On what operating system?
Dart Editor version 1.0.0_r30798 (STABLE)
Dart SDK version 1.0.0.10_r30798
Windows 7 SP1 Professional
Please provide any additional information below.
Is there any way of recovering the code I lost here, I could not find it in the project file. I fair amount of time writing that code, I do not want to have to rewrite it.

This issue has been moved to dart-lang/pub#823.

repo: dart-lang/sdk


text:
Analyzer should report a hint when a Future.then() callback fails to return the proper type.

The following code illustrates a mistake I have made frequently, and would like the analyzer to issue a hint about:
import 'dart:async';
Future foo(int i) {
  return new Future(() {});
}
Future<int> bar() {
  return foo(1).then(() {
    /* return */ foo(2).then(() {
      return 5;
    });
  });
}
I meant to create a method bar() that would perform foo(1) asynchronously, then perform foo(2) asynchronously, then complete the result with the number 5.  However, because I left out a "return" keyword, the closure passed to foo(1).then() returns null.  (Uncommenting the "return" keyword fixes the mistake).
It would be nice if the analyzer could infer that the closure passed to foo(1).then() was intended to return either an int or a Future<int>, and issue a warning to tell me that it's returning null instead.

This has been fixed. The analyzer now reports errors for the above example:
$ dartanalyzer --version
dartanalyzer version 2.0.0-dev.63.0
$ cat 18531.dart
import 'dart:async';

Future foo(int i) {
  return new Future(() {});
}

Future<int> bar() {
  return foo(1).then(() {
    /* return */ foo(2).then(() {
      return 5;
    });
  });
}
$ dartanalyzer 18531.dart
Analyzing 18531.dart...
  error • The argument type '() → Null' can't be assigned to the parameter type '(dynamic) → FutureOr<int>' at 18531.dart:8:22 • argument_type_not_assignable
  error • The argument type '() → int' can't be assigned to the parameter type '(dynamic) → FutureOr<dynamic>' at 18531.dart:9:30 • argument_type_not_assignable
2 errors found.


text:
Add hint for incorrect assignment based on propagated types.

This code:
  class A {}
  class B {}
  main() {
    Map<String, A> c = <String, A>{};
    c['foo'] = new B();
  }
produces the warning "A value of type 'B' cannot be assigned to a variable of type 'A'", as expected.
This code should produce the equivalent hint, but it doesn't:
  class A {}
  class B {}
  main() {
    var c = <String, A>{};
    c['foo'] = new B();
  }

This works with Strong Mode. https://dartpad.dartlang.org/2df2687e6c8032763f26d8af5e53b213


text:
Specification: Non-nullable type annotation syntax

This is the specification-specific issue for #27231. That issue has the details.

No milestone now: This is not part of Dart 2, but will be considered later.

authors: eernstg munificent


text:
Change the default of pub get/update --packages-dir to false

[ ] Complete https://github.com/dart-lang/pub/issues/1450
[ ] Pull latest pub into SDK

How is this going?

See https://github.com/dart-lang/pub/issues/1450#issuecomment-249731843 – @munificent is "on it"

https://codereview.chromium.org/2377063002/

Marking pub work item complete. We still need to merge this to 1.20.

Closed by https://github.com/dart-lang/sdk/commit/c20e2a0883a43a75ac18d5db6f8d32dfabf34f83

authors: kevmoo mit-mit munificent


text:
dart2js should overwrite the output file rather than appending

Currently, if the output file passed to dart2js already exists, it will append to that file rather than overwriting it. This doesn't match the behavior of most other compilers, and can cause unexpected problems in build systems. We ran into this in Dart Sass, and the only visible symptom was that repeated compilations got slower and slower for no discernable reason.

Sorry, false alarm—something else (as yet undetermined) was causing the slowdown.


text:
building dartlang on solaris fails

I try to build dartlang following https://github.com/dart-lang/sdk/wiki/Building but it fails on my solaris zone.
Everything is available (g++, python27,....) but the build fails with:
Unknown platform: sunos5
There are a lot of files which are checking on the os, so a list where solaris would need to be added int the code (or a new build script) would be nice.

Solaris is not a supported operating system for Dart VM at the moment.

Difficult to understand. Solaris doesn't necessary mean sparc. It should be possible to run this on x86_64 solaris as well.
If the code available, it should be possible to build it.
Can you indicate which specific code is needed, as this can be easily foreseen.

There are plenty of OS specific code in the VM in addition to architecture specific one.
Certainly, supporting x86_64 Solaris is simpler than supporting Sparc. But you will still need to look through a bunch of OS specific code and make it work for Solaris, e.g. some files that need to be copied and specialized for Solaris.
https://github.com/dart-lang/sdk/blob/master/runtime/vm/os_linux.cc
https://github.com/dart-lang/sdk/blob/master/runtime/vm/os_thread_linux.cc
https://github.com/dart-lang/sdk/blob/master/runtime/bin/process_linux.cc
https://github.com/dart-lang/sdk/blob/master/runtime/bin/socket_base_linux.cc
https://github.com/dart-lang/sdk/blob/master/runtime/platform/utils_linux.h

I will check out this code.
I also saw that in a lot of py-scripts the os check is done over and over again.
Which ones are necessary to adapt to add the sunos5 so that the build can succeed?

The best strategy might be just to search for linux in build scripts and put sunos5 next to it everywhere.

authors: mraleph vanpupi


text:
DartDevCompiler fails without any helpful information

While attempting to build a library using the Dart Dev Compiler:
pub build --web-compiler=dartdevc --all --verbose --trace

an error is encountered with no hint on where the issue is:
#left out other parts for brevity
MSG : Building dartdevc modules...
FINE: Building dartdevc modules finished (20.385s).
ERR : Build failed.

This library passes the static analysis checks by the Analyzer package with strong-mode enabled.  Is there any other way to get more details on the reason this is failing?

@jakemac53 - this should be filed on build?

This looks like its going through pub not build, can you run with --verbose and link a gist with the entire contents of the log?

@dustinlessard-wf We don't expect to offer much support for DDC through pub build/serve at this point, due to the fact that build_runner offers much better (and long-term) support. Could you try that instead?
(Feel free to re-open if I'm mistaken)

authors: dustinlessard-wf jakemac53 matanlurey vsmenon


text:
Observatory: stop using pkg:browser

Should no longer be needed.
<script async type="application/dart" src="main.dart"></script>
<script async src="packages/browser/dart.js"></script>

to
<script defer src="main.dart.js"></script>

Also cleanup
runtime/observatory/.packages:6:browser:../../third_party/observatory_pub_packages/packages/browser/lib/
runtime/observatory/pubspec.lock:30:      path: "../../third_party/observatory_pub_packages/packages/browser"
runtime/observatory/pubspec.yaml:26:    path: ../../third_party/observatory_pub_packages/packages/browser
runtime/observatory/web/index.html:10:  <script async src="packages/browser/dart.js"></script>

We're going to deprecated pkg:browser – we need to remove all usage from the SDK
CC @a-siva @devoncarew

This seems to be breaking dart2js tests on firefox and safari, including observatory tests.

/cc @terrylucas @sigmundch

I made a comment on the CL - I believe the issue is not about removing package:browser, but about removing web_components. The webcomponents polyfill is necessary in old browsers like IE11 for anything using custom elements, so my guess is that removing that package is causing a lot of the issues.
@kevmoo - I recall you were trying to remove all dependencies on web_components, but can we? It seems it was removed from the DEPS a while back, but indirectly it was available in the repo is because it was pulled in by the observatory-packages dependency. We however have tests in html that rely on the polyfill script, so my thinking is that we should bring it back as an explicit dependency in DEPS. Separately, if observatory tests need it too, we should probably include it in the observatory_pub_dependencies as well.

I don't believe observatory needs it, we did a round of clean up couple of year back to remove all usage of web components from observatory.
If some other test was using the side effect that observatory-packages was pulling in web_components then the tests need to ensure that web_components is explicitly pulled in.

I believe the cleanup remove all uses of polymer, but it seems that observatory is built using a lot of custom elements (classes that extend HtmlElement), and use the custom element API (https://github.com/dart-lang/sdk/blob/master/runtime/observatory/lib/src/elements/helpers/tag.dart#L23)
While this is part of dart:html, this API works in modern browsers out of the box, but it needs the webcomponents-lite.js polyfill on old browsers.

Just had a quick chat with @kevmoo to clarify about next steps.
Here are my suggestions:

readd web_components to observatory - since it is used by the app and it is needed to test it in old browsers, it seems accurate to keep it around. If we do this first, we can unbreak the broken bots faster, then we can work on disentangling this dependency between html & observatory (which I also believe shouldn't exist).
make webcomponents-lite.js available in the SDK repo to use it in html tests (and hence remove that odd dependency). One way would be to readd web_components to sdk/DEPS (note that it is now moved to the dart-archive org, so we'd need to create a mirror for it pointing at https://github.com/dart-archive/web-components). Other ideas are to copy webcomponents-lite.js directly in the SDK repo.

Thoughts?

regarding use of custom elements API in observatory, we would like to get rid of it but yes until that is done we will add web_components back to observatory-packages.
and yes please open a new issue for adding web_components to sdk/DEPS so we don't go through the same failures when we remove web components from observatory after cleaning up custom elements API..

Thanks Siva. Filed: https://github.com/dart-lang/sdk/issues/32742

One other item to consider, do we really want observatory to be supported on older browsers?

I don't think we need to support older browsers.

good point, I agree - this is mostly a development tool. Similary DDC only works on modern browsers that support ES6

I will turn off these observatory tests on the older browsers and not bother with restoring web_components.

Turning off of observatory_ui tests on older browsers has landed. Closing this issue.

Fixed the dart2js builders requesting this file in https://dart-review.googlesource.com/c/sdk/+/60845

authors: a-siva dgrove kevmoo sigmundch whesse


text:
Default to "sync async" for DDC

Today, apps must opt into the new behavior.  Internally, there is a small whitelist of apps that don't work with the new behavior (and don't opt in).  We can stage as follows:

[x] Explicitly set behavior to on / off (rather than opt-in only command).
[ ] Internally, always explicitly set on or off based on whitelist (ignoring default).
[ ] Change default to new behavior.
[ ] Remove flag when whitelist is empty (won't need to gate Dart 2)

We may need to provide an opt-out in package:build once we flip the default.  Would rather avoid if possible though.  @jakemac53 @natebosch

Is there a flag already in dartdevc? If so we could add the option ahead of time and release the default flip  as a breaking change in build_web_compilers which would be preferable.

@vsmenon

@jakemac53 once you have an SDK with the above commit, then you can add:
dart_sdk.dart.setStartAsyncSynchronously(false);

here:
https://github.com/dart-lang/build/blob/a8565263013d1215bd6ed16337d24edd98c6b479/build_web_compilers/lib/src/dev_compiler_bootstrap.dart#L126
to preserve the old behavior, and flip to true to enable.

authors: dgrove jakemac53 vsmenon


text:
Missing error on static vs. non-static setter/member

It is a compile-time error if a class contains a static getter and a non-static member with matching name, or vice versa.
Caught by language_2/bad_override_test.

I think this issue can be closed. We've implemented the stuff from https://github.com/dart-lang/sdk/issues/33235#issue-326617285. Those should be the new rules.
Re-open if I'm wrong.

authors: askeksa-google jensjoha


text:
Const error when chaning method invocations

I'm not sure if this is a bug or not, but it seems strange to me.  The following program will fail to compile:
class Foo {
  const Foo();

  void resolve() {
    print('resolve');
  }
}

void main() {
  const Foo foo = Foo()..resolve();
}

But final Foo foo = const Foo()..resolve(); is ok, and const Foo foo = const Foo(); foo.resolve(); are both ok.  I would expect the compiler to desugar the failing version into the last version there.

Duplicate of https://github.com/dart-lang/language/issues/542#issuecomment-525019371

I don't think this is a duplicate of #542, but it is similar.
The reason this program does not compile is that Foo()..resolve() is not a compile-time constant expression. It cannot be because calling a method never is.
The problem is not (as in #542) that the initializer refers to the new variable foo before it's initialized. There is also no risk that the const Foo() object is seen in an uninitialized state because it is immutable when created.
Rather, the problem is that const declarations really do not occur as part of the normal evaluation flow.
For a local variable, we could perhaps find a way to split Foo()..resolve() into a constant part and a run-time part which is evaluated ... every time the declaration is evaluated? Even if we find a way to specify it meaningfully, it's a very specialized feature which is unlikely to pay for its own complexity cost.
It would not work for a top-level const variable. If you did const foo = Foo()..resolve(); at top-level (or as a static), it's very, very unclear when and how often the resolve would get called.

I assumed because of how constant evaluation is described in the linked issue, this would require the other

authors: dnfield lrhn


text:
dart2js crash in set literal following throw

main() {
  [throw 1, {...[], ...[]}];
}

Unhandled exception:
Assertion failure: Cannot find value local(main#null) in () for j:method(main).
#0      failedAt (package:compiler/src/diagnostics/invariant.dart:49:3)
#1      LocalsHandler.readLocal (package:compiler/src/ssa/locals_handler.dart:331:11)
#2      KernelSsaGraphBuilder.visitVariableGet (package:compiler/src/ssa/builder_kernel.dart:3482:29)
#3      VariableGet.accept (package:kernel/ast.dart:2996:44)
#4      KernelSsaGraphBuilder.visitBlockExpression (package:compiler/src/ssa/builder_kernel.dart:3620:16)
#5      BlockExpression.accept (package:kernel/ast.dart:5306:44)

Fixed via a85c8b75204ba8879b0220bf232b94d1ff8bf48c


text:
Include a simple README.txt in $HOME/.dart

Tell folks looking around their directory

when this directory was created
by what tool
What it's for – with a link to our web site?

CC @jwren @mit-mit

If the idea sounds good to you @mit-mit, please provide a the README.md contents here and I'll make the change.

Current contents:
$ ls -l ~/.dart
total 8
-rw-r--r--  1 mit  primarygroup   0 Jun 25 14:56 analytics.json
-rw-r--r--  1 mit  primarygroup  97 Jun 25 14:49 dartdev.json

So this is basically a Dart-wide directory containing various settings.

So this is basically a Dart-wide directory containing various settings.

Yup.
One could argue we could/should move .pubCache here, too! Not now...but...someday

Draft CL in https://dart-review.googlesource.com/c/sdk/+/158521

@mit-mit @kevmoo any changes before LGTM so we can land the PR and close out this issue?

+1'ed the cl

authors: jwren kevmoo mit-mit


text:
Analyzer Feedback from IntelliJ

Not able to see FlatButton properities
Version information

IDEA AI-202.7660.26.42.7322048
2.12.2
AI-202.7660.26.42.7322048, JRE 11.0.8+10-b944.6842174x64 JetBrains s.r.o, OS Windows 10(amd64) v10.0 , screens 1920x1080

Thanks for filing an issue. Please explain in more detail the issue you are facing. How do you reproduce the issue? What is the expected behavior? What is the actual behavior?


text:
How to make the result of strong-mode (implicit-casts/implicit-dynamic) analysis as warning, instead of error?

My project has been developed for almost a year, and I didn’t use stricter static checks at the beginning, which resulted in a lot of bugs.
Now I want to add strong-mode (implicit-casts/implicit-dynamic) to analysis_options.yaml, But after that, 5k errors were displayed!

I can't modify them all at once. Can I display this error as warning? In this way, I will pay attention to this warning in future development, and the historical code not need modified temporarily.
thanks.

I haven't solved this problem. It was closed because a new issue was created in dart-lang/sdk.
There are some problems with adding analysis_options.yaml in sub-directory:

We also need to add pubspec.yaml in the subdirectory, otherwise the analyzer will not execute.
If pubspec.yaml is added, the sub-directory will as a package.

@wigl you can change the severity of different error codes. See this page. You might set some codes to ignore temporarily, and you can work your way through the different codes enabled by those analysis options.

authors: srawlins wigl


text:
language server: Option to auto-fill required param keyword for non-null arguments

Wanted to open an issue for discussing a potential flag to the language server / analysis server for automatically filling the required keyword when non-nullable arguments are specified.
As far as I know, this would be the first semantic-source-altering feature for the analysis server. I don't know if there has been an explicit decision to never implement something of the sort.
Discussion of language-level solutions:

required argument inference: https://github.com/dart-lang/language/issues/878#issuecomment-925136971
f({ int! param }) syntax alternative: https://github.com/dart-lang/language/issues/1103#issuecomment-1009115504

The auto-fill feature should be discussed first, but as an aside, I think an elegant approach to such a feature would be to have it be bi-directional, so that an editor could toggle between "final source" and "truncated/tersified source." I doubt that would make sense if this is the only syntax rewrite that ever gets implemented though.

I think there are some interesting challenges to implementing something like this in the analysis server, and it isn't clear to me that doing so would be good for the user experience, but we do have another tool (dart fix) that might be a reasonable place for implementing such behavior.
We already have a fix to add the required keyword, but it isn't currently enabled for bulk application. The reason is that there are other equally valid fixes so we can't know which one the user is likely to want to have applied in a given situation. If we had a way for the user to specify which fix to apply for a given diagnostic, then we could apply that fix as part of bulk fixing.

wherever it ends up, as long as it  can be applied efficiently on file save it should be fine / satisfy this use-case, but I feel like that might be awkward to do with dart fix unless there are pre-existing plans to expand its API surface area / intended usage.

authors: bwilkerson micimize


text:
pkg/modular_test/test/* flaky -> RuntimeError

The tests
pkg/modular_test/test/find_sdk_root1_test RuntimeError (expected Pass)
pkg/modular_test/test/find_sdk_root2_test RuntimeError (expected Pass)
pkg/modular_test/test/loader/loader_test RuntimeError (expected Pass)

are failing on configurations
unittest-asserts-release-mac
unittest-asserts-release-win
unittest-asserts-release-linux

Example log:
--- Command "vm" (took 760ms):
DART_CONFIGURATION=ReleaseX64 CHROME_PATH="/opt/s/w/ir/third_party/browsers/chrome/Google Chrome.app/Contents/MacOS/Google Chrome" xcodebuild/ReleaseX64/dart-sdk/bin/dart --enable_asserts --sound-null-safety -Dtest_runner.configuration=unittest-asserts-release-mac --ignore-unrecognized-flags --packages=/opt/s/w/ir/.dart_tool/package_config.json /opt/s/w/ir/pkg/modular_test/test/find_sdk_root1_test.dart

exit code:
255

stdout:
unittest-suite-wait-for-done

stderr:
Unhandled exception:
error: cannot find the root of the Dart SDK
#0      findRoot (package:modular_test/src/find_sdk_root.dart:19:7)
#1      main.<anonymous closure> (file:///opt/s/w/ir/pkg/modular_test/test/find_sdk_root1_test.dart:13:63)
#2      asyncTest (package:async_helper/async_helper.dart:85:11)
#3      main (file:///opt/s/w/ir/pkg/modular_test/test/find_sdk_root1_test.dart:12:3)
#4      _delayEntrypointInvocation.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:291:19)
#5      _RawReceivePort._handleMessage (dart:isolate-patch/isolate_patch.dart:184:12)

--- Re-run this test:
python3 tools/test.py -n unittest-asserts-release-mac pkg/modular_test/test/find_sdk_root1_test

/cc @sigmundch

Is web-infrastructure the right tag for issues like these?

Could be web-dev-compiler? We wrote the modular_test framework just to have coverage of modular builds that are so important for DDC. We then leveraged it to start testing modular and sequential dart2js builds as well. So, thus far, only the web backend teams are making use of this framework.

Seems this is failing due to a change to infrastructure. The test are extremely simple: they are looking at the directory that contains the test file, and they expected to be of the form some/path/to/sdk/pkg/modular_test/test/...
Then they look up the sdk segment. It seems these bots checked-out the sdk without the sdk folder for some reason
@athomas @whesse is that expected? It is possible to preserve the SDK structure in these runners, or do we need to amend the logic used by these tests somehow?

Looking at the history: https://dart-ci.web.app/#showLatestFailures=false&test=pkg/modular_test/test/find_sdk_root1_test&configurations=unittest-asserts-release-mac
It appears the flakes/failures started occurring as we started running these tests in shards. Seems the main runner has the right folder name, but that the shards copy the contents of the sdk into a folder that is not named "sdk". I can imagine this being labeled as flaky depending on how the sharding split the tests: if the test was picked up by the main test runner (since it acts as one of the shards) the test would pass, if it was picked up by a shard, it would fail.

Fix underway: https://dart-review.googlesource.com/c/sdk/+/316642

authors: alexmarkov sigmundch srujzs


text:
[dart2js] fix precedence issues in js_ast

The js_ast implementation currently doesn't respect precedence in several scenarios:

new expressions: new a.b().c() vs (new a.b()).c()
exponentiation: -1 ** 2 vs (-1) ** 2
for-initializers: for (i = (0 in o) || (1 in o); x; y) vs for (i = (0 in o || 1) in o; x; y)

We never came across these issues in the past because they can't be triggered directly by dart2js generated code. However, they can be triggered by new code patterns introduced via dart:js_interop and its corresponding lowering operations. The precedence for new expression is especially critical because it's a result of a common declaration for factory interop constructors.
Here are example test repros for each case. Note that some of these issues only occur with --omit-implicit-checks, which causes subexpressions to be combined into a single JS expression in the JS-ast:
new: only fails with --omit-implicit-checks, instead of calling new (globalThis.f().classA)(), calls (new globalThis.f()).classA():
import 'dart:js_util';

void main() {
  callMethod(globalThis, 'eval', ['''
      class A {}
      self.f = function F() { return {ClassA: A}; }
      ''']);
  final f = callMethod(globalThis, 'f', const []);
  final o = callConstructor(getProperty(f, 'ClassA'), const []);
  print(getProperty(getProperty(o,'constructor'), 'name'));
}

exponentiation: produces a syntax error
import 'dart:js_util';
main() => print(exponentiate<int>(-1, 2));

for loop: produces syntax error because in is parsed as part of the for loop accidentally:
import 'dart:js_util';
main() {
  final x = newObject();
  for(var o = or(hasProperty(x, 0), hasProperty(x, 1)); o != 1; o = 1) print(o);
}

Note: this is a high priority issue that already affected internal and external developers. It causes issues with the new JS interop that is about to be released in Dart 3.3. Fixing this issue will require a cherry-pick to the beta channel as well.
See also: b/313833546

cc @itsjustkevin FYI

If we think this may be a hard fix, do we see the possibility of doing something different in our interop lowering so that this has less of an impact? One option is to expose a _self in js_util just for this purpose for dart2js.

Regarding your suggestion, could you clarify how the _self is different from globalContext? I was imagining it as a getter as well, which could have the same vulnerability we have today.

We used to use globalThis from js_util which is also a getter before we moved to staticInteropGlobalContext. I'm not sure why it avoided the precedence problem before, but I'm assuming it may have to do with either globalThis being an external, patched member or the prefer-inline pragma. I figured if that avoids the issue, we could use a _self to avoid exposing a public member and continue using JS' self until this issue is resolved.

I see - yeah, I had a hard time reproing the issue because sometimes inlining does the right thing and inlines globalContext. The pragma is there, but there is a chance that it is not properly dealt with while patching the SDK. We should also look into that, or add the annotation also on the public interface if that helps get it right.

A small distinction is that we inline using staticInteropGlobalContext and not globalContext, which just reexports the former, which doesn't have the pragma. If the pragma is the difference, we can try adding one to the former member. Maybe we always had this problem, but the pragma made it very rare.

authors: sigmundch srujzs


text:
Dartdoc does not seem to parse [namedtype]:

Consider the following dartdoc:
class Foo {}

/// See also [Foo]:
/// < some more docs >
class Bar {}

In this case, [Foo]: won't be interactive in the IDE:

Removing the : or adding a whitespace after it fixes the issue:
/// See also [Foo] :
class Bar {}

Tested on Dart SDK version: 3.4.0-175.0.dev (dev) (Sat Feb 24 00:03:30 2024 -0800) on "macos_arm64"

@srawlins @kallentu

This was fixed at some point. We have a test case, test_commentReference_followedByColon, for this.

authors: bwilkerson rrousselGit srawlins


text:
Update checked-in Dart SDK

Can the checked-in dart SDK [1] please be bumped to a version that includes the new dart formatter? 3.7.0-151.0.dev or newer would be great so it contains the latest dart_style fixes from https://github.com/dart-lang/sdk/commit/218057a87c2d23770905f722a249339b4f5426f4.
[1] https://github.com/dart-lang/sdk/blob/bb483f34a3d9787d2f891d2edf4438e36e2ad492/DEPS#L53

cc @munificent @athomas

Alex, you're my only hope. :)

authors: goderbauer munificent


text:
Support named arguments in mock proxies

This issue was originally filed by iz...@gmail.com

In line 1338 of mock.dart there's the next TODO:
          // TODO(gram): Replace all this with:
          //     var rtn = invocation.invokeOn(value);
          // once that is supported.
I believe invokeOn is already implemented. Can you remove the TODO and implement it correctly? Otherwise there's no way to mock method with named arguments.
Thanks.

This issue has been moved to dart-lang/mock#11.


text:
Support latest analyzer in analyzer_plugin

I've raised the upper constraint for the analyzer to <0.41.0.

@googlebot I fixed it.


text:
make social link clickable?

how can I make the social links clickable?
I tried both these syntaxes but I failed :(

\href{http://www.link.com}{clickable word}

clickable word \url{http://www.link.com}

thanks in advance

You can use.
\href{http://link.com}{My awesome link} % this must work
\link{http://link.com}{My awesome link} % This should work

authors: PatimaDA darwiin

repo: darwiin/yaac-another-awesome-cv


text:
DSP-932: Don't allow missing StringLiteralV2 value if language tag given

resolves DSP-932
closes #1418

@benjamingeer Thanks for reviewing this. I just added a general test for directly checking StringLiteralV2. I am gonna remove the unnecessary checks in ListMessagesSpec. Is that Ok for you?

OK that looks good!

authors: SepidehAlassi benjamingeer

repo: dasch-swiss/knora-api


text:
feat!: allow using only one index in a query

Issue being fixed or feature implemented

Since GroveDB doesn’t support indexes intersection, we need to validate query to make sure that only one index is being used.
What was done?

allow using only one index in a query
How Has This Been Tested?

Unit tests
Breaking Changes

Only one index can be used in a query
Checklist:

[x] I have performed a self-review of my own code
[x] I have commented my code, particularly in hard-to-understand areas
[x] I have added or updated relevant unit/integration/functional/e2e tests
[x] I have made corresponding changes to the documentation

For repository code-owners and collaborators only

[x] I have assigned this pull request to a milestone

@shuplenkov we got more clarity on how this thing should work:
All properties in where conditions must much exactly one of the indices and sequence of fields used in where must match how they defined in the index.
In orderBy, it can be only one property from the last condition

authors: shumkov shuplenkov

repo: dashevo/platform


text:
Adds undo to the readme

Undo wasn't listed in the keyboard shortcuts.

My bad, thanks for fixing it!

authors: dasilvacontin formap

repo: dasilvacontin/animus

license: mit


text:
Persistent cache with Zarr

This is somewhat related to both #5 and #3 but slightly different. Basically I would like a persistent cache for use when working in a jupyter notebook. The motivation is very similar to https://github.com/rossant/ipycache i.e. if I restart a notebook I don't want to have to repeat any computations that previously finished. However I would like to use Zarr to store cache results not pickle because compression will save disk space. Also I would like to use a memoize function decorator rather than a cell magic, i.e., something more like the cachey memoize decorator and the joblib Memory.cache decorator.
No problem if this is beyond scope for cachey but I thought I'd mention it in case there were any synergies with other requirements. On the technical side there are two main points to consider: one is how to generate a key from function arguments that is stable across python sessions (i.e., doesn't rely on Python's built-in hash function); the second is how to integrate with Zarr (or similar) for storage.

Zict may also be useful here

It seems like the outputs of a jupyter notebook would be more general than Zarr generally consumes.  Is this correct?

Yes, Zarr currently could only cache arrays, although you could hack around to also store scalar values.
FWIW I just hacked something up for my own use, code here: https://gist.github.com/alimanfoo/ed724d207d859ac507696c3f6735fdf9

Any thoughts on http://zict.readthedocs.io/en/latest/

The composability of zict is very nice. But for my use case I am mostly
wanting to cache results which are numpy arrays, and so I wondered about
using Zarr because serialization/deserialization of arrays is very
efficient. Also zict doesn't provide a memoize decorator, although it could
be adapted to do so. Maybe talking at cross-purposes though, not sure where
you were thinking zict to fit in.
On Tuesday, January 17, 2017, Matthew Rocklin notifications@github.com
wrote:

Any thoughts on http://zict.readthedocs.io/en/latest/
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/cachey/issues/7#issuecomment-273213335, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAq8Qjh6jYxq3tR6Uk7QuD57Mmf3Es4rks5rTOclgaJpZM4LlbmQ
.

--
Alistair Miles
Head of Epidemiological Informatics
Centre for Genomics and Global Health http://cggh.org
The Wellcome Trust Centre for Human Genetics
Roosevelt Drive
Oxford
OX3 7BN
United Kingdom
Email: alimanfoo@googlemail.com
Web: http://purl.org/net/aliman
Twitter: https://twitter.com/alimanfoo
Tel: +44 (0)1865 287721

Generally I think the way to go about this is to let cachey consume a MutableMapping rather than construct its own dict.  Then all of the choices of compression/serialization etc. fall on the user.  Zict composes nicely here because it could be used to construct MutableMappings, even from zarr functions.
One could do this now.
from cachey import Cache
c = Cache(...)
c.data = my_mutable_mapping

OK, that makes sense.
The other issue is about making persistent keys, i.e., keys that are stable
across different Python sessions. In particular, for the memoize decorator,
this means making a stable hash of positional and keyword arguments. Would
you see this as in scope for cachey?
In joblib.Memory the general approach is to pickle arguments then use
hashlib (i.e., cryptographic hashing), although there's lots of extra logic
for handling arrays, sets, and trying to normalize positional and keyword
args, which I think is a bit over-complicated. In the zarr_cache.py gist I
just linearized positional and keyword arguments then pickled and passed
through md5, but designed so other methods of producing keys could be
plugged in.
On Tuesday, January 17, 2017, Matthew Rocklin notifications@github.com
wrote:

Generally I think the way to go about this is to let cachey consume a
MutableMapping rather than construct its own dict. Then all of the choices
of compression/serialization etc. fall on the user. Zict composes nicely
here because it could be used to construct MutableMappings, even from zarr
functions.
One could do this now.
from cachey import Cache
c = Cache(...)
c.data = my_mutable_mapping
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/cachey/issues/7#issuecomment-273250737, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAq8Qrpixb_63xbbBdnL-FXG1BMG9SI7ks5rTQRVgaJpZM4LlbmQ
.

Is it possible to keep key generation within the choice of MutableMapping?  At the moment cachey consumes python objects for keys (args, kwargs).  Presumably it asks the MutableMapping for the value at a key like this and the MutableMapping transforms those python objects into the sort of key that it needs in order to access its internal data structure.  If its internal data structure is just a dict then this is trivial.  If it is an on-disk file then it has to be more complex as you suggest.  At the end of the day though, I think that cachey might be able to say "I don't care.  It's up to the MutableMapping how it wants to handle this."

Yes I guess much of this could be pushed behind the MutableMapping
interface (it hides a multitude of sins :-). The only thing then to
standardize on the cachey side is how to form a key from args and kwargs.
Did you mean to just use a tuple of (args, kwargs)?
On Wednesday, January 18, 2017, Matthew Rocklin notifications@github.com
wrote:

Is it possible to keep key generation within the choice of MutableMapping?
At the moment cachey consumes python objects for keys (args, kwargs).
Presumably it asks the MutableMapping for the value at a key like this and
the MutableMapping transforms those python objects into the sort of key
that it needs in order to access its internal data structure. If its
internal data structure is just a dict then this is trivial. If it is an
on-disk file then it has to be more complex as you suggest. At the end of
the day though, I think that cachey might be able to say "I don't care.
It's up to the MutableMapping how it wants to handle this."
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/cachey/issues/7#issuecomment-273474075, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAq8QlFYUbisSi7TjAJMQ1Joif-DM9_9ks5rThKLgaJpZM4LlbmQ
.

--
Alistair Miles
Head of Epidemiological Informatics
Centre for Genomics and Global Health http://cggh.org
The Wellcome Trust Centre for Human Genetics
Roosevelt Drive
Oxford
OX3 7BN
United Kingdom
Email: alimanfoo@googlemail.com
Web: http://purl.org/net/aliman
Twitter: https://twitter.com/alimanfoo
Tel: +44 (0)1865 287721

Yeah, it looks like currently the memoize decorator takes in a key function and that key function takes in args and kwargs.  Here is the default:
def memo_key(args, kwargs):
    result = (args, frozenset(kwargs.items()))
    try:
        hash(result)
    except TypeError:
        result = tuple(map(id, args)), str(kwargs)
    return result

This is not exactly the best possible answer.  Using dask.base.tokenize here might be a good idea:
In [1]: from dask.base import tokenize

In [2]: tokenize([1, 2, 3], {1: 2, 3: {4}})
Out[2]: 'a46c1482d53199982e0211802af8dee6'

In [3]: tokenize([1, 2, 3], {1: 2, 3: {4}})
Out[3]: 'a46c1482d53199982e0211802af8dee6'

The tokenize function from dask looks like a nice solution to the key
generation problem.
One question, how does a token get generated for an arbitrary Python
object? Couldn't make sense of this:
@normalize_token.register(object)
def normalize_object(o):
if callable(o):
return normalize_function(o)
else:
return uuid.uuid4().hex
On Tuesday, January 24, 2017, Matthew Rocklin notifications@github.com
wrote:

Yeah, it looks like currently the memoize decorator takes in a key
function and that key function takes in args and kwargs. Here is the
default:
def memo_key(args, kwargs):
result = (args, frozenset(kwargs.items()))
try:
hash(result)
except TypeError:
result = tuple(map(id, args)), str(kwargs)
return result
This is not exactly the best possible answer. Using dask.base.tokenize
here might be a good idea:
In [1]: from dask.base import tokenize
In [2]: tokenize([1, 2, 3], {1: 2, 3: {4}})
Out[2]: 'a46c1482d53199982e0211802af8dee6'
In [3]: tokenize([1, 2, 3], {1: 2, 3: {4}})
Out[3]: 'a46c1482d53199982e0211802af8dee6'
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/cachey/issues/7#issuecomment-274895530, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAq8QvhNRIJfmaATNf0URSJtrWj_avTwks5rVkWPgaJpZM4LlbmQ
.

--
Alistair Miles
Head of Epidemiological Informatics
Centre for Genomics and Global Health http://cggh.org
The Wellcome Trust Centre for Human Genetics
Roosevelt Drive
Oxford
OX3 7BN
United Kingdom
Email: alimanfoo@googlemail.com
Web: http://purl.org/net/aliman
Twitter: https://twitter.com/alimanfoo
Tel: +44 (0)1865 287721

We serialize functions with cloudpickle and then hash them down.  For objects that we don't know how to hash we generate a random string.  In the context of caching this doesn't make sense and we would just want to not cache.

Would it not be an option to fall back to pickling objects then hashing the
pickle string?
On Wednesday, January 25, 2017, Matthew Rocklin notifications@github.com
wrote:

We serialize functions with cloudpickle and then hash them down. For
objects that we don't know how to hash we generate a random string. In the
context of caching this doesn't make sense and we would just want to not
cache.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/cachey/issues/7#issuecomment-275099399, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAq8Qh4tDOGZsOOLVAl5tuuD6D7YXfBoks5rV0QbgaJpZM4LlbmQ
.

--
Alistair Miles
Head of Epidemiological Informatics
Centre for Genomics and Global Health http://cggh.org
The Wellcome Trust Centre for Human Genetics
Roosevelt Drive
Oxford
OX3 7BN
United Kingdom
Email: alimanfoo@googlemail.com
Web: http://purl.org/net/aliman
Twitter: https://twitter.com/alimanfoo
Tel: +44 (0)1865 287721

That would certainly be an option.  We probably won't do it in Dask for speed and robustness reasons.  We're pretty comfortable with false negatives.

OK, so a possible architecture...
Cachey provides a Cache class.
You can set any MutableMapping as the value of the data property on an instance of Cache.
A Cache instance provides a memoize decorator, which uses (args, kwargs) as the key when setting and getting items from the cache. It's up to the data mapping how to transform (args, kwargs) into a key used internally.
To get caching which works across Python sessions, implement a zict-style composable MutableMapping class (call it Hash for now) which does the job of turning (args, kwargs) key into some hashed key and then passes through get/set operations to an inner MutableMapping which handles storage.
To get persistent storage of cached values, implement a storage MutableMapping class (call it Store for now) which uses whatever strategy to save values to disk. This could be using Zarr only save any array-like values and barf on any non-array-like values, or it could use Zarr to save array-like values and some other strategy to save other types of values. Point is, details of persistence are hidden behind this layer.
So then you could do something like:
cache = cachey.Cache()
cache.data = Hash(Store('/path/to/cache'))

@cache.memoize
def myfunc(...):
   ...

The Hash and Store mappings could be combined into a single class, but then you would lose flexibility to combine different hashing and storage implementations.
To generalize, there is a three-layer Cache/Hash/Store architecture. The top layer Cache class could be cachey.Cache or zict.LRU or something simple that caches everying. I'm assuming the top layer Cache class also provides the memoize decorator, or maybe that could be decoupled, but needs to live somewhere.
Is this sounding like something worth pursuing? In Cachey? In Zict?
What I need for my own use case is a simple, flexible way to memoize a function that efficiently stores array-like and scalar return values and works across Python sessions. The additional logic of choosing when to evict cached values based on computational cost is actually a bit unnecessary when storing cached values on disk, although maybe not always if amount of stored data is large.

In general I like deferring choices to mutable mappings (as you well know). One question I have about the above procedure is if .data can just be a plain dict.  In particular, it will probably we probably can not send just (args, kwargs), but some transformation on (args, kwargs) that is definitely hashable.
Perhaps we just punt on any non-hashable input?  This would removes our ability to cache mutable data structures (like lists or numpy arrays) by default.
So, I'm +1 on deferring custom decisions to other user defined objects, but -1 on requiring users to construct MutableMappings in the common case.

Yes definitely in the common case the user should not have to construct any
mutable mappings. Should be easy to achieve as you suggest via some minimal
transformation on (args, kwargs) in the memoize decorator.
On Wed, 25 Jan 2017 at 19:25, Matthew Rocklin notifications@github.com
wrote:

In general I like deferring choices to mutable mappings (as you well
know). One question I have about the above procedure is if .data can just
be a plain dict. In particular, it will probably we probably can not send
just (args, kwargs), but some transformation on (args, kwargs) that is
definitely hashable.
Perhaps we just punt on any non-hashable input? This would removes our
ability to cache mutable data structures (like lists or numpy arrays) by
default.
So, I'm +1 on deferring custom decisions to other user defined objects,
but -1 on requiring users to construct MutableMappings in the common
case.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/cachey/issues/7#issuecomment-275206821, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAq8QmIKD2xG4Zie-uiGfM4WRfGJdql6ks5rV6GOgaJpZM4LlbmQ
.

--
Alistair Miles
Head of Epidemiological Informatics
Centre for Genomics and Global Health http://cggh.org
The Wellcome Trust Centre for Human Genetics
Roosevelt Drive
Oxford
OX3 7BN
United Kingdom
Email: alimanfoo@googlemail.com
Web: http://purl.org/net/aliman
Twitter: https://twitter.com/alimanfoo
Tel: +44 (0)1865 287721

I just wanted to comment that this would be a very useful feature for me. Currently, I am automating my computational workflows using Snakemake and make, which then calls python scripts which use dask. The advantage of this is that I can "checkpoint" my the analysis at certain locations in case the workflow fails or because the output of these intermediate steps is interesting. The disadvantage is that I am basically reimplementing the dask graph in my Makefiles, which is very tedious since I have to write file IO boiler-plate for every step.
Solving this issue could make dask a suitable replacement for tools like Snakemake, nextflow, make, etc.

Noah, have you taken a look at the caching that Dask already provides?
http://dask.pydata.org/en/latest/caching.html
On Wed, Mar 22, 2017 at 2:29 PM, Noah D Brenowitz notifications@github.com
wrote:

I just wanted to comment that this would be a very useful feature for me.
Currently, I am automating my computational workflows using Snakemake and
make, which then calls python scripts which use dask. The advantage of this
is that I can "checkpoint" my the analysis at certain locations in case the
workflow fails or because the output of these intermediate steps is
interesting. The disadvantage is that I am basically reimplementing the
dask graph in my Makefiles, which is very tedious since I have to write
file IO boiler-plate for every step.
Solving this issue could make dask a suitable replacement for tools like
Snakemake, nextflow, make, etc.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dask/cachey/issues/7#issuecomment-288495728, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszHmCHLccezJVcOmtSPcQLHh0UNOFks5roWifgaJpZM4LlbmQ
.

I have glanced at that page, but it seemed to me that that is in-memory caching only.
I basically want to be able to manually specify the nodes of a dask graph which should be saved to disk. I deally, my desired  syntax would look something like this:
A = ...
B = really_expensive_or_interesting_computation(A).cached_to_disk("B")
C = moderately_expensive(B)
C.to_hdf5("final_output.h5")
Can the link you sent me be used for that?

Perhaps with an on-disk mutablemapping  like shelve or chest.  Regardless,
I don't think that this is related to Zarr, so we should probably move the
discussion to a different issue.  I recommend starting an issue in
dask/dask, perhaps with a simple example of what you would like to achieve.
On Wed, Mar 22, 2017 at 2:38 PM, Noah D Brenowitz notifications@github.com
wrote:

I have glanced at that page, but it seemed to me that that is in-memory
caching only.
I basically want to be able to manually specify the nodes of a dask graph
which should be saved to disk. I deally, my desired syntax would look
something like this:
A = ...
B = really_expensive_or_interesting_computation(A).cached_to_disk("B")
C = moderately_expensive(B)
C.to_hdf5("final_output.h5")
Can the link you sent me be used for that?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dask/cachey/issues/7#issuecomment-288498438, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszBjMpRroaAZsBN4MQ-apqCKONrAKks5roWq0gaJpZM4LlbmQ
.

Okay, I might start another issue, but the memoization syntax that @alimanfoo proposes above would also probably handle my use-case pretty nicely.

Yah, I would be more than happy with
B = cache.memoize(really_expensive_or_interesting_computation)(A)

I'm a fan of joblib.memory. You can implement storage backends with it. But I'd like for something 'smart' to automatically choose the computations to persist.
There's graphchain to look at but I'm not sure what it adds over dask's caching except for the ability to persist. I also don't know how current it is.

authors: alimanfoo majidaldo mrocklin nbren12

repo: dask/cachey


text:
Dask Backlog Priorities

@jrbourbeau and I took a stab at sorting the backlog on the project board by creating a new column: Sorted Backlog

We tried to create a mix of priorities which spanned array, dataframe, admin, and infrastructure.  We decided that sorting 44 issues would not be too helpful and would probably result in increased decision making when adding new tickets to the backlog.  Instead, a new column allows maintainers to quickly see, set, and readjust the priorities.  I've opened this issue to see if maintainers are in agreement and/or want to explore alternate prioritization schemes.

seems like devs are ok with the sorted backlog column in place -- closing

repo: dask/dask


text:
Cannot compute min or max of dates in dask array when converted from dask dataframe using to_dask_array

What happened:
Computing the minimum date from a dask array causes an exception to be thrown. That dask array was converted from a dask dataframe using to_dask_array.
What you expected to happen:
I expected to be able to compute the minimum date from the dask array.
Minimal Complete Verifiable Example:
using to_dask_array raises an exception
import pandas as pd
from datetime import date
import dask.dataframe as dd

dates_df = pd.Series(pd.date_range(date(2014,1,1),date(2015,1,1), freq="M"))
dates_dd = dd.from_pandas(dates_df, npartitions=1)

dates_da = dates_dd.to_dask_array()
print(dates_da.min().compute())

UFuncTypeError: ufunc 'add' cannot use operands with types dtype('<M8[ns]') and dtype('<M8[ns]')
Using from_array works as expected
import pandas as pd
from datetime import date
import dask.dataframe as dd

dates_df = pd.Series(pd.date_range(date(2014,1,1),date(2015,1,1), freq="M"))

dates_da = dd.from_array(dates_df.values)
print(dates_da.min().compute())

2014-01-31 00:00:00
Environment:

Dask version: 2.16.0
Python version: 3.7.6
Operating System: macOS Catalina 10.15.5
Install method (conda, pip, source): conda

Going through dataframe isn't necessary. This issue is the unknown chunks. This operation fails when discovering the output dtype.
In [71]: a = da.ones((4,), dtype='datetime64[ns]')

In [72]: a._chunks
Out[72]: ((4,),)

In [73]: a._chunks = ((np.nan,),)

In [74]: a.min()
---------------------------------------------------------------------------
UFuncTypeError                            Traceback (most recent call last)
<ipython-input-74-8f6244b3691f> in <module>
----> 1 a.min()

~/sandbox/dask/dask/array/core.py in min(self, axis, keepdims, split_every, out)
   1960         from .reductions import min
   1961
-> 1962         return min(self, axis=axis, keepdims=keepdims, split_every=split_every, out=out)
   1963
   1964     @derived_from(np.ndarray)

~/sandbox/dask/dask/array/reductions.py in min(a, axis, keepdims, split_every, out)
    371         dtype=a.dtype,
    372         split_every=split_every,
--> 373         out=out,
    374     )
    375

~/sandbox/dask/dask/array/reductions.py in reduction(x, chunk, aggregate, axis, keepdims, dtype, split_every, combine, name, out, concatenate, output_size, meta)
    184         name=name,
    185         concatenate=concatenate,
--> 186         reduced_meta=reduced_meta,
    187     )
    188     if keepdims and output_size != 1:

~/sandbox/dask/dask/array/reductions.py in _tree_reduce(x, aggregate, axis, keepdims, dtype, split_every, combine, name, concatenate, reduced_meta)
    249         dtype=dtype,
    250         name=(name or funcname(aggregate)) + "-aggregate",
--> 251         reduced_meta=reduced_meta,
    252     )
    253

~/sandbox/dask/dask/array/reductions.py in partial_reduce(func, x, split_every, keepdims, dtype, name, reduced_meta)
    313     if is_arraylike(meta) and meta.ndim != len(out_chunks):
    314         if len(out_chunks) == 0:
--> 315             meta = meta.sum()
    316         else:
    317             meta = meta.reshape((0,) * len(out_chunks))

~/Envs/dask-dev/lib/python3.7/site-packages/numpy/core/_methods.py in _sum(a, axis, dtype, out, keepdims, initial, where)
     36 def _sum(a, axis=None, dtype=None, out=None, keepdims=False,
     37          initial=_NoValue, where=True):
---> 38     return umr_sum(a, axis, dtype, out, keepdims, initial, where)
     39
     40 def _prod(a, axis=None, dtype=None, out=None, keepdims=False,

UFuncTypeError: ufunc 'add' cannot use operands with types dtype('<M8[ns]') and dtype('<M8[ns]')

My only suggestion is to expose a meta to these methods and pass it through as reduced_meta in dask.array.reductions.reduction. Does that sound reasonable?

My only suggestion is to expose a meta to these methods and pass it through as reduced_meta in dask.array.reductions.reduction. Does that sound reasonable?

yeah that sounds great!

This issue is the unknown chunks

That's strange, I tested it the same "to_dask_array" example with lengths=True to compute the chunk sizes and it still raised the exception. Do you know why that would happen?

Oh sorry, I misread the original example. The second example used dd.Series.min(), I thought it was using da.Array. So the missing chunk size is a red herring
In [25]: a = da.ones((4,), dtype='datetime64[ns]')

In [26]: a.min()
---------------------------------------------------------------------------
UFuncTypeError                            Traceback (most recent call last)
<ipython-input-26-8f6244b3691f> in <module>
----> 1 a.min()

~/sandbox/dask/dask/array/core.py in min(self, axis, keepdims, split_every, out)
   1960         from .reductions import min
   1961
-> 1962         return min(self, axis=axis, keepdims=keepdims, split_every=split_every, out=out)
   1963
   1964     @derived_from(np.ndarray)

~/sandbox/dask/dask/array/reductions.py in min(a, axis, keepdims, split_every, out)
    371         dtype=a.dtype,
    372         split_every=split_every,
--> 373         out=out,
    374     )
    375

~/sandbox/dask/dask/array/reductions.py in reduction(x, chunk, aggregate, axis, keepdims, dtype, split_every, combine, name, out, concatenate, output_size, meta)
    184         name=name,
    185         concatenate=concatenate,
--> 186         reduced_meta=reduced_meta,
    187     )
    188     if keepdims and output_size != 1:

~/sandbox/dask/dask/array/reductions.py in _tree_reduce(x, aggregate, axis, keepdims, dtype, split_every, combine, name, concatenate, reduced_meta)
    249         dtype=dtype,
    250         name=(name or funcname(aggregate)) + "-aggregate",
--> 251         reduced_meta=reduced_meta,
    252     )
    253

~/sandbox/dask/dask/array/reductions.py in partial_reduce(func, x, split_every, keepdims, dtype, name, reduced_meta)
    313     if is_arraylike(meta) and meta.ndim != len(out_chunks):
    314         if len(out_chunks) == 0:
--> 315             meta = meta.sum()
    316         else:
    317             meta = meta.reshape((0,) * len(out_chunks))

~/Envs/dask-dev/lib/python3.7/site-packages/numpy/core/_methods.py in _sum(a, axis, dtype, out, keepdims, initial, where)
     36 def _sum(a, axis=None, dtype=None, out=None, keepdims=False,
     37          initial=_NoValue, where=True):
---> 38     return umr_sum(a, axis, dtype, out, keepdims, initial, where)
     39
     40 def _prod(a, axis=None, dtype=None, out=None, keepdims=False,

UFuncTypeError: ufunc 'add' cannot use operands with types dtype('<M8[ns]') and dtype('<M8[ns]')

Ahh of course, no problem! Do you think this is still fixable with exposing meta to to_dask_array and passing it through using reduced_meta in dask.array.reductions.reduction or do you think its a larger issue?
I'm happy to poke around and come up with a solution, just might be a while because I'm fairly busy at the moment. It seemed to me like a type inference issue in to_dask_array

fixable with exposing meta to to_dask_array

I think it's just in Array.min(), .sum(), etc. Anything going through reduction.

@kylejn27 are you still interested in trying to come up with a fix?

@jsignell Ah! Looks like this slipped out of mind. Unless you're asking because you wanted to take this on, I'm happy to start working on this next week

I think that Julia was just poking stale issues.  If you're down to resolve
this Kyle that would be swell.
On Fri, Sep 11, 2020, 10:58 AM Kyle Nicholson notifications@github.com
wrote:

@jsignell https://github.com/jsignell Ah! Looks like this slipped out
of mind. Unless you're asking because you wanted to take this on, I'm happy
to start working on this next week
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/6354#issuecomment-691234294, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AACKZTHYZ37RJVUSWE37E5TSFJQLXANCNFSM4OLO52UA
.

great! I'll pick it up first thing on monday

Julia was just poking stale issues

Indeed I was! All yours @kylejn27 :) If you want to add a meta to to_dask_array while you're at it, then you'll close #6349  as well.

If you want to add a meta to to_dask_array while you're at it,

sure!

I just realized that I misinterpreted what the proposed solution was in my implementation of #6651, that PR doesn't actually pass meta to reduced_meta it just sets the metadata after the reduction happens.
In my local branch I corrected this and tried to pass a metadata in that I thought would help the issue (da.min(a, meta=np.array([1], dtype=timedelta64[ns])) but realized that the real issue isn't really anything to do with metadata

but that that numpy cannot add `datetime64[ns]` and throws a value error when doing so here:

if is_arraylike(meta) and meta.ndim != len(out_chunks):
if len(out_chunks) == 0:
meta = meta.sum() # <----
else:
meta = meta.reshape((0,) * len(out_chunks))

naively I'd like add a try catch block and pass through the value error. This resolves this issue, but I'm wondering if it would have undesirable effects downstream if added in.

if is_arraylike(meta) and meta.ndim != len(out_chunks):
    if len(out_chunks) == 0:
        try:
            meta = meta.sum()
        except ValueError:
            pass
    else:
        meta = meta.reshape((0,) * len(out_chunks))

If you do that then the dimensionality of the meta stops matching the dimensionality of the output so that's not ideal.

The goal of that meta  = meta.sum() is to reduce the dimensionality of the meta. I can't think of a safe general way to do that for datetime. Specifically in this case we need to go from ndim=1 to ndim=0 and the len is 0 so flatten doesn't work, and np.asscalar doesn't either.

Ah thanks for the input! I'll spend some time thinking on this, perhaps in a different PR than #6651 so we can get that merged in

Thinking about @mrocklin's comment a little bit

I wonder if we can replace the use of sum with an understanding of the axis
and keepdims arguments to achieve the same result.  This would probably
require a little bit of work on our part, but might reduce complexity for
others in the future.

Seems that the point of this block of code is to replace meta with either a single value 0 or a tuple of 0's with length out_chunks.
if is_arraylike(meta) and meta.ndim != len(out_chunks):
        if len(out_chunks) == 0:
            # point of this is to get
            meta = meta.sum()
        else:
            meta = meta.reshape((0,) * len(out_chunks))

My understanding of Dask metadata is a bit weak so I could be wrong but wouldn't it be ok to replace meta = meta.sum() with the following? This gives you a scalar metadata of 0 of dtype type(meta)
if is_arraylike(meta) and meta.ndim != len(out_chunks):
        if len(out_chunks) == 0:
            # point of this is to get
            original_type = type(meta) # preserve original type
            meta = np.int64(0) # get meta into the shape you want, single scalar 0 value
            meta = meta.astype(original_type) # Cast scalar 0 metadata to original type
        else:
            meta = meta.reshape((0,) * len(out_chunks)) # this should work regardless of the type

I think going through np is kind of discouraged since ideally dask doesn't care what the array type of the underlying blocks is (cupy for instance should work). Ideally there would be some method on meta that could create a scalar of the same type.

Did anyone ever come up with a solution or workaround for this from a user perspective? I'm working with some very large xarrays and I'd like to stay in dask-land as much as possible, but for part of my program this issue is forcing me to use a numpy min().

A +1 on this issue. It has also bitten me, with a very similar use case to @claytharrison. My minimum reproducible example is below
import dask.array
import numpy as np

times = np.array(
    [
        "1755-10-20T00:00:00.000000000",
        "1760-09-28T00:00:00.000000000",
        "1766-04-05T00:00:00.000000000",
    ],
    dtype="datetime64[ns]",
)
dask.array.from_array(times).min()

# Output
...
numpy.core._exceptions._UFuncBinaryResolutionError: ufunc 'add' cannot use operands with types dtype('<M8[ns]') and dtype('<M8[ns]')

authors: TomAugspurger claytharrison jsignell kylejn27 mrocklin znichollscr


text:
unmanaged old not released after notebook re-run

What happened:
After run scrpit in notebook with some error, it has about 1G unmanaged old, then I re-run the script,
but the unmanaged old is not released.
What you expected to happen:
I expect the unmanaged old should release.
Minimal Complete Verifiable Example:
The script is same in issues/8941
# Put your MCVE code here
dtype_dic= {0: str, 1: str, 2: str, 3: str, 4: str, 5: str, 6: str, 7: str, 8: str, 9: str, 10: str,
11: str, 12: str, 13: str, 14: str, 15: str, 16: str, 17: str, 18: str, 19: str, 20: str}
%time df2=dd.read_csv('/mnt/output202202/output*.csv', encoding='GBK',dtype=dtype_dic)
df2.columns[0]=['smg_req']
%time df2.set_index('smg_req')

Anything else we need to know?:
Environment:

Dask version: 2022.3.0
Python version: 3.8
Operating System: Cento7
Install method (conda, pip, source): k8s docker hub

Cluster Dump State:

We got out of memory error, the k8s node is down.

@RexHuang Thanks for the issue! Perhaps some solutions in this blog post might be helpful: https://coiled.io/blog/tackling-unmanaged-memory-with-dask/
It'll also be great if you can share a minimal reproducible example, using a toy dataset and local cluster is possible. Also, since this is a usage question, I'd encourage you to open an issue on Discourse (maybe as a reply to your other question in case it's related and we can continue this discussion there!

I'll go ahead and close this due to lack of activity, but please feel free to re-open it if needed!

authors: RexHuang pavithraes