text:
Specify a minimum version for numpy in setup.py.

In dask's setup.py there is no minimum version specified for numpy.
13    'dataframe': ['numpy', 'pandas >= 0.19.0', 'toolz >= 0.7.3',
14                  'partd >= 0.3.8', 'cloudpickle >= 0.2.1'],

This allows a users to get into the situation where a previously installed and potentially outdated version of numpy is in the environment. This causes an issue when importing dask if the numpy version is less than 1.10.0 because np.cbrt was introduced in numpy version 1.10.0,
In [1]: import dask.dataframe as dd
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-1-e012ff17e82a> in <module>()
----> 1 import dask.dataframe as dd

/home/test/.virtualenvs/dask/lib/python3.4/site-packages/dask/dataframe/__init__.py in <module>()
      1 from __future__ import print_function, division, absolute_import
      2
----> 3 from .core import (DataFrame, Series, Index, _Frame, map_partitions,
      4                    repartition, to_delayed, to_datetime, to_timedelta)
      5 from .groupby import Aggregation

/home/test/.virtualenvs/dask/lib/python3.4/site-packages/dask/dataframe/core.py in <module>()
     17     Cache = dict
     18
---> 19 from .. import array as da
     20 from .. import core
     21 from ..utils import partial_by_order

/home/test/.virtualenvs/dask/lib/python3.4/site-packages/dask/array/__init__.py in <module>()
     13                        apply_along_axis, apply_over_axes, result_type)
     14 from .reshape import reshape
---> 15 from .ufunc import (add, subtract, multiply, divide, logaddexp, logaddexp2,
     16         true_divide, floor_divide, negative, power, remainder, mod, conj, exp,
     17         exp2, log, log2, log10, log1p, expm1, sqrt, square, cbrt, reciprocal,

/home/test/.virtualenvs/dask/lib/python3.4/site-packages/dask/array/ufunc.py in <module>()
    136 sqrt = ufunc(np.sqrt)
    137 square = ufunc(np.square)
--> 138 cbrt = ufunc(np.cbrt)
    139 reciprocal = ufunc(np.reciprocal)
    140

AttributeError: 'module' object has no attribute 'cbrt'

There was an existing issue #2596 where another user had the same problem which was closed.
Python environment:
Python version: 3.4.3

Libraries:
cloudpickle (0.4.0)
dask (0.15.4)
locket (0.2.0)
numpy (1.9.1)
pandas (0.20.3)
partd (0.3.8)
pip (9.0.1)
python-dateutil (2.6.1)
pytz (2017.2)
setuptools (36.6.0)
six (1.11.0)
toolz (0.8.2)
wheel (0.30.0)

Copy and paste example:
import dask.dataframe as dd
print('test is complete')

Is there a reason as to why the setup.py could not be modified to contain numpy>=1.10.0?
13    'dataframe': ['numpy>=1.10.0', 'pandas >= 0.19.0', 'toolz >= 0.7.3',
14                  'partd >= 0.3.8', 'cloudpickle >= 0.2.1'],

Seems reasonable to me.  It looks like we also don't mention a minimum version in the conda-forge recipe: https://github.com/conda-forge/dask-feedstock/blob/master/recipe/meta.yaml
The next question is probably "what is the minimum version?"
This issue might interest @jakirkham

Agreed. I guess the oldest one we test in CI, which IIRC is NumPy 1.10. Though I do remember seeing compatibility code for earlier versions. Maybe it's time to revisit what compatibility code is still needed and what can be dropped.

There appear to be errors with the masking functionality with numpy <1.12:
>>> import numpy as np
>>> import dask.array as da
>>> a = np.ma.arange(12).reshape(3,4)
>>> d = da.from_array(a, 1, asarray=False)
>>> print d.mean(axis=1).compute()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../lib/python2.7/site-packages/dask/base.py", line 99, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File ".../lib/python2.7/site-packages/dask/base.py", line 206, in compute
    results = get(dsk, keys, **kwargs)
  File ".../lib/python2.7/site-packages/dask/threaded.py", line 75, in get
    pack_exception=pack_exception, **kwargs)
  File ".../lib/python2.7/site-packages/dask/local.py", line 521, in get_async
    raise_exception(exc, tb)
  File ".../lib/python2.7/site-packages/dask/local.py", line 290, in execute_task
    result = _execute_task(task, data)
  File ".../lib/python2.7/site-packages/dask/local.py", line 271, in _execute_task
    return func(*args2)
  File ".../lib/python2.7/site-packages/toolz/functoolz.py", line 466, in __call__
    ret = self.first(*args, **kwargs)
  File ".../lib/python2.7/site-packages/dask/array/core.py", line 517, in _concatenate2
    return concatenate(arrays, axis=axes[0])
IndexError: axis 1 out of bounds [0, 1)

Considering the masked array functionality is relatively new and may have some rough areas, would not suggest setting the NumPy minimum version based on it. Instead would suggest that bugs related to masked array support be raised in fresh issues where they can be addressed more appropriately.

Have added PR ( https://github.com/dask/dask/pull/3283 ) to set the minimum version of NumPy supported to be 1.10.4 as that is the oldest version we run CI against. Also have seen evidence that some things we include (e.g. wrapping cbrt) don't work on NumPy 1.9. We should probably follow this PR with a bit of spring cleaning as there is back compat code for versions of NumPy earlier than 1.10.

authors: bcarrier2010 djkirkham jakirkham mrocklin


text:
Better repartition behavior with npartitions

If you have source files that are not the same size df.repartition doesn't behave in an intuitive way.  For example.   If I specify
df.repartition(npartitions=4)

where one source file is much smaller than the other you get partitons sizes like this:
2020-08-19 12:39:11        622 0.part
2020-08-19 12:39:12        868 1.part
2020-08-19 12:39:22    6779422 2.part
2020-08-19 12:39:22    6785240 3.part

This is because the way that dask choses to create partitions is appears to be as a subset from how the source files are originally partitioned (as the documentation for partition_size says npartitions does not compute the byte size of the partitions).  However, this does not appear to be the case when using "partition_size".  In order to get the behavior I would expect npartitions to create I have to repartition twice so that much smaller partitions are created and then combined:
df.repartition(partition_size=even_worker_partition_size).repartition(npartitions=4)

2020-08-19 13:00:36    3145730 0.part
2020-08-19 13:00:36    3477110 1.part
2020-08-19 13:00:36    3471711 2.part
2020-08-19 13:00:36    3471601 3.part

But has the obvious penatly of having to repartition twice.

If you intend all partitions to be the same size, then it is best to set that explicitly using partition_size. To get your desired outcome it would make sense for you to determine the total memory usage of the dataframe and then set the partition_size to that divided by the desired number of partitions.

That would only be true if the on disk size was the same as in memory which isn't the case.

I would expect something like this to work ddf.repartition(partition_size=ddf.memory_usage(deep=True).sum().compute() // n)  but it doesn't give preciesly n partitions. You might be interested in a similar issue that came up in dask array: https://github.com/dask/dask/pull/6420. Maybe that approach could be expanded for dask dataframe.

Just checking in here, @kyprifog do you have a suggestion for how the current situation could be improved?

Well, i dont really know.  Ideally it would be better if there were a way to get even partitions without having to repartition twice.   I tried rebalance but had no luck with that approach.  I'll have to spend more time playing around with it.

The second repartition is relatively cheap.  I don't see an obvious thing to improve here.  I'm going to close this for now.  If there is more to discuss then please chime in and we'll reopen.

source: gharchive/issue

authors: jrbourbeau jsignell kyprifog mrocklin

repo: dask/dask

license_type: permissive

license_source: github-api


text:
[docs] Fix formatting in random number generation docs

trivial fix for formatting - highlight both BitGenerators and Generators in docs

[x] Passes pre-commit run --all-files

cc @rjzamora  quick fix for formatting

Mind putting them in a note section instead?

Done. Thanks for the comment

@jrbourbeau reminder for this quick fix in formatting. thanks

source: gharchive/pull-request


text:
Pin boost 1.60 in libhdfs3 conda recipe

Addresses #66.

New libhdfs3 package is available on the dask channel on anaconda.org using this recipe.


text:
RFC: add support for querying whether an array library explicitly supports strides

Overview
The concept of strides has been raised several times since the beginning of the Consortium. This history was summarized in https://github.com/data-apis/array-api/issues/641. In short, because several array libraries don't have an explicit concept of "strides" and strides can be considered an implementation detail, the Consortium has been intentional about avoiding any formal standardization of stride APIs.
Nevertheless, for a subset of array consuming libraries (such as SciPy and scikit-learn), being able to leverage (and manipulate) strides can confer performance advantages for the performance inclined. As a result, downstream libraries wanting to squeeze out additional performance have a higher likelihood of special casing the arrays of recognized libraries. This special casing is fine so long as you have a small number of supported array libraries; however, it presents a material disadvantage to those libraries also supporting strides which are not one of the privileged few.
Proposal
Now that the array API standard has an inspection API, this RFC seeks to propose adding support for querying whether a conforming array library has explicit support for array strides. Specifically, the following field should be added to the object returned by info.capabilities():
{
  ...
  "max rank": Optional[int],
+ "strides": bool
}

With this API in place, downstream libraries could query whether a conforming array library supports strides and special case accordingly.
Questions

Does the proposed inspection API provide enough value to downstream libraries to justify inclusion? Notably, this proposal does not seek to standardize any keywords or stride-specific APIs (e.g., stride_tricks), and, thus, downstream libraries would still be left to check for specific API support. But, this stated, this may be enough of a hook to foster de facto community standardization, of which, among stride supporting libraries, there seems to be.

As discussed in https://github.com/data-apis/array-api/issues/571, DLPack allows access to strides; however, this API is a bit cumbersome for downstream libraries to use just to access strides. Nevertheless, maybe this is enough?

A related question is, if we decide to move forward with this proposal, does it make sense to also add support for querying whether an array API supports an array "order" (e.g., "row-major" or "column-major")? This is another implementation detail which downstream array libraries hoping to adopt the standard have raised with particular concern for performance implications. Especially for CPU-bound dense array libraries which commonly interface with libraries written in other languages (C/C++/Fortran) and which assume a specific memory layout (row or column major), ensuring that created arrays have a particular layout can confer performance advantages.

Related

https://github.com/data-apis/array-api/issues/641
https://github.com/data-apis/array-api/issues/571

I'm not sure how this will help, given:

Does the proposed inspection API provide enough value to downstream libraries to justify inclusion? Notably, this proposal does not seek to standardize any keywords or stride-specific APIs [...]

If anything that has to do with strides has to be queried for with hasattr anyway, then doing has_strides = hasattr(x, 'strides') should be fine too.
The discussion in gh-641 points at defining APIs like sliding_window_view for the operations that people want to manipulate strides for. That seems like an important exercise, and is probably needed before deciding on this proposal. A good start would be to try using sliding_window_view for everything in SciPy and scikit-learn that manipulates strides, and seeing how far that gets us.

Does the proposed inspection API provide enough value to downstream libraries to justify inclusion? Notably, this proposal does not seek to standardize any keywords or stride-specific APIs [...]

If anything that has to do with strides has to be queried for with hasattr anyway, then doing has_strides = hasattr(x, 'strides') should be fine too.

From Python library perspective I am supportive for this RFC. hasattr(x, 'strides') assumes there's an attribute named strides, which may or may not be the case. For example, PyTorch makes it a function without "s": torch.Tensor.stride(). While this RFC does not answer the question "if a library has strides, how do I query it?", sometimes a yes/no is enough (and if yes, we could then proceed to get the strides via .strides, from a DLPack capsule, etc). Though, this RFC doesn't answer this question either: "is the returned strides in unit of bytes or counts?", but DLPack has it defined.

If DLPack already exposes strides then is it possible to do whatever stride tricks you would want to do with DLPack?

If we want to say "having __dlpack__ and __dlpack_device__ implemented is the proxy to check if strides exist," I am fine with it too, but we should document this. But then, it seems to be unintuitive for general users. Thus having a yes/no answer queryable to me is a good compromise.

hasattr(x, 'strides') assumes there's an attribute named strides, which may or may not be the case. For example, PyTorch makes it a ~function~ method without "s": torch.Tensor.stride(). While this RFC does not answer the question "if a library has strides, how do I query it?", sometimes a yes/no is enough

That's not really a good reason. A simple utility function like this will do:
def has_strides(x):
    return hasattr(x, 'strides') or hasattr(x, 'stride')

There are significant downsides to introducing strides as a concept in the standard, and zero upsides as far as I can tell - aside from avoiding having to carry a simple utility function like this. Also, this utility function is going to work for older versions of numpy and other array libraries (which everyone needs), while a new inspection API won't.

If we want to say "having __dlpack__ and __dlpack_device__ implemented is the proxy to check if strides exist,

That doesn't work I think. These methods are implementable by most libraries, and strides will then always be defined (not accessible from Python) even if the library does not offer non-contiguous arrays. Concrete example: JAX arrays have a __dlpack__ method, but x[::2] still copies. For the things folks want, it's the "can I do things with strided views efficiently without triggering copies".
Or maybe it's more like there are several different capabilities here that may require determining:

Are non-contiguous ("strided") arrays supported
Is it possible to do memory-efficient data read operations (a la numpy.lib.stride_tricks.as_strided)
... ?

Looks like I dropped the ball... 😅
From my perspective as a low-level library developer trying to interface with Python (e.g. nvmath-python is live), I don't care if DLPack or the array library describes a view or a copy. All I care is to have a way to extract and pass the memory layout from an array/tensor to my C libraries for invoking the math ops, and for that I need the strides information not just the pointer. I don't want/need to pick sides on whether to support views or the strides concept in Python. Does it make better sense?

Does it make better sense?

Yes, I think so. Based on the discussion above, my conclusion is that we should add some language to a section in "Design topics and constraints" on strides, with a pointer for anyone interested in getting strides information (as in Leo's case) to leverage DLPack. This should be enough to satisfy most needs, even if a bit unintuitive, while not introducing the concept of strides to the standard.

authors: asmeurer kgryte leofang rgommers

repo: data-apis/array-api

license: MIT


text:
Add Python API, semantics and implementation details for DLPack

Related to the discussion at https://github.com/data-apis/consortium-feedback/issues/1. This addresses all open items on that discussion. It's kind of tricky to figure out what to add here though, and what to refer to DLPack itself for.
We should upstream some of this documentation to DLPack as well, so this becomes a summary. Right now DLPack doesn't have any documentation on Python-level API, and some of the other content here isn't very clearly documented yet either (it mostly came from explanations of @tqchen in the issue linked above).
Given that reviewers may not want to build the Sphinx docs, here's a screenshot of the last half (which has visuals):

I think this is complete enough for now. https://github.com/dmlc/dlpack/issues/57 is converging. I'd like to merge this PR if everyone is happy with that, so it shows up in the rendered html docs. And then we can focus on summarizing https://github.com/dmlc/dlpack/issues/57 in that repo, I think that should cover the topic.

I think this is complete enough for now. https://github.com/dmlc/dlpack/issues/57 is converging. I'd like to merge this PR if everyone is happy with that, so it shows up in the rendered html docs. And then we can focus on summarizing https://github.com/dmlc/dlpack/issues/57 in that repo, I think that should cover the topic.

I would like to render the document and read through this, please. Please allow me a couple of hours.

I would like to render the document and read through this, please. Please allow me a couple of hours.

I'll try to revisit this tonight.

I'll try to revisit this tonight.

It might be useful to map None to (legacy) default stream (the default configuration of cuda/rocm's stream 0). Since no synchronization might be ambiguous in GPU case.
For the case when producer's data comes from a non-default stream. the producer need to sync the data to the default stream. No syncing would leads to undefined behavior.

It might be useful to map None to (legacy) default stream (the default configuration of cuda/rocm's stream 0). Since no synchronization might be ambiguous in GPU case.
For the case when producer's data comes from a non-default stream. the producer need to sync the data to the default stream. No syncing would leads to undefined behavior.

Thanks @tqchen, makes sense. If that'd really be confusing with Numba as @leofang says, the alternative may be to specify that stream must not be None if the consumer supports CUDA/ROCm.

- None: CUDA/ROCm streams are not supported,

instead of

- None: no synchronization (default),

or

None: legacy default stream (default),

That may be a bit more explicit.

Thanks @tqchen, makes sense. If that'd really be confusing with Numba as @leofang says, the alternative may be to specify that stream must not be None if the consumer supports CUDA/ROCm.

- None: CUDA/ROCm streams are not supported,

instead of

- None: no synchronization (default),

or

None: legacy default stream (default),

That may be a bit more explicit.

Thanks @rgommers .
I believe the main purpose of default behavior is to serve as a recommendation of starting point to the developers. Option1 means we do not have a recommendation for developers in the case of CUDA/Rocm.
On the other hand, there is a common starting point for developers to develop GPU kernels. I learn GPU programming by launching kernel without the stream argument. In those cases, the behavior relies on the legacy default stream. Although the stream 0 have ambiguity in CUDA as @leofang mentioned, the default behavior of 0 still falls back to the legacy default stream. Additionally, it seems that Rocm also only currently support the legacy default stream behavior.
I would still advocate for option3, because from the experience of CUDA/Rocm development, there is a default starting point (of using legacy stream), having the default to match that starting point is good. It also makes default API work for both CPU, cuda and rocm case.

Thanks @rgommers .
I believe the main purpose of default behavior is to serve as a recommendation of starting point to the developers. Option1 means we do not have a recommendation for developers in the case of CUDA/Rocm.
On the other hand, there is a common starting point for developers to develop GPU kernels. I learn GPU programming by launching kernel without the stream argument. In those cases, the behavior relies on the legacy default stream. Although the stream 0 have ambiguity in CUDA as @leofang mentioned, the default behavior of 0 still falls back to the legacy default stream. Additionally, it seems that Rocm also only currently support the legacy default stream behavior.
I would still advocate for option3, because from the experience of CUDA/Rocm development, there is a default starting point (of using legacy stream), having the default to match that starting point is good. It also makes default API work for both CPU, cuda and rocm case.

For example, when producer's data comes from a non-default stream. the producer need to sync the data to the default stream, in order to be consumed. No syncing in such case would leads to undefined behavior. The default behavior of always sync to the default stream would be a good one.

Sorry, @tqchen, I don't get it. I thought having gone through the long discussion in https://github.com/dmlc/dlpack/issues/57 we finally are close to an agreement that we sync over the Consumer's stream, but what you're saying above is just in contrary to everything we've discussed. IIUC you're saying both the Producer and the Consumer need to sync once (the former with the default stream, and the latter with its own stream), or did I misread something?

For example, when producer's data comes from a non-default stream. the producer need to sync the data to the default stream, in order to be consumed. No syncing in such case would leads to undefined behavior. The default behavior of always sync to the default stream would be a good one.

Sorry, @tqchen, I don't get it. I thought having gone through the long discussion in https://github.com/dmlc/dlpack/issues/57 we finally are close to an agreement that we sync over the Consumer's stream, but what you're saying above is just in contrary to everything we've discussed. IIUC you're saying both the Producer and the Consumer need to sync once (the former with the default stream, and the latter with its own stream), or did I misread something?

what you're saying above is just in contrary to everything we've discussed.

I don't think so, nothing really changes. The only difference is whether we let people who do use the legacy default stream say stream=None as an alias for stream=1, or do we force them to say stream=1.

what you're saying above is just in contrary to everything we've discussed.

I don't think so, nothing really changes. The only difference is whether we let people who do use the legacy default stream say stream=None as an alias for stream=1, or do we force them to say stream=1.

Right, I think we have agreed on the API convention. The main discussion point is how do we specify the recommended default value behavior.

Right, I think we have agreed on the API convention. The main discussion point is how do we specify the recommended default value behavior.

Nit:

DLPack describe the memory layout of strided, n-dimensional arrays.

-> "DLPack describes the memory layout of strided, n-dimensional arrays."

Nit:

DLPack describe the memory layout of strided, n-dimensional arrays.

-> "DLPack describes the memory layout of strided, n-dimensional arrays."

So from_dlpack is implemented by "consumer", and __dlpack__ is implemented by "producer". Say producer is cupy, and consumer if numpy.
Cupy's __dlpack__ will rightfully return a capsule with  DLDeviceType set to a CUDA GPU code. CPU only consumer, that is NumPy, will not know what to do with this data, and must raise.
The only way out is to rely on cupy's to_numpy function. The standard should specify how to to deal with this scenario. Have I missed it?

So from_dlpack is implemented by "consumer", and __dlpack__ is implemented by "producer". Say producer is cupy, and consumer if numpy.
Cupy's __dlpack__ will rightfully return a capsule with  DLDeviceType set to a CUDA GPU code. CPU only consumer, that is NumPy, will not know what to do with this data, and must raise.
The only way out is to rely on cupy's to_numpy function. The standard should specify how to to deal with this scenario. Have I missed it?

That's a good question @oleksandr-pavlyk. When the producer and consumer don't overlap in device support at all, then indeed there's little that can be done other than error. Most libraries have a to_numpy-like method, but that's clearly not appropriate for a standard (specific to one library, so why add to_numpy and not to_jax?).
There is one case where things could work in principle, but don't: that is when the producer would support (e.g.) CPU and GPU, and the consumer only CPU. Then for GPU arrays now it doesn't work, but it could be made to work (with a device transfer) if the consumer could signal that device support to the producer.
The stance this protocol takes is that it must be zero-copy, or else only the consumer library can decide to copy (recommended not to though).
We could make the discussion on that in the Semantics section a little more extensive.

That's a good question @oleksandr-pavlyk. When the producer and consumer don't overlap in device support at all, then indeed there's little that can be done other than error. Most libraries have a to_numpy-like method, but that's clearly not appropriate for a standard (specific to one library, so why add to_numpy and not to_jax?).
There is one case where things could work in principle, but don't: that is when the producer would support (e.g.) CPU and GPU, and the consumer only CPU. Then for GPU arrays now it doesn't work, but it could be made to work (with a device transfer) if the consumer could signal that device support to the producer.
The stance this protocol takes is that it must be zero-copy, or else only the consumer library can decide to copy (recommended not to though).
We could make the discussion on that in the Semantics section a little more extensive.

It may be good to ask libraries to implement to_bytearray, or some such, which can be used to deal with exchange between libraries with no overlapping device support. It would be like copy via host mechanism.

It may be good to ask libraries to implement to_bytearray, or some such, which can be used to deal with exchange between libraries with no overlapping device support. It would be like copy via host mechanism.

I am not sure I am very clear on the warning about device_id in DLPack.
I though CUDA_VISIBLE_DEVICES and the like can influence what device run-time associate for device_id=0 for a given process. Once the process has been launched, the mapping is fixed and can not change, but I am open to stand corrected.

I am not sure I am very clear on the warning about device_id in DLPack.
I though CUDA_VISIBLE_DEVICES and the like can influence what device run-time associate for device_id=0 for a given process. Once the process has been launched, the mapping is fixed and can not change, but I am open to stand corrected.

I though CUDA_VISIBLE_DEVICES and the like can influence what device run-time associate for device_id=0 for a given process. Once the process has been launched, the mapping is fixed and can not change, but I am open to stand corrected.

See https://github.com/data-apis/consortium-feedback/issues/1#issuecomment-746581048 and comments below it.

I though CUDA_VISIBLE_DEVICES and the like can influence what device run-time associate for device_id=0 for a given process. Once the process has been launched, the mapping is fixed and can not change, but I am open to stand corrected.

See https://github.com/data-apis/consortium-feedback/issues/1#issuecomment-746581048 and comments below it.

It may be good to ask libraries to implement to_bytearray, or some such, which can be used to deal with exchange between libraries with no overlapping device support. It would be like copy via host mechanism.

In other places I think we have favoured raising exceptions as well, rather than doing device transfers. Once you get an error as a user, doing the manual transfer is probably fine with a library-specific method like to_numpy. to_bytearray is something that won't be used much even if all libraries would want to implement it.

It may be good to ask libraries to implement to_bytearray, or some such, which can be used to deal with exchange between libraries with no overlapping device support. It would be like copy via host mechanism.

In other places I think we have favoured raising exceptions as well, rather than doing device transfers. Once you get an error as a user, doing the manual transfer is probably fine with a library-specific method like to_numpy. to_bytearray is something that won't be used much even if all libraries would want to implement it.

Ah OK @rgommers @tqchen. Sorry for being nerve-wrecked, it's just that this statement

For example, when producer's data comes from a non-default stream. the producer need to sync the data to the default stream, in order to be consumed. No syncing in such case would leads to undefined behavior.

does not look right to me, and so any conclusion for the default such as

The default behavior of always sync to the default stream would be a good one.

is not sound.
I disagree on two things here:

"The default should be always syncing (on whatever stream)": I think once the stream argument is accepted by all parties, it should be fairly straightforward to avoid syncing whenever possible. In particular, there are use cases in which two libraries working on the same stream, which is known to the user. In such cases syncing is just unnecessary and making syncing as default would make it harder to opt out. Though I feel less strongly on this if it's deemed critical to unblock things.
"If we pick a default stream to sync over, it's better stream 0":

On CUDA: It would be very difficult, if not impossible, if Library A is built such that stream 0 = 1 (legacy) but Library B has stream 0 = 2 (per-thread) via setting nvcc --default-stream or -DCUDA_API_PER_THREAD_DEFAULT_STREAM, and a user tries to debug a synchronization bug. It's subtle and likely an implementation detail that users usually don't need to worry about. I believe at least RAPIDS is on the process to making stream 2 their default (@kkraus14 probably can comment on this), so such a scenario could happen in the near future.
On ROCm: Not choosing a default also avoids the problem that ROCm does not have stream 1 or 2 working.

A different question from the above discussion: Are we focusing only on CUDA/ROCm in this PR and leaving other architectures like OpenCL etc for the future, so that we can talk about streams?

Ah OK @rgommers @tqchen. Sorry for being nerve-wrecked, it's just that this statement

For example, when producer's data comes from a non-default stream. the producer need to sync the data to the default stream, in order to be consumed. No syncing in such case would leads to undefined behavior.

does not look right to me, and so any conclusion for the default such as

The default behavior of always sync to the default stream would be a good one.

is not sound.
I disagree on two things here:

"The default should be always syncing (on whatever stream)": I think once the stream argument is accepted by all parties, it should be fairly straightforward to avoid syncing whenever possible. In particular, there are use cases in which two libraries working on the same stream, which is known to the user. In such cases syncing is just unnecessary and making syncing as default would make it harder to opt out. Though I feel less strongly on this if it's deemed critical to unblock things.
"If we pick a default stream to sync over, it's better stream 0":

On CUDA: It would be very difficult, if not impossible, if Library A is built such that stream 0 = 1 (legacy) but Library B has stream 0 = 2 (per-thread) via setting nvcc --default-stream or -DCUDA_API_PER_THREAD_DEFAULT_STREAM, and a user tries to debug a synchronization bug. It's subtle and likely an implementation detail that users usually don't need to worry about. I believe at least RAPIDS is on the process to making stream 2 their default (@kkraus14 probably can comment on this), so such a scenario could happen in the near future.
On ROCm: Not choosing a default also avoids the problem that ROCm does not have stream 1 or 2 working.

A different question from the above discussion: Are we focusing only on CUDA/ROCm in this PR and leaving other architectures like OpenCL etc for the future, so that we can talk about streams?

Thanks @leofang
The current API design allows the producer/consumer to pass in streams, and implement the optimization mentioned in this point. So it won't block any of the features, the producer and consumer are free to implement your proposal, by passing in explicit stream of interest and checking the stream.
The main topic of interest, however, is to also recommend a default case when developers want simpler implementations (e.g. the program only works on default stream since that was a common way to get started). For those developers, None that default to legacy stream makes sense.
Additionally, I believe we are saying "None defaults to (legacy) default stream". In CUDA that means 1(or 0 when you use the default flag in nvcc), and rocm it means 0.  In short, they are the stream choice when you compile a kernel that does not contain a stream field. Having such a choice won't block any of the more advanced use-cases that are being mentioned. But would also help beginners to adopt things without worrying about the choice of the stream argument.

Thanks @leofang
The current API design allows the producer/consumer to pass in streams, and implement the optimization mentioned in this point. So it won't block any of the features, the producer and consumer are free to implement your proposal, by passing in explicit stream of interest and checking the stream.
The main topic of interest, however, is to also recommend a default case when developers want simpler implementations (e.g. the program only works on default stream since that was a common way to get started). For those developers, None that default to legacy stream makes sense.
Additionally, I believe we are saying "None defaults to (legacy) default stream". In CUDA that means 1(or 0 when you use the default flag in nvcc), and rocm it means 0.  In short, they are the stream choice when you compile a kernel that does not contain a stream field. Having such a choice won't block any of the more advanced use-cases that are being mentioned. But would also help beginners to adopt things without worrying about the choice of the stream argument.

@tqchen I think you're mixing up the expectations for developers/maintainers and users. So this is what I think

Users, especially beginners, would likely work on stream 0 (I don't disagree with your earlier statement)
Developers, who implement Producer/Consumer, are more likely knowledgable in CUDA/ROCm and also their libraries

But given that the current API design (__dlpack__ + from_dlpack) does not ask users to worry about which stream to use -- I think we agreed in https://github.com/dmlc/dlpack/issues/57 that users shouldn't have to -- I don't see the reason to worry about simplicity.
Developers, on the other hand, are expected to know CUDA/ROCm much better than average users. If they write a complex runtime with streams flying here and there, they should be able to determine which stream to use (or none at all) during exchange.
In particular, this is contradictory to the spirit of defining an API standard:

the producer and consumer are free to implement your proposal, by passing in explicit stream of interest and checking the stream

We are proposing a protocol for developers. It's natural to ask a clearly defined semantics (which in this case isn't really hard to do --- I am putting my CAI v3 hat again and say it's doable and done), instead of delegating to each library developer to make their own decision with the risk of conflicts with other libraries that should have been considered by now.
Moreover, checking the streams is not practical if we don't rule out stream 0. On a default build (without setting the flags I mentioned) where 0 means 1, the check if (0 == 1): ... would not pass (but semantically they should be equal), and leads us to an unnecessary synchronization. I hope this makes the reason more clear.
Finally,

In short, they are the stream choice when you compile a kernel that does not contain a stream field.

I think I explained how 0 can mean 1 or 2 by changing the way you compile code without changing the code itself (see also here), which also applies to this situation (launching a kernel without specifying a stream would default to stream 0 in CUDA), but as it stands, this stream choice is ambiguous at least in CUDA, and let's keep such an ambiguity from creeping in by being as explicit as we can 🙂

@tqchen I think you're mixing up the expectations for developers/maintainers and users. So this is what I think

Users, especially beginners, would likely work on stream 0 (I don't disagree with your earlier statement)
Developers, who implement Producer/Consumer, are more likely knowledgable in CUDA/ROCm and also their libraries

But given that the current API design (__dlpack__ + from_dlpack) does not ask users to worry about which stream to use -- I think we agreed in https://github.com/dmlc/dlpack/issues/57 that users shouldn't have to -- I don't see the reason to worry about simplicity.
Developers, on the other hand, are expected to know CUDA/ROCm much better than average users. If they write a complex runtime with streams flying here and there, they should be able to determine which stream to use (or none at all) during exchange.
In particular, this is contradictory to the spirit of defining an API standard:

the producer and consumer are free to implement your proposal, by passing in explicit stream of interest and checking the stream

We are proposing a protocol for developers. It's natural to ask a clearly defined semantics (which in this case isn't really hard to do --- I am putting my CAI v3 hat again and say it's doable and done), instead of delegating to each library developer to make their own decision with the risk of conflicts with other libraries that should have been considered by now.
Moreover, checking the streams is not practical if we don't rule out stream 0. On a default build (without setting the flags I mentioned) where 0 means 1, the check if (0 == 1): ... would not pass (but semantically they should be equal), and leads us to an unnecessary synchronization. I hope this makes the reason more clear.
Finally,

In short, they are the stream choice when you compile a kernel that does not contain a stream field.

I think I explained how 0 can mean 1 or 2 by changing the way you compile code without changing the code itself (see also here), which also applies to this situation (launching a kernel without specifying a stream would default to stream 0 in CUDA), but as it stands, this stream choice is ambiguous at least in CUDA, and let's keep such an ambiguity from creeping in by being as explicit as we can 🙂

In other places I think we have favoured raising exceptions as well, rather than doing device transfers. Once you get an error as a user, doing the manual transfer is probably fine with a library-specific method like to_numpy. to_bytearray is something that won't be used much even if all libraries would want to implement it.

I do not argue with raising exception. I am thinking what I would do should I encounter such an exception. Since the consumer (i.e. numpy in my thought experiment) does not know how to access GPU memory, I would be looking for function in the producer library (cupy) to retrieve the memory from GPU. This is where cuda.to_numpy comes in.
Have a commonly named function to manually reach out to for users working with multiple array libraries in one session would be good. I suggested to_bytearray, to avoid dependency on numpy for scenarios where two different vendors GPU libraries need to share the data. Really what one needs is an ability to reduce GPU array to Python-only objects, kind of like pickling does.

In other places I think we have favoured raising exceptions as well, rather than doing device transfers. Once you get an error as a user, doing the manual transfer is probably fine with a library-specific method like to_numpy. to_bytearray is something that won't be used much even if all libraries would want to implement it.

I do not argue with raising exception. I am thinking what I would do should I encounter such an exception. Since the consumer (i.e. numpy in my thought experiment) does not know how to access GPU memory, I would be looking for function in the producer library (cupy) to retrieve the memory from GPU. This is where cuda.to_numpy comes in.
Have a commonly named function to manually reach out to for users working with multiple array libraries in one session would be good. I suggested to_bytearray, to avoid dependency on numpy for scenarios where two different vendors GPU libraries need to share the data. Really what one needs is an ability to reduce GPU array to Python-only objects, kind of like pickling does.

I though CUDA_VISIBLE_DEVICES and the like can influence what device run-time associate for device_id=0 for a given process. Once the process has been launched, the mapping is fixed and can not change, but I am open to stand corrected.
See data-apis/consortium-feedback#1 (comment) and comments below it.

Thank you. Comment by @szha confirms that device_id is guaranteed within the process. That is in Python that loaded array_library1 and array_library2 in the same process, these libraries will communicate using consistent notions of device_id.
Since device_id is not guaranteed across processes, dlpack should not be use for inter-process communication.

I though CUDA_VISIBLE_DEVICES and the like can influence what device run-time associate for device_id=0 for a given process. Once the process has been launched, the mapping is fixed and can not change, but I am open to stand corrected.
See data-apis/consortium-feedback#1 (comment) and comments below it.

Thank you. Comment by @szha confirms that device_id is guaranteed within the process. That is in Python that loaded array_library1 and array_library2 in the same process, these libraries will communicate using consistent notions of device_id.
Since device_id is not guaranteed across processes, dlpack should not be use for inter-process communication.

Followup on the default behavior a bit. I think we are in agreement that the most important thing should be avoid ambiguity. So making "None that maps to stream 0" is ambiguous. But  "None that maps to the legacy default stream is not".
I do want to emphasize though the importance of starting point. We cannot assume the developer automatically guarantees the complicated implementation when possible.  Instead, imagine more developers who just started to learn GPU programming and want to make use of DLPack.  This is a lesson we learnt repetitively in the past. "simple (without compromising the advanced use case) starting point will make the standard more accessible." After all the goal of standardization is to lower the barrier of adoption, rather than create another standard.
I put a bit more thoughts into the options, considering the starting point. Trying to summarizing the options a bit based on @rgommers 's list

K1: None: CUDA/ROCm streams are not supported,
K2: None: the current runtime default stream (as indicated by getCurrentCUDAStream in cuda runtime)
K3: None: legacy default stream (default) of the corresponding API.

K2 is modified from option2 by @rgommers . Again let us come back to the starting point when a developer simply want to write a CUDA kernel(without managing streams). Then in that particular case the cuda runtime system picks up the current cuda stream to launch. This option would support data from another library that also picks the simple setting without synchronization (as both uses the runtime current stream).
K2 would be in particular useful for frameworks that relies on currentCudaStream. PyTorch is one notable example.  Imagine that I build an my_awesome_array  package that implements the exp function, using CUDA C. And I don't want to worry about stream management whatsoever. The function will simply be a cuda kernel launch that takes the DLTensor data structure. Consider the code below:
def my_torch_func(A):
     C = torch.mm(A, A)
     arr = my_awesome_array.from_dlpack(C)
     my_awesome_array.exp(arr)

with torch.cuda.stream(s1):
      my_torch_func(A0):

my_torch_func(A1):

In the above code, my_torch_func is called twice. First in stream s1 and then in default stream. In both cases my_awesome_array.from_dlpack can pass in 'None' under the setting of K2, not worrying about synchronization and still correctly execute in both cases.
I personally find it is really important to support the case of "the developer just want to write GPU library without worrying about stream management".
As a result I now find K2 the best option. It is also mostly consistent with the previous behavior of DLPack GPU exchange.  Additionally, it can generalize to future settings (e.g. DPCPP) if there are standardization around setCurrentContext and getCurrentContext API in those GPU APIs.
I also want to note that all three choices(K1, K2, K3) have no ambiguity.

Followup on the default behavior a bit. I think we are in agreement that the most important thing should be avoid ambiguity. So making "None that maps to stream 0" is ambiguous. But  "None that maps to the legacy default stream is not".
I do want to emphasize though the importance of starting point. We cannot assume the developer automatically guarantees the complicated implementation when possible.  Instead, imagine more developers who just started to learn GPU programming and want to make use of DLPack.  This is a lesson we learnt repetitively in the past. "simple (without compromising the advanced use case) starting point will make the standard more accessible." After all the goal of standardization is to lower the barrier of adoption, rather than create another standard.
I put a bit more thoughts into the options, considering the starting point. Trying to summarizing the options a bit based on @rgommers 's list

K1: None: CUDA/ROCm streams are not supported,
K2: None: the current runtime default stream (as indicated by getCurrentCUDAStream in cuda runtime)
K3: None: legacy default stream (default) of the corresponding API.

K2 is modified from option2 by @rgommers . Again let us come back to the starting point when a developer simply want to write a CUDA kernel(without managing streams). Then in that particular case the cuda runtime system picks up the current cuda stream to launch. This option would support data from another library that also picks the simple setting without synchronization (as both uses the runtime current stream).
K2 would be in particular useful for frameworks that relies on currentCudaStream. PyTorch is one notable example.  Imagine that I build an my_awesome_array  package that implements the exp function, using CUDA C. And I don't want to worry about stream management whatsoever. The function will simply be a cuda kernel launch that takes the DLTensor data structure. Consider the code below:
def my_torch_func(A):
     C = torch.mm(A, A)
     arr = my_awesome_array.from_dlpack(C)
     my_awesome_array.exp(arr)

with torch.cuda.stream(s1):
      my_torch_func(A0):

my_torch_func(A1):

In the above code, my_torch_func is called twice. First in stream s1 and then in default stream. In both cases my_awesome_array.from_dlpack can pass in 'None' under the setting of K2, not worrying about synchronization and still correctly execute in both cases.
I personally find it is really important to support the case of "the developer just want to write GPU library without worrying about stream management".
As a result I now find K2 the best option. It is also mostly consistent with the previous behavior of DLPack GPU exchange.  Additionally, it can generalize to future settings (e.g. DPCPP) if there are standardization around setCurrentContext and getCurrentContext API in those GPU APIs.
I also want to note that all three choices(K1, K2, K3) have no ambiguity.

Since device_id is not guaranteed across processes, dlpack should not be used for inter-process communication.

I agree, this seems to be substantiated by the fact that currently there is no IPC interface defined in DLPack. Being a data exchange protocol I think IPC is definitely out of scope, as it necessarily involves more knowledge in the hardware detail.

Since device_id is not guaranteed across processes, dlpack should not be used for inter-process communication.

I agree, this seems to be substantiated by the fact that currently there is no IPC interface defined in DLPack. Being a data exchange protocol I think IPC is definitely out of scope, as it necessarily involves more knowledge in the hardware detail.

To also catch up on the thread of CPU default exchange. I actually think that a numpy is not a bad dependency to have as a common fallback, most libraries likely have such a function. Although I believe standardizing the to_numpy/asnumpy function name is a separate topic

To also catch up on the thread of CPU default exchange. I actually think that a numpy is not a bad dependency to have as a common fallback, most libraries likely have such a function. Although I believe standardizing the to_numpy/asnumpy function name is a separate topic

@tqchen Thank you very much for laying out your thoughts in great detail. Unfortunately it has becomes more difficult to me to follow up your latest comment as it seems we do not have a consensus on the subjects referred in the discussion. IMHO you're still mixing up users with library developers...😢
Before we continue the stream discussion, can we please first agree that

Users are those who will use the libraries that implement and comply with the Python Array API Standard.
Developers are those who implement and maintain the libraries complying with the Python Array API Standard, in particular those considered as stakeholders. (The core members includes NumPy/TensorFlow/PyTorch/MXNet/JAX/Dask/CuPy.)
The Python Array API Standard is used as a guidance for the Developers

@tqchen Thank you very much for laying out your thoughts in great detail. Unfortunately it has becomes more difficult to me to follow up your latest comment as it seems we do not have a consensus on the subjects referred in the discussion. IMHO you're still mixing up users with library developers...😢
Before we continue the stream discussion, can we please first agree that

Users are those who will use the libraries that implement and comply with the Python Array API Standard.
Developers are those who implement and maintain the libraries complying with the Python Array API Standard, in particular those considered as stakeholders. (The core members includes NumPy/TensorFlow/PyTorch/MXNet/JAX/Dask/CuPy.)
The Python Array API Standard is used as a guidance for the Developers

@leofang I consider developers go beyond the core member in NumPy/TensorFlow/PyTorch/MXNet/JAX/Dask/CuPy and would want to include anyone who like to implement an CUDA array function and expose to python :)  If we have a chance to support more people without sacrificing the correctness and compactness of the standard, so more libraries can use DLPack, why not?
Even if we are restricting to core developers of the the libraries as you stated.
K2 is going to be significantly easier for cases like PyTorch, and compilers (e.g. in the case of TVM where we only cares about compiling code to a simple cuda C, no stream management is being used, and we defer to the cuda runtime). I personally consider myself as a core developer, and would love to see such an option.

@leofang I consider developers go beyond the core member in NumPy/TensorFlow/PyTorch/MXNet/JAX/Dask/CuPy and would want to include anyone who like to implement an CUDA array function and expose to python :)  If we have a chance to support more people without sacrificing the correctness and compactness of the standard, so more libraries can use DLPack, why not?
Even if we are restricting to core developers of the the libraries as you stated.
K2 is going to be significantly easier for cases like PyTorch, and compilers (e.g. in the case of TVM where we only cares about compiling code to a simple cuda C, no stream management is being used, and we defer to the cuda runtime). I personally consider myself as a core developer, and would love to see such an option.

@leofang I consider developers go beyond the core member in NumPy/TensorFlow/PyTorch/MXNet/JAX/Dask/CuPy and want to include anyone who like to implement an CUDA array function and expose to python via the convention :)

This is the current list of main stakeholders: https://data-apis.github.io/array-api/latest/purpose_and_scope.html#stakeholders. Definitely the likes of TVM are in scope.
There is a bit of a difference of "array API standard stakeholders" and "DLPack Python API stakeholders"; the latter is probably a little bit broader. I don't have as strong an opinion as either of you here.
Currently I'm leaning towards simplicity, assuming the correctness is watertight. Which I'm not yet 100% sure about (likely due to my limited CUDA experience):

K2: None: the current runtime default stream (as indicated by getCurrentCUDAStream in cuda runtime)

Given two libraries built with different options - one uses per-thread default stream (say RAPIDS), the other uses legacy default stream (say the new dev who doesn't do any stream handling) - will the stream=None meaning getCurrentCUDAStream not result in producer and consumer using different streams here?

@leofang I consider developers go beyond the core member in NumPy/TensorFlow/PyTorch/MXNet/JAX/Dask/CuPy and want to include anyone who like to implement an CUDA array function and expose to python via the convention :)

This is the current list of main stakeholders: https://data-apis.github.io/array-api/latest/purpose_and_scope.html#stakeholders. Definitely the likes of TVM are in scope.
There is a bit of a difference of "array API standard stakeholders" and "DLPack Python API stakeholders"; the latter is probably a little bit broader. I don't have as strong an opinion as either of you here.
Currently I'm leaning towards simplicity, assuming the correctness is watertight. Which I'm not yet 100% sure about (likely due to my limited CUDA experience):

K2: None: the current runtime default stream (as indicated by getCurrentCUDAStream in cuda runtime)

Given two libraries built with different options - one uses per-thread default stream (say RAPIDS), the other uses legacy default stream (say the new dev who doesn't do any stream handling) - will the stream=None meaning getCurrentCUDAStream not result in producer and consumer using different streams here?

After looking a bit deeper I find that getCurrentCUDAStream is a function in pytorch runtime and I somewhat thought it is a runtime function.  Considering the fact, I now think that K2 is no longer a right option from standardization's PoV. Sorry for the confusion.
The option of K1 and K3 still stands. In which case K1 is still a better solution since it highlights the None maps (legacy) default stream, which is well-defined in both rocm and cuda

After looking a bit deeper I find that getCurrentCUDAStream is a function in pytorch runtime and I somewhat thought it is a runtime function.  Considering the fact, I now think that K2 is no longer a right option from standardization's PoV. Sorry for the confusion.
The option of K1 and K3 still stands. In which case K1 is still a better solution since it highlights the None maps (legacy) default stream, which is well-defined in both rocm and cuda

To suggest a resolution to move on first, How about we adopt S0reduced as in https://github.com/dmlc/dlpack/issues/57#issuecomment-774482973,
e.g. not introducing stream argument for now and start simple. Then we can continue discussing the stream case if ndded

To suggest a resolution to move on first, How about we adopt S0reduced as in dmlc/dlpack#57 (comment),
e.g. not introducing stream argument for now and start simple. Then we can continue discussing the stream case if ndded

I think my preference is still to go for option S0a in that comment. The only thing that needed deciding is the meaning of None. This I think is the best compromise:

let None mean "legacy default stream"
add a recommendation that array library implementers use explicit streams (e.g. 1) in preference of None wherever possible, with the rationale that None is for simplicity and may lead to more synchronization than necessary.

That would satisfy @tqchen's "start simple without thinking about streams at all", and ensure that all array libraries do this more thoroughly (which is what @leofang prefers).

Thanks @rgommers the proposal looks good to me

One last minor thing on establishing the stream ordering: Can we provide an env var, say, PYTHON_ARRAY_API_IGNORE_STREAM, for suppressing any potential synchronization done by Producer? This is for the need of Consumer libraries which cannot provide any meaningful construct to the producer. In revising CUDA Array Interface we left such a "backdoor" to ignore streams:

Use of this exception should be avoided where possible, as it is provided for libraries that cannot implement the synchronization semantics without the involvement of the User - for example, those interfacing with third-party libraries oblivious to the CUDA Array Interface.

The most notable example is mpi4py, in which we have no control over any CUDA functionality (it's handled by either UCX or the underlying MPI library), so as a Consumer we cannot provide any stream to establish the stream order. The synchronization (in both MPI and CUDA senses) must be explicitly done by the Users.

Can we provide an env var, say, PYTHON_ARRAY_API_IGNORE_STREAM, for suppressing any potential synchronization done by Producer?

Your rationale for a backdoor makes sense, but can we just use stream=-1? An env var seems pretty ugly API-wise.

Can we provide an env var, say, PYTHON_ARRAY_API_IGNORE_STREAM, for suppressing any potential synchronization done by Producer?

Your rationale for a backdoor makes sense, but can we just use stream=-1? An env var seems pretty ugly API-wise.

@rgommers You meant Consumer should set it to -1 and Producer should explicitly check if -1 is given, right? Fine with me. An env var gives Users somewhat a handle to control the behavior, but if a User forgets to set it, it's doomed, so this is a tradeoff for which I have no strong opinion.

You meant Consumer should set it to -1 and Producer should explicitly check if -1 is given, right?

Yes indeed.

An env var gives Users somewhat a handle to control the behavior, but if a User forgets to set it, it's doomed, so this is a tradeoff for which I have no strong opinion.

I'm actually not quite sure I understand this. Can you think of cases where an array library will be used both in settings where it does and doesn't know if the user must be involved in stream handling?
In that case the library must expose some API to the user to control its stream handling. Which could be an env var or a regular API (e.g. add disable_stream_handling= to from_dlpack). Either way I think that's up to the library to figure out, and it shouldn't be in the public API of the array API standard I believe.

An env var gives Users somewhat a handle to control the behavior, but if a User forgets to set it, it's doomed, so this is a tradeoff for which I have no strong opinion.

I'm actually not quite sure I understand this. Can you think of cases where an array library will be used both in settings where it does and doesn't know if the user must be involved in stream handling?

As discussed in yesterday's call, I was mainly thinking that an env var could give superusers a knot to tweak the behavior, but it's probably better just restrict to the need of library implementors, for which setting stream to -1 or any sentinel object is enough.

I haven't done a full pass review yet, will do so asap, but in case I forget: I think we better set a minimal requirement for DLPACK_VERSION so that everyone is importing the latest header.

Given that v0.4 is ABI-compatible with v0.2 and v0.3, and that (I assume) an ABI change will come with a major version increase, I guess the required range is 0.2 <= DLPACK_VERSION < 1.0. Latest would be nice, but it shouldn't matter - once we require complex number support, it needs to be >= 0.4 though.

The one other comment on the new __dlpack_device__ was to change the integer for device ID to a Python enum - I'm looking at that now.

Update PR for all remaining comments. I'd like to merge this by the end of the weekend, and if there are more comments after that do a follow-up PR. Reason: this should be visible in the html docs, and we're about to send the adoption proposal for NumPy out for review, where I'm sure there'll be some discussion around DLPack.

Okay in it goes. Thanks @tqchen, @leofang, @kkraus14 and @oleksandr-pavlyk!

Thanks @rgommers @tqchen for carrying out most of the hard work!

authors: leofang oleksandr-pavlyk rgommers tqchen


text:
Feedback milestone 2

Hello,
My name is Suzana__ and I will be giving you some feedback on your project.
Overall, the project is a good starting point, but there is some work that needs to be done to elevate this project to our standards so its meets our expectations.
I have divided up the feedback into several sections so it is easier for you to implement.
Task 1

Good

Task 2

[x] #2

Task 3

[x] #3

Task 4

[x] #4
[ ] Could not find any visualization plots for Maruf's research question (the second research question).

Hi @SuzanaMomo, thanks for writing feedback for our project. I think those were valid points and I had rushed through some sections. I think I have improved my analysis and EDA by leaps and bound. I added lots of stuff and fancied it up, I hope you like it :)).
Please take a look at those changes and let me know if it needs more improvement.
As for Maruf, I think he had to solve his research question, but he was sick last week so he couldn't do so in a timely fashion. He is working on it as we speak. I think he can better speak on that @marufzubery

Hello @SuzanaMomo , I was a bit sick last week and preoccupied with personal work. I will work on it over the coming week to bring my part up to to standard and expectations. Thank you for your feed back

Hello @SuzanaMomo , I was a bit sick last week and preoccupied with personal work. I will work on it over the coming week to bring my part up to to standard and expectations. Thank you for your feed back

authors: AhmadRazaJamal SuzanaMomo marufzubery


text:
Why is there no API documentation?

There have been issues open for several years asking for API docs.
Out of curiosity, why are there none for this project?
If I started a PR that provided API documentation, would people help out / would it be accepted?

Update - been playing with dat.gui a lot. I'll start an API readme doc soon and hopefully people will add to it.

Here's a start, generated from JSDoc with documentation.js:
npx documentation build src/dat/** -f md > api.md

Result: https://gist.github.com/donmccurdy/c08977da5148651f05a9aa4080cead04
It needs a lot of cleaning up to remove cruft, but maybe between the documentationjs settings and some improvements to the JSDoc, we can have API docs without needing to maintain them separately from the source code?

API docs are available now: https://github.com/dataarts/dat.gui/blob/master/API.md
They're generated from JSDocs throughout the code, so if anyone wants to improve the docs PRs are very welcome.

authors: donmccurdy donpinkus

license_source: bigquery


text:
Fix client side lint issues

add separate modes to run_test for fixing and testing lint
fix lint errors and warnings

@qlj-lijuan @fpernice-google fyi - note the new -f option for ./run_test.sh

@qlj-lijuan @fpernice-google fyi - note the new -f option for ./run_test.sh

Got it. Thanks!

authors: beets qlj-lijuan

repo: datacommonsorg/website


text:
Correction for raw affiliation AGIR, UR 4294, Université de Picardie Jules Verne, Amiens, France

Correction needed for raw affiliation AGIR, UR 4294, Université de Picardie Jules Verne, Amiens, France
raw_affiliation_name: AGIR, UR 4294, Université de Picardie Jules Verne, Amiens, France
new_rors: 01gyxrk03
previous_rors: 01gyxrk03;01sjmsj73
works_examples: W4200293325
contact: 09fe517ef2b2e7001675c5cb532031a1:2c200aa6bda83f370f13a3e3a393bfd5 @ u-picardie.fr

This issue was accepted and ingested by the OpenAlex team on 2024-10-10. The new affiliations should be visible within the next 7 days.

repo: dataesr/openalex-affiliations


text:
Correction for raw affiliation Univ Rennes, CHU Rennes, INSERM CIC1414, F-35000 Rennes, France

Correction needed for raw affiliation Univ Rennes, CHU Rennes, INSERM CIC1414, F-35000 Rennes, France
raw_affiliation_name: Univ Rennes, CHU Rennes, INSERM CIC1414, F-35000 Rennes, France
new_rors: 015m7wh34;02vjkv261;05qec5a53;02baj6743
previous_rors: 015m7wh34;02vjkv261;05qec5a53
works_examples: W4381546176
contact: 4117ae43e5e30de6b5e2f622fa66c616:013507ce844b3d3fd2e2ec7ebee40ccb @ univ-rennes.fr

This issue was accepted and ingested by the OpenAlex team on 2024-10-10. The new affiliations should be visible within the next 7 days.


text:
Correction for raw affiliation Centre des Sciences des Littératures en Langue Française; Littératures, Imaginaire, Sociétés

Correction needed for raw affiliation Centre des Sciences des Littératures en Langue Française; Littératures, Imaginaire, Sociétés
raw_affiliation_name: Centre des Sciences des Littératures en Langue Française; Littératures, Imaginaire, Sociétés
new_rors: 059v8sw69;02en5vm52
previous_rors: 059v8sw69
works_examples: W4390868222
contact: 5c980f2a92feda267717a0116d348f99:167ccaf523ecb4ba3450f8943a @ sorbonne-universite.fr

This issue was accepted and ingested by the OpenAlex team on 2024-11-01. The new affiliations should be visible within the next 7 days.


text:
Correction for raw affiliation INRAE - Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement (France); UMR PHIM - Plant Health Institute of Montpellier (CIRAD Campus International de Baillarguet TA A-54/K
34398 Montpellier - France)

Correction needed for raw affiliation INRAE - Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement (France); UMR PHIM - Plant Health Institute of Montpellier (CIRAD Campus International de Baillarguet TA A-54/K
34398 Montpellier - France)
raw_affiliation_name: INRAE - Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement (France); UMR PHIM - Plant Health Institute of Montpellier (CIRAD Campus International de Baillarguet TA A-54/K
34398 Montpellier - France)
new_rors: 05kpkpg04;003vg9w96;03fkrcm40;05y5s3h28
previous_rors: 05kpkpg04;003vg9w96;03fkrcm40
works_examples: W4402824422
contact: b71a854072b759cab0d983c34dfdefe6:63722a4b0aa02b207a995b0e664261dae8a4 @ inrae.fr

This issue was accepted and ingested by the OpenAlex team on 2024-11-18. The new affiliations should be visible within the next 7 days.


text:
Correction for raw affiliation EPI Modemic Inra-Inria, UMR Mistea, SupAgro, 34 060 Montpellier, France; Irstea, UMR Itap, 34196 Montpellier Cedex, France

Correction needed for raw affiliation EPI Modemic Inra-Inria, UMR Mistea, SupAgro, 34 060 Montpellier, France; Irstea, UMR Itap, 34196 Montpellier Cedex, France
raw_affiliation_name: EPI Modemic Inra-Inria, UMR Mistea, SupAgro, 34 060 Montpellier, France; Irstea, UMR Itap, 34196 Montpellier Cedex, France
new_rors: ;01pd2sz18
previous_rors:
works_examples: W1983053736
contact: a660feb7f2ee497dee6050a4812519a9:98d48de95854b944d6da6018cc432b524985 @ inrae.fr

This issue was accepted and ingested by the OpenAlex team on 2024-12-19. The new affiliations should be visible within the next 7 days.


text:
Correction for raw affiliation Toronto Health Economics and Technology Assessment (THETA) Collaborative (Bremner, Cheung, Krahn, Mendlowitz, Sander, Tsui); Toronto General Hospital Research Institute (Lapointe-Shaw); General Internal Medicine (Lapointe-Shaw), Toronto General Hospital; Biostatistics Research Unit (Tomlinson), University Health Network; Hospital for Sick Children (Tsui); Chiefs of Ontario (Mulder); Ontario Ministry of Health and Ontario Ministry of Long-Term Care (Malikov); ICES Central (Croxford, Lapointe-Shaw, Wong, Sander, Walker, Kulkarni, Krahn); Sunnybrook Health Sciences Centre (Del Giudice); Canadian Centre for Applied Research in Cancer Control (Tsui); Divisions of Urology (Cheung, Kulkarni, Perlis) and Surgical Oncology (Cheung, Kulkarni), Department of Surgery, and Department of Family and Community Medicine (Del Giudice), Temerty Faculty of Medicine, and Institute of Health Policy Management and Evaluation (Mendlowitz), University of Toronto; Toronto, Ont.; C...

Correction needed for raw affiliation Toronto Health Economics and Technology Assessment (THETA) Collaborative (Bremner, Cheung, Krahn, Mendlowitz, Sander, Tsui); Toronto General Hospital Research Institute (Lapointe-Shaw); General Internal Medicine (Lapointe-Shaw), Toronto General Hospital; Biostatistics Research Unit (Tomlinson), University Health Network; Hospital for Sick Children (Tsui); Chiefs of Ontario (Mulder); Ontario Ministry of Health and Ontario Ministry of Long-Term Care (Malikov); ICES Central (Croxford, Lapointe-Shaw, Wong, Sander, Walker, Kulkarni, Krahn); Sunnybrook Health Sciences Centre (Del Giudice); Canadian Centre for Applied Research in Cancer Control (Tsui); Divisions of Urology (Cheung, Kulkarni, Perlis) and Surgical Oncology (Cheung, Kulkarni), Department of Surgery, and Department of Family and Community Medicine (Del Giudice), Temerty Faculty of Medicine, and Institute of Health Policy Management and Evaluation (Mendlowitz), University of Toronto; Toronto, Ont.; Canadian Centre for Applied Research in Cancer Control (Zeitouny, Teckle, Peacock, Pataky), BC Cancer Agency; Centre for Health Services and Policy Research (Zeitouny), School of Population and Public Health (McGrail, Pataky), University of British Columbia; University Canada West (Teckle), Vancouver, BC; Queens University (Mulder), Kingston, Ont.; School of Pharmacy, University of Waterloo (Wong), Kitchener, Ont.; Department of Health Research Methods, Evidence, and Impact (Walker), McMaster University, Hamilton, Ont.; Faculty of Health Sciences (Peacock), Simon Fraser University, Burnaby, BC.
raw_affiliation_name: Toronto Health Economics and Technology Assessment (THETA) Collaborative (Bremner, Cheung, Krahn, Mendlowitz, Sander, Tsui); Toronto General Hospital Research Institute (Lapointe-Shaw); General Internal Medicine (Lapointe-Shaw), Toronto General Hospital; Biostatistics Research Unit (Tomlinson), University Health Network; Hospital for Sick Children (Tsui); Chiefs of Ontario (Mulder); Ontario Ministry of Health and Ontario Ministry of Long-Term Care (Malikov); ICES Central (Croxford, Lapointe-Shaw, Wong, Sander, Walker, Kulkarni, Krahn); Sunnybrook Health Sciences Centre (Del Giudice); Canadian Centre for Applied Research in Cancer Control (Tsui); Divisions of Urology (Cheung, Kulkarni, Perlis) and Surgical Oncology (Cheung, Kulkarni), Department of Surgery, and Department of Family and Community Medicine (Del Giudice), Temerty Faculty of Medicine, and Institute of Health Policy Management and Evaluation (Mendlowitz), University of Toronto; Toronto, Ont.; Canadian Centre for Applied Research in Cancer Control (Zeitouny, Teckle, Peacock, Pataky), BC Cancer Agency; Centre for Health Services and Policy Research (Zeitouny), School of Population and Public Health (McGrail, Pataky), University of British Columbia; University Canada West (Teckle), Vancouver, BC; Queens University (Mulder), Kingston, Ont.; School of Pharmacy, University of Waterloo (Wong), Kitchener, Ont.; Department of Health Research Methods, Evidence, and Impact (Walker), McMaster University, Hamilton, Ont.; Faculty of Health Sciences (Peacock), Simon Fraser University, Burnaby, BC.
new_rors: 042xt5161;04374qe70;026pg9j08;00tjpb250;028d12s45
previous_rors: 042xt5161;04374qe70;026pg9j08;00tjpb250
works_examples: W4296511223
contact: e37e7a74d97334cd4d7d1dba5b839a4f:219f76ca @ ourresearch.org

This issue was accepted and ingested by the OpenAlex team on 2024-12-19. The new affiliations should be visible within the next 7 days.


text:
Bug iframeResize

Je viens de créer un nouveau repo pour intégrer l'iframe paramétrable (en lien avec Time for the Planet) et j'ai pu la déployer via Github Page. Mais je rencontre une drôle d'erreur lorsque je réponds aux questions du tutoriel : la page s'agrandit toute seule.
Je suppose donc qu'il y a un soucis avec la fonction iframeResize car le problème n'existe pas sur le site officiel de NGC.
Pour tester directement sur la page : https://timefortheplanet.github.io/nosgestesclimat-iframe/

Ah oui, en effet, je confirme !

Le problème n'apparait pas cependant sur une intégration comme https://datagir.ademe.fr/apps/nos-gestes-climat/
Je vais investiguer

Salut @Benbb96, j'ai mis en ligne une correction, dis-moi si ça le fait :)

Hello @laem oui c'est parfait, je n'ai plus le soucis après avoir vidé le cache !
Notre site avec l'iframe est désormais fonctionnel ici : https://ngc.time-planet.com/ 😃

repo: datagir/nosgestesclimat-site


text:
How to extend the data model on Neo4j

Currently I want to extend a z-index field to the neo4j model, which is mainly used to identify the business level of the dataset in the data warehouse, and to draw a complete data map later.
I wonder to know how to customize it.

Hopefully it's covered by https://github.com/linkedin/datahub/blob/master/docs/how/graph-onboarding.md. Let us know if there's any gap in the documentation.

After reading the relevant docs, I found that I only need to extend datasetUrn to meet biz needs.
What else needs to be done besides modifying DatasetUrn.pdl ?

I'm actually curious what sort of extension are you talking about in regards to DatasetUrn? I'd imagine adding a "z-index" field to existing/new dataset aspect seems more appropriate to me?

My idea is to extend the properties of the existing DatasetUrn, and then set the properties in the implementation of DatasetGraphBuilder. Is there a problem with my understanding? Of course I can also call the method of the DatasetSnapshot class.

See my comments in https://github.com/linkedin/datahub/issues/1848. You really shouldn't be messing with the URN structure.

Ok, I know. I will try to see which methods are provided by the current DatasetSnapshot class.

@cobolbaby can we close this now?

Yes.

Hopefully it's covered by https://github.com/linkedin/datahub/blob/master/docs/how/graph-onboarding.md. Let us know if there's any gap in the documentation.

This link is broken.
Could not find it here https://github.com/datahub-project/datahub/tree/master/docs/how

Could DataHub accommodate code property graphs
Pruned control flow and data flow graphs can be critical for lineage.

https://twitter.com/claudiuvursache/status/1474149858470735873
https://twitter.com/TigerGraphDB/status/1373398189714325512

@shirshanka

authors: SemanticBeeng cobolbaby mars-lan

repo: datahub-project/datahub


text:
WebUi  Error adding data source hive

'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 656, in '
'__connect\n'
'    648  def __connect(self, first_connect_check=False):\n'
' (...)\n'
'    652      # creator fails, this attribute stays None\n'
'    653      self.connection = None\n'
'    654      try:\n'
'    655          self.starttime = time.time()\n'
'--> 656          connection = pool._invoke_creator(self)\n'
'    657          pool.logger.debug("Created new connection %r", connection)\n'
'    ..................................................\n'
'     self = <sqlalchemy.pool.base._ConnectionRecord object at 0x7fcc9dd78310>\n'
'     first_connect_check = True\n'
'     self.connection = None\n'
'     self.starttime = 1652256727.4983974\n'
"     pool._invoke_creator = <function 'DefaultEngineStrategy.create..connect' strategies.py:106>\n"
"     pool.logger.debug = <method 'Logger.debug' of <Logger sqlalchemy.pool.impl.QueuePool (WARNING)> init.py:1424>\n"
'    ..................................................\n'
'\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/engine/strategies.py", line '
'114, in connect\n'
'    106  def connect(connection_record=None):\n'
' (...)\n'
'    110                  dialect, connection_record, cargs, cparams\n'
'    111              )\n'
'    112              if connection is not None:\n'
'    113                  return connection\n'
'--> 114      return dialect.connect(*cargs, **cparams)\n'
'    ..................................................\n'
'     connection_record = <sqlalchemy.pool.base._ConnectionRecord object at 0x7fcc9dd78310>\n'
'     dialect = <pyhive.sqlalchemy_hive.HiveDialect object at 0x7fcc9dda66a0>\n'
'     cargs = []\n'
"     cparams = {'host': '192.168.99.106',\n"
"                'port': 9831,\n"
"                'username': 'biyao',\n"
"                'password': 'biyao}',\n"
"                'database': 'default',\n"
"                'auth': 'LDAP'}\n"
"     dialect.connect = <method 'DefaultDialect.connect' of <pyhive.sqlalchemy_hive.HiveDialect object at 0x7fcc9dda66a0> "
'default.py:506>\n'
'    ..................................................\n'
'\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 508, '
'in connect\n'
'    506  def connect(self, *cargs, **cparams):\n'
'    507      # inherits the docstring from interfaces.Dialect.connect\n'
'--> 508      return self.dbapi.connect(*cargs, **cparams)\n'
'    ..................................................\n'
'     self = <pyhive.sqlalchemy_hive.HiveDialect object at 0x7fcc9dda66a0>\n'
'     cargs = ()\n'
"     cparams = {'host': '192.168.99.106',\n"
"                'port': 9831,\n"
"                'username': 'biyao',\n"
"                'password': 'biyao}',\n"
"                'database': 'default',\n"
"                'auth': 'LDAP'}\n"
"     self.dbapi.connect = <function 'connect' hive.py:120>\n"
'    ..................................................\n'
'\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/pyhive/hive.py", line 126, in connect\n'
'    120  def connect(*args, **kwargs):\n'
' (...)\n'
'    122      arguments.\n'
'    123  \n'
'    124      :returns: a :py:class:Connection object.\n'
'    125      """\n'
'--> 126      return Connection(*args, **kwargs)\n'
'    ..................................................\n'
'     args = ()\n'
"     kwargs = {'host': '192.168.99.106',\n"
"               'port': 9831,\n"
"               'username': 'biyao',\n"
"               'password': 'biyao}',\n"
"               'database': 'default',\n"
"               'auth': 'LDAP'}\n"
"     Connection = <class 'pyhive.hive.Connection'>\n"
'    ..................................................\n'
'\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/pyhive/hive.py", line 267, in init\n'
'    132  def init(\n'
'    133      self,\n'
'    134      host=None,\n'
'    135      port=None,\n'
'    136      scheme=None,\n'
'    137      username=None,\n'
"    138      database='default',\n"
'    139      auth=None,\n'
'    140      configuration=None,\n'
'    141      kerberos_service_name=None,\n'
'    142      password=None,\n'
'    143      check_hostname=None,\n'
'    144      ssl_cert=None,\n'
"    145      http_path='/cliservice/',\n"
'    146      thrift_transport=None\n'
'    147  ):\n'
' (...)\n'
'    263      # "V6 uses binary type for binary payload (was string) and uses columnar result set"\n'
'    264      protocol_version = ttypes.TProtocolVersion.HIVE_CLI_SERVICE_PROTOCOL_V6\n'
'    265  \n'
'    266      try:\n'
'--> 267          self._transport.open()\n'
'    268          open_session_req = ttypes.TOpenSessionReq(\n'
'    ..................................................\n'
'     self = <pyhive.hive.Connection object at 0x7fcc9dd78130>\n'
"     host = '192.168.99.106'\n"
'     port = 9831\n'
'     scheme = None\n'
"     username = 'biyao'\n"
"     database = 'default'\n"
"     auth = 'LDAP'\n"
'     configuration = {}\n'
'     kerberos_service_name = None\n'
"     password = 'biyao}'\n"
'     check_hostname = None\n'
'     ssl_cert = None\n'
"     http_path = '/cliservice/'\n"
'     thrift_transport = None\n'
'     protocol_version = 5\n'
'     ttypes.TProtocolVersion.HIVE_CLI_SERVICE_PROTOCOL_V6 = 5\n'
"     self._transport.open = <method 'TSaslClientTransport.open' of <thrift_sasl.TSaslClientTransport object at 0x7fcc9dd73400> "
'init.py:72>\n'
"     ttypes.TOpenSessionReq = <class 'TCLIService.ttypes.TOpenSessionReq'>\n"
'    ..................................................\n'
'\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/thrift_sasl/init.py", line 95, in '
'open\n'
'    72   def open(self):\n'
' (...)\n'
'    91     # SASL negotiation loop\n'
'    92     while True:\n'
'    93       status, payload = self._recv_sasl_message()\n'
'    94       if status not in (self.OK, self.COMPLETE):\n'
'--> 95         raise TTransportException(type=TTransportException.NOT_OPEN,\n'
'    96           message=("Bad status: %d (%s)" % (status, payload)))\n'
'    ..................................................\n'
'     self = <thrift_sasl.TSaslClientTransport object at 0x7fcc9dd73400>\n'
'     status = 3\n'
"     payload = b'Error validating the login'\n"
"     self._recv_sasl_message = <method 'TSaslClientTransport._recv_sasl_message' of <thrift_sasl.TSaslClientTransport object at "
'0x7fcc9dd73400> _init\n'
'                                _.py:111>\n'
'     self.OK = 2\n'
'     self.COMPLETE = 5\n'
"     TTransportException = <class 'thrift.transport.TTransport.TTransportException'>\n"
'     TTransportException.NOT_OPEN = 1\n'
'    ..................................................\n'
'\n'
'---- (full traceback above) ----\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/datahub/entrypoints.py", line 149, in '
'main\n'
'    sys.exit(datahub(standalone_mode=False, **kwargs))\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/click/core.py", line 1130, in call\n'
'    return self.main(*args, **kwargs)\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/click/core.py", line 1055, in main\n'
'    rv = self.invoke(ctx)\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/click/core.py", line 1657, in invoke\n'
'    return _process_result(sub_ctx.command.invoke(sub_ctx))\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/click/core.py", line 1657, in invoke\n'
'    return _process_result(sub_ctx.command.invoke(sub_ctx))\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/click/core.py", line 1404, in invoke\n'
'    return ctx.invoke(self.callback, **ctx.params)\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/click/core.py", line 760, in invoke\n'
'    return __callback(*args, **kwargs)\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/click/decorators.py", line 26, in '
'new_func\n'
'    return f(get_current_context(), *args, **kwargs)\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/datahub/telemetry/telemetry.py", line '
'317, in wrapper\n'
'    raise e\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/datahub/telemetry/telemetry.py", line '
'269, in wrapper\n'
'    res = func(*args, **kwargs)\n'
'File '
'"/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/datahub/utilities/memory_leak_detector.py", '
'line 102, in wrapper\n'
'    res = func(*args, **kwargs)\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/datahub/cli/ingest_cli.py", line 128, in '
'run\n'
'    raise e\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/datahub/cli/ingest_cli.py", line 114, in '
'run\n'
'    pipeline.run()\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/datahub/ingestion/run/pipeline.py", line '
'214, in run\n'
'    for wu in itertools.islice(\n'
'File '
'"/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/datahub/ingestion/source/sql/sql_common.py", '
'line 705, in get_workunits\n'
'    for inspector in self.get_inspectors():\n'
'File '
'"/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/datahub/ingestion/source/sql/sql_common.py", '
'line 509, in get_inspectors\n'
'    with engine.connect() as conn:\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2263, '
'in connect\n'
'    return self._connection_cls(self, **kwargs)\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 104, in '
'init\n'
'    else engine.raw_connection()\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2369, '
'in raw_connection\n'
'    return self._wrap_pool_connect(\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/engine/base.py", line 2336, '
'in _wrap_pool_connect\n'
'    return fn()\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 304, in '
'unique_connection\n'
'    return _ConnectionFairy._checkout(self)\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 778, in '
'_checkout\n'
'    fairy = _ConnectionRecord.checkout(pool)\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 495, in '
'checkout\n'
'    rec = pool._do_get()\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 140, in '
'do_get\n'
'    self.dec_overflow()\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line '
'68, in exit\n'
'    compat.raise(\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in '
'raise\n'
'    raise exception\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/pool/impl.py", line 137, in '
'_do_get\n'
'    return self._create_connection()\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 309, in '
'_create_connection\n'
'    return _ConnectionRecord(self)\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 440, in '
'init\n'
'    self.__connect(first_connect_check=True)\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 661, in '
'connect\n'
'    pool.logger.debug("Error on connect(): %s", e)\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line '
'68, in exit\n'
'    compat.raise(\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in '
'raise\n'
'    raise exception\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/pool/base.py", line 656, in '
'__connect\n'
'    connection = pool._invoke_creator(self)\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/engine/strategies.py", line '
'114, in connect\n'
'    return dialect.connect(*cargs, **cparams)\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/sqlalchemy/engine/default.py", line 508, '
'in connect\n'
'    return self.dbapi.connect(*cargs, **cparams)\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/pyhive/hive.py", line 126, in connect\n'
'    return Connection(*args, **kwargs)\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/pyhive/hive.py", line 267, in init\n'
'    self._transport.open()\n'
'File "/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/thrift_sasl/init.py", line 95, in '
'open\n'
'    raise TTransportException(type=TTransportException.NOT_OPEN,\n'
'\n'
"TTransportException: Bad status: 3 (b'Error validating the login')\n"
'[2022-05-11 08:12:08,295] INFO     {datahub.entrypoints:176} - DataHub CLI version: 0.8.34.1 at '
'/tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/lib/python3.9/site-packages/datahub/init.py\n'
'[2022-05-11 08:12:08,295] INFO     {datahub.entrypoints:179} - Python version: 3.9.9 (main, Dec 21 2021, 10:03:34) \n'
'[GCC 10.2.1 20210110] at /tmp/datahub/ingest/venv-1652100c-2a46-4666-8519-3ecd2b12cba1/bin/python3 on '
'Linux-3.10.0-693.el7.x86_64-x86_64-with-glibc2.31\n'
"[2022-05-11 08:12:08,295] INFO     {datahub.entrypoints:182} - GMS config {'models': {}, 'versions': {'linkedin/datahub': {'version': "
"'v0.8.34', 'commit': '5c64e9d5419afc85343307a7cfdc51b90c6f5e68'}}, 'managedIngestion': {'defaultCliVersion': '0.8.34.1', 'enabled': "
"True}, 'statefulIngestionCapable': True, 'supportsImpactAnalysis': True, 'telemetry': {'enabledCli': True, 'enabledIngestion': False}, "
"'datasetUrnNameCasing': False, 'retention': 'true', 'noCode': 'true'}\n",
"2022-05-11 08:12:09.294404 [exec_id=1652100c-2a46-4666-8519-3ecd2b12cba1] INFO: Failed to execute 'datahub ingest'",
'2022-05-11 08:12:09.294758 [exec_id=1652100c-2a46-4666-8519-3ecd2b12cba1] INFO: Caught exception EXECUTING '
'task_id=1652100c-2a46-4666-8519-3ecd2b12cba1, name=RUN_INGEST, stacktrace=Traceback (most recent call last):\n'
'  File "/usr/local/lib/python3.9/site-packages/acryl/executor/execution/default_executor.py", line 119, in execute_task\n'
'    self.event_loop.run_until_complete(task_future)\n'
'  File "/usr/local/lib/python3.9/site-packages/nest_asyncio.py", line 89, in run_until_complete\n'
'    return f.result()\n'
'  File "/usr/local/lib/python3.9/asyncio/futures.py", line 201, in result\n'
'    raise self._exception\n'
'  File "/usr/local/lib/python3.9/asyncio/tasks.py", line 256, in __step\n'
'    result = coro.send(None)\n'
'  File "/usr/local/lib/python3.9/site-packages/acryl/executor/execution/sub_process_ingestion_task.py", line 115, in execute\n'
'    raise TaskError("Failed to execute 'datahub ingest'")\n'
"acryl.executor.execution.task.TaskError: Failed to execute 'datahub ingest'\n"]}
Execution finished with errors.

python -m datahub ingest -c hive_to_datahub_rest.yml  no problem

It looks like a connection error is happening, what is the difference between the recipe you ran that failed and your other recipe? Are you able to share the recipe files you are using? Also be sure to mask any sensitive information in stack traces/shared files 😄

It looks like a connection error is happening, what is the difference between the recipe you ran that failed and your other recipe? Are you able to share the recipe files you are using? Also be sure to mask any sensitive information in stack traces/shared files 😄

I've solved it. Thank you

可以问一下如何解决的吗？我现在用ui界面创建也报错

这个是解决办法已经在github上写过了你可以查看一下，地址:https://github.com/datahub-project/datahub/issues/4955
---Original---
From: @.>
Date: Wed, Jul 20, 2022 15:56 PM
To: @.>;
Cc: @.>;"State @.>;
Subject: Re: [datahub-project/datahub] WebUi  Error adding data source hive  (Issue #4896)
可以问一下如何解决的吗？我现在用ui界面创建也报错
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you modified the open/close state.Message ID: @.***>

authors: RyanHolstien YTQ419 wtspring


text:
Make versions in chart explicit

This PR makes the versioning of the chart explicit, so we'll update it to line up with GitHub releases. The version of the chart isn't necessarily exactly the same as datalust/seq, but kept in-sync enough that users should be able to apply the {year}.{minor} versioning scheme to any Seq-related projects they interact with.
The process for making a release would then be:

Merge some changes to the chart definition
Create a release PR that updates the build number appropriately
After merging the release PR, create a GitHub release with the same version

New major/minor Seq versions will run similarly, except they'll likely only need to update the versions in the Chart.yaml file.

Error:  Chart.yaml: appVersion should be of type string but it's of type float64

🙄


text:
Float issue

I got an issue for the file pgsql_big_dedupe_example.py. Line 117, it says "'float' object cannot be interpreted as an integer".
More details :
File "", line 1, in
dedupe.Dedupe(fields, num_cores=4)
File "/Applications/anaconda/lib/python3.5/site-packages/dedupe/api.py", line 181, in init
super(DedupeMatching, self).init(*args, **kwargs)
File "/Applications/anaconda/lib/python3.5/site-packages/dedupe/api.py", line 627, in init
self.data_model = datamodel.DataModel(variable_definition)
File "/Applications/anaconda/lib/python3.5/site-packages/dedupe/datamodel.py", line 23, in init
primary_fields, variables = typifyFields(fields)
File "/Applications/anaconda/lib/python3.5/site-packages/dedupe/datamodel.py", line 158, in typifyFields
field_object = field_class(definition)
File "/Applications/anaconda/lib/python3.5/site-packages/dedupe/variables/categorical_type.py", line 23, in init
self.comparator = CategoricalComparator(categories)
File "/Applications/anaconda/lib/python3.5/site-packages/categorical/categorical.py", line 20, in init
response = responseVector(i, vector_length)
File "/Applications/anaconda/lib/python3.5/site-packages/categorical/categorical.py", line 44, in responseVector
response = numpy.zeros(vector_length)

I just pushed a new version of dedupe to pypi, download it and this should fix your problem. https://pypi.python.org/pypi/dedupe

You also need to upgrade https://pypi.python.org/pypi/categorical-distance

authors: clementcarrier fgregg

repo: datamade/dedupe-examples


text:
🛑 Invidious is down

In 1c4de46, Invidious (https://tube.mha.fi) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Invidious is back up in 3c1a48d.


text:
Querying out irregular companies

The dataset.py file has an "in Brazil" query which is meant to filter out companies that were mistakenly placed outside Brazil by it's coordinates.
The problem with that is that closed companies end up being queried out thus not allowing us to find suspicious reimbursements when considering the hypothesis explored in datasciencebr/serenata-de-amor#195
AFAIK this query is only necessary for traveled speeds classifier since it uses the companies coordinates to find outlier in the distance traveled.
tl;dr: We need a refactor

This was noticed while I was working on #26 and #28 also for #27 to properly work we need this query out of where it is today

repo: datasciencebr/rosie

license: mit


text:
account identity list unexpected data type

The dictionary that we get from client.account.identity.list() has an updated_at value that is an integer, not a datetime object as would be expected.
    {
        "api_key": "dff990e42c14ef5d5aa280b0e9fea9e2",
        "created_at": "Wed, 13 May 2015 10:46:05 GMT",
        "expires_at": null,
        "id": "5dbb799eea004fcb3e2d999d767e0a20",
        "label": "DataSift",
        "master": true,
        "status": "active",
        "updated_at": 1440604653
    }

Calling the API via curl returns all timestamps as integers; I believe this should be the new standard across all new endpoints that are implemented:
➜  ~  curl -X GET 'https://api.datasift.com/v1.2/account/identity' -H 'Auth: USERNAME:API_KEY' | jq .
{
  "count": 2,
  "page": 1,
  "pages": 1,
  "per_page": 25,
  "identities": [
    {
      "id": "ef554f96b...",
      "label": "...",
      "api_key": "1cefe96f4...",
      "status": "active",
      "master": true,
      "created_at": 1431529262,
      "updated_at": 1431535636,
      "expires_at": null
    },
    {
      "id": "a55677c3f622...",
      "label": "...",
      "api_key": "85fcd26e0...",
      "status": "active",
      "master": false,
      "created_at": 1432230266,
      "updated_at": 1432230266,
      "expires_at": null
    }
  ]
}

I'm not sure why created_at would be returned as a DateTime object

The output mapper determines which fields are converted in to DateTime objects. updated_at has now been added to this file and should be available on the next release.
If you wanted all fields to return as timestamps instead of datetimes you just need to edit the output_mapper.py and rebuild.

authors: dugjason paulmozo sojojo

repo: datasift/datasift-python