text:
a bug in automatic chunk? or my misunderstanding??

Hi,
we were trying to varying array chunk-size for a benchmarking study. [https://github.com/pangeo-data/benchmarking/issues/30]
But when choosing automatic chunk, for our case, certain chunk-size we have used does not work.  For example, we set array.chunk-size as 370MB, and set chunks='auto', chunk size created by Dask Array is '(380, 192, 320)' thus it is only 187MB of chunk.
If using chunk-size more than 380MB, this problem does not occur. But less than 370MB (even with around 100MB) this problem occurs.
I copy and past the outputs I get in a simple example below.  I hope you can point me out if this is a bug or not.
Thank you for your help !

import dask.array as da
import dask
dask.config.set({"array.chunk-size": '370MB'})
da.random.RandomState(0).standard_normal((20834, 384, 320), chunks='auto')
output: dask.array<standard_normal, shape=(20834, 384, 320), dtype=float64, chunksize=(380, 192, 320)>
380 * 190 * 320 * 8 = 186777600  => 187MB thus  ##not ok
import dask.array as da
import dask
dask.config.set({"array.chunk-size": '380MB'})
da.random.RandomState(0).standard_normal((20834, 384, 320), chunks='auto')
output: dask.array<standard_normal, shape=(20834, 384, 320), dtype=float64, chunksize=(386, 384, 320)>
380 * 384 * 320 * 8 = 379453440  => ok
import dask.array as da
import dask
dask.config.set({"array.chunk-size": '100MB'})
da.random.RandomState(0).standard_normal((20834, 384, 320), chunks='auto')
output: dask.array<standard_normal, shape=(20834, 384, 320), dtype=float64, chunksize=(232, 192, 160)>
** 232 * 192 * 160 * 8 = 57016320  => 57MB thus not ok**

@tinaok I get the same results as you. I don't think this is a bug but rather how dask tries to keep the chunk shapes relatively the same.   Notice that the dimensions of the differing chunks are off by factors of two 192-> 384 and 320->160.  In your particular case, the 370 doesn't chunk nicely into equally shaped chunk, but 380 does

Ben's assessment is correct.  Dask's auto chunking heuristics are
comfortable accepting chunk sizes down to half of what was requested if it
can make things line up nicely.
On Sun, Sep 29, 2019 at 5:37 PM Benjamin Zaitlen notifications@github.com
wrote:

@tinaok https://github.com/tinaok I get the same results as you. I
don't think this is a bug but rather how dask tries to keep the chunk
shapes relatively the same. Notice that the dimensions of the differing
chunks are off by factors of two 192-> 384 and 320->160. In your
particular case, the 370 doesn't chunk nicely into equally shaped chunk,
but 380 does
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/5439?email_source=notifications&email_token=AACKZTDX7DX3R4VRGKPSFVDQMEU2VA5CNFSM4I3UIX2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD74AQEA#issuecomment-536348688,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACKZTBD5UMVX6ZXZ3HFHILQMEU2VANCNFSM4I3UIX2A
.

Thank you @quasiben and @mrocklin
we made a benchmark study [https://github.com/pangeo-data/benchmarking] comparing 'manual' chunking and automatic chunking by varying chunk size between 64MB-> 128MB -> 256MB -> 512MB for some geophysical operations.
When I set 64MB, automatic chunking gives me chunksize=(199, 192, 160) thus 49MB
128MB ->chunksize=(251, 192, 160) thus 62MB (slightly less than half)
256MB ->chunksize=(317, 192, 160) thus 78MB (much less than half!)
512MB -> chunksize=(520, 384, 320) thus 511MB (good value ;-)
I see for example setting 256MB and having 78MB may deviate too much.  Is there any way to give some kind of parameter to dask that it use chunk size at least for example 0.8 to 1.2 times the size we asked it to use?

Not currently, but the code is here in case you'd like to engage further:
https://github.com/dask/dask/blob/f6c2449a000a72f05e41d0208d80696bc257d2ab/dask/array/core.py#L2474-L2597

Thanks @mrocklin
I look at the code you mentioned, but I could not found the part accepting chunk size down to half, can you plz help me ?
I thought I can may be modify the minimum 'half' part, to somehow larger value, but may be it is not as simple as that?

Unfortunately I don't have great recall over the full codebase.  I would
have to investigate and understand the code, just as you are now doing.
It's also very possible that I am mis-remembering the half aspect.
On Tue, Oct 1, 2019 at 9:24 AM Tina Odaka notifications@github.com wrote:

Thanks @mrocklin https://github.com/mrocklin
I look at the code you mentioned, but I could not found the part accepting
chunk size down to half, can you plz help me ?
I thought I can may be modify the minimum 'half' part, to somehow larger
value, but may be it is not as simple as that?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/5439?email_source=notifications&email_token=AACKZTAZZ6F3OSWMA4ZWIQLQMNMTVA5CNFSM4I3UIX2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEABOPHQ#issuecomment-537061278,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACKZTBOWUSYCW4HXFCNIQ3QMNMTVANCNFSM4I3UIX2A
.

@tinaok were you able to look into the auto chunking code at all? It might be worthwhile putting an import pdb; pdb.set_trace() in that auto_chunks function and stepping through things line-by-line to see how we arrive at the final number.

Hi @TomAugspurger sorry for my late response, I started to look at the code, and understood some part, made some test on change, but then I did not have time to continue.  I'll get back to it next week.

Hi @mrocklin
here is the implementation of the bug fix that I showed you in SC19 conference;
basic idea is indicated in the comments.
and here is the test code which shows that this make the chunk size better.
----test code ----
import dask.array as da
from distributed.utils import parse_bytes
import dask
import numpy  as np
limit_list=['64MB','128MB','256MB','512MB']
timesteps=20834
lat=320
lon=384
shape=(timesteps,lon,lat)
chunks=("auto","auto","auto")
previous_chunks=None
for limit in limit_list:
print('---test with shape=',shape,'and limit=',limit)
dask.config.set({"array.chunk-size": limit})
x =da.random.RandomState(0).standard_normal(shape, chunks=chunks  )
dtype=x.dtype
print('--chunk computed from dask',x.chunksize,' ratio is:',np.prod(x.chunksize)*dtype.itemsize/parse_bytes(limit))

non changed version gives
---test with shape= (20834, 384, 320) and limit= 64MB
--chunk computed from dask (199, 192, 160)  ratio is: 0.76416
---test with shape= (20834, 384, 320) and limit= 128MB
--chunk computed from dask (251, 192, 160)  ratio is: 0.48192
---test with shape= (20834, 384, 320) and limit= 256MB
--chunk computed from dask (317, 192, 160)  ratio is: 0.30432
---test with shape= (20834, 384, 320) and limit= 512MB
--chunk computed from dask (520, 384, 320)  ratio is: 0.9984
with my PR, it gives
---test with shape= (20834, 384, 320) and limit= 64MB
--chunk computed from dask (1894, 64, 64)  ratio is: 0.969728
---test with shape= (20834, 384, 320) and limit= 128MB
--chunk computed from dask (3837, 64, 64)  ratio is: 0.982272
---test with shape= (20834, 384, 320) and limit= 256MB
--chunk computed from dask (4835, 64, 80)  ratio is: 0.7736
---test with shape= (20834, 384, 320) and limit= 512MB
--chunk computed from dask (520, 384, 320)  ratio is: 0.9984
64MB, 128MB, 256MB gives chunk which fits better to defined array.chunk-size.

authors: TomAugspurger mrocklin quasiben tinaok


text:
Issue with trim_excess in dask.array.coarsen

I am trying to use dask.array.coarsen on 3-d arrays and am finding that the result depends on the chunking of the data, which doesn't seem right. The following example illustrates the issue:
In [1]: import numpy as np

In [2]: import dask.array as da

In [3]: array1 = da.random.random((623, 768, 768)).rechunk((322, 256, 256))

In [4]: array2 = da.random.random((623, 768, 768)).rechunk((623, 768, 768))

In [5]: import numpy as np

In [6]: da.coarsen(np.nanmean, array1, {1: 3}, trim_excess=True).shape
Out[6]: (623, 255, 768)

In [7]: da.coarsen(np.nanmean, array2, {1: 3}, trim_excess=True).shape
Out[7]: (623, 256, 768)

Since 768 is divisible by 3, I would expect the final array to have shape (623, 256, 768) as for array2. The result for array1 seems incorrect, so I think this is a bug?

I think the issue is that the coarsen is happening on each chunk so you are getting a different result than expected because 256 is not divisible by 3. That is why you get an error with array1 if you take out trim_excess=True. So I am not entirely sure if this is worth changing or not.
The relevant code is here:
https://github.com/dask/dask/blob/366c7998398bc778c4aa5f4b6bb22c25b584fbc1/dask/array/routines.py#L1303:L1305

I feel like a top-level function such as this should probably return the same result independent of how the data is chunked (unlike a function like say map_blocks which explicitly operates on chunks)

da.coarsen doesn't apply any overlaps between neighboring blocks, so the output is always dependent on the chunking of the input. There may be a bug here in terms of the output shape; I'm not familiar enough with coarsen to know, perhaps @mrocklin remembers - this method is one of the first things added to dask in 2015 and hasn't been touched much since. However, the output values will still differ depending on chunking, since no windowing is used.
For windowed computations that don't depend on how the data is chunked I recommend looking at da.map_overlap.

The result for array1 seems incorrect, so I think this is a bug?

Looks like it to me

I feel like a top-level function such as this should probably return the same result independent of how the data is chunked

+1
I think that the solution here is to verify that all chunks of the coarsened dimensions align with the coarsening, and if not, to call a minimally invasive rechunk automatically within this function.
This shouldn't add too much extra cost, and should make sure that we get correct and consistent results.

If someone wanted to solve this I would recommend the following approach:
Make a function that takes in the chunks for a single dimension and an integer coarsenting value and produces a new chunking that is aligned with that value
def aligned_coarse_chunks(chunks: List[int], multiple: int) -> List[int]:
    """ Returns a new chunking aligned with the coarsening multiple"""

>>> aligned_coarse_chunks((20, 10, 15, 23, 24), 10)
(20, 10, 20, 20, 20, 2)

Note that the sum of the two lists should be the same, all numbers should be multiples of the multiple, and any excess should be at the end.
Then I would apply that function to all of the dimensions that are being chunked with the right multiple.  If the resulting chunking is different from the inputs .chunks value then I would call rechunk with that combined tuple.

Here's my attempt at this function

source: gharchive/issue

authors: astrofrog dcherian jcrist jsignell mrocklin

repo: dask/dask

license_type: permissive

license_source: github-api


text:
[WIP] Fixes Series comparison to NumPy scalars

Fixes #3392. This PR fixes Series comparison to NumPy scalars, which currently fail.

[ ] Tests added / passed
[x] Passes flake8 dask
[x] Fully documented, including docs/source/changelog.rst for all changes
and one of the docs/source/*-api.rst files for new API

Thank you for taking this on @jrbourbeau !
@philippjfr does this resolve your original issue?

The original regression test I added was using the dask.array.greater_equal ufunc, which actually doesn't raise any errors. So I replaced the added test with the original code snippet given in issue #3392.

It looks like there are some genuine testing errors in travis-ci.  I suspect that this fix works for some versions, but not others.

It looks like the NumPy 1.11.0 tests on Travis are failing because I've only modified __array_ufunc__ (which is new in NumPy version 1.13) and not __array_wrap__ (which is what's being used in the 1.11.0 tests).

Looks like there was an issue during the installation process on some of the Travis tests (see https://travis-ci.org/dask/dask/jobs/371219552#L827-L832). Could someone please restart the failed builds?

Done
On Wed, Apr 25, 2018 at 4:36 PM, James Bourbeau notifications@github.com
wrote:

Looks like there was an issue during the installation process on some of
the Travis tests (see https://travis-ci.org/dask/
dask/jobs/371219552#L827-L832). Could someone please restart the failed
builds?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/pull/3436#issuecomment-384425691, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszC1peBFOtr0vwRmaY6xgsQafCqjZks5tsN5UgaJpZM4ThuME
.

Does the same wrapping need to be applied to dd.Index?

Just to clarify, do you mean that dd.Index.__array_wrap__ should also have a check for 0-dimensional ndarrays? Currently, if a 0-dimensional ndarray is given to dd.Index.__array_wrap__ it raises
TypeError: len() of unsized object

How about (np.array(1) + df.index).compute()

Ah that return TypeError, from a NotImplemented.
On Thu, Apr 26, 2018 at 1:54 PM, Martin Durant notifications@github.com
wrote:

How about (np.array(1) + df.index).compute()
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/pull/3436#issuecomment-384752903, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ABQHIvatWdzmh5BdJaXy9N2mSadSlMMQks5tshf4gaJpZM4ThuME
.

Hmmmm I'm able to reproduce Tom's example. I.e.
import numpy as np
import dask.dataframe as dd
import pandas as pd

df = dd.from_pandas(pd.Series([1, 2, 3], index=[0, 2, 3]), 2)

print((df.index + np.array(1)).compute())

Gives Int64Index([1, 3, 4], dtype='int64').
But reversing the addition order changes behavior depending on which numpy version I'm using.

With numpy 1.11.0, np.array(1) + df.index returns a pandas Index Int64Index([1, 3, 4], dtype='int64')

With numpy 1.14.2, np.array(1) + df.index returns a dask Index. After compute() is called it gives Int64Index([1, 3, 4], dtype='int64')

I take it this difference has to do with the fact that __array_ufunc__ is using elemwise while __array_wrap__ simply returns a pandas Index, i.e. pd.Index(array, name=self.name).

Checking in here.  What is the status of this?

Haven't looked closely, but I think everything I looked at last time was fine. I'd be happy with merging this as is, and opening a separate issue to track the (presumably less common) ops on the Index class, if there's even an issue there.

Just updated comment as per Tom's suggestion

Great.  This is in. Thanks @jrbourbeau !

source: gharchive/pull-request

authors: TomAugspurger jrbourbeau martindurant mrocklin


text:
Fix kurtosis calc for single dimension input array

[x] Tests added / passed
[ ] Passes black dask / flake8 dask

Related to #5176

Thanks for the PR, overall this seems fine to me. cc @TomAugspurger for a quick review since he's the last one to touch this code.

authors: andrethrill jcrist


text:
[WIP] Resolve several documentation build warnings

This doesn't get all the warnings yet. I plan to keep pushing to this PR
xref #5610

Woot.
On Fri, Dec 6, 2019 at 2:51 PM James Bourbeau notifications@github.com
wrote:

This doesn't get all the warnings yet. I plan to keep pushing to this PR
xref #5610 https://github.com/dask/dask/issues/5610
You can view, comment on, or merge this pull request online at:
https://github.com/dask/dask/pull/5685
Commit Summary

Resolve several documentation build warnings
Merge remote-tracking branch 'upstream/master' into
array-doc-warnings

File Changes

M dask/array/core.py
https://github.com/dask/dask/pull/5685/files#diff-0 (35)
M dask/array/creation.py
https://github.com/dask/dask/pull/5685/files#diff-1 (11)
M dask/array/percentile.py
https://github.com/dask/dask/pull/5685/files#diff-2 (14)
M dask/array/routines.py
https://github.com/dask/dask/pull/5685/files#diff-3 (3)
M dask/dataframe/core.py
https://github.com/dask/dask/pull/5685/files#diff-4 (2)
M dask/dataframe/io/parquet/core.py
https://github.com/dask/dask/pull/5685/files#diff-5 (2)
M dask/datasets.py
https://github.com/dask/dask/pull/5685/files#diff-6 (4)
M docs/source/array-creation.rst
https://github.com/dask/dask/pull/5685/files#diff-7 (8)
M docs/source/dataframe-api.rst
https://github.com/dask/dask/pull/5685/files#diff-8 (8)
M docs/source/setup/custom-startup.rst
https://github.com/dask/dask/pull/5685/files#diff-9 (6)

Patch Links:

https://github.com/dask/dask/pull/5685.patch
https://github.com/dask/dask/pull/5685.diff

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/pull/5685?email_source=notifications&email_token=AACKZTAMFSEZBZWQ36TIK4LQXLJODA5CNFSM4JXC6C42YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4H6YWGQQ,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACKZTERJXACFEDLZ6KMP5LQXLJODANCNFSM4JXC6C4Q
.

Added scipy as a dependency foe building the docs to resolve stats warnings like:
WARNING: autodoc: failed to import function 'stats.chisquare' from module 'dask.array'; the following exception was raised:
`dask.array.stats` requires `scipy` to be installed.

(ref https://github.com/dask/dask/pull/5685/checks?check_run_id=337542588#step:6:128)

This LGTM. Feel free to merge when ready @jrbourbeau.

authors: jcrist jrbourbeau mrocklin


text:
Walk method doesn't match python standard lib interface

The implementation of walk in s3fs returns a flat list of all paths encountered when traversing the tree. In contrast, os.walk from the standard library returns an iterator of dirname, dirs, files, where dirname is the current directory prefix in the traversal, and fils and dirs are the names (relative, not absolute) of all files and directories respectively.
This difference in behavior is confusing for those familiar with the standard os.walk method, and also makes providing the interface used by pyarrow (which does match the standard library) tricky.
Pyarrow has created a wrapper class that provides the proper method interface, see here. Is there any reason not to just upstream this? We'd potentially need to figure out a good deprecation path, as the current behavior may be used by users.

I use (and vastly prefer!) the flat list returned by S3FileSystem.walk. That said, @jcrist is right: it's different from os.walk, and the difference was confusing.
Making walk more like os.walk would break my current code, but I'm OK with that. It's an easy fix, and I should rewrite it anyway to avoid any possible clash with pyarrow, which I also use heavily.
Does anyone here know how many others would be affected by changing walk? In the docs, it's in the S3FileSystem summary but not the API summary.

In filesystem_spec, I am proposing making walk match the python API, but provide find, which retains the current behaviour (list of paths), similar to the posix find command without the templates. Templates could be added later...

That sounds good to me. All I really need is something like posix find * -type f, but for an S3 bucket.

Implemented in fsspec

authors: jcrist martindurant samkennerly


text:
Data All Architecture

Is Data all monolithic or microservice architecture

Data.all is a monolith in the sense that the entire application shares the same underlying infrastructure to serve API requests and persist data in Aurora DB. For example, when I create a data.all worksheet and/or I link a data.all environment - the same Lambda compute is handling the API request and communicating with the same datastore (Aurora DB).
That being said, the codebase is modularized in the sense that each “data.all module” is isolated from one another and can be plugged in (or out) of your data.all deployment at any time (via config.json file). For example, I can disable the worksheets module and I will no longer have any UI views, API handlers, etc. with respect to worksheets but the other data.all application components will continue to run fine.
Some more in-depth detail on the way data.all modules are physically structured in the codebase and loaded into the application code can be found in our Code Walkthrough documentation.
Please let us know if you have any additional questions on the above, thanks!

Hello, it appears that the question has been addressed. I'll close it for now, but feel free to reopen if you have additional questions.

authors: anmolsgandhi noah-paige sandeephs1

repo: data-dot-all/dataall


text:
[Bug] axis is partial in img download

Describe the bug
axis is partial in image download
To Reproduce
See injured_count_by_accident_year widget in this newsflash
Y axis numbers are "cut"
Expected behavior
axis should be displayed as it does in the web.
Screenshots

Environment
Browser: Chrome
Environment: Local / Staging / etc.

@shaked-hayek can you take a quick look on this one?
I tried to play a little with height, width, margins, and other variables in html2canvas (see branch bug-1136-axis-partial)
However, issue is not fixed (I reverted these changes and they are not in dev nor in prod)

The branch: https://github.com/data-for-change/anyway-newsflash-infographics/tree/bug-1136-axis-partial

@shaked-hayek perhaps using another package of html2canvas will solve the issue?
Just wondering whether it can be as simple as that :)

Perhaps https://github.com/tsayen/dom-to-image library can be a better fit, but needs testing

Also https://github.com/bubkoo/html-to-image is another alternative to check

html-to-image didn't have the built in onclone In order to include removeMapControllers
Hence I used html-to-image for most images and html2canvas for map images

license: MIT


text:
CARDS-2067: Clicking on the grey highlighting of a multiple choice option should have the same effect as clicking on the option text or radio/checkbox

Testing:

Create a new form with multiple choice options (for example a prems survey`)
Hover one of the options - notice a gray highlighting that takes up the whole width of the question box
Click on the highlighted area (but outside of the radiobutton or label) -> the option should be selected
Turn the question into multiselect by setting maxAnswers = 0, then check that clicking on the highlighting works
Add NA/None of the above options, check that clicking on their highlighted area instead of the checkbox or label preserved the NA/None of the above functionality

On a multivalued checkbox list, clicking on the actual text doesn't work. Clicking on the gray part or the actual checkbox itself works, but clicking on the text does nothing. Probably because there are two onclick calls, once for the <ListItem and once for the <Checkbox. Can you try stopping the even on the outer click, or maybe remove the onclick handler from the Radio / Checkbox?

Addressed in 3eff351.

repo: data-team-uhn/cards


text:
Logging & error message formatting on python models

Describe the feature
This is a request for improvements in formatting of tracebacks when exceptions occur. Users have to scroll through the source code of the model definition itself before getting to the error message. There are also some cases in which error messages are logged in a format that's unreadable, e.g.
 Python model failed with traceback as:
  [0;31m---------------------------------------------------------------------------
  [0;31mTypeError                                 Traceback (most recent call last)
  File [0;32m<command--1>:152
  [1;32m    143 [38;5;66;03m# COMMAND ----------[39;00m
  [1;32m    144
  [1;32m    145 [38;5;66;03m# how to execute python model in notebook[39;00m
  [0;32m   (...)
  [1;32m    149
  [1;32m    150 [38;5;66;03m# --- Autogenerated dbt materialization code. --- #[39;00m
  [1;32m    151 dbt [38;5;241m=[39m dbtObj(spark[38;5;241m.[39mtable)
  [0;32m--> 152 df [38;5;241m=[39m model(dbt, spark)
   ...
  [1;32m     87     dfs [38;5;241m=[39m [
  [1;32m     88         time_window_rollup(weeks, df, agg, miss_threshold)
  [1;32m     89         [38;5;241m.[39mwithColumn([38;5;124m"[39m[38;5;124mlabel_index[39m[38;5;124m"[39m, F[38;5;241m.[39mlit(i))
  [1;32m     90         [38;5;241m.[39mwithColumn([38;5;124m"[39m[38;5;124mlabel[39m[38;5;124m"[39m, F[38;5;241m.[39mlit(label))
  [1;32m     91         [38;5;28;01mfor[39;00m i, (label, df, agg, miss_threshold) [38;5;129;01min[39;00m [38;5;28menumerate[39m(kpis)
  [1;32m     92     ]
  [1;32m     93     res [38;5;241m=[39m reduce([38;5;28;01mlambda[39;00m x, y: x[38;5;241m.[39munion(y), dfs)

  [0;31mTypeError: 'tuple' object is not callable

Describe alternatives you've considered
None. This feels like either an adapter issue or a dbt-core issue. Please comment on where this issue should be properly logged.
Additional context
N/A
Who will this benefit?
All users of dbt-databricks seeking to implement python models as part of their DAGs.
Are you interested in contributing this feature?
Potentially.

Hey @andrefurlan-db apologies for the direct ping, but any thoughts on this issue? From our team's understanding, error formatting lives in adapter space, and I've heard a few other mentions of this recently.

repo: databricks/dbt-databricks


text:
Add setitem for loc/iloc with Series.

Support __setitem__ for loc/iloc with Series.
e.g.,
>>> kser = ks.Series([1, 2, 3], index=["cobra", "viper", "sidewinder"])
>>> kser
cobra         1
viper         2
sidewinder    3
Name: 0, dtype: int64
>>> kser.loc[kser % 2 == 1] = -kser
>>> kser
cobra        -1
viper         2
sidewinder   -3
Name: 0, dtype: int64
>>> kser.loc[["viper", "sidewinder"]] = 10
>>> kser
cobra         -1
viper         10
sidewinder    10
Name: 0, dtype: int64
>>> kser.loc["viper"] = 50
>>> kser
cobra         -1
viper         50
sidewinder    10
Name: 0, dtype: int64
>>> kser.loc[:] = 10
>>> kser
cobra         10
viper         10
sidewinder    10
Name: 0, dtype: int64
>>> kser.loc[:"viper"] = 20
>>> kser
cobra         20
viper         20
sidewinder    10
Name: 0, dtype: int64
>>> kser.loc["viper":] = 30
>>> kser
cobra         20
viper         30
sidewinder    30
Name: 0, dtype: int64

or
>>> kser = ks.Series([1, 2, 3], index=["cobra", "viper", "sidewinder"])
>>> kser
cobra         1
viper         2
sidewinder    3
Name: 0, dtype: int64
>>> kser.iloc[[1, 2]] = 10
>>> kser
cobra          1
viper         10
sidewinder    10
Name: 0, dtype: int64
>>> kser.iloc[1] = 50
>>> kser
cobra          1
viper         50
sidewinder    10
Name: 0, dtype: int64
>>> kser.iloc[:] = 10
>>> kser
cobra         10
viper         10
sidewinder    10
Name: 0, dtype: int64
>>> kser.iloc[:1] = 20
>>> kser
cobra         20
viper         10
sidewinder    10
Name: 0, dtype: int64
>>> kser.iloc[1:] = 30
>>> kser
cobra         20
viper         30
sidewinder    30
Name: 0, dtype: int64

Codecov Report

Merging #1367 into master will decrease coverage by 2.18%.
The diff coverage is 91.17%.

@@            Coverage Diff             @@
##           master    #1367      +/-   ##
==========================================
- Coverage   95.24%   93.06%   -2.19%
==========================================
  Files          34       34
  Lines        7724     7775      +51
==========================================
- Hits         7357     7236     -121
- Misses        367      539     +172

Impacted Files
Coverage Δ

databricks/koalas/indexing.py
94.51% <91.17%> (-0.05%)
:arrow_down:

databricks/koalas/usage_logging/__init__.py
24.32% <0.00%> (-71.18%)
:arrow_down:

databricks/koalas/usage_logging/usage_logger.py
50.00% <0.00%> (-50.00%)
:arrow_down:

databricks/koalas/__init__.py
78.72% <0.00%> (-14.90%)
:arrow_down:

databricks/conftest.py
88.67% <0.00%> (-7.55%)
:arrow_down:

databricks/koalas/frame.py
93.81% <0.00%> (-3.02%)
:arrow_down:

databricks/koalas/namespace.py
87.29% <0.00%> (-1.39%)
:arrow_down:

databricks/koalas/plot.py
93.33% <0.00%> (-0.96%)
:arrow_down:

databricks/koalas/generic.py
97.13% <0.00%> (-0.41%)
:arrow_down:

databricks/koalas/missing/series.py
100.00% <0.00%> (ø)

... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 704747f...ec3cbd9. Read the comment docs.

LGTM if these commented tests are passed.

I'd suspect the test failure in the pandas 1.0 env was a pandas bug..
pser.loc[("x", "viper"):"y"] = pser * 20

authors: codecov-io itholic ueshin

repo: databricks/koalas


text:
Add spark_schema and print_schema.

Add spark_schema and print_schema to see the underlying Spark schema since dtypes are sometimes ambiguous.

Codecov Report

Merging #1446 into master will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #1446   +/-   ##
=======================================
  Coverage   94.00%   94.00%
=======================================
  Files          36       36
  Lines        8342     8346    +4
=======================================
+ Hits         7842     7846    +4
  Misses        500      500

Impacted Files
Coverage Δ

databricks/koalas/frame.py
95.57% <100.00%> (+<0.01%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7ea6739...d272227. Read the comment docs.

authors: codecov-io ueshin


text:
Add same_anchor util.

Adding same_anchor util to check if the anchors of the given DataFrame or Series are the same or not.

LGTM

Codecov Report

Merging #1521 into master will decrease coverage by 1.99%.
The diff coverage is 96.29%.

@@            Coverage Diff             @@
##           master    #1521      +/-   ##
==========================================
- Coverage   93.87%   91.87%   -2.00%
==========================================
  Files          36       36
  Lines        8469     8482      +13
==========================================
- Hits         7950     7793     -157
- Misses        519      689     +170

Impacted Files
Coverage Δ

databricks/koalas/utils.py
94.11% <93.33%> (-0.12%)
:arrow_down:

databricks/koalas/base.py
98.02% <100.00%> (ø)

databricks/koalas/frame.py
92.58% <100.00%> (-2.82%)
:arrow_down:

databricks/koalas/groupby.py
90.26% <100.00%> (ø)

databricks/koalas/indexing.py
92.16% <100.00%> (ø)

databricks/koalas/series.py
97.72% <100.00%> (ø)

databricks/koalas/usage_logging/__init__.py
24.32% <0.00%> (-70.28%)
:arrow_down:

databricks/koalas/usage_logging/usage_logger.py
50.00% <0.00%> (-50.00%)
:arrow_down:

databricks/koalas/__init__.py
73.07% <0.00%> (-13.47%)
:arrow_down:

databricks/conftest.py
88.67% <0.00%> (-7.55%)
:arrow_down:

... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eb4d21c...0a23ce8. Read the comment docs.

Thanks! merging.

authors: codecov-commenter itholic ueshin


text:
Date Conversion from string yyyymmdd to redshift date format YYYY-MM-DD

I have source field date coming in YYYYMMDD format. When trying load on redshift it is giving error saying that few character then expected. expected format is YYYY-MM-DD format.
how to convert YYYYMMDD string TO YYYY-MM-DD date format

Looks like this is duplicated by #242, so let's continue the discussion there instead.

Thanks Josh for prompt reply. Is there any solution available.
From: Josh Rosen notifications@github.com
Reply-To: databricks/spark-redshift reply@reply.github.com
Date: Wednesday, July 20, 2016 at 2:06 PM
To: databricks/spark-redshift spark-redshift@noreply.github.com
Cc: Manisha Sojitra Manisha.Sojitra@sheknows.com, Author author@noreply.github.com
Subject: Re: [databricks/spark-redshift] Date Conversion from string yyyymmdd to redshift date format YYYY-MM-DD (#241)
Looks like this is duplicated by #242https://github.com/databricks/spark-redshift/issues/242, so let's continue the discussion there instead.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/databricks/spark-redshift/issues/241#issuecomment-234031834, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ATPW4uUPVbtK9BXGj_6aYGVMm386RlLjks5qXmOtgaJpZM4JQaUi.

repo: databricks/spark-redshift


text:
Update NL integration test goldens

Updates our NL integration test goldens via ./run_test.sh -g.
Per our release docs, our NL tests rely on staging mixer, so the goldens need to be updated after a mixer release to staging. These updated goldens reflect the data changes from this mixer commit: https://github.com/datacommonsorg/mixer/commit/75c03483a4e843bb77ad71c3e0a6694a2ff39dd0

@chejennifer @pradh These changes just look like floating point error-esque differences. Do we need this update?

thanks! but did you mean to include the mixer submodule change?

Yup, wanted to keep the changes together with the mixer version that generated these files -- was waiting on https://github.com/datacommonsorg/website/pull/4522 to merge. The mixer change diffs have gone away after updating the branch with latest changes.

repo: datacommonsorg/website


text:
Correction for raw affiliation Ecole Nationale Supérieure de Chimie de Rennes, Université de Rennes, CNRS, ISCR—UMR 6226, 35000 Rennes, France

Correction needed for raw affiliation Ecole Nationale Supérieure de Chimie de Rennes, Université de Rennes, CNRS, ISCR—UMR 6226, 35000 Rennes, France
raw_affiliation_name: Ecole Nationale Supérieure de Chimie de Rennes, Université de Rennes, CNRS, ISCR—UMR 6226, 35000 Rennes, France
new_rors: 01h0ffh48;015m7wh34;02feahw73;00adwkx90
previous_rors: 01h0ffh48;015m7wh34;02feahw73
works_examples: W4319158004
contact: 42ad736aedc05430c1ebfe09740f64a3:9de20ff15ca5d6283e0ac9a0fb101716 @ univ-rennes.fr

This issue was accepted and ingested by the OpenAlex team on 2024-10-10. The new affiliations should be visible within the next 7 days.

repo: dataesr/openalex-affiliations


text:
Correction for raw affiliation Cedex 9; France.; La Réunion; Peuplements Végétaux et Bioaggresseurs en Milieu Tropical; Saint-Denis Messag; UMR C53 Cirad-Université; Université de La Réunion

Correction needed for raw affiliation Cedex 9; France.; La Réunion; Peuplements Végétaux et Bioaggresseurs en Milieu Tropical; Saint-Denis Messag; UMR C53 Cirad-Université; Université de La Réunion
raw_affiliation_name: Cedex 9; France.; La Réunion; Peuplements Végétaux et Bioaggresseurs en Milieu Tropical; Saint-Denis Messag; UMR C53 Cirad-Université; Université de La Réunion
new_rors: 03mj14b52
previous_rors: 05kxt4539;02s27py21;03mj14b52
works_examples: W4389330451
contact: 736d766631b64d5da085270a26b223a4:ba20e97e893d914a46555cc77b2c48 @ recherche.gouv.fr

This issue was accepted and ingested by the OpenAlex team on 2024-11-01. The new affiliations should be visible within the next 7 days.


text:
Correction for raw affiliation Cirad, UMR G-EAU, Montpellier University, 361 rue Jean-François Breton, 34196 Montpellier, France; INAT, 43 avenue Charles Nicolle, Tunis 1082, Tunisia

Correction needed for raw affiliation Cirad, UMR G-EAU, Montpellier University, 361 rue Jean-François Breton, 34196 Montpellier, France; INAT, 43 avenue Charles Nicolle, Tunis 1082, Tunisia
raw_affiliation_name: Cirad, UMR G-EAU, Montpellier University, 361 rue Jean-François Breton, 34196 Montpellier, France; INAT, 43 avenue Charles Nicolle, Tunis 1082, Tunisia
new_rors: 051escj72;02zdnfn05;020g0rw41
previous_rors: 051escj72;02zdnfn05
works_examples: W4403036248
contact: 09a932684332011620ab18fc762a8e3d:28e0b003a3e7ecba3f142d48b10743c41d52 @ inrae.fr

This issue was accepted and ingested by the OpenAlex team on 2024-12-19. The new affiliations should be visible within the next 7 days.


text:
Add icon for Intellijidea toolbox

now in toolbox it could be seen as

or in recent projects in idea

Codecov Report

Merging #468 (0e392b4) into main (643c7d4) will decrease coverage by 0.13%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##               main     #468      +/-   ##
============================================
- Coverage     93.03%   92.90%   -0.14%
+ Complexity     2101     2100       -1
============================================
  Files           233      233
  Lines          4381     4383       +2
  Branches        450      450
============================================
- Hits           4076     4072       -4
- Misses          186      189       +3
- Partials        119      122       +3

Impacted Files
Coverage Δ

...rc/main/java/net/datafaker/service/FakeValues.java
79.78% <0.00%> (-2.13%)
:arrow_down:

.../java/net/datafaker/service/FakeValuesService.java
86.45% <0.00%> (-0.87%)
:arrow_down:

...java/net/datafaker/providers/base/PhoneNumber.java
100.00% <0.00%> (ø)

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

@bodiam in case you have better icon we can use it (this one is just a result of compression of existing one to decrease file size)

authors: codecov-commenter snuyanzin

repo: datafaker-net/datafaker


text:
README: fix error in Python import statement

I had to adjust this to get the the import to work.

Thanks!

authors: erezsh jaypeedevlin

repo: datafold/data-diff


text:
Road to v0.2

This issue for setting the specific issues we would like completed before a v0.2 release.  My hope is that there can be a release by end of Sept / Beginning of October.

[x] https://github.com/datafusion-contrib/datafusion-dft/issues/137
[x] https://github.com/datafusion-contrib/datafusion-dft/issues/138
[x] https://github.com/datafusion-contrib/datafusion-dft/issues/128
[ ] (REQUIRED) https://github.com/datafusion-contrib/datafusion-dft/issues/144
[ ] (REQUIRED) https://github.com/datafusion-contrib/datafusion-dft/issues/125
[x] https://github.com/datafusion-contrib/datafusion-dft/issues/153
[x] (REQUIRED) https://github.com/datafusion-contrib/datafusion-dft/issues/148
[ ] (REQUIRED) https://github.com/datafusion-contrib/datafusion-dft/issues/158
[x] https://github.com/datafusion-contrib/datafusion-dft/issues/159
[x] https://github.com/datafusion-contrib/datafusion-dft/issues/166
[x] (REQUIRED) https://github.com/datafusion-contrib/datafusion-dft/issues/167
[x] (REQUIRED) https://github.com/datafusion-contrib/datafusion-dft/issues/170
[x] https://github.com/datafusion-contrib/datafusion-dft/issues/168
[x] https://github.com/datafusion-contrib/datafusion-dft/issues/173
[x] https://github.com/datafusion-contrib/datafusion-dft/issues/175
[x] https://github.com/datafusion-contrib/datafusion-dft/issues/177
[x] (REQUIRED) Benchmark from CLI
[ ] (REQUIRED) Analyze from CLI
[x] https://github.com/datafusion-contrib/datafusion-dft/issues/213
[ ] https://github.com/datafusion-contrib/datafusion-dft/issues/224

@alamb let me know if there is anything you would like to add

I think we should consider adding #148  as well.

So I think it turns out that the DDL issue I have is a bug with datafusion registering external tables to a schema.
I have this in my ~/.datafusion/.datafusionrc`
CREATE SCHEMA staging;

CREATE TABLE staging.foo AS VALUES (1);
CREATE EXTERNAL TABLE staging.min_aggs STORED AS PARQUET LOCATION 'ny2://atlas/sip/aggregates/minute_by_ticker_monthly_v2/year_month=2015-08/data.01.parquet';

But when I run SHOW TABLES I get the following:

Notice how the schema isnt picked up for the the external table.  I dont see anything in the docs for CREATE EXTERNAL TABLE that would suggest this shouldnt work but maybe im misunderstanding.
@alamb do you have a view on this?  If you agree its a bug in datafusion then i can create a ticket there.

I agree it is a datafusion bug. I will file a ticket

Is there anything that the current trunk version does worse than v0.1?

@Kinrany i dont think so although i havent used v0.1 in a long time (years).  i took a brief pause for december to work on some fun not work related project but im about to get back into dft in the beginning of the new year.  im going to review where things are again and potentially do a release early on before proceeding to the next round of features i have in mind.

authors: Kinrany alamb matthewmturner

repo: datafusion-contrib/datafusion-dft


text:
Grossir le + de détails et mettre en avant le covoiturage

Nouveau retour (ré)utilisateur qu'on a eu avec @Clem-mrtn sur le fait qu'on ne comprend pas que :

les barres sont cliquables
qu'il est possible d'ajouter des covoitureurs

Proposition : grossir le "+" et ajouter un smiley à côté ex " ➕ 👨 "

C'est mieux ?

Plus clair avec le nouveau design oui.
Mais tu pourrais tester avec un bonhomme à côté du + ?

un tooltip au survol ? ex : Ajouter des passagers

Ou à défaut d'avoir une tête à côté, ajouter une animation ? Genre le bouton tourne ou autre ?
(+1 pour le tooltip aussi)

J'ai ajouté un tooltip

Tu as testé aussi avec un petit smiley à côté du + ? :)
Sans forcément l'implémenter tu penses que tu pourrais juste nous faire une "capture d'écran" de ce que ça donnerait ?

J'ai fait un essai. Pas convaincu que ça soit plus lisible

cool, testons ! on peut voir les interactions spécifiques avec ce bouton là ?

On peut voir le nombre d'ouverture du panneau de configuration mais pas via ce bouton spécifiquement. Après vu que c'est le seul changement, on pourra voir la différence

Cool ! Mais c'est marrant moi je l'aurais plutôt fait dans l'autre sens
" + et ensuite la tête du bonhomme"

authors: Clem-mrtn florianpanchout martinregner

repo: datagir/monimpacttransport


text:
Route Pass Dump - Allow concurrent querying, writing to socket

Wrap the logic that writes out transaction items in a Promise
Dispatch that and move on to the next set of items to query
await on the previous batch to fully write out before starting to write again

Last commit may possibly break frontend / admin UI. Test first to be sure

Tested against admin - already made the requisite changes in datagovsg/beeline-admin@b2ff3ad (see the conversation in this PR) to expect this.
Will test against frontend in due course

authors: LoneRifle xkjyeah

repo: datagovsg/beeline-server


text:
[docs] pushing single CSV file

We want to have documentation about pushing data to datahub. There is already existing document in the getting started section - http://datahub.io/docs/getting-started/pushing-data
Acceptance criteria

[x] Nice and clear article about how to push data (single CSV file)

Tasks

[x] Plan
[x] Write out the content
[x] Put it online under /docs/getting-started/pushing-data

FIXED. See http://datahub.io/docs/getting-started/pushing-data

repo: datahq/frontend

license: mit

license_source: bigquery


text:
MySQLdb.OperationalError

로컬에서 ENTROPY 실행 중 '연계' > in/outbound 선택 후 dataset의 이름과 설명을 가져온 후 발생한 에러 입니다. 테스트 데이터셋은 iris로 시도했습니다.
Error after importing the name and description of the dataset after selecting 'linked' > in/outbound while running ENTROPY locally. Test dataset was attempted with iris.  I tried with iris dataset during the test.

console에 출력된 에러 로그는 다음과 같습니다.
The error log output to the console is as follows.
(MySQLdb.OperationalError) (1054, "Unknown column 'sepal.length' in 'field list'")
[SQL: SELECT sepal.length,sepal.width,petal.length,petal.width,variety FROM iris]
(Background on this error at: https://sqlalche.me/e/20/e3q8)

해당 이슈는 58257b0b8bc414f6631ec74733ac1bed37f3d4eb 커밋으로 해결되었습니다.

repo: dataignitelab/athemic


text:
Timezone default as UTC may not always be correct

Syslog supports timestamp formats that don't specify a timezone. Right now we assume these are in UTC, but they're more likely to actually be in the timezone of the host that generated them. There's no perfect solution here, but we could allow you to configure the input to use the host timezone instead of UTC.
A workaround is also to use server timestamps on the API key that forwards the events into Seq.

We're not planning to move forward with this in the near future, since "Use server timestamps" is a reasonable workaround.

authors: KodrAus nblumhardt

repo: datalust/seq-input-syslog


text:
Restrict updating of installation environment variables via API

Added checks to the PUT /environmentVariables endpoint to restrict the modification of installation environment variables.
Extended the JSON configuration to include installation environment variable information. This enriched JSON config is now used to seed environment configuration into the database on startup

@jugnu-appveen

repo: datanimbus/dnio-user-manager


text:
pdf to svg

Any free options that will work to convert the pdfs to svgs?

Hi there. Not that would work with the current process. Affinity Designer offer a free trial.
Google have now released the data as a CSV file, so we will be stopping updates to this work soon.
https://www.gstatic.com/covid19/mobility/Global_Mobility_Report.csv

authors: Connorpar mshodge


text:
[#2059] feat(client-python): Support Gravitino Virtual FileSystem in Python

What changes were proposed in this pull request?
Support Gravitino Virtual File System in Python so that we can read and write Fileset storage data. The first PR only supports HDFS.
After research, the following popular cloud storages or companies have implemented their own FileSystem based on fsspec(https://filesystem-spec.readthedocs.io/en/latest/index.html):

S3(https://github.com/fsspec/s3fs)
Azure(https://github.com/fsspec/adlfs)
Gcs(https://github.com/fsspec/gcsfs)
OSS(https://github.com/fsspec/ossfs)
Databricks(https://github.com/fsspec/filesystem_spec/blob/master/fsspec/implementations/dbfs.py)
Snowflake(https://github.com/snowflakedb/snowflake-ml-python),

So this PR will implement GVFS based on the fsspec interface.
Why are the changes needed?
Fix: #2059
How was this patch tested?
Add some UTs and ITs.

Hi @xloya I just found that fsspec have their wrapper for PyArrow and HadoopFileSystem, maybe we can develop our HDFS solution based on theirs?

Hi @xloya I just found that fsspec have their wrapper for PyArrow and HadoopFileSystem, maybe we can develop our HDFS solution based on theirs?

Yeah, actually current implementation is customized based on the ArrowWrapper provided by fsspec and HadoopFileSystem provided by PyArrow.

Hi @xloya I just found that fsspec have their wrapper for PyArrow and HadoopFileSystem, maybe we can develop our HDFS solution based on theirs?

Yeah, actually current implementation is customized based on the ArrowWrapper provided by fsspec and HadoopFileSystem provided by PyArrow.

Yeah, but why not just use their implementation and add our features (ex. lock, custom handlers) on them?

Hi @xloya I just found that fsspec have their wrapper for PyArrow and HadoopFileSystem, maybe we can develop our HDFS solution based on theirs?

Yeah, actually current implementation is customized based on the ArrowWrapper provided by fsspec and HadoopFileSystem provided by PyArrow.

Yeah, but why not just use their implementation and add our features (ex. lock, custom handlers) on them?

ArrowWrapper some logics will call the self methods, but our GVFS need check the path in the args if valid such as start with gvfs://, this may cause a path exception, so we need customized the code.

Btw, the code is not complete yet, I need to do more tests about the interface compatibility.

Btw, the code is not complete yet, I need to do more tests about the interface compatibility.

Sure, would love to help if needed.

Does it support read/write now? I would like to do a local test to see the current status.

Does it support read/write now? I would like to do a local test to see the current status.

Currently I have tested the following interfaces and they can all be executed normally:
from gravitino import gvfs

if __name__ == "__main__":
    server_uri = "http://localhost:8090"
    metalake_name = "test_metalake"
    fs = gvfs.GravitinoVirtualFileSystem(server_uri=server_uri, metalake_name=metalake_name)
    print(fs.ls(path="gvfs://fileset/fileset_catalog/tmp/tmp_fileset"))
    print(fs.info(path="gvfs://fileset/fileset_catalog/tmp/tmp_fileset"))
    print(fs.exists(path="gvfs://fileset/fileset_catalog/tmp/tmp_fileset"))
    with fs.open(path="gvfs://fileset/fileset_catalog/tmp/tmp_fileset/test3.txt", mode="wb") as output:
        print(output.writable())
        output.write(b"hello world")

    with fs.open(path="gvfs://fileset/fileset_catalog/tmp/tmp_fileset/test3.txt", mode="rb") as input:
        print(input.readable())
        print(input.read(1024))

Currently, Python integration testings lacks the support for some Docker environments, so I am considering not adding HDFS integration testings to the current PR, but supporting it separately in another PR to reduce the difficulty of code review.

@jerryshao @xunliu @shaofengshi @noidname01 This PR is ready for review, could you take a look when you have time? Thanks.

I will finish the review before this weekend, thanks a lot @xloya for your work.

@noidname01 can you please also help to review?

@noidname01 can you please also help to review?

Sure, I will review on this weekend.

We added several pylint rules to enforce the python code style, you need to rebase and update the codes.

I think that most of the raise RuntimeError can be replaced with GravitinoRuntimeException, which might be more clear that this is the error that we expect to see.

@jerryshao @noidname01 Have addressed the comments, could you take a look again? Thanks.

LGTM, @jerryshao WDYT?

Sorry for late response, I need to take another round of review, will finish this before Tuesday.

authors: jerryshao noidname01 xloya

repo: datastrato/gravitino


text:
How to use lineupjs by es6 import?

Release number or git hash: 3.0.1-beta.13
Web browser version and OS: macos 10.13.3 chrome 67

Steps to reproduce

npm install --save lineupjs@next

import LineUpJS from 'lineupjs';
import 'lineupjs/build/LineUpJS.css'

const arr = []
for (let i = 0; i < 100; i++) {
  arr.push({
    attr1: Math.random() * 10,
    attr2: Math.random() * 10,
    attr3: Math.random() * 10,
  })
}
LineUpJS(document.body, arr)

Observed behavior
Chrome dev tool consoles errors:

Expected behavior

the default export is the LineUp class not a factory function. please use
LineUpJS.asLineUp(document.body, arr)

Thank you very much!!

authors: crcrcry sgratzl

repo: datavisyn/lineupjs


text:
Can't auth

After updating to 2.3 latest git I can no longer auth,i've verified dozens of times and across countless logouts/logins however the scraper sits at "Attempting 1/10" until it reaches 10 then cleans .part files and exits. I ran with --output debug but nothing was shown.
2.1 was working with the same auth file sometime last week.
2FA is not being used.

Latest commit and a new login is working now

repo: datawhores/OF-Scraper


text:
Diagnostic web page broken in 0.50.0-ea7.

Describe the bug
Diagnostic web page is broken in 0.50.0-ea7
To Reproduce
Steps to reproduce the behavior:

kubectl port-forward <ambassador-pod> 8877
Navigate to http://localhost:8877/ambassador/v0/diag/
Browser displays 'server error'

Expected behavior
Browser should show the diagnostic page
Versions (please complete the following information):

Ambassador: 0.50.0-ea7
Kubernetes environment: Docker Desktop Community 2.0.0.0-mac81 (29211)
Version: 1.10.3

Additional context
The following is shown in logs:
2018-12-07 16:31:23 diagd 0.50.0 [P50TThreadPoolExecutor-0_3] INFO: E32627D4-8486-4091-A419-A103CCF431B4: 127.0.0.1 "GET /ambassador/v0/diag/" START
2018-12-07 16:31:23 diagd 0.50.0 [P50TThreadPoolExecutor-0_3] ERROR: 'kind'
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ambassador-0.0.0.dev0-py3.6.egg/ambassador_diag/diagd.py", line 98, in wrapper
    result = f(*args, reqid=reqid, **kwds)
  File "/usr/lib/python3.6/site-packages/ambassador-0.0.0.dev0-py3.6.egg/ambassador_diag/diagd.py", line 299, in show_overview
    aconf = get_aconf(app, "overview")
  File "/usr/lib/python3.6/site-packages/ambassador-0.0.0.dev0-py3.6.egg/ambassador_diag/diagd.py", line 173, in get_aconf
    resources = fetch_resources(latest, app.logger, k8s=app.k8s)
  File "/usr/lib/python3.6/site-packages/ambassador-0.0.0.dev0-py3.6.egg/ambassador/config/acresource.py", line 258, in fetch_resources
    fetcher = ResourceFetcher(config_dir_path, logger, k8s=k8s)
  File "/usr/lib/python3.6/site-packages/ambassador-0.0.0.dev0-py3.6.egg/ambassador/config/acresource.py", line 155, in __init__
    self.load_yaml(serialization, k8s=k8s)
  File "/usr/lib/python3.6/site-packages/ambassador-0.0.0.dev0-py3.6.egg/ambassador/config/acresource.py", line 171, in load_yaml
    self.ocount = self.process_object(obj, rkey)
  File "/usr/lib/python3.6/site-packages/ambassador-0.0.0.dev0-py3.6.egg/ambassador/config/acresource.py", line 220, in process_object
    if obj['kind'] == 'Pragma':
KeyError: 'kind'
2018-12-07 16:31:23 diagd 0.50.0 [P50TThreadPoolExecutor-0_3] ERROR: E32627D4-8486-4091-A419-A103CCF431B4: 127.0.0.1 "GET /ambassador/v0/diag/" 30ms 500 server error

This is known and will be fixed in GA.

Fixed in -rc1.

authors: dgaffuri kflynn nbkrause

repo: datawire/ambassador


text:
Make config dirs tunable

What

Make config dirs for both ambassador and envoy tunable via custom ENV CUSTOM_CONFIG_BASE_DIR. Defaults to current dir ($AMBASSADOR_ROOT) if not set

Also fixed a bug how the PYTHON_EGG_CACHE is set, it was not overwritable and was writing to root of the app while  the original intent was $APPDIR}/.cache

Why
This allows folks to eg. use tmpfs mounts on k8s when writing to read is not permitted.
Open
I didn't quite grasp why both envoy.json and envoy*.json is used, could be cleaned up if latter could be dropped.

Taking this one for the next 0.40.


text:
Update Graph Info Every 5min

Every 5 minutes, the following code should be executed:
loadedStuff = false;
m.redraw()

Probably in a timeout instead of an interval, though if it is an interval, it should be outside of the class' code.

Also, supposedly init() shouldn't be called again when doing a m.redraw(), so maybe I should create something that will work in all cases... :/

repo: datitisev/flarum-ext-admindashboard


text:
How to run this?

I want to run this for World of Tanks Blitz, but I don't know how to run this.

@NTanker123 what kind of issues are you experiencing? I don't have any troubles running this for WoT Blitz

Well, I don't know how to run DAVAFramework.
There is no .exe to run the program.
I want to learn this engine for program and coding learning. (Sorry my bad English)

@NTanker123 You have to compile everything from source. You will be need CMake and Qt to generate and compile projects. I will ask somebody to make detailed building instructions.

Thank you very much!

Heh. Even if you build the engine - it is almost useless without a lot of documentation.

@sergeyreznik it's 3/25 now and still no building instruction..

https://github.com/dava/dava.engine/blob/development/Docs/ResourceEditorPublic.md
You should check "Docs" directory, which usually contains documentation.

We uploaded instruction mentioned by @subdiox

authors: AlekseiKanash NTanker123 klesch sergeyreznik subdiox


text:
Possible issue with large evaluation?

Hi Dave.
During some quick testing on the Swift3 branch noticed that if I do evaluation on an extremely long series of calculations (say simple +-*/ stuff on randomly generated integers where the number of operators might be 1000 or more) I'm seeing crashes in DDMathParser.
The strange thing is that it only appears to happen when running the evaluation on a background thread (either via OperationQueue or DispatchQueue).
Is this expected? Am I using it wrong? Obviously if I break down the long series into an array of short calculations, and handle each item separately, it works fine but seems like I shouldn't need to do this.

I just reproduced this, and my gut feeling is that this is an issue related to blowing out the execution stack. When I saw this, I was nearly 400 frames deep in execution, and I'm wondering if at that point we exceed the amount of space that Swift runtime allocates for the execution stack.

Ah. Interesting. Is this Radar material?

I just had the thought that you could do this by doing a DFS traversal of the expression tree without recursing. The current implementation of evaluation is essentially recursing through the entire tree, and on a huge tree, that blows the stack.

authors: davedelong joshdistler

repo: davedelong/DDMathParser


text:
Hi

i have tested with 24000*1920 image it works fine with jpg but when i try to add  the same level of png at the top of jpg it crashes with oom kindly help

thanks for your beautiful library i have fixed the issue !!!!


text:
add UVs for texture mapping?

Awesome to see a CSG library being made! Quite a nice method of 'p5'ifying the shapes/inputs/dimensions – which in previous discussion with @micaut was torn between going the CSG way or P5 way. Just tested model() in it = sweeet it worked!
Just a small issue to see if you thought of adding the UVs for wrapping CSG output models with texture? At the moment, adding texture() becomes a solid color over the shape – would be really nice to wrap it similar to what sphere/box/torus do. here's the rough progress we made.

I actually think that something like that would be a useful feature in p5.Geometry and not just in this addon! But this could be a good place to try it out first, maybe adding a p5.Geometry.prototype.generateUVs()?

Most definitely – we briefly discussed this on another issue thread in p5.js regarding the p5.Geometry and new p5.buildGeometry(). Unfortunately this whole UVs topic is far beyond my scope and skills, so it's merely a 'wishlist' that my students would greatly benefit from (and have wanted to use in the past).


text:
error： potentially unavailable with '@available'

.../node_modules/react-native-native-mqtt/ios/Libraries/CocoaMQTT/CocoaMQTTTypes.swift:66:6: Enum cases with associated values cannot be marked potentially unavailable with '@available'

Did you solve it? I upgraded to Xcode 13 and also encountered this problem

I am guessing this repo is no longer supported.

Same issue.

@askie @15638228357
try npm i https://github.com/zivost/rn-native-mqtt.git
requires minimum deployment to be 13 and react-native version 0.61

getting same error, how to fix?

try this
https://github.com/emqx/CocoaMQTT/pull/391/files/520d3c30f3792f2cd39a6b95712043adca65abde

repo: davesters/rn-native-mqtt


text:
Question: why is server_host not localhost by default?

Hello,
Why is this line setup to use ip.address() when the comment next to it seems to suggest a security risk?
server_host :  ip.address(), // use string 'localhost' to prevent exposure on local network
Also the only way I can get hot reload to work is when it is set to 'localhost' - I've seen that mentioned in an old bug, but I'm using the latest and still need to set it to 'localhost'.
It seems strange that this setting needs to be changed to be able to get hot reloading to works etc, why isn't it 'localhost' by default with the option for explicit ip.address().

That comment does not suggest a security risk, simply that it is accessible via the local network. The only reason it uses your IP address rather than localhost is so that you can access the server from other devices on your network, such as tablets and phones, for testing. It used to be localhost, but was changed quite a while ago after many user requests.
I don't know why hot reloading doesn't work for you with the ip address; this is the first I'm hearing of any issue. If you think that the usage of an explicit ip address is itself problematic, and not specific to your machine, I'd be more than willing to entertain a PR. Hope that helps, thanks.

FYI - On OSX I need to use 0.0.0.0 to use within a docker container. I'm still in my development environment so no security issues.  Everything works as expected

Hi Dave,
RE: "this is the first I'm hearing of any issue"
There are several issues here when you search hot reload in the Issues for example: https://github.com/davezuko/react-redux-starter-kit/issues/955 where a lot of people are saying the same thing that setting localhost works by not with the I address.
Some of them sound related to other services they are running etc but I am not running anyother service or firewall, osMac Sierra latest node.
I don't get reloads with any file when I edit a component, reducer, css file I always have to manually refresh browser.
I can only get it usable by adding a hack:
// const ip = require('ip')
const ip = { address: () => 'localhost' }

authors: davezuko dbroadhurst rhysburnie

repo: davezuko/react-redux-starter-kit


text:
Can't get fonts working

I'm sorry to ask this question again, this seems to have been answered a few times, but I can't get fonts working.
I think I followed everything mentioned in other issues, but the font seems to be loaded from : http://localhost:3000/fonts/MY_FONT.ttf (the request appears as "Canceled" in Chrome dev tools)
The file is in : src/styles/fonts/MY_FONT.ttf and is referenced like this :
_base.scss
$font-directory: './fonts';

main.scss
@font-face {
  font-family: "Scoreboard";
  font-weight: bold;
  src: url('#{$font-directory}/SCOREBOARD.ttf');
}

Webpack doesn't build any font at all. Is there a wiki page or a FAQ somewhere telling how to deal with fonts ? Sorry to ask, but I've been struggling and digging through the issues for 2 hours now.

To unblock you in the meantime, you should be able to make this work by moving that font file to ./public/fonts/SCOREBOARD.ttf and referencing it via fonts/SCOREBOARD.ttf.
I believe what you have should work. I will try to reproduce today and get back to you.

From a proof of concept level I got this working as such:

Add missing file-loader dependency (fixed in https://github.com/davezuko/react-redux-starter-kit/commit/178fed5fc05a06c60606eb63911a34340033f289)
Place font in ./src/styles/fonts
Reference font as follows:

@font-face {
  font-family: 'Barrio';
  font-style: normal;
  font-weight: 400;
  src: url('./fonts/barrio.woff2');
  unicode-range: U+0100-024F, U+1E00-1EFF, U+20A0-20AB, U+20AD-20CF, U+2C60-2C7F, U+A720-A7FF;
}

body {
  font-family: 'Barrio', cursive;
}

I'm going to try this with a sass variable next, but if you're having issues I would try dropping the $font-directory first.

Thanks for that really really quick answer. Much appreciated !
This doesn't work for me unfortunately. I already added file-loader manually before since without it the project wasn't starting at all.
But still, the font is loaded through GET http://192.168.0.20:3000/fonts/SCOREBOARD.ttf, and the request gets canceled because of some CORS issue.

Edit: That change is for line 16 in ./project.config.js

Ohhh, sorry. This was the issue all along.
I don't know why I was sure the font shouldn't have been served that way. Thus didn't think the CORS was important anyway.
Thank's a lot for your help !

Glad that fixed it, sorry for all the trouble.

authors: aesteve davezuko


text:
Support Live Reload routing natively on Magento 2 environment

Magento 2 can be rather simply configured to serve the Live Reload script by including the following bit in the env.php file and running bin/magento app:config:import:
    'system' => [
        'default' => [
            'design' => [
                'footer' => [
                    'absolute_footer' => '
                        <script id="__lr_script__">//<![CDATA[
                            document.write("<script src=\'/livereload.js?port=443\'/>");
                        //]]></script>
                    '
                ]
            ],
        ],
    ],

Routing requests to this has required using per-project configuration in .warden/warden-env.yml to label the php-fpm container as required to serve the Live Reload JS file and process Web Socket requests when grunt watch is running.
In the upcoming Warden 0.2.0 release Live Reload should require nothing more than adding the script tag to the env.php file or including it via design configuration in the admin.
Example documentation on how to leverage Live Reload:
This project uses the core bundled Grunt workflow. A Gruntfile and package.json are already included. We are ascribing to server-side compilation, in conjunction with the Grunt watch task, so that changes can be applied more rapidly during development.
To get started, run:
warden shell
grunt clean
grunt exec:<theme_name>
grunt less:<theme_name>

Thereafter, a single command should be needed for daily development:
grunt watch

Note: Grunt should be used within the php-fpm container only.
LiveReload is also used to persist changes to your compiled CSS. When you run the watch task, a LiveReload server will be started on port 35729 within the php-fpm container and Traefik will take care of proxying the JavaScript tag and WebSocket requests to this listener. This project's env.php used for local development injects the LiveReload JavaScript tag necessary for WebSocket communication via app:config:import.

I encountered a JS issue today that turned out to be the result of the script tag being added for livereload.js. The browser wasn't properly parsing the self-closing tag, so it saw the following script tag's content as being inside of the livereload.js tag.
It was fixed by replacing the self-closing tag with a proper closing tag:
document.write("<script src='/livereload.js?port=443'>");

@zaximus84 Thanks for the note. That is likely why @erikhansen encountered an issue and suggested a change which is now in the template and docs page on the subject:

https://docs.warden.dev/configuration/livereload.html
https://github.com/davidalger/warden-env-magento2/blob/develop/webroot/app/etc/env.php.init.php#L23-L25

It would be good to reference those if encountering issues in the future. I'll update the above description to correct the bad JS bit as well. Cheers!

repo: davidalger/warden


text:
Ctrl-E doesn't work for me

I really like the idea of this!
However I'm having trouble getting nbterm to execute the contents of cells. I can press enter, enter something in the cell, like print(100) then press ESC to go back to command mode.
But when I then push Ctrl-E or Ctrl-R, nothing happens. No error message, nothing. Other key bindings (such as Ctrl-Q to quit) do work. Am I overlooking something?

Debian testing
Tilix terminal (based on vte like gnome-terminal). I've checked and Ctrl-E does send '\x05' correctly.
Python 3.9.2
nbterm 0.0.7 installed in a venv, using pip3 install nbterm

Do you have a (Python) kernel installed?
$ pip install ipykernel

That solved it, thank you!

Do you have a (Python) kernel installed?
$ pip install ipykernel

Wouldn't this be a requirement then?
It also solved the problem for me , BTW.

But ipykernel is one kernel implementation for Python, why require this one more than another?

But ipykernel is one kernel implementation for Python, why require this one more than another?

You're right. Maybe a mention of nbterm not coming with any kernel is better.

Will do, thanks!

Yes, I think adding a message when trying to execute a that there is no kernel available would be helpful.

Definitely, trying to run without a kernel should raise an error

I think we should have a status bar for that, see #5.

authors: JulesGM davidbrochart nicoguaro vmedea

repo: davidbrochart/nbterm


text:
Setup in Linux using Anaconda Enviroment

Im trying to prepare the pclpy in order to work in a anaconda enviroment on Linux but I have problem with CppHeaderParser in the ~/pcl/octree/octree_iterator.h file.
This is the error:
CppHeaderParser.CppHeaderParser.CppParseError: Not able to parse  on line 78 evaluating "{"
Error around: class OctreeIteratorBase : public std : : iterator <std : : forward_iterator_tag , const OctreeNode , void , const OctreeNode * , const OctreeNode & >
I don't understand what is the problem, does it happen in Windows?  Or does someone have a similar problem?
Thank you.

Hi,
Linux is not supported at the moment...
But the error you're getting would probably be fixed if you used github.com/davidcaron/cppheaderparser
I changed some things in cppheaderparser and this repo is not on github so I didn't submit a pr...

Le 20 mars 2019 à 11:39, Abián Hernández notifications@github.com a écrit :
Im trying to prepare the pclpy in order to work in a anaconda enviroment on Linux but I have problem with CppHeaderParser in the ~/pcl/octree/octree_iterator.h file.
This is the error:
CppHeaderParser.CppHeaderParser.CppParseError: Not able to parse on line 78 evaluating "{"
Error around: class OctreeIteratorBase : public std : : iterator <std : : forward_iterator_tag , const OctreeNode , void , const OctreeNode * , const OctreeNode & >
I don't understand what is the problem, does it happen in Windows? Or does someone have a similar problem?
Thank you.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

Thank you, your CppHeaderParser works great. For your interest, in PCL 1.9 the EuclideanClusterComparator file (euclidian_cluster_comparator.h) contains two class with the same name, in different namespaces, and the parser crashes but in 1.8 everything works great .
Anyway, I know there is no support for Linux, Im trying to contribute because I think your project is interesting and python-pcl doesn't work very well. My idea is to generate an anaconda repository.
I'll leave this issue open until I get to finish.
Thank you

If you want to contribute that's great!
I just pushed some commits that I did a while ago in this branch: https://github.com/davidcaron/pclpy/tree/linux-compilation
I suggest you start by checking this. It's not done, but I already spent some time on it, so you can check it out.

@SolidusAbi @davidcaron Is there any progress on this? I'd really like to have pclpy support on Linux.

@geirbi I'm sorry but I haven't had time to continue, I'm also very interested but I think it's going to take quite some time.

authors: SolidusAbi davidcaron geirbi

repo: davidcaron/pclpy


text:
How to add react components to main react-component (breaking the UI into component hierarchy)?

Thanks for the work you've done here. Really speeded up my list rendering.
My main list component is getting very large, and I would like to break it out into other components. However I run into a number of errors when trying to do so. My code looks something like this (very much simplified):
var header = React.createClass({
	render: function () {
		return (

header row

		);
	}
});

var table = React.createClass({
	render: function () {
		return (

		);
	}
});

Errors:
Warning: render(): Target node has markup rendered by React, but there are unrelated nodes as well. This is most commonly caused by white-space inserted around server-rendered markup.

Warning: React attempted to reuse markup in a container but the checksum was invalid. This generally means that you are using server rendering and the markup generated on the server was not what the client was expecting. React injected new markup to compensate which works but you have lost many of the benefits of server rendering. Instead, figure out why the markup being generated is different on the client or server:
 (client) <table class="c-smart-
 (server) <header data-reactid=".1

Could this be a limitation of ngReact? I hope not!

This is not a limitation of ngReact. One should just remember to write the component name with a capital letter e.g.
sigh

repo: davidchang/ngReact


text:
Revert "Topic/watcher init"

Reverts davidchang/ngReact#57, see #58
cc: @ethul

Thanks

authors: ethul kasperp


text:
[RTM] Sticky navigations and thresholds

[x] Fixovat sidenav a navbar, aby se pohyboval jenom obsah stránky?
[x] Sidenav - změnit práh
[x] Opravit, aby se neshovavalo horni menu pri mensim zobrazeni

LGTM, thank you. :+1:  Works cool.

authors: davidchocholaty vaceksimon

repo: davidchocholaty/ucebni-karticky


text:
Unable to add HostNames or SSL binding to sites

I'm executing this sample, but unable to get host names added to a site and SSL bindings are impossible to add. Function BindCertificateToSite is getting executed, but no change to the site happens.
And I can manually add the host binding and SSL cert to that site, but unable to do it programmatically.
Any ideas?

@x10shun this is the part of the sample you added. Do you know what might be going on?

This was a bug in the sample. Please add set property HostNameSslState.ToUpdate = true, before calling the CreateOrUpdateSite in BindCertificateToSite. I am fixing the sample to reflect that.

Sorry, but that still does not work. If I add domain binding manually and add ToUpdate=true then yes. But in the current state I get error:

Response: 404 Not Found
Hostname 'blah.com' does not exist

I've tried adding just domain name binding without SSL and that does not happen - which explains the error. Any other ideas?
I've found websiteClient.Sites.CreateOrUpdateSiteHostNameBindingAsync - is it not supposed to add domain name binding? but I could not find any documentation on how to use it.

I'm sorry again, I stand corrected. SSL binding does not work even if I manually add the domain.
When I get site object back from websiteClient.Sites.CreateOrUpdateSite(), I inspect HostNameSslStates object and I see 3 objects there - 2 for mysite.azurewebsites.net and mysite.scm.azurewebsites.net and third for the one I've just added, but in all 3 (including the new one) I see SslState: Disabled.
In debugging this looks like this:
{Microsoft.Azure.Management.WebSites.Models.HostNameSslState}
    Name: "subdomain.mydomaincom"
    SslState: Disabled
    Thumbprint: null
    ToUpdate: null
    VirtualIP: null

I suspect this is the same issue I'm seen using the older management library: https://github.com/Azure/azure-sdk-for-net/issues/1461
Any other ideas? Is that server-side issue, not the library/sample?

I have managed to get it working. Host names are to be added by CreateOrUpdateSiteHostNameBindingAsync() rather than site.HostNames:
        var binding = new HostNameBinding()
        {
            Location = "North Europe",
            Name = hostName,
            CustomHostNameDnsRecordType = CustomHostNameDnsRecordType.CName,
            HostNameType = HostNameType.Verified,
            SiteName = siteName,
            Type = "Microsoft.Web/sites/hostNameBindings",
        };
        var result = await websiteClient.Sites.CreateOrUpdateSiteHostNameBindingAsync(resourceGroupName, siteName, hostName, binding);
        site = websiteClient.Sites.GetSite(resourceGroupName, siteName);

        site.HostNameSslStates.Add(new HostNameSslState
        {
            Name = hostName,
            Thumbprint = certificate.Thumbprint,
            SslState = SslState.SniEnabled,
            ToUpdate = true,
        });

        site = await websiteClient.Sites.CreateOrUpdateSiteAsync(resourceGroupName, siteName, site);

Though I'm not sure about all the parameters in HostNameBinding object are required. Can you please verify what properties are required there? (I took the values from Resource Explorer)

authors: davidebbo naveedaz trailmax

repo: davidebbo/AzureWebsitesSamples


text:
Upload (sdist) tarball to pypi with prebuilt version information

I am interested in packaging this project for other systems, so am wondering if it is possible to have a publicly available source distribution (tarball) with pre-completed version information?
If I download the automatically-generated tarball from github release archives, the version information is broken because it tried to generate the VERSION file on-the-fly without git information to rely upon.

The version in source control now does this a different way. Please verify that the issue still exists (not release for this version yet).  Reopen if it still exists.

authors: Alan-R duncanmmacleod

repo: davidjamesca/ctypesgen


text:
Is the service down or is it just me?

[!] Can't connect to service! restart tor service and try again.
I have tried on a number of different machines with same result.

It was just me, nvm.

how you fixed it?

I wasnt running my tor service

authors: bradgcoza daniladyabin

repo: davidtavarez/pwndb


text:
SoldItemsOnly not working

        $soldItemFilter = new ItemFilter();
        $soldItemFilter->name = 'SoldItemsOnly';
        $soldItemFilter->value = ['true'];
        $request->itemFilter[] = $soldItemFilter;

        $response = $service->findItemsAdvanced($request);

This always returns zero results.
I've also tried various others, '1' does not work. '0' returns unsold items as expected.
Is there something i'm doing wrong, or is this an issue in the code?

After lots of playing around, I've got this to work. For those having the same problem, you must use the FindCompletedItemsRequest for this to work.

repo: davidtsadler/ebay-sdk-php


text:
Add methods for object conversion

The class needs to be able to handle more than one conversion direction. It should probably implement the toPHP method from the parent class.

Should extend and implement the methods from here, https://github.com/cakephp/cakephp/blob/master/src/Database/Type/DateTimeType.php

repo: davidyell/TimezoneAwareDateTime


text:
How to config shape_predictor_68_face_landmarks.dat to use GPU

Hi team, may I ask is it possible to config my environment to let shape_predictor_68_face_landmarks.dat only use GPU? Or is there a way to check whether this model supports GPU?

It only runs on the CPU.

repo: davisking/dlib-models


text:
Is the following method can solve the limit of memory?

Hi, I want to use large images set to retrain face landmark model , but I have not enough ram.  I think if I can use the method of training CNN to train face landmark model.
In particular, I choice a batch of images from large dataset randomly, and then call the function dlib.train_shape_predictor in while loop. Stop the loop until epoch reach a special number.  I doubt if the method workable. I want to listen to  opinion from you. Thank you.

I'm dealing with a similar issue, so let me share what I've learnt from reading the code:

The shape predictor uses a forest of regression trees (paper here), so if you specifically want to use a CNN you'll have to figure it out yourself.
train_shape_predictor trains and returns a new shape_predictor object from scratch, so you can't repeatedly call it for each batch/iteration.
If you can fit all the objects (landmarks and bounding boxes) in memory, then it looks like the image data is only used here: https://github.com/davisking/dlib/blob/master/dlib/image_processing/shape_predictor_trainer.h#L330 - while one could read images as needed in this loop, I need to read more of the code to figure out how memory-bound other parts are when you have a large dataset.

I think that because this is a forest of regression trees we should be able to have watch tree created from different partitions of the data?

During each iteration the trees are built on the entire dataset.  So that
sort of thing isn't going to work with this algorithm.
The deeper issue is that you shouldn't be running out of ram. A computer
with 32GB of ram ought to be plenty for normal sized datasets.  So unless
your images are just super huge, there shouldn't be an issue.  If they are
super huge downsample them first.  There isn't any reason to use crazily
high resolution images as training data with this algorithm.

Thanks for the clarification Davis.
I'm working with a small filtered subset of Open Images (full dataset is 9 million images). While images with human faces are only part of the overall dataset, this is still 300k images downsampled to have a max dimension of 1024px.
My aim is to create a massive and open set of face annotations that isn't unduly restricted like ibug, using a bootstrapping process from a smaller set of human labelled annotations.

Oh, sweet.  That would be pretty awesome.
Still though, just get more RAM.  For instance, if each image were 700x700
pixels and grayscale (the algorithm ignores color, so load images as
grayscale) then you could load over half a million images onto a machine
with 256GB of RAM.  You can just rent a VM on google cloud or AWS with that
to do the training if need be.

Thinks for your pretty answer. As ferrouswheel said, we can not solve the problem with loop method. I solve the problem using the computer with 32G ram.

authors: davisking ferrouswheel lvmeng8

repo: davisking/dlib


text:
illegal instruction (core dumped)

Hello,
i'm facing a strange error and i don't know its reason
i'm using this code:
`import sys
import dlib
from skimage import io

# Take the image file name from the command line
file_name = sys.argv[1]

# Create a HOG face detector using the built-in dlib class
face_detector = dlib.get_frontal_face_detector()

win = dlib.image_window()

# Load the image into an array
image = io.imread(file_name)

# Run the HOG face detector on the image data.
# The result will be the bounding boxes of the faces in our image.
detected_faces = face_detector(image, 1)

print("I found {} faces in the file {}".format(len(detected_faces), file_name))

# Open a window on the desktop showing the image
win.set_image(image)

# Loop through each face we found in the image
for i, face_rect in enumerate(detected_faces):

	# Detected faces are returned as an object with the coordinates
	# of the top, left, right and bottom edges
	print("- Face #{} found at Left: {} Top: {} Right: {} Bottom: {}".format(i, face_rect.left(), face_rect.top(), face_rect.right(), face_rect.bottom()))

	# Draw a box around each face we found
	win.add_overlay(face_rect)

# Wait until the user hits <enter> to close the window
dlib.hit_enter_to_continue()`

and when running it,

"illegal instruction (core dumped)"

this error appears,
so i decided to run the code line by line using Python command, and it appears to show this error in this line:
win = dlib.image_window()
and when i deleted this line, the same error appears in this line:
detected_faces = face_detector(image, 1)
so what's wrong?
i also tried using Docker and followed this link: https://medium.com/@ageitgey/machine-learning-is-fun-part-4-modern-face-recognition-with-deep-learning-c3cffc121d78#.giggisuvt
and stuck at another error:

Can't initialize X11 resources

and i don't know how to fix it

http://dlib.net/faq.html#Whywon'tthePythonbindingscompilework


text:
add serialization support to chip_details

I just needed this today.

Sweet

authors: arrufat davisking


text:
There is a stored xss in module_category（栏目管理）

tested in 4.3.1 version

repo: dayrui/xunruicms


text:
Order by column ASC or DESC is case-sensetive, switch to case-insensetive

Hey,
I just upgraded to the newest version and noticed when I use the dbeaver function to order a table or view column, which contains text, the result is case-sensitive like ABCabc, before it was case-insensitive like AaBbCc. When I use a SQL query instead to Order By the column the result is case-insensitive. I couldn't find any information about this. Is there a setting in preferences to chose what dbeaver is using? Thanks a lot!
System information:

Windows 10
Version 22.0.0.202203060510
Postgres 9.6

Hello @ystenberge
DBeaver uses filters from the client's side by default for already loaded a small amount of data. Or reads from the server-side if data isn't fully loaded. Case-insensitive/case-sensitive behavior may be different in these cases.
You can change it in Preferences -> Editors -> Data Editor -> "Results ordering mode"

Hey @LonwoLonwo,
thank you for your help! I changed the settings to 'Always on server'. That worked.
Cheers!

Thanks for the answer @ystenberge

authors: LonwoLonwo ystenberge

repo: dbeaver/dbeaver


text:
Support TiDB

Is your feature request related to a problem? Please describe.
Add TiDB support.
Describe the solution you'd like

Create TiDB type database
At the database tab, should show sequence, and hide triggers, events, and procedures.

Describe alternatives you've considered
Additional context
Add any other context or screenshots about the feature request here.

Which version will plan to release?

It's not planned to some release as we're waiting for the response on the PR from contributors.

I'll give feedback once I can use the Windows device.

authors: E1izabeth Icemap greenpean


text:
Can't use common table expressions (with .. as)

Description
When i run sql request i saw SQL Error 42P01. This request working in pgAdmin
DBeaver Version
22.1.5
Operating System
WIN 10
Database and driver
PostgreSQL 13.2
PostgreSQL JDBC Driver JDBC 42.2.25
Steps to reproduce
I write this SQL request and run him
with tbl as (
select 1
)
select * from tbl
Additional context
org.jkiss.dbeaver.model.sql.DBSQLException: SQL Error [42P01]: ОШИБКА: отношение "tbl" не существует
Позиция: 15
at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.executeStatement(JDBCStatementImpl.java:133)

at org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.executeStatement(SQLQueryJob.java:577)

at org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.lambda$1(SQLQueryJob.java:486)

at org.jkiss.dbeaver.model.exec.DBExecUtils.tryExecuteRecover(DBExecUtils.java:172)

at org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.executeSingleQuery(SQLQueryJob.java:493)

at org.jkiss.dbeaver.ui.editors.sql.execute.SQLQueryJob.extractData(SQLQueryJob.java:894)

at org.jkiss.dbeaver.ui.editors.sql.SQLEditor$QueryResultsContainer.readData(SQLEditor.java:3644)

at org.jkiss.dbeaver.ui.controls.resultset.ResultSetJobDataRead.lambda$0(ResultSetJobDataRead.java:123)

at org.jkiss.dbeaver.model.exec.DBExecUtils.tryExecuteRecover(DBExecUtils.java:172)

at org.jkiss.dbeaver.ui.controls.resultset.ResultSetJobDataRead.run(ResultSetJobDataRead.java:121)

at org.jkiss.dbeaver.ui.controls.resultset.ResultSetViewer$ResultSetDataPumpJob.run(ResultSetViewer.java:4937)

at org.jkiss.dbeaver.model.runtime.AbstractJob.run(AbstractJob.java:105)

at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)

Caused by: org.postgresql.util.PSQLException: ОШИБКА: отношение "tbl" не существует
Позиция: 15
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2565)

at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2297)

at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:322)

at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481)

at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401)

at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:322)

at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:308)

at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:284)

at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:279)

at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.execute(JDBCStatementImpl.java:329)

at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.lambda$0(JDBCStatementImpl.java:131)

at org.jkiss.dbeaver.utils.SecurityManagerUtils.wrapDriverActions(SecurityManagerUtils.java:94)

at org.jkiss.dbeaver.model.impl.jdbc.exec.JDBCStatementImpl.executeStatement(JDBCStatementImpl.java:131)

... 12 more

Hello @AlexeyAntipov98
I see an empty line in your query. Probably, problem is with this empty line - by default DBeaver recognizes it as delimiter.
So. You can remove an empty line from your query.

Or you can highlight all your query and run it

Or you can disable this setting in Preferences

Thank you. Now it working)


text:
Feature Request: Allow to Save Script of Export Query Data

One of my favorite features in DBeaver is the ability to export query data to a different database.
I can not find, however, a way to save the script to file so that it can be executed at a later time.
That would be a great addition since many users need to migrate large databases from one DBMS to another, so it will allow to prepare all of the scripts beforehand, and then execute them when it's time to make the final switch of the migration.
Thank you for your consideration.

@serge-rider Can you estimate the difficulty in implementing this enhancement/feature?  Thanks.

just got DBeaver Enterprise and imediately had the same usecase where i would have liked to export the result of a rather complex query to a query that includes the result as INSERTS. Would have liked it. But maybe i can for the time being use a different format.

Export to SQL is already available

authors: isapir lsascha uslss


text:
Database navigator becomes blank sometimes while deleting tables

Description

DBeaver Version
Version 23.2.4.202311051706
Operating System
Windows 11
Database and driver
No response
Steps to reproduce
I search for some table and select one or two to delete
confirmation pop up will come to delete
I press yes and all of sudden database navigator page goes black
I feel this occurs when dbeaver left idle for sometime and then i try to delete table without refreshing anything. (not sure)
This happened to me multiple times in previous version also and i'm raising this now
Additional context
No response

Thank you for bug report.


text:
Oracle Create Procedure Throws Invalid SQL Statement

There is a bug with Oracle Database CREATE PROCEDURE.  In DBeaver 5.1.2 the code below throws:

SQL Error [900] [42000]: ORA-00900: invalid SQL statement
ORA-00900: invalid SQL statement

But it executes without a problem on the Oracle SqlDeveloper Client.
CREATE PROCEDURE  contens_reset_sequence (
	seqname 	IN NVARCHAR2,
	tblname 	IN NVARCHAR2,
	tblfield 	IN NVARCHAR2
)
IS
	maxtable 	INTEGER;
	sqlstring	VARCHAR2(100);
BEGIN
	-- get max value from table
	sqlstring := 'SELECT MAX(' || tblfield || ') + 1 FROM ' || tblname;
	EXECUTE IMMEDIATE sqlstring INTO maxtable;

	IF (maxtable IS NULL) THEN
		maxtable := 1;
	END IF;

	sqlstring := 'ALTER SEQUENCE ' || seqname || ' RESTART START WITH ' || maxtable;
	EXECUTE IMMEDIATE sqlstring;
END;

Another example :
CREATE OR REPLACE PROCEDURE EnableAllConstraint_sp AUTHID CURRENT_USER AS
/*
||Purpose: Enables all foreign Keys.
||
||Author: Regis Biassala
||Datalex plc (c) 2002.
*/
    TYPE Constraints_CUR IS REF CURSOR;
    v_cons_CUR Constraints_CUR;
    l_Constraint_name user_constraints.constraint_name%TYPE;
    l_table_name user_constraints.table_name%TYPE;
    v_code  NUMBER;
    v_errm  VARCHAR2(64);

BEGIN
    OPEN v_cons_CUR FOR
        'SELECT constraint_name, table_name FROM user_constraints
                WHERE constraint_type = ''R'' ORDER BY constraint_type';
        LOOP
            FETCH v_cons_CUR INTO l_Constraint_name, l_table_name;
            EXIT WHEN v_cons_CUR%NOTFOUND;

            DBMS_OUTPUT.PUT_LINE('Enabling constraint '||l_Constraint_name);
            EXECUTE IMMEDIATE
                'ALTER TABLE'||' '||l_table_name||' '||'ENABLE CONSTRAINT'||' '||l_Constraint_name;

        END LOOP;
    CLOSE v_cons_CUR;
EXCEPTION
    WHEN OTHERS THEN
        v_code := SQLCODE;
        v_errm := SUBSTR(SQLERRM, 1, 64);
        DBMS_OUTPUT.PUT_LINE ('Error enabling constraint -> ' || 'Error code ' || v_code || ': ' || v_errm);
END EnableAllConstraint_sp;
/

already fixed.
ticket closed

authors: Zardoz89 isapir uslss