text:
creating a mock class

Analyzer Feedback from IntelliJ
Version information

IDEA AI-191.8026.42.35.5977832
2.5.0-edge.1ef83b86ae637ffe7359173804cbc6d3fa25e6db
AI-191.8026.42.35.5977832, JRE 1.8.0_202-release-1483-b49-5587405x64 JetBrains s.r.o, OS Mac OS X(x86_64) v10.15.1, screens 2560x1440, 1680x1050  Retina

Exception
Dart analysis server, SDK version 2.5.0-edge.1ef83b86ae637ffe7359173804cbc6d3fa25e6db, server version 1.27.2, error: Analysis failed: /Users/steven/Downloads/AtmosBlocV2/lib/screens/login_screen.dart context: exception_20191119_162349_078
Exception: Element not found in summary: package:flutter_template/server_side/api_helper.dart;package:flutter_template/server_side/api_helper.dart;ApiResponse
#0      SummaryResynthesizer.getElement (package:analyzer/src/summary/resynthesize.dart:338:9)
#1      ElementHandle.actualElement (package:analyzer/src/dart/element/handle.dart:359:42)
#2      ClassElementHandle.actualElement (package:analyzer/src/dart/element/handle.dart:34:43)
#3      ClassElementHandle.type (package:analyzer/src/dart/element/handle.dart:116:29)
#4      _ReferenceInfo._buildType.<anonymous closure> (package:analyzer/src/summary/resynthesize.dart:1043:48)
#5      InterfaceTypeImpl.typeArguments (package:analyzer/src/dart/element/type.dart:1600:26)
#6      InterfaceTypeImpl.substitute2 (package:analyzer/src/dart/element/type.dart:2264:34)
#7      _FunctionTypeImplStrict.substitute2.transformType (package:analyzer/src/dart/element/type.dart:3690:13)
#8      _FunctionTypeImplStrict.substitute2.transformParameter (package:analyzer/src/dart/element/type.dart:3699:34)
#9      _transformOrShare (package:analyzer/src/dart/element/type.dart:29:32)
#10     _FunctionTypeImplStrict.substitute2 (package:analyzer/src/dart/element/type.dart:3709:25)
#11     _FunctionTypeImplStrict.instantiate.transformParameter (package:analyzer/src/dart/element/type.dart:3643:26)
#12     _transformOrShare (package:analyzer/src/dart/element/type.dart:29:32)
#13     _FunctionTypeImplStrict.instantiate (package:analyzer/src/dart/element/type.dart:3656:9)
#14     MethodInvocationResolver._instantiateFunctionType (package:analyzer/src/dart/resolver/method_invocation_resolver.dart:203:27)
#15     MethodInvocationResolver._setResolution (package:analyzer/src/dart/resolver/method_invocation_resolver.dart:763:30)
#16     MethodInvocationResolver._resolveReceiverInterfaceType (package:analyzer/src/dart/resolver/method_invocation_resolver.dart:482:14)
#17     MethodInvocationResolver.resolve (package:analyzer/src/dart/resolver/method_invocation_resolver.dart:127:7)
#18     ElementResolver.visitMethodInvocation (package:analyzer/src/generated/element_resolver.dart:549:31)
#19     MethodInvocationImpl.accept (package:analyzer/src/dart/ast/ast.dart:7233:49)
#20     ResolverVisitor.visitMethodInvocation (package:analyzer/src/generated/resolver.dart:4219:10)
#21     MethodInvocationImpl.accept (package:analyzer/src/dart/ast/ast.dart:7233:49)
#22     ExpressionStatementImpl.visitChildren (package:analyzer/src/dart/ast/ast.dart:3770:18)
#23     ResolverVisitor.visitNode (package:analyzer/src/generated/resolver.dart:4258:10)
#24     UnifyingAstVisitor.visitExpressionStatement (package:analyzer/dart/ast/visitor.dart:3214:59)
#25     ExpressionStatementImpl.accept (package:analyzer/src/dart/ast/ast.dart:3766:49)
#26     NodeListImpl.accept (package:analyzer/src/dart/ast/ast.dart:7670:20)
#27     BlockImpl.visitChildren (package:analyzer/src/dart/ast/ast.dart:1115:17)
#28     ResolverVisitor.visitNode (package:analyzer/src/generated/resolver.dart:4258:10)
#29     UnifyingAstVisitor.visitBlock (package:analyzer/dart/ast/visitor.dart:3128:31)
#30     ScopedVisitor.visitBlock (package:analyzer/src/generated/resolver.dart:5171:13)
#31     BlockImpl.accept (package:analyzer/src/dart/ast/ast.dart:1111:49)
#32     BlockFunctionBodyImpl.visitChildren (package:analyzer/src/dart/ast/ast.dart:1070:13)
#33     ResolverVisitor.visitNode (package:analyzer/src/generated/resolver.dart:4258:10)
#34     UnifyingAstVisitor.visitBlockFunctionBody (package:analyzer/dart/ast/visitor.dart:3131:55)
#35     ScopedVisitor.visitBlockFunctionBody (package:analyzer/src/generated/resolver.dart:5182:13)
#36     ResolverVisitor.visitBlockFunctionBody (package:analyzer/src/generated/resolver.dart:3508:13)
#37     BlockFunctionBodyImpl.accept (package:analyzer/src/dart/ast/ast.dart:1066:49)
#38     FunctionExpressionImpl.visitChildren (package:analyzer/src/dart/ast/ast.dart:5085:12)
#39     ResolverVisitor.visitNode (package:analyzer/src/generated/resolver.dart:4258:10)
#40     UnifyingAstVisitor.visitFunctionExpression (package:analyzer/dart/ast/visitor.dart:3263:57)
#41     ScopedVisitor.visitFunctionExpression (package:analyzer/src/generated/resolver.dart:5533:15)
#42     ResolverVisitor.visitFunctionExpression (package:analyzer/src/generated/resolver.dart:4021:13)
#43     FunctionExpressionImpl.accept (package:analyzer/src/dart/ast/ast.dart:5079:49)
#44     NamedExpressionImpl.visitChildren (package:analyzer/src/dart/ast/ast.dart:7429:18)
#45     ResolverVisitor.visitNode (package:analyzer/src/generated/resolver.dart:4258:10)
#46     UnifyingAstVisitor.visitNamedExpression (package:analyzer/dart/ast/visitor.dart:3345:51)
#47     ResolverVisitor.visitNamedExpression (package:analyzer/src/generated/resolver.dart:4252:11)
#48     NamedExpressionImpl.accept (package:analyzer/src/dart/ast/ast.dart:...

For additional log information, please append the contents of
file:///private/var/folders/kq/qd9zc3ld16q0kx6l8l666qt40000gn/T/report.txt.

This is an issue likely fixed by a new version of summaries, Summary2. Please upgrade Dart to version 2.6.0, and re-open if this issue occurs again.

authors: GitHubStuff srawlins


text:
Generic type parameter doesn't == to actual type

Dart VM Version: Dart VM version: 2.8.0-dev.0.0 (Wed Dec 11 10:31:10 2019 +0100) on "macos_x64"
OS: MacOS.
Minimum Example:
p<T>() {
    print(T == int);
}

main() {
    p<int>();
}

Currently the output is true if the non-nullable experiment is not enabled, and false if it's enabled. Not sure if this is intentional.

It's not intended.
We haven't implemented the full intended Type object equality yet (it's based on type normalization which isn't implemented everywhere yet), but int being equal to int was never intended to not work.

source: gharchive/issue

authors: hongyiweiwu lrhn

repo: dart-lang/sdk

license_type: permissive

license_source: github-api


text:
Don't report RETURN_OF_INVALID_TYPE when already reported ILLEGAL_ASYNC_RETURN_TYPE

Probably also ILLEGAL_ASYNC_GENERATOR_RETURN_TYPE, or ILLEGAL_SYNC_GENERATOR_RETURN_TYPE - check if we report RETURN_OF_INVALID_TYPE in these cases.

Looks like a duplicate of #38188. I also don't think we report duplicate errors in generators, since they're just not supposed to return anything.

Seems different than https://github.com/dart-lang/sdk/issues/38188, even if in the same are.
The other issue is about saying a wrong type, and this one - about reporting an error at all.

authors: scheglov srawlins


text:
CFE: implement change to default type of error variable in catch clauses in null safe code

This pull request specifies a change to the default type of the error variable in catch clauses in code that has opted in to null safety.  This change is ready for implementation.
Tests for this change are out for review here.
cc @eernstg @lrhn @munificent @natebosch

A CL is in progress: https://dart-review.googlesource.com/c/sdk/+/144840

Note that change is breaking the nnbd sdk:
output: org-dartlang-sdk:///sdk_nnbd/lib/_internal/vm/bin/vmservice_server.dart:62:37: Error: The getter 'message' isn't defined for the class 'Object'.
 - 'Object' is from 'dart:core'.
Try correcting the name to the name of an existing getter, or defining a getter or field named 'message'.
        socket.close(idErrorCode, e.message);
                                    ^^^^^^^
org-dartlang-sdk:///sdk_nnbd/lib/vmservice/vmservice.dart:340:14: Error: A value of type 'Object' can't be assigned to a variable of type 'String'.
 - 'Object' is from 'dart:core'.
      return rpcError;
             ^
org-dartlang-sdk:///sdk_nnbd/lib/vmservice/running_isolate.dart:109:14: Error: A value of type 'Object' can't be assigned to a variable of type 'FutureOr<Response>'.
 - 'Object' is from 'dart:core'.
 - 'FutureOr' is from 'dart:async'.
 - 'Response' is from 'dart:_vmservice'.
      return errorResponse;
             ^
org-dartlang-sdk:///sdk_nnbd/lib/_internal/vm/bin/file_patch.dart:148:70: Error: The argument type 'Object' can't be assigned to the parameter type 'OSError?'.
 - 'Object' is from 'dart:core'.
 - 'OSError' is from 'dart:io'.
            "Failed to initialize file system entity watcher", null, e));
                                                                     ^
org-dartlang-sdk:///sdk_nnbd/lib/_internal/vm/bin/file_patch.dart:159:76: Error: The argument type 'Object' can't be assigned to the parameter type 'OSError?'.
 - 'Object' is from 'dart:core'.
 - 'OSError' is from 'dart:io'.
          .addError(new FileSystemException("Failed to watch path", _path, e));
                                                                           ^
org-dartlang-sdk:///sdk_nnbd/lib/io/network_profiling.dart:76:51: Error: The argument type 'Object' can't be assigned to the parameter type 'String'.
 - 'Object' is from 'dart:core'.
          ServiceExtensionResponse.invalidParams, errorMessage));

I think you can change these cases from } catch( to } on dynamic catch( to get unblocked. I'd file an issue for each one to get cleaned up - they seem like potential bugs to me.

https://dart-review.googlesource.com/c/sdk/+/146521

I landed the above change.

authors: johnniwinther leafpetersen natebosch


text:
Migration tool should account for comments when deciding where to add an import

When the migration tool adds an import statement, it should attempt to avoid breaking apart previous declarations/directives from the comments that precede them.  For example, given the following code:
library foo;

// Description
main() {
  ...
}

If the tool has to add an import of IterableExtension from package:collection, it will currently add it between the declaration of main and the comment just above it, resulting in:
library foo;

// Description
import 'package:collection` show IterableExtension;

main() {
  ...
}

As of 1c7fe7196dd7a542f7f91cd6ea2ac81e1a13146c, the null safety migration tool has been removed from active development and retired. No further work on the tool is planned.
If you still need help, or you believe this issue has been closed in error, please feel free to reopen.


text:
pkg/test_runner (an infra package) has not yet migrated to null safety

See also the https://github.com/dart-lang/sdk/issues/49169 meta issue.

Also tracked in b/232495224.

Draft CL: https://dart-review.googlesource.com/c/sdk/+/248008

authors: athomas devoncarew


text:
Compilation error after applying quick fix for prefer_is_empty linter rule

Consider this code:
bool isEmptyNotNull(List<String>? nullableList) {
  return nullableList?.length == 0;
}

The prefer_is_empty linter report reports a warning: Use 'isEmpty' instead of 'length' to test whether the collection is empty. Try rewriting the expression to use 'isEmpty'. When you apply the proposed fix, e.g. by running dart fix --apply, the result is
bool isEmptyNotNull(List<String>? nullableList) {
  return nullableList.isEmpty;
}

which fails to compile with The property 'isEmpty' can't be unconditionally accessed because the receiver can be 'null'.
Expected behavior: if code compiles before dart fix then it should compile after.

Dart version and tooling diagnostic info (dart info): 3.3.0 (stable) (Tue Feb 13 10:25:19 2024 +0000) on "macos_arm64"
Whether you are using Windows, macOS, or Linux (if applicable): macOS
Whether you are using Chrome, Safari, Firefox, Edge (if applicable): N/A

Great catch and awesome report. Thank you!

authors: aemelyanovff pq


text:
Dart analyzer does not infer the correct type for generic record

import 'dart:math';

typedef Setter<T extends num> = void Function(T value);
typedef Getter<T extends num> = T Function();

typedef TestRecord<T extends num> = ({Setter<T> set, Getter<T> get});

class TestClass<T extends num> {
  Setter<T> set = (T value) {};

  Getter<T> get = () => 0 as T;
}

TestRecord<T> getRecord<T extends num>(Type t) {
  return (set: (T value) {}, get: () => 0 as T);
}

TestClass<T> getClass<T extends num>(Type t) {
  return TestClass<T>();
}

void main() {
  final rng = Random();
  TestRecord<int> testRecord = getRecord<int>(int);
  TestRecord<double> testRecord2 = getRecord<double>(double);
  final res = rng.nextDouble() > 0.5 ? testRecord : testRecord2;
  res.set(res.get());

  TestClass<int> testClass = getClass<int>(int);
  TestClass<double> testClass2 = getClass<double>(double);
  final res2 = rng.nextDouble() > 0.5 ? testClass : testClass2;
  res2.set(res2.get());
}

The type of res.set is void Function(Never) instead of void Function(num)

Functions are contravariant with respect to their parameter types. void Function(int) it not a subtype of void Function(num). Consider Liskov's substitution principle: can a void Function(int) be used where void Function(num) is expected? The answer to this question is no: you can't invoke void Function(int) with double value as an argument, but you can invoke void Function(num) with a double value as an argument.
On the other hand void Function(int) and void Function(double) are both subtypes of void Function(Never) - because they can be used anywhere where an instance of void Function(Never) is expected.

authors: mraleph paulmwk


text:
VMService::dispose will complete outstanding requests with RPCErrors that lack stack traces

Discovered while investigating a crasher of the flutter CLI tool: https://github.com/flutter/flutter/issues/153471.
https://github.com/dart-lang/sdk/commit/a7d8707a5964e174f2ce321d075e96e37a9a16ad modified VMService to complete any outstanding requests with errors upon the VM service connection closing. This was achieved by calling VMService::dispose upon the connection closing, which in turn will complete any outstanding request with an RPCError:
https://github.com/dart-lang/sdk/blob/6c92babf4d3a256c15d98f126a5a6c30ed148c7a/pkg/vm_service/lib/src/vm_service.dart#L1739-L1745
Notice that no stack trace is provided. Judging by https://github.com/dart-lang/sdk/commit/9a755f22278cf1d345dd758ce7c162d9b820a51c, should the above code instead be
      request.completeError(RPCError(
        request.method,
        RPCErrorKind.kServerError.code,
        'Service connection disposed',
      ));

?
cc @bkonyi

@derekxu16, can you take a look at this today and prepare a package:vm_service release with the fix?

Thanks for the detailed report @andrewkolos! I've just published package:vm_service 14.2.5 which contains the fix: https://pub.dev/packages/vm_service/versions/14.2.5.

authors: andrewkolos bkonyi derekxu16


text:
pddFlag removal + tests for pdd restricted instance of Analytics

This PR is only focused on removing the flag that prevented the session and log files from being used. There will be a follow up PR that will add the functionality required on opt out which will clear these files.
Reference issue: https://github.com/dart-lang/tools/issues/84

LGTM. Recommend merging after merging a change to clear out the files if possible; that way we are matched up with either v1 or v2 of the pdd at all commits.

Yep good call, this tracking issue has the information for the other pull request to clear everything out https://github.com/dart-lang/tools/issues/84

source: gharchive/pull-request

repo: dart-lang/tools


text:
Color test example

Comfy currently struggles with colors. What many probably attribute to the "comfy look" is in part actually a bug :)
Once the bug is fixed I'd still like to preserve "the comfy look" with a post processing shader, but it should also be possible to render accurate colors.
This includes things like draw_circle(..., RED) actually being true RGBA RED, and not a comfy pink-ish red.
In order to test this, we need an example that draws a bunch of standardized gradients (like egui color test for example here https://github.com/setzer22/godot-egui/issues/14) so that we have a baseline comparison and figure out what's wrong. I suspect something with sRGB, but it'd be great to have this to avoid regressions in the future, and also maybe to test interesting color mapping effects afterwards :)
Note that this issue is not about fixing the actual rendering issue, it's only about creating a color test so we can better diagnose problems with colors, and ultimately lighting as well.

Hi there, I have a few color maps for different pallets that I got to work with the existing colors test, could contribute

This is now implemented in the colors example

authors: darthdeus nuts-rice

repo: darthdeus/comfy


text:
No block sources available

What I did:

Reset network
Manually added nodes from https://www.dashninja.pl/deterministic-masternodes.html#mnlistdetail
Checked that I can reach those nodes on port 9999. I can.

Ideas?
version: v0.12.0.56-7154cdf on Arch Linux.

Well, if you're actually using v0.12.0.56 of Dash Core then that'll be you're issue... You need to be using v19.1.0 https://github.com/dashpay/dash/releases/tag/v19.1.0; please reopen if you continue to have issues.

That's strange, the package was installed via AUR. Source URL is https://github.com/dashpay/dash/releases/download/v19.1.0/dashcore-19.1.0-x86_64-linux-gnu.tar.gz
but the about inside the gui reports v0.12.0.56-7154cdf.

That was a local error on my system. Keep it close

authors: PastaPastaPasta shaygover

repo: dashpay/dash

license: mit

license_source: bigquery


text:
Lazy unique implementation

The current implementation of unique seems to evaluate immediately. Would be nice if unique were rewritten  to be lazy.

Implementation in PR ( https://github.com/dask/dask/pull/2775 ).

repo: dask/dask


text:
warn_dtype_mismatch -method fails sometimes

Not easy to reproduce, but the check in warn_dtype_mismatch-method
if not left.dtypes[lo] is right.dtypes[ro]

fails sometimes. Maybe one should not use 'is', but instead '=='?
https://github.com/dask/dask/blob/master/dask/dataframe/multi.py#L371-L374

When does this arise?  No objection from me on changing the is to ==, but
it would be nice to know when this is important.
On Fri, Sep 27, 2019 at 5:56 AM JukkaKeisala notifications@github.com
wrote:

Not easy to reproduce, but the check in warn_dtype_mismatch-method
if not left.dtypes[lo] is right.dtypes[ro]
fails sometimes. One should not use 'is', but instead '=='.
https://github.com/dask/dask/blob/master/dask/dataframe/multi.py#L371-374
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/issues/5437?email_source=notifications&email_token=AACKZTCB6VRITEHXQMIE22DQLXRE7A5CNFSM4I3E5JE2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HOD5HSQ,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACKZTACBHX2LZCVQ7QGWBTQLXRE7ANCNFSM4I3E5JEQ
.

@JukkaKeisala any response to https://github.com/dask/dask/issues/5437#issuecomment-535923484?
One problem with using == is that pandas' extension dtypes can't be safely compared to NumPy dtypes
In [4]: np.dtype('int') == pd.CategoricalDtype()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-ec85c3b44381> in <module>
----> 1 np.dtype('int') == pd.CategoricalDtype()

TypeError: data type not understood

@mrocklin @TomAugspurger
This happened when tried to merge two dask dataframes. Left and right dtypes were the same in asciitable (showed in log/output). Because dtypes were the same I deduced that the error shouldn't have been raised.
Of course it is possible that the printing of asciitable hides the problem...
On my local environment I went to code and changed is to == and the warning was gone.
The main thing is how to safely compare datatypes, as @TomAugspurger said.
And the main thing, I've tried to make easy reproducable example of this but I've failed. This goes back to that that sometimes is work's as ==. This usually happens in more complicated examples, not in short exampled as far as I know.

I saw somewhere (maybe in NumPy-related stuff) that the comparison was made by using try-except to add more choices. Of course, the datatype comparison is wider topic than this case.

We could try using pandas.api.types.is_dtype_equal instead, which should be robust.
@JukkaKeisala do you have an example where this comes up? Ideally we would have a test case for it.

The following JSON data lies in web (server through REST API in my case):
'{"microsecondtimestamp": {"0": 1568981552099000, "1": 1568981799149000, "2": 1568981814126000, "3": 1568981896929000, "4": 1568982049422000, "5": 1568982099443000, "6": 1568982866835000, "7": 1568982875109000, "8": 1568982891157000, "9": 1569220182183000}, "values": {"0": 59.0, "1": 59.0, "2": 59.0, "3": 59.0, "4": 61.2, "5": 61.4, "6": 61.4, "7": 61.4, "8": 69.1, "9": 74.1}}'

at endpoint myendpoint/data1
and
'{"microsecondtimestamp": {"0": 1569224231024000, "1": 1569225379890000, "2": 1569225399425000, "3": 1569225425104000, "4": 1569323898216000, "5": 1569325297211000, "6": 1569325349491000, "7": 1569325580105000, "8": 1569325587558000, "9": 1569325641823000}, "values": {"0": 54, "1": 54, "2": 55, "3": 56, "4": 56, "5": 99, "6": 99, "7": 99, "8": 99, "9": 99}}'

at endpoint myendpoint/data2
If I run the following script on TCPServer (using TCPServer.serve_forever + parameters)
df1 = dd.read_json('myendpoint/data1', orient='columns')
df2 = dd.read_json('myendpoint/data2', orient='columns')

id1 = id(df1.dtypes['timestamp'])
id2 = id(df1.dtypes['timestamp'])

the id's are not the same but the datatypes are (i.e., ==). This is the reason why I raised this issue.
I cannot reproduce this in iPython or Jupyter Notebook.

Thanks @JukkaKeisala, that and https://github.com/numpy/numpy/issues/6295 was enough to get a reproducer
In [102]: df1 = pd.DataFrame({"a": np.array([1, 2], dtype=np.dtype(np.int64))})

In [103]: df2 = pd.DataFrame({"a": np.array([1, 2], dtype=np.dtype(np.longlong))})

In [104]: dd.multi.warn_dtype_mismatch(df1, df2, 'a', 'a')
/Users/admin/miniconda3/envs/dask-dev/lib/python3.7/site-packages/dask/dataframe/multi.py:387: UserWarning: Merging dataframes with merge column data type mismatches:
+---------------+------------+-------------+
| Merge columns | left dtype | right dtype |
+---------------+------------+-------------+
| ('a', 'a')    | int64      | int64       |
+---------------+------------+-------------+
Cast dtypes explicitly to avoid unexpected results.
  ).format(col_tb)

I don't really know what np.longlong is, but the repr is dtype('int64') for me, and the np.iinfo() looks the same. I'll see if we can use is_dtype_equal instead there.

Nice reproducer you found @TomAugspurger, I think is_dtype_equal will work here. I cannot check it for a couple of days but will do that as soon as possible.

No need. I have an open PR and the tests all passed.

On Oct 12, 2019, at 12:39 PM, JukkaKeisala notifications@github.com wrote:

Nice reproducer you found @TomAugspurger, I think is_dtype_equal will work here. I cannot check it for a couple of days but will do that as soon as possible.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.

authors: JukkaKeisala TomAugspurger mrocklin


text:
Could not serialize object of type tuple error using tldextract library

import dask.bag as db
from tldextract import extract

def main():
    db.read_text(INPUT_PATH).map(json.loads).map(lambda x: extract(x['uri']).domain)).compute()

if __name__ == '__main__':
    main()

Using the extract method from tldextract inside of a Dask processing pipeline yields an error.
TypeError: ('Could not serialize object of type tuple.', "(<function reify at 0x109b0c430>,
Environment:

Dask version: 2.30.0
Python version: 3.8.5
Operating System: macOS
Install method (conda, pip, source): Source

Does that tuple contain non-serializable elements? You can try calling cloudpickle.dumps on the result locally.

Thanks

authors: TomAugspurger kevchn


text:
Poor performance when combining many blockwise operations

The Problem
I was doing some performance tests and ran into some pretty poor behavior when a lot (>100) of Blockwise operations are applied in series. The same example framed with a single map_partitions call performs much better. A minimal reproducer:
import distributed
import dask.dataframe as dd
from dask.datasets import timeseries

# A dataframe with a lot of columns to mess with
ddf = timeseries(
    dtypes={
        **{f"name-{i}": str for i in range(25)},
        **{f"price-{i}": float for i in range(25)},
        **{f"id-{i}": int for i in range(25)},
        **{f"cat-{i}": "category" for i in range(25)},
    },
    freq="5s"
)

def transform(df):
    """
    Transform the columns of a pandas or dask dataframe.
    """
    transform = {}
    for c in df.columns:
        dtype = str(df[c].dtype)
        if dtype == "object":
            transform[c] = df[c].str.upper()
        elif dtype == "int64":
            transform[c] = df[c] * 2
        elif dtype == "float64":
            transform[c] = df[c] / 2
    return df.assign(**transform)

ddf.map_partitions(transform).compute()  # does okay!
transform(ddf).compute()  # does *very* badly

This reproduces on both local and distributed schedulers, but a comparison of the two from the dask dashboard is shown here (purple is map_partitions, teal uses the top-level dataframe API):

Not only does using the top-level Dask APIs take several times longer, but it also takes several times as much memory, and it's not hard to knock over workers with the second approach.
A bit of exploration and reflection reveals what's going on: the Blockwise layers are being correctly fused into a single layer as we want during optimization. But the resulting local subgraph is pretty complex, and has a ton of intermediate results! If I drill into the layer and plot the subgraph, I get the following:

The performance characteristics of this graph are way worse than just bundling them into the one map_partitions call and letting pandas handle intermediate results internally.
Does this matter? I think so. I don't think what I'm doing is too far from "best practices". It is embarrassingly parallel, has immutable operations, and batches all of my transformations together into a single compute() call. From an API design perspective, I want to be able to tell users to use the top level API and use map_partitions as an escape hatch. I'm not really sure what to do about this behavior right now. It's not necessarily a bug, since Blockwise is more-or-less doing the right thing. But it's performing so much worse that I thought I'd write this up for discussion. Fixing cases like this might be worth considering in any HLG redesign we do (cf #8980, #9076).
Appendix: why am I using `DataFrame.assign?
I don't see too many people using DataFrame.assign like I am here. But I like it for a few reasons:

It is immutable as opposed to __setitem__
It makes it easier to use method chaining
I understand it, and __setitem__ is too complicated.

I want to be able to tell users to use the top level API and use map_partitions as an escape hatch

I personally view map_partitions as both and "escape hatch" and an optimization. I agree that we should treat pathological cases like this seriously, but I have a feeling we may have better luck encouraging the use of map_partitions when the user/dev knows munual fusion makes the most sense .

but I have a feeling we may have better luck encouraging the use of map_partitions when the user/dev knows munual fusion makes the most sense .

Yeah, that's fair. I have a general desire to make pandas best-practices also work with dask dataframes, but it may be unrealistic to expect too much. That being said, I don't think computations like these (lots of embarrassingly parallel transformations) are particularly unusual in an ETL context, so it's a pathological case that I expect to see in the wild.

I definitely think of this as a bug. Would applying standard graph optimizations to the SubgraphCallable help here? Or is the subgraph being constructed in some pathological way by Blockwise fusion?

authors: gjoseph92 ian-r-rose rjzamora


text:
'mode' argument passed to bokeh.output_file() (#7034)

Hello,
Here is a simple modification for my 1rst PR in github. (Please be indulgent)
Q1 : The #7034 in the title is related to my post in Issues. Is it the way to do it ?
Q2 : When my PR will be accepted, who will be in charge of the final commit ?
Comment : I confirm what I read in the panda documentation
To the new user, working with Git is one of the more daunting aspects of contributing
Hope I did the right things.
Patrick

Hmm.. Linting has failed and I have no idea about the meaning of the error msg.

Hmm.. Linting has failed and I have no idea about the meaning of the error msg.

Thanks for the PR @patquem it'll be included in the 2021.01.1 release on Friday.

Thanks for the PR @patquem it'll be included in the 2021.01.1 release on Friday.

authors: jsignell patquem


text:
Fixed deprecated syntax in StyledComponent.

The master branch was giving off a warning about the "extend" API. This replaces it with the new "styled" API.
For reference here is the original error:
ThemeProvider.js:28 Warning: The "extend" API will be removed in the upcoming v4.0 release. Use styled(StyledComponent) instead. You can find more information here: https://github.com/styled-components/styled-components/issues/1546
(anonymous) @ ThemeProvider.js:28

@SirRujak Thank you for the PR.

authors: SirRujak martinheidegger

repo: dat-land/dat-desktop

license: MIT


text:
Support multi-catalog

resolves #95
Description
Supports multi-catalog.

This is based on #94.

Thanks! merging.

repo: databricks/dbt-databricks


text:
Implement Koalas Missing APIs

Help wanted! A few popular pandas APIs are missing in Koalas. We are going to implement them!
Please use this thread to comment on which function you will be working so we don't duplicate work. Please mention this issue in your PR so that the list below can be updated.

DataFrame

[ ] cov
[x] first
[x] itertuples
[ ] resample
[x] combine_first
[x] between_time
[ ] boxplot
[x] insert
[x] swapaxes

Series

[ ] ewm
[ ] resample
[x] factorize
[x] last
[x] swapaxes

Index

[x] to_list / tolist
[ ] map

SeriesGroupBy and DataFrameGroupBy

[x] median
[x] tail

Hi, I would like to help. I was planning on picking combine_first functionality.
Can you let me know if I can work on it. Thanks!

Please go ahead @AishwaryaKalloli!

Certainly, thank you @AishwaryaKalloli!

Just finished the set up in my local, hopefully will have some updates soon!

I have committed the code, let me know if it is in the right direction. If it is I'll add the test cases and docs.

Hi @ueshin, @HyukjinKwon can I proceed with the following Dataframe APIs -

cov
first
between_time
insert
I'll start with the dev once you give me the approval.

@shril sure, please go ahead! Thanks!

@ueshin I was going through this blog post of yours - https://databricks.com/blog/2020/03/31/10-minutes-from-pandas-to-koalas-on-apache-spark.html, and it suggested using map_in_pandas as a better alternative workaround because it does not require moving data into a single client node and potentially causing out-of-memory errors.
Do you suggest to proceed with using the map_in_pandas API?

@shril I don't have a strong opinion on it. If you can implement it without apply_batch(), you don't need to use it.

Hi @ueshin, I am slightly confused. We don't have DatetimeIndex like pandas, so can we convert our index to pandas and use it's indexer_between_time function.
i = pd.date_range('2018-04-09', periods=2000, freq='1D1min')
ts = ks.DataFrame({'A': ['timestamp']}, index=i)
indexer = ts.index.to_pandas().indexer_between_time(start_time='0:15', end_time='0:45')
result = ts.copy().take(indexer)

This is the small implementation I tried. Do you think that to_pandas() might result in out-of-memory errors?

Is iterating through groups on the roadmap for API coverage? I would find that helpful.

Is iterating through groups on the roadmap for API coverage? I would find that helpful.

@chogg Thanks for the suggestion! We'll look into this and keep you updated.

@chogg Thanks for the suggestion! We'll look into this and keep you updated.

Hi all, I was going to look at implementing functionality for last Is anyone looking at this?

@awdavidson Certainly, please feel free to do so! Your PR seemed to be closed.

@xinrong-databricks I'll reopen currently still working on it. Opened the PR to check build etc as had issue running a few things locally - didn't want to clutter your PR tab. Local environment is now working so should be able to completely test :)

@awdavidson Thanks! You might mark it as a draft PR until it's ready for review. Let us know if you have any questions :)

@ueshin @xinrong-databricks has there been any discussion around how to implement Index.map? I have a few idea's that maybe useful, however, do not want to step on anyones toes! If you have any information or documentation that would be useful! :)

@awdavidson As we have not been working on it, you can go ahead.
One thing on the example implementation, using self._index.values is not a good idea because it collects all data into a Driver node which could cause OOM.
Thanks!

authors: AishwaryaKalloli HyukjinKwon awdavidson chogg shril ueshin xinrong-databricks

repo: databricks/koalas


text:
Include Dask license again

The Dask license was removed in #284, where the user-facing Dask code got deleted, but there are still bits of Dask in the codebase, at least in tests. As noted in #12:

The license allows us to copy the code, if we put the proper attribution in the README

And since these tests are still in place, I suggest the license gets included again.
(I'm not a licensing expert, I'm not sure what the implications of code copying are, I'm just following the quote from the issue above.)

No, the tests are completely different if you refer this file:
https://github.com/dask/dask/blob/master/dask/dataframe/tests/test_dataframe.py
vs
https://github.com/databricks/koalas/blob/master/databricks/koalas/tests/test_dataframe.py
and we now have anything copied from Dask.

Look at the issue I linked, where the author talks explicitly about adapting Dask code. And the code is still live in your test suite, see e.g.:
https://github.com/dask/dask/blob/e734544ef74aa0b1db1144d3f0a193b8f313d9d1/dask/dataframe/tests/test_dataframe.py#L156
https://github.com/databricks/koalas/blob/465861cbb70dbe98ba8b5ac5477ed7c99b184b2d/databricks/koalas/tests/test_dataframe.py#L307

Here's the test you pointed out:

I don't think we should add a license when it borrows an idea.

authors: HyukjinKwon kokes


text:
Add yaml compilation test

Added test to make sure the action.yml file compiles correctly.
This is how the test fails when I added locally a bad indentation to ./action.yml:

From above build-test / build-and-test workflow

repo: databricks/run-notebook


text:
tracking non-commit GitHub actions

I've just learned about this interesting tool. I was looking for something to help with a different project. Instead of looking at the activity around a particular open source project, I wanted to track the activity of a cohort of GitHub users. In particular, those that attend some sort of training like Software Carpentry or other university-run training in good software practices. We would like to know if these training efforts are useful, and one way of measuring that success is to track the student's activity levels on GitHub over time. Do they continue using GitHub for their projects? Do they contribute to others? etc.
I see that bigbang can deal with commits on GitHub and email conversations, but it's not clear to me yet from your documentation if bigbang tracks other kinds of activities on GitHub, like posting issues or comments (on any repository). I'm also not sure if the GitHub API allows for this. For instance, if you knew only my username, would you be able to find this issue?
If bigbang does support this, maybe it can be called out in the documentation a bit more clearly.
If not, consider this a feature request.

It doesn't yet work with those data sources but yes that's a great feature.

One application of this feature would be to help informed BigBang's own governance process. It's been suggested that issue tracker activity should counted as a participation metric for determining membership of the Steering Committee.
http://lists.ghserv.net/pipermail/bigbang-dev/2016-October/000042.html
See also:
https://github.com/datactive/bigbang/wiki/Governance

authors: cranmer sbenthall

repo: datactive/bigbang


text:
FIX nb2: clarify that "selecting subsets" are not cumulative

We select different subsets from the same base dataframe, the notebook should be clear that we are not iteratively refining the selection (ie. we are not selecting a smaller subset each time).

Solved on branch

repo: datactivist/scpo-data-science-bootcamp


text:
Correction for raw affiliation MoISA, Université de Montpellier, Cirad, Ciheam-IAMM, Inrae, Institut Agro, IRD, Montpellier, France.

Correction needed for raw affiliation MoISA, Université de Montpellier, Cirad, Ciheam-IAMM, Inrae, Institut Agro, IRD, Montpellier, France.
raw_affiliation_name: MoISA, Université de Montpellier, Cirad, Ciheam-IAMM, Inrae, Institut Agro, IRD, Montpellier, France.
new_rors: 05kpkpg04;051escj72;05q3vnk25;03rnk6m14;0005r2j17;011xg1225
previous_rors: 05kpkpg04;051escj72;05q3vnk25;03rnk6m14;0005r2j17
works_examples: W4402512292
contact: 3028d62fc7a12badf634b585960ccc8e:d1b9e57eb45835dc @ supagro.fr

This issue was accepted and ingested by the OpenAlex team on 2024-12-19. The new affiliations should be visible within the next 7 days.

repo: dataesr/openalex-affiliations


text:
Correction for raw affiliation UMR 1083 SPO, SUPAGRO, INRA, Montpellier, France

Correction needed for raw affiliation UMR 1083 SPO, SUPAGRO, INRA, Montpellier, France
raw_affiliation_name: UMR 1083 SPO, SUPAGRO, INRA, Montpellier, France
new_rors: 03rnk6m14;04w07sc25
previous_rors: 03rnk6m14
works_examples: W2158770972
contact: 27d2c598aab364e44d642ab0385783ea:9efcb8060ab35254aa3d8e9417b2bcf9c1d6 @ inrae.fr

This issue was accepted and ingested by the OpenAlex team on 2024-10-09. The new affiliations should be visible within the next 7 days.


text:
Correction for raw affiliation Aeres Applied University Wageningen, Wageningen, The Netherlands

Correction needed for raw affiliation Aeres Applied University Wageningen, Wageningen, The Netherlands
raw_affiliation_name: Aeres Applied University Wageningen, Wageningen, The Netherlands
new_rors: 03jnx2c74
previous_rors:
works_examples: W4386004792
contact: 4636de9b5c31d785d5113cc65268e355:056e4cb80b5d06a7e960 @ hotmail.com

This issue was accepted and ingested by the OpenAlex team on 2024-10-10. The new affiliations should be visible within the next 7 days.


text:
Nettoyer la hierarchie des fichiers du site

Actuellement, on hérite d'une structure des fichiers qui est complexe, avec une partie des composants dans source/sites/publicodes, et l'autre dans components. Elle n'a aucun sens aujourd'hui.
Plutôt facile à corriger... mais surtout si plus aucune PR en attente.

Oups, déjà ici #165

repo: datagir/nosgestesclimat-site


text:
fix(docs): remove link-checker from CI

It had too many false positives, so can't be a blocking step in CI. However, it was still super useful so we'll just need to manually run it every now and then.
Checklist

[ ] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
[ ] Links to related issues (if applicable)
[ ] Tests for the changes have been added/updated (if applicable)
[ ] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
[ ] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

Merging through smoke test failure

repo: datahub-project/datahub


text:
Reaper does not work when Cassandra authentication is used

Consider the following example CassandraDatacenter manifest:
apiVersion: cassandra.datastax.com/v1beta1
kind: CassandraDatacenter
metadata:
  name: dc1
spec:
  clusterName: cluster1
  serverType: cassandra
  serverVersion: "3.11.6"
  managementApiAuth:
    insecure: {}
  size: 3
  storageConfig:
    cassandraDataVolumeClaimSpec:
      storageClassName: server-storage
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
  reaper:
    enabled: true
  config:
    cassandra-yaml:
      authenticator: org.apache.cassandra.auth.PasswordAuthenticator
      authorizer: org.apache.cassandra.auth.CassandraAuthorizer
      role_manager: org.apache.cassandra.auth.CassandraRoleManager

Cass Operator will create a C* cluster with authentication enabled. Reaper talks to Cassandra using both JMX and CQL. The CQL calls will fail because no credentials are passed to Reaper.
It might be good for .spec.reaper to allow you to specify a C* user/role for Reaper to use. I think it is sufficient for now though for the operator to auto generate a role for Reaper.

This may require some more effort and changes that I initially anticipated. For its readiness probe, Reaper attempts to connect to its storage backend, which is Cassandra. In order for the readiness check to succeed, the reaper_db schema needs to exist. Cass Operator deploys a k8s job to create the keyspace. The Casandra pods which includes both Cassandra and Reaper are deployed before the job. There may be some Reaper container restarts until the job has a chance to run.
When authentication is enabled, the operator needs to create a C* role before the schema initialization job can run. This makes things more complicated. In the reconciliation loop today the schema initialization job does not run the statefulsets are ready. The statefulsets however cannot reach the ready state until both the Cassandra and Reaper containers reach the ready state. Chicken meet egg.
I should also point out that in my prototyping I am creating a separate C* role for Reaper. I am trying to piggyback off the existing user creation code. Specifically, I have modified the GetUsers function in reconcile_racks.go to add a user for Reaper when Reaper and authentication are enabled. GetUsers is called by CreateUsers.
One option is to create the Reaper in advance of and separately from the call to CreateUsers. The statefulsets won't be ready but this should be fine as long as all of the Cassandra containers are ready. There are at least a couple downsides to this approach. First, we will probably see even more container restarts with Reaper as we wait for both the role to get created and for the schema to get initialized. This creates a lot of noise for users which will make debugging that more difficult. The second issue is that CreateUsers is not written in a way to be easily reused. I would likely need to refactor both GetUsers and CreateUsers in order to avoid a lot of duplication.
Alternatively, we could take an install and update approach. We deploy the statefulsets but do not include the Reaper container initially. Then the operator creates users including the Reaper user. After user creation, we run the schema initialization job. At this point we can update the PodTemplateSpec in order to deploy Reaper. We may want to consider adding a status field (or fields) for Reaper. It would indicate that the schema is initialized. When this field gets set, the operator would know it is is safe to deploy Reaper.
This brings me to another point about Reaper schema initialization. Suppose a user deletes the reaper_db keyspace. Reaper's liveness probes will start to fail. I am not sure if the operator can get things back to the desired state in this situation.  I will need to do some testing.

There are similar-but-different issues (creds needed, etc) when TLS and/or mTLS are enabled on the cassandra cluster. user/role won't solve that problem - but cass operator creates user, pw and certs - can't cass operator just let reaper mount those secrets and use them and solve both user/role auth and (m)TLS at the same time?

can't cass operator just let reaper mount those secrets and use them and solve both user/role auth and (m)TLS at the same time?

I do not think that this would solve the auth problem. The start up order of the cassandra and reaper containers is non-deterministic. Reaper cannot get to a ready state until:

The Reaper role is created in Casssandra and
The Reaper schema is initialized in Cassandra

Reaper's health checks depend on both of the above being completed. As things are implemented, this results in lot of liveness and readiness probe failures.
Without changes in Reaper, its sidecar mode is not a good fit for integration with Cass Operator. I am working on a different approach for integration that won't hit these issues.

But waiting for state to settle is kubernetes super power.
Maybe kick off the reaper initialization job before creating the stateful sets but with configuration about where to look for secrets when they show up. There cassandra containers will start and eventually the job that sets up the reaper_db will get creds and have an actual cassandra to connect to. Then the reaper containers will successfully start.
Feels like liveness / readiness failures don't really matter provided startup order is documented and state eventually settles.
I do think the install and update approach is also a decent way to go. Or quick anyway :-)
And a reaper operator would certainly be more flexible - best of luck!

authors: jsanda underrun

repo: datastax/cass-operator


text:
Add Trino use case in the Jupyter notebook

Currently, Gravitino playground support Jupyter notebook, We can add a new notebook to explain how to use Trino to federation query mysql, postgres and hive, like this, http://github.com/datastrato/gravitino-playground?tab=readme-ov-file#example
I think we can use these Trino python library

https://github.com/CybercentreCanada/jupyterlab-sql-editor/blob/main/example/Trino.ipynb
https://github.com/trinodb/trino-python-client

Preferably, we can design a federated query scenario.

I'll work on it.

repo: datastrato/gravitino-playground


text:
Added ExcelDateTime.java

Added ExcelDateTime.java

LGTM

repo: datautils-io/tableutils


text:
package management on VM

@dkapitan
I am having a difficulty with package management on the vm instance. Namely, when trying to run:
$python nl_open_data/flows/cbs/register_statline_bq_flow.py
I get an error message saying the Dask bag requirements are not installed. This is the full error message:
(nl-open-data-VUruowue-py3.8) amitgalmail@nl-open-data-vm:~/nl-open-data$ python nl_open_data/flows/cbs/register_statline_bq_flow.py
Traceback (most recent call last):
  File "/home/amitgalmail/.cache/pypoetry/virtualenvs/nl-open-data-VUruowue-py3.8/lib/python3.8/site-packages/dask/bag/__init__.py", line 2, in <module>
    from .core import (
  File "/home/amitgalmail/.cache/pypoetry/virtualenvs/nl-open-data-VUruowue-py3.8/lib/python3.8/site-packages/dask/bag/core.py", line 41, in <module>
    from ..bytes import open_files
  File "/home/amitgalmail/.cache/pypoetry/virtualenvs/nl-open-data-VUruowue-py3.8/lib/python3.8/site-packages/dask/bytes/__init__.py", line 9, in <module>
    raise ImportError(
ImportError: fsspec is required to use any file-system functionality. Please install using
conda install -c conda-forge 'fsspec>=0.3.3'
or
python -m pip install 'fsspec>=0.3.3'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "nl_open_data/flows/cbs/register_statline_bq_flow.py", line 18, in <module>
    from statline_bq.utils import (
  File "/home/amitgalmail/.cache/pypoetry/virtualenvs/nl-open-data-VUruowue-py3.8/lib/python3.8/site-packages/statline_bq/utils.py", line 10, in <module>
    import dask.bag as db
  File "/home/amitgalmail/.cache/pypoetry/virtualenvs/nl-open-data-VUruowue-py3.8/lib/python3.8/site-packages/dask/bag/__init__.py", line 26, in <module>
    raise ImportError(str(e) + "\n\n" + msg) from e
ImportError: fsspec is required to use any file-system functionality. Please install using
conda install -c conda-forge 'fsspec>=0.3.3'
or
python -m pip install 'fsspec>=0.3.3'

Dask bag requirements are not installed.

Please either conda or pip install as follows:

  conda install dask               # either conda install
  python -m pip install "dask[bag]" --upgrade  # or python -m pip install

However, fsspec is installed, as is apparent by wither running poetry show fsspec:

(nl-open-data-VUruowue-py3.8) amitgalmail@nl-open-data-vm:~/nl-open-data$ poetry show fsspec
name         : fsspec
version      : 0.8.5
description  : File-system specification

or even just python -m pip install 'fsspec>=0.3.3, as stated by the error log:

(nl-open-data-VUruowue-py3.8) amitgalmail@nl-open-data-vm:~/nl-open-data$ python -m pip install 'fsspec>=0.3.3'
Requirement already satisfied: fsspec>=0.3.3 in /home/amitgalmail/.cache/pypoetry/virtualenvs/nl-open-data-VUruowue-py3.8/lib/python3.8/site-packages (0.8.
5)

On my local machine that does not happen, but I actually seem to remember something similar happening in the beginning, but I cannot remember what exactly - just that this is not the first time I ran into issues with fsspec...

Any thoughts are welcome :)

update
This has to do with statline-bq somehow, as the error occurs when importing from it:

update
This has to do with statline-bq somehow, as the error occurs when importing from it:

Seems to be an issue with the bz2 module.

Seems to be an issue with the bz2 module.

did you install bz2 on the os level? poetry doesn't handle that. so:
sudo apt get install bz2
should help
On Tue, Jan 26, 2021 at 3:01 PM Amit Gal notifications@github.com wrote:

Seems to be an issue with the bz2 module.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dataverbinders/nl-open-data/issues/50#issuecomment-767560063,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA7GYDTVEIYLGCAE6JNQXATS33DMFANCNFSM4WTKSYSA
.

did you install bz2 on the os level? poetry doesn't handle that. so:
sudo apt get install bz2
should help
On Tue, Jan 26, 2021 at 3:01 PM Amit Gal notifications@github.com wrote:

Seems to be an issue with the bz2 module.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dataverbinders/nl-open-data/issues/50#issuecomment-767560063,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA7GYDTVEIYLGCAE6JNQXATS33DMFANCNFSM4WTKSYSA
.

Thanks! I'm actually just reinstalling some things now, hopefully I got it - I think I missed a prerequisite for pyenv that is supposed to take care of it. Sorry to bother you, I've actually been using this thread to log for the future if needed (btw - are you getting an Email per comment on this thread, or just the first mention?)

Thanks! I'm actually just reinstalling some things now, hopefully I got it - I think I missed a prerequisite for pyenv that is supposed to take care of it. Sorry to bother you, I've actually been using this thread to log for the future if needed (btw - are you getting an Email per comment on this thread, or just the first mention?)

i get emails for everything ... don't you worry, I will manage my own
happiness and turn notifications off when it's too much :-)
On Tue, Jan 26, 2021 at 3:36 PM Amit Gal notifications@github.com wrote:

Thanks! I'm actually just reinstalling some things now, hopefully I got it

I think I missed a prerequisite for pyenv that is supposed to take care
of it. Sorry to bother you, I've actually been using this thread to log for
the future if needed (btw - are you getting an Email per comment on this
thread, or just the first mention?)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dataverbinders/nl-open-data/issues/50#issuecomment-767582865,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA7GYDXH5ELYYGZHHVKV6C3S33HNVANCNFSM4WTKSYSA
.

i get emails for everything ... don't you worry, I will manage my own
happiness and turn notifications off when it's too much :-)
On Tue, Jan 26, 2021 at 3:36 PM Amit Gal notifications@github.com wrote:

Thanks! I'm actually just reinstalling some things now, hopefully I got it

I think I missed a prerequisite for pyenv that is supposed to take care
of it. Sorry to bother you, I've actually been using this thread to log for
the future if needed (btw - are you getting an Email per comment on this
thread, or just the first mention?)

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dataverbinders/nl-open-data/issues/50#issuecomment-767582865,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AA7GYDXH5ELYYGZHHVKV6C3S33HNVANCNFSM4WTKSYSA
.

:)
OK, issue solved. Indeed, some prerequisites of pyenv were not properly installed, leading to bz2 to not be installed, leading to import error on fsspec, and so on.
This all happened because I copied a segment for dependencies from a guide:
root@ubuntu:~# apt install -y make build-essential libssl-dev zlib1g-dev \
> libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev\
> libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python-openssl\
> git

and did not notice that pasting it into the VM terminal, it automatically adds the ">" on its own, making it into:
apt install -y make build-essential libssl-dev zlib1g-dev \
> > libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev\
> > libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python-openssl\
> > git

🤦
A day's work for a couple of very important lessons... Not that bad :)
Also - we now have full remote capabilities, meaning I've successfully used the VM to register&run a flow, uploading a CBS dataset. It's strange to see the prefect UI running without the usual fan noise in the background ;)

:)
OK, issue solved. Indeed, some prerequisites of pyenv were not properly installed, leading to bz2 to not be installed, leading to import error on fsspec, and so on.
This all happened because I copied a segment for dependencies from a guide:
root@ubuntu:~# apt install -y make build-essential libssl-dev zlib1g-dev \
> libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev\
> libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python-openssl\
> git

and did not notice that pasting it into the VM terminal, it automatically adds the ">" on its own, making it into:
apt install -y make build-essential libssl-dev zlib1g-dev \
> > libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev\
> > libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev python-openssl\
> > git

🤦
A day's work for a couple of very important lessons... Not that bad :)
Also - we now have full remote capabilities, meaning I've successfully used the VM to register&run a flow, uploading a CBS dataset. It's strange to see the prefect UI running without the usual fan noise in the background ;)

repo: dataverbinders/nl-open-data


text:
JSON output option

Write output objects to .markdowndb/ e.g.
.markdowndb/
  files.json  # array of file objects

  # BONUS - don't need to do these yet
  tags.json  # tags indexed by tag names with a list of files
  links.json   # array of link objects

Why do this? Simplest thing that a web dev could use ... may also solve the hot reloading need in #45

@rufuspollock
The code for this is complete and ready to be pushed. Please let me know what our next steps will be.

@mohamedsalem401 there is no point pushing fileTags imo. I'd just push the fiiles.json with a list of tags (fileTags only makes sense for sql where you need a join table).
In fact, i'm not sure we even need the separate links or tags json dumps (why do you think we need them?)

@mohamedsalem401 and note how in the issue i said we didn't need the tags.json or links.json yet 😄

@rufuspollock
Yes, I will put a PR next thing tomorrow.
And I know that you wanted only files.json, I just added them in the screenshot for you to see how it would look like if we ever wanted added them all but the PR would have just files.json.

@mohamedsalem401 great 😄

This is done https://github.com/datopian/markdowndb/pull/73 , should I close the issue?

@mohamedsalem401 yes let's do that!

repo: datopian/markdowndb


text:
20200813 - meeting

Just an hour hanging out and talking about DAT stuff.
:point_right: Post a comment in this github issue with any topics you'd like to cover this week
:alarm_clock: Time: 13 Aug 2020, 21:00 Asia/Tokyo
:link: https://zoom.us/j/181036238 (Native)
:link: https://zoom.us/wc/join/181036238?pwd= (Web)

Timezone
Date/Time

Sydney
13 Aug, 10 PM

Tokyo
13 Aug, 9 PM

Berlin
13 Aug, 10 AM

Buenos Aires
13 Aug, 5 AM

New York
13 Aug, 4 AM

Los Angeles
13 Aug, 1 AM

Or in your local time: https://www.timeanddate.com/worldclock/fixedtime.html?msg=Dat+Comm-Comm+Meeting+2020-08-13&iso=20200813T21&p1=671&ah=1

I was zoning out working on post conference tasks and completely forgot about the session. 🙇 - very, very, very sorry. @zootella tried to join but found the room empty, it must have been weird, sorry about that!

repo: datproject/comm-comm


text:
Tamilnadu 2022-23 - Districts to  be updated.

Missing district name is "Mayiladuthurai". Total - 38 districts. But, currently having only 37 districts.

updated! please cgeck


text:
Mac installation with brew

Or pip, etc.

Documented in ee8264e27783bf16a7ec5a5cc760e5ee77aa0cee
Brew repo: https://github.com/dave1010/homebrew-clipea


text:
cross compiling libraries

I am using you code to cross compile for arm. I am now using the libraries

"github.com/google/gopacket"
"github.com/google/gopacket/layers"
"github.com/google/gopacket/pcap"

They are missing from the cross compiled libraries, is there a way to get it to cross compile those libraries aswell? It currently does the standard library.
I am using the Revel framework, so to do the cross compilation I am running
GOOS=linux GOARCH=arm revel package tcpscanner
Thanks
Alex

If those libraries are referenced by the package you are compiling, they will be compiled as well.
Can you please provide some output, along with an explanation of what you expect to see and what you actually see.
Thanks
Dave

I didn't submit this, but with Go 1.8.3 on Mac OS X I get this:
GOOS=linux GOARCH=amd64 go build
repo/url/projectname
./main.go:20: undefined: pcap.Handle
./main.go:65: undefined: pcap.OpenLive
I worked around this by cloning the repo inside a CentOS virtual machine and building it there.

No more issues please. This project is no longer needed

authors: amlwwalker davecheney tutley

repo: davecheney/golang-crosscompile


text:
Delay closing the app until the user confirms they have seen the error.

If a user runs their bot with an error, for example not having a config file, the app simply closes before they have a chance to read the error message. This commit fixes that.

I don't know how important this is. In the console or terminal, you can still read what went wrong and it's not like the user really can do anything about it at run time if the config-file is missing, for example.
I'm not against it, but I would see more utility in better error logging, like dumping it in a file.

I am not going to pull this, since users can simply run the bot from the command line to see the outout, and the solution you have proposed is annoying to deal with if you're doing any sort of automation.

authors: IanGallacher davechurchill pimmen89

repo: davechurchill/commandcenter


text:
Update PyPi

Howdy,
Looks like it has been a while since PyPi has been updated (May 2013).  Can you update with the latest version so I can get access to the --query argument?  Thanks!

Done.
https://pypi.python.org/pypi/django-fixture-magic
On Tue, Mar 24, 2015 at 8:52 AM Bill Blanchard notifications@github.com
wrote:

Howdy,
Looks like it has been a while since PyPi has been updated (May 2013). Can
you update with the latest version so I can get access to the --query
argument? Thanks!
—
Reply to this email directly or view it on GitHub
https://github.com/davedash/django-fixture-magic/issues/31.

Now that's service!  Thanks for a great library!

authors: chromakey davedash

repo: davedash/django-fixture-magic


text:
Docker: Specifiy Interface for Server to bind to

Is it possible to bind the server to a specific interface? Right now the Server chooses the wrong interface for outgoing UDP (in a specific scenario with our internal phones).

wrong project => drachtio-server

repo: davehorton/drachtio-srf


text:
Vaccine breakthrough and reinfections

Model vaccine breakthroughs.  Model should consider vaccine type, time since vaccination, actor demographics (i.e. age)
The actor should track a history of prior infections
This is related to vaccine waning.
Reinfections occur when an actor is infected with the same or a different variant.

The existing code already has a vaccine efficacy rate. However, cross variant protection is not modeled.

Started update of vaccine efficacy by making it per variant.  Next step will be to have it wane over time.

authors: rstager tedkyi


text:
How do you define and access environment variables?

I need a place to store my backend's host, port, API key, etc.  Issues 451 and 437 suggest adding these to config.globals, but I'm not sure what to do with them from there.  How do I access these values inside one of my modules?  Thanks!

They are exposed as global variables, so if you have a global of, say, API_HOST, you can simply reference it as you would any other variable:
// in some module:
const api = `${API_HOST}:${API_PORT}`

The config.globals object is passed to https://github.com/webpack/docs/wiki/list-of-plugins#defineplugin, if that helps: https://github.com/davezuko/react-redux-starter-kit/blob/master/build/webpack.config.js#L48

Also, just to be pedantic and 100% clear, globals are not actually exposed as global variables on window, for example. Webpack simply looks for references to those globals in your modules and replaces them with the value you defined. This is why dead code removal works, since if you have a globally defined (via webpack) __DEV__ variable (as this repo does), then the following:
if (__DEV__) {
  debug('something')
}

gets transformed to (pretending we're in production):
if (false) {
  debug('something')
}

which allows UglifyJS to completely omit that block since it will always evaluate to false.

Nice, this is a lot simpler than I expected.  Thanks for the thorough explanation- it's a huge help!

Building on this question, I've tried to add global variables that depend on the deployment environment. So I added a variable in each of the _development.js and _production.js files. And also added the global variable. However, it refuses to override depending on the deployment.
_development.js
export default (config) => ({
  ...
  my_api   : 'localhost:8080'
})

_production.js
export default () => ({
  ...
  my_api   : 'www.myapi.com:8080'
})

And then in globals I've added the new global variable trying to read from the config variable.
_base.js
config.globals = {
  ...
  '__MY_API__' : JSON.stringify(config.my_api)
}

But this doesn't seem to work, if I give a default value in _base.js that value will always be the value of the global variable - if I remove the default in _base.js then the global variable is undefined.

Answering my own question by reading #437 again...
.eslinrc
"globals" : {
   ...
    "__MY_API__" : true
  },

_base.js
const config = {
  env : process.env.NODE_ENV || 'development',
  flex_api   : 'api:8080',
  ...
}

config.globals = {
  ...
  '__MY_API__' : JSON.stringify(config.my_api)
}

_production.js
export default () => ({
  ...
  my_api   : 'www.myapi.com:8080'
})

_development.js
export default (config) => ({
  ...
  my_api   : 'localhost:8080'
})

authors: agoransson davezuko dfalling

repo: davezuko/react-redux-starter-kit


text:
about multiple object tracking

Hello, I was wondering whether the code can be changed to track multiple objects per frame? I tried to modify the code to let it accept multiple objects positions input (for each line in the groundtruth.txt I wrote 8, 16 or 24... numbers to input multi-positions), but then I found the network input is just an image cropped by bounding-box(it is just my understanding...), and I can not find out how to change it.... Do you have any ideas about that? @davheld

did anyone find solution to this?

I don't think there's a great way of doing this at the moment.  You can change the network to accept multiple images as input, but then you'd need to retrain it.  I think the best solution at the moment is to track each object in the frame independently; you can change the code to feed the images as a "batch" and output predictions for each image in the batch.  Anything else that you wanted to do would probably be a research project.

repo: davheld/GOTURN


text:
Improve feedback during graph generation

The process of generating the graph can take a while, improve feedback via logging

added progress for converting rdf to ttl


text:
Initial electron app. #npm start

Work towards #278
does nothing clever but gets it running in electron.
The webgl seems to work fine for me.
npm install
npm run build
npm start

closing as this PR is quite old, though electron is still something we should look at. sorry we didn't have a go at this sooner

authors: rumblesan stretchyboy

repo: davidedc/livecodelab


text:
Make links in all content providers external i.e target="_blank"

IT's annoying that jabbr navigates when content providers have links

Fixed

repo: davidfowl/JabbR


text:
Add DX-FORTH for CP/M (a FORTH compiler, rather than interpreter)

There are several FORTH interpreters floating around, but DX-FORTH is an actively maintained (!) FORTH compiler that produces actual runnable binaries.
version 4.30 is the easiest to find on the Internet, but 4.56 is the most recent version released in 2024.
(Note, I'm not certain if you're just collecting these things, or if you're generally focusing on stuff you're already using in your existing performance/unit tests. Please feel free to treat this PR as suits your intentions.)

Thanks for this. It'll be a good test case for my emulator.

authors: davidly majick


text:
hassan you might want to see this?

https://www.youtube.com/watch?v=cSWI4faOktA
some guy has bots called meybots & is using a cookie stealer lol
just saying... you might not be bothered..but thought id share lol

What is the problem with it? Hassan is my friend and Im the owner of meybots

LOL

LMAO
RIP COOKIES

You cant steal cookies if only one person can use one account

@SrNicolas Your bots are not disconnecting, how ?

@hassanila97 there not yours right lol :) just seen it & thought it was cheeky lol

I made new version out of them.. I use base code only as base and have completely refractored the code...  so I can say that I have my own version out of em :D

hassanila97 i tested them there good :) nice work


text:
Design pluggable system for extending available transforms

We'd like to make it possible to extend Amaxa's transform system with simple pluggable modules.
Initial design sketch:

An operation could specify an import path to a callable
That callable is a factory, which accepts some metadata about the transform it's being asked to do (such as the field name and type)
It returns a dict of transform names and callables that Amaxa can then apply.

Genesis of this proposal is discussion between David and Kyle Crouse about the infrastructure needed to support 3rd-party data obfuscation capabilities.
See also #29.

See also #29: we should also support transform functions being parameterized with options that are set in YAML markup on a per-field basis.

I just merged in code that I think will hit this mark. It adds prefix and suffix transforms, as well as allowing for further extensibility in separate modules.

repo: davidmreed/amaxa


text:
Update build procedures. Add rpm and deb package scripts.

Better folder layout, cleans up/re-arranges files/folders in the repo.
Adds build scripts for deb and rpm packages. Packages build, but they're as-of-now untested.
Also adds build script for macOS package.
Cleans up local install and uninstall procedures.

Tested on macos and bionic. Updated wiki as well after reproducing a build on a fresh install of Ubuntu 18.04.

repo: davidnewhall/unifi-poller


text:
Review Joss - Tulio

Hey David,
First of all, I'm happy to see your work and review it. I'm a physical therapist and I really think that Sports2D can be super useful for clinicians.
I was reading and using your colab notebook and everything worked super well with the example file.
I tried to use one old video that I have measured it using others software, but I had a small bug (maybe I did something wrong).
I used the upload button and I the data was fully uploaded but didn't appeared in the folder videos, on purpose I tried a file with spaces i the name ( gait patient 1.mp4 - as a clinician would do). Then I removed the spaces but didn't worked also:

I did something wrong (I have two folders of Sports2D)? Also, it's possible to previously upload this data in Google Drive use the folder path in Sports2D ? Maybe can be nice to receive the path of a folder (maybe the person already have it).
Thank you,
Tulio

Interesting, thanks!
I'll read it more carefully and fix it as soon as possible, but in the meantime you can write ['test.mp4'] instead and it should work.

authors: davidpagnon tuliofalmeida


text:
Possibility of adding user defined language sets

Switching between different languages, it would be useful to have different pre-defined or user-defined language sets.
The default English set would contain information about the text strings "Equation ", Figure ", "Table " whereas a Danish (relevant in my case) would contain the strings "Ligning ", "Figur " and "Tabel ".
The picture below shows an example of how the existing configuration GUI could be enhanced to support additional language sets

Clicking the "+" sign next to "Danish" lets the user add a new language. The default "text"s could simply be a copy the English ones
Clicking the up/down arrows next to "+" lets the user choose between the created languages (only the default  if he hasn't added extra ones)

That would be a great feature!
Thanks

authors: florencioq justnielsen


text:
Add data augumentation

Add random translation and horizontal flipping of images

This has been added some time ago (facenet_train.py --random_crop --random_flip ...). However it does not give any imporvement on the results.

repo: davidsandberg/facenet


text:
head: canonical link only if required

Currently the canonical link is set on EVERY page, which is not required. It makes sense to add canonical only to pages that have multiple URLs (including forwards) that they can be reached at.

non-issue.

repo: davidsneighbour/hugo-modules


text:
Efficiency improvement

2s too slow. Search inside a screenshot of the game board instead.

https://pyautogui.readthedocs.io/en/latest/screenshot.html#the-screenshot-function

repo: davidzhengyes/ChessFENAgain

license: Unlicense


text:
Update libpng to version1.6.37 and zlib to version 1.2.11

There have been several vulnerability fixes in libpng, so I decided to update it, and while at it, also zlib.
I updated the files and cherry-picked all extra commits that fixed warnings (notably on GCC and MSVC).
I pushed every change in a separate commit (libpng, zlib, and each fix) so that it's easy to keep track later on. It certainly helped me a lot to find those changes. So maybe, exceptionally, we shouldn't squash this PR before merging into master.
Please don't hesitate to close if you don't think this is needed.
PS: I'd like to update the Pybind11 at some point too, but that seems a bit more complicated/time-consuming...

Here is a suggestion: how about embed libpng, libjpeg and libz in dlib using the same trick Davis suggested for kissFFT. I.e, rename the .c files to .cpp files and put everything in the dlib namespace then those libraries are no longer dependencies, you can update them freely and won't conflict with user code if they are linking to libpng and co themselves.

Ah, maybe I need to try that, it seems like it's not being able to build on Travis...

@davisking will probs have an opinion on this.
These days, disk space is cheap, so having duplicate libraries or building a few extra C files (well in this case refactored C files) isn't an issue.

Those libraries might have fortran code though....

I had to run ./configure to update the apparently half-baked pnglibconf.h.

Here is a suggestion: how about embed libpng, libjpeg and libz in dlib using the same trick Davis suggested for kissFFT. I.e, rename the .c files to .cpp files and put everything in the dlib namespace then those libraries are no longer dependencies, you can update them freely and won't conflict with user code if they are linking to libpng and co themselves.

Na I really don't want to do that. People should link to the proper system installs of libpng. I've only got these here in external because so many Windows users are like "what's linking/what's a system version of a library?"
The stuff in external is just a fallback for people who don't have a working libpng cmake can find.
And like, I say this specifically for the reason this stuff here needs to be updated. People who use the normal method of linking to the system libpng will always be appropriately updated.
Kissfft is special. It's ultra simple and also doesn't have a stable ABI like libpng. So "go link to the system kissfft" isn't a thing anyone would want or care about.

Like if I had my way dlib wouldn't include copies of these libraries. They always get out of date so people shouldn't be using them. But I added them after like O(a billion) people kept complaining to me that they didn't know How to link to libpng. Or how to install libpng.

Fair enough

Ah, so let's not touch this then. I guess we should not encourage using this.

Ha no this PR is good. These things should be updated periodically for the reasons you mentioned. Like people do use this :)
I'm just saying I wish they didn't. But it's here and it really should be updated every now and then.
The other reason to not change the code and dlib/c++ify it is that doing that makes updating it hard. Right now you should be able to update it by mostly just copying new files over the old ones.

Understood. Yes, that's what I did, I just updated the files.

Understood. Yes, that's what I did, I just updated the files.

Yeah, looks good.  I'll merge when the tests all pass :)   There aren't any other changes you think you need to add first?
And to be clear, my objection to this stuff being in dlib is of the same flavor as saying that hospitals shouldn't exist.  The world would be way better if people just didn't get injured or sick.  But the world is what it is and we need hospitals.  And we need this stuff in external/ too.  No getting away from it.
But It's good that dlib depend on the normal public APIs of libraries like libpng/libjpeg/etc.  This way people who want to (and most people should want to but some just can't for various reasons, some of which are their own fault) can use dlib with the normal versions of these libraries.  Either the system installed ones or their own builds they have in their own environment, whatever that is.  I know plenty of people who would be upset if dlib started to always build in its own copy of libpng.  Since they would then be unable to just update libpng when there are security vulnerabilities and get their whole software system fixed.

Ha, nice analogy. I completely understand it.
Actually, I've always used dlib with the systems' libpng, so this PR is not useful to me. I got somehow curious about libpng and remembered that dlib had one version in external. I checked, and it was quite old, so I decided to update it.

I will admit to using the external ones sometimes :)  Only on windows though where it's a pain to do otherwise though.

Ah, I've never had the pleasure of working with dlib in Windows :sweat_smile:

Ah, I've never had the pleasure of working with dlib on Windows :sweat_smile:

authors: arrufat davisking pfeatherstone

repo: davisking/dlib


text:
Couch bot not giving streamers list

!cb streamer list is not working at the moment .  Why is couch bot offline ? is it having an update at the moment ?

Thanks for notifying me - he crashed. Brought him up after I saw this. All good.

authors: CanadianKittieKat dawgeth

repo: dawgeth/CouchBot


text:
CSS

CSS
User Story
As a boot camp student
I want the prework notes to be structured on a webpage
So that I can easily find and read the information
Acceptance Criteria
GIVEN a Prework Study Guide website
WHEN I view the study guide
THEN I see a dark blue header and footer, and four boxes with a shadow

completed module 3.2.7 on 7/30/24

repo: dawsofd/prework-study-guide


text:
Python 3 version

"Simple" request :).
Thanks!

Hi @jonovate, we are working on that. You could also contribute if you have managed to support python 3.

I'd like to contribute :)
On Tue, 13 Mar 2018, 17:42 Daxeel Soni, notifications@github.com wrote:

Hi @jonovate https://github.com/jonovate, we are working on that. You
could also contribute if you have managed to support python 3.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/daxeel/blockshell/issues/17#issuecomment-372644402,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AP0xxJjawfmR0CDH6ZUQjIN5l0v6LFcNks5td7eQgaJpZM4Sm-ml
.

If you are looking for someone to contribute please tag issue with help
wanted.
On Tue, 13 Mar 2018 at 18:06, Siddharth Prajosh notifications@github.com
wrote:

I'd like to contribute :)
On Tue, 13 Mar 2018, 17:42 Daxeel Soni, notifications@github.com wrote:

Hi @jonovate https://github.com/jonovate, we are working on that. You
could also contribute if you have managed to support python 3.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/daxeel/blockshell/issues/17#issuecomment-372644402,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AP0xxJjawfmR0CDH6ZUQjIN5l0v6LFcNks5td7eQgaJpZM4Sm-ml
.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/daxeel/blockshell/issues/17#issuecomment-372650359,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AJu3KkvSJ3eh1Wh1Eb_VDB8ni402-nX3ks5td70_gaJpZM4Sm-ml
.

@sprajosh You are welcome to contribute for python3 support.
@smitthakkar96 Thanks. This issue is not now tagged.

I would like to contribute

Hello,
We have an exciting opportunity for you! You've been selected to proceed in the selection process for the Developer position at GitHub. Congratulations on your achievement!
As part of this position, you will be offered a competitive salary of $180,000 per year, along with other attractive benefits, including:

Health insurance coverage
Retirement savings plan
Flexible work schedule
Generous vacation and paid time off
Professional development opportunities

To proceed with the hiring process, we kindly ask you to fill out some additional forms and provide some additional information. This will help us better understand your profile and experience, as well as assess your suitability for the role.
Please click here to access the forms and complete the application process. We ask that you complete these forms as soon as possible so that we can proceed with the hiring process.
Important: You have 24 hours to complete the application process.
If you have any questions or need further information, please don't hesitate to contact us.
Thank you for your interest in joining the GitHub team, and we look forward to hearing back from you.
Best regards,
GitHub Recruitment Team
MatejLach, @daltonjorge, @lupengkai, @nunonunes, @karanganesan, @Dub22, @vrajanap, @savekirk, @mjm-buzz, @jackmcpickle, @suriyaa, @llalueza, @ArthurJordao, @luislard, @theCoderCat, @yairmark, @achermannagement, @samuelnair, @thiagosf, @BhargavMahajan

authors: KanhaiyaPandey daxeel elton jonovate smitthakkar96 sprajosh

repo: daxeel/blockshell


text:
feat(NpmBundleSizeBadge): add new NpmBundleSizeBadge

This commit thus adds a new badge which uses Bundlephobia to tell
how many KB (or even MB) you will be adding with a new dependency.
This new badge is accessible as <NpmBundleSizeBadge pkg={pkg} />.
This pull request solves #8.

Thanks! Looks great! :)

@tancredosouza I just realized that I hadn't set up CLAs for this repository yet. If you don't mind, it would be great if you could sign one for your contribution:
https://cla-assistant.io/dbartholomae/jsx-readme
Thanks! :)

No problem! I signed it.

authors: dbartholomae tancredosouza

repo: dbartholomae/jsx-readme


text:
Uncaught TypeError: fs.existsSync is not a function(…)

Hey folks,
thanks for your awesome work. I was looking for something like that for ages.
Works like a charm, when using in command line. Thats where I tested its capabilities.
Now i tried to make it run in my chrome extension using browserify and npm 3.10.8.
There it throws multiple errors all surrounding the fs.exists functions.
Any idea how I could resolve this issue?
Thanks! :)

Sorry, no idea there. Likely that isn't supported in whatever environment you are porting it to? Regardless, it clearly is a node function.
Going to close this out. Conversation can keep taking place here if need be, but it is very unlikely this is a textract issue.

authors: dbashford oliversauter

repo: dbashford/textract


text:
DBeaver 7.3.0 - MySQL 5.7 ROW_COUNT() returns -1

System information:

Operating system (distribution) and version
Ubuntu Linux - 20.04
DBeaver version
7.3.0
Additional extensions
None

Connection specification:

Database name and version
MySQL 5.7.23
Driver name
MySQL 5
Do you use tunnels or proxies (SSH, SOCKS, etc)?
No. Issues replicates on localhost

Describe the problem you're observing:
When using the ROW_COUNT() function after INSERT/UPDATE or SELECT, it always returns -1
This behavior is not seen with MySQL workbench to the same server and database
Connection configuration is as generic as possible. Out-Of-Box experience.
As behaving, DBeaver can not be used for any scripts that require using those values in logic.
Steps to reproduce, if exist:
Use any database table. Script it out to recreate it and edit script to give new table name. Run script to create empty destination table for testing.  Copy data from one table to the other, using the ROW_COUNT() function immediately afterward.
Example script that has the issue on my server:
insert into OLDER_EVENT
select * from OLD_EVENT where EVENT_ID is not NULL ;
select ROW_COUNT();
Results (for my table)

DBeaver: -1
MySQL Workbench: 30   (which is the actual number of rows copied)

Second example:
delete from OLDER_EVENT;
select ROW_COUNT();
Results (for my table)

DBeaver: -1
MySQL Workbench: 30   (which is the actual number of rows deleted)

Include any warning/errors/backtraces from the logs

No errors generated. This is a behavior issue.

Alternate but still undesired behavior: By using variables, the variable is set to "0". Example below:
`select @mycount := 1;
delete from OLDER_EVENT;
select @mycount := ROW_COUNT();
select @mycount;
insert into OLDER_EVENT
select * from OLD_EVENT where EVENT_ID is not NULL ;
select @mycount := ROW_COUNT();
select @mycount;`

Alternate but still undesired behavior: By using variables, the variable is set to "0". Example below:
`select @mycount := 1;
delete from OLDER_EVENT;
select @mycount := ROW_COUNT();
select @mycount;
insert into OLDER_EVENT
select * from OLD_EVENT where EVENT_ID is not NULL ;
select @mycount := ROW_COUNT();
select @mycount;`

Hello @bsherwoodofdaptiv,
I don't think ROW_COUNT function should be used in this case, since DBeaver can display statistics of your executed query, including the total number of updated rows. If you really want to use this function, you can execute your queries as a script, by using Ctrl+E shortcut.
Please see our wiki: https://github.com/dbeaver/dbeaver/wiki/SQL-Execution

Hello @bsherwoodofdaptiv,
I don't think ROW_COUNT function should be used in this case, since DBeaver can display statistics of your executed query, including the total number of updated rows. If you really want to use this function, you can execute your queries as a script, by using Ctrl+E shortcut.
Please see our wiki: https://github.com/dbeaver/dbeaver/wiki/SQL-Execution

authors: ShadelessFox bsherwoodofdaptiv

repo: dbeaver/dbeaver


text:
DBeaver doesn't work with the param "-reuseWorkspace"

System information:

Windows 10
DBeaver 21.0.5

Issue:
DBeaver doesn't work with the param "-reuseWorkspace". When I removed that parameter, Dbeaver open sucessfully.
Steps to reproduce:

Create a windows shortcut with the command: "C:\path\to\install\dbeaver.exe -reuseWorkspace"
DBeaver open the starter dialog and close it in a few seconds.

Remove the param reuseWorkspace from the shortcut: "C:\path\to\install\dbeaver.exe"
DBeaver open normally.

thanks for the report

Fixed (it was broken when we started to use the same parameter for exclusive workspace mode mark)

verified

authors: AdrianoBortoloto serge-rider uslss


text:
TLS 1.0 and TLS 1.1 support

System information:

Operating system (distribution) and version:  Windows 10 20H2 (build 19042.985)
DBeaver version: Community 21.1.0.202105300349
Additional extensions

Connection specification:

Database name and version: SQL Server 2008 10.50.6000.34
Driver name: Microsoft JDBC Driver for SQL Server 8.2.0.jre11
Do you use tunnels or proxies (SSH, SOCKS, etc)? : No

Describe the problem you're observing:
After upgrading DBeaver to the most recent version, the connection to a SQL Server 2008 DB is no longer working. When I try to connect I get the error message

The server selected protocol version TLS10 is not accepted by client preferences [TLS13, TLS12]

It worked fine in the previous minor release, v21.0.1??? something like this from memory. I usually keep DBeaver up to date to the latest version offered. The latest upgrade works just fine for other connections to SQL Server 2016 for example.
Include any warning/errors/backtraces from the logs
!ENTRY org.jkiss.dbeaver.model 4 0 2021-05-31 15:44:02.645
!MESSAGE The server selected protocol version TLS10 is not accepted by client preferences [TLS13, TLS12]
!SUBENTRY 1 org.jkiss.dbeaver.model 4 0 2021-05-31 15:44:02.645
!MESSAGE The server selected protocol version TLS10 is not accepted by client preferences [TLS13, TLS12]
!STACK 0
javax.net.ssl.SSLHandshakeException: The server selected protocol version TLS10 is not accepted by client preferences [TLS13, TLS12]
	at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
	at java.base/sun.security.ssl.Alert.createSSLException(Unknown Source)
	at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
	at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
	at java.base/sun.security.ssl.TransportContext.fatal(Unknown Source)
	at java.base/sun.security.ssl.ServerHello$ServerHelloConsumer.onServerHello(Unknown Source)
	at java.base/sun.security.ssl.ServerHello$ServerHelloConsumer.consume(Unknown Source)
	at java.base/sun.security.ssl.SSLHandshake.consume(Unknown Source)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(Unknown Source)
	at java.base/sun.security.ssl.TransportContext.dispatch(Unknown Source)
	at java.base/sun.security.ssl.SSLTransport.decode(Unknown Source)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(Unknown Source)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(Unknown Source)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
	at com.microsoft.sqlserver.jdbc.TDSChannel.enableSSL(IOBuffer.java:1802)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectHelper(SQLServerConnection.java:2558)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.login(SQLServerConnection.java:2216)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.connectInternal(SQLServerConnection.java:2067)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.connect(SQLServerConnection.java:1204)
	at com.microsoft.sqlserver.jdbc.SQLServerDriver.connect(SQLServerDriver.java:825)
	at org.jkiss.dbeaver.model.impl.jdbc.JDBCDataSource.lambda$0(JDBCDataSource.java:176)
	at org.jkiss.dbeaver.model.impl.jdbc.JDBCDataSource.openConnection(JDBCDataSource.java:195)
	at org.jkiss.dbeaver.model.impl.jdbc.JDBCExecutionContext.connect(JDBCExecutionContext.java:101)
	at org.jkiss.dbeaver.model.impl.jdbc.JDBCRemoteInstance.initializeMainContext(JDBCRemoteInstance.java:100)
	at org.jkiss.dbeaver.model.impl.jdbc.JDBCRemoteInstance.<init>(JDBCRemoteInstance.java:59)
	at org.jkiss.dbeaver.model.impl.jdbc.JDBCDataSource.initializeRemoteInstance(JDBCDataSource.java:109)
	at org.jkiss.dbeaver.model.impl.jdbc.JDBCDataSource.<init>(JDBCDataSource.java:97)
	at org.jkiss.dbeaver.model.impl.jdbc.JDBCDataSource.<init>(JDBCDataSource.java:89)
	at org.jkiss.dbeaver.ext.mssql.model.SQLServerDataSource.<init>(SQLServerDataSource.java:69)
	at org.jkiss.dbeaver.ext.mssql.SQLServerDataSourceProvider.openDataSource(SQLServerDataSourceProvider.java:129)
	at org.jkiss.dbeaver.registry.DataSourceDescriptor.connect(DataSourceDescriptor.java:914)
	at org.jkiss.dbeaver.runtime.jobs.ConnectJob.run(ConnectJob.java:70)
	at org.jkiss.dbeaver.runtime.jobs.ConnectJob.runSync(ConnectJob.java:98)
	at org.jkiss.dbeaver.ui.actions.datasource.DataSourceHandler.connectToDataSource(DataSourceHandler.java:113)
	at org.jkiss.dbeaver.ui.actions.datasource.UIServiceConnectionsImpl.initConnection(UIServiceConnectionsImpl.java:70)
	at org.jkiss.dbeaver.model.navigator.DBNDataSource.initializeNode(DBNDataSource.java:158)
	at org.jkiss.dbeaver.model.navigator.DBNDatabaseNode.getChildren(DBNDatabaseNode.java:217)
	at org.jkiss.dbeaver.model.navigator.DBNDatabaseNode.getChildren(DBNDatabaseNode.java:1)
	at org.jkiss.dbeaver.model.navigator.DBNUtils.getNodeChildrenFiltered(DBNUtils.java:77)
	at org.jkiss.dbeaver.ui.navigator.database.load.TreeLoadService.evaluate(TreeLoadService.java:49)
	at org.jkiss.dbeaver.ui.navigator.database.load.TreeLoadService.evaluate(TreeLoadService.java:1)
	at org.jkiss.dbeaver.ui.LoadingJob.run(LoadingJob.java:88)
	at org.jkiss.dbeaver.ui.LoadingJob.run(LoadingJob.java:72)
	at org.jkiss.dbeaver.model.runtime.AbstractJob.run(AbstractJob.java:105)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)

I have the same issue after upgrading to the latest release

Hello @azmau @sroui,
I believe that's the result of upgrading the version of bundled JRE to 11.0.11 (#12067).
Starting from this version, support for TLS 1.0/1.1 was disabled by default, as mentioned in the roadmap:
https://java.com/en/jre-jdk-cryptoroadmap.html
You can try using previous JRE versions or refer to this article on how to enable support for TLS 1.0 manually:
https://java.com/en/configure_crypto.html#DisableTLS
Let me know if your problem was solved.

Hi @ShadelessFox, Thank you, It worked now.
As you have mentioned,  the bundled JRE does not support TLS 1.0/1.1
To fix the problem in my ubuntu machine,  I have modified the file /usr/share/dbeaver/jre/conf/security/java.security by deleting the TLSv1 and TLSv1.1 from the value of property jdk.tls.disabledAlgorithms

same problem here (windows 10)..solved using the above solution by editing \jre\conf\security\java.security . Thanks

or reinstall DBeaver 21.0.5, free.. Thanks

Same problem with 21.1.0.202105300349 (installed via snap)

Eu também removi os protocolos como o @sroui falou ali acima, e funcionou.
Neste arquivo: "C:\Program Files\DBeaver\jre\conf\security\java.security"

We will update java.security in the bundled JDK in the next version to enable TLS1.0/1.1

Verified

Thanks @ShadelessFox and sorry for the late confirmation but indeed the fix works.
Seems like the same solution still applies to Java 16 (https://www.oracle.com/java/technologies/javase/16-relnotes.html#JDK-8202343), if anyone is interested.

In my case the file was located at /usr/share/dbeaver/jre/conf/security/java.security - removing TLSv1, TLSv1.1, from jdk.tls.disabledAlgorithms worked.

In my case the file was located at /etc/java-openjdk/security/java.security - removing TLSv1, TLSv1.1, from jdk.tls.disabledAlgorithms worked.
Distro OS: Manjaro - Pahvo 21.1.1

In my case the file was located at /etc/java11-openjdk/security/java.security. The fix - removing TLSv1 and TLSv1.1 from jdk.tls.disabledAlgorithms -  worked!
DBeaver version: 21.1.5
OS: Arch Linux 5.13.13

@kafene - Thanks Man. Removing TLSv1, TLSv1.1 from jdk.tls.disabledAlgorithms helped me to fix my issue.

I am using snap version of dbeaver-ce.
In version 21.2.2.202110030714 which is the latest one the TLS1.0 and TLS1.1 are disabled by default.
But in previous version 21.2.1.202109181446 they are not disabled.
The corresponding config file is /snap/dbeaver-ce/147/usr/share/dbeaver-ce/jre/conf/security/java.security but it is readonly, I cannot modify it.

I am using snap version of dbeaver-ce.
In version 21.2.2.202110030714 which is the latest one the TLS1.0 and TLS1.1 are disabled by default. But in previous version 21.2.1.202109181446 they are not disabled.
The corresponding config file is /snap/dbeaver-ce/147/usr/share/dbeaver-ce/jre/conf/security/java.security but it is readonly, I cannot modify it.

Could you try to open the file with "sudo" privileges? This works fine for me.

Not works, it seems snap mounted the "package" as readonly fs.
df -ah
/dev/loop5      249M  249M     0 100% /snap/dbeaver-ce/147

cat /proc/mounts  | grep -i dbeaver
/dev/loop5 /snap/dbeaver-ce/147 squashfs ro,nodev,relatime 0 0

The squashfs is a compressed read-only filesystem for Linux. So I can't edit this file.

In my case the file was located at /etc/java-openjdk/security/java.security - removing TLSv1, TLSv1.1, from jdk.tls.disabledAlgorithms worked.
Distro OS: Manjaro - Pahvo 21.1.1

Saved my life, thanks a lot!

The java.security file might need to be updated again.
For those using flatpak it's located at /var/lib/flatpak/app/io.dbeaver.DBeaverCommunity/current/active/files/jre/conf/security/java.security

I am using snap version of dbeaver-ce.
In version 21.2.2.202110030714 which is the latest one the TLS1.0 and TLS1.1 are disabled by default. But in previous version 21.2.1.202109181446 they are not disabled.
The corresponding config file is /snap/dbeaver-ce/147/usr/share/dbeaver-ce/jre/conf/security/java.security but it is readonly, I cannot modify it.

yes! same experience cannot editing any files in snap version.

authors: Kokolokoli Matvey16 ShadelessFox azmau fabianMendez kafene kred-jon lucastafarelbs moonpole puvi7722 ricgavira rubenflorez serge-rider silvestreR sroui timofas tomboe311


text:
我的DBearver 的解析执行计划按钮是灰色的 无法使用

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.

@onedriver3 please add more info about your issue (dbeaver version, database you use, what's going wrong).
Also, please, write in English.
Thanks in advance.

there is no update on the issue for a long time.
If it is still actual for you - feel free to reopen it with additional info


text:
Support for PolarDB

Description
In China's current environmental impact, Chinese companies are migrating to extend Support for PolarDB
DBeaver Version
24.0
Operating System
W10
Database and driver
No response
Steps to reproduce
No response
Additional context
No response

Let's keep only one issue #23206 , the first one


text:
Double info in General User Guide

FYI

Thank you for the bug report.
Fixed.

authors: Spoelle serge-rider