text:
Raspberry Pi VS Code does not recognize Dart SDK folder

Dart 2.10.5 (Linux ARMv7)
Raspberry Pi OS (32 bit)
VS Code 1.53.2
Flutter is NOT installed.
In .bashrc and .bash_profile I added
export PATH=$PATH:/usr/lib/dart/bin/
so dart is working fine from Terminal.
After installing Dart extension to VS Code it can not see where is Dart (but dart is in the PATH already), so I locate folder manually, but then it says:
That folder does not appear to be a Dart SDK.
The log file:
[9:40:28 PM] [General] [Info] Searching for SDKs...
[9:40:28 PM] [General] [Info] Environment PATH:
[9:40:28 PM] [General] [Info]     /usr/local/sbin
[9:40:28 PM] [General] [Info]     /usr/local/bin
[9:40:28 PM] [General] [Info]     /usr/sbin
[9:40:28 PM] [General] [Info]     /usr/bin
[9:40:28 PM] [General] [Info]     /sbin
[9:40:28 PM] [General] [Info]     /bin
[9:40:28 PM] [General] [Info]     /usr/local/games
[9:40:28 PM] [General] [Info]     /usr/games
[9:40:28 PM] [General] [Info]     /usr/lib/dart/bin/
[9:40:28 PM] [General] [Info] Searching for flutter
[9:40:28 PM] [General] [Info]     Looking for flutter in:
[9:40:28 PM] [General] [Info]         /home/pi/snap/flutter/common/flutter
[9:40:28 PM] [General] [Info]         /home/pi/snap/flutter/common/flutter/bin
[9:40:28 PM] [General] [Info]         /usr/local/sbin
[9:40:28 PM] [General] [Info]         /usr/local/bin
[9:40:28 PM] [General] [Info]         /usr/sbin
[9:40:28 PM] [General] [Info]         /usr/bin
[9:40:28 PM] [General] [Info]         /sbin
[9:40:28 PM] [General] [Info]         /bin
[9:40:28 PM] [General] [Info]         /usr/local/games
[9:40:28 PM] [General] [Info]         /usr/local/games/bin
[9:40:28 PM] [General] [Info]         /usr/games
[9:40:28 PM] [General] [Info]         /usr/games/bin
[9:40:28 PM] [General] [Info]         /usr/lib/dart/bin/
[9:40:28 PM] [General] [Info]     Found at:
[9:40:28 PM] [General] [Info]     Candidate paths to be post-filtered:
[9:40:28 PM] [General] [Info]     Returning SDK path undefined for flutter
[9:40:28 PM] [General] [Info] Searching for dart
[9:40:28 PM] [General] [Info]     Looking for dart in:
[9:40:28 PM] [General] [Info]         /usr/local/sbin
[9:40:28 PM] [General] [Info]         /usr/local/bin
[9:40:28 PM] [General] [Info]         /usr/sbin
[9:40:28 PM] [General] [Info]         /usr/bin
[9:40:28 PM] [General] [Info]         /sbin
[9:40:28 PM] [General] [Info]         /bin
[9:40:28 PM] [General] [Info]         /usr/local/games
[9:40:28 PM] [General] [Info]         /usr/local/games/bin
[9:40:28 PM] [General] [Info]         /usr/games
[9:40:28 PM] [General] [Info]         /usr/games/bin
[9:40:28 PM] [General] [Info]         /usr/lib/dart/bin/
[9:40:28 PM] [General] [Info]     Found at:
[9:40:28 PM] [General] [Info]         /usr/lib/dart/bin/
[9:40:28 PM] [General] [Info]     Candidate paths to be post-filtered:
[9:40:28 PM] [General] [Info]         /usr/lib/dart
[9:40:28 PM] [General] [Info]     Returning SDK path undefined for dart
[9:40:28 PM] [General] [Info] !! PLEASE REVIEW THIS LOG FOR SENSITIVE INFORMATION BEFORE SHARING !!

Dart Code extension: 3.19.2

App: Visual Studio Code
Version: 1.53.2
Platform: linux

Workspace type: Unknown (No SDK)
Analyzer type: DAS
Multi-root?: undefined

Dart SDK:
    Loc: undefined
    Ver: undefined
Flutter SDK:
    Loc: undefined
    Ver: undefined

HTTP_PROXY: undefined
NO_PROXY: undefined
[9:44:36 PM] [General] [Info] Searching for dart
[9:44:36 PM] [General] [Info]     Looking for dart in:
[9:44:36 PM] [General] [Info]         /usr/lib/dart/bin
[9:44:36 PM] [General] [Info]     Found at:
[9:44:36 PM] [General] [Info]         /usr/lib/dart/bin
[9:44:36 PM] [General] [Info]     Candidate paths to be post-filtered:
[9:44:36 PM] [General] [Info]         /usr/lib/dart
[9:44:36 PM] [General] [Info]     Returning SDK path undefined for dart

Restarting, rebooting and reinstalling did not help.

What folder did you use when locating it manually? You should use /usr/lib/dart and not /usr/lib/dart/bin.

Both options do not work. I also tried to locate it through json setting in the VS Code Dart extension like this:
{
    "dart.sdkPath": "/usr/lib/dart/"
}

It also does not work with 'bin' and without 'bin' at the end.
By the way, when I click 'locate manually' it opens '/tmp' folder by default.

cc @DanTup, are you aware of any issues here?

I think this is the same as https://github.com/dart-lang/sdk/issues/41845. The armv7 build is missing the analysis_server snapshot. I'm not familiar with the build process, so I'm not sure where to look to see the reason for that though.

Yes, looks like a duplicate, closing.


text:
evaluate_activation_in_method_class_test type 'DartError' is not a subtype of type 'Instance'

  type 'DartError' is not a subtype of type 'Instance'

https://dart-ci.appspot.com/log/app-kernel-linux-release-x64/app_jitk-linux-release-x64/10684/service_2/evaluate_activation_in_method_class_test/dds
https://dart-ci.appspot.com/log/app-kernel-linux-release-x64/app_jitk-linux-release-x64/10684/service_2/evaluate_activation_in_method_class_test/service
There are new test failures on Bump dart_internal to allow 2.12.0....[ CLI ] Fix issue where analytics were not correctly being disabled.
The tests
service_2/evaluate_activation_in_method_class_test/dds RuntimeError (expected Pass)
service_2/evaluate_activation_in_method_class_test/service RuntimeError (expected Pass)

are failing on configurations
app_jitk-linux-release-x64

cc @bkonyi

/cc @bkonyi

source: gharchive/issue

authors: a-siva dcharkes

repo: dart-lang/sdk

license_type: permissive

license_source: github-api


text:
No CyclicInitializationError in CFE

The following code produces errors in Analyzer and no issues in CFE
f(func) {}

class C {
  static var sVar = f(() => sVar); // analyzer error - The type of 'sVar' can't be inferred because it depends on itself through the cycle: [sVar].  - top_level_cycle

  static final sFinal = f(() => sFinal); // analyzer error - The type of 'sFinal' can't be inferred because it depends on itself through the cycle: [sFinal].  - top_level_cycle
}

main() {
  C? c;
}

@eernstg please confirm that there must be an error in CFE as well
Tested on Dart SDK version: 2.14.0-edge.b9a0c0dfba993e726c36c255470af9868b65a6e5 (be) (Thu May 20 00:50:07 2021 +0000) on "linux_x64"

@stefantsov This sounds related to the top level inference of local functions that we discussed.

please confirm that there must be an error in CFE as well

We need to settle the rules about the treatment of cyclic dependencies during inference, cf. https://github.com/dart-lang/language/issues/1650. So I'll mark this issue as blocked on that language issue.

@stereotype441 @stefantsov This is related to #45029

authors: eernstg johnniwinther sgrekhov


text:
Improve error message for type mismatches between record types

When there's a type mismatch involving records, it'd be nice if the error message could include some more specific details about the diff between the actual and expected record shapes.
For example this is the current error message for a small typo (see if you can spot it 😉)
typedef Server = ({
  String hostName,
  ServerConfig config,
});

typedef ServerConfig = ({
  int terabytesOfDisk,
  int gigabytesOfRAM,
  int cpus,
});

String describeServer(Server server) => '$server';

void main() {
  final small = (
    hostName: 'foo.bar.com',
    config: (
      terabytesOfDisk: 4,
      gigabtyesOfRAM: 16,
      cpus: 8,
    ),
  );
  print(describeServer(small));
  //                   ^^^^^
}

IDE (IntelliJ):

CLI:
$ dart analyze ex1.dart
Analyzing ex1.dart...

  error • ex1.dart:23:24 • The argument type '({({int cpus, int gigabtyesOfRAM, int terabytesOfDisk}) config, String hostName})' can't be assigned to the parameter type '({({int cpus, int gigabytesOfRAM, int terabytesOfDisk})
          config, String hostName})'. • argument_type_not_assignable

1 issue found.

The same typo in the class-version of the code above produces an example of a better error message (though obviously its a different kind of error because classes and records are different):
class Server {
  String hostName;
  ServerConfig config;
  Server({required this.hostName, required this.config});
}

class ServerConfig {
  int cpus;
  int gigabytesOfRAM;
  int terabytesOfDisk;
  ServerConfig({
    required this.cpus,
    required this.gigabytesOfRAM,
    required this.terabytesOfDisk,
  });
}

String describeServer(Server server) => '$server';

void main() {
  final small = Server(
    hostName: 'foo.bar.com',
    config: ServerConfig(
         // ^^^^^^^^^^^^
      cpus: 8,
      gigabytesOfRAM: 16,
      terabtyesOfDisk: 4,
   // ^^^^^^^^^^^^^^^
    )
  );
  print(describeServer(small));
}

$ dart analyze ex2.dart
Analyzing ex2.dart...

  error • ex2.dart:23:13 • The named parameter 'terabytesOfDisk' is required, but there's no corresponding argument. Try adding the required argument. • missing_required_argument
  error • ex2.dart:27:7 • The named parameter 'terabtyesOfDisk' isn't defined. Try correcting the name to an existing named parameter's name, or defining a named parameter with the name 'terabtyesOfDisk'. • undefined_named_parameter

2 issues found.

Improved message changes landed in above mentioned CL.


text:
Dart SDK docker image 3.4 stable unavailable

https://hub.docker.com/_/dart

https://github.com/docker-library/official-images/pull/16769#event-12806918700

authors: itsjustkevin mnordine


text:
[CP]  [dart2wasm] Only share type parameter fields if nullability allows

Commit(s) to merge
e6e9b4e5953dc554c513932c64f57d18ad08cb90
Target
stable
Prepared changelist for beta/stable
https://dart-review.googlesource.com/369241
Issue Description
A bug in the dart2wasm compiler can lead to a type parameter type to have incorrect nullability
What is the fix
Fix sharing of type parameter values with super types if nullability is changed.
Why cherry-pick
External user hit this (https://github.com/dart-lang/sdk/issues/55741)
Risk
Low
Issue link(s)
https://github.com/dart-lang/sdk/issues/55741
Extra Info
No response

Limited impact, has coverage, LGTM

authors: athomas mkustermann


text:
DWX_ClosePartial function works with order without selecting it via OrderSelect

https://github.com/darwinex/dwx-zeromq-connector/blob/f416e3cc569dfe9b3ce59253f3a118308dd0fab7/v2.0.1/mql4/DWX_ZeroMQ_Server_v2.0.1_RC8.mq4#L917
When this function starts - it reads the data from Metatrader about order which is selected by OrderSelect function, like in DWX_ModifyOrder function.
But, as order with ticket we send into this function is not selected - EA start to partial close some other order.
So it should be selected first.

Hi,
thank you for raising this issue. If the ticket sent from python is not zero, it will return false if the order could not be selected:
if(ticket != 0) {
  zmq_ret = zmq_ret + "'_action': 'CLOSE', '_ticket': " + IntegerToString(ticket);

  if (OrderSelect(ticket, SELECT_BY_TICKET)) {
     zmq_ret = zmq_ret + ", '_response': 'CLOSE_PARTIAL'";
  } else {
     zmq_ret = zmq_ret + ", '_response': 'CLOSE_PARTIAL_FAILED'";
     return(false);
  }
}

If the ticket is zero, then it would indeed close the last selected order because the function DWX_ClosePartial() is used in DWX_CloseAtMarket() and before that one is executed the order is selected.
So the only wrong case I can think of is when you submit a partial close from python with ticket=0. We could simple add a check at
case 4: // CLOSE PARTIAL
and only execute if ticket != 0.
Or is there another case where it would lead to a problem?

I made a small update to fix the case that I explained in my last post.

authors: elvinex mashida

repo: darwinex/dwx-zeromq-connector


text:
testing for code?

Which programs shall I add some basic testing too or do some error handling for. I had done some research last week but wasn't sure which programs needed it applied to specifically.

I think error handling is completed.
The code for testing should be made for solving_tdse.py, not sure if tdse_cmd.py needs it.

Indeed, its all handled by warning now, should i close this issue?

damn I had code planned for that. close the issue i guess

license: MIT


text:
map_label_to_target should init zero tensor

Your map_label_to_target for SICK dataset init random tensor.
def map_label_to_target(label,num_classes):
    target = torch.Tensor(1,num_classes) # this is not zero tensor
    ceil = int(math.ceil(label))
    floor = int(math.floor(label))
    if ceil==floor:
        target[0][floor-1] = 1
    else:
        target[0][floor-1] = ceil - label
        target[0][ceil-1] = label - floor
    return target

However, in treelstm , the author init zero tensor
local targets = torch.zeros(batch_size, self.num_classes)
for j = 1, batch_size do
  local sim = dataset.labels[indices[i + j - 1]] * (self.num_classes - 1) + 1
  local ceil, floor = math.ceil(sim), math.floor(sim)
  if ceil == floor then
    targets[{j, floor}] = 1
  else
    targets[{j, floor}] = ceil - sim
    targets[{j, ceil}] = sim - floor
  end

Hi @ttpro1995
Good catch!
I just tried generating some random tensors, and it seems the values are usually quite small, of the order of 1e-37, although occasionally, things might shoot up to very large values, of the order of 1e+22. I believe because the random initialisation mostly resulted in small-ish numbers, I was lucky enough to get some reasonable results.
I have made the change to zero initialisation in the map_label_to_targetfunction, and am running a job now to see the new results. Will push the new changes as soon as the job finishes. Thanks a ton for spotting the bug!
P.S. Great work porting over the sentiment classification part of the paper as well!

Hi @dasguptar
sadly there are another few bugs I have found (((
https://github.com/dasguptar/treelstm.pytorch/blob/master/model.py#L57
there is subtraction of 1 from tree.idx. Though I believe you should use just tree.idx instead.
Also, subsequent runs produce different word2index mapping in vocabulary sine you do not sort words and just use plain set with random hash
https://github.com/dasguptar/treelstm.pytorch/blob/master/utils.py#L49
I suggest to use sorted():
https://github.com/Inilien/treelstm.pytorch/blob/master/utils.py#L48
I fixed these (and maybe others) in my fork here: https://github.com/Inilien/treelstm.pytorch
But I also switched code to Python 3.
After all those fixes I was able to achieve 0.859269 Pearson coefficient

Hi, @ttpro1995 @Inilien ,
I fixed the label target tensor initialisation via commit https://github.com/dasguptar/treelstm.pytorch/commit/66a951b545ddae4dd5922e80658982523b03c8b5
Thanks a lot for pointing it out!

Also, @Inilien , I added commit https://github.com/dasguptar/treelstm.pytorch/commit/ddec16e7968d709ecde4e89cc61b1931193fad4b to fix the indexing bug you found in the model. I was porting from the original Lua code, which is 1 indexed, and probably forgot to account for that!
Regarding the other points:

I am testing out the fix you suggested to reproduce the same word2index mapping every time. I am not sure whether that affects performance or not, but it would be nice to have reproducibility.
Also, could you submit a PR with the fixes needed to port the code to Python3? I believe you had submitted PR #8 , but that was too many things in one go. Just a small PR with the Python3 fixes would be highly appreciated! :smile:

Closing for lack of response, and since most of the issues pointed out here seem to have been fixed!

authors: Inilien dasguptar ttpro1995

license: mit

license_source: bigquery


text:
UnhandledPromiseRejectionWarning: Error: Invalid instant lock proof signature during Identity creation

Using v21 sdk/dashmate I got an error during identity registration:
(node:771073) UnhandledPromiseRejectionWarning: Error: Invalid instant lock proof signature
I sent a L1 tx to the wallet to pay for the credits, the tx had matured 10 mins or so. Creating a new wallet and sending a fresh L1 tx worked.

Thanks @dashameter - issue added to triage sheet.

Could you describe steps to reproduce it?

authors: SamKirby22 dashameter shuplenkov

repo: dashevo/dashmate


text:
Installation path clashes with another application called Dash

The application is installing all its data into /Users/<username>/Library/Application\ Support/Dash under OSX.
Unfortunately there is another application called Dash using the same path, so they ended up using the same folder to install the files.
I guess the application should do a minimum check to avoid clashes and accidental data loss.

Good find, many thanks. We have a similar clash in the Linux world,as there is a package called 'dash' too, which is a Shell application
$ apt-cache search dash
dash - POSIX-compliant shell

Yep, we could move default datadir e.g.
Windows: %APPDATA%\DashCore\
Mac OS: ~/Library/Application Support/DashCore/
Unix/Linux: ~/.dashcore/

This however will require yet another migration for users.
Same for Linux package name - dashcore. We could probably even rename this repo?
Basically we need to come up with some naming convention and maybe it's better to follow the principle that is already declared right there in README.md (and imo it actually makes sense):
Dash is an experimental new digital currency ...
Dash Core is the name of open source software which enables the use of this currency.

DashCore +1

DashCore sounds good, is (more) unique and leaves room for other wallet/daemon implementations (DashThis, DashThat, ...).
+1

DashCore +1

fixed by #905

authors: UdjinM6 andyfreer crowning- pasine schinzelh

repo: dashpay/dash


text:
bug(dashmate): local nodes fails to start after dashmate reset

Expected Behavior

Local nodes should start successfully after reset operation
Current Behavior

Nodes fails to start due to missing environment variables that are not set up during dashmate group reset
Possible Solution

Steps to Reproduce (for bugs)

dashmate setup local
dashmate group reset
dashmate group start

Context

Your Environment

Version used:
Environment name and version (e.g. Chrome 39, node.js 5.4):
Operating System and version (desktop, server, or mobile):
Link to your project:

https://github.com/dashpay/platform/issues/99

The error pops in the every drive_abci container:
/platform/.yarn/cache/dotenv-safe-npm-8.2.0-f1bcdebce9-8b73770330.zip/node_modules/dotenv-safe/index.js:31
            throw new MissingEnvVarsError(allowEmptyValues, options.path || '.env', example, missing, dotenvResult.error);
                  ^
MissingEnvVarsError: The following variables were defined in .env.example but are not present in the environment:
  DPNS_MASTER_PUBLIC_KEY, DPNS_SECOND_PUBLIC_KEY, DASHPAY_MASTER_PUBLIC_KEY, DASHPAY_SECOND_PUBLIC_KEY, FEATURE_FLAGS_MASTER_PUBLIC_KEY, FEATURE_FLAGS_SECOND_PUBLIC_KEY, MASTERNODE_REWARD_SHARES_MASTER_PUBLIC_KEY, MASTERNODE_REWARD_SHARES_SECOND_PUBLIC_KEY
Make sure to add them to .env or directly to the environment.

If you expect any of these variables to be empty, you can use the allowEmptyValues option:
require('dotenv-safe').config({
  allowEmptyValues: true
});
    at Object.config (/platform/.yarn/cache/dotenv-safe-npm-8.2.0-f1bcdebce9-8b73770330.zip/node_modules/dotenv-safe/index.js:31:19)
    at Object.<anonymous> (/platform/packages/js-drive/scripts/abci.js:1:49)
    at Module._compile (node:internal/modules/cjs/loader:1155:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1209:10)
    at Object.require$$0.Module._extensions..js (/platform/.pnp.cjs:11822:33)
    at Module.load (node:internal/modules/cjs/loader:1033:32)
    at Function.require$$0.Module._load (/platform/.pnp.cjs:11645:22)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:169:29)
    at ModuleJob.run (node:internal/modules/esm/module_job:193:25)
    at async Promise.all (index 0) {
  missing: [
    'DPNS_MASTER_PUBLIC_KEY',
    'DPNS_SECOND_PUBLIC_KEY',
    'DASHPAY_MASTER_PUBLIC_KEY',
    'DASHPAY_SECOND_PUBLIC_KEY',
    'FEATURE_FLAGS_MASTER_PUBLIC_KEY',
    'FEATURE_FLAGS_SECOND_PUBLIC_KEY',
    'MASTERNODE_REWARD_SHARES_MASTER_PUBLIC_KEY',
    'MASTERNODE_REWARD_SHARES_SECOND_PUBLIC_KEY'
  ],
  sample: '.env.example',
  example: '.env.example'
}

Fixed in https://github.com/dashpay/platform/pull/991

repo: dashpay/platform


text:
Kick-off doc section about common work-arounds.

This is about https://github.com/dask/dask-jobqueue/issues/356 and also because recently #196 led me to realise there was no nice place where to add a proposed work-around for using job arrays

I am going to merge this one to kick off this common work-arounds page.
@guillaumeeb or @willirath if you happen to have five minutes to look at this page and have some suggestions, this would be more than welcome!

Thanks @lesteve.
I was also wondering were to put the documentation for #122. But I'm not sure this is a "work-around", this sounds more like a "how to that" chapter. Do you think we should add another section, or just rename this new one to something more general?

#122 feels like a work-around to me but I agree work-around is not a great name.
This could be made more general like cookbook or advanced tips and tricks or a better name if you can think of one ...

Yeah, I prefer something like "Advanced tips and tricks" or something like that,
"Advanced pratices" ?
just "Tips and Tricks" ?
"Advanced use cases" ?

Advanced tips and tricks works for me, let me do the renaming.

Renaming done in https://github.com/dask/dask-jobqueue/commit/bb89969d9c6cabcae66c900409872bb29e060e62.

source: gharchive/pull-request

authors: guillaumeeb lesteve

repo: dask/dask-jobqueue


text:
Instalation

Hi, there.
I am trying to install dask-ml on anaconda 3.6, ubuntu 16.04
with:
conda install -c conda-forge dask-ml
I receive the message:
conda install -c conda-forge dask-ml
Solving environment: done
All requested packages already installed.
But after import:
import dask_ml.datasets
from dask_ml.wrappers import ParallelPostFit
The messages are:
/home/libardo/anaconda3/lib/python3.6/site-packages/dask_ml/__init__.py in <module>()
      1 from pkg_resources import get_distribution, DistributionNotFound
      2 try:
----> 3     __version__ = get_distribution(__name__).version
      4 except DistributionNotFound:
      5     # package is not installed

ContextualVersionConflict: (dask 0.13.0 (/home/libardo/anaconda3/lib/python3.6/site-packages), Requirement.parse('dask>=0.14.0'), {'dask-searchcv'})
I would appreciate your help to solve my issue.

BR,
Libardo

ContextualVersionConflict: (dask 0.13.0 (/home/libardo/anaconda3/lib/python3.6/site-packages),
Requirement.parse('dask>=0.14.0'), {'dask-searchcv'})

Updating dask to >0.14 should fix it.
Though I don't understand why it wasn't updated automatically by conda  since dask-searchcv has this requirement. Did you install dask with pip initially?

Thanks @rth. @Libardo1 does explicitly upgrading dask, and then upgrading dask-ml fix your problem?
We should probably set lower bounds in on our dependencies in dask-ml itself, once we figure out what those are :)

OK, Thanks so much.
It is working.

Thaks so much. It is working fine
On Thu, Apr 19, 2018 at 9:01 PM, Tom Augspurger notifications@github.com
wrote:

Thanks @rth https://github.com/rth. @Libardo1
https://github.com/Libardo1 does explicitly upgrading dask, and then
upgrading dask-ml fix your problem?
We should probably set lower bounds in on our dependencies in dask-ml
itself, once we figure out what those are :)
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask-ml/issues/160#issuecomment-382942236, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AGyREezBSuIia6wgDBsXBWNKAdXeVmV9ks5tqUGLgaJpZM4TXJit
.

--
LIBARDO LÓPEZ GUZMÁN
QR vCARD
http://chart.apis.google.com/chart?cht=qr&chs=350x350&chld=M&choe=UTF-8&chl=BEGIN%3AVCARD
N%3ALIBARDO+LOPEZ+GUZMAN
TEL%3A3176397899
EMAIL%3Alilopezg%40gmail.com
ADR%3ABOGOTA+COLOMBIA
NOTE%3ABI%2C+DM%2C+Marketing+Especialist
END%3AVCARD

authors: Libardo1 TomAugspurger rth

repo: dask/dask-ml


text:
Difference between dklearn.Pipeline and sklearn.Pipeline

What's the difference between dklearn.Pipeline and sklearn.Pipeline?  Does dklearn cache the intermediate results?
Reasin I ask -- I'm trying to fit a large number of sklearn Pipelines in parallel, comparing building the computational graph w/ dask with a handrolled (way less sophisticated) joblib implementation.  dask is ~50% slower, and I wonder whether that's a result of the fact that joblib memmap's large objects to reduce communication overhead.  Any thoughts?  Happy to share code if it's useful.

Pipelines are inherently sequential, so on their own they are not very valuable.  They become valuable when combined with other pipelines.  When computing the same pipeline with many parameters you may see some benefits from de-duplication.  This blogpost from @jcrist may be informative here.
I don't know why the dklearn pipeline would be slower in your case.  If possible I recommend trying out both systems with a single thread and then profiling with cProfile, snakeviz, or the prun ipython magic.
dask.set_options(get=dask.async.get_sync)  # use a single threaded scheduler

Sorry for the lack of response here. This library has seen a lot of work over the last couple weeks.

The library is renamed to dask-searchcv and has been released on pypi and (soon) conda-forge
The Pipeline class is completely removed. Only public facing api is GridSearchCV and RandomizedSearchCV.
A lot of work has gone into improving graph building times. Building graphs for 500,000 candidates now completes in seconds (not that fitting this many candidates is a good idea)
In my simple benchmarks this library should at worst be ~equivalent to scikit-learn's implementation. For certain grids/pipelines, we can be a lot faster. For more information, see this blogpost

Please let us know if you try it and find it significantly slower than scikit-learn. There is no reason why we shouldn't be able to at least be equivalent in all cases.

@bkj I'm creating my own parallel pipeline and curious what approach you took.
To me there are two types of pipelines: 1) Large permutation of steps, ie scale, transform then predict or transform, scale then predict.  2) Parameter search within a pipeline.  Ie try transform param 1,2,3,etc.
Case 2 is applies to GridSearchCV while case 1 I'm looking to parallelize.

authors: bkj bscully jcrist mrocklin

repo: dask/dask-searchcv


text:
Azure Functions using dask raise unexpected "AttributeError: 'function' object has no attribute 'fs'" for io operations

Describe the issue:
I'm not sure if this is really a "dask" problem, but thought it was worth raising. Dask appears to be incompatible with azure functions (note that from what I can determine the exact same code will not raise a bug in a normal running on linux environment)
Minimal Complete Verifiable Example:
Appologies, as since this bug is so context dependent - this example may not be particularly helpful.
Testing some azure functions that make use of dask, I found that IO operations seem to fail with an "AttributeError: 'function' object has no attrribute 'fs'" - this error only happens when I'm testing running things in an azure function, and not when running locally.
I haven't tracked down the exact line yet (multiple reads of csvs/json etc all seem to throw the same error) - it seems to be isolated around io operations, although interestingly doesn't include reading parquet (only reading csv, json etc) - I'm guessing because parquet uses a seperate process with the pyarrow engine.
I think this is the simplest code that would throw the error:
(
    dd.read_csv(filepath, storage_options=storage_options)
    .to_parquet(another_filepath, storage_options=storage_options)
)

Anything else we need to know?:
This is only happening on azure functions from what I can observe, and not running locally.
Environment:

Dask version: 2024.3.1
Python version: 3.10
Operating System: Linux / Azure Functions
Install method (conda, pip, source): pip (I think, although I don't really know what azure functions do under the hood for installation)

Can you give us a traceback and the dependencies in your environment?

Just seen this but will do! (I've since pinned back versions so will need to do a little digging to get a full traceback and example)

Been looking into this just now, and can't reproduce the issue, even using the exact same environment (using a lockfile, so pretty sure the dependencies are exactly as they where when I saw the issues).
My best guess is that this didn't ever have anything to do with dask (especially since the same code worked locally with the exact same dependencies and versions) and either:

something has since changed on azure functions
the initial error I saw was with something odd that happened during the building of the functions and changing the dask version fixed the bug only because it retriggered the build and not because of any dask code.

Either way - everything seems to be working fine with the same version so I'll put this down to either me or azure functions doing something janky.

Spoke to soon! Managed to hunt one down (although I think the issue is specific to dask 2024.3 and resolve in the latest 2024.4 so might be fine to close anyway. Here's a full traceback:
/home/site/wwwroot/yeti/azure_function_helpers.py\", line 20, in wrapped_function\n    return azure_function_to_wrap(*args, **kwargs)\n  File \"/home/site/wwwroot/function_app.py\", line 116, in inspect\n    success, message = data_pipeline.inspect_date_range(\n  File \"/home/site/wwwroot/yeti/pipeline.py\", line 217, in inspect_date_range\n    success, detail = self.inspect_date(\n  File \"/home/site/wwwroot/yeti/pipeline.py\", line 178, in inspect_date\n    success, expectations_df = self._inspect_date(query_date)\n  File \"/home/site/wwwroot/yeti/pipeline.py\", line 91, in _inspect_date\n    success, expectations_df = flexpectations.validate(\n  File \"/home/site/wwwroot/yeti/inspect/flexpectations.py\", line 92, in validate\n    df_to_check = datalake.clipped_ge_dataset(filename, storage_options)\n  File \"/home/site/wwwroot/yeti/datalake.py\", line 423, in clipped_ge_dataset\n    return_df_size = len(return_df)\n  File \"/home/site/wwwroot/.python_packages/lib/site-packages/dask_expr/_collection.py\", line 381, in __len__\n    return new_collection(Len(self)).compute()\n  File \"/home/site/wwwroot/.python_packages/lib/site-packages/dask_expr/_collection.py\", line 452, in compute\n    out = out.optimize(fuse=fuse)\n  File \"/home/site/wwwroot/.python_packages/lib/site-packages/dask_expr/_collection.py\", line 488, in optimize\n    return new_collection(self.expr.optimize(fuse=fuse))\n  File \"/home/site/wwwroot/.python_packages/lib/site-packages/dask_expr/_expr.py\", line 93, in optimize\n    return optimize(self, **kwargs)\n  File \"/home/site/wwwroot/.python_packages/lib/site-packages/dask_expr/_expr.py\", line 2877, in optimize\n    return optimize_until(expr, stage)\n  File \"/home/site/wwwroot/.python_packages/lib/site-packages/dask_expr/_expr.py\", line 2828, in optimize_until\n    expr = result.simplify()\n  File \"/home/site/wwwroot/.python_packages/lib/site-packages/dask_expr/_core.py\", line 371, in simplify\n    new = expr.simplify_once(dependents=dependents, simplified={})\n  File \"/home/site/wwwroot/.python_packages/lib/site-packages/dask_expr/_core.py\", line 332, in simplify_once\n    out = child._simplify_up(expr, dependents)\n  File \"/home/site/wwwroot/.python_packages/lib/site-packages/dask_expr/io/parquet.py\", line 622, in _simplify_up\n    _lengths = self._get_lengths()\n  File \"/home/site/wwwroot/.python_packages/lib/site-packages/dask_expr/io/parquet.py\", line 1311, in _get_lengths\n    self._update_length_statistics()\n  File \"/home/site/wwwroot/.python_packages/lib/site-packages/dask_expr/io/parquet.py\", line 1333, in _update_length_statistics\n    stat[\"num-rows\"] for stat in _collect_pq_statistics(self)\n  File \"/home/site/wwwroot/.python_packages/lib/site-packages/dask_expr/io/parquet.py\", line 1554, in _collect_pq_statistics\n    fs = expr._io_func.fs\nAttributeError: 'function' object has no attribute 'fs'\n"

Ok, so I'm pretty confident that this issue isn't present in dask 2024.4 (probably because updating to that also bumps dask-expr from 1.0.5 to 1.0.11) so this can probably be closed

I've just been digging around the codebase for dask-expr to try and figure the source of this out and this section is a little confusing to me and I think the source of the error? (still in the current codebase)
    @property
    def _io_func(self):
        if self._plan["empty"]:
            return identity
        dataset_info = self._dataset_info
        return ParquetFunctionWrapper(
            self.engine,
            dataset_info["fs"],
            dataset_info["base_meta"],
            self.columns,
            dataset_info["index"],
            dataset_info["kwargs"]["dtype_backend"],
            {},  # All kwargs should now be in `common_kwargs`
            self._plan["common_kwargs"],
        )

ParquetFunctionWrapper has a call method so it can get used as if it's a function. But it also has an fs object variable, which gets inspected during _collect_pq_statistics.
That's all good, except that when the self._plan["empty"] is truthy, the _io_func is getting set instead just to functoolz identity function. That's an actualy function, so it doesn't have the fs method, which is what's causing the bug.
I think I was wrong to assume earlier that it was related to azure functions, and was probably because azure functions is somehow shortcutting the turning off of query planning I has.
Will dig some more.

Closing here for now, the other issue is most likely the cuplrit

authors: benrutter phofl

repo: dask/dask


text:
read_sql_table - lower case columns

Is there preference of lower-case or upper-case field names when using df.read_sql_table?
pd.read_sql works fine with upper-case field names but df.read_sql_table does not.
df = pd.read_sql(query, con_oracle, index_col='FIELD')

Works.
df = dd.read_sql_table(table='TABLE', uri=uri, index_col='FIELD', npartitions=8)
In [6]: df = dd.read_sql_table(table='TABLE', uri=uri, index_col='FIELD', npartitions=8)
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-6-22f4a20b882a> in <module>
----> 1 df = dd.read_sql_table(table='TABLE', uri=uri, index_col='FIELD')

~\AppData\Local\Continuum\anaconda3\lib\site-packages\dask\dataframe\io\sql.py in read_sql_table(table, uri, index_col, divisions, npartitions, limits, columns, bytes_per_chunk, head_rows, schema, meta, engine_kwargs, **kwargs)
    108         table = sa.Table(table, m, autoload=True, autoload_with=engine, schema=schema)
    109
--> 110     index = table.columns[index_col] if isinstance(index_col, str) else index_col
    111     if not isinstance(index_col, (str, elements.Label)):
    112         raise ValueError(

~\AppData\Local\Continuum\anaconda3\lib\site-packages\sqlalchemy\util\_collections.py in __getitem__(self, key)
    192
    193     def __getitem__(self, key):
--> 194         return self._data[key]
    195
    196     def __delitem__(self, key):

KeyError: 'FIELD'

Does not work.
df = dd.read_sql_table(table='TABLE', uri=uri, index_col='field', npartitions=8)

Works.

Do you know if sqlalchemy is normalizing the fields somehow?

@raybellwaves are you able to look into this any more? Or are you able to provide a minimal example that uses just sqlite so that someone else is able to take a look?:

@TomAugspurger thanks for the prompt.
I tried to create a MCVE on my windows laptop but in this case I didn't get an issue.
I'll take another look otherwise i'll close.
from sqlalchemy import create_engine
import pandas as pd
import dask.dataframe as dd

uri = 'sqlite:///C:\\sqlitedbs\\database.db'
engine = create_engine(uri)

df = pd.DataFrame({'UUID' : [1, 2, 3], 'NAME' : ['User 1', 'User 2', 'User 3']})
print(df)
   UUID    NAME
0     1  User 1
1     2  User 2
2     3  User 3

df.to_sql('USERS', con=engine)
engine.execute("SELECT * FROM USERS").fetchall()
[(0, 1, 'User 1'), (1, 2, 'User 2'), (2, 3, 'User 3')]

df = pd.read_sql("SELECT * FROM USERS", engine, index_col='UUID')
print(df)
      index    NAME
UUID
1         0  User 1
2         1  User 2
3         2  User 3

df = dd.read_sql_table(table='USERS', uri=uri, index_col='UUID')

authors: TomAugspurger raybellwaves


text:
Provide proper like value for array_safe in percentiles_summary

While toying around with some shuffle operations that reached this fallback in percentiles_summary, I noticed that we're passing a dtype object through as the like argument for this array_safe call, instead of the array-like that's expected - this hasn't caused any errors up until now, but I imagine would cause problems on GPU if we reached this fallback and failed to specify that the constructed array should be on GPU? Will try to find a motivating test case to add

[ ] Closes #xxxx
[ ] Tests added / passed
[ ] Passes pre-commit run --all-files

Seems like in practice, this wouldn't really come up on GPUs? The main case that I think this would account for is if we were shuffling by a string column, but cuDF currently doesn't allow us to grab an underlying array for them:
https://github.com/rapidsai/cudf/blob/d5aad2f42a766821ee7b11c321104d99f02fce2a/python/cudf/cudf/core/column/string.py#L5556-L5560
So the fallback would fail anyways when trying to create the array. That being said, I imagine this would be a blocker for shuffling by string columns if/when that support is eventually added.


text:
Raise error when replace=False in da.choice

In the current implementation of dask.array.choice, a random sample from the entire dask array is sampled for each chunk in the output dask array. These random samples are then merged together to form the returned dask array. This means that when replace=False you won't choose the same sample for a given chunk, but different chunks can sample the same items from the full dask array. For example,
import dask.array as da
import numpy as np

a = da.arange(10, chunks=3)
for replace in [True, False]:
    c = da.random.choice(a, size=a.shape[0], replace=replace, chunks=3)
    print('replace: {}'.format(replace))
    print('c.compute() = {}'.format(c.compute()))

outputs something like
replace: True
c.compute() = [3 3 7 0 4 5 7 9 5 8]
replace: False
c.compute() = [4 0 7 3 2 9 5 1 7 1]

We can see there are repeats when replace=True (as expected), but there are also repeats when replace=False (unexpected).
I'll keep thinking about how to re-work the da.choice implementation to support replace=False. In the meantime, this PR adds a check for replace=False and raises a NotImplementedError accordingly.

[ ] Tests added / passed
[ ] Passes flake8 dask

Am I right in thinking that passing a numpy array or a one-partition dask array works fine with replace?

Right yes, agreed.

authors: jrbourbeau martindurant


text:
How the API is expected to be used

In today's meeting it was discussed what's the goal of the API, and which are its target users.
@maartenbreddels and @devin-petersohn, if I understood correctly, see the API we're defining here, as something they'd like to implement internally in Vaex and Modin, but not making it their public API. Not sure what's pandas point of view on that.
I think that's perfectly fine, and it makes sense. But I have the question of whether would make sense if those public API's would be independent wrappers, in the same way Seaborn wraps Matplotlib, or HoloViews wraps Bokeh. Let me expand on what I mean here.
For the discussions we had, I think people mentioned that they were interested in defining a more "pure" and less "magic" API, than the existing one. Not sure if the previous sentence makes a lot of sense, but I guess some of the principles for the API could be:

Explicit is better than implicit
There should be one-- and preferably only one --obvious way to do it
Avoid ambiguity
In general, avoid making the library having to make guesses

Personally, I think this API should be great for software developers. Like developers of libraries like us, who want to build on top of it. Or developers of downstream software. And I'd say, also to data engineers, and people who want to write production code with dataframes.
Then, I understand that some users (e.g. data analysts) prefer more "magic" API's, that automatically fix problems they don't want to care about. As an example, let's think of the dataframe constructor.
As a data analyst, or non-software people, I think the next code working is very reasonable/convenient:
DataFrame({'a': [1, 2], 'b': [3, 4]})
DataFrame([{'a': 1, 'b': 3}, {'a': 3, 'b': 4}])
DataFrame(json.loads(value))

But as software engineer, I may want to have a more explicit and less magic syntax, for example:
DataFrame.load(kind='dict', {'a': [1, 2], 'b': [3, 4]})
DataFrame.load(kind='list_of_dict', [{'a': 1, 'b': 3}, {'a': 3, 'b': 4}])
DataFrame.load(kind='dict', json.loads(value))

Correct me if I'm wrong, but I think there is mostly agreement that what we want to focus in the consortium API in the latter style. If Vaex, Modin, pandas... provide this API, then there is easy compatibility in the ecosystem. For example, Scikit-learn or Matplotlib can  get a "dataframe" as a parameter, and operate with it, since they know it will follow the standard API.
But then, implementations like Modin, Vaex, or pandas, may want to keep their existing API's. Or provide a different user API, more targeted to specific users (e.g. data analysts, who want the library making guesses, that make their lives easier).
Then my question is, does it make sense that this alternative API live in the implementations? For example, let's consider I see pandas as this API on top of numpy, Vaex on top of memory maps, and Modin on top of Ray (excuse the simplification). Then, if Modin wants to implement an SQLite-like API. Could make sense that this is an independent project, of an SQLite-like API that wraps the standard API? Instead of a Modin API? I guess that could make sense.
Then, I guess there is the case, of an implementation, let's say pandas, which is planning to expose the API to users, but it's going to add some extra magic (let's say that the standard for filter is df.filter(condition) but pandas wants to keeps supporting df[condition] for backward compatibility. Or Vaex having some specific syntax for expressions in top of the standard API.
I see there is a whole range between these options:

All implementations offer exactly the same API
Implementations offer the standard API, but add some functionality to it (for their target users, or specific to the backens)
Backends (e.g. dataframes over numpy, over ray, over memory maps, over Arrow...) implement the same API, but users use libraries built on top of it. For example, the existing pandas API, could be a layer on top of the standard API, and work on top of Vaex, Modin...

Would be great to know other people thoughts. I think most people have an idea on how this API is expected to be used, but not sure if we're all in the same page.

we now have a standardised way of opting into the standard (__dataframe_consortium_standard__), it's up to implementations where the implementation itself lives or what they do with their main API
I suspect that nobody will change their main API, but will just expose a thing wrapped around their main API to comply with the standard
closing then, as I think this is now addressed

authors: MarcoGorelli datapythonista

repo: data-apis/dataframe-api


text:
Return namespace

having to keep typing __dataframe_namespace__ is quite inconvenient, shall we just return it right away (as is done in the array api)?

In my experience it doesn't really matter, when you use this it's anyway a little unusual to call a dunder method directly. I think what you end up doing as writing a single convenience function that does more things (input validation, processing multiple dataframes at once, etc.). E.g.: https://github.com/scipy/scipy/blob/45e875d819c24a3adaa187a1d7050c164d3e937d/scipy/_lib/_array_api.py#L74-L108
This would also necessitate updates to these methods in Pandas/Polars/etc, right? Is it all really worth doing?

This would also necessitate updates to these methods in Pandas/Polars/etc, right?

they just defer to dataframe-api-compat, and are typed to return Any anyway, so I can just handle it in dataframe-api-compat directly
(if I'd had to make changes upstream then I would definitely not have suggested this 😄)

rejected on the call (this isn't actually what the array api does)

authors: MarcoGorelli rgommers


text:
CARDS-1852: Bold fonts in Firefox are not bold enough

Draft PR; to test: build, run, does the page look too heavy?

Nope, nope. I like the 300 default especially in the Questionnaire wizard, please don't tinker with it.

Why we have scrollbars in Chrome and not in FF?

Why we have scrollbars in Chrome and not in FF?

Good question, but it's certainly not because of this PR.
This only changes the default font weight, but it changes it globally. In addition to the MDEditor, the fonts in livetables, materialtables, and questionnaire wizard values will be affected (probably others as well).

authors: marta- sashaandjic sdumitriu

repo: data-team-uhn/cards


text:
CARDS-2146 - Statistics: support multivalued variables; CARDS-2148: Use the displayed value instead of the raw value as labels in Statistics

Can be tested on branch CARDS-2146_CARDS-2148_prems_stats where prems stats are turned on (includes changes from #1390).
Reviewing the code commit by commit is recommended.

QUESTION: In the screenshot below, the red Unspecified relates to the patient who is unsubscribed and will not receive the survey. Currently in CARDS we generate surveys for such patients, however my understanding is that this will not happen in production?
If that is true, I have no complains. The red one should count only historical responses from unsubscribed patients.

QUESTION for @marta- : In the screenshot below, the red Unspecified relates to the patient who is unsubscribed and will not receive the survey. Currently in CARDS we generate surveys for such patients, however my understanding is that this will not happen in production? If that is true, I have no complains. My thinking is that the red one should count only historical responses from the unsubscribed patients.

Surveys will still be generated for unsubscribed patients in production. We only read the "unsubscribed" flag when getting ready to send emails, not at the time of importing the data.
This is a good observation, please create a separate task for it ("prevent the import of visits and the creation of surveys for unsubscribed patients").

authors: marta- sashaandjic


text:
w.dbfs.copy("file:/...","dbfs:/...") is much slower than using REST api with requests library

While this library is easy to use,
I found copying files ranging from 200 kb to 5MB files from local laptop storage to dbfs to be much slower with this library when compared to the requests library or curl.
Curl was about 15.3 seconds for a 5MB file.
This library didn't finish the 5MB file after 20 minutes.

@dmoore247 please provide debug logs and code to reproduce this issue

Following up - is this issue still relevant?

Log wise...
> * Host: https://xxxxxxxxxxxxxxx.cloud.databricks.com
> * User-Agent: unknown/0.0.0 databricks-sdk-py/0.1.8 python/3.8.12 os/darwin auth/pat
> * Accept-Encoding: gzip, deflate
> * Accept: application/json
> * Connection: keep-alive
> * Content-Length: 112
> * Content-Type: application/json
> * Authorization: Bearer dapidtokentokentokentoken
> {
>   "data": "QHByZWZpeCBiZW5jaDogPGh0dHA6Ly9sb2NhbGhvc3Qvdm9jYWJ1bGFyeS9iZW5jaC8+IC4K",
>   "handle": 7925567539485178
> }

What I see is that 100k short blocks (< 300 bytes) are being sent as the upload.

Thank you for the feature request! Currently, the team operates in a limited capacity, carefully prioritizing, and we cannot provide a timeline to implement this feature. Please make a Pull Request if you'd like to see this feature sooner, and we'll guide you through the journey.

repo: databricks/databricks-sdk-py


text:
Adding {withr} to improve tests that involve envvars/files.

As per #68

@RafiKurlansik please review (but do not click merge). 🙏

repo: databrickslabs/brickster


text:
Admin implicit props cannot be modified, class=fvBD, prop=bcastP

cisco@cisco:~/Downloads$ python arya_tenant_json_discovery.py
/home/cisco/.local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)

/home/cisco/.local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:768: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
InsecureRequestWarning)
Traceback (most recent call last):
File "arya_tenant_json_discovery.py", line 53, in
md.commit(c)
File "/usr/local/lib/python2.7/dist-packages/acicobra-1.3_2i-py2.7.egg/cobra/mit/access.py", line 58, in commit
return self._accessImpl.post(configObject)
File "/usr/local/lib/python2.7/dist-packages/acicobra-1.3_2i-py2.7.egg/cobra/internal/rest/accessimpl.py", line 175, in post
return self.__parseError(rsp, CommitError, rsp.status_code)
File "/usr/local/lib/python2.7/dist-packages/acicobra-1.3_2i-py2.7.egg/cobra/internal/rest/accessimpl.py", line 181, in __parseError
parseXMLError(rsp.text, errorClass, httpCode)
File "/usr/local/lib/python2.7/dist-packages/acicobra-1.3_2i-py2.7.egg/cobra/mit/xmlcodec.py", line 15, in parseXMLError
raise errorClass(int(errorCode), errorStr, httpCode)
cobra.mit.request.CommitError: Admin implicit props cannot be modified, class=fvBD, prop=bcastP
cisco@cisco:~/Downloads$

Code and JSON document attached.
Archive.zip

Did you ever figure this out?

Is this not expected behaviour?
You are trying to set the bcastP value of an fvBD (Bridge Domain) object, which is an implicit parameter.
This is the line below from your Python code:
fvBD = cobra.model.fv.BD(fvTenant, multiDstPktAct=u'bd-flood', seg=u'16252846', unicastRoute=u'yes', unkMcastAct=u'flood', descr=u'', llAddr=u'::', vmac=u'not-applicable', bcastP=u'225.1.94.192', mac=u'00:22:BD:F8:19:FF', epMoveDetectMode=u'', ownerTag=u'', ownerKey=u'', name=u'bd2', unkMacUcastAct=u'proxy', arpFlood=u'no', limitIpLearnToSubnets=u'no', mtu=u'inherit', pcTag=u'32770')
This is the Cisco APIC Model Reference (https://your-apic-ip/doc/html/) definition for bcastP of fvBD:
bcastP

Type: mcast:Group
Primitive Type: address:IPv4

Units: null
Encrypted: false
Access: implicit
Category: TopLevelRegular
Comments:
Outer multicast group IP address.

As an implicit parameter it is inherited from another object, in this case fvL2Dom.
When I have created fvBD objects I did not provide a bcastP parameter and it allocated a multicast IP address from the pool configured on my APIC.

authors: NathanDotTo glennake lmalhoit

repo: datacenter/arya


text:
Correction for raw affiliation Archéologie et Philologie d'Orient et d'Occident; Textes, Représentations, Archéologie, Autorité et Mémoires de l'Antiquité à la Renaissance - UR UPJV 4284

Correction needed for raw affiliation Archéologie et Philologie d'Orient et d'Occident; Textes, Représentations, Archéologie, Autorité et Mémoires de l'Antiquité à la Renaissance - UR UPJV 4284
raw_affiliation_name: Archéologie et Philologie d'Orient et d'Occident; Textes, Représentations, Archéologie, Autorité et Mémoires de l'Antiquité à la Renaissance - UR UPJV 4284
new_rors: 05nsvn336;01gyxrk03
previous_rors: 05nsvn336;04wz9m339
works_examples: W4390899785;W4390927881;W4310371949
contact: 354c5ec2e3c4bfbf96dae30b5dbc64c4:8bedc3bd98d4aaa8f3768eb0d3f5b7d2 @ u-picardie.fr

This issue was accepted and ingested by the OpenAlex team on 2024-10-10. The new affiliations should be visible within the next 7 days.

repo: dataesr/openalex-affiliations


text:
Correction for raw affiliation Institut d'Études Politiques [IEP] - Toulouse; Laboratoire des Sciences Sociales du Politique; Université Toulouse Capitole; Université de Toulouse

Correction needed for raw affiliation Institut d'Études Politiques [IEP] - Toulouse; Laboratoire des Sciences Sociales du Politique; Université Toulouse Capitole; Université de Toulouse
raw_affiliation_name: Institut d'Études Politiques [IEP] - Toulouse; Laboratoire des Sciences Sociales du Politique; Université Toulouse Capitole; Université de Toulouse
new_rors: 022scmy89;004raaa70;0443n9e75
previous_rors: 03jdg7625;01b5nw197;004raaa70;0443n9e75
works_examples: W4388520664
contact: 041b0f8c2099ce3f2eed770ed2ab19b7:8fe65be2ed2ffbf9020acd845272b7ce2a7dda4ba1 @ recherche.gouv.fr

This issue was accepted and ingested by the OpenAlex team on 2024-11-18. The new affiliations should be visible within the next 7 days.


text:
Correction for raw affiliation Université de Liège, UMRt BioEcoAgro 1158-INRAE, équipe Métabolites Secondaires d’Origine Microbienne, TERRA Teaching and Research Centre, Gembloux Agro-Bio Tech, B-5030 Gembloux, Belgium

Correction needed for raw affiliation Université de Liège, UMRt BioEcoAgro 1158-INRAE, équipe Métabolites Secondaires d’Origine Microbienne, TERRA Teaching and Research Centre, Gembloux Agro-Bio Tech, B-5030 Gembloux, Belgium
raw_affiliation_name: Université de Liège, UMRt BioEcoAgro 1158-INRAE, équipe Métabolites Secondaires d’Origine Microbienne, TERRA Teaching and Research Centre, Gembloux Agro-Bio Tech, B-5030 Gembloux, Belgium
new_rors: 00bmzhb16;00afp2z80;032v67m28
previous_rors: 00bmzhb16;00afp2z80
works_examples: W4389514011
contact: e1a09d9865e783c1d25c2025659f8e67:5305b6c98a6d47104a39d67cdedc67d7 @ u-picardie.fr

This issue was accepted and ingested by the OpenAlex team on 2024-10-09. The new affiliations should be visible within the next 7 days.


text:
Correction for raw affiliation UMR CBGP (INRA/IRD/Cirad/Montpellier SupAgro), Cirad, Montpellier, France; UR633 Zoologie Forestière, INRA, Orléans, France

Correction needed for raw affiliation UMR CBGP (INRA/IRD/Cirad/Montpellier SupAgro), Cirad, Montpellier, France; UR633 Zoologie Forestière, INRA, Orléans, France
raw_affiliation_name: UMR CBGP (INRA/IRD/Cirad/Montpellier SupAgro), Cirad, Montpellier, France; UR633 Zoologie Forestière, INRA, Orléans, France
new_rors: 05kpkpg04;05q3vnk25;03rnk6m14;05h7ddb14
previous_rors: 05kpkpg04;05q3vnk25;03rnk6m14
works_examples: W2611867704;W2610439647
contact: 30db8438202ac81dce8052ba831db7e5:27069d6cb8dc5d60147d90ff8ccbb5748229 @ inrae.fr

This issue was accepted and ingested by the OpenAlex team on 2024-11-18. The new affiliations should be visible within the next 7 days.


text:
Correction for raw affiliation CRAL, UMR 5574, CNRS, Université de Lyon, ENS, 9 avenue Charles André, F-69561, Saint Genis Laval Cedex, France

Correction needed for raw affiliation CRAL, UMR 5574, CNRS, Université de Lyon, ENS, 9 avenue Charles André, F-69561, Saint Genis Laval Cedex, France
raw_affiliation_name: CRAL, UMR 5574, CNRS, Université de Lyon, ENS, 9 avenue Charles André, F-69561, Saint Genis Laval Cedex, France
new_rors: 029brtt94;02feahw73;0084x3h80
previous_rors: 029brtt94;02feahw73
works_examples: W4388126130
contact: 78b5a776a4cdd9ee60eb759330fb749b:3e905f1915b6592fbc7ee3ea7c0e @ ens-lyon.fr

This issue was accepted and ingested by the OpenAlex team on 2024-12-19. The new affiliations should be visible within the next 7 days.


text:
Support Avro Schema in Flink SQL

Need to support to deal with Avro type of data if Kafka Connect enable
key.converter.schemas.enable=true
key.converter.schemas.enable=true

This issue was moved to datafibers-community/df_data_service#19


text:
fix: Use After Free in PacketReader

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/
Summary
related issue: https://github.com/datafuselabs/opensrv/issues/66
Refactor packet_reader.rs to avoid Use After Free due to dropping Vec while still holding packets of u8 slice from it.

impl NomBytes type which allow nom to parse bytes::Bytes which is smart pointer to array of u8, hence avoid uaf
reuse buffer whenever possible to improve performance
make sure packet still have a lifetime arg, so to prevent breaking change to api
correct both impl of next() and next_async() to prevent UAF or unsafe code
add tests that run to those large packet dealing logics(Which disclosure another bug about having packet size of exactly U24_MAX would also cause nom to refuse parse packets due to lack of opt, fixed)
cargo test is passed

@PsiACE might want to take a look?

The style warning is all in clickhouse/*, I'm not sure if I should fix it in case might break something,
The MacOS CI failure seems to be related to wrong CI setting in Mac env?

Thanks @discord9 . I will check tonight, I am now a community contributor.

repo: datafuselabs/opensrv


text:
Ajout vidéo tutoriel page accueil mode Conférence

La vidéo tutorielle du mode Conférence serait plus pertinente si elle était intégrée dans la page d'accueil du mode Conférence
On pourrait avoir un message en haut ou bas de page "découvrez très rapidement avec cette vidéo comment utiliser le mode conférence"
@laem @martinregner qu'en pensez-vous ?

La vidéo est dans la doc Datagir :)

authors: Benjamin-Boisserie-ABC Clemog

repo: datagir/nosgestesclimat-site


text:
Ajout tracker swipe page de fin

L'idée est d'avoir une idée du nombre de swipe par simulation.
Plusieurs réserves :

Je ne suis pas sur que ce soit un bon indicateur (l'idée est plus de savoir si toutes les pages ont été visité et non combien de swipe l'utilisateurs a fait)
La proposition actuelle est imparfaite (on compte 1 la page initiale sans même avoir changé de page) -> se servir le state tracking ?

Ça me semble intéressant de savoir si l'utilisateur a swipé en complément des URL des pages fin qu'on suit déjà.
Par contre, pourquoi ne pas faire le suivi directement sur les fonctions next et previous ?

Yes carrément, le pb du premier comptage est résolu aussi !

authors: Clemog laem


text:
Petit déjeuner pas de case pour le salé

De plus en plus de personnes mangent salé le matin au petit déjeuner. Je mange ce qui reste dans le frigo ou je me fais des œufs...
Depuis la page : https://nosgestesclimat.fr/simulateur/bilan

Ce ticket a été créé automatiquement par notre robot depuis notre page de contribution.

Merci, c'est pris en compte dans #453

authors: laem publicodes

repo: datagir/nosgestesclimat


text:
Filters: custom filters API enhancements

There are two enhancements to introduce:

[ ] a global event to listen to filter addition (click on the "+" icon in the panel that adds filters), e.g., onFilterAdding, to be able to replace the default filter with the custom one (when the users configure/enable a custom filter, this choice needs to be saved on the subsequent addition of filters for the particular column)
[x] store the name/id of a filter to use in a hidden tag of the column and make the filter panel work with this tag (if present, prioritize the filter specified in the tag)

Wiki: https://datagrok.ai/help/visualize/viewers/filters#column-tags, DG.TAGS.CUSTOM_FILTER_TYPE

To test use SPGI

Right-click the Structure column's header and select Column Properties
Click the Plus icon to add the tag: .ignore-custom-filter = true
Click OK
Go to the Filter Panel and delete the Structure tab.
Drag'n'drop the structure column's header to the Filter Panel to add a filter tab. It should be a categorical filter
Add a tag .custom-filter-type = Chem:substructureFilter
Go to the Filter Panel and delete the Structure tab.
Drag'n'drop the structure column's header to the Filter Panel to add a filter tab. It should be a substructure

filter

authors: ahadzhanian onuf

repo: datagrok-ai/public


text:
Show scatter plot labels above all formula and regression lines

Scatter plot labels are shown below formula lines in case formula line is displayed above markers, and below regression lines:

Expected: labels should be shown above all lines: formula lines (regardless if the line is shown above markers or not), regression lines.

This issue has been mirrored in Jira: https://reddata.atlassian.net/browse/GROK-17050

authors: dnillovna katerina-stepanova


text:
[WIP] feat(cli): Make consistent use of DataHubGraphClientConfig

Deprecates get_url_and_token() in favor of a more complete option: load_graph_config() that returns a full DatahubClientConfig.
This change was then propagated across previous usages of get_url_and_token so that connections to DataHub server from the client respect the full breadth of configuration specified by DatahubClientConfig.
I.e: You can now specify disable_ssl_verification: true in your ~/.datahubenv file so that all cli functions to the server work when ssl certification is disabled.
Checklist

[x] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
[x] Links to related issues (if applicable)
[ ] Tests for the changes have been added/updated (if applicable)
[ ] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
[x] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

This is a GREAT contribution for code quality. @hsheth2 what needs to happen to get this in?

repo: datahub-project/datahub


text:
feat(dbt): support prefer_sql_parser_lineage with sources enabled

Checklist

[ ] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
[ ] Links to related issues (if applicable)
[ ] Tests for the changes have been added/updated (if applicable)
[ ] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
[ ] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@coderabbitai summary


text:
Update files

Added flyway schema for adapter table to grant permission
ZoneDateTime to OffsetDateTime is changed

Can one of the admins verify this patch?

ok to test

repo: datakaveri/iudx-resource-server


text:
FOI: what consumes/requires space in the image - minimize somehow?

ATM I see
❯ docker images
REPOSITORY                             TAG                            IMAGE ID       CREATED         SIZE
datalad-registry                       dev                            0fd3d4b208cf   8 days ago      1.14GB
datalad-registry                       debug                          3bff492fbced   8 days ago      1.14GB

so it is over a GB image. Sounds like too much ...
I have ran it and installed ncdu and debian-goodies to investigate. Here are the heaviest debian pkgs:
root@44d72652f233:/app# dpigs -n 20
73283 git-annex
51125 gcc-10
44132 guile-2.2-libs
35133 python3-pycryptodome
35084 git
33152 libicu67
27869 libperl5.32
27441 g++-10
25613 cpp-10
19944 libpython3.9-dev
17752 libstdc++-10-dev
17478 coreutils
17455 perl-modules-5.32
16014 locales
14502 libc6-dev
13897 binutils-common
13474 libgcc-10-dev
12833 libc6
9631 libasan6
8926 libtsan0

may be we could remove -dev and all the build tooling...
ncdu shows that lots coming from /usr/local/lib/python3.9/dist-packages
--- /usr/local/lib/python3.9/dist-packages -------------------------------------------------------------------------------------------------------
                         /..
   88.6 MiB [##########] /scipy
   60.7 MiB [######    ] /pandas
   37.8 MiB [####      ] /bids
   36.3 MiB [####      ] /numpy.libs
   35.5 MiB [####      ] /numpy
   34.2 MiB [###       ] /scipy.libs
   17.4 MiB [#         ] /sqlalchemy
   13.0 MiB [#         ] /pip
   11.3 MiB [#         ] /pydicom
   11.1 MiB [#         ] /nibabel
   10.1 MiB [#         ] /boto
    9.4 MiB [#         ] /pydantic
    8.1 MiB [          ] /datalad
..

here is what we have in /usr
--- /usr -----------------------------------------------------------------------------------------------------------------------------------------
                         /..
  451.3 MiB [##########] /local
  442.6 MiB [######### ] /lib
  201.4 MiB [####      ] /share
  143.9 MiB [###       ] /bin
   23.4 MiB [          ] /include
    4.2 MiB [          ] /sbin
   16.0 KiB [          ] /libexec
e   0.0   B [          ] /src
e   0.0   B [          ] /games

so just filing for our information (FOI) but not worth probably messing with it ATM

may be we could remove -dev and all the build tooling...

-dev is just the naming of the image. We didn't actually install any packages specified in requirements_.txt or .[dev] in this image. Can packaging the datalad-registry app in to a package help eliminating the need of installing those build tooling in the image?
We can also consider basing our image on one of the official Python images.

may be we could remove -dev and all the build tooling...

-dev is just the naming of the image.

I was talking about -dev debian packages like libpython3.9-dev

I was talking about -dev debian packages like libpython3.9-dev

I was talking about -dev debian packages like libpython3.9-dev

I just checked. Both libpq-dev and python3-dev are specifically required by psycopg2, the DB driver for Postgres.
I think we will have to find other ways to reduce the size of the image.

most likely they are required only during installation so it could be built but then could be uninstalled...  E.g. if we were installing all components as already built and packaged in debian (e.g. python3-psycopg2 etc) and not with pip install - those -dev would not be needed and size would have been smaller.
But as I have mentioned -- that is more of an information for us, we aren't really at demand to reduce the size atm.  I just was surprised a little that it is that large.  Let's close for now

authors: candleindark yarikoptic

repo: datalad/datalad-registry


text:
DlButton - added active prop to button

Added active prop so we can make the button look like it's being hovered over from outside. This allows persisting the hover styles from outside so we can have it look like it's pressed even on leaving the mouse
https://opentechdev.atlassian.net/jira/software/projects/DLCL/boards/17?selectedIssue=DLCL-899

@prodan-eric this needs attention

authors: fadiDL prodan-eric

repo: dataloop-ai/components


text:
ignore _book and _bookdown_files

Optimistic this will finally work this time

My optimism was misplaced! But I may have eventually gotten it? Will try a new PR shortly to confirm.

repo: datalorax/sds-r


text:
🛑 Aadhaar Login is down

In 078adb5, Aadhaar Login (https://ssup.uidai.gov.in/ssup/login.html) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Aadhaar Login is back up in 746d6c6.

repo: datameet/upptime


text:
[TEMP] Fix to the CI - all tests passing

Thought process
Something causes these tests to hang on the GitHub action but not locally. It doesn't seem super consistent as to which tests cause which processes to break to step though to work it out.

[x] Don't run the IPU tests to make sure the install is actually working okay - verify the install
[x] Run the non-IPU tests (w/o dime-net) to make sure those tests are okay.
[x] Add in the low hanging - i.e. non-pop art tests - these should be easy to verify
[x] Add in the other tests one by one

Merging this as it's only skipping a couple of the IPU specific tests - more important to have the CI running smoothly. Will address these in #410

repo: datamol-io/graphium


text:
Create Cassandra CDC Source

http://cassandra.apache.org/doc/latest/operating/cdc.html

I have a question on the design of this source. It really has more to do with the Cassandra CDC implementation, but I think it's relevant here.
How will this source implementation handle the case where a given Cassandra node fails? As I read the CDC document, the SST files are written to a directory on a given node. Are these SST files the same across all nodes? If not, it seems like it'd be impossible to fail over from one node to the other without some special logic to figure out in which SST the source should pick up from. If the two SST directories ARE identical, then we can probably just run Kafka connect in distributed mode on N of the Cassandra nodes--a failure would cause Kafka connect to rebalance, and the connector would pick up where it left off.

@criccomini
Chris we are yet to start the implementation. we are going to find out the answer to your question soon.

Checking in on this.

checking on this.

@AkshathPatkar We have started figuring out the internal mutations written to the logs so it's in progress.

@andrewstevenson 👍

@criccomini
Early feed appreciated http://docs.datamountaineer.com/en/latest/cassandra-cdc.html

Woot! Reading now

@andrewstevenson
Cassandra CDC

messages sent to Kafka are in AVRO format.

Shouldn't the connector just be using SourceRecord, and allow Kafka Connect config to define the serialization format? We use Avro+Confluent Schema Reg.

Once the disk space limit is reached, writes to CDC enabled tables will be rejected until space is freed.

What mechanism is cleaning old legs from the CDC folder? Does Cassandra do this? Or the Connector? Or a CRON job? The docs read like the connector handles deletion after it processes the CDC file. Just want to confirm.

and will push them to the target topic

How does the mapping work between Cassandra tables and Kafka topics?  The docs read like there is just a single topic. We would much rather the mapping be one table->one topic. For one thing, this gives us the ability to enable compatibility checks in Confluent schema registry. It’s also much cleaner, since consumers can pick just the tables (topics) that they care about.

"changeType": //The type of change in Cassandra

The Key should not contain the change type. You shouldn’t have any dynamic data in the key (i.e. fields that change between insert/update/delete). The reason for this is that it is a common pattern to use Kafka log compacted topics with CDC feeds. In order for log compaction to work properly, the keys for a given row must be identical across mutations.

"timestamp" : //the timestamp of when the change was made in Cassandra

Same as above.

"deleted_columns": //which columns have been deleted. We will expand on the details

Same as above.

It is expected that Kafka Connect worker will run on the same node as the Apache Cassandra node.

In distributed mode or standalone?

Upon flushing the memtable to disk, all the commit log segments containing data for CDC-enabled tables are moved to the configured cdc_raw directory

What is the normal frequency with which this occurs? I’m trying to get a feel for how much latency there will be.

The connector is driven by the configurations and not by the list of all the tables with CDC enabled. (Might be a feature, change to do)

+1 Would very much like to have this as a feature.

"connect.cassandra.kcql":"INSERT INTO orders-topic SELECT * FROM datamountaineer.orders",

How would we configure the connector if we wanted to consume more than one topic? For example, if we had two tables: orders, and payments, and we wanted the orders table to go to the orders topic, and the payments table to go to the tables topic? Would this require two instances of the connector, each with different config, or is there a way to configure this for a single Cassandra CDC connector?

The maximum number of Cassandra mutation to buffer. As it reads from the Cassandra CDC files the mutations are buffered before they are handed over to Kafka Connect.

You always flush the buffer before the CDC file is deleted, right?

The worker CDC thread will read a CDC file checking if any of the processed files are ready to be deleted (that means the records have been sent to kafka). Rather than waiting for a read to complete we can delete the files while reading a CDC file. a CDC file. You can disable it for faster faster reads by setting the value to false.

wat? :)

connect.cassandra.cdc.single.instance.port ra.cdc.single.i

This looks like nonsense. Probably a typo?
Other questions:

How does a node failure get handled?
Is there just one connect CDC process running on a single node? Cassandra is sharded, so I’m assuming you have to have the connector running on every node. How does this work in Kafka connect distributed mode? I guess I’m just a bit confused about the topology and deployment relationship between Kafka connect, the connector, and the Cassandra nodes. Based on some hints in the configuration table, it sounds like  you force one task per worker by grabbing a port. What happens when a worker is down for a period of time? Does that extra worker just bounce from worker to worker causing rebalances in Kafka connect?
Want to confirm that there’s no possibility for data loss (even when failure occurs).

Kafka Connect framework doesn’t allow yet configuration where you are running only one task per worker

Is there a KAFKA JIRA for this?

Another question: does the Cassandra CDC connector support snapshotting of existing tables? For example, if I have a table that's existed for a year, and I alter it to enable CDC, then configure the connector to consume it, how do I get the old data?

@criccomini
i will try to clarify some of your statements.

messages sent to Kafka are in AVRO format.
To clarify how connect works.  The connector does create a SourceRecord and sets the value/key schema to a Connect schema. Since the converter is set to AvroConverter  it will end up calling AvroData (see confluent schema-registry repo for its implementation) which converts the Connect Schema to an Avro Schema and Connect Struct to an Avro GenericRecord.  The AvroConverter will work with SchemaRegistry to establish the schema id which is added at the beginning of the binary array which results as part of the serialization. This is how the translation happens. So the connector does all the right thing hence the payload on the topic is AVRO.

What mechanism is cleaning old logs from the CDC folder? Does Cassandra do this? Or the Connector? Or a CRON job? The docs read like the connector handles deletion after it processes the CDC file. Just want to confirm.
I think you missed the documentation How does it work section. "The Connect source will process the files in the order they were created and one by one. This ensures the change sequence is retained. Once the records have been pushed to Kafka the CDC file is deleted."

3The Key should not contain the change type. You shouldn’t have any dynamic data in the key (i.e. fields that change between insert/update/delete). The reason for this is that it is a common pattern to use Kafka log compacted topics with CDC feeds. In order for log compaction to work properly, the keys for a given row must be identical across mutations.
Valid point we will look to change.

What is the normal frequency with which this occurs? I’m trying to get a feel for how much latency there will be. the documentation says it can't be predicted. when the memtable is filled that's when it is flushed. depending on Cassandra settings and volumes it could be days. A user might need to schedule these fluses.

5 +1 Would very much like to have this as a feature. it already does - it's implemented.

How would we configure the connector if we wanted to consume more than one topic? For example, if we had two tables: orders, and payments, and we wanted the orders table to go to the orders topic, and the payments table to go to the tables topic? Would this require two instances of the connector, each with different config, or is there a way to configure this for a single Cassandra CDC connector?  I think the documentation specifies connect.cassandra.kcq : ..  It describes which CDC tables are monitored and the target Kafka topic for Cassandra CDC information. we will make it clearer one can say  INSERT INTO payments SELECT * FROM datamountaineer.payments; INSERT INTO orders SELECT * FROM datamountaineer.orders

You always flush the buffer before the CDC file is deleted, right? yes otherwise we can lose data

"wat? :)"  -> a file is read one by one and the buffer is filled. there is a dedicated thread that does that. Meanwhile Connect thread will asks for records from this buffer. It could happen as the a CDC file is read the records for the previous file to be consumed and thus the prev CDC file is eligible to be deleted. this option allows to check while a file is read if the previous file/-s can be deleted. this check for files to be deleted happens anyway but not while a CDC file is read.

This looks like nonsense. Probably a typo? yes. should be connect.cassandra.cdc.single.instance.port

How does a node failure get handled? if its connect failure we retain the file and mutation location thus on restart it is picked up from where it left. if Cassandra node goes down that connect node will consume whatever is left there and then there won't be much to do since no more CDC files will be spit out on that machine. If the machine fails totally the CDC logs contain the data even on replica nodes ( Cassandra CDC documentation)

Is there just one connect CDC process running on a single node? Cassandra is sharded, so I’m assuming you have to have the connector running on every node. How does this work in Kafka connect distributed mode?...   yes the documentation should be clear enough stating there has to be a 1-1 relationship between Cassandra process and Connect worker process. The reason for the port grabbing is to allow one instance of task per worker really active. We can't have two threads reading the CDC files. If a machine goes down Connect will rebalance and 2 tasks will be allocated on one worker. Then only 1 can be active!

Want to confirm that there’s no possibility for data loss (even when failure occurs). That's what we worked out based on the implementation we have so far. however more testing is required. remember the version we have is BETA!!

How does the connector handle the fact that some Cassandra nodes have data that's a replica from another node (or nodes)? Does it send a ton of duplicate rows, or does it de-duplicate such that each row is only sent once?  Connect doesn't deduplicate it can't by its nature. you don't want two Connect tasks on different machines to start synchronising.  Unfortunately Cassandra having a masterless architecture and the implementation choosing this file based logging (rather than say a TCP listener) the dedup needs to happen at consumption. But processing the records sent on the topic eventual consistency is achieved.

Please remember the implementation so far is only BETA. We would like early adopters in DEV to give it a try and help with testing and any potential problems

If the memtable has been flushed to the disk and CDC wasn't enable you won't get those commits. That's what i understand. but that's a question for the Apache Cassandra community more than us.

If the memtable has been flushed to the disk and CDC wasn't enable you won't get those commits. That's what i understand. but that's a question for the Apache Cassandra community more than us.

This is a pretty big deal. Basically, it precludes anyone from using this connector if they are already running Cassandra. In addition, it means you'll have to t
To be frank, supporting a snapshot feature when tables are first added is a pretty big priority. This is something that Debezium added early on, and without it, it's going to severely limit the usefulness of the connector.

the dedup needs to happen at consumption

This is what I was afraid of. Do you have a hint on how to do this? Do we use the timestamp, and highest timestamp wins?

As far as snapshots go, perhaps this can be leveraged?
http://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsAboutSnapshots.html

We can add a read all table upfront and then start the connector. but to enable the cdc you need to restart your cassandra cluster. you can do it node by node to avoid downtime but that means the snapshot is not accurate. We might build that tool as well but it still down to the user to decide if this is required. This shouldn't be enforced.
I fully understand the concerns but this is still work in progress and we appreciate early feedback hence we made it public.
But there are a few things to bare in mind :

it's open source hence people can contribute.
it might require some work from the Apache Cassandra community (if there are limitations and not just gaps in our understanding)
development is still in progress

dedup: you have the key for the cassandra record and the mutation timestamp. you can use the timestamp. But if you process let them to be processed you will still end up with the right value after all records have been processed. this avoid the check for 'have-i-processed-this-entry'

We can add a read all table upfront and then start the connector.

+1 This is what I was getting at. This is how Debezium does it. One thing to consider is how the read all works vs how the CDC works. In Debezium land (on MySQL), there's a SELECT * that runs (similar to MySQL dump), then it flips to the MySQL binlog at exactly the point that the SELECT * ran. Given that Cassandra is distributed and there's no de-dupe, I anticipate that the dump/CDC flip will be more complicated, but it's worth thinking through a bit.

but to enable the cdc you need to restart your cassandra cluster.

Understood. I don't think that this is something you guys should solve for. Make your lives easier, and ops folks can determine the best way to make this change.
Totally get that this is open source, and beta. Don't get me wrong, I'm stoked that you guys are doing this. My main motivation is in making sure you guys are aware of all these edge cases, and what's going to be required to really use this thing in production. We've already gone through this dance with the Debezium community on the MySQL side of things, so I'm trying to apply learnings from that to this problem.

we appreciate the feedback so far

NP.
It's also still unclear to me how your current solution is going to be have when a node is down for an extended period of time. IIRC, you are using a port-binding trick to force one task per node. But if you have 10 nodes and 10 tasks, and then a node dies, the task on that node will be rebalanced to a new node, right? This will go on until the dead node comes back. This could go on for quite some time, and the rebalancing will just keep happening because there are now 9 nodes and 10 tasks. This can be really disruptive since a rebalance in Kafka connect currently (at least as of 0.10) rebalances ALL tasks every time there's a rebalance.
This is vs. running the connector in standalone mode, one per node. When a node goes down, both the Cassandra node and the connector on that box are then down. No rebalance occurs. When the node comes back, both Cassandra and the standalone connector are started up again.

' the task on that node will be rebalanced to a new node, right? This will go on until the dead node comes back' -> that is not true. the rebalance will complete without requiring the node which died to be back up.
We chose the distributed version to be able to monitor it as one connector. think of a 100nodes cluster. you will have 100 connectors.
on the dedup: if the target can store and reading the ts we are good thus having the streaming dedup is quite an overhead. plus as i said earlier some might choose to consume the change data since eventually is consistent.

that is not true. the rebalance will complete without requiring the node which died to be back up.

Can you describe how that will happen? I think I'm lacking some understanding in this area. What I described above w.r.t. the 10 task/9 node problem seems to apply.

Also, regarding my "how frequently do the CDC files get shifted", there's an interesting patch floating in the Cassandra JIRA slated for a 4.x release:
https://issues.apache.org/jira/browse/CASSANDRA-12148

Connect is using the Kafka consumer group semantics for the workers. Hence if a worker has died the remaining 9 will rebalance and distribute the tasks amongst themselves. if 9 workers are running you will see one of the workers spinning up 2 tasks.
http://docs.confluent.io/current/connect/concepts.html should give you more details

if 9 workers are running you will see one of the workers spinning up 2 tasks.

Yes, but if one worker has two tasks, the first task will "grab the port". The second task on the node will fail to "grab the port" because it's already bound to the first task, right? I'm assuming this causes the second task to fail, which would trigger a rebalance, wouldn't it?

Let me clarify that part. If it fails we capture the exception and we don't initialise the cdc logic. that task sits there basically doing nothing. Thus no new rebalance is triggered

Ah ha. That's the piece I was missing. In that case, though, I don't think Kafka connect is smart about balancing tasks across its nodes. If we have 10 nodes and 10 tasks, what stops KC from assigning 3 tasks to node 0 and no tasks to nodes 1 or 2. We see this happen all the time with our production Kafka connect cluster--things can be imbalanced. If the tasks that fail to bind don't get rebalanced, then some Cassandra nodes won't have any tasks at all on them, right?

Connect round robins the tasks onto workers,  i know there was some talk of changing this. So if we have 10 workers with one on each Cassandra node and 10 tasks we should be ok. We should be able to test this once we get our integration tests going.
From: Chris Riccomini
Sent: Thursday, 3 August, 19:02
Subject: Re: [datamountaineer/stream-reactor] Create Cassandra CDC Source (#162)
To: datamountaineer/stream-reactor
Cc: Andrew Stevenson, Mention
Ah ha. That's the piece I was missing. In that case, though, I don't think Kafka connect is smart about balancing tasks across its nodes. If we have 10 nodes and 10 tasks, what stops KC from assigning 3 tasks to node 0 and no tasks to nodes 1 or 2. We see this happen all the time with our production Kafka connect cluster--things can be imbalanced. If the tasks that fail to bind don't get rebalanced, then some Cassandra nodes won't have any tasks at all on them, right?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/datamountaineer/stream-reactor/issues/162#issuecomment-320030050, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJwvZReucQkV_wRB_kLDkKp4IDxd1Rcoks5sUf0VgaJpZM4M-BUL.

Sounds good.

Another thought: Debezium's value payload has both a "before" and "after" to show the diff of what changed. Not sure if the CDC files expose that (I kind of doubt it), but if so, it'd be useful to expose both the before and after fields.

I am affraid debezium leverage the product and its capabilities. Unfortunately Cassandra CDC doesn't provide that. But someone consuming ( there will be a consumer) can work out  the prev value.

Ya, that's what I figured. Just wanted to confirm. Consumers can always maintain state if they want to know the previous value.

Adding for reference @criccomini blog
https://medium.com/@criccomini/so-you-want-to-build-a-kafka-connector-source-edition-357dce7d55f4

@criccomini We'll get in the 0.3.1 release, we are busy with http:://kafka-lenses.io and 0.3.0  at the moment.

closing as won't be implemented

authors: AkshathPatkar Antwnis andrewstevenson criccomini stheppi

repo: datamountaineer/stream-reactor


text:
[PRED-3419] Remove redundant check for encoding

BATCH SCORING PULL REQUEST
Remove encoding check
RATIONALE
We using opening file in text mode with encoding later - so this check redundant
CHANGES
TESTING

Coverage decreased (-0.3%) to 83.886% when pulling 75321454672443a880f961c9acbb79f548b619ee on andriy-popovych/PRED-3419 into b8a67863ec67d18e1d98408e727b0138b8ea5a7b on master.

Coverage decreased (-0.3%) to 83.886% when pulling 75321454672443a880f961c9acbb79f548b619ee on andriy-popovych/PRED-3419 into b8a67863ec67d18e1d98408e727b0138b8ea5a7b on master.

authors: coveralls falkerson

repo: datarobot/batch-scoring


text:
export-gemini fails using Python 3.9.0

When trying to run an export of gemini formated schema, the run fails when using python 3.
You can see the problem here:
jeffdinoto@jdinoto-rmbp16 adelphi % pyenv local 3.9.0
jeffdinoto@jdinoto-rmbp16 adelphi % pip install .
Processing /Users/jeffdinoto/github/public/datastax/adelphi/python/adelphi
Collecting cassandra-driver~=3.24.0
  Using cached cassandra-driver-3.24.0.tar.gz (283 kB)
Requirement already satisfied: click~=7.1.0 in /Users/jeffdinoto/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from adelphi==0.1.0) (7.1.2)
Requirement already satisfied: PyGithub~=1.45 in /Users/jeffdinoto/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from adelphi==0.1.0) (1.53)
Requirement already satisfied: six>=1.9 in /Users/jeffdinoto/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from cassandra-driver~=3.24.0->adelphi==0.1.0) (1.15.0)
Requirement already satisfied: geomet<0.3,>=0.1 in /Users/jeffdinoto/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from cassandra-driver~=3.24.0->adelphi==0.1.0) (0.2.1.post1)
Requirement already satisfied: pyjwt in /Users/jeffdinoto/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from PyGithub~=1.45->adelphi==0.1.0) (1.7.1)
Requirement already satisfied: requests>=2.14.0 in /Users/jeffdinoto/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from PyGithub~=1.45->adelphi==0.1.0) (2.25.0)
Requirement already satisfied: deprecated in /Users/jeffdinoto/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from PyGithub~=1.45->adelphi==0.1.0) (1.2.10)
Requirement already satisfied: idna<3,>=2.5 in /Users/jeffdinoto/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from requests>=2.14.0->PyGithub~=1.45->adelphi==0.1.0) (2.10)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/jeffdinoto/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from requests>=2.14.0->PyGithub~=1.45->adelphi==0.1.0) (1.26.2)
Requirement already satisfied: chardet<4,>=3.0.2 in /Users/jeffdinoto/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from requests>=2.14.0->PyGithub~=1.45->adelphi==0.1.0) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /Users/jeffdinoto/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from requests>=2.14.0->PyGithub~=1.45->adelphi==0.1.0) (2020.11.8)
Requirement already satisfied: wrapt<2,>=1.10 in /Users/jeffdinoto/.pyenv/versions/3.9.0/lib/python3.9/site-packages (from deprecated->PyGithub~=1.45->adelphi==0.1.0) (1.12.1)
Using legacy 'setup.py install' for adelphi, since package 'wheel' is not installed.
Using legacy 'setup.py install' for cassandra-driver, since package 'wheel' is not installed.
Installing collected packages: cassandra-driver, adelphi
    Running setup.py install for cassandra-driver ... /
done
    Running setup.py install for adelphi ... done
Successfully installed adelphi-0.1.0 cassandra-driver-3.24.0
WARNING: You are using pip version 20.2.4; however, version 20.3.3 is available.
You should consider upgrading via the '/Users/jeffdinoto/.pyenv/versions/3.9.0/bin/python -m pip install --upgrade pip' command.
jeffdinoto@jdinoto-rmbp16 adelphi % python -V
Python 3.9.0
jeffdinoto@jdinoto-rmbp16 adelphi % adelphi --hosts=localhost --username=cassandra --password=cassandra --anonymize export-gemini
WARNING:cassandra.cluster:Downgrading core protocol version from 66 to 65 for 127.0.0.1:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version
WARNING:cassandra.cluster:Downgrading core protocol version from 65 to 4 for 127.0.0.1:9042. To avoid this, it is best practice to explicitly set Cluster(protocol_version) to the version supported by your cluster. http://datastax.github.io/python-driver/api/cassandra/cluster.html#cassandra.cluster.Cluster.protocol_version
INFO:cassandra.policies:Using datacenter 'datacenter1' for DCAwareRoundRobinPolicy (via host '127.0.0.1:9042'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes
INFO:cassandra.cluster:Cassandra host ::1:9042 removed
INFO:adelphi:Excluding system keyspaces system_auth,system_schema,system_distributed,system,system_traces
INFO:adelphi:Processing the following keyspaces: datastax_iot_demo
Traceback (most recent call last):
  File "/Users/jeffdinoto/.pyenv/versions/3.9.0/bin/adelphi", line 269, in <module>
    export(obj={}, auto_envvar_prefix="ADELPHI")
  File "/Users/jeffdinoto/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/jeffdinoto/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/jeffdinoto/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/jeffdinoto/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/jeffdinoto/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/jeffdinoto/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/decorators.py", line 21, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/jeffdinoto/.pyenv/versions/3.9.0/bin/adelphi", line 231, in export_gemini
    export_keyspaces(ctx.obj, export_gemini_schema, keyspace_map, metadata)
  File "/Users/jeffdinoto/.pyenv/versions/3.9.0/bin/adelphi", line 200, in export_keyspaces
    print(export_fn(keyspace_map.keys(), metadata, options))
  File "/Users/jeffdinoto/.pyenv/versions/3.9.0/lib/python3.9/site-packages/adelphi/gemini.py", line 30, in export_gemini_schema
    keyspace = keyspace_objs[0]
TypeError: 'dict_keys' object is not subscriptable

The problem does no occur when running with python 2.7.18

A quick test on master looks like this is only a problem on the branch being used for the #71 fixes

repo: datastax/adelphi


text:
[Improvement] The REST API of List metalakes should sort by created time as default

What would you like to be improved?
The Rest API of List metalakes should sort by created time as defaultTo achieve this default sorting behavior, the API endpoint for listing metalakes can be designed to automatically sort the results based on the "created time" field in ascending order.
This way, users accessing the API will receive a sorted list by default, simplifying their experience and ensuring consistency in how the data is presented.
By setting this as the standard behavior, users can rely on the API to consistently deliver results in a predictable and organized manner.
How should we improve?
Modify the return results of the List metalakes API.

Why not sort alphabetically by default?

Why not sort alphabetically by default?

Sorting by alphabetical order is also fine, because the current list looks disorganized.

@mchades @ch3yne is anyone working on this? can i pick up?

Shall we do it on the server side or on the UI side, what is your opinion @mchades @ch3yne ?

Shall we do it on the server side or on the UI side, what is your opinion @mchades @ch3yne ?

Currently, it seems that only the UI side has a sorting requirement, so I suggest doing it only on the UI side.

Shall we do it on the server side or on the UI side, what is your opinion @mchades @ch3yne ?

Currently, it seems that only the UI side has a sorting requirement, so I suggest doing it only on the UI side.

When loading with large amounts of data and complex sorting logic, it's generally recommended to have the backend return sorted results.
However, since we currently don't have a large volume of data, the frontend can handle sorting when initially loading the list.
I think the REST APIs need to provide flexible sorting parameters to accommodate different use cases.

@ch3yne so can this be picked up? if yes can you point me to the rest api classes ?

@ch3yne Currently, it is a heavy work for the backend to do the sorting and paging, we don't consider this in our API design, it requires the API redesign to support this. So I would suggest to do this in the frontend for now if urgent. For the backend, we should rethink the whole API design to see how to well support this.

I think we can start improve by the front end.

Sorting by ascending order would put new items at the bottom of the list, which would be inconvenient for a user if there's a long list.

Sorting by ascending order would put new items at the bottom of the list, which would be inconvenient for a user if there's a long list.

I agree to sort by time in descending order, and if the list is too long, we can filter by name or time range.

repo: datastrato/gravitino


text:
[FEAT] Render ERD with the actual fully qualified tablename

Is your feature request related to a problem? Please describe.
Currently the ERD generator will output the ERD with the name of the node within the dbt project, i.e. model.project.name. It would be nice if there was an option to render the ERD with the actual table name itself like database.schema.table or just schema.table instead of the name of the dbt node. Not everyone who views an ERD is involved with dbt development so the node names can throw people off.
Describe the solution you'd like
The ability to be able to output an ERD with the actual tablename instead of the dbt node name.

Thanks @dylan-murray for another good request 👍
I will work it this weekend, and initially the solution will looks like this:
dbterd run --entity-name-format resource,package,model # default
dbterd run --entity-name-format database,schema,table # with fqn of the physical tables
dbterd run --entity-name-format schema,table # with schema.table only
dbterd run --entity-name-format table # with table name only

Let me know what you think.
Thanks

This looks great to me! Flexible for every use case. Thanks for tackling my ideas! 😄

It should be done in dbterd 1.7.0 now!

authors: datnguye dylan-murray

repo: datnguye/dbterd


text:
Make merkle-tree-stream generic over hash types

Right now a hash is always a buffer. This is not ideal because before comparing we need to cast it back to the original Hash type to ensure that we have constant-time comparisons (which is an essential security feature).
Therefore I propose we make this module a generic over a Hash type so we can get away with fewer conversions. Thanks!

Maybe we should even make the Node types generic. We can provide a default implementation, but it's super useful to be able to extend them with custom traits.

This has been done!

repo: datrs/merkle-tree-stream