text:
CI Infra Docs

PR adds Infrastructure documentation including info on gpuCI .
This PR may be reformatted as there is an ongoing effort to reorganize the Docs in https://github.com/dask/dask/pull/7984
cc @jrbourbeau

Thanks for the feedback @jrbourbeau -- I'll update with another commit shortly addressing the rest of your thoughts

@jrbourbeau do you have preferences in showing the gputester message ?

Said this here, but it might be nice to have a condensed version of this (particularly the comment system) in a PR template on both repos; okay with handling that in follow up PRs depending on what direction we go with gpuCI

If you don't mind, I'd prefer to hold off on merging this until https://github.com/dask/distributed/issues/5154 is resolved

@jrbourbeau do you have preferences in showing the gputester message ?

Neat, I personally like the image version. Do you have a preference?

Said this here, but it might be nice to have a condensed version of this info (particularly how the comments work) in a PR template on both repos;

I'd prefer to point to our developer guide instead. That will minimize the number of places we need to update when something related to our development process changes.

If you don't mind, I'd prefer to hold off on merging this until dask/distributed#5154 is resolved

Yup, no rush.   I think I like the image better

I have a preference, can the bot message be: "Can one of the admins verify this patch for GPU testing?"
If we don't include "for GPU testing" then new contributors are likely to still get confused. I don't think it's enough to expect people to look at the bot name and put two and two together, that's too subtle.
Thanks for doing all of this @quasiben

I have a preference, can the bot message be: "Can one of the admins verify this patch for GPU testing?"

I'm +1 on this change as well and I'll ask OPs about our options

Thanks for doing all of this @quasiben

Thanks should go to @charlesbluca and @raydouglass for connecting all of this up

Was there anything else needed here?

Was there anything else needed here?

@jrbourbeau suggested we wait for a resolution on dask/distributed#5154 before merging this; that being said, I'm not sure how long the current solution (ops changing the bot message) will take, so I'm in favor of merging this now and then updating the docs later on as gpuCI changes.

Thanks all! 😄

authors: GenevieveBuckley charlesbluca jakirkham jrbourbeau quasiben


text:
Unable to use RVC, torchaudio not found

Good day,
I've tried to set up and use your webui and have been unable to use RVC. It seems that the script to install RVC on Linux is incorrect (should be using / path separators or os.path.join to build path). I changed this locally and the installation fails due to this error.
ERROR: Could not find a version that satisfies the requirement torchaudio==0.13.1+cu117 (from versions: 2.0.0+cu117, 2.0.1+cu117, 2.0.2+cu117)
ERROR: No matching distribution found for torchaudio==0.13.1+cu117

Hi, I was planning to update the lib and upgrade torch dependencies to 2.1.x soon

Thanks for letting me know, I look forward to it and trying this out.

source: gharchive/issue

license: MIT

license_type: permissive

license_source: github-api


text:
x-axis is cropped if y-axis is hidden

http://localhost:3300/profile/040AF00217#CropsBySupply
(probably a d3plus-plot fix)

requires an rm -r node_modules/ && npm i to work

repo: data-africa/data-africa-site

license: mit

license_source: bigquery


text:
[CDAP-19970] Fixing null pointer error in prepareRun

https://cdap.atlassian.net/browse/CDAP-19970

LGTM, Please squash commits before merge.

source: gharchive/pull-request

authors: itsankit-google saimukkamala

repo: data-integrations/dlp-plugins


text:
Allow to use without required prefix

Some use cases will need this for legacy compatibility

Problem is that there is no fixed prefix for sys entries

Can not use a fixed prefix, it varies and needs legacy compatibility😟

Setting an empty string disables prefix enforcement.

And entries written as --sys can not be overwritten by user?

They shouldn't be possible to overwrite. However no test coverage for that scenario without the prefix.
Having multiple values may cause confusion:
$ nvram --sys --set TESTSYS 123
$ nvram --set TESTSYS 456
$ nvram --list
TESTSYS=123
TESTSYS=456
$ nvram --get TESTSYS
123

authors: hcl-dr ninjafinne

repo: data-respons-solutions/nvram


text:
[Question] Can I delete just one tag that references an image with multiple tags?

When two tags point to an image at the same time, does deleting one affect the other?
For example: v1.1, abcedf -> sha256:66666.....
I just want to remove abcdef and don't want to affect v1.1, is this possible?

Thank you! Your work is excellent, I have successfully deleted thousands of images today!

Thanks!  That's great to hear.

authors: GZTimeWalker rohanmars

repo: dataaxiom/ghcr-cleanup-action


text:
Feat: Support show columns from table SQL parser.

What is changed and how it works?
ref: #61 .
What's Changed:
Related changes

Support show columns from table SQL parser.

LGTM

repo: database-mesh/pisanix


text:
Support for Key-Value Pair Query Results, Streamed Data Access

Currently, the npm databend-driver has the following limitations that impact usability and efficiency:

Array of Arrays for Results: Query results are returned as arrays of arrays containing only values. To fetch field names (keys) alongside these values, the same query must be executed again using a different function. This approach is inefficient and cumbersome.
Variant Types in String Format: Variant types are returned in string format instead of actual objects or arrays. This requires explicit parsing for each record, adding unnecessary overhead and complexity.
Lack of Streaming Support: The absence of streaming options makes handling large datasets inefficient.
Async Query Execution: There is no support for asynchronous query execution (e.g., async/await), which is a common pattern in modern database drivers like those for MySQL and Snowflake.

Ex:- const results = await conn.exec('SELECT * FROM test;');
results = [{ id: 1 }, { id: 2 }]
Proposed Features:

Key-Value Pair Rows: An option to fetch query results as objects with field names (keys) and corresponding values (e.g., JSON format).
Proper Handling of Variant Types: Return variant types as actual objects or arrays instead of strings to eliminate the need for explicit parsing.
Streaming Support: Introduce support for streaming query results to efficiently handle large datasets.
Async Query Execution: Add methods for executing queries using async/await to align with modern asynchronous programming patterns.

These features would significantly improve the usability, performance, and developer experience of the databend-driver. Thanks!

Thank you for your detailed feedback on the npm databend-driver. We are pleased to inform you that the mentioned limitations have been addressed:

Array of Arrays for Results: You can now use row.data() instead of row.values() to fetch query results with field names.
Variant Types in String Format: Support has been added for returning variant types as actual objects or arrays by setting the parameter { variantAsObject: true } for row or client.
Lack of Streaming Support: Our queryIter() method already returns rows as a streaming result.
Async Query Execution: All methods in our Node.js client already support async/await.

We hope these improvements enhance your experience with the databend-driver. If you have any further questions or feedback, please feel free to share.
Thank you again for your valuable input!

@everpcpc Thanks for your support. We encountered an issue with the output of the record. For example, when we call queryIter or any method to execute a query, the output should ideally be in its respective data type, correct? However, for bigint, the output is returned as '7n' instead of 7 (even though it’s not a very large value). Using the MySQL handler, it provides the correct result. Does the output remain consistent across types—for instance, varchar as a string, and number and boolean in their proper formats? Am I correct?

@Shanmugavel-J
Thank you for your feedback. I'd like to clarify the data type mappings between Databend and JavaScript native types. You can find a full list of these mappings in our documentation here: Databend Driver Type Mapping.
Regarding your specific concern, '7n' is indeed the correct output for bigint in JavaScript. In JavaScript, '7n' represents a BigInt, which is a primitive type. For more details on BigInt, you can refer to the MDN documentation: BigInt in JavaScript.
To answer your query about consistency across types:

varchar is returned as a string.
number and boolean are returned in their proper formats.
bigint is returned as a BigInt (e.g., '7n').

This ensures that the output remains consistent and adheres to JavaScript's handling of different data types. If you have any further questions, feel free to ask!

@everpcpc Thanks! I indeed knew it but since the mysql handler port gave in a different way which made me ask it.

@Shanmugavel-J Oh, I see. The type mapping depends on how the driver converts the response from the database. We recommend using the native Databend driver for better support. The MySQL handler is intended for transitional use and is just a compatible implementation. It has not been fully tested, so there may be some corner cases that are not covered.

@everpcpc Yes, I noticed the features have been released. I’m now updating to the databend-driver package. Instead of an iterator for streams, I had requested a Node.js Readable stream, similar to what Snowflake and mysql provides.

@Shanmugavel-J  I get it. We will add support for Node.js Readable stream in upcoming releases, making data streaming operations more flexible and convenient.

@everpcpc That sounds great please let me know once that feature is added for fetching the rows in stream format.
Thanks again for your support!!

@Shanmugavel-J Hi, we have released a new version with Readable stream support. You could refer to the examples: https://www.npmjs.com/package/databend-driver?activeTab=readme#usage

@everpcpc I will check and get back to you. Thanks!

@everpcpc Please find my below observations,

queryIter Method

Doc:
// get rows of map
const rows = await conn.queryIter("SELECT * FROM test");
let row = await rows.next();
while (row) {
  console.log(row.data());
  row = await rows.next();
}

Actually when i execute this. I get rows.next() is not an function. I do get only a single row rather it should ideally provide me all the rows.
TypeError: te.next is not a function

queryIterExt Method

When i execute the above method i could see its comment saying the rows, schema & stats could be fetched but i don't get the rows as output when i execute this method but i get the stats (execution time etc). but Ideally it should provide me with the rows, schema & the stats am I correct?

Stream It works as expected. I can use the nodejs stream in any way right other than that mentioned in the doc ?.

One observation was like, I couldn't see any types for stream. Could you please update it so that it would be great to see what that function would return?
example code:
const rows = await conn.queryIter("SELECT * FROM my_table1;");
const stream = rows.stream();
stream.on('data', (data) => {
  console.log(data.data());
});
stream.on('end', async () => {
  console.log('end');
});
stream.on('error', (err) => {
  console.log(err);
});

Object (Variant Data) one I will try and let you know. Thanks!

cc @youngsofun please check this error:

thread '' panicked at core/src/client.rs:894:13:
there is no reactor running, must be called from the context of a Tokio 1.x runtime
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
thread '' panicked at core/src/panicking.rs:221:5:
panic in a function that cannot unwind

Hi, @Shanmugavel-J

We've tested this driver with Node versions 18, 20, and 22, but not with 16 since it is deprecated. However, it might still work with version 16 as it is a native plugin, so feel free to try it out and report any bugs.
queryIterExt is not intended for customer use, but it is okay to use it. This function returns either a row or statistics with each fetch.
rows.stream() returns a common implementation for node:stream:Readable with Row as the item in objectMode.
This error will be fixed in the next release.

@everpcpc Thanks! The variantObject option works as well.

authors: Shanmugavel-J everpcpc

repo: databendlabs/bendsql


text:
Merge staging ingestion into main

This is a follow-up to:

https://github.com/databricks/databricks-sql-python/pull/67
https://github.com/databricks/databricks-sql-python/pull/64

It merges the staging ingestion feature into main so we can release it to Pypi.
We need to re-lock our dependencies because the poetry.lock file is different between these branches. To do this:

Merge main back into si (taking the version of poetry.lock from main). At this point, the lock file and pyproject.toml disagree with one another and will raise an error.
rm poetry.lock
poetry install which regenerates a new lockfile from pyproject.toml

e2e tests pass

repo: databricks/databricks-sql-python


text:
Add setting of UC to the project template

Description
Adds setting of UC to the project template:

When using UC:

$ dbt init
...
[1] use Unity Catalog
[2] not use Unity Catalog
Desired unity catalog option (enter a number): 1
catalog (initial catalog): cat
...

When not using UC:

$ dbt init
...
[1] use Unity Catalog
[2] not use Unity Catalog
Desired unity catalog option (enter a number): 2
...

Thanks! merging.

repo: databricks/dbt-databricks


text:
DataFrame.reindex supports named columns index

This PR would close #1872.

Codecov Report

Merging #1876 into master will decrease coverage by 2.72%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1876      +/-   ##
==========================================
- Coverage   94.18%   91.45%   -2.73%
==========================================
  Files          40       40
  Lines        9853     9762      -91
==========================================
- Hits         9280     8928     -352
- Misses        573      834     +261

Impacted Files
Coverage Δ

databricks/koalas/frame.py
93.58% <100.00%> (-3.15%)
:arrow_down:

databricks/koalas/usage_logging/__init__.py
25.66% <0.00%> (-66.65%)
:arrow_down:

databricks/koalas/usage_logging/usage_logger.py
47.82% <0.00%> (-52.18%)
:arrow_down:

databricks/koalas/__init__.py
71.66% <0.00%> (-18.34%)
:arrow_down:

databricks/conftest.py
89.83% <0.00%> (-10.17%)
:arrow_down:

databricks/koalas/spark/functions.py
88.88% <0.00%> (-7.41%)
:arrow_down:

databricks/koalas/accessors.py
86.13% <0.00%> (-6.94%)
:arrow_down:

databricks/koalas/generic.py
90.25% <0.00%> (-5.11%)
:arrow_down:

databricks/koalas/namespace.py
78.08% <0.00%> (-4.93%)
:arrow_down:

... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5a5c468...dcef750. Read the comment docs.

LGTM.

Thank you for raising the issue and the solution!

Thanks! merging.

Nice !

authors: LucasG0 codecov-io itholic ueshin xinrong-databricks

repo: databricks/koalas


text:
[WIP] Cleanups to groupy

This simplifies somewhat the calls to GroupBy.

Will take this over

authors: HyukjinKwon thunterdb


text:
packaging of project

Hey, since for example many teams within organizations does not have access to public github, will u package this repo in order to initialize project structure and everything

Hi @goldstein0101 this project can now be initialized with the Databricks CLI, so this should get around the public Github aspect!

repo: databricks/mlops-stacks


text:
Changes for 0.3.0 release

Updates build and readme files only.

Current coverage is 88.25%

Merging #57 into master will not affect coverage as of 5150b6f

@@            master     #57   diff @@
======================================
  Files           12      12
  Stmts          613     613
  Branches        82      82
  Methods          0       0
======================================
  Hit            541     541
  Partial          0       0
  Missed          72      72

Review entire Coverage Diff as of 5150b6f

Powered by Codecov. Updated on successful CI builds.

Merged.

authors: HyukjinKwon codecov-io

repo: databricks/spark-xml


text:
Add pubsub as an option for server

Resolves https://github.com/datacontract/cli/issues/58

Seems to solve our needs for using pubsub

authors: adaminsta jochenchrist

repo: datacontract/datacontract-specification


text:
Correction for raw affiliation EA7516 CHIMERE, Picardie Jules Verne University, Amiens, France; Laboratory of Biochemistry, University Hospital, Amiens, France

Correction needed for raw affiliation EA7516 CHIMERE, Picardie Jules Verne University, Amiens, France; Laboratory of Biochemistry, University Hospital, Amiens, France
raw_affiliation_name: EA7516 CHIMERE, Picardie Jules Verne University, Amiens, France; Laboratory of Biochemistry, University Hospital, Amiens, France
new_rors: 01gyxrk03;010567a58
previous_rors: 01gyxrk03;01thw2g46
works_examples: W3043245156
contact: e1f540515d5e1c8913f2da925073e663:1329ef0a314adb9ae489a66afa21db69 @ u-picardie.fr

This issue was accepted and ingested by the OpenAlex team on 2024-10-10. The new affiliations should be visible within the next 7 days.

repo: dataesr/openalex-affiliations


text:
Correction for raw affiliation CP2M - Catalyse, Polymérisation, Procédés et Matériaux (CPE LYON
Bâtiment 308F
43 boulevard du 11 Novembre 1918
69616 VILLEURBANNE CEDEX - France)

Correction needed for raw affiliation CP2M - Catalyse, Polymérisation, Procédés et Matériaux (CPE LYON
Bâtiment 308F
43 boulevard du 11 Novembre 1918
69616 VILLEURBANNE CEDEX - France)
raw_affiliation_name: CP2M - Catalyse, Polymérisation, Procédés et Matériaux (CPE LYON
Bâtiment 308F
43 boulevard du 11 Novembre 1918
69616 VILLEURBANNE CEDEX - France)
new_rors: 02f6tst70
previous_rors: 01cbtr271;034tcbt19
works_examples: W4324123608;W4366216181;W4377825005;W4290861317;W4385408652
contact: aed2218eaf90a50bc594ffeb9538ba37:b303d09a5df53a4912c61d5eeb143e @ recherche.gouv.fr

This issue was accepted and ingested by the OpenAlex team on 2024-10-10. The new affiliations should be visible within the next 7 days.


text:
Correction for raw affiliation LIP (UMR 5668 CNRS ENS Lyon UCBL INRIA), ENS Lyon, Université de Lyon, France

Correction needed for raw affiliation LIP (UMR 5668 CNRS ENS Lyon UCBL INRIA), ENS Lyon, Université de Lyon, France
raw_affiliation_name: LIP (UMR 5668 CNRS ENS Lyon UCBL INRIA), ENS Lyon, Université de Lyon, France
new_rors: 04zmssz18;029brtt94;02feahw73;04msnz457
previous_rors: 04zmssz18;029brtt94;02feahw73
works_examples: W2152414370;W107018433;W2173116115
contact: 9181ef488f825f84f4f8773752bfe8ac:2825856eb432506894ce5f193610 @ ens-lyon.fr

This issue was accepted and ingested by the OpenAlex team on 2024-12-19. The new affiliations should be visible within the next 7 days.


text:
Expand --cloud output

The --dbt --cloud output is extremely basic at the moment:
Running with data-diff=0.5.0

INTEGRATION.BEERS_test_custom_schema.simple_example <> DEV.DBT_DEV_DAN_test_custom_schema.simple_example
    Diff in progress:
    https://app.datafold.com/datadiffs/<diff_id>/overview

INTEGRATION.BEERS.simple_example_downstream <> DEV.DBT_DEV_DAN.simple_example_downstream
    Diff in progress:
    https://app.datafold.com/datadiffs/<diff_id>/overview

Diffs Complete!

Describe the solution you'd like
Add functionality to poll /api/v1/datadiffs/<diff_id>/summary_results and also display high level results in the command line

@williebsweet feel free to change/add anything

I'm addressing https://github.com/datafold/data-diff/issues/387 alongside this issue

@dlawin Is there a PR for this?

authors: dlawin williebsweet

repo: datafold/data-diff


text:
Chore: bump deps to latest version

Summary:

[x] bump deps to latest version
[x] upgrade rust toolchain to nightly-2024-07-02 that same as databend.

Checklist

[x] Updated guide with pertinent info (may not always apply).
[x] Squash down commits to one or two logical commits which clearly describe the work you've done.
[x] Unittest is a friend:)

This change is

This unnecessarily allocates the vector. Even though it's just an example, I'd suggest to do it properly. No idea what the type of bs is, so no suggestion.

Fixed.

Are these changes correct? Doesn't look correct to me...

That's correct, the original docs were:
[`getting_started`](crate::docs::getting_started)

rust complain:
error: redundant explicit link target
 --> openraft/src/docs/docs.md:4:23
  |
4 | - [`getting_started`](crate::docs::getting_started).
  |    -----------------  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ explicit target is redundant
  |    |
  |    because label contains path that resolves to same destination
  |
  = note: when a link's destination is not specified,
          the label is used to resolve intra-doc links
  = note: `-D rustdoc::redundant-explicit-links` implied by `-D warnings`
  = help: to override `-D warnings` add `#[allow(rustdoc::redundant_explicit_links)]`
help: remove explicit link target
  |
4 | - [`getting_started`].
  |   ~~~~~~~~~~~~~~~~~~~

after removing these redundant links, it will say:
error: unresolved link to `getting_started`
  --> openraft/src/lib_readme.md:1:1
   |
1  | /
2  | |
3  | |

4  | |
Openraft

...  |
   |
   = note: the link appears in this line:

           - [`getting_started`].
              ^^^^^^^^^^^^^^^^^
   = note: no item named `getting_started` in scope
   = help: to escape `[` and `]` characters, add '\' before them like `\[` or `\]`
   = note: `-D rustdoc::broken-intra-doc-links` implied by `-D warnings`
   = help: to override `-D warnings` add `#[allow(rustdoc::broken_intra_doc_links)]`

so we need to escape it.

Why not just use #[derive(Debug)]

the trait derive_more::Debug is not implemented for fixtures::RPCRequest<openraft_memstore::TypeConfig>, which is required by derive_more::TryIntoError<fixtures::RPCRequest<openraft_memstore::TypeConfig>>: derive_more::Debug

repo: datafuselabs/openraft


text:
Two formula lines with same formula but different min/max cannot be displayed

v1.22.2
Steps to reproduce:

Add a scatter plot to the table
Configure two formula lines with the same formula but different min/max

Expected results: both lines should be displayed.
Actual results: only one line is displayed:

This issue has been mirrored in Jira: https://reddata.atlassian.net/browse/GROK-17269

authors: dnillovna katerina-stepanova

repo: datagrok-ai/public


text:
Search page should not search by connecting words (at least incide the README), like:  in, at, on, a, the, ...

Reported here: datahq/datahub-qa#69
We want to improve our search, so it will not return search by connecting words (at least inside the README), like:  in, at, on, a, the, ...
Acceptance criteria

[x] it should found 3 datasets with 'Mauna Loa' in the readme.
[x] it should not found 109 datasets with 'the' in the readme

Tasks
Analysis
How to reproduce:
Go to '\search' page and search 'the Mauna Loa'
Output
'the Mauna Loa' request found 109 datasets (should be 3)

FIXED/DUPLICATE of https://github.com/datahq/metastore/issues/28

authors: Mikanebu zelima

repo: datahq/frontend


text:
feat(ui) Adding support for deleting Tags and Domains via the UI

Summary
In this PR, we introduce the capability of deleting Domains and Tags in the UI. To delete either, you must have a Manage X Platform privilege, or a Delete Metadata privilege.
Note that this represents a full hard delete, including dangling references that exist in the graph.
With this PR, deletes will be supported for the primary Metadata Entities on DataHub: tags, terms, domains, groups, users, assertions, secrets, ingestion sources.
Changes

Refactoring EntityDropdown to generalize for wider use (beyond Entity Profile)
Implement UI for deleting tags from Profile
Implement UI for deleting terms from Profile + List domains
Misc refactoring and improvements (e.g. in GraphQL layer to use EntityClient instead of EntityService), check for exists

Screenshots

Checklist

[ ] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
[ ] Links to related issues (if applicable)
[ ] Tests for the changes have been added/updated (if applicable)
[ ] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
[ ] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

Hi John,
"Note that this represents a full hard delete, including dangling references that exist in the graph."
does this mean that if a dataset has the tag, the tag would also disappear from the dataset profile page?

repo: datahub-project/datahub


text:
feat(ingest/elastic): reduce number of calls made

Checklist

[ ] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
[ ] Links to related issues (if applicable)
[ ] Tests for the changes have been added/updated (if applicable)
[ ] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
[ ] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

Merged as failure is related to download failure and ingestion tests passed.


text:
feat(kafka): meta-mapping for labels from Avro schemas on Kafka ingest

Integrating this old PR: https://github.com/datahub-project/datahub/pull/7381/files as per current code.
Since Avro permits non-spec field/value pairs, a common pattern is to use this to embed business metadata. Being able to map this directly to DataHub's Owners, Tags, and Terms allows metadata to be kept closer to source, avoiding the need for Data Producers to configure transformers.
Mapping an Avro list-of-strings to DataHub Tags
Consider the following Avro schema:
{
  "name": "sampleRecord",
  "type": "record",
  "tags": ["tag1", "tag2"],
  "fields": [{
    "name": "field_1",
    "type": "string",
    "tags": ["tag3", "tag4"]
  }]
}

The following new config allows this to be ingested as DataHub tags:
config:
  schema_tags_field: tags
  tag_prefix: kafka

These config values are currently the default (so they wouldn't actually be needed).
Maybe it would make more sense for the default value of schema_tags_field to be something like _datahub_tags?
Field-level tags do not automatically become Dataset-level tags.
Meta-mapping implementation
I've re-used the meta-mapping implementation that already exists for dbt (with a small extension to support referencing nested fields).
Consider the following Avro schema:
{
  "name": "sampleRecord",
  "type": "record",
  "owning_team": "@Data-Science",
  "data_tier": "Bronze",
  "fields": [{
    "name": "field_1",
    "type": "string",
    "gdpr": {
      "pii": true
    }
  }]
}

This can now be mapped to DataHub metadata with the following config:
config:
  meta_mapping:
    owning_team:
      match: "^@(.*)"
      operation: "add_owner"
      config:
        owner_type: group
    data_tier:
      match: "Bronze|Silver|Gold"
      operation: "add_term"
      config:
        term: "{{ $match }}"
  field_meta_mapping:
    gdpr.pii:
      match: true
      operation: "add_tag"
      config:
        tag: "pii"

Note the ability to reference nested fields.
Implementation
The implementation is mostly just re-using the OperationProcessor created for the dbt source. Otherwise:
ConfluentSchemaRegistry(KafkaSchemaRegistryBase) to support returned multiple Aspects. The original method for backward compatibility.
Checklist

[ ] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
[ ] Links to related issues (if applicable)
[ ] Tests for the changes have been added/updated (if applicable)
[ ] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
[ ] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@Deepankarkr looks like CI is failing - please take a look

Closing this PR. @mayurinehate


text:
Program individually queries every GitHub repository, leading to problems

When looking for DataLad datasets on GitHub, most of the information about the datasets found is returned from search requests, which list multiple repositories per page.  However, for various reasons, the program still ends up making a separate GitHub API request for each & every dataset, both those found by searching and those previously recorded but not found in the current run.

Search results for some reason do not include the number of stars each repository has (nor the timestamps needed by #40), so, for each repository found via searching, we need to make a separate request to get its details and extract the number of stars.

If a repository was previously recorded in datalad-repos.json but was not returned in the current search results, we make a request for that repository's information in order to check whether it has been either deleted or else replaced by a new repository with the same name but a different ID.

Note that, although datalad-repos.json currently contains 5250 known datasets, searching for datasets on GitHub only returns 438 results.  I believe this is because (as stated in the docs), "Only repositories that have had activity or have been returned in search results in the last year are searchable."

As a result, in a run of find_datalad_repos where no new datasets are found, we end up making (in addition to the search requests) a separate request for each & every dataset already recorded in datalad-repos.json, which causes the program to exceed GitHub's 5000/hour request limit.
At this point, I can't tell what exactly is happening, as the log level in CI runs is only INFO rather than DEBUG, and some of the historical CI run times are all over the place.  However, one important factor is that, since PR #31, the program uses my ghreq library for querying GitHub, which (by default) will not spend more than 5 minutes total sleeping on a request — and, since the release on 2023-12-17, if waiting for the rate limit to reset would take more than five minutes, ghreq doesn't both waiting at all and just immediately reraises the 403 error.  Unfortunately, as per PR #11, 403 responses when getting a respository's details are treated the same as 404's, so it's possible that a number of repositories are erroneously being marked "gone" when they shouldn't be.
One possible solution to this would be to use the GraphQL API to query repository details, which allows making requests for multiple repositories at once.  However, the way the GraphQL rate limit works is a bit inscrutable, so this might not work either, and the GraphQL API uses different IDs for repositories than the ones that are recorded in datalad-repos.json.
CC @yarikoptic

re hitting the limits.  I think there is no huge need to check every repo every day for either it is still there or to update its stars.  What if we add last-checked datetime, and then check only up to 1000 least recently checked repos on each run? Also we can avoid checking altogether if last checked e.g. less than a week ago.  Ideally it should be not 1000 but "until we hit the limit", so that we do test a good bunch, save results, and then would check only next ones (or none if all checked and nothing within week period) in the next run. WDYT?

@yarikoptic

Note that, as stated above, there are two code paths that lead to querying individual repositories: when a repo is returned in a search result, we get its stars; and when going through all the repositories after a search, any that weren't returned in the search are checked for existence.  Do you want this "delayed checking" to apply to both kinds of checks?  Note that #42 modifies the code so that both checks will now update both stars and the "last updated" timestamp.

Do you want to continue treating (non-rate-limit-related) 403 responses when querying repository details the same as 404s?  That behavior was added in #11 in order to possibly deal with private repositories, but I'm pretty sure that GitHub returns 404 for those (so as not to leak even the existence of private repos to people who shoudn't know about them).

authors: jwodder yarikoptic

repo: datalad/datalad-usage-dashboard


text:
Addresses in Texas Like '# Country Road 212' should be parsed differently

So I am attempting to parse the address: "1 COUNTRY ROAD 212"
Which is giving the following output: (note I am parsing the result it into JSON format)
[{"AddressNumber": "1"}, {"StreetName": "COUNTRY"}, {"StreetNamePostType": "ROAD"}, {"OccupancyIdentifier": "212"}]

When in Texas it uses the Country Road # as a street name for the rural areas.
Output should be something like:
[{"AddressNumber": "1"}, {"StreetName": "COUNTRY ROAD 212"}]

I assume that something Country Rd 212 should be parsed with the same idea as above.
This is kind of a corner case, but I thought I should post an issue report anyways :+1:

thanks for reporting - we've run into issues with weird rural Texas roads before. what's the source of this address? do you have more addresses where this came from?

I can list out a bunch of addresses that are similar:
1901 Country Road 183 Carbon TX 76435
10 Country Road 3668 Splendora TX 77372
1954 Country Road 43 Greenville TX 75401
118 Country Road 46 Hondo TX 78861
14442 Country Road 546 Nevada TX 75173
It looks like from the data that I have there are also some with named country roads:
109 Country Road Ranchette Corsicana TX 75109
104 Country Road Ranchette Corsicana TX 75109
116 Country Road Ranchette Corsicana TX 75109
If I find any more, I will post them up in this same post.

didn't train on the country road ranchette addresses b/c I was unable to find the addresses on google maps, but now the parser is pretty good at recognizing 'country road' as StreetNamePreType. let us know if you run into other patterns that stump the parser, though

Awesome, looks great! Thanks a lot :+1:

authors: REWDevinMcBeth cathydeng

repo: datamade/usaddress


text:
P.O. Box in Box Elder, Montana

A curve ball for the parser:
ORIGINAL STRING:  PO BOX 195; BOX ELDER, MT 59521-0269
PARSED TOKENS:    [(u'PO', 'USPSBoxType'), (u'BOX', 'USPSBoxType'), (u'195;', 'USPSBoxID'), (u'BOX', 'USPSBoxType'), (u'ELDER,', 'PlaceName'), (u'MT', 'StateName'), (u'59521-0269', 'ZipCode')]
UNCERTAIN LABEL:  USPSBoxType

Parsed tokens should be:
[(u'PO BOX', 'USPSBoxType'), (u'195;', 'USPSBoxID'), (u'BOX ELDER', 'PlaceName'), (u'MT', 'StateName'), (u'59521-0269', 'ZipCode')]

hey @rsimoes - thanks for reporting. do you have any more addresses from box elder? or any other addresses where 'box' is an address token other than USPSBoxType?

Well, this is a fun edge case, isn't it? Added this and a few more addresses from Box Elder, MT to the training data. Thanks @rsimoes!

authors: cathydeng jeancochrane rsimoes


text:
Clicking on blog menu header results in download of file

Weird issue...accessing https://blog.datasaturdays.com/ either directly or from menu item on https://datasaturdays.com auto-downloads livedevelop.mp4. Brave, Edge, Chrome all exhibit this behaviour but Safari does not #weird browsers are on MacOS on an M2 Mac, everything up to date etc.

The file comes from https://blog.robsewell.com/assets/uploads/2021/livedevelop.mp4
It's weird. I couldn't find that file referenced anywhere in the datasaturdays repository, so it must be coming from the jekyll files.
@SQLDBAWithABeard any ideas?

Well that is very weird. The blog is hosted in a different repo - let me check what is going on

very strange. The blog is in the repo here https://github.com/dataplat/datasaturdays-blog
The mp4 is on this page and loaded vvia an iframe
https://github.com/dataplat/datasaturdays-blog/blob/d634e712628efc7c4c951208a9d967042ce1f440/_posts/2021-04-15-locally-viewing-github-pages-new-data-saturdays.md?plain=1#L71
So have changed that

Thank you Rob!

Appears to be fixed now with 92b64a2080cfaf34f8fde8c9f751fb8201cc958a

repo: dataplat/DataSaturdays


text:
I am looking for a guy to fuck ❤

I am looking for a guy to fuck ❤

This is escalating… curious to see what comes next honestly

On Feb 28, 2020, at 1:32 AM, helen78989 notifications@github.com wrote:
I am looking for a guy to fuck ❤
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub https://github.com/datasciencemasters/go/issues/173?email_source=notifications&email_token=ALLYOWYWSOSX6S2ZK6O63BLRFCVWNA5CNFSM4K5J7QOKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IRAND3A, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALLYOW32KTCLKQEBQ25QVO3RFCVWNANCNFSM4K5J7QOA.

repo: datasciencemasters/go

license: unlicense


text:
Carrltz disable commands not verified

Inaktiverar kommando användande utöver "/verify" för alla som inte har "verified" rollen.
Skapad till ett Issue av Niklas.

Thanks for the constructive feedback Niklas! I've now updated the file accordingly.
Just tell me if there's something more to change!

Realized that we can improve some other code based on this check. Will write some tomorrow.

Realized that we can improve some other code based on this check. Will write some suggestions tomorrow.

As promised.
So we do only want interaction with the bot to be allowed in Guilds. Doing this check that you make initially here makes a lot of sense. But we do currently not make use of this in our handlers. They are defined to take ChatInputCommandInteraction as an argument. That type has the property guild?: Guild, but we can now use a type where Guild is not undefined. I highly doubt that this augmented type is natively available in the library. But we could instead define a new type
type GuildChatInputCommandInteraction = Omit<ChatInputCommandInteraction, 'guild'> & {
   guild: Guild
}

and use this to type our functions. I think you have to use Omit here to override the property (@Hampfh do you know if there is a prettier way of doing this?). Also, you can probably come up with a more concise name if you want to. Once this is done, then we can remove the check if(!interaction.guild) {...} from other files!

type GuildChatInputCommandInteraction = Omit<ChatInputCommandInteraction, 'guild'> & {
   guild: Guild
}

I think this is the best way of doing it unless guild is the only optional field in the object, if that's the case we could use Required<ChatInputCommandInteraction>

We can't mark every field required. We only want to override Guild right now I think.

authors: Carlltz Hampfh niklasvatn

repo: datasektionen/harmony


text:
How to update secrets accessKeyID and secretAccessKey?

Hello, I'm using dataset with secret-name in spec.local. Datashim operator creates it's own secrets using data from my secret.
But then go to update accessKeyID and secretAccessKey in my secret - it does not updates in secrets, that have been created by datashim operator.

@vasili-ii At the moment, we do not allow incorporate updates to the secrets once they have been used in the Dataset object. This might not be possible to do so because we will have to recreate the PVs/PVCs which may be potentially in   use by pods

repo: datashim-io/datashim


text:
Finding *-Statistics.db in multiple 'data_file_directories' locations fails

ds-collect latest (v2.0.2) / Ubuntu 16
when running the collector I’ve noticed in the logs the following error:
	executing `find /storage/cassandra/data,/storage2/cassandra/data,/storage3/cassandra/data -maxdepth 3 -name *-Statistics.db -exec cp --parents {} /tmp/datastax/20db1.lax1.gogii.net_artifacts_2022_04_21_1232_1650569548/sstable-statistics/ ; > `… find: ‘/storage/cassandra/data,/storage2/cassandra/data,/storage3/cassandra/data’: No such file or directory
failed

It seems like an find is trying to search in the one unique path called /storage/cassandra/data,/storage2/cassandra/data,/storage3/cassandra/data instead of 3 different paths.
I tried manually searching for Statistic files and we have plenty:
root@20db1:~# find /storage/cassandra/data -maxdepth 3 -name *-Statistics.db | wc -l
2400
root@20db1:~# find /storage2/cassandra/data -maxdepth 3 -name *-Statistics.db | wc -l
2266
root@20db1:~# find /storage3/cassandra/data -maxdepth 3 -name *-Statistics.db | wc -l
2401
root@20db1:~#

If I run the find the same way the collector does, it fails:
root@20db1:~# find /storage/cassandra/data,/storage2/cassandra/data,/storage3/cassandra/data -maxdepth 3 -name *-Statistics.db
find: ‘/storage/cassandra/data,/storage2/cassandra/data,/storage3/cassandra/data’: No such file or directory
root@20db1:~#

I believe the issue is with the find cmd concatenating multiple search paths, you must do that without concatenating:
root@20db1:~# find /storage/cassandra/data /storage2/cassandra/data /storage3/cassandra/data -maxdepth 3 -name *-Statistics.db | wc -l
7063
root@20db1:~#

I've narrowed down the issue to:
https://github.com/datastax/diagnostic-collection/blob/fe13ab0ba54269d2975e2d089e2251c6a70afee4/ds-collector/ds-collector#L556
The fix for that specific line to have a result of 3 different paths instead of one for $cassandra_data_dir would be to replace:
tr $'\n' ','
with
tr $'\n' ' '
    cassandra_data_dir=$(sed -n '/^data_file_directories:/,/^[^- ]/{//!p;};/^data_file_directories:/d' "$configHome/cassandra.yaml" | grep -e "^[ ]*-" | sed -e "s/^.*- *//" | tr $'\n' ' ' | sed -e "s/.$/\n/")

The problem with that fix is that for sure it's going to break things further down, as later some other parsing does expect the ',' there to continue working correctly... so changes are required in multiple places or add new code to address this.
Thanks!
Joel.

duplicate of https://github.com/datastax/diagnostic-collection/issues/100
though the analysis and write up here is super valuable, thanks @joelsdc !
this presents itself a super simple fix here: https://github.com/datastax/diagnostic-collection/blob/master/ds-collector/ds-collector#L477
  export data_dir=${cassandra_data_dir/,/ }

this is because the script can be left to work with the value as comma-separated, while then the rust command can now work with it as space-separated.

Awesome! Running it again with that fix!
Do you want me to open a PR or you got it?
Thanks @michaelsembwever !!

Yes, I'd definitely rather see a commit with your name on it 😎

I tried with your suggestion and we almost had it:
	executing `find /storage/cassandra/data /storage2/cassandra/data,/storage3/cassandra/data -maxdepth 3 -name *-Statistics.db -exec cp --parents {} /tmp/datastax/node14.example.com_artifacts_2022_04_22_143
0_1650663034/sstable-statistics/ ; > `… find: ‘/storage2/cassandra/data,/storage3/cassandra/data’: No such file or directory
failed

I think the good-good one will be:
  export data_dir=${cassandra_data_dir//,/ }

(Notice the double // after the variable name to enforce "replace-all" instead of "replace-first-occurrence")
I'll test tonight and if it works as expected I'll submit a PR.
👍

(Notice the double // after the variable name to enforce "replace-all" instead of "replace-first-occurrence")

ooh! TIL

authors: joelsdc michaelsembwever

repo: datastax/diagnostic-collection


text:
Caching codec fixes

Fixes in the same vein as #1643 adapted to play nicely with recent changes to the Java types representing CQL vectors.

Okay, this should largely be ready for validation from everybody involved.  I still need an explicit test for CqlVector itself; that's next, but my main goal was to get the API in front of everyone so that we could get buy-off on the general idea.
As of the most recent round of commits this should build a usable artifact.  @eolivelli I'm most interested in feedback from you confirming that this will address the use case you were addressing with the change in #1643.  There should be equivalent changes to what you had there in this PR but I'd like explicit confirmation from you before we proceed with this thing.

Excellent, thanks @hhughes!
Given all of the feedback above and the various conversations in various other venues I'm going to call this good and merge this change now.  There are several downstream efforts which depend on a released driver that includes these changes and finalizing this and getting it merged is necessary to make that happen.  A huge thanks to everybody who was involved in this process; thanks for making the Java driver's support for vectors even better!

repo: datastax/java-driver


text:
[#2241] feat(client,server): Support the permission operations for both server and client side.

What changes were proposed in this pull request?
Support the permission operations for both server and client side.
Why are the changes needed?
Fix: #2241
Does this PR introduce any user-facing change?
Yes, I will add the document later.
How was this patch tested?
Add new uts.

@jerryshao All comments are addressed.

repo: datastrato/gravitino


text:
data-sync from different repo

In a dvc repository some of data-files were synced to cloud. Then the repo was cloned.
But I failed to sync the file:
$ dvc data-sync data/summary.txt
Downloading cache file from S3 "nlx-shared/tag_classifier/.cache/summary.txt_6405e23"
Traceback (most recent call last):
  File "/Users/dmitry/src/dvc/dvc2.py", line 77, in <module>
    Runtime.run(CmdDataSync, args_start_loc=argv_offset)
  File "/Users/dmitry/src/dvc/dvc/runtime.py", line 45, in run
    sys.exit(instance.run())
  File "/Users/dmitry/src/dvc/dvc/command/data_sync.py", line 60, in run
    return self.sync_symlink(data_item)
  File "/Users/dmitry/src/dvc/dvc/command/data_sync.py", line 84, in sync_symlink
    self.sync_from_cloud(data_item)
  File "/Users/dmitry/src/dvc/dvc/command/data_sync.py", line 184, in sync_from_cloud
    return self._sync_from_cloud_aws(item)
  File "/Users/dmitry/src/dvc/dvc/command/data_sync.py", line 106, in _sync_from_cloud_aws
    key.get_contents_to_filename(item.resolved_cache.relative, cb=percent_cb)
  File "/usr/local/lib/python2.7/site-packages/boto/s3/key.py", line 1714, in get_contents_to_filename
    os.remove(filename)
OSError: [Errno 2] No such file or directory: '.cache/summary.txt_6405e23'

Names are correct. File exist in the cloud.
Another issue - exception should be caught and outputted as a single line (or two lines).

The problem was in .cache dir which was not created because there were no files.
Possible fix - create empty files when initializing the directories.

repo: dataversioncontrol/dvc


text:
关于joyrl，初学者的困惑，请作者帮忙答疑解惑

网上很多能找到的教程都是利用已有的环境来教学，但实际上对于很多非强化学习专业研究者来说如何快速利用强化学习包来解决自己领域内场景的问题才是最棘手的，因此希望作者可以出一个教程甚至可以录一个视频，从下载joyrl，到编写自己的一个环境，到调用joyrl对应的rl算法，到最后解决，能够写出一整个完整的教程，我觉得这是对于其他教程来说一个很好的突破。希望作者可以考虑一下并尽快抓住这个初学者的痛处，毕竟现在考研结束了，很多研0的同学都开始找自己感兴趣的方向了，如果趁这个时候可以做出一个好的教程，那么对于想要学习rl的研0同学一定是帮助很大的。

网上很多能找到的教程都是利用已有的环境来教学，但实际上对于很多非强化学习专业研究者来说如何快速利用强化学习包来解决自己领域内场景的问题才是最棘手的，因此希望作者可以出一个教程甚至可以录一个视频，从下载joyrl，到编写自己的一个环境，到调用joyrl对应的rl算法，到最后解决，能够写出一整个完整的教程，我觉得这是对于其他教程来说一个很好的突破。希望作者可以考虑一下并尽快抓住这个初学者的痛处，毕竟现在考研结束了，很多研0的同学都开始找自己感兴趣的方向了，如果趁这个时候可以做出一个好的教程，那么对于想要学习rl的研0同学一定是帮助很大的。

算法理论部分目前有配套的joyrl book，你说的交叉领域问题我们会在一周内出一个初版的自定义环境教程，敬请期待

作者你好，我想知道为什么我将.yaml文件为preset里的其他文件之后，运行会一直出错。难道只能运行你给的例子吗？其他的很多.yaml文件都运行不了

repo: datawhalechina/joyrl


text:
hasActive method for EntityQuery

PR Checklist
Please check if your PR fulfills the following requirements:

[x] The commit message follows our guidelines: https://github.com/datorama/akita/blob/master/CONTRIBUTING.md#commit
[x] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

PR Type
What kind of change does this PR introduce?

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build related changes
[ ] CI related changes
[ ] Documentation content changes
[ ] Other... Please describe:

What is the current behavior?

To check for an active entity you have to getActiveId and evaluate the result.
What is the new behavior?
A method to check if there is an active entity in the EntityQuery.
Does this PR introduce a breaking change?
[ ] Yes
[x] No

Other information

@NetanelBasal can you give me a hint how I can add the method to the docs as well?

I will add it, thanks. You forget to add the null type tosetActive?.

:tada: This PR is included in version 1.10.0 :tada:
The release is available on npm package (@latest dist-tag)
Your semantic-release bot :package::rocket:

authors: NetanelBasal twittwer

repo: datorama/akita


text:
Implemented Molecule.from_cif(), .from_mol(), .to_mol()

Implemented the following methods:

Molecule.from_cif(),
Molecule.from_mol(), and
Molecule.to_mol().

Implementation notes following.
Molecule.from_cif()

Pretty slow.
I didn't find a standardized way in which .cif files may include a name for the described molecule. Provisionally, the method  takes the basename of the passed file and uses that as the created molecule's name.
The .cif file specification allows a fair bit of variety in the way data is presented, but the parser can't exactly cope with it all. As such, there are valid .cif files which the parser will cowardly refuse to deal with and return None. Notably:

Some .cif files represent atom coordinates in the atom_site loop (table) as absolute coordinates (identified as cartn_x etc.), but others use fractional coordinates w.r.t. the unit cell's dimensions (identified as fract_x etc.). I don't know exactly how involved conversion between fractional and absolute coordinates would be, but either way, the parser does not carry it out, and instead refuses to parse a file if the atom_site loop doesn't have absolute coordinates. PDB-specific .cif file specifications appear to mandate using cartn_x etc. over fract_x etc.
.cif files may represent the element of a given atom in the atom_site loop differently. The parser requires the element name to be contained in the type_symbol column specifically. PDB-specific .cif file specifications appear to mandate including type_symbol.
While the fields in a .cif file loop are whitespace-separated, the specification allows fields to contain whitespace, by putting the entire field in quote - e.g. the record AUTHOR    'A. Jackson'     1 has 3 fields. Blindly treating each contiguous whitespace as a separator would instead net 4 fields, which is incorrect, and what the parser does. Fortunately, the parser can detect when this has potentially occurred and ceases operation in such cases. For what it's worth, I haven't found any .cif files whose atom_site loop actually contained quoted fields.
.cif files provide a way of expressly stating a data value as missing/non-applicable, and this includes an atom's coordinates/element. If the parser encounters a missing value for one of those items, it will refuse to create the molecule.

Not tested very extensively. The parser can identify and report on the above 4 issues, but if a .cif file has something else that that the parser can't naturally deal with, it'll probably be difficult to debug.

Molecule.from_mol()

Pretty slow.
Might work for .sdf files too.
.mol files can carry various information about atoms, including non-standard atomic mass, valence, and implicit hydrogens. The parse ignores most of it - the only atom information other than coordinates and element it makes use of is the atomic charge.
The 'bond' block can describe bonds of type other than currently representable by the Bond class (see page 52 of this PDF. The parser degenerates all of those exotic cases to a single bond.
The only information extracted from the 'header' block is the first line, which may or may not contain the molecule name.
The 'properties' block of the .mol file is completely ignored.

Molecule.to_mol()

Given how much information Molecule.from_mol() disregards, parsing a .mol file to a Molecule instance, and then immediately back into a .mol file (i.e. Molecule.from_mol(file).to_mol()) is very likely to be lossy.
Bonds within the 'bond' block might get reordered when compared to the original file, but otherwise remain intact.

If we could, we can also add a method for reading PDB files - their format is very similar to what we already have, so it shouldn't take long to implement (we can support both ATOM and HETATM with CONECT information).

Applied the suggested fixes etc. Comments to each fix below.

Yes, that line was a debug message I put there for help with testing. A message of two unannotated numbers is pretty useless, and we don't log debug messages consistently elsewhere, so I just removed the message.
Amended as suggested. Also added import os.
Amended as suggested. Mistake on my part, thank you for noticing.
Amended as suggested.
Slightly changed how the atom_site header is dealt with.
Reimplemented the parenthesis removal to a one-liner, using re. Also unrolled the loop over x, y, and z.

I'm holding with the implementation of a potential Molecule.from_pdb() method for now.


text:
Update sbt to 1.4.9

Updates org.scala-sbt:sbt from 1.3.13 to 1.4.9.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "org.scala-sbt", artifactId = "sbt" } ]

labels: library-update, semver-minor

Superseded by #422.

repo: davenverse/cormorant


text:
Update sbt-tpolecat to 0.1.18

Updates io.github.davidgregory084:sbt-tpolecat from 0.1.16 to 0.1.18.
GitHub Release Notes - Version Diff
I'll automatically update this PR to resolve conflicts as long as you don't change it yourself.
If you'd like to skip this version, you can just close this PR. If you have any feedback, just mention me in the comments below.
Configure Scala Steward for your repository with a .scala-steward.conf file.
Have a fantastic day writing Scala!

Ignore future updates
Add this to your .scala-steward.conf file to ignore future updates of this dependency:
updates.ignore = [ { groupId = "io.github.davidgregory084", artifactId = "sbt-tpolecat" } ]

labels: sbt-plugin-update, semver-patch

Superseded by #266.


text:
DOCS: remove braces from variables for unofficial style guide

What this PR does / why we need it:
remove braces from variables
Which issue (if any) this PR fixes:
Fixes # n/a
Any other useful info:
unofficial ansible-nas style guide:
braces are NOT used when stating Ansible variables in other documentation

Good spot! :)

repo: davestephens/ansible-nas


text:
Goturn trained on ALOV only

Hi,
I ran a Goturn(Tensorflow) implementation on ALOV dataset only. When I run it on a test set I find all the bounding boxes at the centre of the image. Does this make sense? Since among majority of the search/target(where obj is centred) pairs in ALOV there is not too much change.

Did you find a solution?

My code generates synthetic image "shifts" by cropping the image in different ways, so this bias in ALOV shouldn't cause too much of a problem.

Also, at test time the bounding box is typically initialized at the location of the object and then the output prediction is a relative motion, relative to the location of the box in the previous frame.  So the box at test time shouldn't necessarily go to the center of the image unless that's where it starts.

repo: davheld/GOTURN


text:
mixcloud not scrobbling. tried restart, and disconnected/reconnected connector but no songs are being scrobbled from mixcloud.

Expected behavior

Current behavior

Possible solution

Steps to reproduce (for bugs)

Context

Steps for resolving the issue

[ ] Reauthenticate on Last.fm

[ ] Try to use unpacked extension from latest master

[ ] Check other issues for duplicates

Environment

Version used:
Browser name and version:

Can't reproduce:

Confirm can't scrobble.

How to debug the extension.

hello,
For me too, it doesn't work.
I used Firefox ESR 52.5.2 64 bit.
Is it compatible with this version of Firefox?

@BenTen it is 'should' or 'may' work, but use latest stable Firefox Quantom, because it is based on web-extensions (which this addon for sure was made for)

@bykidi Quantum
ESR have webext support under flag
But this ext won't work https://github.com/web-scrobbler/web-scrobbler/issues/1538#issuecomment-352815907

authors: BenTen OwnageIsMagic alexesprit bykidi kustek rudegirlb

repo: david-sabata/web-scrobbler


text:
Add scrollable header will be error

When add scrollable header will be error @davideas
mAdapter.addScrollableHeader(new LanguagesHeader("H" ,"Hello", "World"));
What did i do wrong? How should I correct it? Thank you
The following is the current code
LanguagesFragment
public class LanguagesFragment extends BaseFragment {

    RecyclerView mRecyclerView;

    SmartAdapter<BaseItem> mAdapter = new SmartAdapter<>(this);

    int mActivatedPosition = -1;

    @Override
    protected int getLayoutId() {
        return R.layout.fragment_languages;
    }

    @Override
    protected void startUi(Bundle state) {
        super.startUi(state);

        final List<Locale> locales = LocaleChanger.getSupportedLocales();

        for (int i = 0; i < locales.size(); i++) {
            final String language = locales.get(i).getLanguage();
            final String country = locales.get(i).getCountry();
            final LanguagesItem item = new LanguagesItem("I" + i, language, country);

            mAdapter.addItem(item);
        }
        mAdapter.setMode(SelectableAdapter.Mode.SINGLE);
        mAdapter.addScrollableHeader(new LanguagesHeader("H" ,"Hello", "World"));

        mRecyclerView = ViewUtil.getRecyclerView(this, R.id.recyclerView);

        final RecyclerView.LayoutManager layoutManager = ViewUtil.newLinearLayoutManager(getContext());
        final DefaultItemAnimator itemAnimator = new DefaultItemAnimator();

        ViewUtil.setRecycler(mRecyclerView, mAdapter, layoutManager, itemAnimator, null);

        final Locale locale = LocaleChanger.getLocale();
        final int index = LocaleChanger.getIndexFromLocale(locale);
        setActivatedPosition(index);
    }

    @Override
    public boolean onItemClick(View view, int position) {
        if (position != mActivatedPosition) {
            setLocale(position);
        }
        return true;
    }

    private void setLocale(int position) {
        setActivatedPosition(position);

        final Locale locale = LocaleChanger.getLocaleFromIndex(position);
        LocaleChanger.setLocale(locale);

        ActivityRecreationHelper.recreate(getParent(), true);
    }

    private void setActivatedPosition(int position) {
        mActivatedPosition = position;
        mAdapter.toggleSelection(position);
    }
}

LanguagesItem
public class LanguagesItem extends BaseItem<LanguagesItem.LanguagesViewHolder> {

    private String mLanguage;
    private String mCountry;

    public LanguagesItem(String id, String language, String country) {
        this.id = id;
        this.mLanguage = language;
        this.mCountry = country;
    }

    @Override
    public int getLayoutRes() {
        return R.layout.item_material_dense_two_line_list_with_icon;
    }

    @Override
    public LanguagesViewHolder createViewHolder(View view, FlexibleAdapter adapter) {
        return new LanguagesViewHolder(view, adapter);
    }

    @Override
    public void bindViewHolder(FlexibleAdapter adapter, LanguagesViewHolder holder, int position, List payloads) {
        final String displayName = new Locale(mLanguage, mCountry).getDisplayName(new Locale(mLanguage, mCountry));
        final String displayLanguage = new Locale(mLanguage, mCountry).getDisplayLanguage();
        final int accentColor = Colorful().getAccentColor().getColorPack().light().asInt();

        holder.mTextViewPrimary.setText(displayName);
        holder.mTextViewSecondary.setText(displayLanguage);
        ViewUtil.setImageResource(holder.mImageViewIcon, R.drawable.ic_check_black_24dp);
        ViewUtil.setImageColor(holder.mImageViewIcon, accentColor);

        if (adapter.isSelected(position)) {
            ViewUtil.visible(holder.mImageViewIcon);
        } else {
            ViewUtil.gone(holder.mImageViewIcon);
        }

        Timber.d("Payloads: %s", payloads);
    }

    @Override
    public boolean filter(Serializable constraint) {
        return false;
    }

    static final class LanguagesViewHolder extends FlexibleViewHolder {

        final AppCompatTextView mTextViewPrimary;
        final AppCompatTextView mTextViewSecondary;
        final AppCompatImageView mImageViewIcon;

        LanguagesViewHolder(View view, FlexibleAdapter adapter) {
            super(view, adapter);
            mTextViewPrimary = view.findViewById(R.id.primary_text);
            mTextViewSecondary = view.findViewById(R.id.secondary_text);
            mImageViewIcon = view.findViewById(R.id.icon);
        }
    }
}

LanguagesHeader
public class LanguagesHeader extends BaseItem<LanguagesHeader.LanguagesViewHolder> {

    private String mLanguage;
    private String mCountry;

    public LanguagesHeader(String id, String language, String country) {
        this.id = id;
        this.mLanguage = language;
        this.mCountry = country;
    }

    @Override
    public int getLayoutRes() {
        return R.layout.item_material_dense_two_line_list_with_icon;
    }

    @Override
    public LanguagesViewHolder createViewHolder(View view, FlexibleAdapter adapter) {
        return new LanguagesViewHolder(view, adapter);
    }

    @Override
    public void bindViewHolder(FlexibleAdapter adapter, LanguagesViewHolder holder, int position, List payloads) {

        holder.mTextViewPrimary.setText(mLanguage);
        holder.mTextViewSecondary.setText(mCountry);
    }

    @Override
    public boolean filter(Serializable constraint) {
        return false;
    }

    static final class LanguagesViewHolder extends FlexibleViewHolder {

        final AppCompatTextView mTextViewPrimary;
        final AppCompatTextView mTextViewSecondary;
        final AppCompatImageView mImageViewIcon;

        LanguagesViewHolder(View view, FlexibleAdapter adapter) {
            super(view, adapter);
            mTextViewPrimary = view.findViewById(R.id.primary_text);
            mTextViewSecondary = view.findViewById(R.id.secondary_text);
            mImageViewIcon = view.findViewById(R.id.icon);
        }
    }
}

SmartAdapter
public class SmartAdapter<T extends BaseItem> extends BindingFlexibleAdapter<T> {

    private View.OnClickListener clickListener;
    private View.OnLongClickListener longClickListener;

    public SmartAdapter(Object listener) {
        super(listener);
        if (listener instanceof View.OnClickListener) {
            clickListener = (View.OnClickListener) listener;
        }
        if (listener instanceof View.OnLongClickListener) {
            longClickListener = (View.OnLongClickListener) listener;
        }
    }

    public View.OnClickListener getClickListener() {
        return clickListener;
    }

    public View.OnLongClickListener getLongClickListener() {
        return longClickListener;
    }

    @Override
    public boolean addItem(@NonNull T item) {
        if (contains(item)) {
            updateItem(item);
            return true;
        } else {
            return super.addItem(item);
        }
    }

    @Override
    public boolean addItem(int position, @NonNull T item) {
        if (contains(item)) {
            updateItem(item);
            return true;
        } else {
            return super.addItem(position, item);
        }
    }

    public boolean addItem(@NonNull T item, Comparator<IFlexible> comparator) {
        if (contains(item)) {
            updateItem(item);
            return true;
        } else {
            return super.addItem(calculatePositionFor(item, comparator), item);
        }
    }

    @Override
    public void toggleSelection(int position) {
        super.toggleSelection(position);
        notifyItemChanged(position);
    }

    public int getPosition(T item) {
        return getGlobalPositionOf(item);
    }

    public boolean isAnySelected() {
        return getSelectedItemCount() > 0;
    }

    public void addSelection(T item, boolean selected) {
        int position = getGlobalPositionOf(item);
        if (position >= 0) {
            if (selected) {
                addSelection(position);
            } else {
                removeSelection(position);
            }
            notifyItemChanged(position);
        }
    }

    public void selectAll() {
        super.selectAll();
        notifyDataSetChanged();
    }

    @Override
    public void clearSelection() {
        super.clearSelection();
        notifyDataSetChanged();
    }

    public boolean isSelected(T item) {
        return isSelected(getGlobalPositionOf(item));
    }

    public List<T> getSelectedItems() {
        if (getSelectedItemCount() > 0) {
            List<Integer> positions = getSelectedPositions();
            List<T> items = new ArrayList<>(positions.size());
            for (int position : positions) {
                items.add(getItem(position));
            }
            return items;
        }
        return null;
    }
}

BaseItem
public abstract class BaseItem<VH extends FlexibleViewHolder> extends AbstractFlexibleItem<VH> implements IFlexible<VH>, IFilterable, Serializable {

    protected String id;
    protected Color color;
    protected int layoutId;
    protected boolean flag;

    protected BaseItem() {
        color = ColorUtil.getRandColor();
    }

    @Override
    public boolean equals(Object inObject) {
        if (inObject instanceof BaseItem && this.id != null) {
            BaseItem item = (BaseItem) inObject;
            return this.id.equals(item.id);
        }
        return false;
    }

    @Override
    public int hashCode() {
        return Objects.hashCode(id);
    }

    public void setId(String id) {
        this.id = id;
    }

    public void setColor(Color color) {
        this.color = color;
    }

    public void setLayoutId(int layoutId) {
        this.layoutId = layoutId;
    }

    public void setFlag(boolean flag) {
        this.flag = flag;
    }

    public String getId() {
        return id;
    }

    public Color getColor() {
        return color;
    }

    public int getLayoutId() {
        return layoutId;
    }
    public boolean isFlag() {
        return flag;
    }
}

Error Log
2018-04-09 10:24:13.814 30925-30925/com.changrui.tt E/AndroidRuntime: FATAL EXCEPTION: main     Process: com.changrui.tt, PID: 30925     java.lang.ClassCastException: com.changrui.tt.data.adapter.items.LanguagesHeader$LanguagesViewHolder cannot be cast to com.changrui.tt.data.adapter.items.LanguagesItem$LanguagesViewHolder         at com.changrui.tt.data.adapter.items.LanguagesItem.bindViewHolder(LanguagesItem.java:21)         at eu.davidea.flexibleadapter.FlexibleAdapter.onBindViewHolder(FlexibleAdapter.java:1769)         at android.support.v7.widget.RecyclerView$Adapter.bindViewHolder(RecyclerView.java:6714)         at android.support.v7.widget.RecyclerView$Recycler.tryBindViewHolderByDeadline(RecyclerView.java:5647)         at android.support.v7.widget.RecyclerView$Recycler.tryGetViewHolderForPositionByDeadline(RecyclerView.java:5913)         at android.support.v7.widget.RecyclerView$Recycler.getViewForPosition(RecyclerView.java:5752)         at android.support.v7.widget.RecyclerView$Recycler.getViewForPosition(RecyclerView.java:5748)         at android.support.v7.widget.LinearLayoutManager$LayoutState.next(LinearLayoutManager.java:2232)         at android.support.v7.widget.LinearLayoutManager.layoutChunk(LinearLayoutManager.java:1559)         at android.support.v7.widget.LinearLayoutManager.fill(LinearLayoutManager.java:1519)         at android.support.v7.widget.LinearLayoutManager.onLayoutChildren(LinearLayoutManager.java:614)         at android.support.v7.widget.RecyclerView.dispatchLayoutStep2(RecyclerView.java:3812)         at android.support.v7.widget.RecyclerView.dispatchLayout(RecyclerView.java:3529)         at android.support.v7.widget.RecyclerView.onLayout(RecyclerView.java:4082)         at android.view.View.layout(View.java:19659)         at android.view.ViewGroup.layout(ViewGroup.java:6075)         at android.widget.FrameLayout.layoutChildren(FrameLayout.java:323)         at android.widget.FrameLayout.onLayout(FrameLayout.java:261)         at android.view.View.layout(View.java:19659)         at android.view.ViewGroup.layout(ViewGroup.java:6075)         at android.widget.FrameLayout.layoutChildren(FrameLayout.java:323)         at android.widget.FrameLayout.onLayout(FrameLayout.java:261)         at android.view.View.layout(View.java:19659)         at android.view.ViewGroup.layout(ViewGroup.java:6075)         at android.support.design.widget.HeaderScrollingViewBehavior.layoutChild(HeaderScrollingViewBehavior.java:132)         at android.support.design.widget.ViewOffsetBehavior.onLayoutChild(ViewOffsetBehavior.java:42)         at android.support.design.widget.AppBarLayout$ScrollingViewBehavior.onLayoutChild(AppBarLayout.java:1361)         at android.support.design.widget.CoordinatorLayout.onLayout(CoordinatorLayout.java:894)         at android.view.View.layout(View.java:19659)         at android.view.ViewGroup.layout(ViewGroup.java:6075)         at android.widget.FrameLayout.layoutChildren(FrameLayout.java:323)         at android.widget.FrameLayout.onLayout(FrameLayout.java:261)         at android.view.View.layout(View.java:19659)         at android.view.ViewGroup.layout(ViewGroup.java:6075)         at android.widget.FrameLayout.layoutChildren(FrameLayout.java:323)         at android.widget.FrameLayout.onLayout(FrameLayout.java:261)         at android.view.View.layout(View.java:19659)         at android.view.ViewGroup.layout(ViewGroup.java:6075)         at android.widget.FrameLayout.layoutChildren(FrameLayout.java:323)         at android.widget.FrameLayout.onLayout(FrameLayout.java:261)         at android.view.View.layout(View.java:19659)         at android.view.ViewGroup.layout(ViewGroup.java:6075)         at android.widget.LinearLayout.setChildFrame(LinearLayout.java:1791)         at android.widget.LinearLayout.layoutVertical(LinearLayout.java:1635)         at android.widget.LinearLayout.onLayout(LinearLayout.java:1544)         at android.view.View.layout(View.java:19659)         at android.view.ViewGroup.layout(ViewGroup.java:6075)         at android.widget.FrameLayout.layoutChildren(FrameLayout.java:323)         at android.widget.FrameLayout.onLayout(FrameLayout.java:261)         at com.android.internal.policy.DecorView.onLayout(DecorView.java:761)         at android.view.View.layout(View.java:19659)         at android.view.ViewGroup.layout(ViewGroup.java:6075)         at android.view.ViewRootImpl.performLayout(ViewRootImpl.java:2496)

@tcqq, the 2 ViewHolders (Item and Header) have same layout Id.

Thank you very much. It is not have error now.  if it allowed Item and Header to use same Layout will be better, I will not need to create an additional identical layout:-)

If you need same layout, then you need to override FlexibleItem.getItemViewType().

I checked the sample code, sample used "GridLayoutManage", but I am using "LinearLayoutManager" now. How do I correct it now?
    @Override
    protected GridLayoutManager createNewGridLayoutManager() {
        GridLayoutManager gridLayoutManager = new SmoothScrollGridLayoutManager(getActivity(), mColumnCount);
        gridLayoutManager.setSpanSizeLookup(new GridLayoutManager.SpanSizeLookup() {
            @Override
            public int getSpanSize(int position) {
                // NOTE: If you use simple integers to identify the ViewType,
                // here, you should use them and not Layout integers
                switch (mAdapter.getItemViewType(position)) {
                    case R.layout.recycler_scrollable_expandable_item:
                    case R.layout.recycler_scrollable_header_item:
                    case R.layout.recycler_scrollable_footer_item:
                    case R.layout.recycler_scrollable_layout_item:
                    case R.layout.recycler_scrollable_uls_item:
                    case R.layout.progress_item:
                        return mColumnCount;
                    default:
                        return 1;
                }
            }
        });
        return gridLayoutManager;
    }

How should I modify getItemViewType is correct?

authors: davideas tcqq

repo: davideas/FlexibleAdapter


text:
Model reducer state location

The biggest hurdle with RRF I'm trying to understand as I get started is the model reducer: why it's needed, how it can be configured, etc.
In my application, I have a number of reducers already in place which handle actions eg. asynchronously retrieving data.  Let's say I have a page to edit a user which has a form.  When I visit the page, the first thing it does is fire an action to fetch the user record based off an ID in the URL.  Setting an initial state for the corresponding reducer would be useless.  When that async operation completes, it fires a FETCH_USER_SUCCESS action which hits two reducers: one which sets an isFetching state value to FALSE and one that sets the user data state value to the HTTP response payload.
Given RRF's convention, the user data actually needs to be stored in a separate area of the app state from eg. the isFetching value, correct?   I see that the model property on <Form> can take a function. Is there any way to use my existing state location for my form model?.

I think we have a similar question here that I would love to get insight on. I asked the question on StackOveflow:
https://stackoverflow.com/questions/47951142/how-can-i-share-a-reducer-s-initial-state-with-another-reducer?noredirect=1#comment82872787_47951142

I think the thing to realise is that the purpose of the form reducer is to store the state of your form - and that's it. That's not the same purpose as your user profile reducer, which is to store a representation of your user. Although it might look like a duplication of data, this distinction is super useful. Along with your initial form states, theres a $form key that react-redux-form uses to track validation and touched states amongst other things, and the separation means you can replay mutation actions on your form state without affecting anything else.
With that in mind, I think what you both want is to initialise a form with data asynchronously, rather than reusing existing redux state. I solve this problem by dispatching actions.merge(model, <async data>) then actions.setPristine(model) after all my async data has finished loading.
Another thing to experiment with would be the "dynamic" initialisation of a form state. You don't need to create initial states up front - you could just dispatch actions.setInitial(anyModelPathYouLike, <data>) after your user profile has finished loading.

authors: im1dermike madeinspace rikkit

repo: davidkpiano/react-redux-form


text:
fix ascii/bcd instructions

I'm not sure if a lot of software relied on these "archaic" BCD-related instructions, but they had some particulars that contradicted the official documentation, so all these changes can be thought as "undefined behavior", although without them the results are just incorrect in certain cases.

Awesome. Thanks!

I ran my test suite of 50 DOS compilers for various languages with a bunch of apps and there were no regressions.

Awesome!

authors: Arcnor davidly

repo: davidly/ntvdm


text:
Updating stringify signature to match JSON.stringify signature

The current call signature for stringify in index.d.ts does not allow for a replacer or space argument even though they are supported.  Trying to use these arguments produces Typescript compilation errors.
declare function stringify(data: any): string;

I updated the signature to match that of JSON.stringify as this is the proper signature and allows for the supported arguments.
declare function stringify(value: any, replacer?: (key: string, value: any) => any, space?: string | number): string;

Here is a workaround for anyone else experiencing this issue.
(stringify as any)(value, replacerFn, space)

thank you!

repo: davidmarkclements/fast-safe-stringify


text:
FlexLayouts & Card Carousels

Background
I tried creating a commonly requested control that has taken me weeks to build in the past. Here is a drawing of the control I tried to emulate using Xamarin Visual

Control Description
Carousel
The top is a Carousel View which is going to use the collection view Carousel View coming in Xamarin.Forms 4.0. In the past I have used a combination of the following libraries, but the implementation has always been fairly difficult to get it just right:

alexrainman/CarouselView
luberda-molinet/FFImageLoading

Material Entries
The material entries have been either 100% custom built or using the XFX Controls. The idea is identical to what we now have accessible in the built in Visual Material. Including the animated effect when the placeholder slides up and shrinks.

XamFormsExtended/Xfx.Controls

Material Buttons
The custom styles for buttons haven't been that difficult to customize in the past, it is still nice to be supported out of the box.
Screenshots
I went ahead and built out the control in Android first and then iOS and just about everything I tried worked on the first attempt. I really couldn't believe it!
After I got the main control screen working, which only really took 90 minutes I added a new flex layout of tiles to simulate an overview page.
Android

iOS

(No screenshot of carousel)

Overview
Prior to completing the Visual Challenge I had only seen snippets and minor videos about Shell and Visual. This put me in a perfect position to see how long it will take for me to build something I am commonly requested to build which takes multiple weeks in the past. From beginning to end I spent about 90 minutes when in the past I had spent 40-60 hours building custom renderers and styles.
Overall, my experience using Visual and Shell was really great and I can't wait until it is Generally Available so I can start using it on projects.
Some Problems
Even though this was easy to implement I did run into several problems that I wanted to report.
Carousel View (iOS)
The Carousel View did not work at all on iOS, but it worked great on Android
Entry & Editor
The Entry & Editor control worked exactly how I expected. This is a control that I have custom built in the past and have used XfxControls. It was really cool seeing this just work as expected on both iOS and Android.
Datepicker
The DatePicker is another control that was impressive to see it working out of the box, but my biggest complaint is there is no placeholder or label. It makes it difficult to keep consistency when other controls such as Editor or Entry have a placeholder to be displayed above the control when there is a value.
The big problem I noticed with this control is the BackgroundColor only worked on iOS and it did nothing in Android.
Live Streamed
@davidortinau if you are interested in the process I went through I Live Streamed my entire experience on Twitch.tv.  https://www.twitch.tv/videos/397555572

@ahoefling, this is outstanding, thanks! I'll give the recording a look.

Glad you enjoyed my write-up. I am still new to the live coding streams and realized the other night my videos will expire. I have uploaded this one to YouTube in case you try and take a look after the expiration.
https://youtu.be/UN_ZuFxhhko
Just be sure to ignore the 20-30 minutes I spent trying to setup the HotReload plugin

Thanks for the video! I scanned through it tonight. So glad these controls are saving you time. More goodness to come, and thanks in large part to feedback and participation like this!

authors: ahoefling davidortinau

repo: davidortinau/VisualChallenge


text:
Link popup cutoff

Ok just fixed that by adding:
data: () => ({
  options: {
     bounds: '.parent-class' // add the wrapper class or id of vue-editor
  }
})

And the html part
	<vue-editor :editor-options="options" />


text:
Validate on LFW with 20161030 accuracy is 67%

I  test validate on LFW with pre-trained model.
OS : Ubuntu 14 64bit. CPU mode.
~/work/facenet/facenet/src$ python validate_on_lfw.py ~/work/facenet/facenet/data/lfw_mtcnnpy_182 ~/work/facenet/facenet/model/20161030-023650 --lfw_pairs=~/work/facenet/facenet/data/pairs.txt
Model directory: /home/apollo/work/facenet/facenet/model/20161030-023650
Metagraph file: model-20161030-023650.meta
Checkpoint file: model-20161030-023650.ckpt-80000
Runnning forward pass on LFW images
Accuracy: 0.678+-0.016
Validation rate: 0.05905+-0.01552 @ FAR=0.00131
What did I wrong?

It usually looks like this when running Validate on LFW on unaligned images.
Did you align the LFW images as described in Validate on LFW?

Yes, I did align it.
But I got previous result with phase_train_placeholder set False.
And I try test with phase_train_placeholder set True, I got good result.
Runnning forward pass on LFW images
Accuracy: 0.979+-0.008
Validation rate: 0.91245+-0.01793 @ FAR=0.00136
What's the matter?
When test graph without train, is setting phase_train_placeholder  with True Ok?
Moreover If I want to use this graph in really face feature extraction, then is phase_train_placeholder True or False?

Yes, it's a known problem that has been reported in #56. Not sure if this is that the batch norm is used in the wrong way or if it's a bug in it.

authors: apollo-time davidsandberg

repo: davidsandberg/facenet


text:
Can't use GPU in getting embedding layers?

Versions:
OS: Ubuntu 16.04 LTS
OS RAM: 15gb
GPU: Tesla k80 w/ 12gb memory
TF: 1.8.0
Keras: 2.1.5
Facenet: Latest (git cloned 5/11/2018)
CUDA: Cuda compilation tools, release 9.0, V9.0.176
To make sure TF/Keras can access the GPU, i ran
from keras import backend as K; print(K.tensorflow_backend._get_available_gpus())

2018-05-12 05:27:28.496005: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that thi
s TensorFlow binary was not compiled to use: AVX2 FMA
2018-05-12 05:27:28.584137: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from S
ysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-05-12 05:27:28.584531: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:00:04.0
totalMemory: 11.17GiB freeMemory: 11.09GiB
2018-05-12 05:27:28.584556: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-05-12 05:27:28.891873: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor w
ith strength 1 edge matrix:
2018-05-12 05:27:28.891948: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-05-12 05:27:28.891958: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-05-12 05:27:28.892256: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:loc
alhost/replica:0/task:0/device:GPU:0 with 10749 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000
:00:04.0, compute capability: 3.7)
/home/T/anaconda3/lib/python3.6/site-packages/h5py/init.py:34: FutureWarning: Conversion of the second argument of i
ssubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).t\ ype.
from ._conv import register_converters as _register_converters
Using TensorFlow backend.
['/job:localhost/replica:0/task:0/device:GPU:0']

import tensorflow as tf;sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))

2018-05-12 05:26:45.383271: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that thi
s TensorFlow binary was not compiled to use: AVX2 FMA
2018-05-12 05:26:45.471990: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from S
ysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-05-12 05:26:45.472361: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties:
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:00:04.0
totalMemory: 11.17GiB freeMemory: 11.09GiB
2018-05-12 05:26:45.472382: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-05-12 05:26:45.757933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor w
ith strength 1 edge matrix:
2018-05-12 05:26:45.757980: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929]      0
2018-05-12 05:26:45.757996: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0:   N
2018-05-12 05:26:45.758367: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:loc
alhost/replica:0/task:0/device:GPU:0 with 10749 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000
:00:04.0, compute capability: 3.7)
2018-05-12 05:26:45.949442: I tensorflow/core/common_runtime/direct_session.cc:284] Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability
: 3.7
/home/T/anaconda3/lib/python3.6/site-packages/h5py/init.py:34: FutureWarning: Conversion of the second argument of i
ssubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).t\ ype.
from ._conv import register_converters as _register_converters
Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability
: 3.7

Problem:
When i run this:
CUDA_VISIBLE_DEVICES=0 python compare.py  (... rest of parameters ...)
Watching the output of nvidia-smi, i see that load_and_align_data() seems to getting the GPU memory spiked.. but during the sess.run(), GPU memory is at zero.. which makes me wonder

why is the GPU being used ONLY for load_and_align_data?
if not, how do I get the GPU to be used for generating the embedding layers as well?

Ok I solved my own case. Turns out to be CUDA/CUDNN version issue
As of this post's date... for anyone who's having similar issues, make sure your CUDA toolkit version is 9.0.x.. and your CUDNN is 7.0.X...   (note that latest CUDA toolki is 9.1'ish.. and CUDNN is 7.1'ish)
But one question I had @davidsandberg  - load_and_align_data() is quite slow . 80 images took 10 seconds. Particularly, align.detect_face.detect_face() took 7.5 second.
Is there any way to run that in parallel?

I found bulk_detect_face, can't quite make out what the return values are

@taewookim I am sure my CUDA toolkit version is 9.0.176 and CUDNN is 7.0.5, but when I ran the recommand :
python3 align/align_dataset_mtcnn.py align/lfw/ align/lfw_mtcnnalign_160 --image_size 160 --gpu_memory_fraction 0.6
My computer fan is very loud and the display does not have graphics output signal which is black screen. I am sure my environment is no problem for other cv project. Do you know the reason?

authors: Dawson-huang taewookim


text:
detecting a face through webcam using "real_time_face_recognition.py"

I have trained the classifier and tried to use that to detect faces with the help of a webcam.
I used the "real_time_face_recognition.py" code from "contributed" section.
So, here the issue is that it is not detecting the face if there's a slight change in the pose.
Will this code detect faces if there are pose variations in a frame?
If yes, how many training images I have to use to build a classifier so that it can detect a face with pose variation?

Are you pre processing your test images with MTCNN ? check out https://github.com/AshutoshDongare/FaceNet-IOT

That works fine brother @AshutoshDongare
Thank you.

authors: AshutoshDongare RohithPendyala


text:
Fix ReferenceError: Element is not defined in Node.js

Since there's no Element global in Node.js, it's not safe to assume that Element.prototype exists. Looks like this broke in 3.0.0.
Tabbable isn't actually useful in server-side code, but universal React components may still end up loading Tabbable via focus-trap-react when rendering on the server.

Should be released now in 3.1.1 👍

authors: davidtheclark rgrove

repo: davidtheclark/tabbable


text:
Add raw data support

Thank you so much for the library! It was (almost) exactly what we needed. Let us know if we can help further.
This commit adds the ability to return the raw topic and message buffer,
allowing easier handling of binary data from mqtt inside the graphql endpoint
itself.
Enable by setting the rawData option. If it is not set, it behaves
identically to the existing JSON/String path.
It also upgrades Mocha to 7.0.1, to resolve multiple security
vulnerabilities with Mocha.

It looks like node 6 fails on modern versions of Mocha, but I suspect it is reasonable to no longer have support for it. Happy to do whatever makes sense for you.
It also appears like we are the first people to check in the package-lock - happy to remove it if you would rather not.

repo: davidyaha/graphql-mqtt-subscriptions


text:
Add support for additional file types

photosorcery, the underlying library that handles file conversion for Alchemy, added support for converting to and from additional file types. We should handle these types as well.

Closed with #23

repo: dawnlabs/alchemy


text:
Gitlab Git Provider seems to ignore subgroups when constructing the repo URL

Describe the bug
When creating a workspace from a Gitlab repo that is part of a subgroup (e.g. https://gitlab.com/<ORG>/<GROUP>/<REPO_NAME>) the (sub)group is left out of the repo URL which makes it malformed.
To Reproduce
Steps to reproduce the behavior:

daytona create <GITLAB_REPO_IN_GROUP>
Observe that the repo can not be cloned and that the url in the repository object is missing the group.

Expected behavior
The group should be part of the repository URL and the repo should be clonable.
Desktop (please complete the following information):

OS: any
Daytona Version: v0.27.0

@Tpuljak i would like to work on this issue

@quest-bot embark

@s-vamshi go for it. No need to get assigned

hey @Tpuljak , I am not able to replicate the issue, when i use this command with gitlab url go run main.go create https://gitlab.com/gitlab-com/gl-infra/production-engineering, seems clone complete if i am not wrong

@s-vamshi you're correct. The issue seems to have been resolved in the meantime. Thanks for the testing.
@quest-bot tip @s-vamshi 15

authors: Tpuljak s-vamshi

repo: daytonaio/daytona


text:
今年的双十一毫无波澜 | 区块链罗宾

https://dbarobin.com/2020/11/11/1111/
文/Robin好吧，标题党了，今年的双十一确实毫无波澜。往年的双十一通常都有交易额战报，往往从 11 月 11 号凌晨 0 点，隔 1 个小时，就会大张旗鼓地宣传。今年的双十一，阿里巴巴从 11 月 1 号就开始了。很多商家的活动，在十月底就开始了。也不清楚是否是今年疫情影响，经济受到影响的关系，今年的双十一在有...

只是资本市场商品过剩，广告的推销让人们的欲望无限放大。其实我们真的不需要那么多，极简就挺好的，做还是要做，只是需要的不用那么多。

@AmbroseRen
只是资本市场商品过剩，广告的推销让人们的欲望无限放大。其实我们真的不需要那么多，极简就挺好的，做还是要做，只是需要的不用那么多。

是的，我看了下我之前京东的购买订单，确实挺多都是冲动性消费。

authors: AmbroseRen dbarobin


text:
Sql scripting

Draft implementation of sql batch scripting
Examples
CREATE TABLE t1 (a INT) USING parquet;
CREATE TABLE t2 (a INT) USING parquet;
DECLARE totalInsertCount = 0;
WHILE SELECT COUNT(*) < 2 FROM t1; DO
  INSERT INTO t1 VALUES (1);
  SET VAR totalInsertCount = totalInsertCount + 1;
  WHILE SELECT COUNT(*) < 2 FROM t2; DO
   INSERT INTO t2 VALUES (1);
   SET VAR totalInsertCount = totalInsertCount + 1;
   SELECT COUNT(*) as T2Count FROM t2;
  END WHILE;
  TRUNCATE TABLE t2;
END WHILE;
SELECT COUNT(*) as t1FinalCount FROM t1;
SELECT COUNT(*) as t2FinalCount FROM t2;
SELECT totalInsertCount;

Note - this is just testing, proof of concept syntax. Actual syntax will likely be different.
High level design

Implementation of a new tiny parser that supports multi statements, IF/ELSE, variable scope, WHILE loops. Support for TRY/CATCH, cursors and PROCEDURES is TBD (but path forward should be somewhat clear). Main goal of parser is to detect either batch constructs or fallback to main spark parser for statement parsing.
Interpreter implementation. Parser output is handled by a builder that creates new AST-like structures. For each spark statement we first build a logical plan that is stored as leaf operator in aforementioned AST. Interpreter implements iterator interface that returns stream of dataframes. By walking the iterator consumer (that can also be a debugger) gets dataframes for each of spark statements executed by interpreter.

Code pointers

Parser and Lexer are defined in CiglaBaseLexer.g4 and CiglaBaseParser.g4. Lexer can either be further simplified or we can end up reusing main spark lexer. Parser at this point checks leading word to detect spark statement. This also can be further simplified.
Parser to AST logic is in CiglaLangBuilder. AST consists of class hierarchy that derives from NodeStatement (newly introduced interface). Procedural constructs are also defined from this class. e.g.

case class CiglaWhileStatement(
   condition: BoolEvaluableStatement,
   whileBody: CiglaLangNestedIteratorStatement,
   evaluator: StatementBooleanEvaluator) extends NonLeafStatement {

ProceduralLangInterpreter implements Iterator[RewindableStatement] where each statement hosts a LogicalPlan of underlying spark statement if any and, in future, debug information to support debug attach. Code is executed by iterating over this iterator.
Good staring point are tests. CiglaLangSuiteE2E tests E2E functionality. CiglaLangSuite tests new language constructs without spark statements. CiglaLangParserSuite implements parser tests.

This approach is similar to what BigQuery did here: https://github.com/google/zetasql/tree/master/zetasql/scripting.

authors: dbatomic dtenedor

repo: dbatomic/spark


text:
Close or minimize to tray [feature request]

I'd love to be able to natively close or minimize to Windows tray and have a preferencs toggle to set up tray behaviour.
Not sure if there is any similar functionality in other oses, I am purely Windows guy.

I have the same request, but on Linux

repo: dbeaver/dbeaver


text:
[Trino] Forbid creation of objects

Objects cannot be created in Trino. We need to disable their creation in UI

verified

authors: FedorovaDarya uslss


text:
Can't create driver instance, Error during driver instantiation, Missing driver

Description
DBeaver forgets user defined drivers between restarts.
Using the Driver Manager I created a custom definition for a JDBC driver that is not available online. The driver is in my local hard drive.
I defined a DB connection using this driver and was able to connect to the database, no problem at all. The next day after restarting my PC and opening DBeaver again, I got this error message when tried to connect to the database:

Opened the Driver Manager and noticed that the driver I defined before was renamed to a UUID name and shows a "Missing driver" message under the driver selection panel.

Used the Edit button and found that its Libraries list was empty. Added the jar file of the JDBC driver and selected the Driver class, then clicked OK.  After doing this I was able to connect to the database again. However, after restarting DBeaver again the exact same problem occurred.
I have tried these versions of DBeaver and had the same issue with all of them:

ce-21.1.5-win32.win32.x86_64
ce-22.1.4-x86_64-setup
ce-23.1.3-x86_64-setup

DBeaver Version
Community Edition 22.1.4
UPDATE: it hasn't happened with the latest version 23.1.3, but it happened when upgraded from the previous version to the latest.
Operating System
Windows 10
Database and driver
No response
Steps to reproduce

Create a User Defined driver
Create a DB connection that uses it
Confirm that DB sessions can be opened with it
Restart DBeaver and try to connect to the database again. The connection may fail with "Can't create driver instance" and the driver appears as "missing" despite the jar file being available.
Review the driver definition, set the jar library again and save the changes
Attempt to connect to the DB. It should work.
Go back to step 4. The issue reappears after every restart.

Additional context
No response

@LightDye Have you modified the file drivers.xml on directory C:\Users\user\AppData\Roaming\DBeaverData\workspace6.metadata.config  ? And is it reproduced in version 23.1.4?

Hi @MashaKorax, thanks for your reply. I haven't edited any config files manually. I noticed that this started to happen few versions back every time that I upgraded from one version to the next. In a recent version it didn't keep the configuration between restarts. I have now updated to the latest version 23.1.4 but the issue didn't occur. The drivers appear correctly in the Driver Manager and I had no issue opening database sessions. I think that the issue is probably gone now. I can't reproduce it anymore.

authors: LightDye MashaKorax