text:
Querying Lisp Data-frames?

How much effort would it be for cl-duckdb to query Lisp-Stat data frames?  They are a hashtable of vectors and it looks from the duckdb documentation these might be suitable for direct query scanning, in the same way that Pandas, Julia and R data frames can be scanned.

I was also thinking about something like this. Looking at the Julia bindings this seems to be implemented via a combination of table functions  and replacement scans. These are exposed in the C API, but I didn't get around to looking at them yet: I'll add them to the roadmap.
I'll keep this issue open so related discussion can continue here.

@snunez1: I've cobbled together a proof of concept for table functions based on the C API testcases in DuckDB (see example below, needs the table-functions branch). This is still pretty rough, but I think I can see how table functions could be used to integrate CL data structures more or less directly with DuckDB. I'm not exactly sure how copying data could be avoided altogether, but I'll take a look at what the other bindings are doing.
(in-package :duckdb-api)

(defcstruct my-bind-data-struct
  (size :int64))

(defcallback free :void ((ptr :pointer))
  (foreign-free ptr))

(defcallback my-bind :void ((info duckdb-bind-info))
  (let ((type (duckdb-create-logical-type :duckdb-bigint)))
    (duckdb-bind-add-result-column info "forty_two" type)
    (with-foreign-object (p-type '(:pointer duckdb-logical-type))
      (setf (mem-ref p-type 'duckdb-logical-type) type)
      (duckdb-destroy-logical-type p-type)))
  (let* ((my-bind-data (foreign-alloc '(:struct my-bind-data-struct)))
         (param (duckdb-bind-get-parameter info 0)))
    (with-foreign-slots ((size) my-bind-data (:struct my-bind-data-struct))
      (setf size (duckdb-get-int64 param)))
    (with-foreign-object (p-param '(:pointer duckdb-value))
      (setf (mem-ref p-param 'duckdb-value) param)
      (duckdb-destroy-value p-param))
    (duckdb-bind-set-bind-data info
                               my-bind-data
                               (callback free))))

(defcstruct my-init-data-struct
  (pos :int64))

(defcallback my-init :void ((info duckdb-bind-info))
    (let ((my-init-data (foreign-alloc '(:struct my-init-data-struct))))
      (with-foreign-slots ((pos) my-init-data (:struct my-init-data-struct))
        (setf pos 0))
      (duckdb-init-set-init-data info
                                 my-init-data
                                 (callback free))))

(defcallback my-function :void ((info duckdb-bind-info)
                                (output duckdb-data-chunk))
  (let* ((bind-data (duckdb-function-get-bind-data info))
         (init-data (duckdb-function-get-init-data info))
         (bind-data-size (foreign-slot-value bind-data
                                             '(:struct my-bind-data-struct)
                                             'size))
         (ptr (duckdb-vector-get-data (duckdb-data-chunk-get-vector output
                                                                    0))))
    (with-foreign-slots ((pos) init-data (:struct my-init-data-struct))
      (loop :until (>= pos bind-data-size)
            :for i :from 0
            :do (setf (mem-aref ptr :int64 i)
                      (if (zerop (mod pos 2)) 42 84))
            :do (incf pos))
      (duckdb-data-chunk-set-size output 3))))

(defun register-table-function (connection)
  (let ((function (duckdb-create-table-function))
        (type (duckdb-create-logical-type :duckdb-bigint)))
    (duckdb-table-function-set-name function "my_function")
    (duckdb-table-function-add-parameter function type)
    (with-foreign-object (p-type '(:pointer duckdb-logical-type))
      (setf (mem-ref p-type 'duckdb-logical-type) type)
      (duckdb-destroy-logical-type p-type))
    (duckdb-table-function-set-bind function (callback my-bind))
    (duckdb-table-function-set-init function (callback my-init))
    (duckdb-table-function-set-function function (callback my-function))

    (duckdb-register-table-function connection function)
    (with-foreign-object (p-function '(:pointer duckdb-table-function))
      (setf (mem-ref p-function 'duckdb-table-function) function)
      (duckdb-destroy-table-function p-function))))

;; (ddb:initialize-default-connection)
;; (register-table-function (ddb::handle ddb:*connection*))
;; (ddb:query "SELECT * FROM my_function(3) LIMIT 15" nil)
;; => (("forty_two" . #(42 84 42 42 84 42 42 84 42 42 84 42 42 84 42)))

That was fast progress.  I haven't used CFFI for vectors, but I think that a pointer to them can be passed that will avoid copying.  There seems to be a few different ways.  Tamas Papp described two of them in his blog on FFA.  There's also static-vectors, which is actively maintained.  Maybe cffi:make-shareable-byte-vector is worth looking at too?

@snunez1 thanks for your suggestions, static-vectors look really promising for this. Currently I can't seem to figure out how to avoid copying data via the C API, but I've worked a bit on integrating table functions and the basics seem to be working.
Here's how they can be used. I've stuck with alists for now, but data frames seem very similar in structure.

I took some time to speed up the copying and before even appenders were much faster, now there's a huge difference in favor of table functions:
Results for benchmark MEASURE-INTEGER-APPEND-SUM
-                SAMPLES  TOTAL     MINIMUM   MAXIMUM   MEDIAN    AVERAGE     DEVIATION
REAL-TIME        100      0.459004  0.004     0.01      0.005     0.00459004  7.361473e-4
RUN-TIME         100      0.473135  0.004524  0.010109  0.004642  0.00473135  5.572423e-4
USER-RUN-TIME    100      0.455859  0.00264   0.008146  0.004619  0.00455859  5.4765237e-4
SYSTEM-RUN-TIME  100      0.017198  0         0.002022  0         1.7198e-4   4.369875e-4
PAGE-FAULTS      100      0         0         0         0         0           0.0
GC-RUN-TIME      100      0         0         0         0         0           0.0
BYTES-CONSED     100      18673168  162560    1076832   162560    186731.69   91499.75
EVAL-CALLS       100      0         0         0         0         0           0.0

Results for benchmark MEASURE-STATIC-TABLE-INTEGER-SUM
-                SAMPLES  TOTAL     MINIMUM  MAXIMUM   MEDIAN   AVERAGE    DEVIATION
REAL-TIME        100      0.079001  0        0.008     0.001    7.9001e-4  8.5200014e-4
RUN-TIME         100      0.090439  7.72e-4  0.008312  8.11e-4  9.0439e-4  7.469862e-4
USER-RUN-TIME    100      0.088499  5.0e-6   0.008286  8.04e-4  8.8499e-4  7.50846e-4
SYSTEM-RUN-TIME  100      0.00165   0        8.05e-4   0        1.65e-5    8.094547e-5
PAGE-FAULTS      100      0         0        0         0        0          0.0
GC-RUN-TIME      100      0         0        0         0        0          0.0
BYTES-CONSED     100      1926848   0        1371728   0        19268.48   136555.05
EVAL-CALLS       100      0         0        0         0        0          0.0

It's still not very robust: https://github.com/ak-coram/cl-duckdb/issues/9#issue-1447050963

@snunez1 I've fixed the issue mentioned above, but I'm afraid we currently can't avoid copying data using the C API. I've opened a DuckDB feature request issue for this: https://github.com/duckdb/duckdb/issues/5339

Do you think duckdb needs another nudge on the zero-copying issue?

I don't think they currently have much interest in this, for cl-duckdb it would also result in some portability challenges. Performance via table functions is already pretty great, so I'm not too concerned about this either right now. Feel free to nugde them though if you think this is important. I'll gladly look into adapting it if it becomes available in the API.

How is it that Julia can do scan's without copying?

How is it that Julia can do scan's without copying?

It doesn't as far as I can tell:
https://github.com/duckdb/duckdb/blob/master/tools/juliapkg/src/data_frame_scan.jl#L79
I think the Python driver does it by calling the C++ API.

Maybe the Julia docs are out of sync?

The DuckDB Julia package also provides support for querying Julia DataFrames. Note that the DataFrames are directly read by DuckDB - they are not inserted or copied into the database itself.

Maybe it's a question of phrasing, the data is not copied "into the database" I guess. I agree that "directly read by DuckDB" makes it seem like it's a zero-copy implementation.

I'm going to close this issue: since there are multiple alternatives for data-frame libraries, I think it's best if the data-frame itself or a third library is used to integrate with cl-duckdb.
See vellum-duckdb as an example of how this could be done: https://github.com/sirherrbatka/vellum-duckdb
Regarding the the zero-copy implementation of table-functions there's not much we can do until DuckDB itself facilitates it in the C API.

repo: ak-coram/cl-duckdb


text:
[Snyk] Upgrade tap from 11.1.5 to 19.0.2

This PR was automatically created by Snyk using the credentials of a real user.![snyk-top-banner](https://github.com/andygongea/OWASP-Benchmark/assets/818805/c518c423-16fe-447e-b67f-ad5a49b5d123)
Snyk has created this PR to upgrade tap from 11.1.5 to 19.0.2.
:information_source: Keep your dependencies up-to-date. This makes it easier to fix existing vulnerabilities and to more quickly identify and fix newly disclosed vulnerabilities when they affect your project.

⚠ Warning: This PR contains major version upgrade(s), and may be a breaking change.

The recommended version is 241 versions ahead of your current version.

The recommended version was released on 22 days ago.

Issues fixed by the recommended upgrade:

Issue
Score
Exploit Maturity

Prototype PollutionSNYK-JS-JSONSCHEMA-1920922
149
No Known Exploit

Prototype PollutionSNYK-JS-MIXINDEEP-450212
149
Proof of Concept

Denial of Service (DoS)SNYK-JS-DECODEURICOMPONENT-3149970
149
Proof of Concept

Prototype PollutionSNYK-JS-AJV-584908
149
No Known Exploit

Incomplete List of Disallowed InputsSNYK-JS-BABELTRAVERSE-5962463
149
Proof of Concept

Prototype PollutionSNYK-JS-HANDLEBARS-173692
149
No Known Exploit

Prototype PollutionSNYK-JS-HANDLEBARS-174183
149
No Known Exploit

Inefficient Regular Expression ComplexitySNYK-JS-MICROMATCH-6838728
149
No Known Exploit

Prototype PollutionSNYK-JS-UNSETVALUE-2400660
149
No Known Exploit

Prototype PollutionSNYK-JS-SETVALUE-1540541
149
Proof of Concept

Prototype PollutionSNYK-JS-SETVALUE-450213
149
Proof of Concept

Prototype PollutionSNYK-JS-SETVALUE-1540541
149
Proof of Concept

Prototype PollutionSNYK-JS-SETVALUE-450213
149
Proof of Concept

Regular Expression Denial of Service (ReDoS)SNYK-JS-HOSTEDGITINFO-1088355
149
Proof of Concept

Regular Expression Denial of Service (ReDoS)SNYK-JS-MINIMATCH-3050818
149
No Known Exploit

Reverse TabnabbingSNYK-JS-ISTANBULREPORTS-2328088
149
No Known Exploit

Regular Expression Denial of Service (ReDoS)SNYK-JS-PATHPARSE-1077067
149
Proof of Concept

Server-side Request Forgery (SSRF)SNYK-JS-REQUEST-3361831
149
Proof of Concept

Prototype PollutionSNYK-JS-YARGSPARSER-560381
149
Proof of Concept

Prototype PollutionSNYK-JS-YARGSPARSER-560381
149
Proof of Concept

Denial of Service (DoS)npm:mem:20180117
149
No Known Exploit

Prototype PollutionSNYK-JS-TOUGHCOOKIE-5672873
149
Proof of Concept

Validation BypassSNYK-JS-KINDOF-537849
149
Proof of Concept

Release notes

Package name: tap

19.0.2 - 2024-05-25

19.0.1 - 2024-05-24

19.0.0 - 2024-05-21

18.8.0 - 2024-05-13

18.7.3 - 2024-05-06

18.7.2 - 2024-03-26

18.7.1 - 2024-03-08

18.7.0 - 2024-01-25

18.6.1 - 2023-11-16

18.6.0 - 2023-11-16

18.5.8 - 2023-11-13

18.5.7 - 2023-11-04

18.5.6 - 2023-11-03

18.5.5 - 2023-11-03

18.5.4 - 2023-10-31

18.5.3 - 2023-10-28

18.5.2 - 2023-10-16

18.5.1 - 2023-10-15

18.5.0 - 2023-10-11

18.4.6 - 2023-10-10

18.4.5 - 2023-10-09

18.4.4 - 2023-10-08

18.4.3 - 2023-10-05

18.4.2 - 2023-10-02

18.4.1 - 2023-10-01

18.4.0 - 2023-09-29

18.3.1 - 2023-09-29

18.3.0 - 2023-09-28

18.2.2 - 2023-09-28

18.2.1 - 2023-09-28

18.2.0 - 2023-09-27

18.1.5 - 2023-09-26

18.1.4 - 2023-09-24

18.1.3 - 2023-09-24

18.1.2 - 2023-09-24

18.1.1 - 2023-09-23

18.1.0 - 2023-09-23

18.0.4 - 2023-09-18

18.0.3 - 2023-09-17

18.0.2 - 2023-09-17

18.0.1 - 2023-09-15

18.0.0 - 2023-09-15

18.0.0-26 - 2023-09-13

18.0.0-25 - 2023-09-10

18.0.0-24 - 2023-09-05

18.0.0-23 - 2023-09-03

18.0.0-22 - 2023-09-01

18.0.0-21 - 2023-08-29

18.0.0-20 - 2023-08-27

18.0.0-19 - 2023-08-21

18.0.0-18 - 2023-08-20

18.0.0-17 - 2023-08-18

18.0.0-16 - 2023-08-17

18.0.0-15 - 2023-08-15

18.0.0-14 - 2023-08-12

18.0.0-13 - 2023-08-09

18.0.0-12 - 2023-08-07

18.0.0-11 - 2023-08-07

18.0.0-10 - 2023-08-07

18.0.0-9 - 2023-08-07

18.0.0-8 - 2023-08-07

18.0.0-7 - 2023-08-07

18.0.0-6 - 2023-08-06

18.0.0-5 - 2023-08-06

18.0.0-4 - 2023-08-04

18.0.0-3 - 2023-08-04

18.0.0-2 - 2023-08-04

18.0.0-1 - 2023-08-04

16.3.10 - 2023-12-15

16.3.9 - 2023-09-27

16.3.8 - 2023-07-30

16.3.7 - 2023-06-22

16.3.6 - 2023-06-04

16.3.5 - 2023-06-04

16.3.4 - 2023-01-16

16.3.3 - 2023-01-09

16.3.2 - 2022-11-16

16.3.1 - 2022-11-13

16.3.0 - 2022-06-17

16.2.0 - 2022-05-04

16.1.0 - 2022-04-25

16.0.1 - 2022-03-22

16.0.0 - 2022-03-05

15.2.3 - 2022-03-05

15.2.2 - 2022-03-05

15.2.1 - 2022-03-05

15.2.0 - 2022-03-05

15.1.6 - 2022-01-04

15.1.5 - 2021-11-26

15.1.4 - 2021-11-26

15.1.3 - 2021-11-26

15.1.2 - 2021-11-18

15.1.1 - 2021-11-17

15.1.0 - 2021-11-16

15.0.10 - 2021-09-21

15.0.9 - 2021-05-07

15.0.8 - 2021-05-07

15.0.7 - 2021-05-06

15.0.6 - 2021-04-29

15.0.5 - 2021-04-24

15.0.4 - 2021-04-20

15.0.3 - 2021-04-20

15.0.2 - 2021-04-05

15.0.1 - 2021-04-01

15.0.0 - 2021-03-31

15.0.0-3 - 2021-03-26

15.0.0-2 - 2021-02-23

15.0.0-1 - 2021-02-18

15.0.0-0 - 2021-02-17

14.11.0 - 2020-11-16

14.10.8 - 2020-07-21

14.10.7 - 2020-03-20

14.10.6 - 2020-01-14

14.10.5 - 2019-12-18

14.10.4 - 2019-12-16

14.10.3 - 2019-12-16

14.10.2 - 2019-11-25

14.10.2-unbundled - 2019-11-26

14.10.2-totally-bundled.1 - 2019-11-27

14.10.2-totally-bundled - 2019-11-27

14.10.1 - 2019-11-21

14.10.0 - 2019-11-21

14.9.2 - 2019-11-06

14.9.1 - 2019-10-30

14.9.0 - 2019-10-30

14.8.3 - 2019-10-30

14.8.2 - 2019-10-22

14.8.1 - 2019-10-22

14.8.0 - 2019-10-21

14.7.3 - 2019-10-20

14.7.2 - 2019-10-18

14.7.1 - 2019-10-16

14.7.0 - 2019-10-14

14.6.9 - 2019-09-30

14.6.8 - 2019-09-30

14.6.7 - 2019-09-29

14.6.6 - 2019-09-29

14.6.5 - 2019-09-23

14.6.4 - 2019-09-15

14.6.3 - 2019-09-15

14.6.2 - 2019-09-10

14.6.1 - 2019-08-04

14.6.0 - 2019-08-04

14.5.0 - 2019-07-28

14.4.3 - 2019-07-22

14.4.2 - 2019-07-17

14.4.1 - 2019-07-03

14.4.0 - 2019-07-02

14.3.1 - 2019-06-25

14.3.0 - 2019-06-25

14.2.5 - 2019-06-22

14.2.4 - 2019-06-19

14.2.3 - 2019-06-14

14.2.2 - 2019-06-05

14.2.1 - 2019-06-04

14.2.0 - 2019-05-28

14.1.11 - 2019-05-24

14.1.10 - 2019-05-23

14.1.9 - 2019-05-23

14.1.8 - 2019-05-23

14.1.7 - 2019-05-22

14.1.6 - 2019-05-22

14.1.5 - 2019-05-22

14.1.4 - 2019-05-22

14.1.3 - 2019-05-21

14.1.2 - 2019-05-21

14.1.1 - 2019-05-20

14.1.0 - 2019-05-20

14.0.0 - 2019-05-18

13.1.11 - 2019-05-17

13.1.10 - 2019-05-17

13.1.9 - 2019-05-14

13.1.8 - 2019-05-11

13.1.7 - 2019-05-10

13.1.6 - 2019-05-09

13.1.5 - 2019-05-08

13.1.4 - 2019-05-08

13.1.3 - 2019-05-08

13.1.2 - 2019-05-01

13.1.1 - 2019-04-29

13.1.0 - 2019-04-28

13.0.4 - 2019-04-27

13.0.3 - 2019-04-26

13.0.1 - 2019-04-26

13.0.0 - 2019-04-26

13.0.0-rc.30 - 2019-04-24

13.0.0-rc.29 - 2019-04-24

13.0.0-rc.28 - 2019-04-24

13.0.0-rc.27 - 2019-04-24

13.0.0-rc.26 - 2019-04-24

13.0.0-rc.25 - 2019-04-20

13.0.0-rc.24 - 2019-04-17

13.0.0-rc.23 - 2019-04-06

13.0.0-rc.22 - 2019-04-06

13.0.0-rc.21 - 2019-04-06

13.0.0-rc.20 - 2019-03-31

13.0.0-rc.19 - 2019-03-29

13.0.0-rc.18 - 2019-03-26

13.0.0-rc.17 - 2019-03-26

13.0.0-rc.16 - 2019-03-25

13.0.0-rc.15 - 2019-03-25

13.0.0-rc.14 - 2019-03-22

13.0.0-rc.13 - 2019-03-22

13.0.0-rc.12 - 2019-03-21

13.0.0-rc.11 - 2019-03-20

13.0.0-rc.10 - 2019-03-18

13.0.0-rc.9 - 2019-03-18

13.0.0-rc.8 - 2019-03-18

13.0.0-rc.7 - 2019-03-17

13.0.0-rc.6 - 2019-03-17

13.0.0-rc.5 - 2019-03-16

13.0.0-rc.4 - 2019-03-16

13.0.0-rc.3 - 2019-03-13

13.0.0-rc.2 - 2019-03-13

13.0.0-rc.1 - 2019-03-11

13.0.0-rc.0 - 2019-03-11

12.7.0 - 2019-04-28

12.6.6 - 2019-04-26

12.6.5 - 2019-04-24

12.6.4 - 2019-04-24

12.6.3 - 2019-04-22

12.6.2 - 2019-04-17

12.6.1 - 2019-03-22

12.6.0 - 2019-03-07

12.5.3 - 2019-02-15

12.5.2 - 2019-02-08

12.5.1 - 2019-02-01

12.5.0 - 2019-01-30

12.4.1 - 2019-01-30

12.4.0 - 2019-01-23

12.3.0 - 2019-01-23

12.2.1 - 2019-01-23

12.2.0 - 2019-01-23

12.1.4 - 2019-01-22

12.1.3 - 2019-01-22

12.1.2 - 2019-01-21

12.1.1 - 2018-12-12

12.1.0 - 2018-11-13

12.0.2 - 2018-11-12

12.0.1 - 2018-05-17

12.0.0 - 2018-05-17

11.1.5 - 2018-05-09

    from tap GitHub release notes

[!IMPORTANT]

Warning: This PR contains a major version upgrade, and may be a breaking change.
Check the changes in this PR to ensure they won't cause issues with your project.
This PR was automatically created by Snyk using the credentials of a real user.
Max score is 1000. Note that the real score may have changed since the PR was raised.

Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open upgrade PRs.
For more information:

🧐 View latest project report
📜 Customise PR templates
🛠 Adjust upgrade PR settings
🔕 Ignore this dependency or unsubscribe from future upgrade PRs

🎉 Snyk hasn't found any issues so far.
✅ code/snyk check is completed. No issues were found. (View Details)

source: gharchive/pull-request

repo: akanchhaS/github-goof-npm

license_type: permissive

license_source: github-api


text:
Authentication using local and social accounts

fixes #1

Review please @jainmickey

Coverage remained the same at 100.0% when pulling 645313e79027d94300dbed26d031d419c1ab00ff on feature/authentication into c3ffaea898fbefbbcdeff037f174d3cc75d3e2fd on develop.

Coverage remained the same at 100.0% when pulling a170b79778a4ce361fa4f08c721f6684b400fbd4 on feature/authentication into c3ffaea898fbefbbcdeff037f174d3cc75d3e2fd on develop.

LGTM.

authors: akarambir coveralls jainmickey

repo: akarambir/askcoding

license: mit

license_source: bigquery


text:
Invalid rpc nodes

We were doing some testing with the proxy and came across connectivity issues. Some of the rpc nodes under the proxy are either returning errors or an html page.
Here's the list of faulty nodes I found:

https://public.stakewolle.com/cosmos/akash/rpc
https://rpc-akash-ia.cosmosia.notional.ventures/
https://akash-rpc.publicnode.com/

should we maybe remove these nodes from https://raw.githubusercontent.com/cosmos/chain-registry/master/akash/chain.json ?
I noticed that while working on this, assumed it was a temporary error or that I needed some sort of api key...

we could store status codes in a histogram or in some buckets and use that as an heuristic to determine if the node is healthy or not, along with response time.
do you think that makes sense?

Yea, I think keeping track of the status of each nodes makes the most sense to me. I don't think we want to manually filter which ones currently work or not.

forgot to mention, but #12 should do it

source: gharchive/issue

repo: akash-network/rpc-proxy


text:
1.全球指数数据获取的建议2.中债指数获取需求。

1.全球指数数据获取的建议
目前AK的的全球指数据来自英维，一是需要代理二是对方服务器也有限制。建议改用东财网站的数据。
东财网页地址如下，比如英国富时100FTSE，规则就是/gb/zs加指数代码，页面也同国内指数类似。
http://quote.eastmoney.com/gb/zsFTSE.html
2.中债指数获取需求
国内债基指数基本是以中债指数为基准，网页如下，请求AK进行新接口开发。非常感谢。
https://yield.chinabond.com.cn/cbweb-mn/indices/single_index_query

1.全球指数数据获取的建议 目前AK的的全球指数据来自英维，一是需要代理二是对方服务器也有限制。建议改用东财网站的数据。 东财网页地址如下，比如英国富时100FTSE，规则就是/gb/zs加指数代码，页面也同国内指数类似。 http://quote.eastmoney.com/gb/zsFTSE.html
2.中债指数获取需求 国内债基指数基本是以中债指数为基准，网页如下，请求AK进行新接口开发。非常感谢。 https://yield.chinabond.com.cn/cbweb-mn/indices/single_index_query

感想反馈问题；

东财的接口数据较少且非正常数据格式
已新增部分接口，目前正在更新中，升级到 AKShare 1.7.26 版本查看

repo: akfamily/akshare

license: MIT


text:
No need for specification of category colours any more

0.7.0 contains code that will automatically assign a random bgColor and color if none is passed in.
Something to keep in mind for the next version of the importer :)

gotcha!

authors: akhoury julianlam

repo: akhoury/nodebb-plugin-import


text:
[BUG] Duplicates in cmp suggestions

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior
If flutter-tools is enabled there are duplicates in cmp suggestions.
Disabling flutter-tools makes it work correctly.
Here's my config:
lazy
{
	"akinsho/flutter-tools.nvim",
	lazy = false,
	dependencies = {
		"nvim-lua/plenary.nvim",
		"stevearc/dressing.nvim", -- optional for vim.ui.select
	},
	ft = "dart",
	config = true,
},

lsp.lua
local capabilities = vim.lsp.protocol.make_client_capabilities()
capabilities = require("cmp_nvim_lsp").default_capabilities(capabilities)
-- Setup language servers.
local lspconfig = require("lspconfig")
lspconfig.pyright.setup({})
-- lspconfig.dartls.setup({})

-- Global mappings.
-- See `:help vim.diagnostic.*` for documentation on any of the below functions
vim.keymap.set("n", "<leader>lD", vim.diagnostic.open_float)
vim.keymap.set("n", "[d", vim.diagnostic.goto_prev)
vim.keymap.set("n", "]d", vim.diagnostic.goto_next)
vim.keymap.set("n", "<leader>ld", vim.diagnostic.setloclist)

-- Use LspAttach autocommand to only map the following keys
-- after the language server attaches to the current buffer
vim.api.nvim_create_autocmd("LspAttach", {
	group = vim.api.nvim_create_augroup("UserLspConfig", {}),
	callback = function(ev)
		-- Enable completion triggered by <c-x><c-o>
		vim.bo[ev.buf].omnifunc = "v:lua.vim.lsp.omnifunc"

		local opts = { buffer = ev.buf }
		vim.keymap.set("n", "gD", vim.lsp.buf.declaration, opts)
		vim.keymap.set("n", "K", vim.lsp.buf.hover, opts)
		vim.keymap.set("n", "gi", vim.lsp.buf.implementation, opts)
		vim.keymap.set("n", "<C-k>", vim.lsp.buf.signature_help, opts)
		-- vim.keymap
		--     .set('n', '<Leader>sa', vim.lsp.buf.add_workspace_folder, opts)
		-- vim.keymap.set('n', '<Leader>sr', vim.lsp.buf.remove_workspace_folder,
		--                opts)
		-- vim.keymap.set('n', '<Leader>sl', function()
		--     print(vim.inspect(vim.lsp.buf.list_workspace_folders()))
		-- end, opts)
		-- vim.keymap.set('n', '<Leader>D', vim.lsp.buf.type_definition, opts)
		vim.keymap.set("n", "<Leader>lr", vim.lsp.buf.rename, opts)
		vim.keymap.set({ "n", "v" }, "<Leader>la", vim.lsp.buf.code_action, opts)
		-- vim.keymap.set('n', 'gr', vim.lsp.buf.references, opts)
		vim.keymap.set("n", "<Leader>lf", function()
			vim.lsp.buf.format({ async = true })
		end, opts)
	end,
})

cmp.lua
local cmp = require("cmp")
local luasnip = require("luasnip")

cmp.setup({
	snippet = {
		-- REQUIRED - you must specify a snippet engine
		expand = function(args)
			-- vim.fn["vsnip#anonymous"](args.body) -- For `vsnip` users.
			require("luasnip").lsp_expand(args.body) -- For `luasnip` users.
			-- require('snippy').expand_snippet(args.body) -- For `snippy` users.
			-- vim.fn["UltiSnips#Anon"](args.body) -- For `ultisnips` users.
		end,
	},
	window = {
		completion = cmp.config.window.bordered(),
		documentation = cmp.config.window.bordered(),
	},
	mapping = cmp.mapping.preset.insert({
		["<C-b>"] = cmp.mapping.scroll_docs(-4),
		["<C-f>"] = cmp.mapping.scroll_docs(4),
		["<C-Space>"] = cmp.mapping.complete(),
		["<C-e>"] = cmp.mapping.abort(),
		["<CR>"] = cmp.mapping.confirm({ select = true }),
		["<Tab>"] = cmp.mapping(function(fallback)
			if cmp.visible() then
				cmp.select_next_item()
			elseif luasnip.expand_or_jumpable() then
				luasnip.expand_or_jump()
			else
				fallback()
			end
		end, { "i", "s" }),
		["<S-Tab>"] = cmp.mapping(function(fallback)
			if cmp.visible() then
				cmp.select_prev_item()
			elseif luasnip.jumpable(-1) then
				luasnip.jump(-1)
			else
				fallback()
			end
		end, { "i", "s" }),
	}),
	sources = cmp.config.sources({
		{ name = "luasnip" },
		{ name = "nvim_lsp" },
	}, { { name = "buffer" }, { name = "nvim_lsp_signature_help" } }),
})

-- Set configuration for specific filetype.
cmp.setup.filetype("gitcommit", {
	sources = cmp.config.sources({
		{ name = "cmp_git" }, -- You can specify the `cmp_git` source if you were installed it.
	}, { { name = "buffer" } }),
})

-- Use buffer source for `/` and `?` (if you enabled `native_menu`, this won't work anymore).
cmp.setup.cmdline({ "/", "?" }, {
	mapping = cmp.mapping.preset.cmdline(),
	sources = { { name = "buffer" } },
})

-- Use cmdline & path source for ':' (if you enabled `native_menu`, this won't work anymore).
cmp.setup.cmdline(":", {
	mapping = cmp.mapping.preset.cmdline(),
	sources = cmp.config.sources({ { name = "path" } }, { { name = "cmdline" } }),
})

-- Set up lspconfig.
local capabilities = require("cmp_nvim_lsp").default_capabilities()
require("lspconfig")["tsserver"].setup({ capabilities = capabilities })
require("lspconfig")["dartls"].setup({ capabilities = capabilities })

Expected Behavior
Onlu unique options in cmp
Steps To Reproduce
.
Environment
- OS: MacOS 13.2.1
- Flutter version: 3.13.1
- Is flutter in $PATH: yes
- neovim version: v0.9.1

Anything else?
No response

My bad. I missed that there's require("lspconfig")["dartls"].setup({ capabilities = capabilities }) in cmp.lua.
Removed that and it works fine.
Sorry for redundant issue.


text:
Make sure that we're using the new protobuf backed serializer

See the actual problem here: https://github.com/akkadotnet/akka.net/issues/3811
Example implementation in MongoDb: https://github.com/akkadotnet/Akka.Persistence.MongoDB/pull/71

I've had some pushback on this issue before from @to11mtm and others, but I'm generally of the opinion that using the Protobuf envelopes is a good idea so we can future proof against future schema migration changes. Open to suggestions though.

We'll stick with the current serialization regime for Akka.Persistence.Sql.

Adding some flavor for those who may see this issue and have questions... also to respond to other comments here;
In short, JVM Akka tried the Protobuf backed serializer but wound up moving away from it. Last I looked at any of the 'sanctioned' plugins for JVM (i.e. before they moved away from Apache and onto BSL,) none of them used protobuf envelopes and instead had some level of optimization for the specific vagaries of their store.
Best reasons I could divine (since here we do things the way JVM did,) and I agree with these in principle as well as practice:

The way information is stored in Journals/snapshots, we already have all of the data that we care about AFA Serializer ID, Serializer Manifest, ETC.

IIRC we may not be handling Event Manifest yet (will need to look at the code) but do have it provisioned in the table

Using a Protobuf envelope unfortunately would be an additional layer of indirection (i.e. double-serialization and double-deserialization), and while I can't speak for the JVM world, .NET Protobuf implementations tend to be pretty allocatey and not as fast as other solutions...

but I'm generally of the opinion that using the Protobuf envelopes is a good idea so we can future proof against future schema migration changes.

I mean, they do make some things easier. Won't argue that.
However, as counterpoints:

The protobuf envelopes would lead to data dupe:

For SQL Implementations, we already need to have PersistenceId and SequenceNumber as columns in the DB for core functionality.
If I actually needed to troubleshoot a serialization issue, I'd definitely prefer having the SerializerId/SerializerManifest as columns like they are now vs them living in an envelope I have to decode

The kinda-nice thing about AkkaPersistenceDataConnectionFactory is it does let us handle a -lot- on our end via config flags/etc.

And yes I know we already have a lot of... violent code in there and other parts AFA config flags, but that is almost entirely because we theoretically support 'live' migration to tag-tables (something not even the JVM did, go us!) as well as backwards compatibility with old SQL providers.
FWIW, there -are- ways to do 'sensing' via Linq2Db. Let's imagine we added a string property, Baz into PersistentRepr. In the connection factory you could do a query like sensingContext.GetTable<JournalRow>().Select(r=>r.Baz).FirstOrDefault() in a try-catch... if it didn't throw you know the column is there and can then act appropriately, otherwise ignore column in mapping and IIRC Linq2Db can be finagled into providing a 'default' in those cases...
Yes I still know it's not always pretty. Alas balancing performance versus code violence is often tricky.

authors: Aaronontheweb Arkatufus to11mtm


text:
Is it supposed to do a new note per highlight?

Curious if I have something misconfigured, or if it's intended to create a new note for each highlight? Thanks!

Whoops, misunderstood the purpose of the plugin 😅


text:
error: ‘long unsigned int newsbeuter::openssl_mth_id_function()’ defined but not used

When attempting to compile newsbeuter (https://github.com/akrennmair/newsbeuter/commit/20330eaf3441a47b1f1c0d0a134a2052fdc1ac74) I get the following errors:
g++ -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -std=c++11 -ggdb -Iinclude -Istfl -Ifilter -I. -Irss -Werror -Wall -Wextra -Wunreachable-code -DLOCALEDIR=\"/usr/local/share/locale\" -DGIT_HASH=\"20330\" -I/usr/include/libxml2  -I/usr/include/json-c  -D_GNU_SOURCE -D_DEFAULT_SOURCE  -DHAVE_OPENSSL=1 -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong -fno-plt -o src/cache.o -c src/cache.cpp
src/utils.cpp:1149:22: error: ‘long unsigned int newsbeuter::openssl_mth_id_function()’ defined but not used [-Werror=unused-function]
 static unsigned long openssl_mth_id_function(void) {
                      ^~~~~~~~~~~~~~~~~~~~~~~
src/utils.cpp:1135:13: error: ‘void newsbeuter::openssl_mth_locking_function(int, int, const char*, int)’ defined but not used [-Werror=unused-function]
 static void openssl_mth_locking_function(int mode, int n, const char * file, int line) {
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

Recently we switched to openssl 1.1.0e which is likely the cause for this breakage.
% openssl version
OpenSSL 1.1.0e  16 Feb 2017

Is there any other information or any suggestions about what I can do next?

This is probably related to this entry in the changelog for 1.1.0:

*) OpenSSL now uses a new threading API. It is no longer necessary to
set locking callbacks to use OpenSSL in a multi-threaded environment. There
are two supported threading models: pthreads and windows threads. It is
also possible to configure OpenSSL at compile time for "no-threads". The
old threading API should no longer be used. The functions have been
replaced with "no-op" compatibility macros.
[Alessandro Ghedini, Matt Caswell]

Thanks for reporting! I'll look into fixing this tomorrow.

I should really stop making such promises. Large PR got merged and ate all my computer time for the evening. This issue will have to wait a bit more.

I'm in no hurry, don't worry.  Thank you for looking into at all \o

I don't have 1.1.0, so I can't test this locally (without building OpenSSL and Curl from source, at least). Can you please pull down bugfix/522-openssl-new-threading-api branch and see if it works for you?

With that branch I see two new errors of the same calibur:
src/utils.cpp:1152:22: error: ‘long unsigned int newsbeuter::openssl_mth_id_function()’ defined but not used [-Werror=unused-function]
 static unsigned long openssl_mth_id_function(void) {
                      ^~~~~~~~~~~~~~~~~~~~~~~
src/utils.cpp:1138:13: error: ‘void newsbeuter::openssl_mth_locking_function(int, int, const char*, int)’ defined but not used [-Werror=unused-function]
 static void openssl_mth_locking_function(int mode, int n, const char * file, int line) {
             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~

Perhaps you added one too many 0's?

Definitely looks like that! I pushed another commit removing last two zeroes—can you try it?

Compiling https://github.com/akrennmair/newsbeuter/commit/9e44cb564b31e21ecfc03582bc3b8a314e166a59 I get no errors or warnings and newsbeuter seems to function fine pulling feeds from https urls.  From what little I know, this seems to be a success. o/

authors: Earnestly Minoru

repo: akrennmair/newsbeuter


text:
[STYLE CHANGE] : Adjust the positioning of search bar on ISKCON Page

Description:
The searchbar area on the ISKCON Page looks very chaotic.
Current Styling:
[Describe the current styling that needs to be changed.]
Proposed Styling:
Margin and padding needs to be modified
Screenshots or Design Mockups (optional):

Please assign me this issue for hacktoberfest

Hi @akshitagupta15june , Please assign me this issue under JWoC 2024

Hi!! I am Anshdeep singh, contributor in GSSoC'24. I would like to address this issue and make the respective changes.


text:
[Feature Enhancement] Dark Mode for Blog Page (GSSoC'23)

Is your feature request related to a problem? Please describe.
The problem is that the blog page of a website is currently in white mode and not compatible with dark mode. The navbar, changes to dark mode but the background of the blog page doesn't.
Describe the solution you'd like
I would like the blog page of the website to be compatible with dark mode. This means changing the background color and font color of the blog page to a dark color that is suitable for viewing in dark mode.
Describe alternatives you've considered
One alternative solution would be to manually change the background color of the blog page to a dark color using custom CSS styles. Another alternative would be to use a CSS framework or library that provides built-in support for dark mode and can easily apply the necessary styles to the blog page.
Additional context
Dark Mode only changes the Navbar color

@akshitagupta15june Kindly assign this issue to me for gssoc'23!

I would like to work on this issue. Please assign me under GSSOC'23.

authors: CodeWizard-Ishu prernamittal


text:
Actions key being called but nothing happend

I'm using v3.40.1.
In the error part of axios interceptor I expect when I get for example 403 (refresh token has expired) to use Actions.sceneName() and to be rerouted to home page where user can login and get new tokens. Right now when I get to point where this Actions.sceneName() is being called in debug mode it just step over and scene is not being change and I get white screen. Can someone help me with this issue or point me where might be error?

Are you using redux-saga? You will probably yield call/put or dispatch this action to have it working.

Yes, I've already tried that but nothing happens. Action is dispatched and called but no rerouting to the login screen.

Do you mind showing me some code of how are you calling it inside your saga?

authors: daviscabral rahmanmarija

repo: aksonov/react-native-router-flux


text:
Can the Drawer slide the content right/left instead of overlaying it?

Version
"react-native-router-flux": "^4.0.0-beta.21",
"react-native": "0.48.4",
Expected behaviour
I want the content to be slid instead of being overlayed
Actual behaviour
Drawer overlays content
Steps to reproduce
import {
    Router,
    Scene,
    Stack,
    Drawer,
} from 'react-native-router-flux';

on the render side
                <Stack key="root" hideNavBar>
                  <Scene
                    key="login"
                    component={LoginScreen}
                    title="Login Screen"
                    />
                  <Drawer
                    hideNavBar
                    key='home'
                    contentComponent={Sidebar}
                    >
                    <Scene
                      key="todo"
                      component={TodoScreen}
                      title="Todo Screen"
                      />
                  </Drawer>
                </Stack>

Sidebar
export class Sidebar extends Component {
    render () {
        return (
            <View style={styles.container}>
              <Text style={{color: 'black'}}>
                Test
              </Text>
            </View>
        );
    }
}

hi bro, did you solve this?

Not with router-flux.
I switched entirely to react-navigation, which is what router-flux is based on.
If you're interested, here is what i did:
import React, { Component } from 'react';
...
import Router from './Router.js';
...
            <Provider store={store}>
              <Router/>
            </Provider>

Instead of Stacks and Scenes create your own router:
Router.js
import React, { Component } from 'react';
import { connect } from 'react-redux';

import { StackNavigator } from 'react-navigation';

import LoginScreen from './LoginScreen';
import ForgotPasswordScreen from './ForgotPasswordScreen';
import TodoScreen from './TodoScreen';
import TodoScreenWithTasks from './TodoScreenWithTasks';

const Main = StackNavigator(
  {
    Todo: {screen: TodoScreen},
    TodoWithTasks: {screen: TodoScreenWithTasks},
  }, {headerMode: () => null}
);

const Auth = StackNavigator(
  {
    Login: {screen: LoginScreen},
    ForgotPassword: {screen: ForgotPasswordScreen},
  }, {headerMode: () => null}
);

export const Router = StackNavigator(
  {
    Auth: {screen: Auth},
    Main: {screen: Main}
  }, {headerMode: () => null}
);

import { addNavigationHelpers } from 'react-navigation';

import {
  Sidebar,
  Colors,
} from './components';

@connect(
  ({ router }) => ({
    nav: router
  })
)
export default class App extends Component {
  constructor(props){
    super(props);

    this.state = {
      isOpen: false
    };
  }

  render() {
    return (
      <View style={{flex:1, backgroundColor: Colors.Primary}}>
        <Sidebar>
          <Router
            navigation={addNavigationHelpers({
              dispatch: this.props.dispatch,
              state: this.props.nav
            })}/>
        </Sidebar>
      </View>
    );
  }
};

Sidebar.js
import React, {Component} from 'react';
...

class Menu extends Component {
    render () {
        return (
            <View style={styles.container}>
              <Text style={{color: 'black'}}>
                Test
              </Text>
            </View>
        );
    }
}

import Drawer from 'react-native-drawer';

import {
  sidebarOpen,
  sidebarClose,
} from '../actions';

import { connect } from 'react-redux';

@connect(
  ({ sidebar: {
    isEnabled,
    isOpen
  } }) => ({
    isEnabled,
    isOpen
  }), {
    sidebarOpen,
    sidebarClose,
  }
)
export class Sidebar extends Component {
  render () {
    return (
      <Drawer
        open={this.props.isOpen}
        enabled={this.props.isEnabled}
        onOpen={this.props.sidebarOpen}
        onClose={this.props.sidebarClose}
        type="displace"
        content={<Menu />}
        tapToClose={true}
        openDrawerOffset={0.2}
        panCloseMask={0.2}
        negotiatePan={true}
        tweenHandler={Drawer.tweenPresets.parallax}
        >
        {this.props.children}
      </Drawer>
    );
  }
}

Of course the @connect decorator is optional just like the redux state handling the sidebar, you could use a singleton/global variable
The react-navigation documentation is pretty good IMO

So is there really no way we can add this animation (slide content to right) using react-native router flux ? Because my production app depends wholly on Flux and I don't want to make so many changes to the architecture just for the sake of adding one animation.

authors: LaloHao Mimicx kckunal2612


text:
Search field in navbar loses focus after animation

Version

react-native-router-flux v3.26.16
react-native v0.26.3

My problem is this - I'm using RNRF in an android app and I have a search <TextInput> in the navbar, which has autofocus set, so that after transitioning to the Scene, user can enter query right away. The problem is, the focus is lost upon the end of animation.
Expected behaviour
As the scene transitions, I'd like the keyboard appear and stay there.
Actual behaviour
While transitioning to the scene, keyboard goes up but upon the ed of animation, <TextInput> loses focus and keyboard gets hidden again. :/
Turning the animation off doesn't help.
Steps to reproduce

use a custom navbar through static renderNavigationBar(props)
put a textinput with autofocus in the navbar
call Actions.searchScene()

I would welcome any thoughts on how to solve this in a nicer way if possible, thanks.
Note I cannot easily access the component through this.refs because the navbar is rendered ins static context. Hm maybe not doing it static will help

solved by not using the static renderNavigationBar and focusing the field upon the end of animation

Hi what did you use instead of the static renderNavigationBar?

I just rendered the search field as part of the content.

Hi, my whole android app's TextInput has the same phenomenon, I would be very grateful if you'd elaborate me on this solution.  Thank You @vonovak

@MirGhojam I honestly don't remember. I switched to ex-navigation soon after trying out RNRF. The next, hopefully final switch, will be to react-navigation :)

@vonovak oh I see, Thanks anyway :)

authors: MirGhojam ajoshdee vonovak


text:
Add README encoding examples

Fixes #156

oops, forgot to request a review lol

oops, forgot to request a review lol

And I am on Java User Group conference, so missed the PR :)


text:
Disable auto-selection on first row?

Hi!
Is there a way to disable auto-selection and remove the possibility to select rows as well?
Didn't find it in the docu...
Cheers,
Greg

@kgrzesiu  do you find a solution?

@lopenchi No, still did not. A simple property for disable row selection would solve this, but I think there is none.

The property is st-select-row. add "st-select-row="fn(row)". If the row should be selectable have that function return the row.

authors: amfsyn kgrzesiu lopenchi


text:
TabView with ScrollView content overflows with height equal to the TabBar's

🐛 Bug Report

To Reproduce
Steps to reproduce the behavior:
Create a TabView with ScrollViews inside the Tabs on iPhone 11 with react-navigation-material-bottom-tabs on the bottom.
    <Layout>
      <TabView
        selectedIndex={selectedIndex}
        onSelect={setSelectedIndex}
      >
        <Tab title='A'>
          <ScrollView ... />
        </Tab>
        <Tab title='B'>
          <ScrollView ... />
        </Tab>
      </TabView>
    </Layout>

Expected behavior

The content should scroll up to the bottom tabs. Instead, it overflows below the bottom tabs and the bottom of the content and the bottom of the scroll bar are hidden below the bottom tabs.
Link to runnable example or repository (highly encouraged)

UI Kitten and Eva version

Package
Version

@eva-design/eva
^1.3.1

@ui-kitten/components
4.4.0-beta.2

Environment information

  System:
    OS: macOS 10.15.3
    CPU: (8) x64 Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz
  Binaries:
    Node: 10.18.0 - /usr/local/bin/node
    Yarn: 1.19.1 - ~/.yarn/bin/yarn
    npm: 3.5.3 - /usr/local/n/versions/0.12.4/bin/npm
  SDKs:
    iOS SDK:
      Platforms: iOS 13.2, DriverKit 19.0, macOS 10.15, tvOS 13.2, watchOS 6.1
    Android SDK:
      API Levels: 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26
      Build Tools: 19.1.0, 21.1.2, 23.0.1, 23.0.2, 23.0.3, 24.0.3, 25.0.0, 25.0.2, 26.0.2, 26.0.3, 28.0.2, 28.0.3
  IDEs:
    Android Studio: 3.2 AI-181.5540.7.32.5056338
    Xcode: 11.3.1/11C504 - /usr/bin/xcodebuild
  npmPackages:
    react-native: https://github.com/expo/react-native/archive/sdk-36.0.0.tar.gz => 0.61.4
  npmGlobalPackages:
    react-native-cli: 2.0.1

Can you please provide the full code? TabView is designed to be used as top tabs, and not bottom. For bottom tabs - there is a BottomNavigation.

I'm using them as top tabs. I use the mentioned package for bottom tabs.

@garrett-gottlieb sorry I can't review this now, can you please provide a full code of creating bottom tab navigator and screens for it?
Also, there is an option to configure TabBar with top navigator instead of using TabView if it will cover your use-case.

@garrett-gottlieb I'm unable to reproduce it by running almost same code. Any more details?
const renderItem = (info: ListRenderItemInfo<number>): React.ReactElement => (
  <Layout style={{ height: 200, backgroundColor: info.item % 2 === 0 ? 'red' : 'green' }}/>
);

<TabView>
  <Tab title='Tab 1'>
    <List data={[0, 1, 2, 3, 4]} renderItem={renderItem}/>
  </Tab>
  <Tab title='Tab 2'>
    <List data={[0, 1, 2, 3, 4]} renderItem={renderItem}/>
      </Tab>
</TabView>

https://i.imgur.com/u6CgwLM.gif

Im experiencing the same issue. I if you take a look at your imgur video, ull see that after scrolling to the end, if you keep scrolling you would revel one more item. whose behind the bottom tab. I think we are expecting to render all items inside the Tab content space.

@artyorsh Your example does reproduce the bug.  Part of the red box at the bottom of the list isn't visible when you scroll to the bottom.  If you just render the List component in your example, the entire portion of the red box is visible.
Here's a video of what I mean:
https://user-images.githubusercontent.com/12564184/124843158-8b278c00-df4e-11eb-94ee-217062597f70.mov
I am having the same issue when trying to render a list of users inside a tab view.  The last one or two users are always hidden when scrolling to the bottom of the list.  I've had to add a View wrapper around it, calculate the height of the view and subtract 75... I couldn't figure out where the extra space was coming from.
I tried setting the height of the tab bar to 50, and subtracting that from the height of the View I wrapped my List in, but that didn't seem to work.  Maybe the tab indicator is also taking up space??
This is what I am rendering inside my tab view, in order to see the last item when scrolling to the bottom of the list:
<View
  style={styles.container}
  onLayout={({ nativeEvent }) => setHeight(nativeEvent.layout.height - 75)}>
    <View style={{ height }}>
      <List data={users} renderItem={renderItem} style={styles.list} />
    </View>
</View>

Is there a better solution for this?

authors: artyorsh danielanthonyl garrett-gottlieb nyoung697

repo: akveo/react-native-ui-kitten


text:
米家智能插座2 蓝牙网关版功率显示异常

6W 的功率会被显示成 0.06W

实际上是 5W

https://github.com/al-one/hass-xiaomi-miot/issues/1253#issuecomment-1676666715

repo: al-one/hass-xiaomi-miot


text:
micloud 0.3 package requirement conflicts with home assistant

Home Assistant now includes xiaomi_miio component as standard, this has a package requirement of micloud 0.4 since https://github.com/home-assistant/core/commit/e34aed743cf1a1448166a990bfdeefd654f0a452
xiaomi_miot requires 0.3 exactly (not >=)
https://github.com/al-one/hass-xiaomi-miot/blob/68815169735fd5db4435a6ce02dfd919bd1e2524/custom_components/xiaomi_miot/manifest.json#L17
Because xiaomi_miio component is now built in it will fail to initialise the component even if the component is not used spamming as 0.3 does not contain the MiCloudAccessDenied exception class.
Logger: homeassistant.config_entries
Source: config_entries.py:711
First occurred: 16:10:40 (12 occurrences)
Last logged: 16:21:58

Error occurred loading configuration flow for integration xiaomi_miio: cannot import name 'MiCloudAccessDenied' from 'micloud.micloudexception' (/usr/local/lib/python3.9/site-packages/micloud/micloudexception.py)

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 709, in async_create_flow
    integration.get_platform("config_flow")
  File "/usr/src/homeassistant/homeassistant/loader.py", line 530, in get_platform
    cache[full_name] = self._import_platform(platform_name)
  File "/usr/src/homeassistant/homeassistant/loader.py", line 535, in _import_platform
    return importlib.import_module(f"{self.pkg_path}.{platform_name}")
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/src/homeassistant/homeassistant/components/xiaomi_miio/__init__.py", line 72, in <module>
    from .gateway import ConnectXiaomiGateway
  File "/usr/src/homeassistant/homeassistant/components/xiaomi_miio/gateway.py", line 6, in <module>
    from micloud.micloudexception import MiCloudAccessDenied
ImportError: cannot import name 'MiCloudAccessDenied' from 'micloud.micloudexception' (/usr/local/lib/python3.9/site-packages/micloud/micloudexception.py)

Thanks! It is fixed in the latest commit: https://github.com/al-one/hass-xiaomi-miot/commit/427417a284b90ca2b065aa2ca7cc40f0834a680b


text:
Changing refresh/update interval on Air Purifier sensor

Is it possible to change the update interval on Xiaomi Air Purifier devices?
The Pm2.5 sensor is getting updated roughly every 15 minutes which is a really long time and I feel like I am not getting a realistic number in home assistant because of this. I would like a much lower interval, maybe ~1 minute.
Thank you

Did you manage to fix it? Im Struggling with the same. sadly i dont understand the above advice to customize the device (im new)

authors: GergelyMagyar Tinteex


text:
Error In Targeting Loop

Hello - relatively newbie here, recently downloaded the full repository on a clean install and am giving it a shot in BN8.1 (Ghost of Wall Street).
After launching daemon.js either with or without arguments, I shortly get the warning:
"daemon.js: WARNING: Caught an error in the targeting loop: TypeError: Cannot read properties of undefined (reading 'name')"
This error pops continuously every 20 seconds or so, and it's not clear to me if it's a serious issue or even what it's trying to tell me.  Since 8.1 restricts all profit except from stock trading, I'm wondering if it's an issue with not buying or using player servers or hacknet nodes?
stockmanager.js appears to be functioning normally - no issues there
Any idea what I might be doing wrong to cause this error?  I have not purchase or delete any servers on this bitnode - certainly not while the script was running.
Thanks!

Hello - relatively newbie here, recently downloaded the full repository on a clean install and am giving it a shot in BN8.1 (Ghost of Wall Street).
After launching daemon.js either with or without arguments, I shortly get the warning:
"daemon.js: WARNING: Caught an error in the targeting loop: TypeError: Cannot read properties of undefined (reading 'name')"
This error pops continuously every 20 seconds or so, and it's not clear to me if it's a serious issue or even what it's trying to tell me. Since 8.1 restricts all profit except from stock trading, I'm wondering if it's an issue with not buying or using player servers or hacknet nodes?
stockmanager.js appears to be functioning normally - no issues there
Any idea what I might be doing wrong to cause this error? I have not purchase or delete any servers on this bitnode - certainly not while the script was running.
Thanks!

@cullumar Yeah, the bug was introduced in one of the recent commits, but it has been reverted for now, until a proper patch is going to be applied. Just re-download the daemon.js, and it should work just fine.

Can confirm - I think I just got very unlucky with my download timing.  Re-downloading daemon.js appears to have fixed the issue.  Thank you!

authors: Randshot cullumar

repo: alainbryden/bitburner-scripts


text:
scenario generation

Hello!
I would like to know if there is a sample csv for scenario generation? I tried using csv files on my own, but there is an error while looking up the table in overflier_climber.py file.

Hi! I've had a look but I don't see any csv files included in the repo.
If you run the tests from the develop branch though, I can see that some sample files are produced (although I think they contain dummy data).
$ pytest
...

$ find ./aviary -name "*.csv"
aviary/test/integration/scripts/cruise_speed.csv
aviary/test/integration/scripts/climb_time.csv
aviary/test/integration/scripts/downtrack_distance.csv

@thobson88 do you have any suggestions?

@arunbalas @rkm Apologies - I haven't got around to adding the csv files to the public repo yet.
To explain:

the full lookup tables are proprietary so can't be shared publicly
the tests on the develop branch use dummy data (from aviary/test/conftest.py)
my intention is to create some new sharable tables, more representative of the original ones but containing dummy data. These will contain the full list of flight levels and a small number of different aircraft types (probably two).

I'll create the new tables later today and post here when they're available.

Oh ok, thank you for the response. Because, when I ran pytest, the sample csv files were not generated in my system, so I was just curious...

I've added some approximate lookup tables in the aviary/resources directory, plus an explanatory note in the README (all on the develop branch).
The documentation is still minimal, but I hope their meaning can be understood. See the comments in the LookupTrajectoryPredictor class. Also note that the physical units can be inferred from the CSV filenames.
Let me know if anything is unclear.

I haven't written any code to read the tables but they can be passed directly to the overflier_climber.py script. (I haven't actually tested this but they are the same format as the proprietary tables, so should work.) Please feel free to update the LookupTrajectoryPredictor class to ingest them if you like.

authors: arunbalas rkm thobson88

repo: alan-turing-institute/aviary


text:
[ENH] Save/Load Functionality for both Classical and DL Estimators

Reference Issues/PRs

Fixes #3022
What does this implement/fix? Explain your changes.

The Save/Load function for classical estimators is from #3336 and has been expanded for DL models along the same design. Keras is known to be problematic for certain kinds of (de)serialization, so this uses model.save() from keras along with pickle.dump(). The result is a directory, as passed inside our_model.save(folder_loc).
Components of this directory include:

Metadata
Object
Keras History
Keras Model subdirectory

The first two components are the same as previously mentioned PR, stored differently to avoid this problem. This PR does not attempt to save everything under a single file and is left for a future PR.
All the attributes are saved and restored (optimizers/history etc.) appropriately.
Does your contribution introduce a new dependency? If yes, which one?

No
What should a reviewer concentrate their feedback on?

One thing in __getstate__() and __setstate__() might be a source of contention, I've explained why it was necessary and I believe it is the simplest solution.
Any other comments?

PR checklist

For all contributions

[x] The PR title starts with either [ENH], [MNT], [DOC], or [BUG] indicating whether the PR topic is related to enhancement, maintenance, documentation, or bug.

Following is a small code-snippet to showcase how "states are preserved" / "contract is maintained":
import sktime.classification.deep_learning as dl
from sktime.base import load, load_dl
import numpy as np
from pprint import pprint
from pathlib import Path

if __name__ == "__main__":
    save_folder_location = "temp_saved"
    train_X = np.random.randn(15, 24, 16)
    train_y = np.random.randint(0, 2, size=(15, ))
    test_X = np.random.randn(5, 24, 16)

    cnn = dl.cnn.CNNClassifier(n_epochs=1)
    cnn.fit(train_X, train_y)

    cnn.save(save_folder_location)

    new_cnn = load_dl(save_folder_location)

    pprint(cnn.__dict__)
    pprint(cnn.model_.optimizer)
    pprint(cnn.optimizer)
    pprint(cnn.optimizer_)
    pprint(cnn.history)

    print("--" * 50)

    pprint(new_cnn.__dict__)
    pprint(new_cnn.model_.optimizer)
    pprint(new_cnn.optimizer)
    pprint(new_cnn.optimizer_)
    pprint(new_cnn.history)

    print("Does both give same result to same data?", np.allclose(
        cnn.predict(X=test_X),
        new_cnn.predict(X=test_X)
    ))

The output:
{'_X_metadata': {'has_nans': False,
                 'is_empty': False,
                 'is_equal_length': True,
                 'is_equally_spaced': True,
                 'is_one_series': False,
                 'is_univariate': False,
                 'mtype': 'numpy3D',
                 'n_instances': 15,
                 'scitype': 'Panel'},
 '_callbacks': [],
 '_class_dictionary': {0: 0, 1: 1},
 '_estimator_type': 'classifier',
 '_is_fitted': True,
 '_network': CNNNetwork(),
 '_tags_dynamic': {},
 '_threads_to_use': 1,
 'activation': 'sigmoid',
 'avg_pool_size': 3,
 'batch_size': 16,
 'callbacks': None,
 'classes_': array([0, 1]),
 'fit_time_': 3424,
 'history': <keras.callbacks.History object at 0x000001F77ECD9BB0>,
 'input_shape': (16, 24),
 'kernel_size': 7,
 'label_encoder': LabelEncoder(),
 'loss': 'mean_squared_error',
 'metrics': None,
 'model_': <keras.engine.functional.Functional object at 0x000001F778B355E0>,
 'n_classes_': 2,
 'n_conv_layers': 2,
 'n_epochs': 1,
 'onehot_encoder': OneHotEncoder(sparse=False),
 'optimizer': None,
 'optimizer_': <keras.optimizers.optimizer_v2.adam.Adam object at 0x000001F778B95760>,
 'random_state': None,
 'use_bias': True,
 'verbose': False}
<keras.optimizers.optimizer_v2.adam.Adam object at 0x000001F778B95760>
None
<keras.optimizers.optimizer_v2.adam.Adam object at 0x000001F778B95760>
<keras.callbacks.History object at 0x000001F77ECD9BB0>
----------------------------------------------------------------------------------------------------
{'_X_metadata': {'has_nans': False,
                 'is_empty': False,
                 'is_equal_length': True,
                 'is_equally_spaced': True,
                 'is_one_series': False,
                 'is_univariate': False,
                 'mtype': 'numpy3D',
                 'n_instances': 15,
                 'scitype': 'Panel'},
 '_callbacks': [],
 '_class_dictionary': {0: 0, 1: 1},
 '_estimator_type': 'classifier',
 '_is_fitted': True,
 '_network': CNNNetwork(),
 '_tags_dynamic': {},
 '_threads_to_use': 1,
 'activation': 'sigmoid',
 'avg_pool_size': 3,
 'batch_size': 16,
 'callbacks': None,
 'classes_': array([0, 1]),
 'fit_time_': 3424,
 'history': <keras.callbacks.History object at 0x000001F7022DC430>,
 'input_shape': (16, 24),
 'kernel_size': 7,
 'label_encoder': LabelEncoder(),
 'loss': 'mean_squared_error',
 'metrics': None,
 'model_': <keras.engine.functional.Functional object at 0x000001F70113F850>,
 'n_classes_': 2,
 'n_conv_layers': 2,
 'n_epochs': 1,
 'onehot_encoder': OneHotEncoder(sparse=False),
 'optimizer': None,
 'optimizer_': <keras.optimizers.optimizer_v2.adam.Adam object at 0x000001F70300C0A0>,
 'random_state': None,
 'use_bias': True,
 'verbose': False}
<keras.optimizers.optimizer_v2.adam.Adam object at 0x000001F70300C0A0>
None
<keras.optimizers.optimizer_v2.adam.Adam object at 0x000001F70300C0A0>
<keras.callbacks.History object at 0x000001F7022DC430>
1/1 [==============================] - 0s 253ms/step
1/1 [==============================] - 0s 174ms/step
Does both give same result to same data? True

@AurumnPegasus  I would also like for you to code-review this PR and give some suggestion for possible tests.

Hey @achieveordie ,
Thanks for picking this up! I was not able to find a solution to adding keras history and loading it from Zip file (from Franz's PR #3336 ) and am glad you did.
In case they would be helpful, I had a design doc in work for the same as a STEP Document, though as mentioned above, there were some solutions I was not able to find.
I went through your PR and I think the code is good. I agree with the __getstate__ and __setstate__ point, and think that it is neccessary (generally required because you cannot pickle lambda functions, and keras.optimizers use them in there implementation, and hence there is no way around it.) I like the idea of having a seperate load_dl as well (since I think working with Zip files was proving troublesome for DL)
About the tests, there were a few I had in mind:

Idempotent test: Compare the predictions of the fitted model before and after saving the model. This will ensure that the model loaded is the one which was saved.
This is an existing test which had been ignored for all DL Estimators because we were not getting exact results before and after saving (this was done before optimizer was added as a parameter, and hence we could work this out with pickle). The reason it was ignored was although the answers were very similar, they were not able to satisfy the threshold for similarity (which was around 1e-6) due to randomness of DL Estimators.
Saving with other parameters: Although I do not think any of the current parameters will give this issue, I still think we should test by sending some value of all parameters (and not taking None in any of them) so as to check they dont give the lambda issue optimizer gives us. It might be more useful for a future use case where someone adds a parameter which uses the lambda function.

Thanks for the review!
My only problem with a separate load_dl() was that we'd have to modify tests to call load_dl() instead of load() for relevant cases, not to mention it might cause a bit of confusion on the user's end.
The easiest solution that I can think of is if we modify the code to only have load() as an entry point which would either call _load_classical() or _load_dl() depending if path is a file or a directory, or simply add another parameter where the user passes in that information.
I think @fkiraly would be against the latter solution since it would require the user to have some pre-information about the model.
As for tests for the current implementation, I agree with your points. However, the idempotent test seems to be testing for fitting model twice as opposed to save/load. I might be wrong since I just glanced at it. For the second point, I was wondering if there is already an implementation that I could look at in the existing tests?

The easiest solution that I can think of is if we modify the code to only have load() as an entry point which would either call _load_classical() or _load_dl() depending if path is a file or a directory

This looks quite easy to do, and should not require user to know anything about either the estimator or the saved file/directory. Maybe the python command os.path.isdir(path) would be of help? I am sure it works as required for unix-like systems, and unsure about windows since I have not used it on that.
This would allow an easy (although "hacky") solution to the problem

The easiest solution that I can think of is if we modify the code to only have load() as an entry point which would either call _load_classical() or _load_dl() depending if path is a file or a directory

This is the same idea that I was pursuing, I was just attatching these as overrides to the class they are relevant for.
So, yes, but _load_classical already is in the base class in my design, as methods load_from, etc, and _load_dl should simply override the load_from_ methods in the neural network base class.
load should be universal and not require extension or modification - if at current state it does, then it means we need to move more in the class specific loaders.

If we add new estimators that require a different logic than already provided in base classes(having custom objects for example), then the author would have to expand the class method load_from_path()  / load_from_serial(). The same is true for your design. The private functions that I've added are simply modularized, done to reduce boilerplate code. These external functions call class methods similar to your original design.

Hm, can you explain why you added more code in load then?
My concern is that subject to your design, the same might be necessary if, say, we add a pytorch adapter.
Though I admit I have perhaps not fully understood how you see the extension contract, so I'm happy to be convinced that this is the last time something has to be added to load.

I've updated the STEP to clarify what I had in mind:
https://github.com/sktime/enhancement-proposals/pull/27

Perhaps we don't understand each other. The additional code is due to the fact that non-DL and DL estimators are being stored in different ways. For the sake of discussion, if we were to save non-DL estimators in a directory (not zip them), then we can get rid of _load_classical() and _load_dl() functions since they'd have the same workflow.
Going from zip to directory-based for non-DL estimators seems easy, whereas going from directory to file-based storage (for DL estimators) needs more exploration. It might be easy/difficult/impossible. We need to discuss this further.

Perhaps we don't understand each other.

Yes, I suppose so.

The additional code is due to the fact that non-DL and DL estimators are being stored in different ways.

I'm not saying "there should not be additional/different code for DL estimators".
I'm saying "there will likely be different/additional code, and it should live in the base class of the estimator, not in the load function, because we want to go with the strategy object orientation pattern".

question: is there a way to avoid the creation of the temp folder?
This may upset virus scanners etc.

question: is there a way to avoid the creation of the temp folder? This may upset virus scanners etc.

Wouldn't this be a problem with keras's folder creation too? Or it's problematic because the code creates AND deletes it?

Or it's problematic because the code creates AND deletes it?

I was just thinking to minimize file write/delete actions, as heuristic detection may find more steps more odd, but that's just a gut feeling.

hm, seems like save/load is failing when you try to run it for some deep learning estimators - could you kindly have a look?

It was based on the implementation of #3336, not branched from it - sorry I didn't explicitly mention that.
I had removed the tests here since they are being worked on in #3413. You'd see similar error traces in job failures in both cases. Looking through it, I sense that there's a problem with how scenarios run. Since there are plenty of files and you have authored them, I think we can discuss why it's failing tomorrow in the meeting. I strongly suspect something is not working the way it's supposed to in the _testing/ directory. The error indicates .fit() was probably not called before the proceeding method.
PS: there's also a doctest failure.

I strongly suspect something is not working the way it's supposed to in the _testing/ directory.

Ok, how does that explain then that everything on main (i.e., the PR #3336) is passing?

I could be wrong but the reason why #3336 passes and this doesn't might be because all (non-DL) estimators comply with expected test specifications. In contrast, DL estimators probably don't because the tests were written with non-DL estimators in mind.
You'd say the functionalities should be altered to satisfy tests and not the other way around - and you'd be right. I'm saying that there could be certain assumptions in tests which are not true DL estimators. If that's the case we would have to update the testing strategy, assuming implementation is correct for DL estimators (which, after manual testing, I think it is).
In any case, I'll try to come up with specific questions regarding how scenarios work by tomorrow and that could help us identify the problem.

I see we've been discussing two different tests here. Let's break the conversation down:
There are two 'problematic' test functions that we were initially skipping - test_persistence_via_pickle() & test_fit_idempotent(). You were talking about the former while I was talking about the latter.
Easy to debug first: test_persistence_via_pickle() is failing because the default in-memory serialization from #3336 doesn't take into account foreign objects that need to be serialized, in our case that is the keras model, hence the error. I did look around to try and find a way to serialize keras model into memory and found this. Kindly look into it and LMK.
test_fit_idempotent() needs discussion, will update the comment post that.

Yes, I think you are right on two points that I missed:

there should be a test for save/load from file, indeed - we do not cover that properly! I agree we need to.
I missed that test_fit_idempotent was failing for another reason. We should probably rewrite this in terms of save/load instead of using deepcopy.

for test_persistance_via_pickle, I think we can leave the test as is (but we should probably rename the test, this may require deprecation since users may be referring to it via check_estimator), and we should implement the save/load interfaces against it.

So here's what I propose - we do not change test_persistence_via_pickle, just update the doc to explicitly mention that we are checking for in-memory serialization. We add a test_file_persistence_via_pickle where estimators are saved to files. This way we avoid deprecation.

I did look around to try and find a way to serialize keras model into memory and found this. Kindly look into it and LMK.

As the first step, I'll start updating the implementation as per the link, which should resolve test_persistence_via_pickle() (and the new test_file_persistence_via_pickle).
After that, we can update the runner to solve test_fit_idempotent(). I'll also close #3413 since that PR was aimed at fixing this.
LMK if this sounds good so I can start.

yes, sounds good!
Minor comment, I would avoid using pickle in the test name. The old test requires deprecation since it might be in use (let´s do that separately), but we can avoid use of the term pickle in the new test.

@fkiraly When we're saving to memory, I don't think it's possible to serialize two objects into a single variable, so we'd have to modify the in-memory serialization as:
if path is None:
    type(self), pickle.dumps(self), None

for non-DL estimators. The last variable above will store the serialized keras model.
However, if you think this is suboptimal or breaks any user contract, I'd like to hear a different approach to do this.

I'm making an early commit to explain how keras serialization is supposed to work for in-memory.
As explained in my comment above, we'd now return a tuple of size three, with the first two elements same as before. The third element will be None for non-DL estimators whereas it'll be a tuple of size two for DL estimators.
The first element will be in-memory serialized keras model produced via h5py whereas the second element will be in-memory history produced via pickle.dumps().
The following cases now work:

Manually loading the in-memory-saved keras model:

from sktime.classification.deep_learning import CNNClassifier
import h5py
from keras.models import load_model
import numpy as np

X = np.random.randn(15, 28, 16)
y = np.random.randint(0, high=2, size=(15, ))

cnn = CNNClassifier(n_epochs=1)

cnn.fit(X, y)

# Manual loading check
a, b, c = cnn.save()

# Only care about seeing if keras-model
# loads correctly, so skip a, b, c[1]
with open("temp_file", 'wb') as raw_file:
    raw_file.write(c[0])

with h5py.File("temp_file", 'r') as h5file:
    loaded_model = load_model(h5file)

X = X.transpose([0, 2, 1])
print(
    np.allclose(
        cnn.model_.predict(X),
        loaded_model.predict(X)
    )
)
# True

Using the dedicated load from sktime:

from sktime.classification.deep_learning import CNNClassifier
import numpy as np
from sktime.base import load

X = np.random.randn(15, 28, 16)
y = np.random.randint(0, high=2, size=(15, ))

cnn = CNNClassifier(n_epochs=1)

cnn.fit(X, y)

serial = cnn.save()
loaded_cnn = load(serial)

print(
    np.allclose(
        cnn.predict(X),
        loaded_cnn.predict(X))
)
# True

As explained in my comment above, we'd now return a tuple of size three, with the first two elements same as before. The third element will be None for non-DL estimators whereas it'll be a tuple of size two for DL estimators.

This is violating the strategy pattern - you are changing the base interface to accomodate for DL estimators. Again, think about the extension case. If we add a third estimator type (e.g., super-DL), would you extend the tuple size to four?
A simple solution to keep interface homogeneity is to return a tuple of size two, and the second element can be a tuple of size two. Also has three elements, but the general format is the same ("what is in the second element" being type specific).

perhaps a very stupid question: is there a way to avoid the h5py dependency? I thought HDF5 was outdated for tensorflow, is there a way to use SavedModel?

ps: if you want the tests to run, you need to add the soft dependency (also, please isolate it)

A simple solution to keep interface homogeneity is to return a tuple of size two

Sure thing, I'll modify the code to do that.

is there a way to use SavedModel ?

I'm not sure. SavedModel is a tensorflow exclusive whereas h5py has a more extensive scope and dedicated support. Plus SavedModel makes a directory so we'd have to deal with serializing multiple objects in memory and then make sure they communicate the same way they do when stored on disk.

(also, please isolate it)

How do I do that? I didn't update the setup since I just wanted to show how it'll work. Will add the softdeps and other things today.

Things added:

in-memory serialization of deep estimators.
test to save an estimator as a file.
in-memory serialization of deep regression estimators.

Tests should still fail either because of faulty individual estimators (see #3538 and #3539) or because of current test_fit_idempotent() implementation(see comment).
I suggest excluding these again (was removed during this PR) and work on them separately.

because of current test_fit_idempotent() implementation

Should we change that to use deepcopy, or save/load? (not sure which one would be better)

Tests should still fail either because of faulty individual estimators

I'd separate fixing these from the general framework issue, i.e., except them in the _config and link to the issue number in a comment

Should we change that to use deepcopy, or save/load? (not sure which one would be better)

AFAIK currently, deepcopy is being used under the hood for test_fit_idempotent. We'd have to diagnose why exactly it's failing. I'll add an issue regarding it in the upcoming days.

except them in the _config and link to the issue number in a comment

Could you please add the exceptions in the configuration file? I think all the current failures are to be excluded because these are problems with the test/estimator rather than save/load interface. But I might be wrong and so it'll be great if you could confirm it.

Could you please add the exceptions in the configuration file? I think all the current failures are to be excluded because these are problems with the test/estimator rather than save/load interface.

I added excepts for the failures that are covered by currently linked issues, to the best of my knowledge.

But I might be wrong and so it'll be great if you could confirm it.

I do think the FCNClassifier failure is not covered by any known issue, and in fact the tests pass on main.

test_fit_idempotent() was failing for FCNClassifier. Since we know that this test is problematic, how come it passed in the main branch?

When this PR was created, FCNClassifier was skipping test_fit_idempotent() like other DL estimators, as seen in the files changed. When was it added back?

When this PR was created, FCNClassifier was skipping test_fit_idempotent() like other DL estimators, as seen in the files changed. When was it added back?

Don't know - probably a side effect of copy-pasting instead of branching. If it was skipped on main, feel free to add it back.

quickly looked into it - it's not the merging as I claimed, but we removed the skipped tests for deep learning estimators in order to test the new save/load for deep learning.
So, this would be a genuine failure of the new interface, but it seems to be estimator specitic?

This is the output from the latest run. All failures in CNNRegressor is due to the known issue #3538. There seems to be a genuine problem in MLPClassifier for test_fit_idempotent. The outputs are close but they don't satisfy the criteria of decimal=6. This is probably sporadic, we might have to look more into it.

can you please explain what is going on with CNNRegressor, to the best of your understanding?
Re MLPClassifier, does that mean the random_state parameter does not work properly?

My comment here explains the problem with CNNRegressor.
AFAIK, All other failures have test_fit_idempotent() as a common denominator. There seem to be two main reasons why this is failing.

Type-1: Assert failures are sporadic failures that occur when the idempotent property of a method doesn't satisfy the decimal accuracy we want.
Type-2: Attribute failures are due to (faulty?) implementation of the test function. Under the hood, deepcopy is being employed to store parameters which in turn uses __getstate__(), similar to pickle. This is problematic because the function during which this is called is fit, and at that point of execution, self.model_ isn't defined, hence we get an error. It's weird that this isn't happening always. One would probably need to understand how deepcopy works, but this is what I get from those errors.

My comment here explains the problem with CNNRegressor.

This links to this very PR, not a specific comment. Do you mean this issue? https://github.com/sktime/sktime/issues/3538

Type-1: Assert failures are sporadic failures that occur when the idempotent property of a method doesn't satisfy the decimal accuracy we want.

In theory, this should not be happen - if a random_state is set, the same fit should lead to the same output with high decimal accuracy. That indicates to me that random_state is not working as intended.

Type-2: Attribute failures are due to (faulty?) implementation of the test function. Under the hood, deepcopy is being employed to store parameters which in turn uses getstate(), similar to pickle. This is problematic because the function during which this is called is fit, and at that point of execution, self.model_ isn't defined, hence we get an error. It's weird that this isn't happening always. One would probably need to understand how deepcopy works, but this is what I get from those errors.

This sounds odd, I thought deepcopy worked with any python object, including those that are not pickleable. But I may be wrong.
In this case, would replacing deepcopy with save or load work?
Either way, we need to reduce the problem to minimal code, e.g., applying deepcopy to a fitted estimator, or our understanding will remain obscure and based on conjecture.
You could test yours by applying deepcopy to a suitable fitted estimator and seeing whether it breaks in the same way?

From https://github.com/sktime/sktime/issues/3538, it seems to become apparent that you had assumed in your implementation of save/load that all deep learning estimators have an optimizer attribute.
I don't think that is a good assumption to make? It is certainly not a documented interface assumption. Or do we expect all deep learning estimators to have an optimizer attribute? If yes, why?

I'll make the changes where the code doesn't expect optimizer to be present for a DL estimator. It seemed reasonable to me that all DL estimators have an optimizer since we employ compile from keras in our current interface. Even if this interface changes radically in the future (say the user has the ability to define their own custom training step), optimizer would still be pivotal, unless the interface stops using tensorflow altogether.
Besides, I don't think there is any contention in the fact that CNNRegressor should have an optimizer parameter in its constructor. If not for the current implementation, this minor bug would probably have gone unnoticed for longer.

Besides, I don't think there is any contention in the fact that CNNRegressor should have an optimizer parameter in its constructor. If not for the current implementation, this minor bug would probably have gone unnoticed for longer.

Ok - just to spell this out, I'm less concerned about individual estimators than making this an interface expectation on all deep learning estimators.
If we want to make optimizer an interface expectation on all deep learning estimators, it should be a documented interface point in the base classes, perhaps even a property.

After taking care of the issue with optimizer, there are two kinds of failures:

TapNet failures are due to the known issue #3539
Failures from the problematic test_fit_idempotent()

If you think that either of them is in the scope of this PR, LMK, otherwise one can skip these tests and work on it in another PR.

@fkiraly It seems that the only errors remaining are the AssertionError from test_fit_idempotent(). Could you look into them and tell me how you would like them to be handled? If it is beyond the scope we'd be better off creating an issue, if not then we'd have to look more into it.
Do leave some comments about the added tests for base deep classifiers, these tests were meant to check if  __getstate__() and __setstate__() work properly in edge cases or not. I might add more tests in future.

Thanks! looks great!
Did you check (by deactivating --matrixdesign locally) whether TapNet and FCN are the only estimators affected?

@fkiraly It seems that the only errors remaining are the AssertionError from test_fit_idempotent().

These look like genuine bugs to me, i.e., random_state not working properly for the two estimators TapNet and FCN.
I would open two bug issues, skip the tests in _config, and reference the bug issues in the skip comment.

Do leave some comments about the added tests for base deep classifiers, these tests were meant to check if  __getstate__() and __setstate__() work properly in edge cases or not. I might add more tests in future.

Tests look good to me!
Stupid question: now that we got set/getstate to work, and therefore pickle, do we really still need a custom save/load for deep learning estimators? I might be missing something obvious.

Did you check (by deactivating --matrixdesign locally) whether TapNet and FCN are the only estimators affected?

If you remember in one of the conversations I mentioned that my system fails to run the entire suite. It just hangs till all the worker crash. So I couldn't try this out. I think that all the deep estimators are affected (see CNNClassifier failing on this in the latest ubuntu runner).

deal somehow with the issue that test suite should not break

I don't know the feasibility of doing this here's a possible solution - We create a fixture which runs only when certain soft-deps are included in the built. Otherwise, the tests in this fixture are just skipped. For now, we might have to skip all those tests regardless.

do we really still need a custom save/load

Yes. We are deleting the most essential part of the object - the keras model - before pickling. So the custom save/load ties up the work that __getstate__() / __setstate__() is doing (by using pickle) along with saving the model/history/optimizer associated with that model.

So if I were to re-iterate, our revised remaining items will be:

Skipping tests that require soft dependency till there's a solution for that.
Skip all the deep estimators for test_fit_idempotent().
Add ImportError in case soft dep h5py is missing during import

If we agree, then I can move forward and make these changes in the next commit.

Skipping tests that require soft dependency till there's a solution for that.

Not sure what you mean by this, can you explain?

Skip all the deep estimators for test_fit_idempotent().

I thought with getstate solved issus for most. Are we having AssertionError for all deep learning models?
That would mean that random_state is not working at all for these, no?

Add ImportError in case soft dep h5py is missing during import

Yes

Not sure what you mean by this, can you explain?

As you said that user's test suite may not have soft dependencies installed. So we can skip those tests. Specifically, any test that uses h5py (so in-memory serialization tests etc). I don't have a concrete idea if that's the best thing to do or not, but I'll leave that decision-making to you.

I thought with getstate solved issus for most. Are we having AssertionError for all deep learning models?

The updated getstate solves the Attribute Failure. AssertionFailure seems more of an issue with the test function / scenario rather than the estimators, but that's just my initial hypothesis. In any case, it is an entirely different issue and is not concerned with save/load code.

@ilkersigirci thanks for the much-needed input. I think we were expecting to find some bugs since this has just been implemented. What I would suggest is that we should tackle this on a new issue. So maybe open a new issue (possibly with reproducible code) and let's take on it from there.
Thanks!

Ofc, I will open a bug issue. BTW, in the sktime.BaseObject instead of using pickle, if we use cloudpickle or dill, all of this issues will be solved. This is because, we won't need to modify __get__state__ and __set__state__. cloudpickle and dill can successfully serialize optimizer and lambda functions. Hence, I am in favor for this approach which changes pickle as discussed in #3672

authors: AurumnPegasus achieveordie fkiraly ilkersigirci

repo: alan-turing-institute/sktime


text:
Why we need to use number for createdAt in BaseEntity class

Why we use number instead of Date Object or Datetime string?
Any explaination on this?
Thank you

hey @galih56
This was because the dates are stored as timestamps, which are numbers instead of dates. Both options are valid, it would be up to the API implementation how to implement it.

If i want to use Datetime string to serve date from an endpoint, i must convert the dates into Date Object for every fetch?
For now, i have to make converter that i use in axios interceptor. But i'm still curious to ask this question

If you really want dates as date objects, consider treating them that way at the API level, so you don't do any conversion on the FE.
But if you don't control the API data and really want it, then transforming the response makes sense. In that case, you can also consider using zod for transforming and validating your responses.


text:
dequantize op

The matlab and python dequantize op assumes the input has been convolved and therefore multiplies by the scale factor twice.
The functions should be changed to only multiply the scale factor once.
Other code should compute the new range after convolution outside the dequantize op function

Dequantize and quantize ops have been fixed.  They should be implementing the same algorithm as the tensor flow quantize ops for arbitrary ranges.


text:
MQTT boolean conversion issue

Version: v3.15.2
Configuration
Teslamate MQTT
Describe the bug
Boolean entities are not updated correctly from MQTT (e.g. sentry mode, is_climate_on). I think it's due to ReceiveMessage returning msg.payload as a string, which is then casted to bool. But this doesn't work since bool("false") or even bool("False") always returns True.

That's exactly what's the problem, thanks for figuring that out. I've added the following method to teslamate.py:
def cast_bool(val: str) -> bool:
    """Convert bool string to actual bool"""
    return val.lower() in ['true', 'True']

and changed the MAP_*_STATE maps to use that instead. That makes my lock entity change state flawlessly whenever Teslamate returns a changed value. I'm also intending to update some other entities that are currently not updated from MQTT using the same mechanism and then add a PR here.

Thanks for reporting. Happy to take a PR to resolve this.

authors: alandtse cguedel oakedcorn

repo: alandtse/tesla


text:
Better display of blank nodes in default interface

It is hard to understand the data when there are lots of blanks nodes in it.
Idea: Provide a services that can be called in a blank node to retrieve data via javascript and display it upon request

or build the initial query to handle getting the extra step past the bnodes.

authors: alangrafu timrdf

repo: alangrafu/lodspeakr


text:
callback seems not work, i must use settimeout() .

targz().extract('/bkp/backup.tar.gz', '/home/myuser', function(err){
if(err){
console.log('Something is wrong ', err.stack);
callback err.stack
}
console.log('Job done!');
callback();
});

Facing the same issue. Call back gets called after the first file is extracted. (vs. after the whole archive is extracted). Had to use setTimeout to solve this.

Had the same issue, I eventually switched to the 'decompress' module since there was no response to this issue after a month.

Having the same issue. setTimeout as a workaround kinda works, but I will have to look for another package until this is fixed.

Hello folks, sorry for the delay. It seems like fstream is not triggering the event I wanted, I'll take a look as soon as I can

Thanks for taking a look at this. Once it's fixed I will jump back to your package.

Having same issue. setTimeout is temporary solution. I'm just curious, so please correct me if I'm wrong - problem is that all input stream is finished(in this stage we get our callback executed) but output is not yet done?

One place to look might be that the underlying write stream that handles the finish event (https://github.com/alanhoff/node-tar.gz/blob/master/index.js#L158) is created in another method (https://github.com/alanhoff/node-tar.gz/blob/master/index.js#L154), and in that method, the write stream that is returned is actually the initial write stream that decompresses the archive, not the write stream that extracts the decompressed contents (https://github.com/alanhoff/node-tar.gz/blob/master/index.js#L62).
I'm wondering if this issue isn't caused by a brief 'blip' between the time the first stream has completed and the second one has completed, which would explain why setTimeout seems to help by providing a buffer (and some other bug reports that indicate the archive sometimes appears to be partially-extracted when the callback is fired).
The fix is a bit more complicated and would likely be best done by @alanhoff or someone else familiar with the underlying intention of the codebase, mostly because you still need the initial stream to be able to pipe the content from the archive into it. It's just that you probably only want to consider things 'done' once the second stream has fired a finish event, but it seems like we're actually getting the first one.
If this is the cause, the fix is somewhat straightforward, but it would likely involve more refashioning of how the combined write stream is created and returned so that you can feed the content from the file into the first stream but listen for the finish event on the second, certainly not a one-line fix. That said, I feel somewhat confident as I was originally using this library on a project, and ended up having to use a very similar approach to what I've described here as a workaround to using this library in my own code.

@fjmilens3 can you write a PR for this? I really don't care about code refactoring if this issue gets fixed.

fixed by #28

But how do we get callback to this code ? Tried everything - no end or finish event is emmited ?
var request = require('request');
var targz = require('tar.gz');
// Streams
var read = request.get('https://nodejs.org/dist/v0.12.7/node-v0.12.7.tar.gz');
var write = targz().createWriteStream('/some/directory');
read.pipe(write);

license: isc


text:
AjaxLoadInitialReport looks like invers logic

First thanks a lot on effort making this.
I hope that this is going to be more and more robust.
There is AjaxLoadInitialReport
But for me looks like that has inverse logic.
When AjaxLoadInitialReport is set to false . Reports load initially on lending on view.
It is just a whatnot but It took me for while.
I came to complain about missing feature and before Subnit Issue i Try to set it to false.
And it is woking as shoud
Thaks again
Please keep going

@adopilot, thanks for reaching out to me and thanks for checking out my code!
The AjaxLoadInitialReport is used to determine if the report should be generated when the page loads immediately (AjaxLoadInitialReport = false) or should the report be generated after the initial page load (AjaxLoadInitialReport = true).
I think what you're really getting at is that I'm trying to load the report before you've filled in any parameters, is that correct? In my instances of reports I usually fill defaults into the parameter fields so that the report can still load the first time. I suppose
I should look at giving people the option to not have the report load until after you've filled out parameters.
Alan

It is perfectly  fine to have control will report be rendered or not.
But I try only to point that this should   be default.
I was confused because it has to be  AjaxLoadInitialReport   = false to preform auto load.
For me more logic is to AjaxLoadInitialReport   = true preform initial load.

authors: adopilot alanjuden

repo: alanjuden/MvcReportViewer


text:
Bypass cache for logged in status

This adds a flag to bypass the cache for logged in users when they directly view the status page.  Hopefully logged in users should be relatively low traffic and by restricting it to when they explicitly load the status page, we only refresh when we're pretty sure they're interested, vs. just viewing the repo.
[fixes #91]

This is great, thanks @ForbesLindesay

authors: ForbesLindesay alanshaw

repo: alanshaw/david-www


text:
Feat/list language

add list language api
optimaze new sonarclient logic

LGTM

authors: adolphlwq chengjingtao

repo: alauda/bergamot