text:
Azure Data Explorer e2e tests are failing

Report

>>> scalers/azure-data-explorer.test.ts.retry.log <<<

  ✖ Replica count should be scaled out to 4 replicas [Pod Identity] Replica count should be 4 after some time
  ✔ Replica count should be scaled in to 0 replicas [Pod Identity] (43***ms)
  ✔ Replica count should be scaled out to 4 replicas [clientId & clientSecret] (7.5s)
  ─

  Replica count should be scaled out to 4 replicas [Pod Identity]

  scalers/azure-data-explorer.test.ts:***49

   ***48:     ***
   ***49:     t.is(desiredReplicaCount, currentReplicaCount, `Replica count shoul…
   ***50: ***

  Replica count should be 4 after some time

  Difference:

  - '4'
  + '0'

  › testDeploymentScale (scalers/azure-data-explorer.test.ts:***49:7)
  › scalers/azure-data-explorer.test.ts:86:5

  ─

  *** test failed

Expected Behavior
e2e tests fails
Actual Behavior
e2e test pass
Steps to Reproduce the Problem

run e2e

Logs from KEDA operator
2022-03-16T15:32:10.5426920Z ***.6474444552630084e+09	INFO	controller.scaledobject	Reconciling ScaledObject	***"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "test-azure-data-explorer-scaled-object", "namespace": "test-azure-data-explorer-ns"***
2022-03-16T15:32:10.5427947Z ***.6474444552630453e+09	INFO	controller.scaledobject	Adding Finalizer for the ScaledObject	***"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "test-azure-data-explorer-scaled-object", "namespace": "test-azure-data-explorer-ns"***
2022-03-16T15:32:10.5429208Z ***.647444455306***287e+09	INFO	controller.scaledobject	Detected resource targeted for scaling	***"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "test-azure-data-explorer-scaled-object", "namespace": "test-azure-data-explorer-ns", "resource": "apps/v***.StatefulSet", "name": "test-azure-data-explorer-sts"***
2022-03-16T15:32:10.5430615Z ***.6474444553062053e+09	INFO	controller.scaledobject	Creating a new HPA	***"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "test-azure-data-explorer-scaled-object", "namespace": "test-azure-data-explorer-ns", "HPA.Namespace": "test-azure-data-explorer-ns", "HPA.Name": "keda-hpa-test-azure-data-explorer-scaled-object"***
2022-03-16T15:32:10.5431732Z ***.64744445532658***2e+09	INFO	controller.scaledobject	Initializing Scaling logic according to ScaledObject Specification	***"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "test-azure-data-explorer-scaled-object", "namespace": "test-azure-data-explorer-ns"***
2022-03-16T15:32:10.5432216Z 2022/03/***6 ***5:27:35 KUSTO UNSAFE SETIINGS HAVE BEEN EXPLICITLY ENABLED. If this has not been done on purpose, remove the import
2022-03-16T15:32:10.5432659Z of the kusto/unsafe package and all reference to functions or methods with "Unsafe" in their title. Unsafe
2022-03-16T15:32:10.5433269Z methods have the potential for SQL-like injection attacks from service clients. If you still intend to use
2022-03-16T15:32:10.5433710Z unsafe methods, be sure to scrub your data inputs before sending them to Kusto. This message can be suppressed.
2022-03-16T15:32:10.5434723Z ***.647444455336868e+09	INFO	controller.scaledobject	Reconciling ScaledObject	***"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "test-azure-data-explorer-scaled-object", "namespace": "test-azure-data-explorer-ns"***
2022-03-16T15:32:10.5438087Z ***.6474444554636867e+09	ERROR	scalehandler	Error getting scale decision	***"scaledobject.Name": "test-azure-data-explorer-scaled-object", "scaledObject.Namespace": "test-azure-data-explorer-ns", "scaleTarget.Name": "test-azure-data-explorer-sts", "error": "failed to get azure data explorer metric value: failed to get azure data explorer metric result from query print result = ***8: Op(OpQuery): Kind(KInternal): azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to ***/v2/rest/query: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: ***\"error\":\"invalid_request\",\"error_description\":\"Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request\"*** Endpoint http://***69.254.***69.254/metadata/identity/oauth2/token?api-version=20***8-02-0***&resource=***"***
2022-03-16T15:32:10.5438464Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
2022-03-16T15:32:10.5438731Z 	/workspace/pkg/scaling/scale_handler.go:278
2022-03-16T15:32:10.5439097Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
2022-03-16T15:32:10.5439368Z 	/workspace/pkg/scaling/scale_handler.go:***49
2022-03-16T15:32:10.5440390Z ***.6474444703620303e+09	INFO	controller.scaledobject	Reconciling ScaledObject	***"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "test-azure-data-explorer-scaled-object", "namespace": "test-azure-data-explorer-ns"***
2022-03-16T15:32:10.5443533Z ***.6474444854765646e+09	ERROR	scalehandler	Error getting scale decision	***"scaledobject.Name": "test-azure-data-explorer-scaled-object", "scaledObject.Namespace": "test-azure-data-explorer-ns", "scaleTarget.Name": "test-azure-data-explorer-sts", "error": "failed to get azure data explorer metric value: failed to get azure data explorer metric result from query print result = ***8: Op(OpQuery): Kind(KInternal): azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to ***/v2/rest/query: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: ***\"error\":\"invalid_request\",\"error_description\":\"Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request\"*** Endpoint http://***69.254.***69.254/metadata/identity/oauth2/token?api-version=20***8-02-0***&resource=***"***
2022-03-16T15:32:10.5443895Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
2022-03-16T15:32:10.5444265Z 	/workspace/pkg/scaling/scale_handler.go:278
2022-03-16T15:32:10.5444614Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
2022-03-16T15:32:10.5444902Z 	/workspace/pkg/scaling/scale_handler.go:***49
2022-03-16T15:32:10.5448076Z ***.6474445***545256***4e+09	ERROR	scalehandler	Error getting scale decision	***"scaledobject.Name": "test-azure-data-explorer-scaled-object", "scaledObject.Namespace": "test-azure-data-explorer-ns", "scaleTarget.Name": "test-azure-data-explorer-sts", "error": "failed to get azure data explorer metric value: failed to get azure data explorer metric result from query print result = ***8: Op(OpQuery): Kind(KInternal): azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to ***/v2/rest/query: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: ***\"error\":\"invalid_request\",\"error_description\":\"Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request\"*** Endpoint http://***69.254.***69.254/metadata/identity/oauth2/token?api-version=20***8-02-0***&resource=***"***
2022-03-16T15:32:10.5448529Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
2022-03-16T15:32:10.5448809Z 	/workspace/pkg/scaling/scale_handler.go:278
2022-03-16T15:32:10.5449149Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
2022-03-16T15:32:10.5449441Z 	/workspace/pkg/scaling/scale_handler.go:***49
2022-03-16T15:32:10.5452510Z ***.6474445454682274e+09	ERROR	scalehandler	Error getting scale decision	***"scaledobject.Name": "test-azure-data-explorer-scaled-object", "scaledObject.Namespace": "test-azure-data-explorer-ns", "scaleTarget.Name": "test-azure-data-explorer-sts", "error": "failed to get azure data explorer metric value: failed to get azure data explorer metric result from query print result = ***8: Op(OpQuery): Kind(KInternal): azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to ***/v2/rest/query: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: ***\"error\":\"invalid_request\",\"error_description\":\"Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request\"*** Endpoint http://***69.254.***69.254/metadata/identity/oauth2/token?api-version=20***8-02-0***&resource=***"***
2022-03-16T15:32:10.5452874Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
2022-03-16T15:32:10.5453159Z 	/workspace/pkg/scaling/scale_handler.go:278
2022-03-16T15:32:10.5453501Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
2022-03-16T15:32:10.5453784Z 	/workspace/pkg/scaling/scale_handler.go:***49
2022-03-16T15:32:10.5456905Z ***.6474445754665952e+09	ERROR	scalehandler	Error getting scale decision	***"scaledobject.Name": "test-azure-data-explorer-scaled-object", "scaledObject.Namespace": "test-azure-data-explorer-ns", "scaleTarget.Name": "test-azure-data-explorer-sts", "error": "failed to get azure data explorer metric value: failed to get azure data explorer metric result from query print result = ***8: Op(OpQuery): Kind(KInternal): azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to ***/v2/rest/query: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: ***\"error\":\"invalid_request\",\"error_description\":\"Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request\"*** Endpoint http://***69.254.***69.254/metadata/identity/oauth2/token?api-version=20***8-02-0***&resource=***"***
2022-03-16T15:32:10.5457270Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
2022-03-16T15:32:10.5457531Z 	/workspace/pkg/scaling/scale_handler.go:278
2022-03-16T15:32:10.5457883Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
2022-03-16T15:32:10.5458229Z 	/workspace/pkg/scaling/scale_handler.go:***49
2022-03-16T15:32:10.5461296Z ***.64744460545***502e+09	ERROR	scalehandler	Error getting scale decision	***"scaledobject.Name": "test-azure-data-explorer-scaled-object", "scaledObject.Namespace": "test-azure-data-explorer-ns", "scaleTarget.Name": "test-azure-data-explorer-sts", "error": "failed to get azure data explorer metric value: failed to get azure data explorer metric result from query print result = ***8: Op(OpQuery): Kind(KInternal): azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to ***/v2/rest/query: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: ***\"error\":\"invalid_request\",\"error_description\":\"Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request\"*** Endpoint http://***69.254.***69.254/metadata/identity/oauth2/token?api-version=20***8-02-0***&resource=***"***
2022-03-16T15:32:10.5461732Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
2022-03-16T15:32:10.5461995Z 	/workspace/pkg/scaling/scale_handler.go:278
2022-03-16T15:32:10.5462353Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
2022-03-16T15:32:10.5462626Z 	/workspace/pkg/scaling/scale_handler.go:***49
2022-03-16T15:32:10.5465812Z ***.6474446354853964e+09	ERROR	scalehandler	Error getting scale decision	***"scaledobject.Name": "test-azure-data-explorer-scaled-object", "scaledObject.Namespace": "test-azure-data-explorer-ns", "scaleTarget.Name": "test-azure-data-explorer-sts", "error": "failed to get azure data explorer metric value: failed to get azure data explorer metric result from query print result = ***8: Op(OpQuery): Kind(KInternal): azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to ***/v2/rest/query: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: ***\"error\":\"invalid_request\",\"error_description\":\"Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request\"*** Endpoint http://***69.254.***69.254/metadata/identity/oauth2/token?api-version=20***8-02-0***&resource=***"***
2022-03-16T15:32:10.5466184Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
2022-03-16T15:32:10.5466447Z 	/workspace/pkg/scaling/scale_handler.go:278
2022-03-16T15:32:10.5466790Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
2022-03-16T15:32:10.5467077Z 	/workspace/pkg/scaling/scale_handler.go:***49
2022-03-16T15:32:10.5470241Z ***.6474446654606526e+09	ERROR	scalehandler	Error getting scale decision	***"scaledobject.Name": "test-azure-data-explorer-scaled-object", "scaledObject.Namespace": "test-azure-data-explorer-ns", "scaleTarget.Name": "test-azure-data-explorer-sts", "error": "failed to get azure data explorer metric value: failed to get azure data explorer metric result from query print result = ***8: Op(OpQuery): Kind(KInternal): azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to ***/v2/rest/query: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: ***\"error\":\"invalid_request\",\"error_description\":\"Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request\"*** Endpoint http://***69.254.***69.254/metadata/identity/oauth2/token?api-version=20***8-02-0***&resource=***"***
2022-03-16T15:32:10.5470591Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
2022-03-16T15:32:10.5470866Z 	/workspace/pkg/scaling/scale_handler.go:278
2022-03-16T15:32:10.5471212Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
2022-03-16T15:32:10.5471493Z 	/workspace/pkg/scaling/scale_handler.go:***49
2022-03-16T15:32:10.5474686Z ***.6474446954549336e+09	ERROR	scalehandler	Error getting scale decision	***"scaledobject.Name": "test-azure-data-explorer-scaled-object", "scaledObject.Namespace": "test-azure-data-explorer-ns", "scaleTarget.Name": "test-azure-data-explorer-sts", "error": "failed to get azure data explorer metric value: failed to get azure data explorer metric result from query print result = ***8: Op(OpQuery): Kind(KInternal): azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to ***/v2/rest/query: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: ***\"error\":\"invalid_request\",\"error_description\":\"Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request\"*** Endpoint http://***69.254.***69.254/metadata/identity/oauth2/token?api-version=20***8-02-0***&resource=***"***
2022-03-16T15:32:10.5475043Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
2022-03-16T15:32:10.5475314Z 	/workspace/pkg/scaling/scale_handler.go:278
2022-03-16T15:32:10.5475722Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
2022-03-16T15:32:10.5476009Z 	/workspace/pkg/scaling/scale_handler.go:***49
2022-03-16T15:32:10.5477149Z ***.647444720898***369e+09	INFO	controller.scaledobject	Reconciling ScaledObject	***"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "test-azure-data-explorer-scaled-object", "namespace": "test-azure-data-explorer-ns"***
2022-03-16T15:32:10.5478273Z ***.6474447209***27045e+09	INFO	controller.scaledobject	Initializing Scaling logic according to ScaledObject Specification	***"reconciler group": "keda.sh", "reconciler kind": "ScaledObject", "name": "test-azure-data-explorer-scaled-object", "namespace": "test-azure-data-explorer-ns"***
2022-03-16T15:32:10.5481438Z ***.64744472***038***93e+09	ERROR	scalehandler	Error getting scale decision	***"scaledobject.Name": "test-azure-data-explorer-scaled-object", "scaledObject.Namespace": "test-azure-data-explorer-ns", "scaleTarget.Name": "test-azure-data-explorer-sts", "error": "failed to get azure data explorer metric value: failed to get azure data explorer metric result from query print result = 0: Op(OpQuery): Kind(KInternal): azure.BearerAuthorizer#WithAuthorization: Failed to refresh the Token for request to ***/v2/rest/query: StatusCode=400 -- Original Error: adal: Refresh request failed. Status Code = '400'. Response body: ***\"error\":\"invalid_request\",\"error_description\":\"Multiple user assigned identities exist, please specify the clientId / resourceId of the identity in the token request\"*** Endpoint http://***69.254.***69.254/metadata/identity/oauth2/token?api-version=20***8-02-0***&resource=***"***
2022-03-16T15:32:10.5481808Z github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers

KEDA Version
No response
Kubernetes Version
No response
Platform
No response
Scaler Details
Azure Data Explorer
Anything else?
No response

@yasiboni-max PTAL

We need to specify MSI client id when creating kusto client if there's more than one managed identity assigned to the cluster.
I'm working on PR that adds ManagedIdentityClientId as parameter when user choose to authenticate through pod identity.

@yasiboni-max PTAL

Looks like we need to specify MSI client id when creating kusto client if there's more than one managed identity assigned to the cluster.
I'm working on PR that adds ManagedIdentityClientId as parameter when user choose to authenticate through pod identity.

@zroubalik Can you please assign me this issue?
keda PR: #2822
keda docs PR: https://github.com/kedacore/keda-docs/pull/731

Meanwhile, we solved the problem consistently, I'll disable that e2e test

Closing as the e2e tests are working now.

authors: JorTurFer v-shenoy yasiboni-max zroubalik

repo: kedacore/keda


text:
Metric keda_scaler_errors_total is absent

Report
I have installed keda and the metric keda_scaler_errors_total is missing.
I have deployed the grafana dashboard and noticed that the leftmost panel is not showing any data.

The data displayed is sum by(job) (rate(keda_scaler_errors_total{}[5m]))
I checked the exposed metrics of the operator using a port-forward and could not find any value for keda_scaler_errors_total.
Expected Behavior
The /metrics endpoint should contain keda_scaler_errors_total 0. The dashboard shoud then display the left panel graph with value 0.
Actual Behavior
keda_scaler_errors_total is not exposed.
Steps to Reproduce the Problem

Install keda and configure a ScaledObject.
Check /metrics

Logs from KEDA operator
No response
KEDA Version
2.10.0
Kubernetes Version
1.24
Platform
Microsoft Azure
Scaler Details
Apache Kafka
Anything else?
No response

Not sure if I read the code correctly. I think the counter is not initialized until the first call of RecordScalerError when this line is called.
https://github.com/kedacore/keda/blob/ee28bf69389bca7bf6d5edd83e2bc3940254114c/pkg/prommetrics/prommetrics.go#L151

source: gharchive/issue

license_type: permissive

license_source: github-api


text:
Add support for a trigger based on a queue in an ActiveMQ Artemis cluster

Proposal
As detailed here: https://github.com/artemiscloud/activemq-artemis-operator/issues/892
The ActiveMQ Artemis management endpoint that KEDA uses works alright for 1 instance but fails for a cluster. The issue is that each artemis node only returns its local queue length (not the queue for the entire cluster). Unfortunately, they do not provide an endpoint for returning the queue length for the entire cluster.
Could we add a configuration where we provide multiple target hosts and the the queue lengths get aggregated together to get the total queue length?
Use-Case
Allowing proper scaling based on ActiveMQ Artemis queues in a cluster configuration.
Is this a feature you are interested in implementing yourself?
No
Anything else?
No response

Is maybe scaling modifiers what will work for you ?

ah yeah, that looks like it would work

authors: lavifb rickbrouwer


text:
adding the script to validate changelog

adding the script to validate changelog by make validate-changelog and updating github action
Checklist

[ ] When introducing a new scaler, I agree with the scaling governance policy
[ ] I have verified that my change is according to the deprecations & breaking changes policy
[ ] Tests have been added
[ ] Changelog has been updated and is aligned with our changelog requirements
[ ] A PR is opened to update our Helm chart (repo) (if applicable, ie. when deployment manifests are modified)
[ ] A PR is opened to update the documentation on (repo) (if applicable)
[ ] Commits are signed with Developer Certificate of Origin (DCO - learn more)

Fixes #3190

Also would be nice to add a way to skip the Changelog validation for a certain PR. (eg. we have the skip-e2e label for skipping e2e test check)

I'm not sure if we should be able to skip Changelog validation, it's more like other CI checks than e2e test IMO, we cannot skip static checks (and this is one static check more). Personally, I'm fine without skipping it, but I prefer to execute it as part of pre-commit if it's possible. If somebody is using pre-commit, it'll cover this new check automatically
WDYT?

Also would be nice to add a way to skip the Changelog validation for a certain PR. (eg. we have the skip-e2e label for skipping e2e test check)

I'm not sure if we should be able to skip Changelog validation, it's more like other CI checks than e2e test IMO, we cannot skip static checks (and this is one static check more). Personally, I'm fine without skipping it, but I prefer to execute it as part of pre-commit if it's possible. If somebody is using pre-commit, it'll cover this new check automatically
WDYT?

Good point!

hey @JorTurFer @JorTurFer Please take a look.

Thanks for this! The changelog is sorted, but we also tend to put General: before all other values.

@zroubalik To my understanding as there are other values as well checking for General specifically will not give us any result. As we than also need to allow for other example name also. If we can limit those names than that check would be a good fit. WDYT?

Thanks for this! The changelog is sorted, but we also tend to put General: before all other values.

@zroubalik To my understanding as there are other values as well checking for General specifically will not give us any result. As we than also need to allow for other example name also. If we can limit those names than that check would be a good fit. WDYT?

General is the only value that has to be prioritized. This is because usually General affects to all the users and that's why we place it on top of all the sections. The style rules are here: https://github.com/kedacore/keda/blob/main/CONTRIBUTING.md#Changelog
Said that, maybe we can add 2 steps in the extract_and_check function, the first step checks for General items and the second one for the rest. WDYT?

@JorTurFer sounds great, i will add it.

This is nice! My only question is if we can check all the changelog and not just unreleased. I say this because during cherry-picks for hotfix releases we can make mistakes

we can but we need to change all previous to correct order as well first.

we can but we need to change all previous to correct order as well first.

If we can, it'd be nice. The changelog has been chaotic in the past 😞

we can but we need to change all previous to correct order as well first.

If we can, it'd be nice. The changelog has been chaotic in the past disappointed

I will try.

hey @JorTurFer I have tried the script to validate the change in each version. The script is:
#!/bin/bash

SCRIPT_ROOT=$(dirname "${BASH_SOURCE[0]}")/..

# Define filename
filename="$SCRIPT_ROOT/CHANGELOG.md"

# Check if file exists
if [[ ! -f "$filename" ]]; then
    echo "Error: $filename does not exist."
    exit 1
fi

# Storing the version to be checked
mapfile -t versions < <(awk '/## History/{flag=1;next}/## /{flag=0}flag' "$filename" | grep -o '\[[^]]*\]' | sed 's/[][]//g')

# Define a function to extract and sort sections
function extract_and_check() {
  local section=$1
  local content_block=$2
  local content=$(awk "/### $section/{flag=1;next}/### /{flag=0}flag" <<< "$content_block" | grep '^- \*\*')

  # Skip if content does not exist
  if [[ -z "$content" ]]; then
    return
  fi

  # Separate and sort the **General**: lines
  local sorted_general_lines=$(echo "$content" | grep '^- \*\*General\*\*:' | sort)

  # Sort the remaining lines
  local sorted_content=$(echo "$content" | grep -v '^- \*\*General\*\*:' | sort)

  # Check if sorted_general_lines is not empty, then concatenate
  if [[ -n "$sorted_general_lines" ]]; then
      sorted_content=$(printf "%s\n%s" "$sorted_general_lines" "$sorted_content")
  fi

  # Check pattern and throw error if wrong pattern found
  while IFS= read -r line; do
      echo "Error: Wrong pattern found in section: $section , line: $line"
      exit 1
  done < <(grep -Pv '^(-\s\*\*[^*]+\*\*: .*\(\[#(\d+)\]\(https:\/\/github\.com\/kedacore\/(keda|charts)\/(pull|issues)\/\2\)\))$' <<< "$content")

  if [ "$content" != "$sorted_content" ]; then
      echo "Error: Section: $section is not sorted correctly. Correct order:"
      echo "$sorted_content"
      exit 1
  fi
}

# Extract release sections, including "Unreleased", and check them
for version in "${versions[@]}"; do
  release_content=$(awk "/## $version/{flag=1;next}/## v[0-9\.]+/{flag=0}flag" "$filename")

  if [[ -z "$release_content" ]]; then
    echo "No content found for $version Skipping."
    continue
  fi

  echo "Checking section: $version"

  # Separate content into different sections and check sorting for each release
  extract_and_check "New" "$release_content"
  extract_and_check "Improvements" "$release_content"
  extract_and_check "Fixes" "$release_content"
  extract_and_check "Deprecations" "$release_content"
  extract_and_check "Other" "$release_content"

done

But the problem is, some changes are not logged properly, and not easy to search for in the pr and issues, may be they are named differently there.
For example:
Checking section: v2.11.0
Error: Wrong pattern found in section: Improvements , line: - **General**: Enable secret scanning in GitHub repo

But the problem is, some changes are not logged properly, and not easy to search for in the pr and issues, may be they are named differently there.

Let me take a look, we haven't been following the same rules since the beginning, let me check it changelog for trying to limit the scope 😄

There are a lot of issues on the changelog T.T
Can I fix them as part of your PR? As you enabled maintainer to edit the PR, I can commit into your branch if you agree

Based on your script, I have done some small modifications (to exclude v1.* and to support multiple repetitions of the pattern) and I have fixed all the issues on the changelog. After that, it works really nice ❤
As I said, if you agree, I can commit the changes to your branch directly

@JorTurFer yes please. Thankyou

Done!
Please take a look 🙏 I'm terrible using bash and probably I have done any mistake. I started from your code and update it a bit, so I hope that I didn't do too much shity code xD

Done! Please take a look 🙏 I'm terrible using bash and probably I have done any mistake. I started from your code and update it a bit, so I hope that I didn't do too much shity code xD

Thanks for your kind words, It looks good to me.

Why is failing in this:
https://github.com/kedacore/keda/pull/4731/commits/d6c54d552c7beb3c3b2d61b10cb72730369b01a9
Same is running fine locally.

Why is failing in this:
d6c54d5
Same is running fine locally.

That failure that the commit solves is related with missing new line at the end of the file. It's this rule: https://github.com/kedacore/keda/blob/main/.pre-commit-config.yaml#L16C7-L16
About why the script that validates the changelog failed, it looks related with some small differences between grep and sort commands in Windows, Linux and Mac. For example, the grep modifier -P doesn't exist in Mac (I have replaced it with -E) and sort has different default behaviours (in Windows using WSL, default behaviour is case insensitive but in Mac is case sensitive...)

Everything is green! 🎉 🎉 🎉 🎉

@zroubalik PTAL

source: gharchive/pull-request

authors: JorTurFer nitishchauhan0022 zroubalik


text:
feat: Improve hurl file structure + full hurl coverage

Feature Request
Describe the Feature Request

Improve file structure + full hurl coverage before hurl CI
Describe Preferred Solution

Describe Alternatives

Related Code

Additional Context

If the feature request is approved, would you be willing to submit a PR?
(Help can be provided if you need assistance submitting a PR)

[ ] Yes
[ ] No

Merged

repo: keep-starknet-strange/beerus

license: MIT


text:
[feat] open a good first issue

Hi I would like to contribute more.
My issue where I contribute is currently stucked by another PR.
I want to avoid rebase conflict. could you create and assign me to a good first issue please?

Please can I be assigned to this issue

I would love to contribute again. Could you let me just finish a very hard PR on another PR before contributing again please?

I am applying to this issue via OnlyDust platform.
My background and how it can be leveraged
first Find an existing "good first issue":
If you're contributing to an open-source project on platforms like GitHub, most repositories have a "good first issue" label to help contributors find beginner-friendly tasks. You can search for these issues by filtering through the repository's issue tracker and applying the "good first issue" label.
Create a new "good first issue":
If you have sufficient permissions on the project, you can create a new issue that qualifies as a "good first issue." This should be a smaller, self-contained task, like fixing documentation, improving test coverage, or addressing minor bugs.
Avoid rebase conflicts:
Work on the new issue in a separate branch.
Once your original PR is unblocked, you can merge the latest changes into your branch to minimize potential conflicts.
How I plan on tackling this issue
I usually follow the above steps on my organisation projects, hope it finds you informative...

I am applying to this issue via OnlyDust platform.
My background and how it can be leveraged
I'm an undergraduate in Computer Science and am well experienced in coding across different programming languages. Along the way, I have enjoyed building efficient algorithms, practicing data structures, and tackling problems in clean, optimized code. So far, I've had fun using best practices to enhance performance and readability in projects.
I'm excited about this project because it will take my coding abilities, problem-solving skills, and passion for learning to the benefit of improving and maintaining code quality. With my strong foundation in computer science and eagerness to collaborate on impactful projects, you can expect to receive valuable contributions from me.

Hello.
I am looking for a good first issue in cryptography. I am not very strong in cryptography but it is a field where I would like to learn. I am strong in coding but I would LOVE doing cryptogrtaphy.

I am applying to this issue via OnlyDust platform.
My background and how it can be leveraged
i am a frontend and blockachain developer with a strong background in cario also have experience with html, css, react, solidity, javascript,typescript .
How I plan on tackling this issue
Request a "good first issue" from the project maintainers and ask to be assigned, explaining your current contribution is blocked by another PR and you'd like to avoid rebase conflicts.

I am applying to this issue via OnlyDust platform.
My background and how it can be leveraged
I am a MERN stack developer with expertise in MongoDB, Express.js, React, and Node.js. I also have experience with JavaScript, TypeScript, and frontend development tools, making me well-versed in building and debugging efficient, scalable applications. My technical background enables me to approach this task with a strong understanding of modern development practices and low-level debugging concepts.
How I plan on tackling this issue
I plan to request a "good first issue" from the project maintainers and ask to be assigned. This approach ensures I can get started on a manageable task and gradually familiarize myself with the project's codebase. I will also communicate openly with maintainers, explaining if my current contribution is blocked by another PR, to avoid potential rebase conflicts. I aim to implement the solution in a systematic manner, ensuring all changes are tested and aligned with the project's standards.


text:
ARCamera compatibility?

Question: Can your UnityBarcodeScanner work with ARCamera from Vuforia? It works wonders in normal mode, how can I use it for ARCamera?

Hi, I've create a class that works with Vuforia AR, check #30

Does not seem to work on iOS/Android device
ArgumentException: The Assembly zxing.unity is referenced by Assembly-CSharp-firstpass ('Library/ScriptAssemblies/Assembly-CSharp-firstpass.dll'). But the dll is not allowed to be included or could not be found.
UnityEditor.AssemblyHelper.AddReferencedAssembliesRecurse (System.String assemblyPath, System.Collections.Generic.List`1 alreadyFoundAssemblies, System.String[] allAssemblyPaths, System.String[

@Maximetinu  did you have to use a different zxing.a or .dll for iOS?
@kefniark in your readme, you mention this works on iOS - however, though testing in editor, a code reads, i cannot get a code to read on actual iOS device. how did you set it up (different zxing libraries?) to run on iOS/Android devices?

authors: Maximetinu Pavel-Habzansky yosun

repo: kefniark/UnityBarcodeScanner

license: WTFPL


text:
Added new formats Behind the Iron Gate and Sink or Swim

Added new formats

No checksum at all on Sink Or Swim? Not even an unchecked CRC or something after the data (sometimes there even if teh loader doesn't check it)?

I checked and didn't see a checksum.  I can send you the dumps if you want to take a look.
I figured out the issue with eye of horus and will update my repo this weekend.  Bytes per sector isn't always a multiple of 4, so the memcopy of the header in the read method is incorrect for these instances (tracks 60 and 70 for example).
Sent from Surface
From: keirf
Sent: Wednesday, January 7, 2015 12:11 AM
To: keirf/Disk-Utilities
Cc: Keith Krellwitz
No checksum at all on Sink Or Swim? Not even an unchecked CRC or something after the data (sometimes there even if teh loader doesn't check it)?
—
Reply to this email directly or view it on GitHub.

Email me over the dump and I'll have a look. I think we have one or two other formats with no checksum at all, so it's quite likely there really isn't one.

authors: keirf kkrellwitz

repo: keirf/Disk-Utilities

license: unlicense

license_source: bigquery


text:
xdftool command not found

Hi. I think I found a little error in the documentation. The amitools have to be installed with "sudo pip install amitools" and not with "pip install --user amitools", otherwise the command xdftool cannot be found. (My system: Ubuntu 18.04.1 LTS). Maybe someone else can confirm that...

You need ~/.local/bin on your PATH. I should mention that, or just recommend the system-wide install.

license: Unlicense


text:
Tailwind UI 🤔

https://tailwindui.com (paid)
It's pretty shiny, hey.
How does it really compare to material-ui from a React components point of view?
https://www.reddit.com/r/reactjs/comments/uasd7v/i_made_a_complete_team_support_in_react_for_my/
AB#30

Naaaaaaaaah.

repo: keithamoss/demsausage


text:
How to change download location path in TagUI - use hack to override

Hi @kensoh
I am trying to download file from chrome using xpath. My TagUi was installed in C:\apps\TagUi
I am having my code in C:\apps\Codes, when i download the file my file is being downloaded in C:\apps\Codes. Can i change the download location to something like C:\Files.
Thanks in advance.

Hi @sandeep14897 there are a few issues with the answer, one of them is - https://github.com/kelaberetiv/TagUI/issues/469
TagUI wasn't designed to change the download folder to another folder from the script folder, so the hack below will let users override the download location to their desired location. In this case I think flowpath can be replaced with C:\\Files. Make sure user account has write access to folder.
chrome_step('Page.setDownloadBehavior',{behavior: 'allow', downloadPath: 'flow_path'});

Another way, is after downloading, you can use the run step to run cmd / c move source target to move the file to the target folder, or if more adventurous can use JavaScript fs module that comes packaged with PhantomJS engine (used in TagUI) - https://phantomjs.org/api/fs/method/move.html

Closing issue for now until further input

@kensoh . Thank you. That is working fine.
Can we convert any type of file to  .pdf using tagui, without opening the file. My case is i will download a file from chrome (.pptx) and it should be converted back to pdf and upload the same .pdf file.
will that work out for me. Please, help me on this.
Thank you

I'm afraid TagUI does not have built-in file conversion capabilities. You can find out how to do that using command-line tools available on the internet and use the run step to do that. Or maybe use some free online convertor and add some TagUI workflow to do that. Or maybe write TagUI workflow to do that using Powerpoint and export. Method is visual automation but very tricky. But is find other command-line tools to do it.

Also, for different issues, raise a new issue so other users can learn from your question.

Sure @kensoh . Thank you.

Hi @sandeep14897 there are a few issues with the answer, one of them is - #469
TagUI wasn't designed to change the download folder to another folder from the script folder, so the hack below will let users override the download location to their desired location. In this case I think flowpath can be replaced with C:\\Files. Make sure user account has write access to folder.
chrome_step('Page.setDownloadBehavior',{behavior: 'allow', downloadPath: 'flow_path'});

Another way, is after downloading, you can use the run step to run cmd / c move source target to move the file to the target folder, or if more adventurous can use JavaScript fs module that comes packaged with PhantomJS engine (used in TagUI) - https://phantomjs.org/api/fs/method/move.html

Hey there!
I've been trying to use this hack but everytime the flow downloads the file i need, it fails. The folder where it is supposed to save in is inside the same folder as the script. How can i make sure the user account has write access to folder?
Thanks!

Hi @andyguins to save in same folder as script, there is no need to use this hack because the default download behaviour is saving in the same folder as the script. However, if you need to change to another folder, you can now use the download to step here - https://tagui.readthedocs.io/en/latest/reference.html#download-to

Hi @kensoh, what i meant was that the folder where i want to save the file is inside the same folder as the script.
C:/tagui/flows/rpa  script location
C:/tagui/flows/rpa/filesfolder     files location
Anyways, i tried the dowload to step but it kept saving the files on the same folder.

Oic... Can you do tagui update to get the latest version of TagUI and then do below to see if it helps?
download to c:\tagui\flows\rpa\filesfolder

If it still does not work, can you attach the file tagui\src\tagui_chrome.log here to see if there is any error message from Chrome. The user will need write access to the folder to save there. If your user does not have write access you can pick another location where you for sure has write access, for eg your user Desktop folder.

I solved it, the problem was that i wrote the download to  step as the last step of the flow when i should have written at the beginning. It was a pretty dumb problem :P  I realised when i read the tagui_chrome.log that you mentioned.
Thanks for your help Ken!

No probs Andy! Yes it has to be before the actual download, otherwise the download would have happened at original folder.

authors: andyguins kensoh sandeep14897

repo: kelaberetiv/TagUI


text:
network: Add support for address sets

As spark scaled, the number of ACLs that OVN had to process grew at O(n^2)
as the number of workers grew.  At a scale of 64 nodes this would cause OVN
to spend several minutes processing ACLs before the network would work
properly.
This patch dramatically reduces the number of ACLs by adding support for
address sets to the Kelda Daemon.  Address sets allow ACLs to be express
groups of IP addresses, instead of requiring them to be exploded out into a
separate ACL per IP.
This patch adds full support for address sets to the deployment engine, and
partial support to the Javascript bindings.  Even with partial support for
address sets, ACLs are no longer a bottleneck when running Spark.  However,
full support will need to be added later if we want to scale to large
numbers of containers.

Also note that the build failed (in case you didn't see this already)

authors: ejj kayousterhout

repo: kelda/kelda

license: mit


text:
conflict resolution should be opt-in, no longer default

It is too hard to do all at once.

👍

I see --ignore-conflicts in README.md

@Gaurav0 Yes, it should be --resolve-conflicts, so without the option it doesn't do anything for you.

authors: Gaurav0 kellyselden rwjblue

repo: kellyselden/ember-cli-update


text:
fixed example worm.json

The Example worm.json dosn't work becouse it tries to download the twitter and facebook sharelink as chapters, this changes the chapter_selector to fix that.

I merged #51 instead, which changed the way Worm is downloaded entirely.

I merged #51 instead, which changed the way Worm is downloaded entirely.

authors: KeinNiemand kemayo

repo: kemayo/leech


text:
Enhance navigation panel

Events are now published as the selected item in the navigation panel changes. This will allow, for example, KeyStrokeHandlers to activate/deactivate themselves as appropriate.

Coverage increased (+0.6%) to 94.976% when pulling 80fe4c38bc0d07fbb0ba62d0906454d1c0755ebe on enhance-navigation-panel into 94f0ec12e164c3af24d2a98de5d01c3290ad6477 on develop.

authors: coveralls kemitix

repo: kemitix/ldap-user-manager


text:
Make the swapCache actually work

As the SW can be killed before this call, we might need to store the responses like in an indexedDB, if these can be stored at all. Then remove existing entries and adding the new ones.

Didn't look what you do there yet, but why not simply store in tmp cache?

the SW fetch event seems to consult all caches. It might be that a new one isnt consulted but then I would need to update the SW. Maybe it is easy, I just didn't attempt this yet

authors: NekR kenchris

repo: kenchris/jakecache


text:
Check the copytodns && copytossl buttons on resolved

Seems they cleanup the cache after adding :O

The cache was erroneously cleaned without user direct intervention-removed

repo: kendarorg/HttpAnsweringMachine


text:
How can I get generic available devices in IOS without pair? (Help needed)

Mobile Device Environment

Device: iPhone6s
OS: iOS14.3

Application Environment

React Native version: v0.63.4
RN Bluetooth Classic version: 1.60.0-rc.6

In iOS platform, How can I get generic available devices without pair?
Generic device like : Windows 10 Bluetooth, Mac mini Bluetooth etc...
The Android app works as expected but iOS needs help.
If I am not wrong we don't need an MFi string for this type of device. Right?
I am asking it because I have successfully paired  Windows 10 Bluetooth through the IOS Bluetooth setting screen. and after call getBondedDevices() method but I am not getting any parried device. it is always retune blank array.

I'm not sure I understand your use case.
Let's say for example that we updated the getBondedDevices to use the CoreBluetooth library which might make this possible.  To what end?
You still cannot make a serial connection with the device due to the lack of MFi protocols.

I'm not sure I understand your use case.
Let's say for example that we updated the getBondedDevices to use the CoreBluetooth library which might make this possible.  To what end?
You still cannot make a serial connection with the device due to the lack of MFi protocols.

I've tried looking into whether the new changes to Core Bluetooth to allow for classic connection would be helpful:
https://developer.apple.com/documentation/corebluetooth/using_core_bluetooth_classic
But haven't had much time to actually play with the sample to see if it would fit in.   I see no point providing a list of devices to the user, which they cannot actually connect.
func centralManagerDidUpdateState(_ central: CBCentralManager) {
		// In your application, you would address each possible value of central.state and central.authorization
		switch central.state {
		case .resetting:
			os_log("Connection with the system service was momentarily lost. Update imminent")
		case .unsupported:
			os_log("Platform does not support the Bluetooth Low Energy Central/Client role")
		case .unauthorized:
			switch central.authorization {
			case .restricted:
				os_log("Bluetooth is restricted on this device")
			case .denied:
				os_log("The application is not authorized to use the Bluetooth Low Energy role")
			default:
				os_log("Something went wrong. Cleaning up cbManager")
			}
		case .poweredOff:
			os_log("Bluetooth is currently powered off")
		case .poweredOn:
			os_log("Starting cbManager")
// It might be possible to use the BTConstant for SSP here? Although who knows if the devices support it.
			let matchingOptions = [CBConnectionEventMatchingOption.serviceUUIDs: [BTConstants.sampleServiceUUID]]
			cbManager.registerForConnectionEvents(options: matchingOptions)
		default:
			os_log("Cleaning up cbManager")
		}
	}

	func centralManager(_ central: CBCentralManager, connectionEventDidOccur event: CBConnectionEvent, for peripheral: CBPeripheral) {
		os_log("connectionEventDidOccur for peripheral: %@", peripheral)
		switch event {
		case .peerConnected:
			cbPeripherals.append(peripheral)
		case .peerDisconnected:
			os_log("Peer %@ disconnected!", peripheral)
		default:
			if let idx = cbPeripherals.firstIndex(where: { $0 === peripheral }) {
				cbPeripherals.remove(at: idx)
			}
		}
		tableView.reloadData()
	}

Although this would require changes to the IOS side so that the devices were managed using both Core Bluetooth and EAccessory getBondedDevices() which again, doesn't help much since even if you see the device, you cannot connect to it.
Although, with that said, it is possible now to write your own connection logic, so there would be nothing stopping you from wrapping a BLE connection within this library now.
Let me think about it.

I've tried looking into whether the new changes to Core Bluetooth to allow for classic connection would be helpful:
https://developer.apple.com/documentation/corebluetooth/using_core_bluetooth_classic
But haven't had much time to actually play with the sample to see if it would fit in.   I see no point providing a list of devices to the user, which they cannot actually connect.
func centralManagerDidUpdateState(_ central: CBCentralManager) {
		// In your application, you would address each possible value of central.state and central.authorization
		switch central.state {
		case .resetting:
			os_log("Connection with the system service was momentarily lost. Update imminent")
		case .unsupported:
			os_log("Platform does not support the Bluetooth Low Energy Central/Client role")
		case .unauthorized:
			switch central.authorization {
			case .restricted:
				os_log("Bluetooth is restricted on this device")
			case .denied:
				os_log("The application is not authorized to use the Bluetooth Low Energy role")
			default:
				os_log("Something went wrong. Cleaning up cbManager")
			}
		case .poweredOff:
			os_log("Bluetooth is currently powered off")
		case .poweredOn:
			os_log("Starting cbManager")
// It might be possible to use the BTConstant for SSP here? Although who knows if the devices support it.
			let matchingOptions = [CBConnectionEventMatchingOption.serviceUUIDs: [BTConstants.sampleServiceUUID]]
			cbManager.registerForConnectionEvents(options: matchingOptions)
		default:
			os_log("Cleaning up cbManager")
		}
	}

	func centralManager(_ central: CBCentralManager, connectionEventDidOccur event: CBConnectionEvent, for peripheral: CBPeripheral) {
		os_log("connectionEventDidOccur for peripheral: %@", peripheral)
		switch event {
		case .peerConnected:
			cbPeripherals.append(peripheral)
		case .peerDisconnected:
			os_log("Peer %@ disconnected!", peripheral)
		default:
			if let idx = cbPeripherals.firstIndex(where: { $0 === peripheral }) {
				cbPeripherals.remove(at: idx)
			}
		}
		tableView.reloadData()
	}

Although this would require changes to the IOS side so that the devices were managed using both Core Bluetooth and EAccessory getBondedDevices() which again, doesn't help much since even if you see the device, you cannot connect to it.
Although, with that said, it is possible now to write your own connection logic, so there would be nothing stopping you from wrapping a BLE connection within this library now.
Let me think about it.

Any word on this, or should I consider it working as intended?

Any word on this, or should I consider it working as intended?

Any word on this, or should I consider it working as intended?

Thanks for the reply,
I have changed my mind and used "react-native-ble-plx" library instead of 'react-native-bluetooth-classic' and it is working for me without 'MFi'.

Any word on this, or should I consider it working as intended?

Thanks for the reply,
I have changed my mind and used "react-native-ble-plx" library instead of 'react-native-bluetooth-classic' and it is working for me without 'MFi'.

if you can use BLE with your devices that's probably for the best.
Thanks for the update.

if you can use BLE with your devices that's probably for the best.
Thanks for the update.

Hello @kenjdavidson ,
sorry to reopen this issue, but i'm facing a similar issue.
What i'm trying to do : get generic available devices around with iOS (and Android but it seems easier), concretely i would like to know if the user is near his car (which often uses old bluetooth versions, and assuming that he had already paired his iOS phone with his car).
if I'm not mistaken, with this librairie i can't do that ?  ( getBondedDevices return an empty array, and getConnectedDevices() return [TypeError: this._nativeModule.getConnectedDevices is not a function. (In 'this._nativeModule.getConnectedDevices()', 'this._nativeModule.getConnectedDevices' is undefined)] )

getBondedDevices returns a list of (yup) the currently bonded devices.  These are the devices that the user has gone through the process of pairing with.  How this works is pretty straight forward:
Android

User pairs device
User sees device as paired in System / Bluetooth
User calls getBondedDevices()
User gets a list of devices

Now, where you're having the problem (and there is literally nothing I can do)
IOS

User pairs the device with IOS (sweet it works)
User sees device in System / Bluetooth (great)
User calls getBondedDevices()
Empty

This is due to how Apple manages Bluetooth Classic devices through MFi protocols.   Yes, that device is connected and paired, but Apple won't let you see / use it without the MFi Protocols.
So lets take your example:

You pair your phone to your car (sweet it works)
Behind the scenes if your car (Ford for example) is MFi compliant, which it probably has to be, there will be protocols that Apple is aware of, say com.ford.apple-protocol, com.ford.manage-contacts, com.ford.manage-audio, etc.
As a vendor, you need to go to Ford and ask for those.  Without them you don't get to work with the connected device (hence you see an empty list)

This is just how Mfi works, it's annoying and stupid and yet another reason why I can't stand Apple as a company.  But such is life.

Feel free to browse through:

https://developer.apple.com/accessories/Accessory-Design-Guidelines.pdf
https://mfi.apple.com/en/how-it-works.html

How MFi Certification Works
Here is an overview of the steps required to create an MFi accessory, from conception to production.

1. Product Plan
Submit a Product Plan for an accessory that integrates MFi licensed technology.

2. Development
Design, develop, and test the accessory. Procure any MFi components as needed.

3. Certification
Use Apple’s certification tools to help ensure optimal user experience and interoperability. Submit production-ready samples and packaging materials for review.

4. Mass Production
Upon completion of certification approved by Apple, begin manufacturing and sales.

https://medium.com/theengineeringgecko/external-accessory-framework-what-why-how-d0a9fad52271

Essentially, unless you go to every car manufacturer and ask/pay for their protocol string, you're out of luck.  Or you use BLE, which doesn't have these issues.

With regards to the getConnectedDevices, you are correct.  I missed implementing that function when I moved the Android logic over to IOS.  My bad.  I'll try to get to it at some point, I've opened #92 for this purpose.
Although, getConnectedDevices will/should only return the Device information for those devices you've already established an RFCOMM socket with - which requires MFi protocols anyhow.
It's really just a convenience method when connecting to multiple devices (which to be honest no one has really even documented that they are using).  In generally most applications connect to 1 device at a time, and therefore don't really even need it:

They remember the Address which was connected and calls isDeviceConnected(address) instead
Just loop through getBondedDevices() and re-connect to the device they were last connected

Thank you for your detailed answer , i will try to use BLE but my guess is that all cars may not have bluetooth 4.0

No problem man, it's been a "wonderous" ride attempting to learn about Apple and how they deal with Bluetooth Classic.  So I feel your pain and sadness.
Good luck!! Hit me up anytime, I'll do my best to help.

authors: WO-Mehul kenjdavidson psyzz


text:
Content Streaming

@mitsuhiko

:sparkles: :cake: :sparkles:

repo: kennethreitz/requests


text:
styleguide

Which do you prefer? .eslint or .jshint single or double quotes etc etc.

I definitely prefer eslint. For my full configuration preferences, see eslint-config-kentcdodds

authors: benhalverson kentcdodds

repo: kentcdodds/starwars-names


text:
Adding Sli spec link

This PR fixes #1497

thank you for your contribution, it seems like there is a merge conflict based on your previous contribution :) can you please take a short look.
The change itself looks fine, but can you do this for the 1.0.x version and not for 0.19.x version, as the 1.0.x is the most recent one.
Besides that, it looks good to me :)

thank you for your contribution, it seems like there is a merge conflict based on your previous contribution :) can you please take a short look.
The change itself looks fine, but can you do this for the 1.0.x version and not for 0.19.x version, as the 1.0.x is the most recent one.
Besides that, it looks good to me :)

Done!


text:
Remediation action for slowing down requests

Implement a remediation action, which slows down requests from a specific IP.

Implementation on keptn side finished. Next, do integration.

repo: keptn/keptn


text:
mock image availability check responses

previously, we called external services such as docker.io or quay.io in the TestCheckImageAvailablity unit test. These endpoints have now been mocked to ensure our unit tests are not impacted by potential downtimes of those services

Codecov Report

Merging #1813 into master will decrease coverage by 0.00%.
The diff coverage is 71.42%.

@@            Coverage Diff             @@
##           master    #1813      +/-   ##
==========================================
- Coverage   32.12%   32.11%   -0.01%
==========================================
  Files          99       99
  Lines        7579     7581       +2
==========================================
  Hits         2435     2435
- Misses       4843     4844       +1
- Partials      301      302       +1

Impacted Files
Coverage Δ

cli/pkg/docker/docker_utils.go
68.75% <60.00%> (-4.59%)
:arrow_down:

cli/cmd/install.go
24.44% <100.00%> (ø)

cli/cmd/send_event_newArtifact.go
78.12% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4ef57f7...0a90280. Read the comment docs.

authors: bacherfl codecov-commenter


text:
Auto-update kubernetes-utils to latest version

:robot: Beep boop I am a bot
This is an automatically created PR to change keptn/kubernetes-utils to version cfcfe1b629aa9faa6e1248af6c41a72449cf0de0.
Please consult https://github.com/keptn/kubernetes-utils/actions?query=workflow%3A"Auto+PR+to+keptn%2Fkeptn" for more information.

Codecov Report

Merging #2854 (7a2a8b0) into master (265eef7) will decrease coverage by 7.22%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2854      +/-   ##
==========================================
- Coverage   36.08%   28.86%   -7.23%
==========================================
  Files         104       10      -94
  Lines        6460      887    -5573
  Branches      197        0     -197
==========================================
- Hits         2331      256    -2075
+ Misses       3821      604    -3217
+ Partials      308       27     -281

Flag
Coverage Δ

moduleA
?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

bridge/client/app/_models/event-labels.ts

...vice/event_handler/configure_monitoring_handler.go

...ouse-service/event_handler/evaluate_sli_handler.go

..._components/ktb-markdown/ktb-markdown.component.ts

helm-service/controller/release_handler.go

bridge/client/app/_services/http-state.service.ts

...b-expandable-tile/ktb-expandable-tile.component.ts

...nts/ktb-project-list/ktb-project-list.component.ts

...ponents/ktb-event-item/ktb-event-item.component.ts

helm-service/controller/handler_base.go

... and 84 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 265eef7...7a2a8b0. Read the comment docs.

authors: christian-kreuzberger-dtx codecov-io


text:
fix(installer): Remove obsolete openshift-route-service (#6272)

Closes #6272
The openshift-route-service has been used for setting up the configuration needed for enabling Istio injection in a namespace within an openshift cluster, as described here: https://istio.io/latest/docs/setup/platform-setup/openshift/ - in a production environment this might not be favorable because this requires the openshift route service to have cluster admin rights. Therefore, I propose to remove this service from Keptn, since it also never been touched/used/tested in the last couple of months.

Found some additional occurrences of the OS route service in codecov.yml, cleanup_docker_images.sh and detect_stale_images.sh

authors: bacherfl mowies


text:
Is The WGAN Wasserstein Loss Function Correct?

I'm struggling to understand why the Wasserstein Loss function is correct, as seen below:
def wasserstein_loss(y_true, y_pred):
    return K.mean(y_true * y_pred)

The comment in the code says that this works if your labels are -1 for generated images and 1 for real images. If we assume that disc_real and disc_fake are the outputs of the discriminator with only real and fake image, respectively, then:

y_true * y_pred = 1 * disc_real - 1 * disc_fake

Therefore, the loss functions that this defines are

loss_discriminator = 1 * disc_real - 1 * disc_fake
loss_generator = - 1 * disc_fake (since the generator only updates on fake images)

The reason that I'm skeptical of this is that in the tensorflow implementation from the Improved Training of Wasserstein GANs the authors define the following losses:
# Standard WGAN loss
gen_cost = -tf.reduce_mean(disc_fake)
disc_cost = tf.reduce_mean(disc_fake) - tf.reduce_mean(disc_real)

So in other words, it would be:

loss_discriminator = - 1 * disc_real + 1 * disc_fake
loss_generator = - 1 * disc_fake (since the generator only updates on fake images)

With the sign flipped on the discriminator loss, you would need to maximize instead of minimize loss right?
Have the authors of this implementation seen success with it? @the-moliver @Rocketknight1 @DavidS3141

Your conclusion is not quite correct for the implementation in keras. As you can see from the definition of the Wasserstein loss, it clearly depends on the labels we feed into the model. So for the discriminator the we feed +1 as label for real and -1 for fake images, here your conclusion was correct. This is indeed opposite to the TF implementation, but the sign actually does not matter. As you noted, it is important, that the generator has the opposite sign for the fake images. This is in the current implementation the case, as we feed +1 as label for the fake images (no real images are trained, so no label for them needed). So we have
loss_generator = + 1 * disc_fake

I hope this cleared things up for you!

It did, thanks for pointing that out!

Sorry but I still do not understand this:

This is indeed opposite to the TF implementation, but the sign actually does not matter.

I don't know why sign does not matter.
Let's think about only training of discriminator.
Assume that a label of 2 images is (-1, 1), which means (real, fake)
If the discriminator gives perfectly correct result, which is (-1, 1), the average loss will be 1.
If the discriminator cannot distinguish the inputs so its prediction is (0, 0), the loss will be 0.
Since training tries to minimize the loss, the better discriminator should has a smaller loss right? But in the case that I said it is not.

The paper calls this second network not a discriminator but a critic. The important difference is, that the network does not try to match the labels y_true, but tries to minimize the loss K.mean(y_true * y_pred).

If the discriminator gives perfectly correct result, which is (-1, 1), the average loss will be 1.
Dont know how you reasoned this, but essential the critic tries to minimize the output y_pred for real images (this is +1 * y_pred) and maximize the output for fake images (as it tries to minimize -1 * y_pred). If you change the sign in this definition, the critic simply has to learn a sign flip compared to the previous setting, which should be not a problem. Also if the critic can't distinguish, we have that the loss is indeed 0, as the expectation value of y_pred is independent of y_true, and as half of your data is real (y_true=1) and half is fake (y_true=-1) the loss will be 0. If the critic is able to distinguish, it will have a loss smaller than 0, if it minimizes.

I think you got the right idea, yes!
But to be precise, the actual answers to your questions would be "no". The reason for this is, that the critic can have an arbitrary offset, so if you have a critic C(x) with loss L, and another critic C'(x)=C(x)+K, which has just a shifted output by K, the loss would be still L, the same as for C, because the +K term would cancel for all the fake and real data. Because of this, it is possible, that a perfectly trained critic can have only positive or only negative values. And than there is also the gradient condition, which is making sure, that the y_pred cant get arbitraily large. So your second statement is in general true, but this scale absolute(y_pred) for which you have a good critic depends heavily on the used dataset.

So I understand that we can define the losses with either sign, which would be why its ok to define the losses as disc_real - disc_fake instead of the author's disc_fake - disc_real if you also change the sign on the generator's loss, because you are essentially saying disc_fake-disc_real = - (disc_real-disc_fake).
But considering that the actual loss function is disc_fake - disc_real + grad_pen and the gradient penalty is defined the same way in both implementations, shouldn't we have to subtract the gradient penalty loss in this implementation (ie, multiply it by -1) in order to achieve an equivalent function?

authors: DavidS3141 JiamingJiao justinessert

repo: keras-team/keras-contrib


text:
Loops in Keras backend.

I'm searching for the discussion somewhere but if i remember correctly somewhere in this video https://www.youtube.com/watch?v=90-S1M7Ny_o Jeff Dean talked about the fact that TensorFlow do not support loops, they basically unfold the scan each time. I think we must do the same for our backend if we don't want fragmentation.
In theano should be possible to unfold a scan. and so in tensor flow. In my opinion we should do the same.

btw, I updated my TF LSTM example to be independent of batch size. The price was an initial hidden state needing to be defined as input, which I believe is not too bad. https://github.com/EderSantana/TwistedFate/blob/master/mnist_lstm.py

I am going to write a loss function in which a piece-wise differentiable function will have to be implemented, that is, converting different element in y_true into another value depending on its own value. Is therefore it possible to implement a for-loop in tf backend?

authors: EderSantana dbonadiman yourtheron

repo: keras-team/keras


text:
BatchNormalization layer fails with torch backend on GPU with scale=False, center=True

Here is the colab for reproducing it.
link

The bug is fixed by https://github.com/keras-team/keras/commit/9815ac1c81f0f24cf3ca24638e4ab11c5fc95bd1.
However, the torch BN is converging slower than other backends. Needs further investigation.


text:
Loading model Conv2DTranspose from h5 file results in error complaining about group:1 parameter

Loading a .h5 model created in TensorFlow.keras containing Conv2DTranspose layer in keras-core results in an error:
Exception encountered: Unrecognized keyword arguments passed to Conv2DTranspose: {'groups': 1}
A hacky fix is to edit keras_core/src/ops/operation.py in the method from_config() so:
    try:
        if cls.__name__=="Conv2DTranspose" and "groups" in config:
            config.pop("groups")
        return cls(**config)
....

I've yet to confirm the resultant network behaves exactly the same as the original, but it seems to work. It's obviously not the ideal solution. I'm not quite sure what that parameter does, as it's not something I used when creating the network. I imagine most legacy networks using Conv2DTranspose will have the same issue.
Model is here if helpful:
https://drive.google.com/file/d/1Jn9C4vTRVWgHCqAXnMfO_i-B3pK4c14I/view?usp=sharing
Note:it also has the issue raised here about Lambda layers: https://github.com/keras-team/keras/issues/19151
(I proposed a hack there, and it looks like a better solution is being worked on)
Derek

Can you try to create a model in Keras 3 instead of keras_core and try to load it in the same version and let us know the outcome of it. Thanks

I'm a bit confused about keras versions. I thought Keras 3 and keras-core were basically the same thing? The model I'm having issues with was created by tf-keras, and I'm trying to load into the latest version of keras-core (installed via pip as keras-core). Do you actually want me to try to install keras 3 (if so how), or do you wnat me to try to generate a model in keras-core (this is on the todo list for next week anyway).

keras-core was a pre-release for Keras 3, now Keras 3 is available and all the latest updates will be made available in the Keras 3 release.
you can install it though pip install -U keras

I'm only getting keras 2.15.0 when I do that? I'm on windows with Python 3.8. Am I doing something wrong, or has Keras 3 not reached windows yet?

Also, 2.15.0 doesn't actually allow you to import keras as it fails with a tensorflow related error. The maximum tensorflow version that seems to be available is 2.13.1, which works with keras 2.13.1.

I managed to install keras3 by starting with a fresh installation of python 3.11 and just installing pytorch and keras (plus packaging and pandas which appear to be dependencies of keras but aren't configured as such). The same problem(s) exist with this version as with keras_core loading historic models from tf-keras, and the same hacky fix(es) work to get the model loaded. I'm going to port our model generation code over now, and see if a h5 generated by keras3 will have the same issue (not that that solves anything for legacy models). I'll report back.

Could you please provide some sample reproducible code to replicate the reported behavior. Thanks

The minimal code below will throw that error with the model I linked above. However, first you need to fix https://github.com/keras-team/keras/issues/19151. I proposed a hack for that there, but I see there are some posts, so I'll go respond to that now.
import os
os.environ["KERAS_BACKEND"] = "torch"

import keras
from keras.layers import Lambda

@keras.saving.register_keras_serializable()
class Linear(keras.layers.Layer):
    def __init__(self, units=32, input_shape=None, name=None,**kwargs):
        super(Linear, self).__init__(trainable=True, name=name)
        self.op_shape = input_shape
        self.units=units
        self.name=name

    def build(self, input_shape):

        self.b = self.add_weight(
            name="b",
            shape=(self.units,),
            initializer=keras.initializers.RandomNormal(mean=0.0, stddev=0.01),
            trainable=True
        )
        super(Linear, self).build(input_shape)

    def get_config(self):
        config = super(Linear, self).get_config()
        config['units'] = self.units
        config['input_shape'] = self.op_shape
        return dict(list(config.items()))
    @classmethod
    def from_config(cls, config):
        units_config = config.pop("units")
        units = keras.saving.deserialize_keras_object(units_config)
        input_shape_config = config.pop("input_shape")
        input_shape = keras.saving.deserialize_keras_object(input_shape_config)
        name = config.pop("name")

        return cls(units=units,input_shape=input_shape,name=name)

    def call(self, inputs):

        return keras.layers.LeakyReLU(negative_slope=0.01)(inputs + keras.ops.broadcast_to(self.b,[self.op_shape[0],self.op_shape[1],self.units]))

with keras.utils.custom_object_scope({'Linear': Linear}):
        model = keras.models.load_model('old_keras_model.h5',compile=False, custom_objects={'LeakyReLU': keras.layers.LeakyReLU(negative_slope=0.01),'Linear':Linear})

Unfortunately this code snippet is not reproducible since it refers to 'old_keras_model.h5'. Do you have a reproducible code snippet? You can attach files to GitHub comments.

@fchollet I attached the model above. For convenience it is here: https://drive.google.com/file/d/1Jn9C4vTRVWgHCqAXnMfO_i-B3pK4c14I/view?usp=sharing

Adding this solves the problem (click me.)

# hack to change model config from keras 2->3 compliant
import h5py
f = h5py.File("/content/old_keras_model.h5", mode="r+")
model_config_string = f.attrs.get("model_config")
if model_config_string.find('"groups": 1,') != -1:
    model_config_string = model_config_string.replace('"groups": 1,', '')
    f.attrs.modify('model_config', model_config_string)
    f.flush()
    model_config_string = f.attrs.get("model_config")
    assert model_config_string.find('"groups": 1,') == -1

f.close()

As it was indicated in #19151, in this comment.
Is the it fixed for you @dmagee ?
I include a GIST that downloads and instantiates the model and problem (removing the snippet I provided above from it).
CC @fchollet

try using tensorflow version 2.12.1, worked for me.

authors: CS-DaviMagalhaes dmagee fchollet ghsanti sachinprasadhs


text:
why is my siamese network not working ?

import tflearn
from tflearn.layers.conv import conv_2d, max_pool_2d
from tflearn.layers.core import input_data, dropout, fully_connected
from tflearn.layers.estimator import regression
import tflearn.datasets.mnist as mnist

train_x1, train_y1, test_x1, test_y1 = mnist.load_data(one_hot=True)
train_x2, train_y2, test_x2, test_y2 = mnist.load_data(one_hot=True)
train_x1 = train_x1.reshape ([-1,28,28,1])
test_x1 = test_x1.reshape ([-1,28,28,1])
train_x2 = train_x2.reshape ([-1,28,28,1])
test_x2 = test_x2.reshape ([-1,28,28,1])

# define the first network
def tower_network(reuse = False):
        network = input_data(shape=(None,28,28,1))
        network = conv_2d(network, 32,1, activation='relu',reuse=reuse, scope='conv1')
        network = conv_2d(network, 64,1, activation='relu',reuse=reuse, scope='conv2')
        network = conv_2d(network, 128,1, activation='relu',reuse=reuse, scope='conv3')
        network = max_pool_2d(network, 2, strides=2)
        network = fully_connected(network, 512, activation='relu',reuse=reuse, scope='fc1')
        network = dropout(network, 0.5)
        return network

def similarity_network( net1, net2):
    num_classes = 2
    network = tflearn.merge([net1,net2], mode='concat', axis=1, name='Merge') # merge net1 and net2 networks
    # fully connected layers
    network = tflearn.fully_connected(network, 2048, activation='relu')
    network = tflearn.dropout(network, 0.5)
    network = tflearn.fully_connected(network, 2048, activation='relu')
    network = tflearn.dropout(network, 0.5)
    # softmax layers
    network = tflearn.fully_connected(network, num_classes, activation='softmax')
    return network

if __name__ == "__main__":
    net1 = tower_network()
    net2 = tower_network(reuse=True)
    network = similarity_network( net1, net2)
    network = fully_connected(network, 10, activation='softmax')
    network = regression(network, optimizer='adam', learning_rate=0.01, loss='categorical_crossentropy', name='targets')
    model = tflearn.DNN(network, tensorboard_verbose=3)
    model.fit([train_x1,train_x2], [train_y1,train_y2],n_epoch=1)

Hi!
Tensorflow doesn't use keras functions, thus you'll have way more answers by submitting the issue to the Tensorflow repository or (even better) stackoverflow. The issues in the keras repo arre to report bugs in the keras codebase. For example, I know keras well, but not Tensorflow, so I can't help you. Just tell you where to ask for help.
I hope it helps you.

authors: gabrieldemarmiesse ufdul


text:
Fix ImageDataGenerator preprocessing_function

preprocessing_function must run before any other modification on image (by documentation)
by issue
Discussed on keras-user-googlegroup, I suggested preprocessing function's option parameter for executing before resize.
But It must run before any modification according to documentation.

Which resize is being used, bilinear or nearest neighbor? Have you tried bilinear interpolation?
This other preprocessing issue may also be related:
https://github.com/keras-team/keras/issues/8916

@ahundt In keras image preprocess function - load_img(target_shape) uses PIL.Image.NEAREST.
Haven't tried bilinear option. Thanks for the notice.

On your google-doc you add a argument to specify the order, why is it not the case here?

@Dref360 By DataImageGenerator Document, preprocessing_function should be run before any other modification on it.
I noticed issue - (resize before preprocessing) isn't the considered case, but it was a bug.
So I canceled argument suggestion and correct a bug.
In my opinion, google-doc's argument should be added after discussing change description of DataImageGenerator - preprocessing_function

@davinnovation, @Dref360
I started getting the following exception, [I am pretty sure] due to this PR:
Traceback (most recent call last):
  File "oleg/ResNet50.NewClasses.py", line 118, in <module>
    validation_data=test_generator
  File "/usr/local/lib/python3.5/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/training.py", line 2192, in fit_generator
    generator_output = next(output_generator)
  File "/usr/local/lib/python3.5/dist-packages/keras/preprocessing/image.py", line 860, in __next__
    return self.next(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/keras/preprocessing/image.py", line 1283, in next
    return self._get_batches_of_transformed_samples(index_array)
  File "/usr/local/lib/python3.5/dist-packages/keras/preprocessing/image.py", line 1233, in _get_batches_of_transformed_samples
    img = self.image_data_generator.preprocessing_function(img)
  File "/usr/local/lib/python3.5/dist-packages/keras/applications/imagenet_utils.py", line 178, in preprocess_input
    mode=mode)
  File "/usr/local/lib/python3.5/dist-packages/keras/applications/imagenet_utils.py", line 132, in _preprocess_symbolic_input
    x = x[..., ::-1]
TypeError: 'JpegImageFile' object is not subscriptable

When using
train_datagen = image.ImageDataGenerator(preprocessing_function = resnet50.preprocess_input)
train_generator = train_datagen.flow_from_directory("/dir/with/jpegs")

doing print(img) before line 1233 prints
<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=224x224 at 0x7F6AD4FC2D30>

@ozabluda thanks for reporting.
By keras documentation, Imagedatagenerator’s preprocessing function must return numpy tensor, but resnet50.preprocessing_input function returns ‘JpegImageFile’ type.

I’ll look for this. Thanks for review. @ozabluda

Hi guys
Great ideas you are working on. One question: When fine tuning existing models like Inception V3, there is also a preprocessing_input function that cares about appropriate rescaling and centering. This is typically done as the last step of input preparation. How does this fit in your framework? Would it even be possible to have "preprocess_function" (applied before resizing) and "postprocess_function" (applied as final step)?

@ozabluda I confirm this is a side effect of this PR. The preprocessing function is supposed to receive and return numpy arrays not Image Objects. There are more side-effects explained here: https://github.com/keras-team/keras/commit/738819de0b7e6bc45abed8d0640f02b81c6ac4e9

This seems to be a massive issue, all the pretrained networks are unusable with the image generator? https://github.com/keras-team/keras/issues/9624

@kmader Indeed, it was agreed to rollback the change.
You can read more on the details here: https://github.com/keras-team/keras/commit/738819de0b7e6bc45abed8d0640f02b81c6ac4e9
I guess this PR will be reverted soon.

v2.1.5 is not fully satisfactory. It describes preprocessing_function as
            The function should take one argument:
            one image (Numpy tensor with rank 3),
            and should output a Numpy tensor with the same shape.

but _get_batches_of_transformed_samples uses it on image and expects image to be returned (to be used in img.resize). Either the doc or the code has to be fixed.
I do all the preprocessing in my own preprocessing_function which does cropping and image normalisation. It looks to me that doing this at the very beginning is the best and I wish I could totally bypass Keras preprocessing if I want. As of 2.1.5 I can't because target_size is not allowed to be None.
I agree with @datumbox that the expected action of preprocessing_function should be maintained compatible with keras.applications.imagenet_utils.preprocess_input but in 2.1.5 it is not.

Is this issue resolved? I get different answers when I use preprocess_input of pretrained networks of keras.applications and when I use _get_batches_of_transformed_samples without the preprocess. The second one seems to be the correct one, the first gives skewed predictions which I fail to understand.

Nope. I've made a gist here to demonstrate this issue. It sucks!

Is this bug corrected yet? Reading the current documentation of preprocessing_function seems like it is not. (https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/image/ImageDataGenerator)
Is there a workaround to this bug?
I need to apply the preprocessing function first to make my images square after filling them with empty spaces on the edges and then resize them.

authors: Dref360 ahundt crearo datumbox davinnovation ink1 joshinkumar kmader mayer79 ozabluda tonmoyborah


text:
Add SkiplistSet implementation

Add SkiplistSet, which is identical to Skiplist but only permits one copy of each key.

Codecov Report

Merging #5 into master will decrease coverage by 3.37%.
The diff coverage is 89.47%.

@@            Coverage Diff             @@
##           master       #5      +/-   ##
==========================================
- Coverage   91.04%   87.67%   -3.38%
==========================================
  Files           4        4
  Lines         134      146      +12
==========================================
+ Hits          122      128       +6
- Misses         12       18       +6

Impacted Files
Coverage Δ

src/Skiplists.jl
100.00% <ø> (ø)

src/core.jl
100.00% <ø> (ø)

src/node.jl
79.59% <73.33%> (-3.75%)
:arrow_down:

src/list.jl
91.11% <95.23%> (-3.01%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5287d33...14ac42a. Read the comment docs.

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@5287d33). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master       #5   +/-   ##
=========================================
  Coverage          ?   87.67%
=========================================
  Files             ?        4
  Lines             ?      146
  Branches          ?        0
=========================================
  Hits              ?      128
  Misses            ?       18
  Partials          ?        0

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5287d33...14ac42a. Read the comment docs.

Fixed #1

authors: codecov-commenter kernelmethod


text:
Persisting serializers/deserializers to an assembly for better startup time

This is a great library. I have a suggestion for an issue we're seeing in a production environment with a decent-sized object graph.
Here's the scenario:

On startup, the app performs serialization/deserialization pairs on dummy object graphs (async in a Task)
This process takes several seconds, is heavy on CPU, and memory usage climbs to over 1GB.
Once done, most of the memory is reclaimed and serialization/deserialization performance is excellent as would be expected.

What I'd like to accomplish:

During the build process, I want to launch a step that generates the serializers/deserializers I need and write them to an assembly.
At runtime, I want to specify this assembly in some type of Jil initialization method. Jil would then look first to this assembly and use the prebuilt serializers/deserializers as needed.
I will assume all risks for impedance between the object in question and the prebuilt assembly.

I can take a crack at this change and send a pull request, unless you see a problem with this request or advise against it for whatever reason. The end goal is to reduce the startup initialization time so how that's done isn't particularly important.

How about using weaver like Fody and provide a plugin which does that? Having one assembly instead of *.Serializers is better imho.

I'd also like something like this, but for a slightly different use case, .net native. If you could pre-generate the serializers and put them in an assembly and then reference that, .net native would have an easier time using them.
i'd love to take a stab at it but i cant commit to a timeframe unfortunately :/

The work required to do this, at the current time, is ... extensive. You
can't do it using the platform-specific builds, as many (core-clr, etc)
lack emit to dll options. You can't do it using the regular .NET 4.* build,
as IL-emit doesn't allow cross-targeting (it will reference the wrong
runtimes). The way I do it in protobuf-net is to use IKVM as a bridge, as
that has a cloned IL-emit API that permits cross-targeting... but to do
that would require a rewrite of Sigil, or the removal of Sigil completely.
In the general case, at that point, I'm (from bitter-sweet experience)
reaching the conclusion that the better option is C# code generation. There
aren't currently any friendly ways to deploy code-gen tools, though, if you
expect them to work in various build tools.
On 9 December 2015 at 08:41, Allan Lindqvist notifications@github.com
wrote:

I'd also like something like this, but for a slightly different use case,
.net native. If you could pre-generate the serializers and put them in an
assembly and then reference that, .net native would have an easier time
using them.
i'd love to take a stab at it but i cant commit to a timeframe
unfortunately :/
—
Reply to this email directly or view it on GitHub
https://github.com/kevin-montrose/Jil/issues/88#issuecomment-163151237.

--
Regards,
Marc

I havent looked super closely at the code yet, but do you think it would be possible to codegen all the tricks that jil does to speed up serialization? i have my own internal hobby project were i've been thinking about codegening serializers but it hasnt really materialized into something useful and i wasn't aware of jil at the time, so i'm curious to learn from this project :)

Can it be done? yes, mostly (see below)
Can it be done with the existing code: no
Code-gen is radically different to il-emit, and it would essentially be a
rewrite of the core. I'm not saying that rules it out, but it at least sets
the context.
Note also that code-gen doesn't allow all the tricks, because the
compiler is going to want to enforce things like accessibility rules. If
Jil currently allows access to private members (and I can't remember
whether or not it does), that won't work. Also, some of what Jil does: it
does by performing IL analysis of code that isn't necessarily in the
current project. For example, it actually does magic to figure out whether
the return value from a property is going to be constant value, and it
does analysis of things like constructors to determine intent. Those things
won't necessarily work the same, unless a way can be found to do equivalent
functionality. Core-CLR already has to make some compromises here, because
IL of other methods isn't as readily available (although I have some code I
need to try, there) - so it won't be a first. Likewise, Rosyln potentially
provides access to much of that info, but only for in-project code.
On 9 December 2015 at 15:24, Allan Lindqvist notifications@github.com
wrote:

I havent looked super closely at the code yet, but do you think it would
be possible to codegen all the tricks that jil does to speed up
serialization? i have my own internal hobby project were i've been thinking
about codegening serializers but it hasnt really materialized into
something useful and i wasn't aware of jil at the time, so i'm curious to
learn from this project :)
—
Reply to this email directly or view it on GitHub
https://github.com/kevin-montrose/Jil/issues/88#issuecomment-163289914.

--
Regards,
Marc

authors: Scooletz aL3891 craigajohnson mgravell

repo: kevin-montrose/Jil


text:
aiohttp stub records the full url in the recorded data regardless of parameter filtering

When using the filter_query_parameters argument, the aiohttp stub still records the full url given by the response object for later reconstruction of the yarl response.  This is an unexpected behavior and can lead to secrets being leaked.  Based on a review of the code, I think the Tornado stub may also have this same issue (but I haven't verified that).  This leads us to having to manually scrub filtered values from cassettes to avoid secrets being committed into repositories.
Here is an example test case and recorded cassettes. The query parameter sometoken is intended to be dropped from storage (please ignore the fact that that the test target, httpbin, includes the query parameters in the response body).
import os
import json
import asyncio
import unittest

import vcr
import aiohttp
import requests

URL = 'https://httpbin.org/get?arg1=val1&sometoken=sekrit'
ASSETDIR = './'

FILTER_QUERY_PAREMETERS = ['sometoken']

class FilterQueryParamTest(unittest.TestCase):
    # Helper code, replicated here
    def _getVcrArgs(self):
        kwargs = getattr(self, 'vcr_kwargs', {})
        return kwargs

    def getVcr(self, **kwargs):
        # simplification for the vcr-unittest library.
        fn = '{0}.{1}.yaml'.format(self.__class__.__name__,
                                   self._testMethodName)
        fp = os.path.join(ASSETDIR, fn)
        _kwargs = self._getVcrArgs()
        _kwargs.update(kwargs)
        myvcr = vcr.VCR(**_kwargs)
        cm = myvcr.use_cassette(fp)
        return cm

    def test_requests(self):
        with self.getVcr(filter_query_parameters=FILTER_QUERY_PAREMETERS):
            resp = requests.get(URL)
            d = resp.json()
        self.assertIsInstance(d, dict)

    def test_aiohttp(self):
        async def coro():
            with self.getVcr(filter_query_parameters=FILTER_QUERY_PAREMETERS):
                async with aiohttp.ClientSession() as session:
                    async with session.get(URL) as response:
                        text = await response.text()
            return text

        result = asyncio.run(coro())
        d = json.loads(result)
        self.assertIsInstance(d, dict)

This is the requests output which does not have the filtered value present in any vcrpy constructs:
interactions:
- request:
    body: null
    headers:
      Accept:
      - '*/*'
      Accept-Encoding:
      - gzip, deflate
      Connection:
      - keep-alive
      User-Agent:
      - python-requests/2.23.0
    method: GET
    uri: https://httpbin.org/get?arg1=val1
  response:
    body:
      string: "{\n  \"args\": {\n    \"arg1\": \"val1\", \n    \"sometoken\": \"sekrit\"\n
        \ }, \n  \"headers\": {\n    \"Accept\": \"*/*\", \n    \"Accept-Encoding\":
        \"gzip, deflate\", \n    \"Host\": \"httpbin.org\", \n    \"User-Agent\":
        \"python-requests/2.23.0\", \n    \"X-Amzn-Trace-Id\": \"Root=1-5e849e8a-908aad86267b92fe372fa1da\"\n
        \ }, \n  \"origin\": \"8.8.8.8\", \n  \"url\": \"https://httpbin.org/get?arg1=val1&sometoken=sekrit\"\n}\n"
    headers:
      Access-Control-Allow-Credentials:
      - 'true'
      Access-Control-Allow-Origin:
      - '*'
      Connection:
      - keep-alive
      Content-Length:
      - '385'
      Content-Type:
      - application/json
      Date:
      - Wed, 01 Apr 2020 14:00:42 GMT
      Server:
      - gunicorn/19.9.0
    status:
      code: 200
      message: OK
version: 1

This is the aiohttp response which does include the unfiltered token in the url key of the response:
interactions:
- request:
    body: null
    headers: {}
    method: GET
    uri: https://httpbin.org/get?arg1=val1
  response:
    body:
      string: "{\n  \"args\": {\n    \"arg1\": \"val1\", \n    \"sometoken\": \"sekrit\"\n
        \ }, \n  \"headers\": {\n    \"Accept\": \"*/*\", \n    \"Accept-Encoding\":
        \"gzip, deflate\", \n    \"Host\": \"httpbin.org\", \n    \"User-Agent\":
        \"Python/3.7 aiohttp/3.6.0\", \n    \"X-Amzn-Trace-Id\": \"Root=1-5e849e89-2fdc0c40f507cd205231bb40\"\n
        \ }, \n  \"origin\": \"8.8.8.8\", \n  \"url\": \"https://httpbin.org/get?arg1=val1&sometoken=sekrit\"\n}\n"
    headers:
      Access-Control-Allow-Credentials: 'true'
      Access-Control-Allow-Origin: '*'
      Connection: keep-alive
      Content-Length: '387'
      Content-Type: application/json
      Date: Wed, 01 Apr 2020 14:00:41 GMT
      Server: gunicorn/19.9.0
    status:
      code: 200
      message: OK
    url: https://httpbin.org/get?arg1=val1&sometoken=sekrit
version: 1

Edit:
This behavior was tested against the current master of the vcrpy project, but has also been observed in the latest 4.x release as well.

Just wanted to note here that this can be remedied "in the mean time" with something like this:
def filter_response(response):
    response["url"] = ''  # hide the URL
    return response

..., and then making your own VCR instance, passing filter_response to the before_record_response kwarg. This doesn't seem to impact the later response playback in any negative way.

Here's a more elaborate implementation I've been using that does the actual response["url"] query param filtering:
https://github.com/scop/pytekukko/blob/c43533c8c8ff46f5fc1114e0d19a9cde58c89ae9/tests/test_pytekukko.py#L36-L50
Haven't tried just emptying the whole URL myself, nice if that works, but this one FTR in case someone finds a problem with the emptying approach.

authors: DevilXD scop vEpiphyte


text:
Account for no files in table

mastDownload.py throws an error when there are no files found.

@taylorbell57 I decided to take your suggestion and convert the level 4 plot to level 3.

@taylorbell57 I decided to take your suggestion and convert the level 4 plot to level 3.

You'll need to change the docs pages accordingly