text:
Add bfloat16 floating-point format support based on AMP

Description
Bfloat16 is wildly used in Deep Learning especially on training to get a better performance.
This PR is to add bf16 support based on mxnet AMP(Automatic Mixed Precision) module .
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[done ] Changes are complete (i.e. I finished coding on this PR)
[done ] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[ done] To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

This PR has passed unitest and preci test in local machine.
Unit tests are added for this PR.
@ZhennanQin @ElaineBao @xinyu-intel @TaoLv @PatricZhao

For bfloat16 training, which loss scalar is recommended? Do we also need to perform NaN checks?

Bfloat16 has the same dynamic range as float32, since they have the same exponent bits. So it can represent gradients directly, it doesn't require loss scaling like fp16.

@ptrendx thanks for your review.  Feel free to let me know if you have other concerns, we are going to merge PR recently.

Hi @leezu, @larroy, this PR can pass the builds for ARM but always hit the time out of the test.
http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Fedge/detail/PR-17265/29/pipeline
We don't have any environment to reproduce. Could you please take a look or have any suggestion for further debugging?

@larroy Do you have idea how to display more logs for the edge tests? It consistently fail at this stage.

I am merging now. If there're any other comments, we can resolve by new PR.

authors: ElaineBao TaoLv pengzhao-intel rongzha1 zhreshold


text:
Fix interleave matmul doc

Description
Fix interleave matmul doc
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[ ] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[ ] To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

@mxnet-bot run ci [all]

source: gharchive/pull-request

repo: apache/incubator-mxnet

license_type: permissive

license_source: github-api


text:
apps: update licenses to Apache

Summary
Gregory Nutt has submitted the SGA and  Haltian Ltd has submitted the SGA
as a result we can migrate the licenses to Apache.
Impact
LICENSE
Testing
NONE

@xiaoxiang781216 I always do it in, when it makes sense, a separate patch

repo: apache/incubator-nuttx-apps


text:
xtensa/esp32: Refactor ESP32 WiFi driver to support station and softAP coexistence

Summary
Refactor ESP32 WiFi driver to support station and softAP coexistence.
Add Wi-Fi station and softAP coexistence config file to test.
Impact
Testing
# connect to target AP
ifup wlan0
wapi psk wlan0 mypasswd 1
wapi essid wlan0 myssid 1
renew wlan0
# You can use ifconfig to check the local IP address.

# Start softAP and DHCP server
wapi psk wlan1 19901101 1
wapi essid wlan1 nuttx-softap 1
dhcpd wlan1
# You can use phone or compute to connect AP whose name is nuttx-softap and password is 19901101

Draft: The code is mainly done, but some friends would not like to merge into master before mar 15, so I set this PR to be Draft.

repo: apache/incubator-nuttx


text:
Export FtpBackend

opendal = "0.36.0"
Could we export opendal::services::ftp::backend::FtpBackend ?

No, we can't. It's a violation of our VISION. Would like to share more details on why you want to use FtpBackend directly?

source: gharchive/issue

authors: Xuanwo gamife

repo: apache/incubator-opendal


text:
pull request

Create a new pull request from @rabbah

Hi. Did you mean to open the PR against this repo?

repo: apache/incubator-openwhisk-release


text:
discard util_test.go

Move the following two test functions to contentreader_test.go:
func TestLocalReader_ReadLocal(t *testing.T) {
...
}

func TestURLReader_ReadUrl(t *testing.T) {
...
}

Move the following one test function to misc_test.go:
func TestGetEnvVar(t *testing.T) {
...
}

create contentreader_test.go and misc_test.go if they don't exist.

no longer applicable.

repo: apache/incubator-openwhisk-wskdeploy


text:
[cdc] Refactor mysql debezium json event parser

Purpose

1.Deserialize debezium json data using debezium event entity class
2.Integrate scheme build method
3.fix database and set ddl

Tests

API and Format

Documentation

@yuzelin Thanks.

repo: apache/incubator-paimon


text:
[flink] ignore expiration check interval of partition in batch mode.

Purpose

Linked issue: close #2969

If the execution time of a batch job is short, and it's less than the interval for partition expiration checks, the partition expiration operations will not be executed. This PR aims to ignore partition expiration check interval for batch job.
Tests

API and Format

Documentation

Hi @Aitozi , could you please take a look at this PR? Thanks.


text:
require PR reviews and linear history

Based on https://cwiki.apache.org/confluence/display/INFRA/git+-+.asf.yaml+features
The changes may need tweaking - I won't know if the changes will be applied as expected - so I'll fix any issues where expected updates don't actually get applied
https://github.com/apache/incubator-pekko/issues/51

requires only 1 PR review - https://lists.apache.org/thread/0hohnn9v0xhxn0qblb196p7mcq7jzz00 and https://cwiki.apache.org/confluence/display/PEKKO/Processes requires 1 review for trivial PRs
tries to enforce that Scalafmt workflow passes - we can add more later
linear git history

Is it possible to add the header check in conjunction with scalafmt?

Which of these workflows is the 'header check'? Do we need to add a 'header check' workflow first?

Which of these workflows is the 'header check'? Do we need to add a 'header check' workflow first?

its already there, its called  Pull Requests / Check / Headers,

@Claudenw Is it possible for you to re-review this?

I'll close this as it is stale

authors: mdedetrich pjfanning

repo: apache/incubator-pekko


text:
Helix exceptions when running integration tests

Looks like this is happening after we added leadControllerResource. Lets clean up other Helix exceptions as well when we look into this.
2019/06/22 09:05:16.929 WARN [BestPossibleStateCalcStage] [HelixController-async_tasks-OfflineClusterIntegrationTest] Event aaadf5d1_DEFAULT : Failed to calculate best possible states for 1 resources.
2019/06/22 09:05:21.954 WARN [AutoRebalancer] [HelixController-pipeline-default-OfflineClusterIntegrationTest] Resource leadControllerResource has tag controller but no configured participants have this tag
2019/06/22 09:05:21.954 ERROR [CRUSHPlacementAlgorithm] [HelixController-pipeline-default-OfflineClusterIntegrationTest] 1 nodes of type INSTANCE were requested but the tree has only 0 nodes!
2019/06/22 09:05:21.955 ERROR [BestPossibleStateCalcStage] [HelixController-pipeline-default-OfflineClusterIntegrationTest] Event 512cdf47_DEFAULT : Error computing assignment for resource leadControllerResource. Skipping.
java.lang.IllegalStateException: null
	at org.apache.helix.controller.rebalancer.strategy.crushMapping.CRUSHPlacementAlgorithm$Selector.select(CRUSHPlacementAlgorithm.java:308) ~[helix-core-0.9.0.jar:0.9.0]
	at org.apache.helix.controller.rebalancer.strategy.crushMapping.CRUSHPlacementAlgorithm.select(CRUSHPlacementAlgorithm.java:119) ~[helix-core-0.9.0.jar:0.9.0]
	at org.apache.helix.controller.rebalancer.strategy.CrushRebalanceStrategy.doSelect(CrushRebalanceStrategy.java:174) ~[helix-core-0.9.0.jar:0.9.0]
	at org.apache.helix.controller.rebalancer.strategy.CrushRebalanceStrategy.select(CrushRebalanceStrategy.java:140) ~[helix-core-0.9.0.jar:0.9.0]
	at org.apache.helix.controller.rebalancer.strategy.CrushRebalanceStrategy.computePartitionAssignment(CrushRebalanceStrategy.java:92) ~[helix-core-0.9.0.jar:0.9.0]
	at org.apache.helix.controller.rebalancer.strategy.CrushRebalanceStrategy.computePartitionAssignment(CrushRebalanceStrategy.java:48) ~[helix-core-0.9.0.jar:0.9.0]
	at org.apache.helix.controller.rebalancer.strategy.AbstractEvenDistributionRebalanceStrategy.computePartitionAssignment(AbstractEvenDistributionRebalanceStrategy.java:89) ~[helix-core-0.9.0.jar:0.9.0]
	at org.apache.helix.controller.rebalancer.strategy.AbstractEvenDistributionRebalanceStrategy.computePartitionAssignment(AbstractEvenDistributionRebalanceStrategy.java:49) ~[helix-core-0.9.0.jar:0.9.0]
	at org.apache.helix.controller.rebalancer.AutoRebalancer.computeNewIdealState(AutoRebalancer.java:129) ~[helix-core-0.9.0.jar:0.9.0]
	at org.apache.helix.controller.rebalancer.AutoRebalancer.computeNewIdealState(AutoRebalancer.java:51) ~[helix-core-0.9.0.jar:0.9.0]
	at org.apache.helix.controller.stages.BestPossibleStateCalcStage.computeResourceBestPossibleState(BestPossibleStateCalcStage.java:245) ~[helix-core-0.9.0.jar:0.9.0]
	at org.apache.helix.controller.stages.BestPossibleStateCalcStage.compute(BestPossibleStateCalcStage.java:121) ~[helix-core-0.9.0.jar:0.9.0]
	at org.apache.helix.controller.stages.BestPossibleStateCalcStage.process(BestPossibleStateCalcStage.java:77) ~[helix-core-0.9.0.jar:0.9.0]
	at org.apache.helix.controller.pipeline.Pipeline.handle(Pipeline.java:68) ~[helix-core-0.9.0.jar:0.9.0]
	at org.apache.helix.controller.GenericHelixController.handleEvent(GenericHelixController.java:640) ~[helix-core-0.9.0.jar:0.9.0]
	at org.apache.helix.controller.GenericHelixController.access$400(GenericHelixController.java:117) ~[helix-core-0.9.0.jar:0.9.0]
	at org.apache.helix.controller.GenericHelixController$ClusterEventProcessor.run(GenericHelixController.java:1168) ~[helix-core-0.9.0.jar:0.9.0]

This exception is thrown from Helix code because lead controller resource is disabled for now. After the logic for lead controller resource is committed and it is enabled, this exception will be gone.

This seems like a helix issue to me. A resource disabled by operator should not be throwing exceptions, right? @jackjlli can you file an issue with Helix including the exception stack? Until helix fixes the exception we should ignore it, I suppose.

I see, I also noticed that QuickStart throws exception in the end and most integration tests take longer time to setup the cluster. Is this also related to Helix change?

The issue is filed: https://github.com/apache/helix/issues/322.
The slowness may also come from the new release of Helix, because the logic of Helix controller getting/preparing leadership has been changed in Version 0.9. But as they said it's doing the right approach in the new release.

authors: jackjlli kishoreg mcvsubbu

repo: apache/incubator-pinot


text:
[documentation][deploy] Improve helm deployment script to deploy Pulsar to minikube

Changes

update the helm scripts: bookie/autorecovery/broker pods should wait until metadata is initialized
disable autoRecovery on bookies since we start AutoRecovery in separate pods
enable function worker on brokers
provide a values file for minikube
update documentation for using helm chart to deploy a cluster to minikube

for reviewers:
this change is based on #2362 , so commit 0c49a8d is the change to review in this PR.

@merlimat can you review this? so I can get this move forward.

repo: apache/incubator-pulsar


text:
[WIP] [QUARKS-22] [COMMENTS?] StreamScope

Open for comments on "API", behavior and impl (where not marked as TODO).
The DevelopmentProvider adds StreamScope peekers.
StreamScope's will be registered with a new StreamScopeRegistry runtime service.
The Console will lookup a stream's StreamScope (to enable&configure) via the StreamScopeRegistry by its associated stream's alias or stream id.
Still working out the details of StreamScopes getting registered to make lookupBy streamAlias or id possible.

Add PlumbingStreams.StreamScope (Consumer)
Add PlumbingStreams.StreamScopeRegistry
Add quarks.topology.plumbing.StreamScope oplet (just so console can
present it differently - remove?)
Update DevelopmentProvider to add StreamScopes
Add StreamScope and StreamScopeRegistry tests

Seems like this should be a independent module rather than being part of the core api.

@ddebrunner, does utils/quarks.streamscope and quarks.utils.streamscope.jar sound good for new location?

unrelated websocket test failure

authors: ddebrunner dlaboss

repo: apache/incubator-quarks


text:
[REEF-469] Assign data splits to specific evaluators

This commit allows to assign data splits to the different evaluators using the DataLoadingService.
Several changes were done.
The main one is to inject a strategy into InputFormatLoadingService, which is in charge of assigning splits to evaluators.
New data types needed to be created. For example, DataPartition, which is just a folder, but has a location field, where the user can specify where she would like to load that data into (in certain node or rack).
We have LocationAwareJobConfs now, which binds the typical hadoop JobConf and the DataPartition objects.
I had to create external constructors for them. And removed unnecessary ones.
I also removed the EvaluatorToPartitionMapper class, most of its logic has been placed in the AbstractEvaluatorToSplitStrategy one.
JIRA:
REEF-469
Pull Request:
This closes

@markusweimer changes done. They seem more than what they actually are. I moved some things around. Tested DataLoadingREEF and BGD examples and worked fine.
BTW, the build failed due to the rat issue...

@markusweimer  changes done. Ready to review

I'll do a pass.

I'm kind of done with my pass. One thing I can't wrap my head around is the dualism between InputSplit and DataPartition. I believe DataPartition is the more general concept, right? As in: a DataPartition is a set of InputSplits with a location, referencing the physical location of the data.
However, InputSplit does provide its locations already. And not all of the InputSplits in a DataPartition need to reference the same locations.

@markusweimer
I made the changes we talked about. Renamed bunch of things and clean up some stuff, like the init method in the strategy interface.
Tested DataLoadingREEF, BGD and my app and everything seems to be working.
I also decided not to create a different DataLoader, it would have been much the same code. I tried to reuse as much code as possible (that is why I included that Abstract strategy). I know we should favor composition over inheritance, but I think in this case it is OK to have a base "template" class.
Ready for review again.

@nachocano Thanks. I will do another pass.

I've done another pass. The tests pass, BTW.

@markusweimer I updated the code with all the comments but the one of the getInputSplit(NodeDescriptor nodeDescriptor, String evalId).
That second param is taken from AllocatedEvaluator.getId. What will be the one in EvaluatorDescriptor, EvaluatorDescriptor.getNodeDescriptor.getId?
Thanks!

@markusweimer  this is ready for review again

@nachocano Thanks, I will do another pass.

@nachocano  I'm done with my review. One question left. The code validates and tests just fine.

@markusweimer  again, you are right. I just removed it. I've tested and seems to be working fine.
Thanks

@nachocano I'll test and merge if the tests pass.

authors: markusweimer nachocano

repo: apache/incubator-reef


text:
Init connector-flink-table-store

Purpose of this pull request

Because the source and sink was assigned to different developer, the initial implementation will avoid some conflicts.
Check list

[ ] Code changed are covered with tests, or it does not need tests for reason:
[ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs

@zhaomin1423 It is recommended to submit the complete code of this connector and modify the PR title, thanks

@zhaomin1423 It is recommended to submit the complete code of this connector and modify the PR title, thanks

+1, you can convert it to draft.

authors: Hisoka-X wuchunfu zhaomin1423

repo: apache/incubator-seatunnel


text:
[SCB-260] Add third party notices

Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[x] Each commit in the pull request should have a meaningful subject line and body.
[x] Format the pull request title like [SCB-XXX] Fixes bug in ApproximateQuantiles, where you replace SCB-XXX with the appropriate JIRA issue.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Run mvn clean install to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Coverage remained the same at 95.285% when pulling 70e2d4e80e7354da95c16d7ed775351d3581a258 on yangbor:master into 541202272bdd5f5b76daa11c7b6cf1c1a7d9a505 on apache:master.

authors: coveralls yangbor

repo: apache/incubator-servicecomb-saga


text:
SCB-724 Add scripts for Mac OS release

https://github.com/apache/incubator-servicecomb-service-center/issues/329 Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[x] Each commit in the pull request should have a meaningful subject line and body.
[x] Format the pull request title like [SCB-XXX] Fixes bug in ApproximateQuantiles, where you replace SCB-XXX with the appropriate JIRA issue.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Run go build go test go fmt go vet to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[x] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Codecov Report

Merging #391 into master will decrease coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #391      +/-   ##
==========================================
- Coverage   71.74%   71.71%   -0.03%
==========================================
  Files          39       39
  Lines        4130     4130
==========================================
- Hits         2963     2962       -1
  Misses       1002     1002
- Partials      165      166       +1

Impacted Files
Coverage Δ

server/service/tag.go
68.32% <0%> (-0.63%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2da49ca...a64e7be. Read the comment docs.

Coverage decreased (-0.05%) to 74.605% when pulling a64e7be5fc834f6ba2f8450d78fc10876a4327a3 on asifdxtreme:master into 2da49cae6f2f317049250d900b3ece3ceecf64f9 on apache:master.

authors: asifdxtreme codecov-io coveralls


text:
[type:refactor] optimize logging-elasticsearch&rocketmq unit test

optimize logging-elasticsearch&rocketmq unit test

Make sure that:

[ ] You have read the contribution guidelines.
[ ] You submit test cases (unit or integration tests) that back your changes.
[ ] Your local test passed ./mvnw clean install -Dmaven.javadoc.skip=true.

Codecov Report

Merging #3642 (3132960) into master (f11c8b1) will decrease coverage by 0.42%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master    #3642      +/-   ##
============================================
- Coverage     62.64%   62.21%   -0.43%
+ Complexity     5855     5821      -34
============================================
  Files           888      888
  Lines         24730    24730
  Branches       2263     2261       -2
============================================
- Hits          15492    15387     -105
- Misses         7784     7902     +118
+ Partials       1454     1441      -13

Impacted Files
Coverage Δ

...henyu/plugin/logging/kafka/LoggingKafkaPlugin.java
22.72% <0.00%> (-72.73%)
:arrow_down:

...ging/rocketmq/client/RocketMQLogCollectClient.java
10.00% <0.00%> (-62.00%)
:arrow_down:

...icsearch/client/ElasticSearchLogCollectClient.java
10.00% <0.00%> (-36.00%)
:arrow_down:

...ache/shenyu/plugin/grpc/cache/GrpcClientCache.java
75.00% <0.00%> (-25.00%)
:arrow_down:

...etmq/handler/LoggingRocketMQPluginDataHandler.java
47.91% <0.00%> (-22.92%)
:arrow_down:

...controller/ShenyuClientHttpRegistryController.java
77.77% <0.00%> (-22.23%)
:arrow_down:

...handler/LoggingElasticSearchPluginDataHandler.java
48.93% <0.00%> (-21.28%)
:arrow_down:

...n/logging/kafka/body/LoggingServerHttpRequest.java
0.00% <0.00%> (-20.00%)
:arrow_down:

...gin/logging/kafka/kafka/KafkaLogCollectClient.java
34.69% <0.00%> (-18.37%)
:arrow_down:

.../plugin/logging/kafka/entity/ShenyuRequestLog.java
86.20% <0.00%> (-13.80%)
:arrow_down:

... and 43 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f11c8b1...3132960. Read the comment docs.

authors: HaiqiQin codecov-commenter

repo: apache/incubator-shenyu


text:
recordDataTTL seem not work

Please answer these questions before submitting your issue.

Why do you submit this issue?
[ √] Question or discussion
[ ] Bug
[ ] Requirement
[ ] Feature or performance improvement

Question

What do you want to know?
i set 10 minute to the value of 'recordDataTTL',but i seem not work ,i wait for several hours ,the trace data was not deleted! even though the log print the tips。

i use the 6.0.0-alpha version。
can you answer my question?

Bug

Which version of SkyWalking, OS and JRE?

Which company or project?

What happen?
If possible, provide a way for reproducing the error. e.g. demo application, component version.

Requirement or improvement

Please describe about your requirements or improvement suggestions.

it is a bug or my configuration problem? @peng-yongsheng @wu-sheng

I have just addressed this issue. The time bucket format in segment table is not right.

Fixed

authors: wu-sheng yimison

repo: apache/incubator-skywalking


text:
Apply DashboardFilter to the dashboards in SliceModeView

CATEGORY
Choose one

[ ] Bug Fix
[ X] Enhancement (new features, refinement)
[ ] Refactor
[ ] Add tests
[ ] Build / Development Environment
[ ] Documentation

SUMMARY
Apply the DashboardFilter to the dashboards to only show dashboards that the user
has access to because they have access to at least one slice/chart on it.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

REVIEWERS

Codecov Report

Merging #7475 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #7475      +/-   ##
==========================================
+ Coverage   65.42%   65.42%   +<.01%
==========================================
  Files         435      435
  Lines       21322    21324       +2
  Branches     2347     2347
==========================================
+ Hits        13950    13952       +2
  Misses       7256     7256
  Partials      116      116

Impacted Files
Coverage Δ

superset/views/core.py
73.02% <100%> (+0.03%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6b8bda6...0645556. Read the comment docs.

can this be merged?

@dpgaspar we should think / talk about this, somehow I didn't think much and thought filters would apply across the board, but they didn't for relationships... It's probably really easy to be exposing more that we want in that context.
It would be great if we had a way to apply this in a single place, at the model level maybe? I think that's the way I'd do something like that in Django.

Yes, the filters on Views are applied to the view class only, and distinct for each related field. This enables a higher degree of configuration. Yes we can think of something that could be applied everywhere and probably Model is the best place, yet SQLAInterface is the one responsible for interfacing DB operations, so it would have to check for filters and apply them behind the scenes

authors: codecov-io dpgaspar mistercrunch rahul-rahul-sp

repo: apache/incubator-superset


text:
fix: Emit a warning message rather than an exception on query failure

SUMMARY
Exception messages are frequently bubbled up to error aggregators like Sentry, Airbrake, and the like from web applications. As queries can fail for a wide variety of reasons, most of which are beyond the ability of Superset to catch and fix before execution, I'd like to downgrade the messaging around analytical query failures to the warning level. This will keep the events in log streams but largely remove them from error aggregators.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

Trigger an error in an analytical query
Ensure that it is logged at the "warning" level rather than the "exception" level.

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

Reviewers
@craig-rueda @rusackas @john-bodley

cc @etr2460 for context.

An issue I see with this is the potential swallowing of useful information. Does it make sense to set exc_info=True here? In this way we would still get stack traces, and instrumentation tools would ignore as the threshold would be WARN

I think with exception you get the stack trace in the logs which can be very useful. We use this ~79 times in the codebase
20:51 $ git grep "\.exception(" | wc -l
      79

No strong opinions here one way or another, I can see both sides of the argument. I lean slightly on the side of agreeing with Will as these exceptions are generally going to be coming from the db engine and aren't really something we can resolve on the Superset side

I think Craig's suggestion is the correct balance. I'll set exec_info=True to capture the stacktraces. To @mistercrunch 's point, many of those .exception calls make a lot of sense, but this one is not actionable. We can tune the log levels without loosing the stacktraces.

Codecov Report

Merging #9811 into master will increase coverage by 1.73%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #9811      +/-   ##
==========================================
+ Coverage   64.31%   66.04%   +1.73%
==========================================
  Files         536      583      +47
  Lines       29109    30432    +1323
  Branches     2806     3162     +356
==========================================
+ Hits        18721    20099    +1378
+ Misses      10208    10152      -56
- Partials      180      181       +1

Flag
Coverage Δ

#cypress
?

#javascript
59.29% <ø> (?)

#python
70.79% <100.00%> (-0.24%)
:arrow_down:

Impacted Files
Coverage Δ

superset/connectors/sqla/models.py
88.41% <100.00%> (-0.16%)
:arrow_down:

superset-frontend/src/SqlLab/App.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/explore/App.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/SqlLab/index.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/dashboard/App.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/explore/index.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/dashboard/index.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/setup/setupColors.js
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/setup/setupPlugins.ts
0.00% <0.00%> (-100.00%)
:arrow_down:

superset-frontend/src/chart/ChartContainer.jsx
0.00% <0.00%> (-100.00%)
:arrow_down:

... and 291 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3cc5400...a315090. Read the comment docs.

authors: codecov-commenter craig-rueda etr2460 john-bodley mistercrunch willbarrett


text:
PanicInfo in sgx_tstd

error[E0061]: this function takes 3 arguments but 2 arguments were supplied
--> /root/sgx/sgx_tstd/src/panicking.rs:552:20
|
552 |     let mut info = PanicInfo::internal_constructor(message, location);
|                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -------  -------- supplied 2 arguments
|                    |
|                    expected 3 arguments
when i port a rust crate into rust sgx, why there is something with the sgx_tstd ?

The reason the compilation fails is that the version of rust you are using is incompatible with the version of rust required by sgx_tstd.
sgx_tstd requires rust-nightly-2021-11-01

@volcano0dr thank you very much and i have just solved this problem with your method

repo: apache/incubator-teaclave-sgx-sdk


text:
Add alternate cublaslt library name. CUDA 11.0 uses cublasLt.

I installed cuda 11.0 and cublaslt was not found. The file is named with an upper-case L now, cublasLt so I added that as an optional name and the library is correctly found again.

@tqchen

@jroesch @tkonolige @junrushao1994

Thanks @cbalint13 @junrushao1994 @rkimball

authors: rkimball tqchen

repo: apache/incubator-tvm


text:
[Android] weex 0.26 crashes on .toUpperCase/.toLocaleUpperCase with non-latin chars

Describe the bug
Application crashes when it uses .toUpperCase() or .toLocaleUpperCase() with non-latin chars
To Reproduce
http://dotwe.org/vue/e935dbd48af9a7fb5882774ed6e72219
Environment

OS: Android 9
Weex SDK Version: 0.26.0
Build from source: no

Example
<template>
    <div>
        {{ "本".toLocaleUpperCase() }}
    </div>
</template>

<script>
    module.exports = { }
</script>

Additional context
E/WeexCore: weex_js_connection.cpp:192,server died
E/weex: WXUtils parseFloat illegal value is null
    WXUtils parseFloat illegal value is null
E/WeexCore: script_side_in_multi_process.cpp:314,futex lock pi failed: No such process, 80006d68 27240 (0 0 0 0)
E/WXBridge: reportServerCrash instanceId:8 crashFile: /crash_dump.log
E/WXParams: getCrashFilePath:/data/user/0/com.test.weex.app/app_crash
D/jsengine: callReportCrashReloadPage crashFile:/data/user/0/com.test.weex.app/app_crash/crash_dump.log
E/WeexCore: wx_bridge.cpp:1138,class Name is com/taobao/weex/bridge/WXBridge
    WXBridge_jni.h:1198,Java_WXBridge_reset_clazz class Name is com/taobao/weex/bridge/WXBridge
D/ReportCrash:  commitJscCrashAlarmMonitor errMsg SIG: SIGSEGV (11), fault addr: 0x0
    maps:
    6058d09000-6058d0a000 r-xp 00000000 103:0d 394212                        /data/app/com.test.weex.app-mnba7FzF8VVvB6XwUBbxtA==/lib/arm64/libweexjsb.so
    6058d19000-6058d1a000 r--p 00000000 103:0d 394212                        /data/app/com.test.weex.app-mnba7FzF8VVvB6XwUBbxtA==/lib/arm64/libweexjsb.so
    6058d1a000-6058d1b000 rw-p 00000000 00:00 0
    751d000000-751d400000 rw-p 00000000 00:00 0                              [anon:libc_malloc]
    751d400000-751f400000 rw-p 00000000 00:00 0                              [anon:libc_malloc]
    751f400000-751fe00000 rw-p 00000000 00:00 0                              [anon:libc_malloc]
    751ff85000-751ff86000 ---p 00000000 00:00 0
    751ff86000-75201f2000 rwxp 00000000 00:00 0
    75201f2000-7521f7f000 ---p 00000000 00:00 0
    7521f7f000-7521f86000 rwxp 00000000 00:00 0
    7521f86000-7521f87000 ---p 00000000 00:00 0
    7521f87000-7521f88000 ---p 00000000 00:00 0                              [anon:thread stack guard]
    7521f88000-7522085000 rw-p 00000000 00:00 0
    7522085000-75222d1000 r-xp 00000000 fd:00 3650                           /system/lib64/libicui18n.so
    75222d1000-75222e8000 ---p 00000000 00:00 0
    75222e8000-75222fc000 r--p 0024c000 fd:00 3650                           /system/lib64/libicui18n.so
    75222fc000-75222fd000 rw-p 00260000 fd:00 3650                           /system/lib64/libicui18n.so
    7522333000-75224c4000 r-xp 00000000 fd:00 3472                           /system/lib64/libicuuc.so
    75224c4000-75224e2000 ---p 00000000 00:00 0
    75224e2000-75224f7000 r--p 0019b000 fd:00 3472                           /system/lib64/libicuuc.so
    75224f7000-75224f8000 rw-p 001b0000 fd:00 3472                           /system/lib64/libicuuc.so
    75224f8000-75224f9000 rw-p 00000000 00:00 0                              [anon:.bss]
    7522502000-7522503000 ---p 00000000 00:00 0                              [anon:thread stack guard]
    7522503000-7522600000 rw-p 00000000 00:00 0
    7522600000-7522800000 rw-s 00000000 00:05 2770996                        /dev/ashmem/WEEX_IPC_SERVER (deleted)
    7522800000-7522a00000 rw-s 00000000 00:05 2772067                        /dev/ashmem/WEEX_IPC_CLIENT (deleted)
    7522a00000-7522c00000 rw-p 00000000 00:00 0                              [anon:libc_malloc]
    7522c02000-7522c03000 ---p 00000000 00:00 0                              [anon:thread stack guard]
    7522c03000-7522d00000 rw-p 00000000 00:00 0
    7522d00000-7522d01000 ---p 00000000 00:00 0                              [anon:thread stack guard]
    7522d01000-7522dfe000 rw-p 00000000 00:00 0
    7522dfe000-7522dff000 ---p 00000000 00:00 0                              [anon:thread stack guard]
    7522dff000-7523800000 rw-p 00000000 00:00 0
    7523800000-7523c00000 rw-p 00000000 00:00 0                              [anon:libc_malloc]
    7523c40000-7523c57000 r-xp 00000000 fd:00 3328                           /system/lib64/libz.so
    7523c57000-7523c6f000 ---p 00000000 00:00 0
    7523c6f000-7523c70000 r--p 0001f000 fd:00 3328                           /system/lib64/libz.so
    7523c70000-7523c71000 rw-p 00020000 fd:00 3328                           /system/lib64/libz.so
    7523cb5000-7523d24000 r-xp 00000000 103:0d 393364                        /data/app/com.test.weex.app-mnba7FzF8VVvB6XwUBbxtA==/lib/arm64/libweexjss.so
    7523d24000-7523d33000 ---p 00000000 00:00 0
    7523d33000-7523d37000 r--p 0006e000 103:0d 393364                        /data/app/com.test.weex.app-mnba7FzF8VVvB6XwUBbxtA==/lib/arm64/libweexjss.so
    7523d37000-7523d38000 rw-p 00072000 103:0d 393364                        /data/app/com.test.weex.app-mnba7FzF8VVvB6XwUBbxtA==/lib/arm64/libweexjss.so
    7523d4a000-7523d60000 r-xp 00000000 fd:00 3746                           /system/lib64/liblog.so
    7523d60000-7523d79000 ---p 00000000 00:00 0
    7523d79000-7523d7a000 r--p 0001f000 fd:00 3746                           /system/lib64/liblog.so
    7523d7a000-7523d7b000 rw-p 00020000 fd:00 3746                           /system/lib64/liblog.so
    7523d99000
E/weex: [WXBridgeManager] invokeCallJSBatch: framework.js uninitialized!!  message:{ when=-32ms what=6 obj=8 target=android.os.Handler }
W/weex: app AvailMemory ---->>>1979
E/WXParams: setCrashFilePath: /data/user/0/com.test.weex.app/app_crash
E/weex: getLibLdPath is /data/app/com.test.weex.app-mnba7FzF8VVvB6XwUBbxtA==/lib/arm64:/data/app/com.test.weex.app-mnba7FzF8VVvB6XwUBbxtA==/base.apk!/lib/arm64-v8a
E/WeexCore: params_utils.cpp:130,initFromParam is running
E/weex: getUseSingleProcess is running true
E/WeexCore: params_utils.cpp:197,g_use_single_process is true
E/weex: getLibJscPath is running /data/app/com.test.weex.app-mnba7FzF8VVvB6XwUBbxtA==/lib/arm64
E/WeexCore: params_utils.cpp:232,g_jscSoPath is /data/app/com.test.weex.app-mnba7FzF8VVvB6XwUBbxtA==/lib/arm64
E/weex: getLibJssPath is running /data/app/com.test.weex.app-mnba7FzF8VVvB6XwUBbxtA==/lib/arm64/libweexjss.so
E/WeexCore: params_utils.cpp:245,g_jssSoPath is /data/app/com.test.weex.app-mnba7FzF8VVvB6XwUBbxtA==/lib/arm64/libweexjss.so
E/WXParams: getCrashFilePath:/data/user/0/com.test.weex.app/app_crash
E/WeexCore: params_utils.cpp:257,g_crashFilePath is /data/user/0/com.test.weex.app/app_crash
E/weex: getLibIcuPath is running /system/usr/icu/icudt60l.dat
E/WeexCore: params_utils.cpp:270,g_jssIcuPath is /system/usr/icu/icudt60l.dat
E/weex: getLibLdPath is running /data/app/com.test.weex.app-mnba7FzF8VVvB6XwUBbxtA==/lib/arm64:/data/app/com.test.weex.app-mnba7FzF8VVvB6XwUBbxtA==/base.apk!/lib/arm64-v8a
E/WeexCore: params_utils.cpp:283,lib_ld_path is /data/app/com.test.weex.app-mnba7FzF8VVvB6XwUBbxtA==/lib/arm64:/data/app/com.test.weex.app-mnba7FzF8VVvB6XwUBbxtA==/base.apk!/lib/arm64-v8a
E/WeexCore: script_bridge.h:168,xxx set_core_side is runing and 0x7af7fe7d60
    script_bridge_in_multi_process.cpp:979,ScriptBridgeInMultiProcess
E/WeexCore: multi_process_and_so_initializer.cpp:38,MultiProcessAndSoInitializer IS IN init
E/WeexCore: weex_js_connection.cpp:146,WeexJSConnection g_crashFileName: /data/user/0/com.test.weex.app/app_crash/crash_dump.log
E/WeexCore: weex_js_connection.cpp:266,startupPie :1
E/WeexCore: script_bridge_in_multi_process.cpp:991,ScriptBridgeInMultiProcess finish 0x7af750b0c0 0x0
W/weex: app AvailMemory ---->>>1980
E/WeexCore: render_page.cpp:72,[RenderPage] Delete RenderPage >>>> pageId: 8

I think we have fixed the problem on master branch.
Check it out, thanks.

confirmed
master has no this issue

repo: apache/incubator-weex


text:
Thrift Error occurred during processing of message. org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset [Bug]

Search before asking

[X] I searched in the issues and found nothing similar.

Version
I am running IOTDB 0.13.3 version on a centos 7 machine.
Describe the bug and provide the minimal reproduce step
I have inserted approx 7000 timeseriesinto the databases and trying to get all of them based on a filter time condition. I have increased themax_deduplicated_path_numvalue to 10000. When I am trying to get thetimeseriesI am getting the below error inlogs/log_all.log`
2023-01-25 15:08:21,520 [pool-19-IoTDB-RPC-Client-6] ERROR o.a.t.s.TThreadPoolServer$WorkerProcess:258 - Thrift Error occurred during processing of message. org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset         at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:196)         at org.apache.iotdb.rpc.TElasticFramedTransport.flush(TElasticFramedTransport.java:135)         at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:60)         at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38)         at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)         at java.lang.Thread.run(Thread.java:750) Caused by: java.net.SocketException: Connection reset         at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:115)         at java.net.SocketOutputStream.write(SocketOutputStream.java:155)         at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)         at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:194)         ... 7 common frames omitted 2023-01-25 15:08:21,520 [pool-19-IoTDB-RPC-Client-6] INFO  o.a.i.d.s.b.ServiceProvider:232 - Session-6 is closing
I am getting the below error in the client:
15:07:53.708 [WorkerThread0] DEBUG org.apache.iotdb.rpc.AutoResizingBuffer - org.apache.iotdb.rpc.AutoResizingBuffer@9718f40 expand from 65536 t$  98304, request: 65539 15:07:53.712 [WorkerThread0] DEBUG org.apache.iotdb.rpc.AutoResizingBuffer - org.apache.iotdb.rpc.AutoResizingBuffer@9718f40 expand from 98304 t$  147456, request: 98306 15:07:53.713 [WorkerThread0] DEBUG org.apache.iotdb.rpc.AutoResizingBuffer - org.apache.iotdb.rpc.AutoResizingBuffer@9718f40 expand from 147456 $ o 221184, request: 147478 15:07:53.715 [WorkerThread0] DEBUG org.apache.iotdb.rpc.AutoResizingBuffer - org.apache.iotdb.rpc.AutoResizingBuffer@9718f40 expand from 221184 $ o 331776, request: 221188 Wed Jan 25 15:08:24 IST 2023 - Fail to reconnect to server. Please check server status
I am using Java Native API to query the data.
Is there anything I am missing? Can anyone help me in fixing this?
What did you expect to see?
I expect the program to run without any error and gives me the result of the query
What did you see instead?
I got exception in the server side.
Anything else?
No response
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

https://issues.apache.org/jira/browse/THRIFT-5502

@HTHou Fine i read the issue you pointed out and they said that the changes took place back in V0.17 we are now in V0.19 and on my IoTDB cluster i have plenty of those Error.
To give an example:
2024-02-02 10:45:26,139 [pool-52-IoTDB-ClientRPC-Processor-494] ERROR o.a.t.s.TThreadPoolServer$WorkerProcess:258 - Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:178)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
        at org.apache.iotdb.rpc.TElasticFramedTransport.readFrame(TElasticFramedTransport.java:113)
        at org.apache.iotdb.rpc.TElasticFramedTransport.read(TElasticFramedTransport.java:108)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:463)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:361)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:244)
        at org.apache.iotdb.db.protocol.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:50)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: java.net.SocketException: Connection reset
        at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:313)
        at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:340)
        at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:789)
        at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1025)
        at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:255)
        at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:310)
        at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:382)
        at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:361)
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:176)
        ... 12 common frames omitted
2024-02-02 10:45:27,181 [pool-52-IoTDB-ClientRPC-Processor-494] ERROR o.a.t.s.TThreadPoolServer$WorkerProcess:258 - Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:178)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
        at org.apache.iotdb.rpc.TElasticFramedTransport.readFrame(TElasticFramedTransport.java:113)
        at org.apache.iotdb.rpc.TElasticFramedTransport.read(TElasticFramedTransport.java:108)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:463)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:361)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:244)
        at org.apache.iotdb.db.protocol.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:50)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: java.net.SocketException: Connection reset
        at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:313)
        at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:340)
        at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:789)
        at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1025)
        at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:255)
        at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:310)
        at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:382)
        at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:361)
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:176)
        ... 12 common frames omitted

Additional Info:

IoTDB Version: 1.3.0
IoTDB BuildInfo: fe787ca
I'm using Telegraf to send data to the Cluster paired with HAProxy
Cluster details: 3D 3C

好吧，我阅读了您指出的问题，他们说更改发生在 V0.17 中，我们现在在 V0.19 中，在我的 IoTDB 集群上，我有很多这样的错误。
举个例子：
2024-02-02 10:45:26,139 [pool-52-IoTDB-ClientRPC-Processor-494] ERROR o.a.t.s.TThreadPoolServer$WorkerProcess:258 - Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:178)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
        at org.apache.iotdb.rpc.TElasticFramedTransport.readFrame(TElasticFramedTransport.java:113)
        at org.apache.iotdb.rpc.TElasticFramedTransport.read(TElasticFramedTransport.java:108)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:463)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:361)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:244)
        at org.apache.iotdb.db.protocol.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:50)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: java.net.SocketException: Connection reset
        at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:313)
        at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:340)
        at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:789)
        at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1025)
        at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:255)
        at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:310)
        at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:382)
        at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:361)
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:176)
        ... 12 common frames omitted
2024-02-02 10:45:27,181 [pool-52-IoTDB-ClientRPC-Processor-494] ERROR o.a.t.s.TThreadPoolServer$WorkerProcess:258 - Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:178)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
        at org.apache.iotdb.rpc.TElasticFramedTransport.readFrame(TElasticFramedTransport.java:113)
        at org.apache.iotdb.rpc.TElasticFramedTransport.read(TElasticFramedTransport.java:108)
        at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
        at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:463)
        at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:361)
        at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:244)
        at org.apache.iotdb.db.protocol.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:50)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: java.net.SocketException: Connection reset
        at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:313)
        at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:340)
        at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:789)
        at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1025)
        at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:255)
        at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:310)
        at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:382)
        at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:361)
        at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:176)
        ... 12 common frames omitted

附加信息：

IoTDB版本：1.3.0
IoTDB 构建信息： fe787ca
我正在使用 Telegraf 将数据发送到与 HAProxy 配对的集群
集群详细信息：3D 3C

same question!Have you solved it？

@zhqingyong Hi,
I have solved this issue some times ago same as other open issues...... the community and the developer are not really helpfull.
Anyway to fix your issue you must set the backend check in HAproxis as the following lines:
for port 18080 you can set up health checking as basic
for port 6667 you MUST set up health checking as none
the error "org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset" means that the connection is closed brutally (The HA Proxy basic health check on port 6667 is the cause) changing the settings to none will fix your issue
I really hope the developers will be more active in the future. 🙏

authors: Bishnup1995 HTHou SeanGaluzzi zhqingyong

repo: apache/iotdb


text:
[IOTDB-4568] Added test api for partitions

[IOTDB-4568]Added test api for partitions at confignodes
related document：https://apache-iotdb.feishu.cn/docx/doxcnA4xpA6vrmKssH2GgbeHFhh
The inputs and outputs are defined there.
getRouting
Query a timePartition's corresponding dataRegionIds
getTimeSlotList
Query a seriesSlot's timeSlots between startTime and endTime
getSeriesSlotList
Query a storageGroup's seriesSlots, using TConsensusGroupType to detect the type of returned seriesSlots.

This PR has:

[x] been self-reviewed.

[ ] concurrent read
[ ] concurrent write
[ ] concurrent read and write

[x] added documentation for new or modified features or behaviors.
[ ] added Javadocs for most classes and all non-trivial methods.
[ ] added or updated version, license, or notice information
[x] added comments explaining the "why" and the intent of the code wherever would not be obvious
for an unfamiliar reader.
[ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold
for code coverage.
[x] added integration tests.
[ ] been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR

We also need to add the corresponding SQL for these interfaces

sponding S

Start doing this~

authors: CRZbulabula Caideyipi


text:
OAK-10081: (tests) create temp folders in .target to avoid leftovers …

…on some platforms

The changes in https://github.com/apache/jackrabbit-oak/pull/832 for this issue. But it's a good idea to create temp files in the TemporaryFolder itself which the test class does in a few tests.

authors: amit-jain fabriziofortino

repo: apache/jackrabbit-oak

license_source: bigquery


text:
JAMES-3461 - Fix Mailbox/changes do not take isSubscribe changes into account

resolve https://github.com/linagora/james-project/issues/4147

mpt/impl/imap-mailbox/core/src/main/resources/org/apache/james/imap/scripts/Subscribe.test
Adding:
C: b05 SUBSCRIBE whatever
S: b05 OK SUBSCRIBE completed.

# LIST All subscribed
C: b06 LSUB "" "*"
SUB {
S: \* LSUB \(\) \"\.\" \"whatever\"
S: \* LSUB \(\) \"\.\" \"subscribetest\"
S: \* LSUB \(\) \"\.\" \"subscribetest\.subfolder\"
S: \* LSUB \(\) \"\.\" \"subscribetest1\.subfolder1\"
}
S: b06 OK LSUB completed.

C: b08 UNSUBSCRIBE whatever
S: b08 OK UNSUBSCRIBE completed.

# LIST A subset of subscribed

From line 54... For instance. :green_apple:  on master...

Do we have tests for WebPush, SSE, and websocket when I change subscriptions?

You need to handle serialization / deserialisation for your new event:
11:56:05.627 [ERROR] o.a.j.e.EventDispatcher - error while dispatching event
java.lang.RuntimeException: no Scala conversion known
	at org.apache.james.event.json.ScalaConverter$.toScala(MailboxEventSerializer.scala:202)
	at org.apache.james.event.json.JsonSerialize.toJsonBytes(MailboxEventSerializer.scala:358)
	at org.apache.james.event.json.MailboxEventSerializer.toJsonBytes(MailboxEventSerializer.scala:368)
	at org.apache.james.events.EventDispatcher.serializeEvent(EventDispatcher.java:209)

Please rebase on https://github.com/apache/james-project/pull/1323

Still missing tests for WebPush, websockets, SSE ?

it's already
org.apache.james.jmap.rfc8621.contract.MailboxSetMethodContract#webSocketShouldPushNewMessageWhenChangeSubscriptionOfMailbox
org.apache.james.jmap.rfc8621.contract.MailboxSetMethodContract#sseShouldHasNewEventWhenChangeSubscribeOfMailbox
WebPush is same mechanism, so IMO the unittest in org.apache.james.jmap.change.MailboxChangeListenerTest is enough.
I pushed it in MailboxSetMethodContract because it more near with MailboxSet,

Oh ok, it's just I was expecting those tests in other files.

rebase master & squash & resolve conflict

Please rebase and have a look at the CI failure... @vttranlina

java.lang.RuntimeException:
java.rmi.server.ExportException: Port already in use: 9999; nested exception is:
	java.net.BindException: Address already in use (Bind failed)

I don't think this error relate to here
I run test on local and it pass

Hi! As you probably know, this PR introduces some breaking changes. Our academic tool, BreakBot, helps developers better assess these changes’ impact on client projects. BreakBot checked the top 100 most starred clients of james-project on GitHub and found out that only one is impacted by one of the breaking changes.
You can find the full BreakBot report in our fork repository breakbot-playground/james-project.
We hope this information about the impact of this PR's changes is valuable to you, and apologize otherwise. If you're willing to help, we would kindly ask for your help to fill in a 5-minutes survey about the report. Your feedback will help us improve the tool and help us in our research!

org.apache.james.jmap.rfc8621.memory.MemoryMailboxSetMethodTest.sseShouldHasNewEventWhenChangeSubscribeOfMailbox{GuiceJamesServer}

Assertion condition defined as a lambda expression in org.apache.james.jmap.rfc8621.contract.MailboxSetMethodContract that uses scala.collection.mutable.ListBuffer
Expected size: 1 but was: 0 in:
[] within 1 minutes  40 seconds.

Likely related. Please fix :)

authors: Arsnael chibenwa lmove vttranlina

repo: apache/james-project


text:
JamesMessageHook is not working

I'm trying to use only SMTP features directly from Java application.
I've defined:
SMTPProtocolHandlerChain chain = new SMTPProtocolHandlerChain(new NoopMetricFactory());
chain.addAll(0, handlers);
chain.wireExtensibleHandlers();
Protocol protocol = new SMTPProtocol(chain, smtpConfiguration);
server = new NettyServer.Factory(new HashedWheelTimer())
.protocol(protocol)
.build();
server.setListenAddresses(new InetSocketAddress(properties.getPort()));
server.setTimeout(properties.getTimeout());
server.bind();
one of the handler is JamesMessageQueue because i need the MimeMessage.
It seems the hook is never intercepted.

Hi @mwzero ,
Thanks for the report!
This problem had already been reported in https://issues.apache.org/jira/projects/JAMES/issues/JAMES-2902
We provided quickly some way of writing integration tests for SMTP hooks, and are looking for contributors to:

Write a similar integration tests for JamesMessageHook
Identify and fix the underlying issue.

Would you be interested to do that?
Also in your pull request, I don't see any code written by you. In the future when you encounter an issue with Apache James, please report it on the JIRA.
Best regards,
Benoit Tellier

@ieugen should we consider closing this PR, given that we opened a JIRA for it and it contains no code contribution relevant for the issue?

authors: chibenwa mwzero


text:
KAFKA-14438: Throw error when consumer configured with empty/whitespace-only group.id for AsyncKafkaConsumer

Per KIP-289 (Improve the default group id behavior in KafkaConsumer), the group.id configuration should no longer allow string values that—when trimmed of any whitespace—are zero length. By virtue of the existing AbstractConfig logic, string values are trimmed of whitespace on use.
Prior to this change, the constructors of both LegacyKafkaConsumer and AsyncKafkaConsumer checked the value, and if it was empty, would log a warning. The new AsyncKafkaConsumer will enforce this by throwing an InvalidGroupIdException if the value is empty instead of logging.
There is no attempt to prevent the creation of a ConsumerConfig that contains an invalid group.id value. It is only checked and enforced on use when creating a Consumer instance.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@philipnee—can you add the ctr and KIP-848 labels? Thanks

FYI: there are some failing integration tests that need to be updated to reflect this change.

Even if there was KIP for this change, we cannot put it into a minor release, but need to wait for 4.0.
This change would have been ok for 3.0, but this ship has sailed.

Even if there was KIP for this change, we cannot put it into a minor release, but need to wait for 4.0.

Is there any preparatory steps we need to take now (i.e. in the 3.7 release time frame) to make sure we are free to make this change in 4.0?

@mjsax would you be willing to review this PR? It's small and (hopefully!) straightforward. Thanks!

@cadonna, can you take a look at this change? Thanks!

@kirktrue My PR regarding KAFKA-15281 overlaps a lot with this. So I broke out the relevant parts for the group ID verification from my PR and opened https://github.com/apache/kafka/pull/14872.
That will avoid a lot of merge conflicts.

authors: cadonna kirktrue mjsax

repo: apache/kafka


text:
KAFKA-17058; Extend CoordinatorRuntime to support non-atomic writes

The group coordinator has (internal) write operations that could generate a large number of records (e.g. expiring offsets and groups). At the moment, those operations are limited by the maximum message size. If they hit it, they are basically stuck forever. This patch extends the CoordinatorRuntime to support non-atomic writes and it changes those internal operations to be non-atomic.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@jeffkbkim @dongnuo123 @jolshan Thanks for your comments. I addressed them.


text:
MINOR: Java 10 fixes so that the build passes

Upgrade EasyMock to 3.6 which adds support for Java 10
by upgrading to ASM 6.1.1.

Ensure that Jacoco is truly disabled for the core project.
This was the original intent, since it's in Scala, but it had not
been achieved. This is important because the Jacoco agent
fails when it tries to instrument the classes compiled by
scalac with Java 10.

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@rajinisivaram trivial change that makes it possible to test against Java 10. Once we merge this, I will switch the PR jobs to use Java 10 instead of Java 9.


text:
MINOR: Fix version in environment setup

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

I tested the fix by running Streams' upgrade tests. Before they were failing.

Call for review @abbccdda @ableegoldman @mjsax @vvcephei @bbejeck @guozhangwang

Thanks for the fix @cadonna LGTM.  Can you add the link for the passing system tests to the comments?

http://testing.confluent.io/confluent-kafka-branch-builder-system-test-results/?prefix=2019-06-26--001.1561552929--cadonna--fix_version_211_upgrade--2dabfde/

Waiting for a green build to complete for merging

retest this please

Merged #7004 into 2.2

Thanks for the fix @cadonna!

authors: bbejeck cadonna vvcephei


text:
[Minor] fix new consumer heartbeat reschedule bug

This commit fixes a minor issue introduced in the patch for KAFKA-2123. The schedule method requires the time the task should be executed, not a delay.

LGTM. Also did a quick check to make sure none of the other usages of the method have the same error.

LGTM.

This commit fixes a minor issue introduced in the patch for KAFKA-2123. The schedule method requires the time the task should be executed, not a delay.

This commit fixes a minor issue introduced in the patch for KAFKA-2123. The schedule method requires the time the task should be executed, not a delay.

This commit fixes a minor issue introduced in the patch for KAFKA-2123. The schedule method requires the time the task should be executed, not a delay.

This commit fixes a minor issue introduced in the patch for KAFKA-2123. The schedule method requires the time the task should be executed, not a delay.

authors: b-goyal ewencp guozhangwang hachikuji


text:
KAFKA-8403: Suppress needs a Materialized variant

This is a completely new implemetation from the previous PR.

KAFKA-8403: Suppress needs a Materialized variant
KIP-508: Make Suppression State Queriable

Here are some guidelines:

9bd279f ~ 37060b6: Refactor the TimeOrderedKeyValueStore into new interfaces, with separating Serdes from BytesStore. (see MeteredSuppressionBuffer)
798af5c ~ 0a684e0: Implement the KTable#suppress variant and make it queriable.
8f2e2c8  ~ 3387e27: Make SuppressionBuffer accessable by the users.
821014f: Implement a testing functionality.

Please have a look when you are free. Thanks in advance! :smile:
cc/ @mjsax @bbejeck @vvcephei @guozhangwang
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Retest this please.

Wow that's a really large PR :) cc @vvcephei for a quick look.

Hey @dongjinleekr , from the discussion on KIP-508, it seems this is not the approach we're going to take. Is that your impression as well?

@vvcephei Sorry for being late, I just got out from my last project; I will have a look.

@vvcephei Here is the fix. The implementation you saw was the draft before the discussion - Now it is updated, with rebasing onto the latest trunk. Don't hesitate to talk to me if you think we need additional tests like a queriable store.
Thanks again for your detailed review! :smile:

Hello. I updated the KIP and rebased the PR onto the latest trunk. Please have a look when you are free.
cc/ @vvcephei @mjsax

Rebased onto the latest trunk.
@vvcephei Could you have a look? It this implementation is okay, then I will open the vote.
cc/ @mjsax @bbejeck

@vvcephei Could you have a look? It this implementation is okay, then I will open the vote.

@dongjinleekr you don't have the implementation completed to call for a vote.  Maybe you can ping the discussion thread one last time to see if everything is settled and if its completed go ahead and call for a vote.

test this please

Rebased onto the latest trunk.

@vvcephei Here is the update. it now consists of 4 stages:

Separating instantiating/serde/metric related operations from InMemoryTimeOrderedKeyValueBuffer.
For InMemoryTimeOrderedKeyValueBuffer needs record collector and its changelog topic to implement flush operation, it has had a monolithic architecture dislike to the other StateStores. And this approach is also making it difficult to support iterator.
As a laying the groundwork, TimeOrderedKeyValueBufferBuilder, MeteredTimeOrderedKeyValueBuffer were separated into independent classes. InMemoryTimeOrderedKeyValueBuffer now concentrates on Bytes, byte[] operations without any serdes/metric operations.

By changing InMemoryTimeOrderedKeyValueBuffer#index's type into NavigatableMap, it now supports all, range operations.

KtableImpl takes queriableStoreName into account.

KtableImpl now makes queriable TimeOrderedKeyValueBuffer instance.

Please have a look when you are free.

@vvcephei Here is the update. it now consists of 4 stages:

Separating instantiating/serde/metric related operations from InMemoryTimeOrderedKeyValueBuffer.
For InMemoryTimeOrderedKeyValueBuffer needs record collector and its changelog topic to implement flush operation, it has had a monolithic architecture dislike to the other StateStores. And this approach is also making it difficult to support iterator.
As a laying the groundwork, TimeOrderedKeyValueBufferBuilder, MeteredTimeOrderedKeyValueBuffer were separated into independent classes. InMemoryTimeOrderedKeyValueBuffer now concentrates on Bytes, byte[] operations without any serdes/metric operations.

By changing InMemoryTimeOrderedKeyValueBuffer#index's type into NavigatableMap, it now supports all, range operations.

KtableImpl takes queriableStoreName into account.

KtableImpl now makes queriable TimeOrderedKeyValueBuffer instance.

Please have a look when you are free.

authors: bbejeck dongjinleekr guozhangwang vvcephei


text:
KAFKA-9748: Extend Streams integration tests for EOS beta

The original PR was only considering EosIntegrationTest. However, there are actually a few more integration test with eos-alpha enabled that we missed to update.
Call for review @abbccdda @guozhangwang

Both Jenkins runs passed. Magic!

Build failed with
23:01:46 [ant:checkstyle] [ERROR] /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk11-scala2.13@2/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerCoordinator.java:332:5: NPath Complexity is 588 (max allowed is 500). [NPathComplexity]

Seems to be fixed via #8447
Can retest after more reviews.

Retest this please.

Java 8 passed.
Java 11: kafka.api.ConsumerBounceTest.testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
Retest this please.

Java 8 passed.
Java 11: kafka.api.SaslGssapiSslEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
Different test as before. Also run the tests locally multiple times without issues. Merging this.


text:
MINOR: Add comments to constrainedAssign and generalAssign method

Recently, I tried to read the codes to understand what the constrainedAssign and generalAssign method is doing, but it is so difficult and suffering due to the complexity of the algorithm. And then I traced back to the JIRA ticket and KIP to get much more information about them. So, I think we should put at least the algorithm goal and main steps in the code comments, to let other developers better understand them and better do trouble shooting if any.
reference for constrainedAssign algorithm:
https://issues.apache.org/jira/browse/KAFKA-9987?focusedCommentId=17106832&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17106832
reference for generalAssign algorithm:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-54+-+Sticky+Partition+Assignment+Strategy
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@ableegoldman @vahidhashemian , please help review this PR to add comments to the assign methods. Thanks.

retest this please

retest this please

retest this please

Thanks @ableegoldman . I've updated based on your suggestion. Thanks.

retest this please

retest this please

Only flaky test failures.

authors: abbccdda chia7712 showuon


text:
[KARAF-7068] add instance:package command to compress an instance into zip/tar.gz

Not sure if this one was up for grabs since @jbonofre is already assigned on Jira?

@awrb good one ! Let me review it, but it's exactly what I had in mind. Thanks !

authors: awrb jbonofre

repo: apache/karaf


text:
KNOX-3075 - Handling unlimited token expiration in JDBC TSS properly

What changes were proposed in this pull request?
As described in KNOX-3075, when the token TTL was set to -1 (which allows creating never-expiring tokens), JDBC TSS threw an UnknownTokenException. This is wrong. It should have returned the configured -1 value.
How was this patch tested?
I updated existing JUnit tests to handle the path when fetching tokens for a user. I also added 2 new test cases to ensure this issue never occurs again.

Hi @moresandeep, @pzampino - Can any of you please check this out for me? Thanks!

repo: apache/knox


text:
fix the dead loop when slave blocked by election campaign

controller resume and suspend cluster checker
use gopkg.in/yaml.v3

@greatsharp You can run go mod tidy (if you didn't) after upgrading the dependency.

@greatsharp You can run go mod tidy (if you didn't) after upgrading the dependency.

done, thanks for your remention

test the case  of delete the lease from etcd side.
test the case of delete the leader key from etcd side.
test the case of shutdown etcd.

@greatsharp Is this PR ready to review?

authors: git-hulk greatsharp torwig

repo: apache/kvrocks-controller


text:
Added Import Snapshot and Describe Import Snapshot to EC2 compute driver

Added Import Snapshot and Describe Import Snapshot to EC2 compute driver
Description
On Importing Snapshot in aws, it returns the import snapshot task which needs to be monitored by describe import snapshot task until it is ready. Hence, added a wait block method. Finally, it returns a volume snapshot.
Status

done, ready for review

Checklist (tick everything that applies)

[x] Code linting (required, can be done after the PR checks)
[x] Documentation
[x] Tests
[ ] ICLA (required for bigger changes)

perfect 👍

authors: nirzari tonybaloney

repo: apache/libcloud


text:
Reporting functions for dimension data driver

Reporting functions for dimension data driver

[ ] Code linting (required, can be done after the PR checks)
[ ] Documentation
[x] Tests
[x] ICLA (required for bigger changes)

well that broke everything :fire:

done. merging


text:
[Question] exchangis execute write from hive to MySQL without obvious error, but fail

Before asking

[X] I have read the How to ask for help (我已阅读如何寻求帮助指引文档).

Your environment

Linkis version used: 1.1.1
Environment name and version:

hdp-3.1.4
hive-3.1.0
spark-2.3.0
scala-2.11.8
jdk 1.8.0_121
sqoop1.4.7
Exchangis1.0.0
-sqoop.version change from 1.4.6 to 1.4.7 before compilation
-hadoop.version change from 2.7.2 to 3.1.1 before compilation
-change class TransformExchangisJob.java [170 line]，Solve the problem of null. Isempty() error
--from：if(!partitionValue.get("ds").isEmpty())
--to：if(StringUtils.isNotBlank(partitionValue.get("ds")))
but bml version is 1.4.6. (..linkis/lib/linkis-engineconn-plugins/sqoop/dist/v1.4.6)

Describe your questions

Q1.  exchangis execute failed. The DSS platform reported unknown error information during execution, while the... / links / logs log reported no error

Eureka service list

Some logs info or acctch file

dss web:
2022-09-06 16:40:58.040 INFO Program is substituting variables for you
2022-09-06 16:40:58.040 INFO Variables substitution ended successfully
2022-09-06 16:40:58.040 WARN The code you submit will not be limited by the limit
Job with jobId : nodeexecution_hadoop_appconn_7 and execID : nodeexecution_hadoop_appconn_7 submitted
2022-09-06 16:40:58.040 INFO You have submitted a new job, script code (after variable substitution) is
************************************SCRIPT CODE************************************
{"id":5793.0,"refProjectId":1.49787087103597389E18}
************************************SCRIPT CODE************************************
2022-09-06 16:40:58.040 INFO Your job is accepted,  jobID is nodeexecution_hadoop_appconn_7 and jobReqId is 155 in ServiceInstance(linkis-cg-entrance, c0:9604). Please wait it to be scheduled
job is scheduled.
2022-09-06 16:40:58.040 INFO Your job is Scheduled. Please wait it to run.
Your job is being scheduled by orchestrator.
2022-09-06 16:40:58.040 INFO job is running.
2022-09-06 16:40:58.040 INFO Your job is Running now. Please wait it to complete.
2022-09-06 16:40:58.040 INFO Job with jobGroupId : 155 and subJobId : 155 was submitted to Orchestrator.
2022-09-06 16:40:58.040 INFO Background is starting a new engine for you,execId astJob_7_codeExec_7 mark id is mark_7, it may take several seconds, please wait
2022-09-06 16:40:58.040 INFO EngineConn local log path: ServiceInstance(linkis-cg-engineconn, c0:32827) /appcom/tmp/hadoop/20220906/appconn/a7ee6eab-a544-40c8-965b-c26ed59feb73/logs
try to execute exchangis.sqoop node, ready to request to http://192.168.70.95:18089/api/rest_j/v1/dss/exchangis/main/appJob/execute/5793
submitted to Exchangis with execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Inited
task8  received asyncReturnExecuteResponse
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Inited
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Scheduled
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Scheduled
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Scheduled
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Scheduled
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Scheduled
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Scheduled
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Scheduled
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Scheduled
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Scheduled
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Scheduled
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Scheduled
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Scheduled
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Scheduled
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Scheduled
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Running
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Running
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Running
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Running
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Failed
ExchangisJob(execId: 8ec0f89a-24dc-462e-a493-5ba02e8b718d) is in state Failed
2022-09-06 16:41:18.041 ERROR Job with execId-nodeexecution_hadoop_appconn_7 + subJobId : 155  execute failed,21304, Task(codeExec_7) status not succeed,is Failed

2022-09-06 16:41:18.041 INFO job is completed.
2022-09-06 16:41:18.041 INFO Task creation time(任务创建时间): 2022-09-06 16:40:58, Task scheduling time(任务调度时间): 2022-09-06 16:40:58, Task start time(任务开始时间): 2022-09-06 16:40:58, Mission end time(任务结束时间): 2022-09-06 16:41:18
2022-09-06 16:41:18.041 INFO Your mission(您的任务) 155 The total time spent is(总耗时时间为): 20.1 秒
2022-09-06 16:41:18.041 INFO Sorry. Your job completed with a status Failed. You can view logs for the reason.

log file:

stderr.log, stdout.log is from path: /appcom/tmp/hadoop/202209xx/appconn/xx/logs
stderr.log
stdout.log
exchangis-server.log
linkis-cg-entrance.log

This question has not been answered for a long time, I will close it first, if necessary, you can turn it back on.

authors: casionone sunriseXDM

repo: apache/linkis


text:
Generate reference documentation and XSD from metadata

Log4j plugins follow a slightly modified JavaBean model. As a first step of #1954 we should:

[x] research the available metadata formats that could be employed to document Log4j plugins,
[ ] choose the most appropriate metadata format based on: maintenance state, available tools and features. An in-house solution should have lowest priority (cf. NIH),
[ ] use an example metadata file to generate an XSD schema,
[ ] use an example metadata file to generate reference documentation pages.

A loose list of metadata formats we might consider:

an XML Schema itself,
a Plexus component descriptor,
a Maven plugin descriptor, which is used by the Maven Plugin plugin to generate documentation,
a Modello model that can be converted in many other formats.

A loose list of metadata formats we might consider:

an XML Schema itself,
a Plexus component descriptor,
a Maven plugin descriptor, which is used by the Maven Plugin plugin to generate documentation,
a Modello model that can be converted in many other formats.

@ppkarwasz, I completely support the research phase you described. Though I will also appreciate some non-Maven-native solutions (besides an XSD) too.

@ppkarwasz, I completely support the research phase you described. Though I will also appreciate some non-Maven-native solutions (besides an XSD) too.

Sure thing, my research is not complete. Maven-related formats were just the easiest to find.

Neat, this is an idea I've had in the past. It would be great if we had proper XSDs that made it easier to edit a log4j configuration file. Might be useful to see if this schema can be represented as a JsonSchema, too, so it can be used for JSON and YAML configuration files.
One potential source of inspiration here would be whatever Spring Boot does for making your various @ConfigurationProperties classes autocomplete when modifying Spring properties configurations.
I should note that various old plugins still use strings for various plugin attributes that should be a different data type, but the initial plugin system only supported string values for attributes as it mimicked the DOM API here. Thus, if you use the type info of a plugin attribute, note that a string type is not necessarily authoritative.

The main plan here is to have a documentation like Maven plugins (e.g. Maven Compiler Plugin). These are generated without a single line of text (besides the *.java files). I would like something like this for each plugin:

Parameter
Type
Description

direct
boolean
Specifies whether to use System.out/err or the file descriptors

layout
Layout
The layout to use. Common layouts: list of links. (all layouts link)

The XSD will be a byproduct of this process, but the metadata must be a little bit larger: in February-March we plan to write an online configuration builder/validation tool. This tool might also spit up a list of Maven dependencies required by the configuration.

Oh that all sounds fantastic!

A second requirement from the metadata format is the availability of tools to generate various formats of documentation:

as far as I can tell OSGi Declarative Services do not have such tools,
the was an Eclipse EMF documentation generator, but it does not seem available at the moment,
Tomcat Configuration Reference, which was the main inspiration for this task is actually generated manually (cf. git repository).

This leaves Modello that has an impressive number of generators and the custom Maven Plugin descriptor as contenders. The documentation of Maven Plugin descriptor is actually generated by Modello.

An experiment performed using Modello (cf. the Log4j Documentation Reference using Modello gist), shows that even if we use a Modello descriptor to describe Log4j plugins, we would need to write the documentation logic ourselves (even for "just" an XSD).
That is why I believe:

we should still use Modello, but to generate our own descriptor that would not approximate our components as Java Beans, but use their natural representation (plugin + plugin builder with a separate hierarchy for each of them).

How would you compare these solutions to simply rendering an XSD/JSON and mapping it to a Map<String, Object> passed to a [FreeMarker?] template where we can generate whatever plain-text (AsciiDoc, etc.) we want? I would rather have full control over model-to-AsciiDoc conversion, since we can tailor the emitted output as we see fit.

log4j-plugins parses Log4j plugins and generates some sort of DOM model already – correct me if I am wrong. How would you compare us generating an XSD out of this DOM model versus decorating plugins with Modello and using Modello to generate an XSD?
When Modello is employed, do we need to use both Log4j and Modello annotations for plugins?

Given XSD is an XML too, I would favor simply passing XSD model to FreeMarker (through a simple Map<String,Object>?) and deal with representation there.

log4j-plugins parses Log4j plugins and generates some sort of DOM model already – correct me if I am wrong. How would you compare us generating an XSD out of this DOM model versus decorating plugins with Modello and using Modello to generate an XSD?

I thought to use Modello only as an implementation detail. log4j-plugins will generate some XML (right now it generates Java code). We'll use Modello to parse these XML files and generate:

A common XSD for all the Log4j (and in the future third-party) modules,
A set of documentation pages.

2. When Modello is employed, do we need to use both Log4j and Modello annotations for plugins?

No, this will be transparent for the user.

Given XSD is an XML too, I would favor simply passing XSD model to FreeMarker (through a simple Map<String,Object>?) and deal with representation there.

The XSD schema will cover only part of our plugins (those in the "Core" namespace).

This translates to
plugins --(Log4j)--> XML --(Modello)--> XSD

How would you compare this to the following?
plugins --(Log4j)--> XSD

Modello will only be used to generate code for XML serialization/deserialization and the model itself. Once that is done it will be:
Source code --(`log4j-plugins-processor`)--> XML --(`log4j-docgen`)--> XSD, Asciidoc et al.

With the submission of the PRs above, I consider this subtask complete.

authors: jvz ppkarwasz vy


text:
[MENFORCER-314] - Warn if EnforcerRuleException has no message

Following this checklist to help us incorporate your
contribution quickly and easily:

[x] Make sure there is a JIRA issue filed
for the change (usually before you start working on it).  Trivial changes like typos do not
require a JIRA issue.  Your pull request should address just this issue, without
pulling in other changes.
[x] Each commit in the pull request should have a meaningful subject line and body.
[x] Format the pull request title like [MENFORCER-XXX] - Fixes bug in ApproximateQuantiles,
where you replace MENFORCER-XXX with the appropriate JIRA issue. Best practice
is to use the JIRA issue title in the pull request title and in the first line of the
commit message.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Run mvn clean verify to make sure basic checks pass. A more thorough check will
be performed on your pull request automatically.

If your pull request is about ~20 lines of code you don't need to sign an
Individual Contributor License Agreement if you are unsure
please ask on the developers list.
To make clear that you license your contribution under
the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.

[x] I hereby declare this contribution to be licenced under the Apache License Version 2.0, January 2004

[ ] In any other case, please file an Apache Individual Contributor License Agreement.

I pushed an alternative approach because I couldn't really decide which one is better: #44
Please merge whichever PR you like best.

I jumped ahead and extended TestEnforceMojo. IMHO this should be sufficient for such a small change.
Please note that I won't be able to put much more effort into this entire issue as I am busy elsewhere.
PS: Yes, I did see that there is a custom MockEnforcerRule but I saw no real sense in extending this with more logic as mockito is just way more flexible at such tasks (and you need to write less code).

@famod great work.
Will commit soon

waiting for CI
https://builds.apache.org/job/maven-box/job/maven-enforcer/job/MENFORCER-314/

You are welcome and thank you, too!

authors: eolivelli famod

repo: apache/maven-enforcer


text:
Mskins 191 dynamic library retrieval

I have modified this branch so the repository no longer contains the JS files used, instead we make use of the com.github.eirslett:frontrend-plugin.
This plug-in will pull down a copy of Node.JS into the build directory, this in turn allows us to call 'npm' and install bootstrap and anchor-js via NPM (since NPM Regsitry is how front end developers "release" libraries). This is performed in the 'generate-resources' phase.
I had to bump the bootstrap version from 2.8.3 to 3.1.1, the classes uses dont appear to have changed, the big change was the removal of jQuery so this was removed from the project.
I then added several executions to the maven-resources-plugin, these run in the 'process-resources' (phase after 'generate-resources'). The com.samaxes.maven:minify-maven-plugin plugin expects all the CSS/JS files to be in the same root directory, they are obviously now in different directories (e.g. src/main/resources and target/node_modules). So we use maven-resources-plugin to move them into a single directory (e.g. target/interim-minify). This lets the minify plug-in create the same build artefacts as before.

I am rejecting this PR for the following reasons:

npm/nodejs bring a huge dependency chain for a simple problem
we don't want to rely on external tools
we rarely change those dependencies

I see very little benefit in doing this for us.

Some comments because I think the issue is legitimate and we are very 'back' on the frontend deliveries for now.
Maybe using download maven plugin or equivalent to grab the resources from a CDN can be sufficient for now but I think we should consider:

NOT packagng the resources in the jar, in particular because we don't own them and we should and will need to upgrade for various reasons (starting by the fact these versions are no more supported and have vulnerabilities)
use node to handle the minification instead of a plugin which does not handle some optimizations of modern web (using browserify at least) sounds like a very good choice for the final delivery so can be worth keeping frontend plugin
(optional) enable to use CDN in the theme which would be probably the best option for most people anyway

The need for an upgrade does not relate to using npm or packaging. Maven sites are supposed to be self-contained because they can be archived, see site:jar. Using CDNs would completely break this. What is the real world benefit of this smart-ass minification? Does it really matter for this simple skin?

I love the idea of making updates of the 3 used js libraries easier, even if the hard work when upgrading is to check that the template still works as expected: at least, it eases having a quick look
like Michael, I feel that introducing npm for that looks too heavy
Could an approach like webjars do the same job? see https://github.com/webjars/bootstrap/blob/master/pom.xml

notice: given updates to the skin requires to check the rendering, I had a look at how easy currently
and I found:

it's easy to see the result of many edge cases: https://maven.apache.org/skins/maven-fluido-skin/ITs.html
but it's not so easy to see result for the sample page, given it uses previous skin version: https://maven.apache.org/skins/maven-fluido-skin/sample.html

perhaps we should inject the sample page content into the 3 main layouts tests https://maven.apache.org/skins/maven-fluido-skin/sidebar/index.html
we never had much contribution to skins, this is time to learn together how to improve that aspect

@hboutemy exploding webjars and copying resources would work but would still rely on outdated plugins for the resources build, this is where using nodejs ecosystem would be a big plus for the theme until we go with CDN as mentionned which would make the game pretty different then. So, IMHO, to summarize it is not only about fetching but more about having a proper front build where node helps.

Main challenge is how to cut template, adjust macros.
including 2 css and .js files doesn't have to be problem here.

I don't want this skin to be rocket science,  but this makes it.

Not sure, using node today is not rocket science (at least no more than using velocity to create a template) but overall it would benefit end users (either using CDN or node to build the bundle). Can avoid the build postprocessing to rewrite the resources in site deployment which care - but I can also hear it is less and less used outside apache so maybe not worth it if we don't care for ourselves.

I am interested in this skin, because maven allows me to incorporate documentation and analysis into a site. The problem is the current site style has dated.
Historically full stack development has JS/HTML/CSS stored in an appropriate resources directory of a java project. This made developing frontends painful.
Node.js/NPM appeared 10 years ago and you could build a mini webserver fairly easy with express and iterate there was much joy. We hit the problem I am addressing in this PR.
The format NPM releases a project is different to Maven.
There are several common approaches to integrate the UI with the backend

write a release time script which calls Maven and generates a JAR when the NPM project releases.
Use the latest java wrapper for JS
Write the front end so its hosted statically in its own webserver and the UI calls to the backend
Use the front end plugin to pull down the dependecies

The problem with 1 is what we see here and requires bootstrap to make a change to support you.
The problem with 2 (e.g. webjars) is Web Development is happy in its ecosystem. So adoption always fails (Personally I loved the maven extension that let me package js).
Approach 3 isn't valid here, fundamentally what this skin offers are some resources and a template file to apache velocity.
The last approach is what I have chosen, we use the frontend plugin to retrieve node/npm which are stored in a temporary directory. We use NPM to retrieve the releases and then the minify plugin to generate the JS artifacts.
The result is no additional dependencies for users of the skin, there should be zero change from their perspective.
I saw the comment on CDN, I have always worked in defense companies. Defense company infosec departments are not reasonable (and iften non technical) meaning this sort of stuff is developed in effectively air gapped environments and I have spent way to much of my life designing and pushing for mirrors of real world infrastructure. One of the reasons I love Maven is as a toolit makes redirecting to local mirrors really easy.
As for why this change?
I have a bootstrap 5 skin supporting a fixed navbar top but its a complete rewrite. Bootstrap has its examples and then there are HTML layouts

authors: hboutemy michael-o rmannibucau slachiewicz stevecrox

repo: apache/maven-fluido-skin


text:
[MRESOLVER-130] Move GlobalSyncContextFactory to a separate module

@elharo Please check documentation for typos and consistency.

@elharo Regarding comments to TrackingFileManager. I have intentionally not touched the class as a whole because I have copied it from maven-resolver-impl. I needed to remove the synchronization only because I consider it to be wrong for several reasons. A JIRA issue will follow. So let's also keep this class as-is (out of scope).

repo: apache/maven-resolver


text:
METRON-2290: [UI] Delaying first auto polling request on app start

Contributor Comments
The state of auto polling on Alerts UI is persisted in the browser's localStorage. Next time the User visiting the UI the auto polling state (on/off, refresh interval) will be automatically restored.
Unfortunately, there is a race condition between the restoration of

persisted auto polling state,
persisted page size and
default time range filter.

To fix this in the simplest possible way I delayed the start of the auto polling when it's automatically starting on app initialization.
With this the application able to restore the persisted page size and set the default time range properly before the auto polling start using these values.
While I was investigating this issue I realized that the restoration of page size itself was also broken, so I fixed that as well (single line of code change).
Testing
If you turn on auto polling and refreshing the browser auto polling should stay turned on.
The queries fired by auto polling should contain page size, from fields in the request and timer range filter in the query string.
Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
For all changes:

[x] Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
[x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

[x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?

[x] Have you included steps or a guide to how the change may be verified and tested manually?

[x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh

[x] Have you written or updated unit tests and or integration tests to verify your changes?

[x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

[x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:
cd site-book
mvn site

[x] Have you ensured that any documentation diagrams have been updated, along with their source files, using draw.io? See Metron Development Guidelines for instructions.

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

+1 pending unfocusing the describe in the unit tests. Tested with full dev. Thanks for fixing this, @tiborm!

authors: sardell tiborm

repo: apache/metron