text:
[Bug] [FTP] FTP SOURCE Unable to return file list under FTP path in the code & FTP SINK NOT CONFIG ROOT PATH

Search before asking

[X] I had searched in the issues and found no similar issues.

What happened
FTP SOURCE, PATH IS NOT FOUND
SeaTunnel Version
dev version
SeaTunnel Config
{
  "env": {
    "execution.parallelism": 2,
    //仅支持BATCH 批处理操作
    "job.mode": "BATCH",
    "checkpoint.interval": 10000
  },

  "source": [
    {
       "plugin_name": "FtpFile",
      // 这个作为流程中的别名
      "result_table_name": "tableA",
      "host":"172.30.13.179",
      "port":"2121",
      "user":"firewolf",
      "password":"P@ssw0rd01!",
      // 数据格式类型
      "file_format_type":"csv",
      // 字段分隔符
      "delimiter":",",
      "path":"/data.csv",
      "schema":{
                "fields":{
                    "id":"int",
                    "job_id":"int",
                  "node_id":"string",
                  "name":"string"
                }
            }
    }
  ],

"sink": [
    {
      "plugin_name": "FtpFile",
      // 写入格式json
      "file_format_type": "csv",
      "host":"172.30.13.179",
      "port":"2121",
      "user":"firewolf",
      "password":"P@ssw0rd01!",
      // 写入文件路径地址
      "path":"/",
      // 列分隔符
      "field_delimiter":",",
      // 行分隔符
      "row_delimiter":"\n",
      // 指定源头是transform中的虚拟表
      "source_table_name": "tableC"
    }
  ]
}

Running Command
curl --location 'http://172.30.13.177:5801/hazelcast/rest/maps/submit-job' \
--header 'Content-Type: application/json' \
--data-raw '{
  "env": {
    "execution.parallelism": 2,
    //仅支持BATCH 批处理操作
    "job.mode": "BATCH",
    "checkpoint.interval": 10000
  },

  "source": [
    {
       "plugin_name": "FtpFile",
      // 这个作为流程中的别名
      "result_table_name": "tableA",
      "host":"172.30.13.179",
      "port":"2121",
      "user":"firewolf",
      "password":"P@ssw0rd01!",
      // 数据格式类型
      "file_format_type":"csv",
      // 字段分隔符
      "delimiter":",",
      "path":"/data.csv",
      "schema":{
                "fields":{
                    "id":"int",
                    "job_id":"int",
                  "node_id":"string",
                  "name":"string"
                }
            }
    }
  ],

"sink": [
    {
      "plugin_name": "FtpFile",
      // 写入格式json
      "file_format_type": "csv",
      "host":"172.30.13.179",
      "port":"2121",
      "user":"firewolf",
      "password":"P@ssw0rd01!",
      // 写入文件路径地址
      "path":"/",
      // 列分隔符
      "field_delimiter":",",
      // 行分隔符
      "row_delimiter":"\n",
      // 指定源头是transform中的虚拟表
      "source_table_name": "tableC"
    }
  ]
}'

Error Exception
FTP SOURCE Unable to return file list under FTP path in the code:
org.apache.seatunnel.connectors.seatunnel.file.exception.FileConnectorException: ErrorCode:[FILE-03], ErrorDescription:[Get file list failed] - Get file list from this path [/data.csv] failed
        at org.apache.seatunnel.connectors.seatunnel.file.ftp.source.FtpFileSource.prepare(FtpFileSource.java:88) ~[?:?]
        at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSource(JobConfigParser.java:84) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSource(MultipleTableJobConfigParser.java:309) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
        at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:182) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
        at org.apache.seatunnel.engine.server.rest.RestJobExecutionEnvironment.getLogicalDag(RestJobExecutionEnvironment.java:76) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
        at org.apache.seatunnel.engine.server.rest.RestJobExecutionEnvironment.build(RestJobExecutionEnvironment.java:99) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
        at org.apache.seatunnel.engine.server.rest.RestHttpPostCommandProcessor.handleSubmitJob(RestHttpPostCommandProcessor.java:138) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
        at org.apache.seatunnel.engine.server.rest.RestHttpPostCommandProcessor.handle(RestHttpPostCommandProcessor.java:77) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
        at org.apache.seatunnel.engine.server.rest.RestHttpPostCommandProcessor.handle(RestHttpPostCommandProcessor.java:55) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
        at com.hazelcast.internal.ascii.TextCommandServiceImpl$CommandExecutor.run(TextCommandServiceImpl.java:402) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
        at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_312]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_312]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
        at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
Caused by: java.io.FileNotFoundException: File /data.csv does not exist.
        at org.apache.seatunnel.connectors.seatunnel.file.ftp.system.SeaTunnelFTPFileSystem.getFileStatus(SeaTunnelFTPFileSystem.java:459) ~[?:?]
        at org.apache.seatunnel.connectors.seatunnel.file.ftp.system.SeaTunnelFTPFileSystem.listStatus(SeaTunnelFTPFileSystem.java:406) ~[?:?]
        at org.apache.seatunnel.connectors.seatunnel.file.ftp.system.SeaTunnelFTPFileSystem.listStatus(SeaTunnelFTPFileSystem.java:391) ~[?:?]
        at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.listStatus(HadoopFileSystemProxy.java:175) ~[?:?]
        at org.apache.seatunnel.connectors.seatunnel.file.source.reader.AbstractReadStrategy.getFileNamesByPath(AbstractReadStrategy.java:93) ~[?:?]
        at org.apache.seatunnel.connectors.seatunnel.file.ftp.source.FtpFileSource.prepare(FtpFileSource.java:85) ~[?:?]
        ... 15 more

FTP SINK NOT ROOT PATH:

java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ftp://T_792686344391360513_528d0e3d11_0_1_0.csvSaaS(å¤%C2%9Aç§%C2%9Fæ%C2%88·)å%C2%8C%C2%96è®¾è®¡ä¸%C2%8Eå®%C2%9Eç%C2%8E°.docxSaaS(å¤%C2%9Aç§%C2%9Fæ%C2%88·)å%C2%8C%C2%96è®¾è®¡ä¸%C2%8Eå®%C2%9Eç%C2%8E°.docx
	at org.apache.hadoop.fs.Path.initialize(Path.java:259) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.4-SNAPSHOT]
	at org.apache.hadoop.fs.Path.<init>(Path.java:157) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.4-SNAPSHOT]
	at org.apache.hadoop.fs.Path.<init>(Path.java:125) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.4-SNAPSHOT]
	at org.apache.seatunnel.connectors.seatunnel.file.ftp.system.SeaTunnelFTPFileSystem.getFileStatus(SeaTunnelFTPFileSystem.java:510) ~[?:?]
	at org.apache.seatunnel.connectors.seatunnel.file.ftp.system.SeaTunnelFTPFileSystem.listStatus(SeaTunnelFTPFileSystem.java:437) ~[?:?]
	at org.apache.seatunnel.connectors.seatunnel.file.ftp.system.SeaTunnelFTPFileSystem.delete(SeaTunnelFTPFileSystem.java:370) ~[?:?]
	at org.apache.seatunnel.connectors.seatunnel.file.ftp.system.SeaTunnelFTPFileSystem.delete(SeaTunnelFTPFileSystem.java:376) ~[?:?]
	at org.apache.seatunnel.connectors.seatunnel.file.ftp.system.SeaTunnelFTPFileSystem.delete(SeaTunnelFTPFileSystem.java:345) ~[?:?]
	at org.apache.seatunnel.connectors.seatunnel.file.hadoop.HadoopFileSystemProxy.renameFile(HadoopFileSystemProxy.java:122) ~[?:?]
	at org.apache.seatunnel.connectors.seatunnel.file.sink.commit.FileSinkAggregatedCommitter.lambda$commit$0(FileSinkAggregatedCommitter.java:53) ~[?:?]
	at java.util.ArrayList.forEach(ArrayList.java:1541) ~[?:?]
	at org.apache.seatunnel.connectors.seatunnel.file.sink.commit.FileSinkAggregatedCommitter.commit(FileSinkAggregatedCommitter.java:45) ~[?:?]
	at org.apache.seatunnel.engine.server.task.SinkAggregatedCommitterTask.notifyCheckpointComplete(SinkAggregatedCommitterTask.java:307) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
	at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointFinishedOperation.lambda$run$0(CheckpointFinishedOperation.java:91) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
	at org.apache.seatunnel.common.utils.RetryUtils.retryWithException(RetryUtils.java:48) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
	at org.apache.seatunnel.engine.server.checkpoint.operation.CheckpointFinishedOperation.run(CheckpointFinishedOperation.java:81) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
	at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:189) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:273) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:248) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:213) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:175) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:139) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.executeRun(OperationThread.java:123) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
	at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) ~[seatunnel-starter.jar:2.3.4-SNAPSHOT]
Caused by: java.net.URISyntaxException: Relative path in absolute URI: ftp://T_792686344391360513_528d0e3d11_0_1_0.csvSaaS(å¤%C2%9Aç§%C2%9Fæ%C2%88·)å%C2%8C%C2%96è®¾è®¡ä¸%C2%8Eå®%C2%9Eç%C2%8E°.docxSaaS(å¤%C2%9Aç§%C2%9Fæ%C2%88·)å%C2%8C%C2%96è®¾è®¡ä¸%C2%8Eå®%C2%9Eç%C2%8E°.docx
	at java.net.URI.checkPath(URI.java:1940) ~[?:?]
	at java.net.URI.<init>(URI.java:757) ~[?:?]
	at org.apache.hadoop.fs.Path.initialize(Path.java:256) ~[seatunnel-hadoop3-3.1.4-uber.jar:2.3.4-SNAPSHOT]
	... 23 more

Zeta or Flink or Spark Version
Zeta
Java or Scala Version
1.8
Screenshots
No response
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

@xiaominyan88  Fixed

repo: apache/seatunnel


text:
How to deploy shenyu gateway in HA multi-clusters?

Question
shenyu如何进行高可用的部署,目前我尝试启动2个admin 2个bootstrap  共用一个数据库,有一些问题.
采用的是nacos数据同步,也同时启用了dubbo插件,2个admin服务都走了同样的逻辑,通过订阅nacos里的服务和配,重复覆盖mysql里的配置.

For admin, please check  https://github.com/apache/shenyu/issues/4881 ;
For bootstrap, you can deploy multiple nodes in the gateway using the same registry.

shenyu admin多节点部署是个问题

数据同步一致性

A new tool for syncing data from different clusters is a good idea.
Can you provider a task in GSoC2024 in shenyu?

The data has been stored in the database, so canel is enough for the migration between different admin clusters.
For gsoc 2024, I'd like to provide KitexPulgin or RpcxPlugin task.
---- 回复的原邮件 ----
| 发件人 | @.> |
| 发送日期 | 2024年01月27日 21:51 |
| 收件人 | apache/shenyu @.> |
| 抄送人 | dragon-zhang @.>,
Comment @.> |
| 主题 | Re: [apache/shenyu] 分布式高可用网关部署问题 (Issue #5388) |
A new tool for syncing data from different clusters is a good idea.
Can you provider a task in GSoC2024 in shenyu?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: @.***>

Biz data syncing in middleware nodes should not depend on db tools.
I still think it make sense for this feature, such as Nacos-Sync.

source: gharchive/issue

authors: MingJunDuan dragon-zhang kimmking wljin321

repo: apache/shenyu

license_type: permissive

license_source: github-api


text:
[Question] Why is dashboard another project

Question
I found that the dashboard is another project, and admin is putting the compiled files. I don't know what the current process is like. Why not link the two through Git submodule, or directly import the dashboard code into the Shenyu main repository through the frontend-maven-plugin?

Of course ,there are a couple of approaches that could be used to manage the integration of the dashboard with the main repository, and you're suggesting using either a Git submodule or the frontend-maven-plugin. Let’s break down these options and the potential advantages or disadvantages of each approach.
1. Using Git Submodule
A Git submodule allows you to keep the dashboard as a separate repository, but link it to the main repository so that changes to the dashboard can be tracked and pulled in as needed.
Benefits:

Separation of concerns: The dashboard can remain in its own repository, which means you can manage it separately from the core gateway code. This is useful if the dashboard has its own release cycle or if it’s used by multiple projects.
Versioning: You can lock the main repository to a specific version of the dashboard, ensuring stability, but also making it easy to update to newer versions when required.
Reuse: If the same dashboard is used by multiple projects, a submodule makes sense for code reuse without needing to duplicate the dashboard code.

Drawbacks:

Complexity: Submodules can be tricky to manage, especially for developers who are not familiar with them. You need to remember to update the submodule (git submodule update --init --recursive) and track changes across two different repositories.
Deployments: Depending on how you're building and deploying the project, ensuring the submodule is pulled and built properly can add complexity to your CI/CD pipeline.
Tight coupling: If changes to the dashboard and the main project are closely linked, using submodules could make the development process more cumbersome, as commits would need to be coordinated across two repositories.

2. Using the frontend-maven-plugin
The frontend-maven-plugin allows you to run Node.js and frontend build tools like npm or webpack as part of your Maven build process. This would involve pulling the dashboard’s code directly into the main repository and building it as part of the Maven build.
Benefits:

Integrated build process: By using the frontend-maven-plugin to build the dashboard, you ensure that the dashboard is always built and compiled along with the main project. This makes the build process more cohesive and reduces the likelihood of version mismatches.
Simplified workflow: Developers don’t have to worry about managing a separate repository or submodule. Everything is in the main repository, and the build process is unified.
Easier deployment: Since the dashboard is built as part of the main project, deployment can be streamlined. The compiled frontend assets can be packaged together with the backend code or served directly from the gateway.

Drawbacks:

Monolithic structure: Combining the dashboard with the main repository could lead to a more monolithic project structure. If the dashboard and the gateway code are developed by different teams or have different lifecycles, this tight coupling may be undesirable.
Frontend dependencies in Maven: You’re introducing Node.js, npm, and potentially a lot of frontend dependencies into your Maven build process. This can increase build times, especially if there are frequent changes to the dashboard, and make the build process more complex.
Scaling challenges: If the dashboard grows independently or has its own release cycle, keeping it in the same repository could be a bottleneck for scaling and could slow down development.

Which Option to Choose?
The decision depends on how closely coupled the dashboard is to the main project and how the team prefers to manage the repositories.

Use a Git submodule if:

The dashboard is a separate, reusable component that may be used across multiple projects.
You want to maintain a clear separation between the dashboard and the main project.
You don’t mind the additional complexity of managing submodules.

Use the frontend-maven-plugin if:

The dashboard is tightly coupled to the main project and is unlikely to be reused in other projects.
You want a more simplified build and deployment process where everything is handled together.
You don’t mind introducing frontend tooling into your Maven build process.

Potential Hybrid Approach:

If you want to keep the dashboard separate for development purposes but still want a unified build process, you could use a combination of both approaches: keep the dashboard in a separate repository and use a Git submodule to pull it into the main repository, and then use the frontend-maven-plugin to handle the build process once the submodule is pulled in.

In summary, both approaches have their pros and cons, and the best choice would depend on how your team prefers to manage the separation of concerns, the complexity of the build process, and future plans for the dashboard’s development.
Are you happy with this answer?

authors: eye-gu yunmaoQu


text:
Can not get value from TraceContext.getCorrelation()

Please answer these questions before submitting your issue.

Why do you submit this issue?
[ ] Question or discussion
[ ] Bug
[ ] Requirement
[ ] Feature or performance improvement

Question

What do you want to know?
How to use TraceContext.putCorrelation() properly?

Bug

Which version of SkyWalking, OS, and JRE?
SkyWalking: 8.2.0
OS: MAC 11.4
JRE: 1.8

Which company or project?
Micro service

What happened?
I am using skywalking to track kafka message.I put the traceId into kafka header, then I get the traceId from kafka header at consumer side. Then, cause I can not get traceId by TraceContext.traceId() from consumer side. I want to put the value like this: TraceContext.putCorrelation("traceId", traceId). So that I can get traceId by TraceContext.getCorrelation("traceId"). However, the value is empty.

Requirement or improvement

Please describe your requirements or improvement suggestions.
Maybe I'm missing something. Anyone can tell me a right way to achieve this? Any help will be appreciated.

Here are some code and logs:
Code:
log.info( "===Headers-traceId==={}", traceId );         TraceContext.putCorrelation( "traceId", traceId );         log.info( "Step0: Start. traceId:{}", TraceContext.getCorrelation( "traceId" ) );
Logs:
===Headers-traceId===b871f6620a76412ab23794b9c11801f2.118.16234074007380001 2021-06-11 18:30:03.862  INFO 17034 --- [ntainer#1-0-C-1] c.s.m.message.TopicConsumer              : Step0: Start. traceId:Optional.empty

Besides, I had tried to put the info from kafka producer side, but it still can not get value from consumer side.

Do you read after consumering? If not, it is not realistic to get.
Two, the trace id is not read by this way. Don't know why you are using this.
Your way of using is not following official document.

I have encountered the same problem. May I know how it was resolved in the end

app-gateWay:
TraceContext.putCorrelation("demo_userId", s);
app-A1:
TraceContext.getCorrelation("demo_userId") return is empty
agent config:
correlation.auto_tag_keys=${SW_CORRELATION_AUTO_TAG_KEYS:demo_userId}
May I know how it was resolved in the end? thanks

authors: ZHANG-GXJ jj2784992 wu-sheng

repo: apache/skywalking

license_source: bigquery


text:
Accessing Solr without using an Ingress

The Solr Operator tutorial wants to use an Ingress so that Solr is available from outside Kubernetes. However, I don't want it to be available outside Kubernetes, only use it internally. Normally with a deployment you would create a Service that other pods connect to, but I can't see any services being installed using Helm.
How can other pods in the same cluster access Solr without having to make Solr available through an Ingress?

You only need the ingress for external traffic. Applications running inside the cluster can use the service name for your cluster. Say you have a SolrCloud deployment called "example" with 2 pods, you will see that there is a -common service created that will load balance between the two pods, and you can also access each node with the -0 and -1 service names.
NAME                                          READY   STATUS                       RESTARTS
pod/example-solrcloud-0                       1/1     Running                      0
pod/example-solrcloud-1                       1/1     Running                      0

NAME                                          TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)
service/example-solrcloud-common              ClusterIP      10.43.185.20    <none>        80/TCP
service/example-solrcloud-0                   ClusterIP      10.43.69.190    <none>        80/TCP
service/example-solrcloud-1                   ClusterIP      10.43.16.234    <none>        80/TCP

Thanks for the swift reply :)
I tried doing a kubectl port-forward to my "example-solrcloud-common" (8080 -> 80), but when I access http://localhost:8080 I get the following error in the shell:
Handling connection for 8080
E1202 15:12:11.516200   32672 portforward.go:400] an error occurred forwarding 8080 -> 8983: error forwarding port 8983 to pod 8a8abe2427d899d7e84f2e524fc43747d9e05ba49d23eda7bf263f731e66d6fa, uid : failed to execute portforward in network namespace "/var/run/netns/cni-8c979efc-93e7-93a6-a2fb-a78a8185e794": failed to dial 8983: dial tcp4 127.0.0.1:8983: connect: connection refused

You can override the image for that container with:
busyBoxImage:
  repository: "busybox"
  tag: "1.28.0-glibc"
  pullPolicy: ""
  imagePullSecret: ""

This image is used for an init container that sets up the data directory/solr.xml.
And yes, the Solr Operator contains defaults that assume access to the default docker registry, docker hub. If you want to use a different repository then you need to set that for each image.
image:
  repository: "solr"
  tag: ""
  pullPolicy: ""
  imagePullSecret: ""

zk:
  provided:
    image:
      repository: "pravega/zookeeper"
      tag: ""
      pullPolicy: IfNotPresent
      imagePullSecret: ""

busyBoxImage:
  repository: "busybox"
  tag: "1.28.0-glibc"
  pullPolicy: ""
  imagePullSecret: ""

How can I do this in the helm install command?

Ah, values.yaml... I remember seeing something about that.
The quickstart tutorial in the documentation said "we will create a yaml" but then proceeds without any yaml, and there's no examples as far as I've been able to find.
Do you have an example values.yaml file available?

Ahh I think that is a sentence left over from before the Solr helm chart existed. The best example values.yaml file is actually the default: https://github.com/apache/solr-operator/blob/v0.5.0/helm/solr/values.yaml
You don't need to (and shouldn't) include everything, since helm will use the default values.yaml as the starting point, then replace anything that is in your override values.yaml. So you can take the default values.yaml and remove anything that you don't want to change. For the example you have provided, you can merely use the following: (sets solr to use 8.4, uses a custom storage class name, and persistent storage for solr, and uses custom repositories for every image)
image:
  repository: "custom.docker.repo/solr"
  tag: "8.4"

dataStorage:
  type: persistent
  persistent:
    pvc:
      storageClassName: "solr-storage"

zk:
  provided:
    image:
      repository: "custom.docker.repo/zookeeper"
    persistence:
      spec:
        storageClassName: "solr-storage"

busyBoxImage:
  repository: "custom.docker.repo/busybox"

Awesome! Thank you so much! :D

It seems there's a problem with the zookeeper image.
I've tried with docker.io/library/zookeeper and index.docker.io/zookeeper but it seems it can't find the 0.2.13 label... which is not really surprising as that label is not listed here.
I also found some references to praveda/zookeeper but that seems to not exist any more.
What would be a correct zookeeper image repository?

https://hub.docker.com/r/pravega/zookeeper is the correct image, it does not use the official zookeeper image.
You should also be using the 0.2.12 image, not 0.2.13. It should be using that by default, so it's surprising that it was trying to use the 0.2.13 label...

I don't know why that link works now. It gave a 404 earlier today. Might've been a glitch at Docker's end.
However, I changed the tag to be 0.2.12 but I still have problems.
The example-solrcloud-zookeeper pods wants to pull the image, based on this event:
Pulling image "index.docker.io/pravega/zookeeper:0.2.12"
But it is immediately followed by Error: ErrImagePull
Can you see any typos in the image name? Something is obviously wrong, but I'm at a loss to what.

I'm unsure what index.docker.io is. maybe try docker.io instead?
Are you being rate limited by docker?

I'm getting ErrImagePull with docker.io as well.
I've been experimenting with these addresses as the Azure Kuberenets Service we're running on doesn't seem to want to pull directly from Docker, so I need to prefix all images with Docker's FQDN, which is not really documented anywhere.
index.docker.io and docker.io seems to work for the Solr images, but it just refuses to get the Zookeeper image.

Closing this as we're way past the original question. Creating a new issue for the nodeSelector issues

authors: HoustonPutman gthvidsten janhoy

repo: apache/solr-operator


text:
[MINOR][DOC] Replace numTasks with numPartitions in programming guide

In programming guide, numTasks is used in several places as arguments of Transformations. However, in code, numPartitions is used. In this fix, I replace numTasks with numPartitions in programming guide for consistency.

Can one of the admins verify this patch?

source: gharchive/pull-request

authors: AmplabJenkins polarke

repo: apache/spark


text:
[SPARK-23647][PYTHON][SQL] Adds more types for hint in pyspark

Signed-off-by: DylanGuedes djmgguedes@gmail.com
What changes were proposed in this pull request?
Addition of float, int and list hints for pyspark.sql Hint.
How was this patch tested?
I did manual tests following the same principles used in the Scala version.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

If it's difficult to make a test, let's do a manual test and describe it in the PR description.

Hi, I added two subtasks (one for Python and one for R) here
Should I close this PR and open a new one or it is ok to just rename it to the new JIRA id?
@felixcheung I don't know a good way to test it, do you have any suggestion?

Jenkins, ok to test

Test build #88297 has started for PR 20788 at commit f5de3b8.

Test build #88297 has finished for PR 20788 at commit f5de3b8.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88297/
Test FAILed.

Test build #88309 has started for PR 20788 at commit f238f26.

Test build #88309 has finished for PR 20788 at commit f238f26.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88309/
Test PASSed.

Test build #88333 has started for PR 20788 at commit b126b13.

Test build #88333 has finished for PR 20788 at commit b126b13.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88333/
Test PASSed.

Hi,
any new feedback about this?
thank you!

Can one of the admins verify this patch?

@DylanGuedes let's add tests.

@HyukjinKwon Sure! I'll be working on this, then. Do you have any specific scenarios in mind, so that I could start with them?

Simply calling it should  be enough. See https://github.com/apache/spark/pull/21649/files

ok to test

Test build #98693 has started for PR 20788 at commit b126b13.

Test build #98693 has finished for PR 20788 at commit b126b13.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98693/
Test PASSed.

@HyukjinKwon I added a test but I checked only the logical plan - is it ok?

Test build #98792 has finished for PR 20788 at commit f1085c3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98792/
Test PASSed.

Test build #98824 has started for PR 20788 at commit de3257f.

Test build #98824 has finished for PR 20788 at commit de3257f.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class UTCOffsetTimezone(datetime.tzinfo):
class ExamplePointUDT(UserDefinedType):
class PythonOnlyUDT(UserDefinedType):
class PythonOnlyPoint(ExamplePoint):
class MyObject(object):
class ReusedSQLTestCase(ReusedPySparkTestCase):
class DataTypeTests(unittest.TestCase):
class SQLTests(ReusedSQLTestCase):
        self.assertRaisesRegexp(AnalysisException, \"Can not load class non_existed_udf\",
        self.assertRaisesRegexp(AnalysisException, \"Can not load class non_existed_udf\",
        self.assertRaisesRegexp(AnalysisException, \"Can not load class non_existed_udaf\",
        class F(object):
        class A():
class HiveSparkSubmitTests(SparkSubmitTests):
class SQLTests2(ReusedSQLTestCase):
class SparkSessionTests(PySparkTestCase):
class UDFInitializationTests(unittest.TestCase):
class HiveContextSQLTests(ReusedPySparkTestCase):
class DataTypeVerificationTests(unittest.TestCase):
class ArrowTests(ReusedSQLTestCase):
class PandasUDFTests(ReusedSQLTestCase):
class ScalarPandasUDFTests(ReusedSQLTestCase):
class GroupedMapPandasUDFTests(ReusedSQLTestCase):
class GroupedAggPandasUDFTests(ReusedSQLTestCase):

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98824/
Test FAILed.

Test build #98825 has started for PR 20788 at commit 4dacff2.

Test build #98825 has finished for PR 20788 at commit 4dacff2.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/98825/
Test PASSed.

Test build #98824 has finished for PR 20788 at commit de3257f.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
class UTCOffsetTimezone(datetime.tzinfo):
class ExamplePointUDT(UserDefinedType):
class PythonOnlyUDT(UserDefinedType):
class PythonOnlyPoint(ExamplePoint):
class MyObject(object):
class ReusedSQLTestCase(ReusedPySparkTestCase):
class DataTypeTests(unittest.TestCase):
class SQLTests(ReusedSQLTestCase):
        self.assertRaisesRegexp(AnalysisException, \"Can not load class non_existed_udf\",
        self.assertRaisesRegexp(AnalysisException, \"Can not load class non_existed_udf\",
        self.assertRaisesRegexp(AnalysisException, \"Can not load class non_existed_udaf\",
        class F(object):
        class A():
class HiveSparkSubmitTests(SparkSubmitTests):
class SQLTests2(ReusedSQLTestCase):
class SparkSessionTests(PySparkTestCase):
class UDFInitializationTests(unittest.TestCase):
class HiveContextSQLTests(ReusedPySparkTestCase):
class DataTypeVerificationTests(unittest.TestCase):
class ArrowTests(ReusedSQLTestCase):
class PandasUDFTests(ReusedSQLTestCase):
class ScalarPandasUDFTests(ReusedSQLTestCase):
class GroupedMapPandasUDFTests(ReusedSQLTestCase):
class GroupedAggPandasUDFTests(ReusedSQLTestCase):

Obs: such a gigantique number of files were listed because a new PR divided tests into multiple files. So I moved the new test to these new organization of tests.

Thanks. @DylanGuedes.

Test build #99055 has started for PR 20788 at commit 00a4888.

@ueshin @felixcheung @viirya hey, could you guys give this one a quick look? I can work more on it, just in case.

retest this please

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/5585/
Test PASSed.

Test build #99521 has started for PR 20788 at commit 00a4888.

Test build #99521 has finished for PR 20788 at commit 00a4888.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/99521/
Test PASSed.

authors: AmplabJenkins DylanGuedes HyukjinKwon SparkQA felixcheung


text:
[SPARK-29951][SQL] Make the behavior of Postgre dialect independent of ansi mode config

What changes were proposed in this pull request?
Fix the inconsistent behavior of build-in function SQL LEFT/RIGHT.
Why are the changes needed?
As the comment in https://github.com/apache/spark/pull/26497#discussion_r345708065, Postgre dialect should not be affected by the ANSI mode config.
During reran the existing tests, only the LEFT/RIGHT build-in SQL function break the assumption, we fix this by following https://www.postgresql.org/docs/12/sql-keywords-appendix.html: LEFT/RIGHT reserved (can be function or type)
Does this PR introduce any user-facing change?
Yes, the Postgre dialect will not be affected by the ANSI mode config.
How was this patch tested?
Existing UT.

Test build #114035 has started for PR 26584 at commit a250886.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18896/
Test PASSed.

Test build #114035 has finished for PR 26584 at commit a250886.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114035/
Test PASSed.

cc @cloud-fan

Test build #114094 has started for PR 26584 at commit 4953646.

Build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18952/
Test PASSed.

Test build #114094 has finished for PR 26584 at commit 4953646.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

Build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114094/
Test FAILed.

Test build #114096 has started for PR 26584 at commit d357a9a.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18954/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18959/
Test PASSed.

Test build #114102 has started for PR 26584 at commit f16b6ea.

Test build #114096 has finished for PR 26584 at commit d357a9a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114096/
Test PASSed.

Test build #114102 has finished for PR 26584 at commit f16b6ea.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114102/
Test PASSed.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/18988/
Test PASSed.

Test build #114130 has started for PR 26584 at commit c05adb9.

Test build #114130 has finished for PR 26584 at commit c05adb9.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114130/
Test FAILed.

Test build #114143 has started for PR 26584 at commit 5a4b2ea.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/19003/
Test PASSed.

Test build #114143 has finished for PR 26584 at commit 5a4b2ea.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/114143/
Test PASSed.

thanks, merging to master!

Thanks for the review.

authors: AmplabJenkins SparkQA cloud-fan xuanyuanking


text:
[SPARK-32884][TESTS] Mark TPCDSQuery*Suite as ExtendedSQLTest

What changes were proposed in this pull request?
This PR aims to mark the following suite as ExtendedSQLTest to reduce GitHub Action test time.

TPCDSQuerySuite
TPCDSQueryANSISuite
TPCDSQueryWithStatsSuite

Why are the changes needed?
Currently, the longest GitHub Action task is Build and test / Build modules: sql - other tests with 1h 57m 10s while Build and test / Build modules: sql - slow tests takes 42m 20s. With this PR, we can move the workload from other tests to slow tests task and reduce the total waiting time about 7 ~ 8 minutes.
Does this PR introduce any user-facing change?
No. This is a test-only change.
How was this patch tested?
Pass the GitHub Action with the reduced running time.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/33318/
Test PASSed.

Test build #128695 has started for PR 29755 at commit 0b2483d.

This is nice. We should mark more tests. Merged to master.
I manually checked the logs and see it's properly excluded/included.

Thank you, @HyukjinKwon !

Test build #128695 has finished for PR 29755 at commit 0b2483d.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/128695/
Test FAILed.

Nice update! (To get a report Report test results, I often wait for the longest job to be completed... )

authors: AmplabJenkins HyukjinKwon SparkQA dongjoon-hyun maropu


text:
[SPARK-34487][K8S][TESTS] Use the runtime Hadoop version in K8s IT

What changes were proposed in this pull request?
This PR aims to use the runtime Hadoop version in K8s integration test.
Why are the changes needed?
SPARK-33212 upgrades Hadoop dependency from 3.2.0 to 3.2.2 and we will upgrade to 3.3.x+.
We had better use the runtime Hadoop version instead of having a static string.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass the K8s IT.
This is tested locally like the following.
KubernetesSuite:
...
- Launcher client dependencies
- SPARK-33615: Launcher client archives
- SPARK-33748: Launcher python client respecting PYSPARK_PYTHON
- SPARK-33748: Launcher python client respecting spark.pyspark.python and spark.pyspark.driver.python
- Launcher python client dependencies using a zip file
...

Test build #135319 has started for PR 31604 at commit a8ecb70.

Test build #135319 has finished for PR 31604 at commit a8ecb70.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135319/

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39899/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39899/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39899/

Thank you, @HyukjinKwon . For 2.7.x, it's difficult because we need to find the corresponding aws-java-sdk.

I landed this test PR to branch-3.1 for K8s GA, too.

authors: AmplabJenkins SparkQA dongjoon-hyun


text:
[SPARK-36783][SQL] ScanOperation should not push Filter through nondeterministic Project

What changes were proposed in this pull request?

ScanOperation collects adjacent Projects and Filters. The caller side always assume that the collected Filters should run before collected Projects, which means ScanOperation effectively pushes Filter through Project.
Following PushPredicateThroughNonJoin, we should not push Filter through nondeterministic Project. This PR fixes ScanOperation to follow this rule.
Why are the changes needed?

Fix a bug that violates the semantic of nondeterministic expressions.
Does this PR introduce any user-facing change?

Most likely no change, but in some cases, this is a correctness bug fix which changes the query result.
How was this patch tested?

existing tests

cc @Ngone51 @viirya

Test build #143357 has started for PR 34023 at commit 718cca4.

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47864/

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47864/

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/47864/

Test build #143357 has finished for PR 34023 at commit 718cca4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/143357/

thanks for the review, merging to master/3.2/3.1/3.0!

@cloud-fan This change also affects PhysicalOperation, causing hive partition pruning to not take effect when there is a non-deterministic project. Is this expected behavior?
like:
select *
from (
  select
    c1, reflect('java.net.URLDecoder', 'decode', c2, 'UTF-8') as c2, dt
  from test) t1
where dt='2024-05-15' limit 10;

Yes, because after pruning, less data are scanned and the non-deterministic function in the SELECT list may return different results if we don't prune ahead.

Yes, because after pruning, less data are scanned and the non-deterministic function in the SELECT list may return different results if we don't prune ahead.

Makes sense, thank you!

authors: AmplabJenkins SparkQA cloud-fan wForget


text:
[SPARK-38357][SQL][3.2] Fix StackOverflowError with OR(data filter, partition filter)

What changes were proposed in this pull request?
If the filter has OR and contains both data filter and partition filter,
e.g. p is partition col and id is data col
SELECT * FROM tmp WHERE (p = 0 AND id > 0) OR (p = 1 AND id = 2)

Spark throws StackOverflowError
Why are the changes needed?
bug fixing
Does this PR introduce any user-facing change?
No
How was this patch tested?
new UT

The failed UTs seems to be related to the current pr, for example SPARK-10829: Filter combine partition key and attribute doesn't work in DataSource scan in ParquetV2FilterSuite

Is this open vs 3.2 because it doesn't exist in master or did I miss another PR?

Thanks @dongjoon-hyun
I just submitted https://github.com/apache/spark/pull/35703 to add this test case to master.

cc @cloud-fan

I merged #35703 . Please address https://github.com/apache/spark/pull/35691#discussion_r817224113 too.

Can you post the stacktrace when overflow? I still don't understand why master and file source v1 is fine.

thanks, merging to 3.2!

Thank you all!

authors: LuciferYang cloud-fan dongjoon-hyun huaxingao srowen


text:
[SPARK-40737][CONNECT] Add basic support for DataFrameWriter

What changes were proposed in this pull request?
This change adds basic support for writes through the Spark Connect API. In todays implementation of the write API from a DataFrame perspective the interface of the DataFrameWriter is as declarative as possible today.
The write support is implemented as a Command and does not return anything.
Why are the changes needed?
Write support through Spark Connect.
Does this PR introduce any user-facing change?
Experimental API
How was this patch tested?
Added new unit tests for the behavior.

@cloud-fan @amaliujia @hvanhovell please take a look!

Can one of the admins verify this patch?

@hvanhovell @cloud-fan @HyukjinKwon can you please have a look?

Merging this one.

authors: AmplabJenkins grundprinzip hvanhovell


text:
[SPARK-47450][INFRA][R] Use R 4.3.3 in windows R GitHub Action job

What changes were proposed in this pull request?
This PR aims to use R 4.3.3 in windows R GitHub Action job.
Why are the changes needed?
R 4.3.3 is the latest release which is released on 2024-02-29.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass the CIs.
Was this patch authored or co-authored using generative AI tooling?
No.

Thank you, @HyukjinKwon ! Merged to master.

Seems like it doesn't work .. let me revert this for now.

Hi, @HyukjinKwon . This has been working for a week.

Oops

This is the yesterday's run.

Could you put it back by revert of reverting?

Let me take a quick look and fix up togehter.

🆗

I just reverted the revert for now. I am sorry it was my bad 🤦

No problem at all~ Let me take a look at Today's failure too~

This seems to fail for some reason Today. It would be transient issue during Windows installation. Let's wait for the next run.
Installing Ghostscript.app...
  Chocolatey timed out waiting for the command to finish. The timeout
   specified (or the default value) was '2700' seconds. Perhaps try a
   higher `--execution-timeout`? See `choco -h` for details.
    Ghostscript.app may be able to be automatically uninstalled.
  The install of Ghostscript.app was NOT successful.

For the record, it passed Today. We are good.

authors: HyukjinKwon dongjoon-hyun


text:
[SPARK-49806][SQL][CONNECT] Remove redundant blank space after show in Scala and Connect clients

What changes were proposed in this pull request?
The pr aims to remove redundant blank space after show in Scala and Connect clients.
Why are the changes needed?
Make the output of show in scala and connect both end with 1 blank space, making it look more consistent in style.
A.Scala (spark-shell)

Before:

Note: Other command end with 1 blank space, while show end with 2 blank spaces.

After:

B.Connect (spark-connect-scala-client)

Before:

Note: Other command end with 1 blank space, while show end with 3 blank spaces.

After:

Does this PR introduce any user-facing change?
Yes, command are separated by 1 blank space between them, and 'show' is no longer abrupt.
How was this patch tested?
Manually check.
Was this patch authored or co-authored using generative AI tooling?
No.

cc @HyukjinKwon @zhengruifeng

the python doctest highly depends on the show format, will this break them?

the python doctest highly depends on the show format, will this break them?

It looks like it is,

If we think it should be fixed, I can update it together.
Because it looks too weird, the separate between command is too inconsistent.

authors: panbingkun zhengruifeng


text:
[SPARK-50437][SS] Reduce overhead of creating deserializers in TransformWithStateExec

What changes were proposed in this pull request?
An optimization that creates deserializers for keys and values at the beginning of a batch.
Why are the changes needed?
Currently, deserializers for key and values are created every time handleInputRows gets called.   The function
ObjectOperator.deserializeRowToObject is called twice do generate the deserializer for the key and value.  The function will do the following things

Generate java code for the deserializer based on input expression and grouping attributes
Compile that java code to java class

The problem is that step 1 will occur every time the function is called.  The compiling the code, i.e. step 2, will only happen once since the generated class will be cached and can be reused.
Step 1 can incur a heavy penalty in situations where handleInputRows will be called many times.  For example, if there are many distinct keys.  Note that handleInputRows is called once per distinct key.
Here is a flamegraph:

Does this PR introduce any user-facing change?
No
How was this patch tested?
Existing tests should suffice
Was this patch authored or co-authored using generative AI tooling?
No

Thanks! Merging to master.

authors: HeartSaVioR jerrypeng


text:
Making class TextFileReader public as it is part of the public API for HDFS Spout

minor change

@roshannaik Duplicated #1461

closing this PR as its a dup

authors: HeartSaVioR roshannaik

repo: apache/storm


text:
fix parent version of storm-kinesis module (1.x)

Builds on PR for 1.x-branch are failing because of this.

Since all builds are failing, and it's not about code itself, I'll just merge this right now. Please revert if you think it should get +1.


text:
STORM-3136: Fix flaky integration test, clean up that code to be more…

… readable
https://issues.apache.org/jira/browse/STORM-3136

Deduplicated the test spouts and bolts a bit.
Made sure all test spouts are waiting a reasonable amount of time between emits. If the log rolls during the tests, the tests will fail. I think this is the reason for the flakiness. It was fixed for some tests in an earlier PR, but I missed that the spout code had so many duplicates.
Added some documentation
Replaced a bunch of Guava code with streams. Deleted unused code. Simplified some classes, e.g. TimeDataWindow

Ran this 4 times on top of the changes from https://github.com/apache/storm/pull/2733 on Travis, runs of the integration test passed.


text:
With hash key option for RedisMapState, only get the values for keys in the requested batch

This commit fixes a bug whereby if the state updater is constructed with a hash key (ie, the state will be stored as a key in a redis hash, versus as a key in the top-level redis space), each call to multiGet would request the entire hash and iterate to extract only the values in the hash relevant to the batch.
This can cause an inordinate amount of network traffic (and actually caused our interfaces to fall over) for states with either a moderately high cardinality or large values. Instead, the call to Redis should be an hmget (hash multiget) that takes the hash key as its first argument and an array of strings as the keys to fetch from that key, thereby retrieving only the requested values.
The change also deprecates and removes buildValuesFromMap.

@dkatten I've just missed it. Thanks! +1
@revans2 Could you take a look, please?

The change looks good to me. I am +1 for the change.  @dkatten if you want to file a JIRA under STORM at issues.apache.org that would be great.  If you would rather I can file it for you.

I'll file it. Thanks!

STORM-703
https://issues.apache.org/jira/browse/STORM-703

@revans2 It's filed to JIRA so seems like we can merge it.

Sorry I really don't follow all the JIRa updates as closely as I should.  @HeartSaVioR thanks for kicking me :) I'll merge it in.

authors: HeartSaVioR dkatten revans2


text:
WW-5084: Add Content Security Policy support to Struts

Hello Struts Devs!
This PR adds Content Security Policy support for Struts 2. A very popular security mitigation against XSS and other injection vulnerabilities. CSP comes in many flavours, but we've chosen to add support for the most robust of them: nonce-based, strict-dynamic CSP.
Here's a summary of these changes:

Allows users to configure whether CSP is enabled in reporting or enforcement modes and lets them set a report URI, where violation reports will be sent by the browser.
Implements a CSP Interceptor that generates a nonce-based, strict-dynamic policy and adds it to HTTP responses according to the user's configuration.
Implements custom <script> and <link> JSP and FTL tags. These (<s:script> in taglib, for instance) set the nonce attribute on script and link blocks automatically, so that they match the nonce set in the policy. This feature allows developers to use both existing and new script blocks that are compatible with CSP with minimal refactoring.
Provides a default implementation of a CSP violation report collection endpoint. This allows developers to see CSP reports as they happen in their logs out of the box, with minimal effort. This behaviour is extensible, so developers can customise the processing of CSP reports.
With these tools, developers can enable CSP in reporting mode, collect reports and identify and refactor existing code that is incompatible with CSP from these reports. Finally, developers will be able to switch CSP to enforcing mode, which will provide a very robust defense against XSS.

Co-authored-by: Ecenaz Jen Ozmen - eo2419@columbia.edu
Co-authored-by: Giannis Chatziveroglou - giannisc@google.com
Co-authored-by: Santiago Diaz - salchoman@gmail.com

Coverage increased (+0.2%) to 49.539% when pulling 4d8bec5d0a56dd8fefaf3783ed57ae65139525cc on salcho:post-ww-5083 into a55e9ed93b3f442a1230be4b9b2ca936155868f5 on apache:master.

That's great news @lukaszlenart! Is there anything else we can do to get these merged? :)

I'm just giving some time to others to allow them review your work :)

@salcho @lukaszlenart Do we need CspReportAction in the Struts itself? Seems like it is too implementation specific.

Hi @aleksandr-m @lukaszlenart,
Re: CspReportAction. In our experience, many developers are put off of adopting CSP in their applications because collecting reports adds more overhead. We decided to add the CspReportAction class with a default implementation to give those developers the option to either use an out of the box (and useful, since the reports can be seen in logs) implementation or subclass the action and implement custom logic.
Alex, I've pushed a new commit that addresses your comment. Thanks for flagging!

Friendly ping! @aleksandr-m @lukaszlenart

I'm fine with the current state, LGTM 👍  (but I will let @aleksandr-m hit the button  :)

@lukaszlenart Hi, for <s: script><s:/script>, if the src of the tags needs variables, what should I do?
<s:script src="${myUrl}"></s:script>

@TheFergus already answered here https://github.com/struts-community-plugins/struts2-jquery/pull/318#issuecomment-1333471447

authors: TheFergus aleksandr-m coveralls lukaszlenart salcho

repo: apache/struts


text:
[dnd] reordering of multiple column/metrics is not working smoothly（superset 1.3 only）

Reordering of multiple column/metrics is not working smoothly, some times can not drag a component to the desired order by dnd
Expected results
can drag component to desired position for ordering
Actual results
sometimes can not drag component to desired position for ordering
Screenshots
https://user-images.githubusercontent.com/81597121/134820334-40d6a271-42f9-4f5d-80d2-540f4f8ec61b.mov
How to reproduce the bug
Precondition: Turn on drag and drop

Go to explore
add multiple metric or add multiple column to filter
reordering the metrics or columns by dnd
See error

Environment
(please complete the following information):

browser type and version:
superset version: superset 1.3 RC2
python version: python --version
node.js version: node -v
any feature flags active:

Checklist
Make sure to follow these steps before submitting your issue - thank you!

[ ] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[x] I have reproduced the issue with at least the latest released version of superset.
[x] I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context
Add any other context about the problem here.

@jinghua-qa I tested it in my local master branch and it works fine. Is there more reproduce context?

authors: jinghua-qa stephenLYZ

repo: apache/superset


text:
"/v1/dashboard/2/filter_state?tab_id=13"   This API prompt “400 Bad Request: The CSRF tokens do not match.”

How can I solve the problem that occurs during iframe embedding?

I'm also having the same issue. Though it's not affecting how I'm using the filters on the dashboard.

Is this using the Embedded SDK or some other form of igrame embed? How are you deploying Superset, and what version? This seems like it's more of a configuration issue than a Superset bug, but it's hard to say the state of affairs without the remainder of the Issue template filled out.
I'll close this as stale since it's been almost 6 months since a comment, but I'm happy to re-open if further details merit it... or if it's a Q&A/config issue that needs tackling, we can take it to Github discussions or slack


text:
fix: Query execution time is displayed as invalid date

SUMMARY
The Time column in the Query History is being displayed as an Invalid date.
The reason being the underlying data type is numeric with precision, to get sub-second precision.
That means the returned json value is a string (to keep the precision) and in turn, it fails the conversion to a moment object.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Before:

After:

TESTING INSTRUCTIONS

Go to SQL lab -> SQL Editor
Run any query
Go to SQL lab -> Query History

Ensure the time column is displaying the proper start time.
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[x] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

/testenv up

/testenv up

🏷 preset:2022.17

authors: diegomedina248 sadpandajoe yousoph


text:
chore: Update Rockset—switching out rockset for rockset-sqlalchemy

chore: bump rockset
docs: update Rockset usage (include virtual instance configuration)
SUMMARY

superset used a very old version of rockset (0.8.10). The latest version is 2.0.0. However, SQLAlchemy support no longer comes with rockset and has been separated into its own package, rockset-sqlalchemy. I have updated setup.py to reference this new package. This change should be completely backward compatible.
This new package comes with the ability to target specific virtual instances through the SQLAlchemy connection URI. I have updated the docs to show this new feature.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Before:

After:

TESTING INSTRUCTIONS

Create a Rockset collection (you can use one of the Rockset public datasets).
Start the superset server and connect to the Rockset database.
Create a dataset from the collection.
Test that chart creation works.
Test that SQL lab works.

ADDITIONAL INFORMATION

[x] Has associated issue: https://github.com/rockset/rockset-python-client/issues/3
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[x] Introduces new feature or API
[ ] Removes existing feature or API

Sorry @gadhagod one more thing. Would you also mind adding a line item to this alphabetically ordered (somehow Trino is in the wrong place) document?

@john-bodley ✅

@john-bodley or @betodealmeida is this going to be a breaking change? For people who use rockset, they'll need to change the driver that they use. Even though the driver itself is backward compatible, the import of the old driver will no longer work given the new requirements in Superset.

@eschutho I'm not sure. The SQLAlchemy URI is unchanged as far as I'm can tell and thus it's no more breaking than other changes which bump an eligible version of a package.

authors: eschutho gadhagod john-bodley


text:
TAJO-1619: JDBC program is stuck after closing

+1 LGTM!!

authors: jihoonson jinossy

repo: apache/tajo


text:
TR can not call crState, crConfig endpoint if TM do not use port 80

Hi,
This is my case, I setup TM with another port, for example 82, and in the traffic_monitor.properties file of TR I declare traffic_monitor.bootstrap.hosts=trafficmonitor.example.com:82. Then start the TR, the TrafficMonitorWatcher show that it can call TM endpoint, but the PeriodicResourceUpdater always call the endpoint at port 80, I have to hard code in /traffic_router/core/src/main/webapp/WEB-INF/applicationContext.xml:
<property name="stateUrl" value="http://trafficmonitor.example.com:82/publish/CrStates" />
<property name="configUrl" value="http://trafficmonitor.example.com:82/publish/CrConfig?json" />

and rebuild the rpm.

What is the TCP Port of your Traffic Monitor servers in Traffic Ops? I think if they are set to 80 in Traffic Ops, your Traffic Router will pick up that 80 from Traffic Ops and use that rather than the bootstrap config (which should really just be for bootstrapping the Traffic Router anyways).

authors: jindov rawlinp

repo: apache/trafficcontrol


text:
Internal Server Error when assigning federation resolver id that doesn't exist

I'm submitting a ...

[X] bug report
[ ] new feature / enhancement request
[ ] improvement request (usability, performance, tech debt, etc.)
[ ] other

Traffic Control components affected ...

[ ] CDN in a Box
[ ] Documentation
[ ] Grove
[ ] Traffic Control Client
[ ] Traffic Monitor
[X] Traffic Ops
[ ] Traffic Ops ORT
[ ] Traffic Portal
[ ] Traffic Router
[ ] Traffic Stats
[ ] Traffic Vault
[ ] unknown

Current behavior:
When making a POST request to federations/{{id}}/federation_resolvers with the request body:
{
	"fedResolverIds": [201]
}

Where the id given in the array doesn't exit/pertain to any federation resolver.
The response returned is:
HTTP/1.1 500 Internal Server Error
Content-Type: application/json

{
    "alerts": [
        {
            "text": "Internal Server Error",
            "level": "error"
        }
    ]
}

Expected / new behavior:

Should return:
HTTP/1.1 404 Not Found
Content-Type: application/json

{
    "alerts": [
        {
            "text": "federation resolver does not exist",
            "level": "error"
        }
    ]
}

Minimal reproduction of the problem with instructions:

Anything else:

I'm not sure if this should be a:
400 Bad Request - "Invalid resolver id(s) provided" or
404 Not Found - "Resolver not found"

I based it off of the server_server_capabilities endpoint, when providing a server id that doesn't exist to this endpoint, it returns a 404 and an alert saying "server 1000000 does not exist".

I vote for 404

404 is fine IMO, but also 409 :eyes:

409 sounds great for api v3.x imo


text:
magick.so: explicitly reference wand/core libraries

Fixes the magick plugin linking for gcc ubuntu 20.04 and debian 11

[approve ci autest]

[approve ci autest]

Cherry-picked to v10.0.x

authors: bneradt cmcfarlen traeak

repo: apache/trafficserver


text:
TS-4488: fix Segmentation fault at HttpSM::tunnel_handler_ua

Returning to PR. It seems I found how to reproduce this bug. It happens when transformation is used (for example from null transform plugin) and client terminates connection before the transformation consumes all data from upstream vconnection.

[approve ci]

FreeBSD build failed! See https://ci.trafficserver.apache.org/job/Github-FreeBSD/220/ for details.

Linux build successful! See https://ci.trafficserver.apache.org/job/Github-Linux/117/ for details.

The FreeBSD repo was in limbo, trying again. [approve ci].

Linux build successful! See https://ci.trafficserver.apache.org/job/Github-Linux/118/ for details.

FreeBSD build successful! See https://ci.trafficserver.apache.org/job/Github-FreeBSD/221/ for details.

maybe should add EPIPE check?
    if (!r || r == -ECONNRESET || r == -EPIPE) {   // SIGPIPE when write on a closed fd.
      vc->write.triggered = 0;
      write_signal_done(VC_EVENT_EOS, nh, vc);
      return;
    }
    vc->write.triggered = 0;
    write_signal_error(nh, vc, (int)-total_written);
    return;

make the iocore callback with EOS

@oknet, no, this isn't helping.
It looks like cache write consumer tries to complete reading data from transformation but server_session already closed because transformation reads data from cache.
BTW, I see this error only on ATS-4.2.x, so I will close this PR and Jira issue

authors: atsci oknet yatsukhnenko zwoop


text:
Create clean exit path for traffic_manager on SIGTERM

Since the removal of traffic_cop, it appears that there is no way to cause traffic_manager to exit cleanly.  SIGTERM causes the cleanup of child processes and a graceful shutdown, however still exits with code 15.  We can ignore this in the systemd unit file, but it's cleaner to have some path to a clean exit of traffic_manager.

It might also be desirable to add a "--manager" option to traffic_ctl server stop, to be parallel with restart?  I'm not entirely clear on the intended operation, however both the init.d file and the systemd unit use signals today, so a clean way to exit traffic_manager via signal is always valuable.

[approve ci fedora]

@jeredfloyd We do plan to get rid of traffic_manager in the next major release (10.0.0).

[approve ci fedora]

@bryancall Thanks! Once I have 9.x in Fedora I will try to follow future plans a little more closely.

Cherry-picked to v9.2.x

authors: bneradt bryancall jeredfloyd randall zwoop


text:
TS-4776: Emulate the effect of O_DIRECTORY if it is not defined

Emulate the effect of O_DIRECTORY with stat() if it is not defined. According to man page of open(), it returns ENOTDIR if the specified pass was not a directory.

ENOTDIR
A component used as a directory in pathname is not, in fact, a directory, or O_DIRECTORY was specified and pathname was not a directory.

@jacksontj Could you take a look at this please? Because the line seems to be added with your PR #653.

FreeBSD build successful! See https://ci.trafficserver.apache.org/job/Github-FreeBSD/607/ for details.

Linux build successful! See https://ci.trafficserver.apache.org/job/Github-Linux/503/ for details.

Status on this?

Waiting +1s.

:+1:
Builds on OmniOS now (however, there's a LuaJIT error now).

authors: atsci maskit zwoop


text:
[Bug] [BYOC] After partitioning through BYOC, the constant memory of the operator in the model is out of order

After partitioning through BYOC, the constant memory in the operator in the model is out of order

I don't think this is a problem. Actually I'm not sure if these numbers represent some ordering. Feel free to reopen if this is causing you a real problem.

repo: apache/tvm


text:
[microNPU] Sum legalization support

Supports legalizing a relay sum operation to an equivalent series of NPU operations. It supports case with int8 output type and channel axis.
cc @leandron, @ekalda, @lhutton1

Thanks @Alexey-Yazev, @sergey-grovety!


text:
[Draft][TIR] Require buffer declaration (DeclBuffer) before use

Part of RFC#70, requiring buffers to be declared prior to their usage.  This is currently a draft PR, to see what errors are found by the CI.

Super excited about this feature!

@tvm-bot rerun

And starting the second step of the gradual implementation described here.  https://github.com/apache/tvm/pull/15243 is the first test that adds DeclBuffer in an existing lowering pass where it didn't exist before.  Hopefully, starting from the end of the lowering flow and working backwards will help to reduce the amount of test failures occurring at once.

authors: Lunderberg junrushao


text:
[Unity] Replace relax_vm/memory_manager with memory/memory_manager

We can unify the runtime/relax_vm/memory_manager and runtime/memory/memory_manager by replacing the former with the latter.

This change has resulted in breakage here in MLC-LLM, due to the removal of the "vm.builtin.memory_manager.clear" packed func.  Is there a renamed version of this function?

authors: Lunderberg yongwww


text:
[6/6] Arm(R) Ethos(TM)-U NPU codegen integration with tvmc

This PR integrates the codegen for Arm® Ethos™-U with tvmc.

Add an ethos-u target
Adds test coverage for the new target

This PR is blocked on merging #8795 , #8806, #8811 and #8849. Since this is built on top of all those PR branches, this contains the accumulation of contents of the all the PRs.
Co-authored-by: Manupa Karunaratne @manupa-arm
cc @manupa-arm @Mousius @grant-arm @gromero @areusch @mbaret  @u99127 for reviews

CI is now green, and all the dependency patches are now merged. Can we merge this, when possible?

Thanks everyone, this is merged.

authors: leandron mbaret


text:
Jenkins CI test

will not be merged.
for test purpose.

Why doesn't our travis work for now?

I have no good idea. https://issues.apache.org/jira/browse/INFRA-14005.

I saw the status of Zeppelin's PR builder in builds.a.o and it doesn't seems to work properly.

PR builder triggered and run normally. But somehow github page does not display build state.

authors: Leemoonsoo jongyoul

repo: apache/zeppelin


text:
[ZEPPELIN-2527] Use thin (normal) cursor.

What is this PR for?
Previously, it's hard to recognize where (before or after) the cursor is since it's too thick.
Also, it's better for fonts which is not monospaced. (e.g Chinese, Korean, ...)
See the attached GIFs for comparison.
These key bindings are still supported in the default mode.

CTRL + P, CTRL+N: go to next / previous line
CTRL + F, CTRL+B: go to next / previous letter
CTRL + A, CTRL+E: go to the beginning / end of the line

What type of PR is it?
[Improvement]
Todos

[x] - Use default mode instead of emacs
[x] - Bind CTRL + P for OSX.

What is the Jira issue?
ZEPPELIN-2527
How should this be tested?

Build: mvn clean package -DskipTests;
Open a paragraph.
Check the cursor

Screenshots (if appropriate)
Before

After

Questions:

Does the licenses files need update?
Is there breaking changes for older versions?
Does this needs documentation?

@zjffdu Could you help review this?

Thanks @1ambda LGTM

Do we need to keep undocumented emacs shortcuts?

Let me find a different way. Changing cursor while keeping emacs shortcut.

Created a PR with css only change so that emacs shortcut can be preserved:
https://github.com/apache/zeppelin/pull/2462

authors: 1ambda malayhm zjffdu


text:
🛑 InterHop (Website) is down

In bbb723d, InterHop (Website) (https://interhop.org) was down:

HTTP code: 0
Response time: 0 ms

Resolved: InterHop (Website) is back up in bb01ef0 after 7 minutes.

license: MIT


text:
[Improvement] addons re version

some add-on version used engine version,
e.g.
milvus 2.3.2
minio 5.2.0
orchestrator 3.2.6
pika 3.5.1
it's better to make it use kb version.

orchestrator version is: 0.8. 0.9, and 3.2.6 (deprecated)
minio version is: 0.9.0, 5.2.0 (non-addon mode, deprecrated)
pika is not maintained by KB.
@loomts only milvus 2.3.2 should be considered.

authors: loomts shanshanying

repo: apecloud/kubeblocks-addons


text:
docs: Correct the per-SP metadata overrides guide

In order to override the SAML2 idP's metadata and/or keys, one needs to
provide the folder structure described in this commit, e.g.
<service_name>-<service_id>
The official guide on CAS' blog is already changed to reflect this
issue [0].

Thanks much for the patch and the earlier blog update, of course!

authors: linosgian mmoayyed

repo: apereo/cas


text:
shim: Support Binary data in Node.js shim

The api gateway supports binary.
However, you can not handle binary using go-apex.
So we added the identifier "__BINARY__" to the Node.js Shim for go-apex.
https://github.com/apex/go-apex/issues/58

hmm I've had no issues with binary in the API Gateway proxy stuff

Currently, you can do this using the proxy function of API Gateway.
But I want to be able to use it without using a proxy.
So I made it.

repo: apex/apex


text:
Legend Formatter?

I'm trying to format the legend on my Donut chart to include some extra data. I see that the source apexcharts docs show that there is a formatter for the legend:

I may have missed it, but don't see an option for this in the wrapper. Any clue on how to do this?

Hello,
The formatter is missing, I have added it and it will be available in the next release

fivex in Release v0.9.19-beta

repo: apexcharts/Blazor-ApexCharts


text:
sorting by data number

is there a way to sory by data number?
https://codepen.io/diomed/pen/BqWBPr
I'd like to be able to sort graph from smallest number to largest
also what could I do to increase height, and to display full text of what is written in bar,
since it gets cut if amount is too small?

Sorting data items is a good suggestion, but in my opinion, needs to be handled externally before injecting data into the chart.
Regarding your second query, there is currently no way to manipulate text other than using label formatters. Also, you can display full text in the tooltip if it gets cropped in the bar data-labels

as you can see, I already have tooltip, so that saves it.
how would this be handled externally?

Something like this
var series = [{
  data: [{
    x: "Apples",
    y: 20
  }, {
    x: "Bananas",
    y: 24
  }]
}]
series[0].data.sort(function(a, b){
  return a.y > b.y;
});

After you sort it, you can then pass that data to the ApexCharts series.

authors: diomed junedchhipa


text:
Added missing project files, gitignore file, and implemented an easier way for custom nodes to be created

Hey there!
I found your post in Facebook the other day, and since I'm interested in seeing this work for a project of mine, I though I'd help by making some stuff better and also adding a few features.
In this pull request I added the Unity project editor files and the meta files of the project.
I coded a way to find via Reflection all classes that implement the Node base class instead of letting users add their custom nodes in the library's code.
Also, I added a new attribute that will allow placing context menu items in hierarchies and also changing their names, if needed.
Feel free to drop me a line anytime :+1:
~ ikromm

Hello ikromm,
thank you for the pull request. I really appreciate that! I thought that it would not be a good idea to use reflection for this project.. but I must agree that the annotation for the nodes is very nice. There a re just two points:
I would like to return a Node and not an object for the CreateNode method. Let us do the casting in this method.
I dont know if it is a good idea to commit a .gitignore  to the project. I usually have my individual one because of different IDE files. If you want we could add it but have to add some file types that are created by my IDE (Rider EAP).
I would like to have a consistent coding style. I am still not sure wich one to use.. but meanwhile let write method names wit a capital letter. (GetItemMenuName).
I will merge the request later and add the changes. Are you ok with that? Thank you for the effort!

Hey there!
Thanks for getting back to me.
Why do you want to avoid Reflection? I suppose the code will work only on the Editor, so there would be no problem if that part of the library is missing from a few of Unity's framework implementations.
I added the meta files and the gitignore because not all people know what to commit. For example I do not use a global gitignore since other projects might have different "rules" for which file to ignore or not.
I'd say IDE files will be bad. Not everyone uses the same IDE and they have no need to download them in the first place. That's why I didn't commit the Visual Studio project files Unity creates for example
As for the coding style, I generally create private methods with the first letter lowercase and protected and public with uppercase. It's just an easy way to recognize the scope of the function when glancing the name.

I thought avoiding reflection would make to code easier for beginners. But I like your solution and I will merge it in.
Yes, IDE files are bad. I do not want to commit them.. so I will add the Rider files to your gitignore.
The coding guideline is a topic that I need to investigate more.. until then let's try to be consistent at least in this project.

excellent!
yes, please add stuff to the gitignore file at will.
update the readme file with your thoughts and I'll create a new pull request when I get a chance :)
I'll stick to first capital letter for all methods defined, no worries.

authors: aphex- ikromm

repo: aphex-/BrotherhoodOfNode

license: mit


text:
OEV Rewards pages

Closes https://github.com/api3dao/oev-docs/issues/97

Will add the dApps in a separate PR


text:
Always getting 503 response when request any possible endpoints

Feature: A

    Scenario: First test
        Given I have valid client TLS configuration
        When I GET https://jsonplaceholder.typicode.com/posts/1
        Then response code should be 200
        And response header content-type should be text/html
        And response body should contain null

I want to use Apickli for API test automation, however when I try to test any possible endpoint it always gives me 503 response and the following info as an error:
 Then response code should be 200 # ..\..\apickli\apickli-gherkin.js:167
       Error: stepContext:
       testOutput:
         success:  false
         expected: 200
         actual:   503

If I am requesting the same endpoint https://jsonplaceholder.typicode.com/posts/1 in any browser - it successfully returns 200 to me
I have configured Client Authentication according to your instruction https://github.com/apickli/apickli#client-authentication
but it did not help :(

The test above is just informative, for better problem understanding, do not pay attention to steps after  "Then response code should be 200"

You should use When I GET /posts/1 and configure the host in init.js as described in the README <When I GET https://jsonplaceholder.typicode.com/posts/1>

authors: Rosohalex laughingbiscuit

repo: apickli/apickli


text:
Order of Operations doesnt work as expected for specs with multiple classifications

Spec has various endpoints grouped under a separate folders, for such specs order of operations is not working when defined the order across folders.

Hi @yajiv789
Currently, we don't support order of operations across folders, mainly because that use-case was not really requested.
2 questions:

Could you share your OpenAPI file and Portman configuration, so we can reproduce the behaviour that you are experiencing?
How would you want to the folders to be organised if you would be mixing requests over multiple folders?

I am expecting the requests across folders to be ordered as per the portman config, under new folder or under contract tests folder.

I am not expecting to be organised in the folders.
I am trying to check the possibility if we could create a new folder and put the mixed requests under that new folder.

Eg.
Folder A -> R1 / R3 / R4
Folder B -> R2 / R5
Contract Tests ->  New Folder ->  R1 / R2 / R3 / R4 / R5

I have been blocked on this, any help much appreciated. Thanks in advance!!

@yajiv789 The folder ordering is complicated to manage in a generic way, so it will take quite a large development effort to figure this out.

hi @yajiv789
We created a https://github.com/apideck-libraries/portman/pull/641 to introduce the option to sort Postman folders based on the "orderOfFolders" configuration. See the PR for more details.
Would this solve your need?

repo: apideck-libraries/portman


text:
Add support for preNavigationHooks in crawlers other than PlaywrightCrawler

This is an extension of #427 - ParselCrawler, BeautifulSoupCrawler and basically everything should support preNavigationHooks as well.

It might be a good idea to wait for #350 to be resolved before going for this.

@Mantisus

authors: janbuchar vdusek

repo: apify/crawlee-python


text:
Introducing Crawlee Python Guru on Gurubase.io

Hello team,
I'm the maintainer of Anteon. We have created Gurubase.io with the mission of building a centralized, open-source tool-focused knowledge base. Essentially, each "guru" is equipped with custom knowledge to answer user questions based on collected data related to that tool.
I wanted to update you that I've manually added the Crawlee Python Guru to Gurubase. Crawlee Python Guru uses the data from this repo and data from the docs to answer questions by leveraging the LLM.
In this PR, I showcased the "Crawlee Python Guru", which highlights that Crawlee Python now has an AI assistant available to help users with their questions. Please let me know your thoughts on this contribution.
Additionally, if you want me to disable Crawlee Python Guru in Gurubase, just let me know that's totally fine.

Hello @kursataktas, kudos for building gurubase.io! We don't mind being there, but we don't want to link to it from our readme. Best of luck with your project!

authors: janbuchar kursataktas


text:
Rewrite local file loading to not use require

require doesn't work for JSON files not having an extension: https://github.com/joyent/node/issues/6073

I forgot I had already done this.


text:
Refactor embedded kotlin compiler out of plugin build system

This is a large change that significantly simplifies the plugin build system by
having the kotlinc handle everything related to script compilation.  Since we
no longer use package name rewriting, the "packageName" option has been dropped
from the gradle script API and has been removed from existing plugins.
In addition to simplifying the build system, it should also significantly speed
up build times.  Previously we ran 2 build processes for every plugin and
didn't have fully working incremental compilation for scripts compared to 1
build process with built-in incremental compilation.
There were also some random pieces of code that referenced kotlinc directly, so
those have either been removed or refactored out.

LGTM

authors: Major- garyttierney

repo: apollo-rsps/apollo

license: isc


text:
apollo密匙不生效问题

[x] 我已经检查过discussions
[x] 我已经搜索过issues
[x] 我已经仔细检查过FAQ

描述bug
apollo所在服务器时间与客户端所在服务器时间不一致时，apollo指定环境添加密匙开启，客户端按照添加的密匙配置以后不生效，会出现两种情况：
1.服务端开启密匙，客户端配置正确密匙，仍然拉取配置失败。
2.服务端开启密匙，客户端配置错误密匙，仍然可以拉取配置。
简洁明了地描述一下bug
复现
通过如下步骤可以复现：

apollo服务器把当前服务器时间调增加12小时，启动apollo服务，完成配置，开启密匙
另一台服务器客户端配置密匙出现拉取失败或者配置错误密匙仍然拉取成功

期望
配置并且开启密匙，客户端密匙未配置或者配置错误，应该拉取不到配置信息。
截图
如果可以，附上截图来描述你的问题
额外的细节和日志

版本：1.2.0
错误日志：
配置：
平台和操作系统：Linux

Could you provider the apollo server-side version and client-side version?

apollo 1.2.0
linux red hat 7.9

apollo server：1.2.0

apollo server 1.2.0

apollo server-side version 1.2.0 , client-side version 1.2.0/2.0.1

accesskey.secret会根据当前时间加密吗，目前三个环境dev uat sit，三个环境分别部署在不同的服务器上，三台机器除了时间其他均保持一致，其中dev 与 sit环境服务器时间与客户端都是保持一致的，没有出现密匙不生效的情况，只有uat所在服务器时间与客户端不一致，出现密匙不生效的情况

apollo server-side version 1.2.0 , client-side version 1.2.0/2.0.1

The AccessKey feature was released in version 1.6.0, are you sure your server-side version is 1.2.0? https://github.com/apolloconfig/apollo/releases/tag/v1.6.0

authors: nisiyong rjzhus zrj-coder

repo: apolloconfig/apollo


text:
chore(deps): update yarn to v4.2.1

This PR contains the following updates:

Package
Change
Age
Adoption
Passing
Confidence

yarn (source)
4.1.0 -> 4.2.1

[!WARNING]
Some dependencies could not be looked up. Check the Dependency Dashboard for more information.

Release Notes

yarnpkg/berry (yarn)
v4.2.1
Compare Source
v4.2.0
Compare Source
v4.1.1
Compare Source

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - "after 8am and before 4pm on tuesday" in timezone Etc/UTC.
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.

This PR has been generated by Renovate Bot.

⚠ Artifact update problem
Renovate failed to update an artifact related to this branch. You probably do not want to merge this PR as-is.
♻ Renovate will retry this branch, including artifacts, only when one of the following happens:

any of the package files in this branch needs updating, or
the branch becomes conflicted, or
you click the rebase/retry checkbox if found above, or
you rename this PR's title to start with "rebase!" to trigger it manually

The artifact failure details are included below:
File name: yarn.lock
➤ YN0000: · Yarn 4.2.1
➤ YN0000: ┌ Resolution step
➤ YN0085: │ + typescript@patch:typescript@npm%3A5.4.3#optional!builtin<compat/typescript>::version=5.4.3&hash=5adc0c, and 1 more.
➤ YN0085: │ - typescript@patch:typescript@npm%3A5.4.2#optional!builtin<compat/typescript>::version=5.4.2&hash=d69c25, and 1 more.
➤ YN0000: └ Completed in 0s 809ms
➤ YN0000: ┌ Post-resolution validation
➤ YN0002: │ @apollo/client-react-streaming@workspace:packages/client-react-streaming [a7905] doesn't provide react-dom (p73f32), requested by react-server-dom-webpack.
➤ YN0002: │ @apollo/client-react-streaming@workspace:packages/client-react-streaming [a7905] doesn't provide react-dom (p89c70), requested by @testing-library/react.
➤ YN0002: │ @apollo/client-react-streaming@workspace:packages/client-react-streaming [a7905] doesn't provide webpack (p92a18), requested by react-server-dom-webpack.
➤ YN0002: │ @apollo/client-react-streaming@workspace:packages/client-react-streaming [ff90e] doesn't provide react-dom (p07efe), requested by react-server-dom-webpack.
➤ YN0002: │ @apollo/client-react-streaming@workspace:packages/client-react-streaming [ff90e] doesn't provide react-dom (pfc924), requested by @testing-library/react.
➤ YN0002: │ @apollo/client-react-streaming@workspace:packages/client-react-streaming [ff90e] doesn't provide webpack (pb9870), requested by react-server-dom-webpack.
➤ YN0002: │ @apollo/client-react-streaming@workspace:packages/client-react-streaming doesn't provide react-dom (p04eb8), requested by @testing-library/react.
➤ YN0002: │ @apollo/client-react-streaming@workspace:packages/client-react-streaming doesn't provide react-dom (p1efc4), requested by react-server-dom-webpack.
➤ YN0002: │ @apollo/client-react-streaming@workspace:packages/client-react-streaming doesn't provide webpack (pd6b69), requested by react-server-dom-webpack.
➤ YN0002: │ @apollo/experimental-nextjs-app-support@workspace:packages/experimental-nextjs-app-support [57d1f] doesn't provide react-dom (p9d83e), requested by @testing-library/react.
➤ YN0002: │ @apollo/experimental-nextjs-app-support@workspace:packages/experimental-nextjs-app-support doesn't provide react-dom (p6ec33), requested by next.
➤ YN0002: │ @apollo/experimental-nextjs-app-support@workspace:packages/experimental-nextjs-app-support doesn't provide react-dom (pe703b), requested by @testing-library/react.
➤ YN0002: │ monorepo@workspace:. doesn't provide webpack (pa8ffb), requested by @size-limit/webpack-why.
➤ YN0086: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code.
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
➤ YN0013: │ 21 packages were added to the project (+ 20.02 MiB).
➤ YN0000: └ Completed in 13s 952ms
➤ YN0000: ┌ Link step

⚠ Artifact update problem
Renovate failed to update an artifact related to this branch. You probably do not want to merge this PR as-is.
♻ Renovate will retry this branch, including artifacts, only when one of the following happens:

any of the package files in this branch needs updating, or
the branch becomes conflicted, or
you click the rebase/retry checkbox if found above, or
you rename this PR's title to start with "rebase!" to trigger it manually

The artifact failure details are included below:
File name: yarn.lock
➤ YN0000: · Yarn 4.2.2
➤ YN0000: ┌ Resolution step
➤ YN0085: │ + typescript@patch:typescript@npm%3A5.4.3#optional!builtin<compat/typescript>::version=5.4.3&hash=5adc0c, and 1 more.
➤ YN0085: │ - typescript@patch:typescript@npm%3A5.4.2#optional!builtin<compat/typescript>::version=5.4.2&hash=d69c25, and 1 more.
➤ YN0000: └ Completed in 0s 711ms
➤ YN0000: ┌ Post-resolution validation
➤ YN0002: │ @apollo/client-react-streaming@workspace:packages/client-react-streaming [a7905] doesn't provide react-dom (p73f32), requested by react-server-dom-webpack.
➤ YN0002: │ @apollo/client-react-streaming@workspace:packages/client-react-streaming [a7905] doesn't provide react-dom (p89c70), requested by @testing-library/react.
➤ YN0002: │ @apollo/client-react-streaming@workspace:packages/client-react-streaming [a7905] doesn't provide webpack (p92a18), requested by react-server-dom-webpack.
➤ YN0002: │ @apollo/client-react-streaming@workspace:packages/client-react-streaming [ff90e] doesn't provide react-dom (p07efe), requested by react-server-dom-webpack.
➤ YN0002: │ @apollo/client-react-streaming@workspace:packages/client-react-streaming [ff90e] doesn't provide react-dom (pfc924), requested by @testing-library/react.
➤ YN0002: │ @apollo/client-react-streaming@workspace:packages/client-react-streaming [ff90e] doesn't provide webpack (pb9870), requested by react-server-dom-webpack.
➤ YN0002: │ @apollo/client-react-streaming@workspace:packages/client-react-streaming doesn't provide react-dom (p04eb8), requested by @testing-library/react.
➤ YN0002: │ @apollo/client-react-streaming@workspace:packages/client-react-streaming doesn't provide react-dom (p1efc4), requested by react-server-dom-webpack.
➤ YN0002: │ @apollo/client-react-streaming@workspace:packages/client-react-streaming doesn't provide webpack (pd6b69), requested by react-server-dom-webpack.
➤ YN0002: │ @apollo/experimental-nextjs-app-support@workspace:packages/experimental-nextjs-app-support [57d1f] doesn't provide react-dom (p9d83e), requested by @testing-library/react.
➤ YN0002: │ @apollo/experimental-nextjs-app-support@workspace:packages/experimental-nextjs-app-support doesn't provide react-dom (p6ec33), requested by next.
➤ YN0002: │ @apollo/experimental-nextjs-app-support@workspace:packages/experimental-nextjs-app-support doesn't provide react-dom (pe703b), requested by @testing-library/react.
➤ YN0002: │ monorepo@workspace:. doesn't provide webpack (pa8ffb), requested by @size-limit/webpack-why.
➤ YN0086: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code.
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
➤ YN0013: │ 3 packages were added to the project (+ 11.43 MiB).
➤ YN0000: └ Completed in 9s 954ms
➤ YN0000: ┌ Link step

repo: apollographql/apollo-client-nextjs


text:
Java-friendly APIs for MockWebServer

I imported these in my spring boot project.
testImplementation project(":testing:unit-testing")
testImplementation("com.apollographql.apollo3:apollo-mockserver:3.3.0")
api "com.apollographql.apollo:apollo-runtime"
For example, when I want to create a MockResponse, there are no builder or set methods. I can only call get methods for these objects. Or when I want to set URL for MockServer,  it gives this error
Required type: Continuation <? super java.lang.String>
Provided: String
I hope its not an obvious question. Am i importing something wrong? or I cannot those in Java anymore?
Thank you in advance!

Hi 👋  Indeed, using MockServer in Java is technically doable but it's a bit awkward. For some background there, we mostly designed MockServer for the Apollo integration tests and published them in hope that it could benefit the community but a lot of the APIs are pretty experimental. And as you found out, it's mainly designed for Kotlin.
If you really want to, you can get mockServer.url with this trick:
    /**
     * Because url doesn't suspend on the JVM, we can just use the return value
     */
    String url = (String) mockServer.url(new Continuation<String>() {
      @NotNull @Override public CoroutineContext getContext() {
        return EmptyCoroutineContext.INSTANCE;
      }

      @Override public void resumeWith(@NotNull Object o) {
      }
    });

Or write a little bit of Kotlin code to bridge the suspend function (see this stackoverflow answer).
All in all, for Java use cases, I would recommend using OkHttp MockWebServer. The main thing that the Apollo MockServer supports is multiplatform but if you're not using that, OkHttp MockWebServer should provide most of the API (and actually Apollo MockServer uses OkHttp MockWebServer under the hood).
Hope this helps!

@ibrhmyzc I just updated the doc to be more explicit about MockServer being mostly Kotlin so far.
I'll keep this is issue around to keep track of improvements to the MockServer APIs and renamed it accordingly, I hope you don't mind.

authors: ibrhmyzc martinbonnin

repo: apollographql/apollo-kotlin


text:
[gradle-plugin] Revive generateSourcesDuringGradleSync as DeprecationLevel.ERROR

Use case
Point users to the AS plugin and double check that the plugin works when a service fails to compile.
One example is when the release schema doesn't have all the debug schema fields yet.
KotlinLang thread
Describe the solution you'd like
No response

Turns out generateSourcesDuringGradleSync was enver removed. The default was changed to false instead, which seems to be working ok so far