text:
[SPARK-5722][SQL] fix for infer long type in python similar to Java long

Please review.

Can one of the admins verify this patch?

Do you mind adding [SQL] to the pull request title, after [SPARK-5722]?

Jenkins, test this please.

Test build #27275 has started for   PR 4521 at commit f53b94c.

This patch merges cleanly.

Test build #27275 has finished for   PR 4521 at commit f53b94c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27275/
Test FAILed.

@dondrake This looks good to me, just one comment, thanks!
In Python, we infer the schema only based on the first few rows, so it's possible to get IntegerType for large Python int after this patch. In this case, user could provide a correct schema to workaround it.

@rxin I updated the title of the pull request.
@davies In regards to inferSchema(), this is a PR for v1.2, I'm going to submit another PR for 1.3 that will use the new DataFrame().
In regards to the comment, where would you like to see that? I understand how inferSchema() works, all rows of my dataset contained had data that was in the edge case.

Test build #598 has started for   PR 4521 at commit f53b94c.

This patch merges cleanly.

@dondrake  After adding a comment, I think it's ready to go.

Test build #598 has finished for   PR 4521 at commit f53b94c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

OK, final changes applied.
Also, I created another PR for branch-1.3 https://github.com/apache/spark/pull/4538

Jenkins, retest this please.

Is there a reason to just not always use long (unless the user says int explicitly)?  The memory savings don't seem worth the chance of throwing confusing errors or truncating.  Especially since parquet / the inmemory cache use zigzag encoding.

Test build #27303 has started for   PR 4521 at commit a694093.

This patch merges cleanly.

Test build #27303 has finished for   PR 4521 at commit a694093.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27303/
Test FAILed.

My bad, a test was accidentally pasted as part of the branch-1.3 commit.  Now fixed, committed and pushed. please test again.

BTW I think Michael has a good point - always use longs. It is cheap to do that, and a lot more robust against inference errors. This doesn't matter much in Python since there isn't compile time type anyway.

@rxin, after thinking about it some more, I agree with @marmbrus .  Let me make another change so _type mappings looks like this:
_type_mappings = {
    type(None): NullType,
    bool: BooleanType,
    int: LongType,
    long: LongType,
    float: DoubleType,
    str: StringType,
    unicode: StringType,
    bytearray: BinaryType,
    decimal.Decimal: DecimalType,
    datetime.date: DateType,
    datetime.datetime: TimestampType,
    datetime.time: TimestampType,
}

I'll remove my if statement and add some comments around that as well.
This also reduces the risk of the inference seeing a small sample and choosing IntegerType.

Thanks for doing it.
@yhuai we should do the same thing for JSON type inference too. Unless we see a decimal, integer types should by default just use LongType.

Sounds good. #4544 is for that.

OK, all int's will be inferred as LongType.  Ready for testing.

@dondrake You could run the python sql tests locally, and see many tests are failing.
The size of int/long in Python is determined by size of machine word, see sys.maxint. In 32-bit machine, they should be Integer and BigInteger. In 64-bit machine, they should be Long and BigInteger.
From user's point view, I think Integer is enough for most of cases, the original patch could help in some cases (nanoseconds and uuid). In the cases that the inferred type is not correct, user could specify the correct type manually. I think this is the behavior without surprise to users.

Test build #599 has started for   PR 4521 at commit 41f363a.

This patch merges cleanly.

Test build #599 has finished for   PR 4521 at commit 41f363a.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

Fix for lint to pass now pushed.
I have tests passing for 1.2, 1.3 does not seem happy in general.  I can't get it to build locally.
Please test.

@dondrake you can build can test locally by:
$ build/sbt assembly
$ python/run_tests

running tests in 1.2 are fine.  1.3 branch won't pass locally tests due to some accumulator errors.  But, the python/pyspark/sql/tests.py that contains my unit tests does pass on its own.
Can we proceed?

Test build #601 has started for   PR 4521 at commit 0d2bcd2.

This patch merges cleanly.

@dondrake you can run any individual docs tests  by
bin/spark-submit python/pyspark/sql/dataframe.py

Test build #601 has finished for   PR 4521 at commit 0d2bcd2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Odd, this failed before it could run any python tests.  Any ideas?

look like a flaky test

test this please

Test build #27410 has started for   PR 4521 at commit 0d2bcd2.

This patch merges cleanly.

Test build #27410 has finished for   PR 4521 at commit 0d2bcd2.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27410/
Test FAILed.

Appears to be another false negative. Next steps?

test this please

Test build #27450 has started for   PR 4521 at commit 0d2bcd2.

This patch merges cleanly.

Test build #27450 has finished for   PR 4521 at commit 0d2bcd2.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27450/
Test FAILed.

This failure comes from my test, but it shouldn't fail when saving a Long with the exception can't convert Integer to Long.
File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/sql.py", line 1475, in pyspark.sql.SQLContext.parquetFile
Failed example:
    srdd.saveAsParquetFile(parquetFile)
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib64/python2.6/doctest.py", line 1253, in __run
        compileflags, 1) in test.globs
      File "<doctest pyspark.sql.SQLContext.parquetFile[4]>", line 1, in <module>
        srdd.saveAsParquetFile(parquetFile)
      File "/home/jenkins/workspace/SparkPullRequestBuilder/python/pyspark/sql.py", line 1906, in saveAsParquetFile
        self._jschema_rdd.saveAsParquetFile(path)
      File "/home/jenkins/workspace/SparkPullRequestBuilder/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__
        self.target_id, self.name)
      File "/home/jenkins/workspace/SparkPullRequestBuilder/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value
        format(target_id, '.', name), value)
    Py4JJavaError: An error occurred while calling o715.saveAsParquetFile.
    : org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 55.0 failed 1 times, most recent failure: Lost task 3.0 in stage 55.0 (TID 147, localhost): java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
    	at scala.runtime.BoxesRunTime.unboxToLong(BoxesRunTime.java:110)
    	at org.apache.spark.sql.catalyst.expressions.GenericRow.getLong(Row.scala:153)
    	at org.apache.spark.sql.parquet.MutableRowWriteSupport.consumeType(ParquetTableSupport.scala:350)
    	at org.apache.spark.sql.parquet.MutableRowWriteSupport.write(ParquetTableSupport.scala:328)
    	at org.apache.spark.sql.parquet.MutableRowWriteSupport.write(ParquetTableSupport.scala:314)
    	at

It appears to be casting (not converting) an Integer to a Long, which you can't do.  But, why does it think this is an Integer in the first place when it's defined as a LongType in Python and the spark Scala code??
I can confirm that I did see this in Spark 1.2.0 which motivated me to start this JIRA, and why I added this additional test case.

@davies can you take a look please?

The int in Python is inferred as LongType, but is not converted into long, then it's pickled and unpickled as Integer in JVM by Pyrolite. So you need to convert the int to long in Python, or convert the Integer to Long in JVM.

I'm struggling with how to handle this.  I would prefer that the saveAsParquet() would handle converting the value to a long for me.  However, I could update the test to store a long datatype, but again that means if I update a SchemaRDD long value in python, I have to guarantee it is a long.  Not very pythonic IMO.
Thoughts?

The parquet write path needs to assume the data matches the schema
otherwise we'll slow down all writing of data to parquet.  Instead I
suggest we check at the jvm side of the Python jvm boundry and convert ints
to longs.
On Feb 14, 2015 11:09 AM, "Don Drake" notifications@github.com wrote:

I'm struggling with how to handle this. I would prefer that the
saveAsParquet() would handle converting the value to a long for me.
However, I could update the test to store a long datatype, but again that
means if I update a SchemaRDD long value in python, I have to guarantee
it is a long. Not very pythonic IMO.
Thoughts?
—
Reply to this email directly or view it on GitHub
https://github.com/apache/spark/pull/4521#issuecomment-74387840.

OK, I'll keep looking, but can you let me know where that is in the code?
-Don

So, in ./sql/core/src/main/scala/org/apache/spark/sql/execution/pythonUdfs.scala
In EvaluatePython.toJava() I added the following:
    case (_, LongType) => {
        println("Converting to LongType" + obj);
        obj.asInstanceOf[Long]
    }

and in EvaluatePython.fromJava() I added the following:
    case (_, LongType) => {
        println("fromJava _ obj=" +obj);
        obj.asInstanceOf[Long]
    }

The fromJava() is never executed, but I do see the toJava() print output.
Unfortunately, this does not force the python value to a long.

@dondrake I just noticed it is for branch 1.2. Do we have a patch for the master?

@yhuai I have commits for branch-1.2 and branch-1.3 for this fix.  Is that not correct?
I am finished changes for branch-1.2 to resolve the test failing. Please advise where my changes for master/1.3 should reside.

@dondrake Can you also create a PR against our master? In our master, _type_mappings is in https://github.com/apache/spark/blob/master/python/pyspark/sql/types.py and your test can go in https://github.com/apache/spark/blob/master/python/pyspark/sql/tests.py.

So, I should not be updating the branch-1.3? Should I just create a branch off of master with my changes?
Are my changes for 1.2.x okay to go on branch-1.2?

In general you should open all PRs against master, and we will backport them manually when merging.  Please suggest this in the comments (or after the fact add the "backport needed" tag on JIRA).
If its a specialized fix that due to conflicts can only be made against a specific release branch, then its okay to make it against that branch.  For Spark SQL at this point master and branch-1.3 are functionally the same.  Regarding branch-1.2, I would not want to backport something that makes a change this large to a maintenance branch since it could break existing code.

OK, this PR, which is against branch-1.2 is now updated and I've verified that the tests are now passing.
I created another branch off of the master (named drake_python_long) that has the changes needed for v1.3.  I'll create another PR for that one.
Please test

ok to test

Test build #27648 has started for   PR 4521 at commit 69ce6d0.

This patch merges cleanly.

Test build #27648 has finished for   PR 4521 at commit 69ce6d0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27648/
Test PASSed.

The long in JVM will be converted into int in Python by Pyrolite, this change sounds reasonable.
long in Python is actually BigInteger, it's much slower than int, also it's slow to pickle/unpickle.
I think it's better to convert the Integer into Long in SQLContext.applySchemaToPythonRDD(), you could have a special case in needConversion() and EvaluatePython.fromJava(). We had do something for ByteType/Short, it's easy to add a case for (Int, LongType).

@dondrake Because we need to cut 1.3-RC1 now, I create another PR for master and 1.3, see #4666

Can this be closed @dondrake ? Looks like this was resolved in https://github.com/apache/spark/pull/4666 and https://github.com/apache/spark/pull/4681

Agreed, closed.  Thanks everyone.

authors: AmplabJenkins SparkQA davies dondrake marmbrus rxin srowen yhuai


text:
SPARK-5815 [MLLIB] Part 2. Deprecate SVDPlusPlus APIs that expose DoubleMatrix from JBLAS

Now, deprecated runSVDPlusPlus and update run, for 1.4.0 / master only

Test build #27555 has started for   PR 4625 at commit 6fd2ca5.

This patch merges cleanly.

Test build #27555 has finished for   PR 4625 at commit 6fd2ca5.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
case class ParquetRelation2(

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27555/
Test PASSed.

source: gharchive/pull-request

authors: AmplabJenkins SparkQA srowen

repo: apache/spark

license_type: permissive

license_source: github-api


text:
[SPARK-49808][SQL] Fix a deadlock in subquery execution due to lazy vals

What changes were proposed in this pull request?
1, Introduce a helper class Lazy to replace the lazy vals
2, Fix a deadlock in subquery execution
Why are the changes needed?
we observed a deadlock between QueryPlan.canonicalized and QueryPlan.references:
The main thread TakeOrderedAndProject.doExecute is trying to compute outputOrdering, it top-down traverse the tree, and requires the lock of QueryPlan.canonicalized in the path.
In this deadlock, it successfully obtained the lock of WholeStageCodegenExec and requires the lock of HashAggregateExec;
Concurrently, a subquery execution thread is performing code generation and bottom-up traverses the tree via def consume, which checks WholeStageCodegenExec.usedInputs and refererences a lazy val QueryPlan.references. It requires the lock of QueryPlan.references in the path.
In this deadlock, it successfully obtained the lock of HashAggregateExec and requires the lock of WholeStageCodegenExec;
This is due to Scala's lazy val internally calls this.synchronized on the instance that contains the val. This creates a potential for deadlocks.
Does this PR introduce any user-facing change?
no
How was this patch tested?
manually test:
before the fix, the deadlock happened twice in first 20 runs;
after the fix, the deadlock didn't happen in consecutive 100+ runs
Was this patch authored or co-authored using generative AI tooling?
no

branch 3.4 + scala 2.12.17

branch 3.4 + scala 2.13.8

@panbingkun thanks a lot for confirming the SerialVersionUID!

cc @JoshRosen would you mind taking another look? thanks

convert to draft for now to avoid merge by mistake

Decompiling with cfr-decompiler generates:
/*
 * Decompiled with CFR 0.152.
 *
 * Could not load the following classes:
 *  scala.Function0
 *  scala.reflect.ScalaSignature
 */
package org.apache.spark.util;

import java.io.Serializable;
import scala.Function0;
import scala.reflect.ScalaSignature;

@ScalaSignature(bytes="\u0006\u0005}2Q!\u0002\u0004\u0001\u00119A\u0001b\t\u0001\u0003\u0002\u0013\u0006I\u0001\n\u0005\u0006e\u0001!\ta\r\u0005\to\u0001A)\u0019)C\u0005q!)Q\b\u0001C\u0001}\tiAK]1og&,g\u000e\u001e'bufT!a\u0002\u0005\u0002\tU$\u0018\u000e\u001c\u0006\u0003\u0013)\tQa\u001d9be.T!a\u0003\u0007\u0002\r\u0005\u0004\u0018m\u00195f\u0015\u0005i\u0011aA8sOV\u0011q\"K\n\u0004\u0001A1\u0002CA\t\u0015\u001b\u0005\u0011\"\"A\n\u0002\u000bM\u001c\u0017\r\\1\n\u0005U\u0011\"AB!osJ+g\r\u0005\u0002\u0018A9\u0011\u0001D\b\b\u00033ui\u0011A\u0007\u0006\u00037q\ta\u0001\u0010:p_Rt4\u0001A\u0005\u0002'%\u0011qDE\u0001\ba\u0006\u001c7.Y4f\u0013\t\t#E\u0001\u0007TKJL\u0017\r\\5{C\ndWM\u0003\u0002 %\u0005Y\u0011N\\5uS\u0006d\u0017N_3s!\r\tReJ\u0005\u0003MI\u0011\u0001\u0002\u00102z]\u0006lWM\u0010\t\u0003Q%b\u0001\u0001B\u0003+\u0001\t\u00071FA\u0001U#\tas\u0006\u0005\u0002\u0012[%\u0011aF\u0005\u0002\b\u001d>$\b.\u001b8h!\t\t\u0002'\u0003\u00022%\t\u0019\u0011I\\=\u0002\rqJg.\u001b;?)\t!d\u0007E\u00026\u0001\u001dj\u0011A\u0002\u0005\u0007G\t!\t\u0019\u0001\u0013\u0002\u000bY\fG.^3\u0016\u0003\u001dB#a\u0001\u001e\u0011\u0005EY\u0014B\u0001\u001f\u0013\u0005%!(/\u00198tS\u0016tG/A\u0003baBd\u0017\u0010F\u0001(\u0001")
public class TransientLazy<T>
implements Serializable {
    private transient T value;
    private final Function0<T> initializer;
    private volatile transient boolean bitmap$trans$0;

    private T value$lzycompute() {
        TransientLazy transientLazy = this;
        synchronized (transientLazy) {
            if (!this.bitmap$trans$0) {
                this.value = this.initializer.apply();
                this.bitmap$trans$0 = true;
            }
        }
        return this.value;
    }

    private T value() {
        if (!this.bitmap$trans$0) {
            return this.value$lzycompute();
        }
        return this.value;
    }

    public T apply() {
        return this.value();
    }

    public TransientLazy(Function0<T> initializer) {
        this.initializer = initializer;
    }
}

based on the Decompiled code, I think the TransientLazy works as expected, would you mind taking another look?
@JoshRosen @cloud-fan

thanks, merging to master!

Since the code before change had @transient already, I don't think there is behavioral change in terms of caching exceptions.

authors: cloud-fan panbingkun tedyu zhengruifeng


text:
[SPARK-5932][CORE] Use consistent naming for size  properties

I've added an interface to JavaUtils to do byte conversion and added hooks within Utils.scala to handle conversion within Spark code (like for time strings). I've added matching tests for size conversion, and then updated all deprecated configs and documentation as per SPARK-5933.

Test build #30520 has started for   PR 5574 at commit 09ea450.

Also will need to update memoryStringToMb to use the new interface.

Test build #30520 has finished for   PR 5574 at commit 09ea450.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30520/
Test FAILed.

Test build #30523 has started for   PR 5574 at commit 851d691.

Test build #30523 has finished for   PR 5574 at commit 851d691.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30523/
Test PASSed.

Test build #30529 has started for   PR 5574 at commit 475370a.

Test build #30530 has started for   PR 5574 at commit 0cdff35.

Test build #30530 has finished for   PR 5574 at commit 0cdff35.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30530/
Test FAILed.

Test build #30532 has started for   PR 5574 at commit 7a6c847.

Test build #30532 has finished for   PR 5574 at commit 7a6c847.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30532/
Test FAILed.

Test build #30533 has started for   PR 5574 at commit 5d29f90.

retest this please

Test build #30533 has finished for   PR 5574 at commit 5d29f90.

This patch fails to build.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30533/
Test FAILed.

Test build #30534 has started for   PR 5574 at commit 928469e.

Test build #30529 has finished for   PR 5574 at commit 475370a.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30529/
Test PASSed.

Test build #30534 timed out     for PR 5574 at commit 928469e     after a configured wait of 150m.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30534/
Test FAILed.

Test build #30535 has started for   PR 5574 at commit 35a7fa7.

retest this please

Test build #30536 has started for   PR 5574 at commit 0f4443e.

retest this please

Test build #30539 has started for   PR 5574 at commit f15f209.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30537/
Test FAILed.

Test build #30535 timed out     for PR 5574 at commit 35a7fa7     after a configured wait of 150m.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30535/
Test FAILed.

Test build #30536 timed out     for PR 5574 at commit 0f4443e     after a configured wait of 150m.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30536/
Test FAILed.

Test build #30539 timed out     for PR 5574 at commit f15f209     after a configured wait of 150m.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30539/
Test FAILed.

retest this please

Test build #30549 has started for   PR 5574 at commit f32bc01.

Test build #30550 has started for   PR 5574 at commit f32bc01.

Test build #30549 has finished for   PR 5574 at commit f32bc01.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30549/
Test PASSed.

Test build #30550 has finished for   PR 5574 at commit f32bc01.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30550/
Test PASSed.

Test build #30631 has started for   PR 5574 at commit c7803cd.

Test build #30631 timed out     for PR 5574 at commit c7803cd     after a configured wait of 150m.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30631/
Test FAILed.

Test build #30677 has started for   PR 5574 at commit fe286b4.

Test build #30677 timed out     for PR 5574 at commit fe286b4     after a configured wait of 150m.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30677/
Test FAILed.

Test build #30683 has started for   PR 5574 at commit d3d09b6.

Test build #30683 timed out     for PR 5574 at commit d3d09b6     after a configured wait of 150m.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30683/
Test FAILed.

Test build #30694 has started for   PR 5574 at commit 84a2581.

Test build #30694 has finished for   PR 5574 at commit 84a2581.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30694/
Test FAILed.

Test build #30705 has started for   PR 5574 at commit 8b43748.

Test build #30705 has finished for   PR 5574 at commit 8b43748.

This patch fails MiMa tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30705/
Test FAILed.

retest this please

Test build #30706 has started for   PR 5574 at commit 8b43748.

Test build #30706 has finished for   PR 5574 at commit 8b43748.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30706/
Test PASSed.

Test build #30762 has started for   PR 5574 at commit e428049.

Test build #30762 has finished for   PR 5574 at commit e428049.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30762/
Test FAILed.

retest this please

Test build #30769 has started for   PR 5574 at commit e428049.

Test build #30769 has finished for   PR 5574 at commit e428049.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30769/
Test FAILed.

retest this please

Test build #30774 has started for   PR 5574 at commit e428049.

Test build #30783 has started for   PR 5574 at commit 3dfae96.

Test build #30774 has finished for   PR 5574 at commit e428049.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30774/
Test PASSed.

Test build #30783 has finished for   PR 5574 at commit 3dfae96.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30783/
Test PASSed.

Test build #30789 has started for   PR 5574 at commit 22413b1.

Test build #30789 has finished for   PR 5574 at commit 22413b1.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30789/
Test PASSed.

@vanzin Thanks for the continued feedback. I've revamped how I handle overflow and added test cases for various boundary conditions. I think this should address the prior concerns.

Test build #30878 has started for   PR 5574 at commit 852a407.

Test build #30878 has finished for   PR 5574 at commit 852a407.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30878/
Test FAILed.

Test build #30887 has started for   PR 5574 at commit fc85733.

Test build #30887 has finished for   PR 5574 at commit fc85733.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30887/
Test FAILed.

Test build #30888 has started for   PR 5574 at commit 2ab886b.

Test build #30888 has finished for   PR 5574 at commit 2ab886b.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30888/
Test FAILed.

Test build #30889 has started for   PR 5574 at commit 49a8720.

Test build #30889 has finished for   PR 5574 at commit 49a8720.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.
This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30889/
Test PASSed.

@andrewor14 what do you think about this in its present state? I think it should be getting near done.

Latest version looks good, thanks for addressing all of the feedback. Just a few nits remaining.

Test build #30946 has started for   PR 5574 at commit 11f6999.

retest this please

retest this please

Test build #31003 has started for   PR 5574 at commit 11f6999.

@andrewor14 How does this look?

Hi @ilganeli sorry for slipping on this. I've been busy with my own patches and I will look at this right now.

LGTM I have merged this into master thanks @vanzin @ilganeli @srowen

authors: AmplabJenkins SparkQA andrewor14 ilganeli vanzin


text:
[SPARK-11896] [SQL] Support Sample in Dataset APIs

Both RDD and DataFrame APIs have these two Sample APIs. Thus, I think they could be useful in DataSet APIs. Also added two test cases.

Test build #46463 has started for PR 9877 at commit 87b5742.

Test build #46463 has finished for PR 9877 at commit 87b5742.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46463/
Test PASSed.

Test build #46495 has started for PR 9877 at commit 7430e61.

Test build #46495 has finished for PR 9877 at commit 7430e61.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46495/
Test PASSed.

close this one and will merge it to https://github.com/apache/spark/pull/9921
Thank you!

authors: AmplabJenkins SparkQA gatorsmile


text:
STORM-2972: Replace storm-kafka with storm-kafka-client in storm-sql-…

…kafka
https://issues.apache.org/jira/browse/STORM-2972
The new kafka scheme syntax is stolen from Apache Camel, since I figure we are doing something very similar to them.
The primary limitation here compared to the storm-kafka version is that you can't include metadata (e.g. partition) in the emitted tuple, because I had to adapt the Scheme API to the RecordTranslator API used in storm-kafka-client. Emitting metadata with a Scheme in the storm-kafka spout is handled by special case instanceof'ing the scheme to see if it fits one that takes metadata, and I'd really prefer not to do that.
If we need to be able to emit metadata as well, we can just allow people to override the RecordTranslatorSchemeAdapter. I'll leave that as a task for later, unless someone objects.

I tried out storm-sql-kafka with a small example similar to the orders example at https://storm.apache.org/releases/2.0.0-SNAPSHOT/storm-sql.html. I adjusted the docs for the storm-sql-example page, but did not go through and run it.

repo: apache/storm


text:
STORM-1218: Use markdown for JavaDoc

JIRA: https://issues.apache.org/jira/browse/STORM-1218
This is a first pass at converting the JavaDoc in storm-core to markdown, using pegdown-doclet. If there is support from the community, I will move on to other components, though any html will just pass through, so it shouldn't garble, and will likely improve any existing javadoc.
The first commit adds the corresponding maven javadoc plugin to the default build. This means it will increase build time when you run mvn install, but it will also spit out warnings about javadoc problems and may encourage developers to fix or improve documentation problems.
The second commit is a quick pass over storm-core to remove unnecessary HTML and some minor javadoc comment linting. Only javadoc comments were modified.
The third commit adds the highlight.js license to both the source and binary LICENSE files, since the generated javadoc includes it. I'm not sure this is necessary, since the JavaDoc is generated, but I will find out if there's support for this patch.
Note: The pegdown-doclet licence is GPL, but it's only used at build time, so that does not seem to present a licensing issue.

Re: Merge conflicts, I'm happy to resolve therm at merge time.

I am +1 on the concept.  I don't see where highlight.js is added to the codebase.  It is automatically inserted into the created javadocs?

@revans2 Yes, it is inserted into the generated HTML. Since javadoc isn't part of the source release, it probably only needs to be added to the LICENSE for the binary distribution.

I removed the highlight.js license from the source release LICENSE file since generated javadoc is not part of a source release.

Note: This pull request originally included the following commit: https://github.com/apache/storm/commit/e03b28ca6222f61471b4acb1a6086aab9b597b8a
Earlier today I inadvertently pushed that commit to master. I have since reverted it in master. The last commit in this PR re-adds what was reverted, but only in my repository.

+1. Thanks this will make doc writing easier.

NB +1

Thanks @ptgoetz merged into master.

authors: harshach ptgoetz redsanket revans2


text:
Chart install extension deployment environment missing SP_COUCHDB_USER

Signed-off-by: huangyanfeng huangyanfeng1992@gmail.com

Purpose

Chart extension deployment environment missing SP_COUCHDB_USER
Remarks

PR introduces (a) breaking change(s): <yes/no>
no
PR introduces (a) deprecation(s): <yes/no>
no

Thanks @yanfeng1992 !

repo: apache/streampipes


text:
Proposed fix for WW-5028 for the 2.5.x branch:

Proposed fix for WW-5028 for the 2.5.x branch:

Disable printing stacktrace on exceptions by the Dispatcher by default.
Printing stacktrace on exception can be re-enabled by configuration.
Now prints stacktrace on exception using LOG, as suggested by A. Mashchenko and the the JIRA reporter.

Hello @lukaszlenart and @aleksandr-m .
This PR separates the fix for WW-5028 from PR#348 into its own PR.
It also addresses @aleksandr-m 's comment and moves the stacktrace print (when enabled) from stdout to the LOG instance for the dispatcher.
Let me know if additional changes are needed.

Coverage increased (+0.004%) to 47.001% when pulling 2faf986c27f467d15948de3aa53d0eb68a1a158a on JCgH4164838Gh792C124B5:localS2_25x_B9 into 02625740952b1e673402e21d52320fe41dbda500 on apache:struts-2-5-x.

A revised commit attempts to address the feedback - removed the separate flag, uses devMode only to control it, only uses debug level output for the stacktrace.
It uses the existing (handleException || devMode) block but still limits the stacktrace to devMode only (as a debug output).  If you'd prefer a change to DefaultDispatcherErrorHandler instead (as per Yasser's suggestion above) the PR can be updated to match that instead.

I'm good with the solution, thank you for all your work :)

repo: apache/struts


text:
Submarine can provide the function of notebook startup and shutdown

由于篇幅问题，考虑到英文可能不一定能表达清楚很多执行的细节，暂时以中文进行描述。
目前的 Submarine 每次创建一个新的 NoteBook 都会在 K8s 的CRD资源上创建一个新的实例。基于 notebook-controller 的服务发现的机制，会创建一个 Jupyter 的 Statefuleset 资源和 Service 资源。这是一个正常的处理流程，用户在 NoteBook 启动后可在 Submarine 上点击名称访问这个实例。
但是每一个实例在不同的使用者创建并使用后并不是一直在运行代码的，很多情况下在执行了一个idea或者验证了一些预想的情景和数据后，这个 NoteBook 实例就会被搁置。虽然用户稍后退出了 Submarine，但是这个实例仍然存在，这对于集群来讲就是一个被占用的无用的 Pod。
我目前的考虑是希望 Submarine 能提供一个针对 NoteBook 启动和停止的功能，这个功能会分为2个部分来实现。

在目前 NoteBook 功能的列表操作部分添加 启动/停止 按钮。用户在确认不使用该 NoteBook 时候可关闭 Pod，但是相关的 workspace 等资源不进行回收；再次启用后可继承原来的资源重启服务。
确认一种自动停止的机制，在用户指定时间不操作后，Submarine 自动停止该实例。在用户再次需要时可再启动 NoteBook。

针对 1 方案，可如描述中提到的添加 启动/停止 按钮，让用户自行控制。
针对 2 方案，目前已知的条件下，可基于 Jupyter 的 metric 指标利用 Promethues 进行采集，然后 Submarine 服务会轮询指标情况，针对指定时长（假定为30分钟）没有过执行的 Pod 做资源回收。我在自己的测试环境下针对既有的 Jupyter 服务做了如下的测试和方案验证：
Jupyter 容器基于 notebook-controller 做定制化资源注册，但是 Service 缺少 lables，暂无法使用 Promethues Operator 的 ServiceMonitor 做服务发现，故使用 PodMonitor做验证。在这里，我创建了一个名称为 test 的 NoteBook 实例。PodMonitor 的描述文件如下：
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: notebook-test
  namespace: submarine
  labels:
    k8s-app-pod: 'true'
spec:
  podMetricsEndpoints:
    - path: /notebook/submarine/test/metrics
      port: notebook-port
  namespaceSelector:
    matchNames:
      - submarine
  selector:
    matchLabels:
      statefulset: test

如图可以看到，采集的指标内，kernel_currently_running_total{type="python3"} 资源可以作为系统运行时的一个判断。
同时，我们也可以添加独立的 Service 做到相同的效果：
kind: Service
apiVersion: v1
metadata:
  name: notebook-test-metric
  namespace: submarine
  labels:
    statefulset: test
spec:
  ports:
    - name: http-test
      protocol: TCP
      port: 8888
      targetPort: 8888
  selector:
    statefulset: test
  type: ClusterIP

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: notebook-test
  namespace: submarine
  labels:
    k8s-app-service: 'true'
spec:
  endpoints:
    - path: /notebook/submarine/test/metrics
      port: http-test
  namespaceSelector:
    matchNames:
      - submarine
  selector:
    matchLabels:
      statefulset: test

基于服务发现后的 Promethues 指标如下如所示：

提供启停的 Rest 服务，具体处理逻辑：

在表 notebook 中新增字段，代表是否启动，默认为 1 启动。

ALTER TABLE submarine.notebook ADD started TINYINT(1) DEFAULT 1;

提供 启动/停止 的 Rest 接口。

关闭接口会将 started 设置为 0。同时，删除指定的 notebooks.kubeflow.org 资源，相关其他资源（例如pvc、configmap、ingress等）不删除。
启动接口会将 started 设置为 1。重新创建 notebooks.kubeflow.org 资源。

新增 NoteBook 的 Status：stopped

如何实现自动启停

配置相关参数 submarine.notebook.prometheus.enable。
是否可支持采集 NoteBook 的 Prometheus 指标。如果开启，在 submitter 为 k8s 的时候，会自动注册 PodMonitor。

配置相关参数 submarine.notebook.prometheus.labels。
配置扫描的标签，默认为空，可支持的描述方式为：k8s-app-pod:true,k8s-app-pod2:true，用逗号隔开

配置相关参数 submarine.server.tasks.autostop.enable。
是否支持 NoteBook 自动关闭，默认为关闭。该参数需要在 submarine.notebook.prometheus.enable 启动后才可实现此服务。

配置相关参数 submarine.server.tasks.autostop.duration。
自动任务扫描不活跃的持续事件，默认为分钟级别，默认为 30 分钟。该参数需要在 submarine.notebook.prometheus.enable 启动后才可实现此服务。

server 端新增可轮训的调度任务，基于分钟间隔进行调度任务。
按照 NoteBook 资源去扫描不活跃的事件，当超过当前不活跃的时间，系统调用关闭接口关闭该 NoteBook。

hi, @cdmikechen good job!
The program is very carefully designed.

source: gharchive/issue

authors: cdmikechen xunliu

repo: apache/submarine


text:
Allow to change number of rows in Time series table

By default time series table retrieve 5000 rows but it does not give the option of changing or show that you reach the limit (in other cases the number of rows are marked red)
I know that it would better to process less number of rows but the chart is misleading if people do not know the limitation

If I filter the rows clearly show the misleading

Expected results
At least a red mark to show that the data hit or Limit of 5K or the ability to change the count
Actual results
Show the chart with missing data in not a particular order so the chart is missliding
How to reproduce the bug
Produce a Timer series chart with more than 5K rows
Environment
0.38
Checklist
Make sure to follow these steps before submitting your issue - thank you!

[X ] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[X] I have reproduced the issue with at least the latest released version of superset.
[ ] I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context
Add any other context about the problem here.

@junlincc It was fixed https://github.com/apache/superset/pull/12887

@eugeniamz we added row limit to time series table. please let me know if this solution is good enough. thanks @maloun96

How can I remove this row limit altogether? Or at least change it? 50000 is insufficient for my histogram, performance is very good.

authors: GeorgePearse eugeniamz junlincc maloun96

repo: apache/superset


text:
How to embed view flow

I want to add a video stream chart to the dashboard for real-time monitoring by the webcam. What should I do?

Closing this since it's not a bug, but you might want to try to start up a conversation in an Ideas thread on Github Discussions, or start up a conversation in #customizing-superset on Superset Slack.
That said, I'm not sure what you mean by monitoring a webcam.... like... bandwidth data? Audio signal?
In any case, Superset isn't set up for realtime data all that well, though it can refresh periodically. The ECharts-based line chart supports a stream chart mode, so you would just need to have it connect to that data (whatever the source is) and refresh frequently.


text:
refactor(native-filters): refactor code of native filters

SUMMARY

No feature implemented
Refactor native filters code:
a. Split filter bar to different components
b. All code of native filters splitted to modal and filter bar

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

Codecov Report

Merging #12889 (0846f72) into master (2adfb85) will decrease coverage by 1.51%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #12889      +/-   ##
==========================================
- Coverage   64.89%   63.38%   -1.52%
==========================================
  Files        1024      489     -535
  Lines       50168    30198   -19970
  Branches     5148        0    -5148
==========================================
- Hits        32557    19140   -13417
+ Misses      17433    11058    -6375
+ Partials      178        0     -178

Flag
Coverage Δ

cypress
?

javascript
?

python
63.38% <ø> (-0.70%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

superset/sql_validators/postgres.py
50.00% <0.00%> (-50.00%)
:arrow_down:

superset/views/database/views.py
62.50% <0.00%> (-25.07%)
:arrow_down:

superset/databases/commands/create.py
83.67% <0.00%> (-8.17%)
:arrow_down:

superset/databases/commands/update.py
85.71% <0.00%> (-8.17%)
:arrow_down:

superset/sql_validators/base.py
93.33% <0.00%> (-6.67%)
:arrow_down:

superset/db_engine_specs/base.py
79.59% <0.00%> (-6.38%)
:arrow_down:

superset/db_engine_specs/presto.py
82.25% <0.00%> (-6.28%)
:arrow_down:

superset/db_engine_specs/sqlite.py
90.62% <0.00%> (-6.25%)
:arrow_down:

superset/views/database/forms.py
83.33% <0.00%> (-5.56%)
:arrow_down:

superset/databases/commands/test_connection.py
84.78% <0.00%> (-4.35%)
:arrow_down:

... and 523 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 38a21bf...0846f72. Read the comment docs.


text:
DO NOT MERGE: for testing 2.1 release only

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

/testenv up

/testenv up

/testenv up

/testenv up

/testenv up


text:
fix: update migration chart_ds_constraint to have seperate batch op

SUMMARY

Getting errors for constraint in same batch operation, so refactoring to commit before adding a new constraint
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

🏷 preset:2023.17

authors: hughhhh sadpandajoe


text:
Migrate dynamic scripted Grafana Dashboards to Scenes

Upgrade to Grafana v10.2.3
Migrate Scripted Dashboards to Grafana Scenes

I've transitioned our existing scripted dashboards to the new Grafana Scenes. The migration also ensures compatibility with future Grafana updates and aligns with Grafana's recommended best practices. Check it here: New in Grafana 10: Grafana Scenes for building dynamic dashboarding experiences

Which Traffic Control components are affected by this PR?

CDN in a Box

What is the best way to verify this PR?

Run CDN in a Box with Grafana
Go to Grafana, go to Apps -> TrafficControl Grafana Scenes
See the dashboards

Check the images below
If this is a bugfix, which Traffic Control versions contained the bug?

PR submission checklist

[x] This PR has tests
[x] This PR has documentation
[x] This PR has a CHANGELOG.md entry
[x] This PR DOES NOT FIX A SERIOUS SECURITY VULNERABILITY (see the Apache Software Foundation's security guidelines for details)

Cache Group dashboard

Delivery Service dashboard

Server dashboard

@jhg03a I also updated the Dockerfile of Grafana, I think you can give me some reviews

This all looks fine on GitHub. I'll carve out some time tomorrow to try to spin it up and see how it actually works (#5939 permitting).

Did you finish with checking?

Did you finish with checking?

Sorry, no, I got busy. And then... I forgot :sweat_smile:

repo: apache/trafficcontrol


text:
quic-latest doesn't work with the latest ngtcp2

I tested the current HEAD of quic-latest against ngtcp2.
I got the below error from ngtcp2.
**$  ./client 172.22.210.27 4433 -i
t=0.001255 TX Initial(0x7e) CID=0x69f845a86f57e64d PKN=2084699087 V=0xff000008
           STREAM(0x12) FIN=0 LEN=1 OFF=0
           stream_id=0x0 fin=0 offset=0 data_length=276
           PADDING(0x00)
           length=939
t=0.403256 TX Initial(0x7e) CID=0x69f845a86f57e64d PKN=2084699088 V=0xff000008
           STREAM(0x12) FIN=0 LEN=1 OFF=0
           stream_id=0x0 fin=0 offset=0 data_length=276
           PADDING(0x00)
           length=939
t=1.205089 TX Initial(0x7e) CID=0x69f845a86f57e64d PKN=2084699089 V=0xff000008
           STREAM(0x12) FIN=0 LEN=1 OFF=0
           stream_id=0x0 fin=0 offset=0 data_length=276
           PADDING(0x00)
           length=939
t=2.805744 TX Initial(0x7e) CID=0x69f845a86f57e64d PKN=2084699090 V=0xff000008
           STREAM(0x12) FIN=0 LEN=1 OFF=0
           stream_id=0x0 fin=0 offset=0 data_length=276
           PADDING(0x00)
           length=939
ngtcp2_conn_write_pkt: ERR_PKT_TIMEOUT
ngtcp2_conn_write_connection_close: ERR_INVALID_STATE
ngtcp2_conn_write_connection_close: ERR_INVALID_STATE
ngtcp2_conn_write_connection_close: ERR_INVALID_STATE
**

From ATS side,
[New Thread 0x7fff9d8e7700 (LWP 18179)]
[Dec  5 18:50:48.411] Server {0x7fff9ea3f700} DEBUG: <QUICPacketHandler.cc:123 (_recv_packet)> (quic_sec) [69f845a86f57e64d] received packet from 172.22.237.232:58808, size=1252
[Dec  5 18:50:48.425] Server {0x7fff9ea3f700} DEBUG: <QUICNetVConnection.cc:875 (_init_flow_control_params)> (quic_flow_ctrl) Connection [c5d7a688791a3646] [LOCAL] 0/2097152
[Dec  5 18:50:48.425] Server {0x7fff9ea3f700} DEBUG: <QUICNetVConnection.cc:878 (_init_flow_control_params)> (quic_flow_ctrl) Connection [c5d7a688791a3646] [REMOTE] 0/0
[Dec  5 18:50:48.425] Server {0x7fff9ea3f700} DEBUG: <QUICNextProtocolAccept.cc:53 (mainEvent)> (quic) [QUICNextProtocolAccept:mainEvent] event 202 netvc 0x7fff9cb19f50
[Dec  5 18:50:48.425] Server {0x7fffeccce700} DEBUG: <QUICNetVConnection.cc:1005 (_switch_to_handshake_state)> (quic_net) [c5d7a688791a3646] Enter state_handshake
[Switching to Thread 0x7fffeccce700 (LWP 18160)]

Breakpoint 1, QUICNetVConnection::state_handshake (this=0x7fff9cb19f50, event=2500, data=0xe01ce0) at QUICNetVConnection.cc:392
392	    QUICPacketUPtr packet = this->_dequeue_recv_packet(result);
(gdb) s
QUICNetVConnection::_dequeue_recv_packet (this=0x7fff9cb19f50, result=@0x7fffecccdb6c: SUCCESS) at QUICNetVConnection.cc:906
906	  QUICPacketUPtr quic_packet = QUICPacketUPtr(nullptr, &QUICPacketDeleter::delete_null_packet);
(gdb) n
907	  UDPPacket *udp_packet      = this->_packet_recv_queue.dequeue();
(gdb)
908	  if (!udp_packet) {
(gdb)
912	  net_activity(this, this_ethread());
(gdb)
915	  ats_unique_buf pkt = ats_unique_malloc(udp_packet->getPktLength());
(gdb)
916	  IOBufferBlock *b   = udp_packet->getIOBlockChain();
(gdb)
917	  size_t written     = 0;
(gdb)
918	  while (b) {
(gdb)
919	    memcpy(pkt.get() + written, b->buf(), b->read_avail());
(gdb)
920	    written += b->read_avail();
(gdb)
921	    b = b->next.get();
(gdb)
918	  while (b) {
(gdb)
923	  quic_packet = this->_packet_factory.create(std::move(pkt), written, this->largest_received_packet_number(), result);
(gdb) s
QUICNetVConnection::largest_received_packet_number (this=0x7fff9cb19f50) at QUICNetVConnection.cc:585
585	  return this->_largest_received_packet_number;
(gdb) s
586	}
(gdb)
std::move<std::unique_ptr<unsigned char, void (*)(void*)>&> (__t=std::unique_ptr<unsigned char> containing 0x7fffa80c8800 "\376i\370E\250oW\346M\377")
    at /usr/include/c++/4.8.2/bits/move.h:102
102	    { return static_cast<typename std::remove_reference<_Tp>::type&&>(__t); }
(gdb) fin
Run till exit from #0  std::move<std::unique_ptr<unsigned char, void (*)(void*)>&> (__t=std::unique_ptr<unsigned char> containing 0x7fffa80c8800 "\376i\370E\250oW\346M\377")
    at /usr/include/c++/4.8.2/bits/move.h:102
0x000000000082ec2f in QUICNetVConnection::_dequeue_recv_packet (this=0x7fff9cb19f50, result=@0x7fffecccdb6c: SUCCESS) at QUICNetVConnection.cc:923
923	  quic_packet = this->_packet_factory.create(std::move(pkt), written, this->largest_received_packet_number(), result);
Value returned is $1 = <unknown type in /opt/ts/bin/traffic_server, CU 0x11b1f15, DIE 0x11ee985>
(gdb) s
std::unique_ptr<unsigned char, void (*)(void*)>::unique_ptr(std::unique_ptr<unsigned char, void (*)(void*)>&&) (this=0x7fffecccdab0,
    __u=<unknown type in /opt/ts/bin/traffic_server, CU 0x11b1f15, DIE 0x11f04b2>) at /usr/include/c++/4.8.2/bits/unique_ptr.h:161
161	      : _M_t(__u.release(), std::forward<deleter_type>(__u.get_deleter())) { }
(gdb) fin
Run till exit from #0  std::unique_ptr<unsigned char, void (*)(void*)>::unique_ptr(std::unique_ptr<unsigned char, void (*)(void*)>&&) (this=0x7fffecccdab0,
    __u=<unknown type in /opt/ts/bin/traffic_server, CU 0x11b1f15, DIE 0x11f04b2>) at /usr/include/c++/4.8.2/bits/unique_ptr.h:161
0x000000000082ec41 in QUICNetVConnection::_dequeue_recv_packet (this=0x7fff9cb19f50, result=@0x7fffecccdb6c: SUCCESS) at QUICNetVConnection.cc:923
923	  quic_packet = this->_packet_factory.create(std::move(pkt), written, this->largest_received_packet_number(), result);
(gdb) s
QUICPacketFactory::create (this=0x7fff9cb1b618, buf=std::unique_ptr<unsigned char> containing 0x7fffa80c8800 "\376i\370E\250oW\346M\377", len=1252, base_packet_number=0,
    result=@0x7fffecccdb6c: SUCCESS) at QUICPacket.cc:625
625	  size_t max_plain_txt_len = 2048;
(gdb) l
620	// QUICPacketFactory
621	//
622	QUICPacketUPtr
623	QUICPacketFactory::create(ats_unique_buf buf, size_t len, QUICPacketNumber base_packet_number, QUICPacketCreationResult &result)
624	{
625	  size_t max_plain_txt_len = 2048;
626	  ats_unique_buf plain_txt = ats_unique_malloc(max_plain_txt_len);
627	  size_t plain_txt_len     = 0;
628
629	  QUICPacketHeader *header = QUICPacketHeader::load(buf.release(), len, base_packet_number);
(gdb) n
626	  ats_unique_buf plain_txt = ats_unique_malloc(max_plain_txt_len);
(gdb)
627	  size_t plain_txt_len     = 0;
(gdb)
629	  QUICPacketHeader *header = QUICPacketHeader::load(buf.release(), len, base_packet_number);
(gdb)
631	  switch (header->type()) {
(gdb)
662	    result = QUICPacketCreationResult::FAILED;
(gdb) p *header
$2 = {
  _vptr.QUICPacketHeader = 0x9498d0 <vtable for QUICPacketLongHeader+16>,
  _buf = 0x7fffa80c8800 "\376i\370E\250oW\346M\377",
  _buf_len = 1252,
  _serialized = '\000' <repeats 16 times>,
  _payload = std::unique_ptr<unsigned char> containing 0x0,
  _type = UNINITIALIZED,
  _key_phase = CLEARTEXT,
  _connection_id = {
    _id = 0
  },
  _packet_number = 0,
  _base_packet_number = 0,
  _version = 0,
  _payload_len = 0,
  _has_key_phase = false,
  _has_connection_id = false,
  _has_version = false
}

@maskit @masaori335
Is it expected? I thought we went through the handshake process with ngtcp2. Is it still true now?

ATS hasn't support draft-08 yet. It should work if you build ngtcp2 on its draft-07 branch.

I see. Thanks! Closing it.

authors: maskit zizhong

repo: apache/trafficserver


text:
fast_signal passed in but not used ProtectedQueue::enqueue(Event *e, bool fast_signal)

Seems fast_signal is deprecated. Better to remove it from the signature.

This was removed between 8.1.x and 9.0.x

@duke8253 did it by https://github.com/apache/trafficserver/pull/6104. Thanks!

authors: bryancall masaori335 zizhong


text:
TXN var config string set. Add nullptr check for non-specific variable string handling.

Without this we will try to store a nullptr which will cause a crash.

[approve ci autest]

[approve ci autest]


text:
TS-5031: Use readdir() instead of readdir_r()

See http://lwn.net/Articles/696474/.

Dupe of #1181 :)

FreeBSD build successful! See https://ci.trafficserver.apache.org/job/Github-FreeBSD/1160/ for details.

Linux build successful! See https://ci.trafficserver.apache.org/job/Github-Linux/1053/ for details.

authors: atsci jpeach zwoop


text:
Remove some autotools leftovers

Follow-up on 332fe3c75be6688f158fa4450eb07446edded7ee

The AuTest subcookie failed with differing gold file output. Probably caused by swapped order of logs.

[approve ci autest 2]

authors: JosiahWI raboof


text:
Remove unused variables

ssn_start_time and ssn_last_txn_time are unused.

Cherry-picked to v9.1.x branch.

authors: maskit zwoop


text:
[Bug] Crash when applying transform_layout with axis separator

When we apply transform_layout on a write buffer generated by cache_write with axis separators, we see a crash where the type_index for the BlockNode stmt in its corresponding sref gets corrupted. The below code reproduces the error:
Expected behavior
The code should work fine without errors and apply the layout transform with axis separators
Actual behavior
Code crashes with different errors like Unknown type index 2106063344 or TypeError: Expects StmtSRef 'block_sref' points to 'Block', but gets: runtime.Object
Environment
Tested on Upstream TVM main with Ubuntu 18.04
Steps to reproduce
import tvm
import tvm.contrib.hexagon
import tvm.testing
from tvm.script import tir as T

@T.prim_func
def add(
    p0: T.Buffer((T.int64(33), T.int64(128)), "float32"),
    p1: T.Buffer((T.int64(33), T.int64(128)), "float32"),
    T_add: T.Buffer((T.int64(33), T.int64(128)), "float32"),
):
    T.func_attr({"global_symbol": "main", "tir.noalias": True})
    # with T.block("root"):
    for ax0, ax1 in T.grid(T.int64(33), T.int64(128)):
        with T.block("T_add"):
            v_ax0, v_ax1 = T.axis.remap("SS", [ax0, ax1])
            T.reads(p0[v_ax0, v_ax1], p1[v_ax0, v_ax1])
            T.writes(T_add[v_ax0, v_ax1])
            T_add[v_ax0, v_ax1] = p0[v_ax0, v_ax1] + p1[v_ax0, v_ax1]

def transform_fn(x, y):
    return [x // 32, y, tvm.te.AXIS_SEPARATOR, x % 32]

def main():
    sch = tvm.tir.Schedule(add)
    block_rv = sch.get_block("T_add")
    sch.cache_write(block_rv, 0, "global")
    sch.transform_layout(block_rv, ("write", 0), transform_fn, pad_value=0.0)

if __name__ == "__main__":
    main()

Triage

needs-triage

A temporary fix to this error is in #13958

Closing in favor of an updated issue #13974

repo: apache/tvm


text:
Auto-scheduler tuned OpenCL kernel produces Invalid Workgroup Size Error

TVM Version: 0.8
OS: Ubuntu 20.04
Auto-scheduler tuned OpenCL kernel produces Invalid Workgroup Size Error.
Below is the script that I used to reproduce the issue. The script basically creates a tflite model, tunes it and run until an error occurs. It tunes 10 times and runs to check for errors.
I've been able to reproduce the issue in about 100 iterations or less.
import numpy as np
import tensorflow as tf
import tvm
from tvm import relay, auto_scheduler
from tensorflow.keras import Model, Input, layers, models, initializers
import tflite
from tvm.contrib import ndk
import tarfile
from tvm.driver.tvmc.runner import run_module
from tvm.auto_scheduler.utils import request_remote
import os
import time

###############################################################################
# RPC configuration
#
RPC_KEY = 'android'
RPC_PORT = 9190
RPC_TRACKER = '0.0.0.0'

###############################################################################
# Tune config
#
SKIP_TUNE = False
NUM_TRIALS = 20000
MIN_TRIALS = 10
EPOCH_CNT = NUM_TRIALS / MIN_TRIALS

def generate_tflite_file(N, H, W, CO, CI, KH, KW):
    def test_model(N, H, W, CO, CI, KH, KW):
        input = Input(shape=(CI, H, W), batch_size=N, name='input')
        x = layers.Conv2D(CO, (KH, KW),
                    kernel_initializer=initializers.RandomUniform(),
                    bias_initializer=initializers.RandomUniform(),
                    activation='relu')(input)
        x = layers.Conv2DTranspose(CO, (KH, KW),
                    kernel_initializer=initializers.RandomUniform(),
                    bias_initializer=initializers.RandomUniform(),
                    activation='relu')(x)
        x = layers.Conv2D(CO, (KH, KW),
                    kernel_initializer=initializers.RandomUniform(),
                    bias_initializer=initializers.RandomUniform(),
                    activation='relu')(x)
        output = layers.Conv2DTranspose(CO, (KH, KW),
                    kernel_initializer=initializers.RandomUniform(),
                    bias_initializer=initializers.RandomUniform(),
                    activation='relu')(x)
        model = Model(input, output)
        return model

    model = test_model(N,H,W,CO,CI,KH,KW)
    # save as tflite
    converter = tf.lite.TFLiteConverter.from_keras_model(model)
    tflite_model = converter.convert()
    params = '_'.join(str(x) for x in (N,H,W,CO,CI,KH,KW))
    tflite_file = f'invalid_ws_test_{params}.tflite'
    with open(tflite_file, 'wb') as f:
        f.write(tflite_model)

    return tflite_file

###############################################################################
# Generate a tflite file
#
tflite_file = generate_tflite_file(1,140,108,64,64,2,2)

###############################################################################
# tune and run until Invalid workgroup size error is generated
#
target_host = tvm.target.Target('llvm -mtriple=arm64-linux-android')
target = tvm.target.Target('opencl')

record_filename = f'{tflite_file}.records'

print('Request remote...')
remote = request_remote(RPC_KEY, RPC_TRACKER, RPC_PORT, timeout=3)
dev = remote.cl()

print(dev.max_clock_rate)
print(dev.max_shared_memory_per_block)
print(dev.max_thread_dimensions)
print(dev.max_threads_per_block)
print(dev.multi_processor_count)
print(dev.warp_size)

INT32_MAX = 2147483647
hardware_params = auto_scheduler.HardwareParams(
    dev.multi_processor_count,
    16,
    64,
    dev.max_shared_memory_per_block,
    INT32_MAX,
    dev.max_threads_per_block,
    int(dev.warp_size / 4) if int(dev.warp_size / 4) > 1 else dev.warp_size,
    dev.warp_size,
)

# wait for remote session to timeout
time.sleep(3)

epoch_cnt = 0
while epoch_cnt < EPOCH_CNT:
    print('===============================================================================')
    print(f'Starting Epoch #{epoch_cnt}')
    print('===============================================================================')

    tf_model_buf = open(tflite_file, "rb").read()
    tflite_model = tflite.Model.GetRootAsModel(tf_model_buf, 0)
    mod, params = relay.frontend.from_tflite(tflite_model, shape_dict=None, dtype_dict=None)

    if not SKIP_TUNE:
        tasks, task_weights = auto_scheduler.extract_tasks(
                            mod["main"], params, target,
                            target_host=target_host,
                            hardware_params=hardware_params)

        runner = auto_scheduler.RPCRunner(key=RPC_KEY, host=RPC_TRACKER, port=RPC_PORT, repeat=1, timeout=50, n_parallel=1)

        builder = auto_scheduler.LocalBuilder(build_func="ndk")

        tuner = auto_scheduler.TaskScheduler(tasks, task_weights, load_log_file=record_filename)
        tune_option = auto_scheduler.TuningOptions(
            num_measure_trials=MIN_TRIALS,
            builder=builder,
            runner=runner,
            measure_callbacks=[auto_scheduler.RecordToFile(record_filename)],
            verbose = 1
        )

        print("Tuning...")
        tuner.tune(tune_option)
        # save best records
        auto_scheduler.measure_record.distill_record_file(record_filename, record_filename+'.best')

    print("Compiling...")
    if os.path.isfile(record_filename):
        config = {'relay.backend.use_auto_scheduler':True}
        with auto_scheduler.ApplyHistoryBest(record_filename):
            with tvm.transform.PassContext(opt_level=3, config=config, disabled_pass=None):
                graph, lib, params = relay.build(mod, target=target, target_host=target_host, params=params)
    else:
        graph, lib, params = relay.build(mod, target=target, target_host=target_host, params=params)

    lib_file = tflite_file + '.so'
    graph_file = tflite_file + '.json'
    param_file = tflite_file + '.params'

    lib.export_library(lib_file, ndk.create_shared)

    with open(graph_file, 'w') as f:
        f.write(graph)
    with open(param_file, 'wb') as f:
        f.write(relay.save_param_dict(params))

    tvm_model_file = tflite_file + '.tar'
    with tarfile.open(tvm_model_file, 'w') as tar:
        tar.add(lib_file, arcname='mod.so')
        tar.add(param_file, arcname='mod.params')
        tar.add(graph_file, arcname='mod.json')

    # Run on target
    print(f'Running {tvm_model_file} on target device...')
    outputs, time = run_module(tvm_model_file, hostname=RPC_TRACKER, port=RPC_PORT, rpc_key=RPC_KEY,
        device='cl', inputs=None, fill_mode='random', repeat=1, profile=False)

    print(time)
    #print(outputs)
    epoch_cnt += 1

With little bit of more effort I was able to capture what auto-scheduler has ran and what the final kernel was.
Kernel compiled during tuning was:
__kernel void default_function_kernel0(__global float* restrict placeholder, __global float* restrict placeholder1, __global float* restrict compute) {
float compute_local[4];
__local float data_pad_shared[3408];
__local float placeholder_shared[1024];
for (int c_c_inner_init = 0; c_c_inner_init < 2; ++c_c_inner_init) {
compute_local[(c_c_inner_init)] = 0.000000e+00f;
compute_local[((c_c_inner_init + 2))] = 0.000000e+00f;
}
for (int dc_outer_outer = 0; dc_outer_outer < 4; ++dc_outer_outer) {
barrier(CLK_LOCAL_MEM_FENCE);
for (int ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer = 0; ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer < 7; ++ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer) {
if (((ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer * 560) + ((int)get_local_id(0))) < 3408) {
data_pad_shared[(((ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer * 560) + ((int)get_local_id(0))))] = (((((1 <= (((((int)get_group_id(0)) & 1) * 70) + (((ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer * 560) + ((int)get_local_id(0))) % 71))) && ((((((int)get_group_id(0)) & 1) * 70) + (((ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer * 560) + ((int)get_local_id(0))) % 71)) < 140)) && (1 <= ((((((int)get_group_id(0)) & 63) >> 1) * 2) + ((((ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer * 560) + ((int)get_local_id(0))) % 213) / 71)))) && (((((((int)get_group_id(0)) & 63) >> 1) * 2) + ((((ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer * 560) + ((int)get_local_id(0))) % 213) / 71)) < 64)) ? placeholder[((((((((dc_outer_outer * 140112) + ((((ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer * 560) + ((int)get_local_id(0))) / 213) * 8757)) + (((((int)get_group_id(0)) & 63) >> 1) * 278)) + (((((ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer * 560) + ((int)get_local_id(0))) % 213) / 71) * 139)) + ((((int)get_group_id(0)) & 1) * 70)) + (((ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer * 560) + ((int)get_local_id(0))) % 71)) - 140))] : 0.000000e+00f);
}
}
for (int ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer1 = 0; ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer1 < 2; ++ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer1) {
if (((ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer1 * 140) + (((int)get_local_id(0)) >> 2)) < 256) {
if (((ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer1 * 280) + (((int)get_local_id(0)) >> 1)) < 512) {
if (((ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer1 * 560) + ((int)get_local_id(0))) < 1024) {
placeholder_shared[(((ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer1 * 560) + ((int)get_local_id(0))))] = placeholder1[((((((dc_outer_outer * 4096) + ((((ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer1 * 140) + (((int)get_local_id(0)) >> 2)) >> 4) * 256)) + ((((int)get_group_id(0)) >> 6) * 64)) + ((((ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer1 * 140) + (((int)get_local_id(0)) >> 2)) & 15) * 4)) + (((int)get_local_id(0)) & 3)))];
}
}
}
}
barrier(CLK_LOCAL_MEM_FENCE);
for (int dc_outer_inner = 0; dc_outer_inner < 8; ++dc_outer_inner) {
for (int dh_outer_inner = 0; dh_outer_inner < 2; ++dh_outer_inner) {
for (int dc_inner = 0; dc_inner < 2; ++dc_inner) {
for (int dw_inner = 0; dw_inner < 2; ++dw_inner) {
for (int c_c_inner = 0; c_c_inner < 2; ++c_c_inner) {
compute_local[(c_c_inner)] = (compute_local[(c_c_inner)] + (data_pad_shared[(((((((dc_outer_inner * 426) + (dc_inner * 213)) + (((((int)get_local_id(0)) % 140) / 70) * 71)) + (dh_outer_inner * 71)) + dw_inner) + (((int)get_local_id(0)) % 70)))] * placeholder_shared[((((((((dc_outer_inner * 128) + (dc_inner * 64)) + ((((int)get_local_id(0)) / 140) * 8)) + (c_c_inner * 4)) + 3) - dw_inner) - (dh_outer_inner * 2)))]));
compute_local[((c_c_inner + 2))] = (compute_local[((c_c_inner + 2))] + (data_pad_shared[(((((((dc_outer_inner * 426) + (dc_inner * 213)) + (((((int)get_local_id(0)) % 140) / 70) * 71)) + (dh_outer_inner * 71)) + dw_inner) + (((int)get_local_id(0)) % 70)))] * placeholder_shared[((((((((dc_outer_inner * 128) + (dc_inner * 64)) + ((((int)get_local_id(0)) / 140) * 8)) + (c_c_inner * 4)) + 35) - dw_inner) - (dh_outer_inner * 2)))]));
}
}
}
}
}
}
for (int c_inner = 0; c_inner < 2; ++c_inner) {
compute[(((((((((((int)get_group_id(0)) >> 6) * 143360) + ((((int)get_local_id(0)) / 140) * 17920)) + (c_inner * 8960)) + (((((int)get_group_id(0)) & 63) >> 1) * 280)) + (((((int)get_local_id(0)) % 140) / 70) * 140)) + ((((int)get_group_id(0)) & 1) * 70)) + (((int)get_local_id(0)) % 70)))] = compute_local[(c_inner)];
compute[((((((((((((int)get_group_id(0)) >> 6) * 143360) + ((((int)get_local_id(0)) / 140) * 17920)) + (c_inner * 8960)) + (((((int)get_group_id(0)) & 63) >> 1) * 280)) + (((((int)get_local_id(0)) % 140) / 70) * 140)) + ((((int)get_group_id(0)) & 1) * 70)) + (((int)get_local_id(0)) % 70)) + 71680))] = compute_local[((c_inner + 2))];
}
}

Kernel compiled using the record written with above tuning results.
__kernel void fused_nn_conv2d_transpose_83_kernel0(__global float* restrict placeholder, __global float* restrict placeholder1, __global float* restrict compute) {

float compute_local[4];
__local float data_pad_shared[3408];
__local float placeholder_shared[1024];
for (int c_c_outer_inner_init = 0; c_c_outer_inner_init < 2; ++c_c_outer_inner_init) {
compute_local[(c_c_outer_inner_init)] = 0.000000e+00f;
compute_local[((c_c_outer_inner_init + 2))] = 0.000000e+00f;
}
for (int dc_outer_outer = 0; dc_outer_outer < 4; ++dc_outer_outer) {
barrier(CLK_LOCAL_MEM_FENCE);
for (int ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer = 0; ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer < 7; ++ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer) {
if (((ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer * 560) + ((int)get_local_id(0))) < 3408) {
data_pad_shared[(((ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer * 560) + ((int)get_local_id(0))))] = (((((1 <= (((((int)get_group_id(0)) & 1) * 70) + (((ax0_ax1_fused_ax2_fused_ax3_fused_outer_o
}
}
for (int ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer1 = 0; ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer1 < 2; ++ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer1) {
if (((ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer1 * 140) + (((int)get_local_id(0)) >> 2)) < 256) {
if (((ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer1 * 280) + (((int)get_local_id(0)) >> 1)) < 512) {
if (((ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer1 * 560) + ((int)get_local_id(0))) < 1024) {
placeholder_shared[(((ax0_ax1_fused_ax2_fused_ax3_fused_outer_outer1 * 560) + ((int)get_local_id(0))))] = placeholder1[((((((dc_outer_outer * 4096) + ((((ax0_ax1_fused_ax2_fused_ax3_fused_out
}
}
}
}
barrier(CLK_LOCAL_MEM_FENCE);
for (int dc_outer_inner = 0; dc_outer_inner < 8; ++dc_outer_inner) {
for (int dh_outer_inner = 0; dh_outer_inner < 2; ++dh_outer_inner) {
for (int c_c_outer_inner = 0; c_c_outer_inner < 2; ++c_c_outer_inner) {
for (int dc_inner = 0; dc_inner < 2; ++dc_inner) {
for (int dw_inner = 0; dw_inner < 2; ++dw_inner) {
compute_local[(c_c_outer_inner)] = (compute_local[(c_c_outer_inner)] + (data_pad_shared[(((((((dc_outer_inner * 426) + (dc_inner * 213)) + (((((int)get_local_id(0)) % 140) / 70) * 71)) + (d
compute_local[((c_c_outer_inner + 2))] = (compute_local[((c_c_outer_inner + 2))] + (data_pad_shared[(((((((dc_outer_inner * 426) + (dc_inner * 213)) + (((((int)get_local_id(0)) % 140) / 70)
}
}
}
}
}
}
for (int c_inner = 0; c_inner < 2; ++c_inner) {
compute[(((((((((((int)get_group_id(0)) >> 6) * 143360) + ((((int)get_local_id(0)) / 140) * 17920)) + (c_inner * 8960)) + (((((int)get_group_id(0)) & 63) >> 1) * 280)) + (((((int)get_local_id(0)) % 1
compute[((((((((((((int)get_group_id(0)) >> 6) * 143360) + ((((int)get_local_id(0)) / 140) * 17920)) + (c_inner * 8960)) + (((((int)get_group_id(0)) & 63) >> 1) * 280)) + (((((int)get_local_id(0)) %
}
}

Thanks for asking the question, please open a new thread on https://discuss.tvm.apache.org/ as we use the forum for related discussions. For this case, could you if check hardware params are correct?


text:
[Hexagon] capture gtest output and return over FFI

Captures Hexagon gtest output in a string and passes back over the FFI.  Captured output mimics standard gtest output:
[==========] Running 27 test(s) from 1 test suite(s).
[----------] 27 test(s) from HexagonBuffer
[ RUN      ] HexagonBuffer.default_scope
[       OK ] HexagonBuffer.default_scope (0 ms)
[ RUN      ] HexagonBuffer.ddr_scope
[       OK ] HexagonBuffer.ddr_scope (0 ms)
[ RUN      ] HexagonBuffer.vtcm_scope
[       OK ] HexagonBuffer.vtcm_scope (0 ms)
[ RUN      ] HexagonBuffer.invalid_scope
[       OK ] HexagonBuffer.invalid_scope (0 ms)
[ RUN      ] HexagonBuffer.micro_copies_corresponding_regions
[       OK ] HexagonBuffer.micro_copies_corresponding_regions (0 ms)
[ RUN      ] HexagonBuffer.micro_copies_src_bigger
[       OK ] HexagonBuffer.micro_copies_src_bigger (0 ms)
[ RUN      ] HexagonBuffer.micro_copies_dest_bigger
[       OK ] HexagonBuffer.micro_copies_dest_bigger (0 ms)
[ RUN      ] HexagonBuffer.micro_copies_src_overlaps_dest_region
[       OK ] HexagonBuffer.micro_copies_src_overlaps_dest_region (0 ms)
[ RUN      ] HexagonBuffer.micro_copies_dest_overlaps_src_region
[       OK ] HexagonBuffer.micro_copies_dest_overlaps_src_region (0 ms)
[ RUN      ] HexagonBuffer.micro_copies_discontiguous_regions
[       OK ] HexagonBuffer.micro_copies_discontiguous_regions (0 ms)
[ RUN      ] HexagonBuffer.micro_copies_invalid_size
[       OK ] HexagonBuffer.micro_copies_invalid_size (0 ms)
[ RUN      ] HexagonBuffer.macro_copies_adjacent_corresponding_regions_merged
[       OK ] HexagonBuffer.macro_copies_adjacent_corresponding_regions_merged (0 ms)
[ RUN      ] HexagonBuffer.macro_copies_discontiguous_regions_not_merged
[       OK ] HexagonBuffer.macro_copies_discontiguous_regions_not_merged (0 ms)
[ RUN      ] HexagonBuffer.macro_copies_overlapping_regions_merged
[       OK ] HexagonBuffer.macro_copies_overlapping_regions_merged (0 ms)
[ RUN      ] HexagonBuffer.copy_from
[       OK ] HexagonBuffer.copy_from (0 ms)
[ RUN      ] HexagonBuffer.copy_from_invalid_size
[       OK ] HexagonBuffer.copy_from_invalid_size (0 ms)
[ RUN      ] HexagonBuffer.copy_from_smaller_size
[       OK ] HexagonBuffer.copy_from_smaller_size (0 ms)
[ RUN      ] HexagonBuffer.nd
[       OK ] HexagonBuffer.nd (0 ms)
[ RUN      ] HexagonBuffer.nd_copy_from
[       OK ] HexagonBuffer.nd_copy_from (0 ms)
[ RUN      ] HexagonBuffer.1d_copy_from_1d
[       OK ] HexagonBuffer.1d_copy_from_1d (0 ms)
[ RUN      ] HexagonBuffer.2d_copy_from_1d
[       OK ] HexagonBuffer.2d_copy_from_1d (0 ms)
[ RUN      ] HexagonBuffer.1d_copy_from_2d
[       OK ] HexagonBuffer.1d_copy_from_2d (0 ms)
[ RUN      ] HexagonBuffer.nd_copy_from_nd_invalid_size
[       OK ] HexagonBuffer.nd_copy_from_nd_invalid_size (0 ms)
[ RUN      ] HexagonBuffer.nd_copy_from_nd_smaller_size
[       OK ] HexagonBuffer.nd_copy_from_nd_smaller_size (0 ms)
[ RUN      ] HexagonBuffer.md_copy_from_nd
[       OK ] HexagonBuffer.md_copy_from_nd (0 ms)
[ RUN      ] HexagonBuffer.copy_to
[       OK ] HexagonBuffer.copy_to (0 ms)
[ RUN      ] HexagonBuffer.nd_copy_to
[       OK ] HexagonBuffer.nd_copy_to (0 ms)
[----------] 27 test(s) from HexagonBuffer (0 ms total)
[==========] 27 test(s) from 1 test suite(s) ran. (0 ms total)
[  PASSED  ] 27 test(s)

cc @mehrdadh

@adstraw how come this doesn't fail since we disabled gtest on hexagon?
https://github.com/apache/tvm/blob/main/tests/scripts/task_build_hexagon_api.sh

@mehrdadh This only works on the simulator. I will create a separate PR to fix #11237 and re-enable hardware tests

This PR is now failing in CI due to name change of the top-level python test from unit_test.py to test_hexagon_unit_tests.py.  This PR is not blocked behind the fix for #11237 which @supersat will fix.

Thought of a better solution that would allow us to land this PR before fixing #11237.  Latest commit skips the unit tests if $USE_HEXAGON_GTEST is not set in the environment.  This should allow CI to pass because we aren't currently setting that flag so test will be skipped.

authors: adstraw mehrdadh supersat


text:
Restore "pytest.mark.gpu" for RELAX tests

Unfortunately, relax GPU tests are excluded from CI right now.
The testing script uses flag "-m gpu" to run gpu tests, but actual tests have no such mark.

Hi @Lunderberg thank you for reviewing this PR.

The tvm.testing.requires_* marks all provide the pytest.mark.gpu flag, but they aren't yet used in all cases.

Yes I know this issue. And I absolutely agree that this PR contains logically incorrect test marks. Some tests are marked like "require_XXX" but in fact they don't. But this incorrectness was introduced not by me. A lot of test from files touched by me verify pattern matching functionality. Pattern matching is pure python code and doesn't require any cmake enabled features like specific code generator or runtime support. But they still has marker like "requires_tensorrt_codegen". So I changed one incorrect marker by another.
The reason of that was in a need to enable tests in CI. I saw 4 way to do that:

To make all test mark correct. Produce one more Jenkins job with enabled compilation with cuda/cublas/cutlass/tensorrt on virtual machine without GPU. And launch there only GPU specific unit tests which doesn't required real GPU. The main question how to filter tests which should be launched there.

Same as first but instead of creation new one Jenkins job integrate all this stuff into "cpu/pr-head". Have to change build script to add all coda dependencies. It introduces incorrectness of meaning, CI job with "cpu" in name builds "gpu" specific code.

Same as second but use "gpu/pr-head". Same problem as for first way. Have to change launch script and add all specific marker filter like "-m "gpu or cudagraph or tensorcore or cutlass or ***". I guess it's impossible to keep this command line correct. The new one tests and "tvm.testing.Features" will be messed in this list.

Just keep situation as is. And use "gpu" marker as a root marker for all type of tests we would like to launch in CI job "gpu/pr-head".

I selected the fourth way as easiest. And I realise that it's imperfect solution. If you have any other ideas how to enable all test in CI and keep testing code correct please point that.

Thank you on the explanation of the different options.  I agree that changing all markers in all locations is far too heavy of a lift for any one PR, and should be avoided.
For long-term solutions, I think it would make sense to have the "cpu" and "gpu" indicate which environment should be used, and not which functionality is being tested.  That way, we could distinguish between a test that compiles TensorRT (doesn't require a GPU) and a test that executes TensorRT (does require a GPU).  The former would run in the cpu/pr-head step, even though it is validating functionality required for GPUs, and the latter would run in the gpu/pr-head step.  However, this will require adding the compile-time support to the CI's CPU container, which is a larger step overall.

(Aside from all my pontificating, I think this PR is ready to merge once the CI passes.)
I took a quick glance at the failing cases, and it looks like this line should be changed from def main(x: R.Tensor): to def main(x: R.Tensor([4], "int64")):.

The test_no_op_for_call_to_tir was added, and passed in local testing.  Due to the markers, it wasn't actually tested in CI.
Recent PR added well-formed checking in TVMScript.
Recent PR added StructInfo for PrimFuncs, allowing a discrepancy between "R.Tensor" and "T.Buffer(T.int64(4), "int64")` to be identified.
This PR fixed the marks, allowing the test to run in CI.

So the test worked when it was added, should have been updated in subsequent PRs, but due to the missing pytest marks, was never noticed until this PR.

Hi, team!
Are we ready to merge this PR? It will be great to restore testing of Relax GPU tests in CI!

@Lunderberg could you please merge this one if you have no objections.

Looks like it is still making its way through CI, but I can merge it after the CI passes.

authors: Lunderberg apeskov ibsidorenko


text:
add zeppelin.livy.share_existing_session option

If you want to share livy session between multiple zeppelin user, There is no option now.
After this commit, setting zeppelin.livy.share_existing_session to true
make LivyInterpreter use an existing livy session
(first session returned by /sessions)
What is this PR for?
add zeppelin.livy.share_existing_session option.
zeppelin.livy.share_existing_session is true, list exsiting livy session and select first session.
by doing so, we can share livy session between zeppelin livy interpreters.
What type of PR is it?
Feature
Todos
What is the Jira issue?
How should this be tested?

setup livy server
run zeppelin server
setup zeppelin livy interpreter(change zeppelin.livy.share_existing_session to true)
make new note
make another note
audit logs/zeppelin-interpreter-*.log

Screenshots (if appropriate)
Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? Yes

@jerryshao may I ask you to review this PR?

@ruseel @swkimme livy session is shared across notes, What kind of scenarios here do you mean ?

I am trying to use livy as shown in https://youtu.be/gtql7z_kj3Y Zeppelin and livy bring multi tenancy to interactive data analysis 20:02.
Share livy session between multiple user.

What I want to accomplish eventually is providing a single spark cluster to four people and interactive query they issued  is run under a single pool, with a fair scheduler.

livy session is shared across notes, What kind of scenarios here do you mean?

I guessed that "Livy session is not shared across notes in zeppelin:master/BaseLivyIntepreter.java" after inspecting source code and exploratory session with my laptop setup. If I opened two notes in zeppelin, LivyServer shows there is two session in their web UI.
Did I missed something? Is it true livy session is shared across notes. Or Did @zjffdu wanted to know about rationale of PR. :)

@ruseel Livy session is shared across notes, as long as the notes belong to one user. Livy interpreter is designed for isolated across users. That means each user own one livy session, and he can share this session across his notes.  The multi-tenancy means users share the same hadoop cluster.

Right, one major reason is security. Sharing SparkContext across users may introduce potential security issue.

well, I think that is the "design" of the open source implementation ;)

A goal of PR is sharing already created livy session, now I see 'there is a concern about security'.
so I am closing this PR.

authors: felixcheung ruseel swkimme zjffdu

repo: apache/zeppelin

license_source: bigquery


text:
[ZEPPELIN-4646]. zeppelin-server doesn't need scala

What is this PR for?
Trivial change which just remove scala dependency from zeppelin-server because it doesn't need these.
What type of PR is it?
[ Improvement ]
Todos

[ ] - Task

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-4646

How should this be tested?

CI pass

Screenshots (if appropriate)
Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

CI pass, will merge if no more comment

LGTM

authors: Leemoonsoo zjffdu


text:
[ZEPPELIN-5510]There are some logic problems, which may lead to thread leaks.

[ZEPPELIN-5510] There are some logic problems, which may lead to thread leaks
What is this PR for?
Fix code issues.
What type of PR is it?
[Improvement]
What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-5510

Thanks for the contribution @NicolasGong , could you update the title to be more specific ? e.g. Thread leakage in JobManager of flink interpreter, and please check the CI, seems flink CI is failed

ping @NicolasGong

ping @NicolasGong

ping @NicolasGong

ping @NicolasGong

Thanks @NicolasGong Could you do rebase ? Recently we fixed flaky CI test.

Ok, I'll try

Sorry, I updated the branch code, but CI still failed. I don't know why. This problem should not be caused by the changed code.

Sorry, I rebase the code and fixed a problem. But I don't know the reason why other CI failed.

CI is passed now, thanks @NicolasGong Sometimes some tests will still fail due to network or other unknown reason, usually rerun will turn CI to green

Will merge if no more comment

authors: NicolasGong zjffdu


text:
🛑 InterHop (Website) is down

In 1c3ead5, InterHop (Website) (https://interhop.org) was down:

HTTP code: 0
Response time: 0 ms

Resolved: InterHop (Website) is back up in d6f0898 after 5 minutes.

license: MIT


text:
Auth mechanism changed?

Made it working within Home Assistant a week ago, but suddenly it started throwing these errors from yesterday:
/node_modules/southern-company-api/dist/main.js:166
                    throw new Error(`Failed to get secondary ScWebToken: Could not find any token matches in headers`);

I think something has changed within auth model: either JWT acquiring mechanism or the way that cookies are used for it

Yep - I'm working on a python implementation of this api: https://github.com/Lash-L/southern_company_api
Sometime last night/ this morning they changed it.
Changing ScWebToken to SouthernJwtCookie will resolve the problem.

@Lash-L @andreifinski, 2.2.2 is now published with the changed auth

Let me know if the fix worked or didn't work

it did, thanks!

authors: Lash-L andreifinski apearson

repo: apearson/southern-company-api


text:
Are elements just hidden or actually not part of the DOM?

Hey, great idea and thanks for taking the time to create it!
I'm wondering, are the hidden elements part of the DOM or truly hidden as if a v-if had been used? I'm wanting to avoid loading assets for mobile that are desktop only and vice-versa.

They are not part of the DOM. They only render if the breakpoint or condition is met
https://github.com/apertureless/vue-breakpoints/blob/master/src/components/showAt.vue#L46-L49

I had the same question, wondering if it's more like a v-show or a v-if.
I suggest you document this distinction in the readme. It makes a difference to me. Also, an example of a rendered DOM fragment for each situation would be helpful.
Anyway, thanks for providing this useful tool!

authors: EvanAgee apertureless suterma

repo: apertureless/vue-breakpoints

license: mit


text:
Loading using Require.js (AMD) broken

Codepen
Just open this codepen and check the dev-console (F12):

Explanation
I think that SVG is not defined because there is something wrong on module bundler.
Do i need to include SVG.js as external library?

The new module bundler configuration (rollup) has broken AMD support after v3.0.0
I will soon update the SVG version which should fix the SVG undefined issues.

In the meantime I can add an AMD build output to the Rollup configuration see if it works.

Looks like its not as plug and play as I thought, I'll investigate more when I have time.

@cstlaurent Thanks!
The main issue I guess is SVG.js current way of exposing the global SVG variable, and this way has caused a lot of problems with the modern build systems (many users have reported it in the past).
They have released 3.0 which promises to fix these issues, and I am in the process of slowly migrating towards 3.0 (many breaking changes, hence getting delayed).

any news  here? here we cannot update the library without this fix.

Hi I am facing the same iisue.
ReferenceError: SVG is not defined
apexcharts.esm.js:24146
at c:\Dev*Project Path*\node_modules\apexcharts\dist\apexcharts.esm.js:24146:1.
does anyone has the solution for this or any workaround?
I am using react-apexcharts in my project .

@Micene09 Unfortunately, no update. I have contacted SVG.js authors regarding plugin compatibility with 3.0. Once, I get a solution, I will upgrade to svgjs@3.0 which hopefully will fix a lot of import issues.

Adding the old Webpack configuration only for amd build as that seems to be the easiest fix for now.
The new file will be
dist/apexcharts.amd.js
Will be released in 3.7.0 this weekend

Can you please add the min version of this file? It is about 1mb.

This "workaround" don't work for me and yes the script is loades before i make my "new ApexCharts":
"Uncaught ReferenceError: ApexCharts is not defined at :1:1"

@junedchhipa, the problem about loading using require is not working. If in the require I put apexcharts.js, the error is "Uncaught ReferenceError: SVG is not defined". If I put apexcharts.amd.js, the error is "ReferenceError: ApexCharts is not defined". Thank's

@Micene09
I have already put the umdNamedDefine in the Webpack configuration.
Also minified.
Are you also facing the same issue in the latest update?
@everyone else, can you please try downgrading and see if the previous versions work? 3.8.0 ?

I stumbled across the same issues, here's my workaround:
require.config({
  baseUrl: ".",
  paths: {
    "ApexCharts": "https://cdn.jsdelivr.net/npm/apexcharts@3.10.1/dist/apexcharts",
    "svg": "https://cdn.jsdelivr.net/npm/svgjs@2.6.2/dist/svg.min"
  }
});

require(['svg', 'ApexCharts'], function (svg, ApexCharts) {
  var myChartOpts = {...};
  var myChart = new ApexCharts(document.querySelector("#myChart"), myChartOpts);
  myChart.render();
});

Hope this helps!

svg.js is already bundled in the apexcharts file, so loading it again may cause issues.
I will shortly look into the Webpack configuration to see what went wrong.

I have changed the Webpack config again and built a umd instead of amd.
The samples directory also includes a requirejs demo.
The next release v3.12.0 along with this change will be released soon.

How is the progress with upgrading to version 4?
I have the same issue "SVG is not defined" in Angular when importing NgApexchartsModule. This works fine in regular Angular build, but does not work in AppShell build when building serverside in NodeJS environment.

Same problem with NgApexchartsModule

@junedchhipa
how can I overcome this problem

@ihordeyneka the solution : https://github.com/apexcharts/ng-apexcharts/issues/78

How is the progress with upgrading to version 4?
I have the same issue "SVG is not defined" in Angular when importing NgApexchartsModule. This works fine in regular Angular build, but does not work in AppShell build when building serverside in NodeJS environment.

Yes it doesn't work in ssr

I'm still not seeing the resolution of this issue on my end. Perhaps it is my introductory knowledge of requireJS... I have a question up on StackOverflow which describes my current setup... Any brave souls willing to help me resolve it?
Having trouble loading ApexCharts using RequireJS

Is this issue fixed ? I am also facing this issue when I am trying to use ApexCharts from Jupyter.
Thanks

authors: Micene09 Teisi byander cjholowatyj cstlaurent epykure ihordeyneka junedchhipa juniorschen shadowintel thetinshusasi wicked539


text:
Example code gives json request parse error

As the old voice_request is marked deprecated, I'm using a text_request to send audio data as instructed here, using a wav file (Unsigned 8 bit, or Signed 16 bit Little Endian, Rate 16000 Hz). However, this gives me the following result:
{
  "error": {
    "code": "002",
    "errorId": "ab6473fe-a57a-49ed-8a08-21a58570be78",
    "message": "json request parse error"
  }
}

What am I doing wrong?

Just forgot about voice request. It won't work for you.

The README says it is possible to do speech recognition through this python api. How do I do that then?

@rokusottervanger No way. This api has been removed.

Apparently, using Google Cloud Speech API is the way to go now. Maybe this should be updated in the documentation...

authors: rokusottervanger sstepashka

repo: api-ai/api-ai-python


text:
Implement/document a way how to pass extra configuration to json.dump()

There is useful configuration to json.dump() which I'd like to pass through await crawler.export_data("export.json"), but I see no way to do that:

ensure_ascii - as someone living in a country using extended latin, setting this to False prevents Python to encode half of the characters as weird mess
indent - allows me to read the output as a mere human
sort_keys - may be useful for git scraping, not sure

The only workaround I can think of right now is something convoluted like:
from pathlib import Path

path = Path("export.json")
await crawler.export_data(path)
path.write_text(json.dumps(json.loads(path.read_text()), ensure_ascii=False, indent=2))

Hi @honzajavorek, thanks for your input. We are going to add a new option for providing additional keyword arguments for export_data.

Thanks for considering this!

Just my $.02 - there is the issue of export_data being the kind of 80:20 helper that decides the output format based on the destination filename. Adding JSON-specific kwargs to this doesn't feel right, and we discussed some other options - everybody, feel free to speak your mind :slightly_smiling_face:

just add kwargs for both json, csv and whatever other format we may add in the future
make a separate export helper for each format on the BasicCrawler, e.g. export_data_json and export_data_csv
keep just export_data, but have it accept something like str | JsonExportOptions | CsvExportOptions... the path could either be part of both option object types, or there could be two parameters - path and options. The decision tree in the implementation may be larger, but manageable and testable
keep everything the way it is and publish an example how to do this now - something like json.dump(crawler.get_data(), dest_file, indent=2) should probably do it

I would go for 2, maybe combined with 1, but mentioning the format-specific methods in the export_data comment would do the job too, if you want to configure something specific to JSON or CSV, it makes sense to use a method specific for that format.

My hunch would be that the current .export_data() method is unnecessarily "magical". It implicitly decides format based on extension, which isn't clear from the outside, and will work 90 % of time, but surely there are edge cases when the extension won't match or won't be, for some reason, either .json or .csv. Also my issue shows that it perhaps does too many things, because once we need to pass JSON export options, it doesn't feel right in the interface, as there's also CSV. Hence I tend to think that this method should be two explicit methods, which have the potential to take care of the specifics of individual formats. Also in the future, adding or deprecating a format is, IMHO, done in a cleaner way with separate functions.

Is this something I can work on?

Is this something I can work on?

Absolutely, open a PR when you're ready.

repo: apify/crawlee-python


text:
Build fails for playwright crawlee initial setup

Which package is this bug report for? If unsure which one to select, leave blank
None
Issue description

Create a new project using npx crawlee create my-crawler
Select the PlaywrightCrawler template project [TypeScript]
Run npm run build or tsc

=> The build crashes with the following error:
> my-crawler@0.0.1 build
> tsc

node_modules/@crawlee/browser-pool/playwright/playwright-browser.d.ts:11:22 - error TS2420: Class 'PlaywrightBrowser' incorrectly implements interface 'Browser'.
  The types returned by 'removeAllListeners(...)' are incompatible between these types.
    Type 'this' is not assignable to type 'Promise<void>'.
      Type 'PlaywrightBrowser' is not assignable to type 'Promise<void>'.
        Type 'PlaywrightBrowser' is missing the following properties from type 'Promise<void>': then, catch, finally, [Symbol.toStringTag]

11 export declare class PlaywrightBrowser extends EventEmitter implements Browser {
                        ~~~~~~~~~~~~~~~~~

Found 1 error in node_modules/@crawlee/browser-pool/playwright/playwright-browser.d.ts:11

Code sample
No response
Package version
crawlee@3.11.4 playwright@1.47.2 tsx@4.19.1
Node.js version
v20.11.0
Operating system
macOS Sonoma 14.7
Apify platform

[ ] Tick me if you encountered this issue on the Apify platform

I have tested this on the next release
No response
Other context
No response

this should be fixed in latest dev version, please try using that

well, that is not the code from latest dev version, we no longer have the implements Browser in there.
https://unpkg.com/browse/@crawlee/browser-pool@3.11.5-beta.11/playwright/playwright-browser.d.ts

also i dont see where you update the dependency. its not enough to use dev version of the CLI to create new project (that will still use the very same templates, using stable versions), you need to update crawlee to dev version inside the created project.

yes, it should, but as i just showed you, the file is different from your screenshot, so you must be still using the old version

it works just fine with the dev version, so closing. we will be shipping new stable version soon, if you have troubles with the dev version, another option is to downgrade playwright, just pin it to the last 1.46 version, they did some breaking changes in 1.47 once again, that's why the build fails for you.

Worked now :D Thank you for helping me out @B4nan

as i said, this is fixed in master, we close issues when they get fixed, not when the fix is released.

repo: apify/crawlee


text:
Get locale from token and browser settings

Current
We read the default language from the config which is applied then.
Idea
Read the language from browser and token and then fallback to default config.
Acceptance
The language should be fetched from the browser settings or the token.
If one of that fails we should fallback to the default from the config file if the language is not available.

Locale will be available from the backend config

https://github.com/apiman/apiman/pull/1657

I would suggest you need to just have a language picker to override the inferred locale, and then store it in the user's profile. Can talk more about this when I get back.

authors: msavy volkflo

repo: apiman/apiman-developer-portal


text:
[Timeline] Reduce the size of the timeline window a tiny bit

Right now it still crops the bottom

closed by #68

repo: apirrone/Memento


text:
Split option

Hi,
The api-ref-resolver module is really powerful for dereferencing $ref.
I wonder if there is also an option to "split" again in $ref after the derefencing?
My use-case:

have a OpenAPI with $ref
derefence them
doing some filtering and cleanup
write back the OpenAPI as $ref files

I'm sorry, I don't know what you mean by "split in $ref" or "write back the OpenAPI as $ref files". Perhaps if you provide a more specific example it will help. However, I think this is likely beyond the scope of what api-ref-resolver does.

repo: apiture/api-ref-resolver

license: ISC