text:
[SPARK-4845][Core] Adding a parallelismRatio to control the partitions num of shuffledRDD

Adding parallelismRatio to control the partitions num of shuffledRDD, the rule is:
 Math.max(1, parallelismRatio * number of partitions of the largest upstream RDD)

The ratio is 1.0 by default to make it compatible with the old version. When we have a good experience on it, we can change this.

Hi @scwf can you elaborate on the motivation for this?

I am also not clear this is a good thing. As a default, it doesn't change anything. There is probably not a globally correct ratio, even if it's not 1, but this implies there is. Is there evidence that a default besides 1.0 is better in most cases? The docs don't even suggest what the tradeoff is here.
Won't this potentially cause more shuffles when the ratio is not 1? I think this is something that must be set on a case-by-case basis, and that can already be done, even as a function of the parent RDD partitions, by the caller.
Can we elaborate on this or close it?

In general, a fixed number of partitions is very difficult to work with when configuring a shuffle.  Suppose I have a job where I know a flatMap is going to blow up the size of my data by two.  If I want to minimize reduce-side spilling in a shuffle that comes after the flatMap, I want the parallelism of the shuffle to be double that of the input stage.  Because the size of my input data could change between different runs of my job, a ratio is a much more natural way to express my needs than a constant.
It's unclear to me whether a global default is useful at all, but a configurable parallelism ratio per shuffle operation definitely is.  (Systems like Crunch take this approach).

You can implement this by expressing parallelism as a function of the parent RDD right? yeah you have to write the expression but does an alternative multiplier arg do much better? yeah mostly I'm questioning a global setting.

@srowen good point.  I think a ratio argument is prettier than an expression, but arguably not enough to warrant clogging up the API.

i do not think that a global default ratio is right. because in a job the size of each stage is different and they are not Increasing or decreasing.  if we define a partition's ratio for per shuffle operation, there are no different between setting a ration and setting partition number for per shuffle operation.

I suggest we close this as I see arguments against, and no replies to those and/or the motivation for this change.

I agree. @scwf would you mind closing this issue?

sorry for delay, my initial idea here is
1 we can set spark.default.parallsim to control the partitions num for shuffle but this config option do not sensitive to data size of rdd, that is for one job with 1T input data the partitions num is x but for the same job with 1K input data the partitions num is also x.
2 if we not set spark.default.parallsim, spark rdd use parent rdd's partitions num as its partitions num, but in this way i found that there maybe some mini-tasks in some case due to the big partitions num of parent rdd, so i think maybe we can give a ratio to control the shuffle partition num
ok, i am closing this

authors: andrewor14 lianhuiwang scwf srowen sryza


text:
[SPARK-40135][PS] Support data mixed with index in DataFrame creation

What changes were proposed in this pull request?

param data accepts ps.DataFrame, since pd.DataFrame(...) also accepts another pd.DataFrame as a data param;
when data is distributed (InternalFrame/SparkDataFrame/ps.DataFrame/ps.Series), apply combine_frames to combine data and index;
when data is a local data (pd.DataFrame/list/...), directly collect index, and apply pd.DataFrame(...) internally

3.1 index should be small, when data fits in memory
3.2 combine_frames can not work with a multi-index for now
3.3 most important, to make sure queries like ps.DataFrame([1, 2], index=ps.Index([1, 2])) returns same result with pd.DataFrame([1, 2], index=pd.Index([1, 2])), if we apply combine_frames in this case, then the results will be different

Why are the changes needed?
to support index in DataFrame creation
Does this PR introduce any user-facing change?
Yes, ps.Index is supported
How was this patch tested?
added UT

cc @HyukjinKwon @ueshin @xinrong-meng

one example:
In [1]:

   ...: from pyspark.sql.types import *
   ...: from pyspark.sql import Column, DataFrame as SparkDataFrame, functions as F
   ...:
   ...: data = np.random.randn(5, 2)
   ...:
   ...: index = ps.Index([1, 3, 5, 7, 9])
   ...:
   ...: pdf = pd.DataFrame(data, columns=["A", "B"])
   ...:
   ...: ps.set_option("compute.ops_on_diff_frames", True)
   ...:
   ...: psdf = ps.DataFrame(data=pdf, index=index)
   ...:

In [2]: psdf

            A         B
NaN  1.283948  1.014323
1.0 -0.093965 -0.752795
NaN -1.254084  1.586583
3.0  1.133467 -0.773754
NaN -2.089362  2.568566
5.0       NaN       NaN
7.0       NaN       NaN
9.0       NaN       NaN

cc @ueshin @HyukjinKwon @xinrong-meng  This PR is ready to review. Thanks!

Looks pretty good to me once https://github.com/apache/spark/pull/37564#pullrequestreview-1088007634 is resolved.

Merged into master, thank you @itholic @HyukjinKwon @ueshin so much!

source: gharchive/pull-request

authors: itholic zhengruifeng

repo: apache/spark

license_type: permissive

license_source: github-api


text:
[SPARK-42226][BUILD] Upgrade versions-maven-plugin to 2.14.2

What changes were proposed in this pull request?
This pr aims upgrade versions-maven-plugin to 2.14.2
Why are the changes needed?
New version bring some improvement like Add a simple cache for ComparableVersions

The full release notes as follows:

https://github.com/mojohaus/versions/releases/tag/2.14.2

Does this PR introduce any user-facing change?
No
How was this patch tested?

GA Dependencies test should work normally
Manually check ./dev/test-dependencies.sh --replace-manifest, run successful

Thanks @dongjoon-hyun


text:
[SPARK-43156][SQL][3.4] Fix COUNT(*) is null bug in correlated scalar subquery

What changes were proposed in this pull request?
Cherry pick fix COUNT(*) is null bug in correlated scalar subquery
cherry pick from #40865 and #40946

Why are the changes needed?
Fix COUNT(*) is null bug in correlated scalar subquery in branch 3.4

Does this PR introduce any user-facing change?
No

How was this patch tested?
add new test.

cc @cloud-fan

I fixed the PR title.
- [SPARK-43156][SQL][3.3] Fix COUNT(*) is null bug in correlated scalar subquery
+ [SPARK-43156][SQL][3.4] Fix COUNT(*) is null bug in correlated scalar subquery

cc @sunchao and @parthchandra

authors: Hisoka-X dongjoon-hyun


text:
[SPARK-47755][CONNECT] Pivot should fail when the number of distinct values is too large

What changes were proposed in this pull request?
Pivot should fail when the number of distinct values is too large
Why are the changes needed?
Following check is missing in Spark Connect
    if (values.length > maxValues) {
      throw QueryCompilationErrors.aggregationFunctionAppliedOnNonNumericColumnError(
        pivotColumn.toString, maxValues)
    }

Does this PR introduce any user-facing change?
pivot will fail when the number of distinct values exceed DATAFRAME_PIVOT_MAX_VALUES
and so will be consistent with classic spark.
How was this patch tested?
added test
Was this patch authored or co-authored using generative AI tooling?
no

Merged to master.

authors: HyukjinKwon zhengruifeng


text:
[SPARK-48780][SQL] Make errors in NamedParametersSupport generic to handle functions and procedures

What changes were proposed in this pull request?

This PR makes errors in NamedParametersSupport generic so that we can reuse that class to handle argument rearrangement both in functions and procedures. It is a subset of changes from PR #47183.
Why are the changes needed?

These changes are needed in preparation for adding support for stored procedures.
Does this PR introduce any user-facing change?

No.
How was this patch tested?

Exists tests.
Was this patch authored or co-authored using generative AI tooling?

No.

cc @cloud-fan @viirya @dongjoon-hyun @huaxingao @sunchao @HyukjinKwon @gengliangwang

Hm, seems like a failure in streaming. I'll check tomorrow.
[info] - SPARK-41224: collect data using arrow *** FAILED *** (39 milliseconds)
[info]   VerifyEvents.this.executeHolder.eventsManager.hasError.isDefined was false (SparkConnectServiceSuite.scala:888)
[info]   org.scalatest.exceptions.TestFailedException:
[info]   at org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:472)
[info]   at org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:471)
[info]   at org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1231)
[info]   at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:1295)

I retried the streaming test and it is all green now.

Merged to master.

authors: HyukjinKwon aokolnychyi


text:
[SPARK-7003] Improve reliability of connection failure detection between Netty block transfer service endpoints

Currently we rely on the assumption that an exception will be raised and the channel closed if two endpoints cannot communicate over a Netty TCP channel. However, this guarantee does not hold in all network environments, and SPARK-6962 seems to point to a case where only the server side of the connection detected a fault.
This patch improves robustness of fetch/rpc requests by having an explicit timeout in the transport layer which closes the connection if there is a period of inactivity while there are outstanding requests.
NB: This patch is actually only around 50 lines added if you exclude the testing-related code.

cc @rxin

Test build #30572 has started for   PR 5584 at commit aa5278b.

Test build #30572 has finished for   PR 5584 at commit aa5278b.

This patch fails RAT tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

public class MapConfigProvider extends ConfigProvider

This patch does not change any dependencies.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30572/
Test FAILed.

Test build #30573 has started for   PR 5584 at commit 37ce656.

All comments addressed.

Test build #30575 has started for   PR 5584 at commit 8699680.

LGTM.

Test build #30573 has finished for   PR 5584 at commit 37ce656.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

public class MapConfigProvider extends ConfigProvider

This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30573/
Test PASSed.

Test build #30575 has finished for   PR 5584 at commit 8699680.

This patch passes all tests.

This patch merges cleanly.

This patch adds the following public classes (experimental):

public class MapConfigProvider extends ConfigProvider

This patch does not change any dependencies.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/30575/
Test PASSed.

Merging into master. We may consider backporting to 1.3 if it turns out this fixes SPARK-6962, which is a pretty nasty bug for those who run into it.

authors: AmplabJenkins SparkQA aarondav rxin


text:
[minor] [build] Use custom temp directory during build.

Even with all the efforts to cleanup the temp directories created by
unit tests, Spark leaves a lot of garbage in /tmp after a test run.
This change overrides java.io.tmpdir to place those files under the
build directory instead.
After an sbt full unit test run, I was left with > 400 MB of temp
files. Since they're now under the build dir, it's much easier to
clean them up.
Also make a slight change to a unit test to make it not pollute the
source directory with test data.

Merged build triggered.

Merged build started.

Test build #34212 has started for   PR 6653 at commit aa92944.

Nice catch. This plays nicely with some Jenkins tuning work that I'm doing, where we're experimenting with using ramdisks to store the Jenkins workspaces for the pull request builders.

+1

Test build #34212 has finished for   PR 6653 at commit aa92944.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test build #883 has started for   PR 6653 at commit aa92944.

Merged build triggered.

Merged build started.

Test build #34219 has started for   PR 6653 at commit 31e2dd5.

Test build #883 has finished for   PR 6653 at commit aa92944.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Update: ramdisk didn't make a difference, so we're not using it.

Test build #34219 has finished for   PR 6653 at commit 31e2dd5.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
public final class UnsafeRow extends BaseMutableRow
public abstract class BaseMutableRow extends BaseRow implements MutableRow
public abstract class BaseRow implements Row
 protected class CodeGenContext
abstract class BaseMutableProjection extends MutableProjection
      class SpecificProjection extends $
class BaseOrdering extends Ordering[Row]
      class SpecificOrdering extends $
abstract class Predicate
      class SpecificPredicate extends $
abstract class BaseProject extends Projection
    class SpecificProjection extends $
    final class SpecificRow extends $

Merged build finished. Test PASSed.

Looks OK to me.

We should probably merge this into branch-1.4 and 1.3 as well, since those are the more active backport branches; I think that Shane will appreciate the reduction in /tmp garbage on the Jenkins boxes :)

Looks like this may have broken a SparkSubmitSuite test: https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-pre-YARN/2654/
We may need to ensure that the temp dir. exists before we run tests or something like that; haven't had time to investigate.  Can someone look at this and either revert this patch or push a hotfix?

Looking. Worked for me locally but may have been a test ordering thing.

I reverted this. Could we file a JIRA for this actually? This is a borderline minor change that would be good to track.

@vanzin @srowen this broke the Spark 1.3 builds, so I revert it in the Spark 1.3 branch.

authors: AmplabJenkins JoshRosen SparkQA andrewor14 harishreedharan pwendell srowen vanzin


text:
STORM-2947: Remove deprecated field and method from Thrift

Part of https://issues.apache.org/jira/browse/STORM-2947

Should I just comment out beginFileDownload in the thrift file, or does it need to stay in Nimbus and LocalCluster commented out as well?

@srdo Only comment it out in the thrift code.  Delete the java code.  The only reason to comment it out instead of deleting it is so there is a visual clue to anyone editing the file in the future that this API used to be here, and we should avoid reusing it.  Just on the off chance that we recreate the same API but the semantics are incompatible with each other and we get really odd errors.

@revans2 Could you revisit this PR?

authors: HeartSaVioR revans2 srdo

repo: apache/storm


text:
WW-5199 Allow forwarding from/to actions

Fixes WW-5199

Thanks a lot! I had no idea that the fix would so easy :)
LGTM 👍

authors: kusalk lukaszlenart

repo: apache/struts


text:
SUBMARINE-947. Deploy website must be limited to apache submarine repository

What is this PR for?

Check website job should be enabled to run on Pull Requests.
Deploy website job must only run on the apache submarine repository and the master branch.

What type of PR is it?
[Improvement]
Todos
What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-947
How should this be tested?
Screenshots (if appropriate)
Questions:

Do the license files need updating? No
Are there breaking changes for older versions? No
Does this need new documentation? No

@pingsutw Thanks for your review

repo: apache/submarine


text:
Multiple tables join queries to produce a single chart

Is your feature request related to a problem? Please describe.
I'm always frustrated when the business department has requirements change, the previously prepared single Table cannot be required (N tables need to be combined). In view of this situation, data can only be processed manually at present, and then feedbacks can be made to the business department.
Describe the solution you'd like
Whether we can consider to realize this function in the future: multiple tables can be aggregated to generate a Charts.
Describe alternatives you've considered
For Chart production, you can select multiple tables and add a drop-down list for association selection (left/right/full join...)..
After the selection, the filter consolidation is passed to the back end for data query.This is just my preliminary idea.If thank you, welcome to discuss and exchange.

@altef  I wonder if you are interested in this topic, or if you can help me with @ related personnel.Thank you

You should be able to achieve this via a SQL view, or by forming your query in SQL Lab and then clicking “explore”

@nytai Later I went to try the way you said, but at the moment our SQLLab or only the admin can operate, so in view of the scene I said, can't let business people operating on a graph, it would greatly increase the workload of the admin staff, if SQLLab open to business people, they do not have time to learn the relevant operation is grammar, this is to let a person helpless.

@amitmiran137 I'd like to know why you closed this issue. Do we have any plans for this issue in the future?

For the time being in order to perform sql join between tables is via a virtual dataset which can be create from the sqlLab screen.
And yes you need to be familiar with SQL.
If you would like
I think it  can be nice to have to support conjuring a join between multiple tables from the explore page  but I don't see it prioritized higher with other critical taks among superset commiters .
If you would like to write a detailed SIP that have accurate definitions on how it would work especially on the frontend wide (explore page) this would help promoting this
But someone would also need to implement that
cc: @villebro @betodealmeida  any thoughts here ?

I mention this briefly in https://github.com/apache/superset/issues/14909, but I would like to improve the semantic layer of Superset so that we can define relationships between tables within Superset. But this would be for simple many-to-one or one-to-one relationships between 2 tables only, so that if you're exploring a fact table (say, sales) you can filter or group by user attributes (age, eg) automatically joining by user_id.

source: gharchive/issue

repo: apache/superset


text:
Example Dashboards: Tabbed Dashboard Charts Don't Show Up

Expected results
Dashboard should load charts properly
Actual results

Charts aren't working in the Tabbed example dashboard
How to reproduce the bug

Click on Dashboards
Click on the Tabbed Dashboard

Environment
(please complete the following information):

superset version: 'master'
python version: python --version
node.js version: node -v

Checklist
Make sure to follow these steps before submitting your issue - thank you!

[ ] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[x] I have reproduced the issue with at least the latest released version of superset.
[x] I have checked the issue tracker for the same issue and I haven't found one similar.

I encountered this message twice before.

with native dashboard filter FF tuned on, it went away after nuking db
changing chart metadata

I will look into it. thanks for reporting! @yousoph

confirmed that this sample dashboard has been broken since/in v1.0
and the dashboard seems got removed as a whole(does not exist anymore)...

authors: junlincc yousoph


text:
Bar chart horizontal orientation out of X axis

Screenshot
Vertical orientation is ok

Horizontal orientation is weird

Description
I using latest version of apache Superset (latest commit on March 22).

Same issue exists in 3.0.0.

Guys, any update on the above problem?

Yeah, that's an odd one. It's still an issue in 4.0, but there seems to be strangely little outcry about this. PRs welcome, of course!

authors: Always-prog prasanjeetdas rusackas zhicwu


text:
update tox testing run to relevant python env

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

Codecov Report

Merging #12748 (33d249e) into master (0fed1e0) will decrease coverage by 3.51%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #12748      +/-   ##
==========================================
- Coverage   66.89%   63.37%   -3.52%
==========================================
  Files        1022      488     -534
  Lines       49994    30091   -19903
  Branches     4892        0    -4892
==========================================
- Hits        33442    19070   -14372
+ Misses      16427    11021    -5406
+ Partials      125        0     -125

Flag
Coverage Δ

cypress
?

javascript
?

python
63.37% <ø> (-0.71%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

superset/db_engines/hive.py
0.00% <0.00%> (-85.72%)
:arrow_down:

superset/databases/commands/create.py
32.65% <0.00%> (-59.19%)
:arrow_down:

superset/sql_validators/postgres.py
50.00% <0.00%> (-50.00%)
:arrow_down:

superset/views/database/mixins.py
59.64% <0.00%> (-22.81%)
:arrow_down:

superset/db_engine_specs/hive.py
73.84% <0.00%> (-17.31%)
:arrow_down:

superset/databases/commands/update.py
85.71% <0.00%> (-8.17%)
:arrow_down:

superset/db_engine_specs/presto.py
81.38% <0.00%> (-7.15%)
:arrow_down:

superset/databases/schemas.py
93.40% <0.00%> (-6.05%)
:arrow_down:

superset/databases/dao.py
94.11% <0.00%> (-5.89%)
:arrow_down:

superset/databases/api.py
86.55% <0.00%> (-5.47%)
:arrow_down:

... and 554 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0fed1e0...33d249e. Read the comment docs.

Codecov Report

Merging #12748 (33d249e) into master (0fed1e0) will decrease coverage by 3.51%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #12748      +/-   ##
==========================================
- Coverage   66.89%   63.37%   -3.52%
==========================================
  Files        1022      488     -534
  Lines       49994    30091   -19903
  Branches     4892        0    -4892
==========================================
- Hits        33442    19070   -14372
+ Misses      16427    11021    -5406
+ Partials      125        0     -125

Flag
Coverage Δ

cypress
?

javascript
?

python
63.37% <ø> (-0.71%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

superset/db_engines/hive.py
0.00% <0.00%> (-85.72%)
:arrow_down:

superset/databases/commands/create.py
32.65% <0.00%> (-59.19%)
:arrow_down:

superset/sql_validators/postgres.py
50.00% <0.00%> (-50.00%)
:arrow_down:

superset/views/database/mixins.py
59.64% <0.00%> (-22.81%)
:arrow_down:

superset/db_engine_specs/hive.py
73.84% <0.00%> (-17.31%)
:arrow_down:

superset/databases/commands/update.py
85.71% <0.00%> (-8.17%)
:arrow_down:

superset/db_engine_specs/presto.py
81.38% <0.00%> (-7.15%)
:arrow_down:

superset/databases/schemas.py
93.40% <0.00%> (-6.05%)
:arrow_down:

superset/databases/dao.py
94.11% <0.00%> (-5.89%)
:arrow_down:

superset/databases/api.py
86.55% <0.00%> (-5.47%)
:arrow_down:

... and 554 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0fed1e0...33d249e. Read the comment docs.


text:
feat: add generic type to column payload

SUMMARY
Adds the generic datatype to the dataset column payload to avoid having to parse the raw datatypes in the frontend. Also bump superset-ui to bring in updates to superset-ui/core that are required to make this function properly: https://github.com/apache-superset/superset-ui/pull/1102
BEFORE
Currently the numeric datatypes on the FCC 2018 example dataset show up as unidentified due to the frontend not being able to detect the generic datatype of the raw datatype from the database ("DOUBLE PRECISION"):

AFTER
Now numeric types are being picked up automatically. In the future only datatypes that the db engine spec doesn't identify are shown as question marks:

TEST PLAN
Added datatype tests for Postgres
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

/testenv up

hmmm still seeing "?"

authors: junlincc villebro


text:
refactor: remove unnecessary dataset queries from dashboard requests

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

table_names is still used here (looks like for FAB CRUD views), should we clean up those as well?

I think it's also safe to remove https://github.com/apache/superset/blob/b3e699b76705185ac9db29d9ee0c4478f02231cc/superset/models/dashboard.py#L164-L167 as that method is not particularly efficient (should've used batch query instead of looping through all slices).

authors: graceguo-supercat ktmud


text:
fix(reports): force data generation in csv reports

SUMMARY
fixes: #22189
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

create table chart
setup csv report on that chart and enable Ignore cache when generating screenshot
update data in datasource without bursting cache
check csv in the report received, should contain new data

ADDITIONAL INFORMATION

[x] Has associated issue: #22189
[x] Required feature flags: ALERTS_REPORTS
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

Should we also change Ignore cache when generating screenshot to Ignore cache when generating report as CSV, json do not need screenshots.

Thanks for the reviews 😀


text:
perf: Lazy load moment-timezone

SUMMARY
moment-timezone library is over 700kb and it is only used in alerts/reports modal. Loading it by default slows down dashboard's loading time.
This PR implements lazy loading this library.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
Notice the lack of ~800kb network activity in the "after" screenshot
Before:

After:

TESTING INSTRUCTIONS

Open a dashboard
Verify that the moment-timezone chunk was not loaded
3 dots menu -> Manage email report -> Set up an email report
Verify that the timezone picker is working correctly

ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Required feature flags:
[ ] Changes UI
[ ] Includes DB Migration (follow approval process in SIP-59)

[ ] Migration is atomic, supports rollback & is backwards-compatible
[ ] Confirm DB migration upgrade and downgrade tested
[ ] Runtime estimates and downtime expectations provided

[ ] Introduces new feature or API
[ ] Removes existing feature or API

Thanks for working on this type of task! That will bring huge benefit to the community.

Glad to see "frontend-infra" getting some ❤ !

@dosubot I'm curious if you can point to libraries that are popular drop-in/easy replacement for moment-timezone and/or moment.js

@mistercrunch @rusackas replacing moment is going to be a big effort as it’s used everywhere, but it’s definitely going to be worthwhile for performance - moment is big

Yeah it's used in 30+ modules ->
$ git grep "import.*moment" superset-frontend/src/
superset-frontend/src/SqlLab/components/QueryTable/index.tsx:import moment from 'moment';
superset-frontend/src/SqlLab/components/SaveDatasetModal/index.tsx:import moment from 'moment';
superset-frontend/src/components/CachedLabel/TooltipContent.test.tsx:import moment from 'moment';
superset-frontend/src/components/CachedLabel/TooltipContent.tsx:import moment from 'moment';
superset-frontend/src/components/Chart/chartAction.js:import moment from 'moment';
superset-frontend/src/components/LastUpdated/index.tsx:import moment, { Moment, MomentInput } from 'moment';
superset-frontend/src/components/ListView/Filters/DateRange.tsx:import moment, { Moment } from 'moment';
superset-frontend/src/components/Timer/Timer.stories.tsx:import moment from 'moment';
superset-frontend/src/components/TimezoneSelector/TimezoneSelector.test.tsx:import moment from 'moment-timezone';
superset-frontend/src/components/TimezoneSelector/index.tsx:import moment from 'moment-timezone';
superset-frontend/src/dashboard/components/Header/index.jsx:import moment from 'moment';
superset-frontend/src/dashboard/components/OverwriteConfirm/OverwriteConfirmModal.tsx:import moment from 'moment';
superset-frontend/src/dashboard/components/SliceHeaderControls/index.tsx:import moment from 'moment';
superset-frontend/src/explore/components/controls/ComparisonRangeLabel.tsx:import moment from 'moment';
superset-frontend/src/explore/components/controls/DateFilterControl/components/CustomFrame.tsx:import { Moment } from 'moment';
superset-frontend/src/explore/components/controls/DateFilterControl/utils/constants.ts:import moment from 'moment';
superset-frontend/src/explore/components/controls/DateFilterControl/utils/dateParser.ts:import moment, { Moment } from 'moment';
superset-frontend/src/explore/components/controls/TimeOffsetControl.tsx:import moment, { Moment } from 'moment';
superset-frontend/src/features/allEntities/AllEntitiesTable.tsx:import moment from 'moment';
superset-frontend/src/features/annotations/AnnotationModal.tsx:import moment from 'moment';
superset-frontend/src/features/datasets/AddDataset/EditDataset/UsageTab/index.tsx:import moment from 'moment';
superset-frontend/src/features/home/ActivityTable.tsx:import moment from 'moment';
superset-frontend/src/pages/AlertReportList/index.tsx:import moment from 'moment';
superset-frontend/src/pages/AnnotationList/index.tsx:import moment from 'moment';
superset-frontend/src/pages/ExecutionLogList/index.tsx:import moment from 'moment';
superset-frontend/src/pages/QueryHistoryList/index.tsx:import moment from 'moment';
superset-frontend/src/preamble.ts:import moment from 'moment';
superset-frontend/src/utils/dates.js:import moment from 'moment';
superset-frontend/src/visualizations/TimeTable/SparklineCell.tsx:import moment from 'moment';

$ git grep "import.*moment" superset-frontend/plugins/
superset-frontend/plugins/legacy-preset-chart-nvd3/src/NVD3Vis.js:import moment from 'moment';
superset-frontend/plugins/plugin-chart-echarts/src/BigNumber/BigNumberPeriodOverPeriod/transformProps.ts:import moment from 'moment';
superset-frontend/plugins/plugin-chart-echarts/src/BigNumber/utils.ts:import moment from 'moment';
superset-frontend/plugins/plugin-chart-handlebars/src/components/Handlebars/HandlebarsViewer.tsx:import moment from 'moment';

Curious how many different functions we use.
From memory working on bundle sizes in the past, I think they have a large i18n module that we don't really use (I think). I'm not sure if we're shaking that off properly or not, but could be something to look at and make sure we're not importing things we don't need.

Yeah most of it is a locale submodule. I’ll see if we can shake it off

I’ll see if we can shake it off

@mistercrunch @rusackas Do we have a list of locales supported in Superset? We can remove the unsupported ones with a webpack plugin (source)

I wonder if they should be 1:1 with the languages that are enabled in the config file. That'd make it dynamic, support expansion when people add languages (which does happen), and keep things as minimal as possible. Or we could even move this to a similar config.

output is a bit messy, but here are our language packs ->
$ ls -d superset/translations/*
superset/translations/__init__.py               superset/translations/it                        superset/translations/sk
superset/translations/__pycache__               superset/translations/ja                        superset/translations/sl
superset/translations/ar                        superset/translations/ko                        superset/translations/tr
superset/translations/babel.cfg                 superset/translations/messages.pot              superset/translations/uk
superset/translations/de                        superset/translations/nl                        superset/translations/utils.py
superset/translations/empty_language_pack.json  superset/translations/pt                        superset/translations/zh
superset/translations/en                        superset/translations/pt_BR                     superset/translations/zh_TW
superset/translations/es                        superset/translations/requirements.txt
superset/translations/fr                        superset/translations/ru

@mistercrunch @rusackas Managed to tree shake the locales 🙂
Before:

After:

authors: kgabryje michael-s-molina mistercrunch rusackas


text:
[DOC][2/2] Documentation for builtin KMeans function

Description:
Adds the runnable code snippet for KMeans
Question: Shall we initialize the number of centroids to 10, instead of 0 (throwing an error in get_sample_maps function)?
This PR Results:
dml:
X = rand (rows = 3972, cols = 972)
kmeans(X = X, k = 20, runs = 10, max_iter = 5000, eps = 0.000001, is_verbose = FALSE, avg_sample_size_per_centroid = 50)

Adds the runnable code snippet.
Log:
# EXPLAIN (RUNTIME):
# Memory Budget local/remote = 637MB/362MB/434MB/152MB
# Degree of Parallelism (vcores) local/remote = 4/4
PROGRAM ( size CP/SP = 443/9 )
--FUNCTIONS
----FUNCTION CALL GRAPH
------MAIN PROGRAM
--------.defaultNS::m_kmeans
----------.defaultNS::get_sample_maps
----FUNCTION .defaultNS::m_kmeans [recompile=true]
------GENERIC (lines 50-66) [recompile=false]
--------CP uasqk+ X.MATRIX.FP64 _Var44.SCALAR.FP64 4
--------CP print BEGIN K-MEANS SCRIPT.SCALAR.STRING.true _Var45.SCALAR.STRING
--------CP print Taking data samples for initialization....SCALAR.STRING.true _Var46.SCALAR.STRING
--------CP extfunct .defaultNS get_sample_maps 3 3 num_records=3972.SCALAR.INT64.true num_samples=10.SCALAR.INT64.true approx_sample_size=1000.SCALAR.INT64.true sample_maps samples_vs_runs_map sample_block_size
--------CP mvvar _Var44 sumXsq
--------CP rmvar _Var45 _Var46 avg_sample_size_per_centroid k runs
------GENERIC (lines 68-79) [recompile=false]
--------CP createvar _mVar47 scratch_space//_pHidden_IP//_t0/temp22 true MATRIX binary 13160 1 1000 -1 copy
--------CP uark+ sample_maps.MATRIX.FP64 _mVar47.MATRIX.FP64 4
--------CP createvar _mVar48 scratch_space//_pHidden_IP//_t0/temp23 true MATRIX binary 13160 972 1000 -1 copy
--------CP ba+* sample_maps.MATRIX.FP64 X.MATRIX.FP64 _mVar48.MATRIX.FP64 4
--------CP createvar _mVar49 scratch_space//_pHidden_IP//_t0/temp24 true MATRIX binary 13160 1 1000 -1 copy
--------CP uarsqk+ _mVar48.MATRIX.FP64 _mVar49.MATRIX.FP64 4
--------CP createvar _mVar50 scratch_space//_pHidden_IP//_t0/temp25 true MATRIX binary 200 972 1000 0 copy
--------CP rand 200.SCALAR.INT64.true 972.SCALAR.INT64.true 1000 0 0 1.0 -1 uniform 1.0 4 _mVar50.MATRIX.FP64
--------CP print Initializing the centroids for all runs....SCALAR.STRING.true _Var51.SCALAR.STRING
--------CP cpvar _mVar47 is_row_in_samples
--------CP cpvar _mVar48 X_samples
--------CP cpvar _mVar49 X_samples_sq_norms
--------CP rmvar _Var51
--------CP cpvar _mVar50 All_Centroids
--------CP cpvar _mVar47 min_distances
--------CP rmvar _mVar47 _mVar48 _mVar49 _mVar50 num_features sample_maps
------GENERIC (lines 0-0) [recompile=true]
--------CP createvar _mVar52 scratch_space//_pHidden_IP//_t0/temp26 true MATRIX binary 13160 -1 1000 -1 copy
--------SPARK chkpoint samples_vs_runs_map.MATRIX.FP64 _mVar52.MATRIX.FP64 MEMORY_AND_DISK
--------CP rmvar samples_vs_runs_map
--------CP mvvar _mVar52 samples_vs_runs_map
------FOR (lines 0-0)
--------CP assignvar 1.SCALAR.INT64.true __pred.SCALAR.INT64
--------CP assignvar 20.SCALAR.INT64.true __pred.SCALAR.INT64
--------GENERIC (lines 84-106) [recompile=true]
----------CP + 180.SCALAR.INT64.true i.SCALAR.INT64.false _Var53.SCALAR.INT64
----------CP createvar _mVar54 scratch_space//_pHidden_IP//_t0/temp27 true MATRIX binary -1 1 1000 -1 copy
----------SPARK seq -1 1 1000 i.SCALAR.INT64.false _Var53.SCALAR.INT64.false 20.SCALAR.INT64.true _mVar54.MATRIX.FP64
----------CP rmvar _Var53
----------CP createvar _mVar55 scratch_space//_pHidden_IP//_t0/temp28 true MATRIX binary 10 1 1000 -1 copy
----------CP seq 10 1 1000 1.SCALAR.INT64.true 10.SCALAR.INT64.true 1.SCALAR.INT64.true _mVar55.MATRIX.FP64
----------CP createvar _mVar56 scratch_space//_pHidden_IP//_t0/temp29 true MATRIX binary 200 10 1000 -1 copy
----------SPARK ctable _mVar54.MATRIX.FP64 _mVar55.MATRIX.FP64 1.0.SCALAR.FP64.true 200.true 10.true _mVar56.MATRIX.FP64 false
----------CP rmvar _mVar54 _mVar55
----------CP * sample_block_size.SCALAR.INT64.false 10.SCALAR.INT64.true _Var57.SCALAR.INT64
----------CP createvar _mVar58 scratch_space//_pHidden_IP//_t0/temp30 true MATRIX binary 1316 10 1000 -1 copy
----------CP rshape min_distances.MATRIX.FP64 sample_block_size.SCALAR.INT64.false 10.SCALAR.INT64.true -1.SCALAR.STRING.true false.SCALAR.BOOLEAN.true _mVar58.MATRIX.FP64
----------CP createvar _mVar59 scratch_space//_pHidden_IP//_t0/temp31 true MATRIX binary 1316 10 1000 -1 copy
----------CP ucumk+ _mVar58.MATRIX.FP64 _mVar59.MATRIX.FP64 4 false
----------CP rmvar _mVar58
----------CP createvar _mVar60 scratch_space//_pHidden_IP//_t0/temp32 true MATRIX binary 1 10 1000 10 copy
----------CP rand 1.SCALAR.INT64.true 10.SCALAR.INT64.true 1000 0.0 1.0 1.0 -1 uniform 1.0 4 _mVar60.MATRIX.FP64
----------CP createvar _mVar61 scratch_space//_pHidden_IP//_t0/temp33 true MATRIX binary 1 10 1000 -1 copy
----------CP rightIndex _mVar59.MATRIX.FP64 sample_block_size.SCALAR.INT64.false sample_block_size.SCALAR.INT64.false 1.SCALAR.INT64.true 10.SCALAR.INT64.true _mVar61.MATRIX.FP64
----------CP createvar _mVar62 scratch_space//_pHidden_IP//_t0/temp34 true MATRIX binary 1 10 1000 -1 copy
----------CP * _mVar60.MATRIX.FP64 _mVar61.MATRIX.FP64 _mVar62.MATRIX.FP64
----------CP rmvar _mVar60 _mVar61
----------CP createvar _mVar63 scratch_space//_pHidden_IP//_t0/temp35 true MATRIX binary 1316 10 1000 -1 copy
----------CP < _mVar59.MATRIX.FP64 _mVar62.MATRIX.FP64 _mVar63.MATRIX.FP64
----------CP rmvar _mVar59 _mVar62
----------CP createvar _mVar64 scratch_space//_pHidden_IP//_t0/temp36 true MATRIX binary 1 10 1000 -1 copy
----------CP uack+ _mVar63.MATRIX.FP64 _mVar64.MATRIX.FP64 4
----------CP rmvar _mVar63
----------CP createvar _mVar65 scratch_space//_pHidden_IP//_t0/temp37 true MATRIX binary 10 1 1000 -1 copy
----------CP r' _mVar64.MATRIX.FP64 _mVar65.MATRIX.FP64 4
----------CP rmvar _mVar64
----------CP createvar _mVar66 scratch_space//_pHidden_IP//_t0/temp38 true MATRIX binary 10 1 1000 -1 copy
----------CP + _mVar65.MATRIX.FP64 1.SCALAR.INT64.true _mVar66.MATRIX.FP64
----------CP rmvar _mVar65
----------CP createvar _mVar67 scratch_space//_pHidden_IP//_t0/temp39 true MATRIX binary 10 1 1000 -1 copy
----------CP seq 10 1 1000 0.SCALAR.INT64.true 9.SCALAR.INT64.true 1.SCALAR.INT64.true _mVar67.MATRIX.FP64
----------CP createvar _mVar68 scratch_space//_pHidden_IP//_t0/temp40 true MATRIX binary 10 1 1000 -1 copy
----------CP +* _mVar66.MATRIX.FP64 sample_block_size.SCALAR.INT64.false _mVar67.MATRIX.FP64 _mVar68.MATRIX.FP64
----------CP rmvar _mVar66 _mVar67
----------CP createvar _mVar69 scratch_space//_pHidden_IP//_t0/temp41 true MATRIX binary 10 -1 1000 -1 copy
----------SPARK rexpand cast=true max=_Var57 ignore=false dir=cols target=_mVar68 _mVar69.MATRIX.FP64
----------CP rmvar _Var57 _mVar68
----------CP createvar _mVar70 scratch_space//_pHidden_IP//_t0/temp42 true MATRIX binary 10 972 1000 -1 copy
----------SPARK mapmm _mVar69.MATRIX.FP64 X_samples.MATRIX.FP64 _mVar70.MATRIX.FP64 RIGHT true SINGLE_BLOCK
----------CP rmvar _mVar69
----------CP createvar _mVar71 scratch_space//_pHidden_IP//_t0/temp43 true MATRIX binary 200 972 1000 -1 copy
----------CP ba+* _mVar56.MATRIX.FP64 _mVar70.MATRIX.FP64 _mVar71.MATRIX.FP64 4
----------CP rmvar _mVar56
----------CP createvar _mVar72 scratch_space//_pHidden_IP//_t0/temp44 true MATRIX binary 200 972 1000 -1 copy
----------CP + All_Centroids.MATRIX.FP64 _mVar71.MATRIX.FP64 _mVar72.MATRIX.FP64
----------CP rmvar _mVar71
----------CP == i.SCALAR.INT64.false 1.SCALAR.INT64.true _Var73.SCALAR.BOOLEAN
----------CP createvar _mVar74 scratch_space//_pHidden_IP//_t0/temp45 true MATRIX binary 10 1 1000 -1 copy
----------SPARK uarsqk+ _mVar70.MATRIX.FP64 _mVar74.MATRIX.FP64 NONE
----------CP createvar _mVar75 scratch_space//_pHidden_IP//_t0/temp46 true MATRIX binary 13160 1 1000 -1 copy
----------SPARK mapmm samples_vs_runs_map.MATRIX.FP64 _mVar74.MATRIX.FP64 _mVar75.MATRIX.FP64 RIGHT false MULTI_BLOCK
----------CP rmvar _mVar74
----------CP createvar _mVar76 scratch_space//_pHidden_IP//_t0/temp47 true MATRIX binary 13160 1 1000 -1 copy
----------CP + X_samples_sq_norms.MATRIX.FP64 _mVar75.MATRIX.FP64 _mVar76.MATRIX.FP64
----------CP rmvar _mVar75
----------CP createvar _mVar77 scratch_space//_pHidden_IP//_t0/temp48 true MATRIX binary 13160 972 1000 -1 copy
----------SPARK mapmm samples_vs_runs_map.MATRIX.FP64 _mVar70.MATRIX.FP64 _mVar77.MATRIX.FP64 RIGHT false MULTI_BLOCK
----------CP rmvar _mVar70
----------CP createvar _mVar78 scratch_space//_pHidden_IP//_t0/temp49 true MATRIX binary 13160 972 1000 -1 copy
----------CP * X_samples.MATRIX.FP64 _mVar77.MATRIX.FP64 _mVar78.MATRIX.FP64
----------CP rmvar _mVar77
----------CP createvar _mVar79 scratch_space//_pHidden_IP//_t0/temp50 true MATRIX binary 13160 1 1000 -1 copy
----------CP uark+ _mVar78.MATRIX.FP64 _mVar79.MATRIX.FP64 4
----------CP rmvar _mVar78
----------CP createvar _mVar80 scratch_space//_pHidden_IP//_t0/temp51 true MATRIX binary 13160 1 1000 -1 copy
----------CP -* _mVar76.MATRIX.FP64 2.SCALAR.INT64.true _mVar79.MATRIX.FP64 _mVar80.MATRIX.FP64
----------CP rmvar _mVar76 _mVar79
----------CP createvar _mVar81 scratch_space//_pHidden_IP//_t0/temp52 true MATRIX binary 13160 1 1000 -1 copy
----------CP * is_row_in_samples.MATRIX.FP64 _mVar80.MATRIX.FP64 _mVar81.MATRIX.FP64
----------CP createvar _mVar82 scratch_space//_pHidden_IP//_t0/temp53 true MATRIX binary 13160 1 1000 -1 copy
----------CP min min_distances.MATRIX.FP64 _mVar80.MATRIX.FP64 _mVar82.MATRIX.FP64
----------CP rmvar _mVar80
----------CP createvar _mVar83 scratch_space//_pHidden_IP//_t0/temp54 true MATRIX binary 13160 1 1000 -1 copy
----------CP ifelse _Var73.SCALAR.BOOLEAN.false _mVar81.MATRIX.FP64 _mVar82.MATRIX.FP64 _mVar83.MATRIX.FP64
----------CP rmvar _Var73 _mVar81 _mVar82 All_Centroids min_distances
----------CP cpvar _mVar72 All_Centroids
----------CP cpvar _mVar83 min_distances
----------CP rmvar _mVar72 _mVar83
--------CP rmvar samples_vs_runs_map centroid_placer distances X_samples_sq_norms min_distances_matrix_form cdf_min_distances threshold_matrix i is_row_in_samples centroid_ids centroids random_row sample_block_size min_distances X_samples
------GENERIC (lines 111-115) [recompile=false]
--------CP createvar _mVar84 scratch_space//_pHidden_IP//_t0/temp55 true MATRIX binary 10 1 1000 0 copy
--------CP rand 10.SCALAR.INT64.true 1.SCALAR.INT64.true 1000 0 0 1.0 -1 uniform 1.0 4 _mVar84.MATRIX.FP64
--------CP print Performing k-means iterations for all runs....SCALAR.STRING.true _Var85.SCALAR.STRING
--------CP cpvar _mVar84 termination_code
--------CP cpvar _mVar84 final_wcss
--------CP cpvar _mVar84 num_iterations
--------CP rmvar _Var85 _mVar84
------GENERIC (lines 0-0) [recompile=false]
------PARFOR (lines 0-0)
--------CP assignvar 1.SCALAR.INT64.true __pred.SCALAR.INT64
--------CP assignvar 10.SCALAR.INT64.true __pred.SCALAR.INT64
--------GENERIC (lines 119-123) [recompile=false]
----------CP - run_index.SCALAR.INT64.false 1.SCALAR.INT64.true _Var86.SCALAR.INT64
----------CP * 20.SCALAR.INT64.true _Var86.SCALAR.INT64.false _Var87.SCALAR.INT64
----------CP rmvar _Var86
----------CP + _Var87.SCALAR.INT64.false 1.SCALAR.INT64.true _Var88.SCALAR.INT64
----------CP rmvar _Var87
----------CP * 20.SCALAR.INT64.true run_index.SCALAR.INT64.false _Var89.SCALAR.INT64
----------CP createvar _mVar90 scratch_space//_pHidden_IP//_t0/temp56 true MATRIX binary 20 972 1000 -1 copy
----------CP rightIndex All_Centroids.MATRIX.FP64 _Var88.SCALAR.INT64.false _Var89.SCALAR.INT64.false 1.SCALAR.INT64.true 972.SCALAR.INT64.true _mVar90.MATRIX.FP64
----------CP rmvar _Var88 _Var89
----------CP assignvar 0.SCALAR.INT64.true iter_count.SCALAR.INT64
----------CP assignvar 0.SCALAR.INT64.true term_code.SCALAR.INT64
----------CP assignvar Infinity.SCALAR.FP64.true wcss.SCALAR.FP64
----------CP cpvar _mVar90 C
----------CP mvvar _mVar90 C_old
--------WHILE (lines 125-161) [in-place=[C, C_old]]
----------CP == term_code.SCALAR.INT64.false 0.SCALAR.INT64.true _Var91.SCALAR.BOOLEAN
----------CP mvvar _Var91 __pred
----------GENERIC (lines 129-152) [recompile=false]
------------CP createvar _mVar92 scratch_space//_pHidden_IP//_t0/temp57 true MATRIX binary 972 20 1000 -1 copy
------------CP r' C.MATRIX.FP64 _mVar92.MATRIX.FP64 4
------------CP createvar _mVar93 scratch_space//_pHidden_IP//_t0/temp58 true MATRIX binary 3972 20 1000 -1 copy
------------CP ba+* X.MATRIX.FP64 _mVar92.MATRIX.FP64 _mVar93.MATRIX.FP64 4
------------CP rmvar _mVar92
------------CP createvar _mVar94 scratch_space//_pHidden_IP//_t0/temp59 true MATRIX binary 3972 20 1000 -1 copy
------------CP * -2.SCALAR.INT64.true _mVar93.MATRIX.FP64 _mVar94.MATRIX.FP64
------------CP rmvar _mVar93
------------CP createvar _mVar95 scratch_space//_pHidden_IP//_t0/temp60 true MATRIX binary 20 1 1000 -1 copy
------------CP uarsqk+ C.MATRIX.FP64 _mVar95.MATRIX.FP64 4
------------CP createvar _mVar96 scratch_space//_pHidden_IP//_t0/temp61 true MATRIX binary 1 20 1000 -1 copy
------------CP r' _mVar95.MATRIX.FP64 _mVar96.MATRIX.FP64 4
------------CP rmvar _mVar95
------------CP createvar _mVar97 scratch_space//_pHidden_IP//_t0/temp62 true MATRIX binary 3972 20 1000 -1 copy
------------CP + _mVar94.MATRIX.FP64 _mVar96.MATRIX.FP64 _mVar97.MATRIX.FP64
------------CP rmvar _mVar94 _mVar96
------------CP createvar _mVar98 scratch_space//_pHidden_IP//_t0/temp63 true MATRIX binary 3972 1 1000 -1 copy
------------CP uarmin _mVar97.MATRIX.FP64 _mVar98.MATRIX.FP64 4
------------CP uak+ _mVar98.MATRIX.FP64 _Var99.SCALAR.FP64 4
------------CP + sumXsq.SCALAR.FP64.false _Var99.SCALAR.FP64.false _Var100.SCALAR.FP64
------------CP rmvar _Var99
------------CP createvar _mVar101 scratch_space//_pHidden_IP//_t0/temp64 true MATRIX binary 3972 20 1000 -1 copy
------------CP <= _mVar97.MATRIX.FP64 _mVar98.MATRIX.FP64 _mVar101.MATRIX.FP64
------------CP rmvar _mVar97 _mVar98
------------CP createvar _mVar102 scratch_space//_pHidden_IP//_t0/temp65 true MATRIX binary 3972 1 1000 -1 copy
------------CP uark+ _mVar101.MATRIX.FP64 _mVar102.MATRIX.FP64 4
------------CP createvar _mVar103 scratch_space//_pHidden_IP//_t0/temp66 true MATRIX binary 3972 20 1000 -1 copy
------------CP / _mVar101.MATRIX.FP64 _mVar102.MATRIX.FP64 _mVar103.MATRIX.FP64
------------CP rmvar _mVar101 _mVar102
------------CP createvar _mVar104 scratch_space//_pHidden_IP//_t0/temp67 true MATRIX binary 1 20 1000 -1 copy
------------CP uack+ _mVar103.MATRIX.FP64 _mVar104.MATRIX.FP64 4
------------CP createvar _mVar105 scratch_space//_pHidden_IP//_t0/temp68 true MATRIX binary 20 3972 1000 -1 copy
------------CP r' _mVar103.MATRIX.FP64 _mVar105.MATRIX.FP64 4
------------CP rmvar _mVar103
------------CP createvar _mVar106 scratch_space//_pHidden_IP//_t0/temp69 true MATRIX binary 20 972 1000 -1 copy
------------CP ba+* _mVar105.MATRIX.FP64 X.MATRIX.FP64 _mVar106.MATRIX.FP64 4
------------CP rmvar _mVar105
------------CP createvar _mVar107 scratch_space//_pHidden_IP//_t0/temp70 true MATRIX binary 20 1 1000 -1 copy
------------CP r' _mVar104.MATRIX.FP64 _mVar107.MATRIX.FP64 4
------------CP createvar _mVar108 scratch_space//_pHidden_IP//_t0/temp71 true MATRIX binary 20 972 1000 -1 copy
------------CP / _mVar106.MATRIX.FP64 _mVar107.MATRIX.FP64 _mVar108.MATRIX.FP64
------------CP rmvar _mVar106 _mVar107
------------CP + iter_count.SCALAR.INT64.false 1.SCALAR.INT64.true _Var109.SCALAR.INT64
------------CP assignvar wcss.SCALAR.FP64.false wcss_old.SCALAR.FP64
------------CP assignvar _Var100.SCALAR.FP64.false wcss.SCALAR.FP64
------------CP assignvar _Var109.SCALAR.INT64.false iter_count.SCALAR.INT64
------------CP rmvar _Var100 _Var109
------------CP cpvar _mVar104 P_denom
------------CP cpvar _mVar108 C_new
------------CP rmvar _mVar104 _mVar108
----------IF (lines 153-160)
------------CP - wcss_old.SCALAR.FP64.false wcss.SCALAR.FP64.false _Var110.SCALAR.FP64
------------CP * 1.0E-6.SCALAR.FP64.true wcss.SCALAR.FP64.false _Var111.SCALAR.FP64
------------CP < _Var110.SCALAR.FP64.false _Var111.SCALAR.FP64.false _Var112.SCALAR.BOOLEAN
------------CP rmvar _Var110 _Var111
------------CP mvvar _Var112 __pred
------------GENERIC (lines 154-154) [recompile=false]
--------------CP assignvar 1.SCALAR.INT64.true term_code.SCALAR.INT64
----------ELSE
------------IF (lines 155-160)
--------------CP >= iter_count.SCALAR.INT64.false 5000.SCALAR.INT64.true _Var113.SCALAR.BOOLEAN
--------------CP mvvar _Var113 __pred
--------------GENERIC (lines 156-156) [recompile=false]
----------------CP assignvar 2.SCALAR.INT64.true term_code.SCALAR.INT64
------------ELSE
--------------IF (lines 157-160)
----------------CP createvar _mVar114 scratch_space//_pHidden_IP//_t0/temp72 true MATRIX binary 1 20 1000 -1 copy
----------------CP <= P_denom.MATRIX.FP64 0.SCALAR.INT64.true _mVar114.MATRIX.FP64
----------------CP uak+ _mVar114.MATRIX.FP64 _Var115.SCALAR.FP64 4
----------------CP rmvar _mVar114
----------------CP > _Var115.SCALAR.FP64.false 0.SCALAR.INT64.true _Var116.SCALAR.BOOLEAN
----------------CP rmvar _Var115
----------------CP mvvar _Var116 __pred
----------------GENERIC (lines 158-158) [recompile=false]
------------------CP assignvar 3.SCALAR.INT64.true term_code.SCALAR.INT64
--------------ELSE
----------------GENERIC (lines 160-160) [recompile=false]
------------------CP mvvar C C_old
----------------CP rmvar P_denom C
--------------CP rmvar P_denom C
------------CP rmvar P_denom C wcss_old
----------GENERIC (lines 160-160) [recompile=false]
------------CP mvvar C_new C
----------CP rmvar P minD P_denom D wcss_old C_new
--------GENERIC (lines 167-170) [recompile=false]
----------CP - run_index.SCALAR.INT64.false 1.SCALAR.INT64.true _Var117.SCALAR.INT64
----------CP * 20.SCALAR.INT64.true _Var117.SCALAR.INT64.false _Var118.SCALAR.INT64
----------CP rmvar _Var117
----------CP + _Var118.SCALAR.INT64.false 1.SCALAR.INT64.true _Var119.SCALAR.INT64
----------CP rmvar _Var118
----------CP * 20.SCALAR.INT64.true run_index.SCALAR.INT64.false _Var120.SCALAR.INT64
----------CP createvar _mVar121 scratch_space//_pHidden_IP//_t0/temp73 true MATRIX binary 200 972 1000 -1 copy
----------CP leftIndex All_Centroids.MATRIX.FP64 C.MATRIX.FP64 _Var119.SCALAR.INT64.false _Var120.SCALAR.INT64.false 1.SCALAR.INT64.true 972.SCALAR.INT64.true _mVar121.MATRIX.FP64
----------CP rmvar _Var119 _Var120
----------CP createvar _mVar122 scratch_space//_pHidden_IP//_t0/temp74 true MATRIX binary 10 1 1000 -1 copy
----------CP leftIndex final_wcss.MATRIX.FP64 wcss.SCALAR.FP64.false run_index.SCALAR.INT64.false run_index.SCALAR.INT64.false 1.SCALAR.INT64.true 1.SCALAR.INT64.true _mVar122.MATRIX.FP64
----------CP createvar _mVar123 scratch_space//_pHidden_IP//_t0/temp75 true MATRIX binary 10 1 1000 -1 copy
----------CP leftIndex termination_code.MATRIX.FP64 term_code.SCALAR.INT64.false run_index.SCALAR.INT64.false run_index.SCALAR.INT64.false 1.SCALAR.INT64.true 1.SCALAR.INT64.true _mVar123.MATRIX.FP64
----------CP createvar _mVar124 scratch_space//_pHidden_IP//_t0/temp76 true MATRIX binary 10 1 1000 -1 copy
----------CP leftIndex num_iterations.MATRIX.FP64 iter_count.SCALAR.INT64.false run_index.SCALAR.INT64.false run_index.SCALAR.INT64.false 1.SCALAR.INT64.true 1.SCALAR.INT64.true _mVar124.MATRIX.FP64
----------CP rmvar termination_code final_wcss num_iterations All_Centroids
----------CP cpvar _mVar121 All_Centroids
----------CP cpvar _mVar122 final_wcss
----------CP cpvar _mVar123 termination_code
----------CP cpvar _mVar124 num_iterations
----------CP rmvar _mVar121 _mVar122 _mVar123 _mVar124 term_code iter_count wcss
--------CP rmvar num_iterations C_old eps run_index term_code iter_count sumXsq wcss max_iter
------GENERIC (lines 175-184) [recompile=false]
--------CP createvar _mVar125 scratch_space//_pHidden_IP//_t0/temp77 true MATRIX binary 10 1 1000 -1 copy
--------CP seq 10 1 1000 1.SCALAR.INT64.true 10.SCALAR.INT64.true 1.SCALAR.INT64.true _mVar125.MATRIX.FP64
--------CP createvar _mVar126 scratch_space//_pHidden_IP//_t0/temp78 true MATRIX binary 10 -1 1000 -1 copy
--------CP ctableexpand _mVar125.MATRIX.FP64 termination_code.MATRIX.FP64 1.0.SCALAR.FP64.true -1.true -1.true _mVar126.MATRIX.FP64 false
--------CP rmvar _mVar125
--------CP mvvar _mVar126 _sbcvar5
------GENERIC (lines 175-184) [recompile=false]
--------CP createvar _mVar127 scratch_space//_pHidden_IP//_t0/temp79 true MATRIX binary 10 3 1000 0 copy
--------CP rand 10.SCALAR.INT64.true 3.SCALAR.INT64.true 1000 0 0 1.0 -1 uniform 1.0 4 _mVar127.MATRIX.FP64
--------CP ncol _sbcvar5.MATRIX.FP64.false _Var128.SCALAR.INT64
--------CP createvar _mVar129 scratch_space//_pHidden_IP//_t0/temp80 true MATRIX binary 10 3 1000 -1 copy
--------CP leftIndex _mVar127.MATRIX.FP64 _sbcvar5.MATRIX.FP64 1.SCALAR.INT64.true 10.SCALAR.INT64.true 1.SCALAR.INT64.true _Var128.SCALAR.INT64.false _mVar129.MATRIX.FP64
--------CP rmvar _mVar127 _Var128
--------CP createvar _mVar130 scratch_space//_pHidden_IP//_t0/temp81 true MATRIX binary 1 3 1000 -1 copy
--------CP uack+ _mVar129.MATRIX.FP64 _mVar130.MATRIX.FP64 4
--------CP createvar _mVar131 scratch_space//_pHidden_IP//_t0/temp82 true MATRIX binary 1 1 1000 -1 copy
--------CP rightIndex _mVar130.MATRIX.FP64 1.SCALAR.INT64.true 1.SCALAR.INT64.true 1.SCALAR.INT64.true 1.SCALAR.INT64.true _mVar131.MATRIX.FP64
--------CP castdts _mVar131.MATRIX.FP64.false _Var132.SCALAR.FP64
--------CP rmvar _mVar131
--------CP castvti _Var132.SCALAR.FP64.false _Var133.SCALAR.INT64
--------CP + Number of successful runs = .SCALAR.STRING.true _Var133.SCALAR.INT64.false _Var134.SCALAR.STRING
--------CP rmvar _Var133
--------CP createvar _mVar135 scratch_space//_pHidden_IP//_t0/temp83 true MATRIX binary 1 1 1000 -1 copy
--------CP rightIndex _mVar130.MATRIX.FP64 1.SCALAR.INT64.true 1.SCALAR.INT64.true 2.SCALAR.INT64.true 2.SCALAR.INT64.true _mVar135.MATRIX.FP64
--------CP castdts _mVar135.MATRIX.FP64.false _Var136.SCALAR.FP64
--------CP rmvar _mVar135
--------CP castvti _Var136.SCALAR.FP64.false _Var137.SCALAR.INT64
--------CP rmvar _Var136
--------CP + Number of incomplete runs = .SCALAR.STRING.true _Var137.SCALAR.INT64.false _Var138.SCALAR.STRING
--------CP rmvar _Var137
--------CP createvar _mVar139 scratch_space//_pHidden_IP//_t0/temp84 true MATRIX binary 1 1 1000 -1 copy
--------CP rightIndex _mVar130.MATRIX.FP64 1.SCALAR.INT64.true 1.SCALAR.INT64.true 3.SCALAR.INT64.true 3.SCALAR.INT64.true _mVar139.MATRIX.FP64
--------CP rmvar _mVar130
--------CP castdts _mVar139.MATRIX.FP64.false _Var140.SCALAR.FP64
--------CP rmvar _mVar139
--------CP castvti _Var140.SCALAR.FP64.false _Var141.SCALAR.INT64
--------CP rmvar _Var140
--------CP + Number of failed runs (with lost centroids) = .SCALAR.STRING.true _Var141.SCALAR.INT64.false _Var142.SCALAR.STRING
--------CP rmvar _Var141
--------CP print _Var134.SCALAR.STRING.false _Var143.SCALAR.STRING
--------CP rmvar _Var134
--------CP print _Var138.SCALAR.STRING.false _Var144.SCALAR.STRING
--------CP rmvar _Var138
--------CP print _Var142.SCALAR.STRING.false _Var145.SCALAR.STRING
--------CP rmvar _Var142
--------CP mvvar _Var132 num_successful_runs
--------CP cpvar _mVar129 termination_bitmap
--------CP rmvar _Var143 _Var144 _Var145 _mVar129 _sbcvar5 termination_code num_runs
------IF (lines 186-210)
--------CP > num_successful_runs.SCALAR.FP64.false 0.SCALAR.INT64.true _Var146.SCALAR.BOOLEAN
--------CP mvvar _Var146 __pred
--------GENERIC (lines 188-207) [recompile=false]
----------CP createvar _mVar147 scratch_space//_pHidden_IP//_t0/temp85 true MATRIX binary 10 1 1000 -1 copy
----------CP rightIndex termination_bitmap.MATRIX.FP64 1.SCALAR.INT64.true 10.SCALAR.INT64.true 1.SCALAR.INT64.true 1.SCALAR.INT64.true _mVar147.MATRIX.FP64
----------CP createvar _mVar148 scratch_space//_pHidden_IP//_t0/temp86 true MATRIX binary 10 1 1000 -1 copy
----------CP * final_wcss.MATRIX.FP64 _mVar147.MATRIX.FP64 _mVar148.MATRIX.FP64
----------CP uamax _mVar148.MATRIX.FP64 _Var149.SCALAR.FP64 4
----------CP * 10.SCALAR.INT64.true _Var149.SCALAR.FP64.false _Var150.SCALAR.FP64
----------CP + _Var150.SCALAR.FP64.false 10.SCALAR.INT64.true _Var151.SCALAR.FP64
----------CP rmvar _Var150
----------CP createvar _mVar152 scratch_space//_pHidden_IP//_t0/temp87 true MATRIX binary 10 1 1000 -1 copy
----------CP - 1.SCALAR.INT64.true _mVar147.MATRIX.FP64 _mVar152.MATRIX.FP64
----------CP rmvar _mVar147
----------CP createvar _mVar153 scratch_space//_pHidden_IP//_t0/temp88 true MATRIX binary 10 1 1000 -1 copy
----------CP +* _mVar148.MATRIX.FP64 _Var151.SCALAR.FP64.false _mVar152.MATRIX.FP64 _mVar153.MATRIX.FP64
----------CP rmvar _Var151 _mVar152
----------CP uamin _mVar153.MATRIX.FP64 _Var154.SCALAR.FP64 4
----------CP rmvar _mVar153
----------CP createvar _mVar155 scratch_space//_pHidden_IP//_t0/temp89 true MATRIX binary 10 1 1000 -1 copy
----------CP == _mVar148.MATRIX.FP64 _Var154.SCALAR.FP64.false _mVar155.MATRIX.FP64
----------CP createvar _mVar156 scratch_space//_pHidden_IP//_t0/temp90 true MATRIX binary 10 1 1000 -1 copy
----------CP ucumk+ _mVar155.MATRIX.FP64 _mVar156.MATRIX.FP64 4 false
----------CP rmvar _mVar155
----------CP createvar _mVar157 scratch_space//_pHidden_IP//_t0/temp91 true MATRIX binary 10 1 1000 -1 copy
----------CP == _mVar156.MATRIX.FP64 0.SCALAR.INT64.true _mVar157.MATRIX.FP64
----------CP rmvar _mVar156
----------CP uak+ _mVar157.MATRIX.FP64 _Var158.SCALAR.FP64 4
----------CP rmvar _mVar157
----------CP + _Var158.SCALAR.FP64.false 1.SCALAR.INT64.true _Var159.SCALAR.FP64
----------CP rmvar _Var158
----------CP castvti _Var159.SCALAR.FP64.false _Var160.SCALAR.INT64
----------CP rmvar _Var159
----------CP + Successful runs:  Best run is .SCALAR.STRING.true _Var160.SCALAR.INT64.false _Var161.SCALAR.STRING
----------CP + _Var161.SCALAR.STRING.false  with Centroid WCSS = .SCALAR.STRING.true _Var162.SCALAR.STRING
----------CP rmvar _Var161
----------CP + _Var162.SCALAR.STRING.false _Var154.SCALAR.FP64.false _Var163.SCALAR.STRING
----------CP rmvar _Var162 _Var154
----------CP + _Var163.SCALAR.STRING.false ;  Avg WCSS = .SCALAR.STRING.true _Var164.SCALAR.STRING
----------CP rmvar _Var163
----------CP uak+ _mVar148.MATRIX.FP64 _Var165.SCALAR.FP64 4
----------CP rmvar _mVar148
----------CP / _Var165.SCALAR.FP64.false num_successful_runs.SCALAR.FP64.false _Var166.SCALAR.FP64
----------CP rmvar _Var165
----------CP + _Var164.SCALAR.STRING.false _Var166.SCALAR.FP64.false _Var167.SCALAR.STRING
----------CP rmvar _Var164 _Var166
----------CP + _Var167.SCALAR.STRING.false ;  Worst WCSS = .SCALAR.STRING.true _Var168.SCALAR.STRING
----------CP rmvar _Var167
----------CP + _Var168.SCALAR.STRING.false _Var149.SCALAR.FP64.false _Var169.SCALAR.STRING
----------CP rmvar _Var168 _Var149
----------CP - _Var160.SCALAR.INT64.false 1.SCALAR.INT64.true _Var170.SCALAR.INT64
----------CP * 20.SCALAR.INT64.true _Var170.SCALAR.INT64.false _Var171.SCALAR.INT64
----------CP rmvar _Var170
----------CP + _Var171.SCALAR.INT64.false 1.SCALAR.INT64.true _Var172.SCALAR.INT64
----------CP rmvar _Var171
----------CP * 20.SCALAR.INT64.true _Var160.SCALAR.INT64.false _Var173.SCALAR.INT64
----------CP rmvar _Var160
----------CP createvar _mVar174 scratch_space//_pHidden_IP//_t0/temp92 true MATRIX binary -1 972 1000 -1 copy
----------CP rightIndex All_Centroids.MATRIX.FP64 _Var172.SCALAR.INT64.false _Var173.SCALAR.INT64.false 1.SCALAR.INT64.true 972.SCALAR.INT64.true _mVar174.MATRIX.FP64
----------CP rmvar _Var172 _Var173
----------CP createvar _mVar175 scratch_space//_pHidden_IP//_t0/temp93 true MATRIX binary 972 -1 1000 -1 copy
----------CP r' _mVar174.MATRIX.FP64 _mVar175.MATRIX.FP64 4
----------CP createvar _mVar176 scratch_space//_pHidden_IP//_t0/temp94 true MATRIX binary 3972 -1 1000 -1 copy
----------CP ba+* X.MATRIX.FP64 _mVar175.MATRIX.FP64 _mVar176.MATRIX.FP64 4
----------CP rmvar _mVar175
----------CP createvar _mVar177 scratch_space//_pHidden_IP//_t0/temp95 true MATRIX binary 3972 -1 1000 -1 copy
----------CP * -2.SCALAR.INT64.true _mVar176.MATRIX.FP64 _mVar177.MATRIX.FP64
----------CP rmvar _mVar176
----------CP createvar _mVar178 scratch_space//_pHidden_IP//_t0/temp96 true MATRIX binary -1 1 1000 -1 copy
----------CP uarsqk+ _mVar174.MATRIX.FP64 _mVar178.MATRIX.FP64 4
----------CP createvar _mVar179 scratch_space//_pHidden_IP//_t0/temp97 true MATRIX binary 1 -1 1000 -1 copy
----------CP r' _mVar178.MATRIX.FP64 _mVar179.MATRIX.FP64 4
----------CP rmvar _mVar178
----------CP createvar _mVar180 scratch_space//_pHidden_IP//_t0/temp98 true MATRIX binary 3972 -1 1000 -1 copy
----------CP + _mVar177.MATRIX.FP64 _mVar179.MATRIX.FP64 _mVar180.MATRIX.FP64
----------CP rmvar _mVar177 _mVar179
----------CP createvar _mVar181 scratch_space//_pHidden_IP//_t0/temp99 true MATRIX binary 3972 1 1000 -1 copy
----------CP uarmin _mVar180.MATRIX.FP64 _mVar181.MATRIX.FP64 4
----------CP createvar _mVar182 scratch_space//_pHidden_IP//_t0/temp100 true MATRIX binary 3972 -1 1000 -1 copy
----------CP <= _mVar180.MATRIX.FP64 _mVar181.MATRIX.FP64 _mVar182.MATRIX.FP64
----------CP rmvar _mVar180 _mVar181
----------CP createvar _mVar183 scratch_space//_pHidden_IP//_t0/temp101 true MATRIX binary -1 3972 1000 -1 copy
----------CP r' _mVar182.MATRIX.FP64 _mVar183.MATRIX.FP64 4
----------CP rmvar _mVar182
----------CP createvar _mVar184 scratch_space//_pHidden_IP//_t0/temp102 true MATRIX binary -1 3972 1000 -1 copy
----------CP ucumk+ _mVar183.MATRIX.FP64 _mVar184.MATRIX.FP64 4 false
----------CP rmvar _mVar183
----------CP createvar _mVar185 scratch_space//_pHidden_IP//_t0/temp103 true MATRIX binary 3972 -1 1000 -1 copy
----------CP r' _mVar184.MATRIX.FP64 _mVar185.MATRIX.FP64 4
----------CP rmvar _mVar184
----------CP createvar _mVar186 scratch_space//_pHidden_IP//_t0/temp104 true MATRIX binary 3972 -1 1000 -1 copy
----------CP == _mVar185.MATRIX.FP64 0.SCALAR.INT64.true _mVar186.MATRIX.FP64
----------CP rmvar _mVar185
----------CP createvar _mVar187 scratch_space//_pHidden_IP//_t0/temp105 true MATRIX binary 3972 1 1000 -1 copy
----------CP uark+ _mVar186.MATRIX.FP64 _mVar187.MATRIX.FP64 4
----------CP rmvar _mVar186
----------CP createvar _mVar188 scratch_space//_pHidden_IP//_t0/temp106 true MATRIX binary 3972 1 1000 -1 copy
----------CP + _mVar187.MATRIX.FP64 1.SCALAR.INT64.true _mVar188.MATRIX.FP64
----------CP rmvar _mVar187
----------CP print _Var169.SCALAR.STRING.false _Var189.SCALAR.STRING
----------CP rmvar _Var169
----------CP print DONE..SCALAR.STRING.true _Var190.SCALAR.STRING
----------CP rmvar _Var189
----------CP cpvar _mVar174 C
----------CP cpvar _mVar188 Y
----------CP rmvar _Var190 _mVar174 _mVar188 termination_bitmap num_successful_runs final_wcss is_verbose X All_Centroids num_centroids
------ELSE
--------GENERIC (lines 210-210) [recompile=false]
----------CP stop No output is produced.  Try increasing the number of iterations and/or runs..SCALAR.STRING.true _Var191.SCALAR.STRING
----------CP rmvar _Var191
--------CP rmvar termination_bitmap num_successful_runs final_wcss is_verbose X All_Centroids num_centroids
----FUNCTION .defaultNS::get_sample_maps [recompile=true]
------GENERIC (lines 222-252) [recompile=false]
--------CP createvar _mVar21 scratch_space//_pHidden_IP//_t0/temp0 true MATRIX binary 1316 10 1000 13160 copy
--------CP rand 1316.SCALAR.INT64.true 10.SCALAR.INT64.true 1000 0.0 1.0 1.0 -1 uniform 1.0 4 _mVar21.MATRIX.FP64
--------CP createvar _mVar22 scratch_space//_pHidden_IP//_t0/temp1 true MATRIX binary 1316 10 1000 -1 copy
--------CP log _mVar21.MATRIX.FP64 _mVar22.MATRIX.FP64 4 false
--------CP rmvar _mVar21
--------CP createvar _mVar23 scratch_space//_pHidden_IP//_t0/temp2 true MATRIX binary 1316 10 1000 -1 copy
--------CP / _mVar22.MATRIX.FP64 -0.2900346193274135.SCALAR.FP64.true _mVar23.MATRIX.FP64
--------CP rmvar _mVar22
--------CP createvar _mVar24 scratch_space//_pHidden_IP//_t0/temp3 true MATRIX binary 1316 10 1000 -1 copy
--------CP + _mVar23.MATRIX.FP64 0.5.SCALAR.FP64.true _mVar24.MATRIX.FP64
--------CP rmvar _mVar23
--------CP createvar _mVar25 scratch_space//_pHidden_IP//_t0/temp4 true MATRIX binary 1316 10 1000 -1 copy
--------CP round _mVar24.MATRIX.FP64 _mVar25.MATRIX.FP64
--------CP rmvar _mVar24
--------CP createvar _mVar26 scratch_space//_pHidden_IP//_t0/temp5 true MATRIX binary 1316 10 1000 -1 copy
--------CP ucumk+ _mVar25.MATRIX.FP64 _mVar26.MATRIX.FP64 4 false
--------CP rmvar _mVar25
--------CP createvar _mVar27 scratch_space//_pHidden_IP//_t0/temp6 true MATRIX binary 1316 10 1000 -1 copy
--------CP <= _mVar26.MATRIX.FP64 3972.SCALAR.INT64.true _mVar27.MATRIX.FP64
--------CP createvar _mVar28 scratch_space//_pHidden_IP//_t0/temp7 true MATRIX binary 13160 1 1000 -1 copy
--------CP rshape _mVar27.MATRIX.FP64 13160.SCALAR.INT64.true 1.SCALAR.INT64.true -1.SCALAR.STRING.true false.SCALAR.BOOLEAN.true _mVar28.MATRIX.FP64
--------CP createvar _mVar29 scratch_space//_pHidden_IP//_t0/temp8 true MATRIX binary 13160 1 1000 -1 copy
--------CP seq 13160 1 1000 13160.SCALAR.INT64.true 1.SCALAR.INT64.true -1.SCALAR.INT64.true _mVar29.MATRIX.FP64
--------CP createvar _mVar30 scratch_space//_pHidden_IP//_t0/temp9 true MATRIX binary 13160 1 1000 -1 copy
--------CP * _mVar28.MATRIX.FP64 _mVar29.MATRIX.FP64 _mVar30.MATRIX.FP64
--------CP rmvar _mVar28 _mVar29
--------CP createvar _mVar31 scratch_space//_pHidden_IP//_t0/temp10 true MATRIX binary 13160 1 1000 -1 copy
--------CP - 13161.SCALAR.INT64.true _mVar30.MATRIX.FP64 _mVar31.MATRIX.FP64
--------CP rmvar _mVar30
--------CP createvar _mVar32 scratch_space//_pHidden_IP//_t0/temp11 true MATRIX binary 13160 1 1000 -1 copy
--------CP seq 13160 1 1000 0.SCALAR.INT64.true 13159.SCALAR.INT64.true 1.SCALAR.INT64.true _mVar32.MATRIX.FP64
--------CP createvar _mVar33 scratch_space//_pHidden_IP//_t0/temp12 true MATRIX binary 13160 1 1000 -1 copy
--------CP / _mVar32.MATRIX.FP64 1316.SCALAR.INT64.true _mVar33.MATRIX.FP64
--------CP rmvar _mVar32
--------CP createvar _mVar34 scratch_space//_pHidden_IP//_t0/temp13 true MATRIX binary 13160 1 1000 -1 copy
--------CP + 0.5.SCALAR.FP64.true _mVar33.MATRIX.FP64 _mVar34.MATRIX.FP64
--------CP rmvar _mVar33
--------CP createvar _mVar35 scratch_space//_pHidden_IP//_t0/temp14 true MATRIX binary 13160 1 1000 -1 copy
--------CP round _mVar34.MATRIX.FP64 _mVar35.MATRIX.FP64
--------CP rmvar _mVar34
--------CP createvar _mVar36 scratch_space//_pHidden_IP//_t0/temp15 true MATRIX binary -1 -1 1000 -1 copy
--------CP ctable _mVar31.MATRIX.FP64 _mVar35.MATRIX.FP64 1.0.SCALAR.FP64.true -1.true -1.true _mVar36.MATRIX.FP64 false
--------CP rmvar _mVar31 _mVar35
--------CP cpvar _mVar36 _sbcvar0
--------CP cpvar _mVar26 _sbcvar3
--------CP cpvar _mVar27 _sbcvar4
--------CP rmvar _mVar26 _mVar27 _mVar36
------GENERIC (lines 222-252) [recompile=true]
--------CP createvar _mVar37 scratch_space//_pHidden_IP//_t0/temp16 true MATRIX binary 1316 10 1000 -1 copy
--------CP * _sbcvar3.MATRIX.FP64 _sbcvar4.MATRIX.FP64 _mVar37.MATRIX.FP64
--------CP createvar _mVar38 scratch_space//_pHidden_IP//_t0/temp17 true MATRIX binary 1316 10 1000 -1 copy
--------CP - 1.SCALAR.INT64.true _sbcvar4.MATRIX.FP64 _mVar38.MATRIX.FP64
--------CP createvar _mVar39 scratch_space//_pHidden_IP//_t0/temp18 true MATRIX binary 1316 10 1000 -1 copy
--------CP +* _mVar37.MATRIX.FP64 3973.SCALAR.INT64.true _mVar38.MATRIX.FP64 _mVar39.MATRIX.FP64
--------CP rmvar _mVar37 _mVar38
--------CP createvar _mVar40 scratch_space//_pHidden_IP//_t0/temp19 true MATRIX binary 13160 1 1000 -1 copy
--------CP rshape _mVar39.MATRIX.FP64 13160.SCALAR.INT64.true 1.SCALAR.INT64.true -1.SCALAR.STRING.true false.SCALAR.BOOLEAN.true _mVar40.MATRIX.FP64
--------CP rmvar _mVar39
--------CP createvar _mVar41 scratch_space//_pHidden_IP//_t0/temp20 true MATRIX binary 13160 3972 1000 -1 copy
--------CP rexpand cast=true max=3972 ignore=false dir=cols target=_mVar40 k=4 _mVar41.MATRIX.FP64
--------CP rmvar _mVar40
--------CP ncol _sbcvar0.MATRIX.FP64.false _Var42.SCALAR.INT64
--------CP createvar _mVar43 scratch_space//_pHidden_IP//_t0/temp21 true MATRIX binary 13160 -1 1000 -1 copy
--------SPARK rightIndex _sbcvar0.MATRIX.FP64 1.SCALAR.INT64.true 13160.SCALAR.INT64.true 1.SCALAR.INT64.true _Var42.SCALAR.INT64.false _mVar43.MATRIX.FP64 MULTI_BLOCK
--------CP rmvar _Var42
--------CP assignvar 1316.SCALAR.INT64.true sample_block_size.SCALAR.INT64
--------CP cpvar _mVar41 sample_maps
--------CP cpvar _mVar43 sample_col_map
--------CP rmvar _mVar41 _mVar43 _sbcvar3 _sbcvar2 approx_sample_size _sbcvar4 num_records _sbcvar1 num_samples _sbcvar0
--MAIN PROGRAM
----GENERIC (lines 1-2) [recompile=false]
------CP createvar _mVar192 scratch_space//_pHidden_IP//_t0/temp107 true MATRIX binary 3972 972 1000 3860784 copy
------CP rand 3972.SCALAR.INT64.true 972.SCALAR.INT64.true 1000 0.0 1.0 1.0 -1 uniform 1.0 4 _mVar192.MATRIX.FP64
------CP extfunct .defaultNS m_kmeans 7 0 X=_mVar192.MATRIX.FP64 k=20.SCALAR.INT64.true runs=10.SCALAR.INT64.true max_iter=5000.SCALAR.INT64.true eps=1.0E-6.SCALAR.FP64.true is_verbose=false.SCALAR.BOOLEAN.true avg_sample_size_per_centroid=50.SCALAR.INT64.true
------CP rmvar _mVar192

BEGIN K-MEANS SCRIPT
Taking data samples for initialization...
Initializing the centroids for all runs...
Performing k-means iterations for all runs...
Number of successful runs = 10
Number of incomplete runs = 0
Number of failed runs (with lost centroids) = 0
Successful runs:  Best run is 4 with Centroid WCSS = 318077.0285004936;  Avg WCSS = 318146.42760973383;  Worst WCSS = 318195.0570530448
DONE.
SystemDS Statistics:
Total execution time:           227.914 sec.
Number of executed Spark inst:  0.

Hi Mark, Can you have a look at this?

Hi @Baunsgaard, can you have a look at this?
Review aid: (Just in case!) 😸

Dev notebook, opens in google colab
The last two code cells of this notebooks are for pasting the snippet.

Hi @j143, i missed this PR and comment.
I think it is fine with the dev notebook, but i don't like that so much of it is setup. Could this not be simplified? Also where do you want notebook to be? somewhere in the documentation?
I don't see a link to such a thing inside the code of the PR.
With regards to changing the K-means algorithm then having something else than 0 as the defaults seems reasonable, but maybe select 2? or better yet, force the user to select an value, since in the end that is something they have to do.
If you look at spark they set it http://spark.apache.org/docs/latest/ml-clustering.html
or R requires the number of clusters https://www.rdocumentation.org/packages/stats/versions/3.6.2/topics/kmeans

Thank you for the review!  @Baunsgaard :)
BTW, the notebook is for testing out dml scripts on master w/o firing it on a PC.
For example, while reviewing #925, I have used a colab notebook to reproduce the results of the PR author and
shared the resulting notebook.

Thank you for the review. :tada:

repo: apache/systemml


text:
convert net order to host order for double type

[ ] Did you create an Apache Jira ticket?  (not required for trivial changes)
[ ] Does your pull request title follow the pattern "THRIFT-NNNN: describe my issue"?  (not required for trivial changes)
[ ] Did you squash your changes to a single commit?
[ ] Did you do your best to avoid breaking changes?  If one was needed, did you label the Jira ticket with "Breaking-Change"

This broke all the lua tests.  My take is that the fix is unnecessary, as the cross-language tests were working before this fix (and they test doubles), and now they are broken.

@jeking3 before the fix. lua version can't send double to java or c# correctly 'cause the order issue. not sure other tests

Our cross language tests send doubles from lua client to java server and back:
java-lua                binary            framed-ip                success
java-lua                binary            buffered-ip              success
On the client lua side here is the code:
https://github.com/apache/thrift/blob/master/test/lua/test_basic_client.lua#L147
On the server java side here is the code:
https://github.com/apache/thrift/blob/master/lib/java/test/org/apache/thrift/server/ServerTestBase.java#L89
The reverse where Java is the client:
https://github.com/apache/thrift/blob/master/lib/java/test/org/apache/thrift/test/TestClient.java#L282
and lua is the server:
https://github.com/apache/thrift/blob/master/test/lua/test_basic_server.lua#L57
HOWEVER we do not seem to be running any tests where lua is the server.  That's a bug, but I still feel the round trip from lua client through java server disproves what you said.
Is there something deficient in these tests we're missing?  We are proving that doubles successfully round-trip with these tests.  Are we testing enough values and enough boundaries to prove or disprove what you are saying?

Closing based on comments.

repo: apache/thrift


text:
Support max http request header size and max http header size configuration separation

Mail link：https://bz.apache.org/bugzilla/show_bug.cgi?id=65866

Please review the code first, if there is no problem, I will support this on other branches as well.

That seems pretty straightforward. I'm slightly skeptical about the real practical use since trying to optimize the sizes too much is risky, but no problem, this is the user's responsibility.

That seems pretty straightforward. I'm slightly skeptical about the real practical use since trying to optimize the sizes too much is risky, but no problem, this is the user's responsibility.

@rmaucher
I saw some discussion about this, you can refer to it:
https://github.com/spring-projects/spring-boot/issues/29382

+1 I see two advantages:

enables cases where "front" tomcat has big incoming headers but "normal" forwarded headers
enables better web server abstractions which often have this configuration

Open question: shouldn't setMaxHttpHeaderSize set both request and response ones and maxHttpHeaderSize field be dropped for backward compat?

Overall, I prefer the PR as it is right now, it's simpler. If the new setters are used, then the behavior could be slightly different, but since this is new API this should be fine.

Overall, I prefer the PR as it is right now, it's simpler. If the new setters are used, then the behavior could be slightly different, but since this is new API this should be fine.

It is not complicated. I forked and edit online to demonstrate the part of change: https://github.com/gmshake/tomcat/commit/75289cf9e7a3920cc6d3ea0463db53a6ab0d090c?diff=split

The proposed patch is still simpler and should work well, so I will merge that one.

@rmaucher
I can patch several other branches, right?

Merged with changed. I didn't understand the changes to TestHttp2Limits, HTTP/2 should use the new request header size and that should be it. Also updated docs and changelog.

@rmaucher
Do i need to resubmit a clean pr?

authors: CrazyHZM gmshake rmannibucau rmaucher

repo: apache/tomcat


text:
Fix misspellings found in documentation

What does this PR (Pull Request) do?

Fixes a few misspellings found in the documentation

[X] This PR is not related to any Issue

Which Traffic Control components are affected by this PR?

Documentation

What is the best way to verify this PR?

Build the docs and check the updated pages.
If this is a bug fix, what versions of Traffic Control are affected?

The following criteria are ALL met by this PR

[X] This PR includes tests OR I have explained why tests are unnecessary
[X] This PR includes documentation OR I have explained why documentation is unnecessary
[X] This PR includes an update to CHANGELOG.md OR such an update is not necessary
[X] This PR includes any and all required license headers
[X] This PR ensures that database migration sequence is correct OR this PR does not include a database migration
[X] This PR DOES NOT FIX A SERIOUS SECURITY VULNERABILITY (see the Apache Software Foundation's security guidelines for details)

Additional Information

Can one of the admins verify this patch?

Add to whitelist

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/4884/

authors: ZugNZwang asf-ci rawlinp

repo: apache/trafficcontrol


text:
Update descriptions of sni.yaml.default

Update the descriptions of sni.yaml.default about the following two things.

Replaceverify_origin_server to verify_server_policy.
verify_origin_server is old setting of ssl_server_name.yaml in version 8.1.x.
Add descriptions about some keys that were not explained.

[approve ci]

Looks good to me.

Cherry-picked to v9.2.x

authors: bneradt shinrich takkitano zwoop

repo: apache/trafficserver


text:
Cleanup: Collect micro-benchmark programs

Cosmetic change to collect Catch2 based micro-benchmark programs in one place (tools/benchmark/) and building them as noinst_PROGRAMS.

Rebased on the latest master.


text:
[ci] Use custom sphinx-gallery in ci_gpu

To enable #10921 this builds sphinx-gallery using @guberti's changes. Patch was created via
git clone https://github.com/sphinx-gallery/sphinx-gallery.git
cd sphinx-gallery
curl -L https://github.com/sphinx-gallery/sphinx-gallery/pull/940.diff | patch -p1 -d .
curl -L https://github.com/sphinx-gallery/sphinx-gallery/pull/941.diff | patch -p1 -d .
git diff > sphinx_gallery_colab_fixes.patch

cc @areusch

Thanks for the patch @driazati! For completeness, it's worth noting I had to make a third change to sphinx-gallery to get Python notebooks working again (https://github.com/sphinx-gallery/sphinx-gallery/pull/943). That change was merged last night though, so we don't need to do anything on our end.

closing for now since we're going to revisit how we handle tutorials in a more generic way and this will make updating versions more difficult

authors: driazati guberti

repo: apache/tvm


text:
[TIR] Expose MMA-related PTX builtins

op list:
ptx_mma
ptx_mma_sp
mma_store
mma_fill
Used and tested for future TVMScript parser
Co-Authored-By: yongwww <55wuyong@163.com>
cc @junrushao1994

I'm glad to provide docstring instructions for this PR and #12428

See our discussion thread here. Let's do one PR at a time :-)


text:
[TIR][Bugfix] Fix AXIS_SEPARATORS in tir.Schedule.transform_layout

Preivously, the block SREF reuse only included a single step of changes, and would have an incorrect mapping if multiple sequential changes to the TIR block occurred.  This could happen if a BufferStore was updated, followed by replacement of Block iter vars/values.  This commit tracks the Block replacements across each usage, to ensure the SREF instances remain valid.

Thank you @Lunderberg for this change!

authors: Lunderberg abhikran-quic


text:
[Unity] Relax VM

This PR implements a flexible register-based VM to execute relax programs with dynamic shape and control flow. Design: https://github.com/tlc-pack/relax/wiki/Relax-VM-Design.
Co-Authored-by: Ziheng Jiang ziheng@apache.org
Co-Authored-by: Ruihang Lai ruihangl@cs.cmu.edu
Co-Authored-by: Sunghyun Park 49998730+sunggg@users.noreply.github.com
Co-Authored-by: Junru Shao junrushao1994@gmail.com
Co-Authored-by: Prakalp Srivastava prakalp@octoml.ai
Co-Authored-by: Yong Wu yongcale@gmail.com
Co-Authored-by: Steven S. Lyubomirsky slyubomirsky@octoml.ai
Co-Authored-by: Tianqi Chen tianqi.tchen@gmail.com
Co-Authored-by: Hongyi Jin 3231950289@qq.com

cc @tqchen

let's fix the lint and get it in!

authors: YuchenJin junrushao


text:
[Unity][CODEGEN] Fix metal codegen when with only single working dim

This PR fixes the metal introduces by a previous commit that removes the workdim remapping that caused issues for kernels with only threadIdx.x and blockIdx.x

cc @Lunderberg was caused by recent merging that bought in #14495 . This is a hotfix, will also need send to main

right i think it should be covered by tests already but was not run due to metal being osx only


text:
ZEPPELIN-1267. PySparkInterpreter doesn't work in spark 2.0

What is this PR for?
PySparkInterpreter doesn't work in spark 2.0 because pyspark and py4j is not distributed to executors.  This PR extract the setup staff for pyspark interpreter into method setupConfForPySpark and use it for both spark1 and spark2. But this is just a short term solution, as I think this should be handled by spark rather than zeppelin, here zeppelin duplicate part of spark's work. In the long term, I'd like to resolve it in ZEPPELIN-1263.
What type of PR is it?
[Bug Fix]
Todos

https://issues.apache.org/jira/browse/ZEPPELIN-1263

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-1267

How should this be tested?
Verify it manually.
Screenshots (if appropriate)

Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

\cc @jongyoul for review

Looks good to me.

LGTM.

Let's merge if there is no further discussion!

@zjffdu I tried current master branch with master set to local[*], Spark standalone and yarn-client both with/without SPARK_HOME set. But somehow I was able to run pyspark interpreter without this patch. I don't know what I missed, could you tell me which environment should I try to make pyspark interpreter fail on master branch?

I also use the latest master and build zeppelin with this command:
```, and then export `SPARK_HOME` to where spark-2.0 is located and run pyspark interpreter using `yarn-client` mode.

@zjffdu thank you for quick response, I was able to reproduce it and tested this patch fixes the issue!
Merge if there is no further discussion

authors: Leemoonsoo bzz jongyoul minahlee zjffdu

repo: apache/zeppelin

license_source: bigquery


text:
[ZEPPELIN-5900] Fix RemoteSchedulerTest

What is this PR for?
This PR changes the RemoteScheduler
The main change is the executor of the RemoteScheduler. It's now a separate FIFO Executor (one Thread). As a result, jobs are now queued in the Zeppelin server and not in the RemoteInterpreter.
Since previously the jobs were queued in the remote interpreter due to parallelism in the Zeppelin server, the cancellation of jobs did not work, which is why the test failed.
The interrupt behaviour has also been improved.
What type of PR is it?
Bug Fix
What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-5900
How should this be tested?

CI

Questions:

Does the license files need to update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

As this PR changes a very important component of Zeppelin, I need an accurate review. If you want me to describe the changes in more detail, please let me know.

Thanks @Reamer  so much for fixing the failed test case ! It's a great job.
It's weird that the RemoteSchedulerTest runs well under Junit 4, Do you know why?

BTW it's weird that the RemoteSchedulerTest used to runs fine on Junit 4, Do you know why?

Unfortunately no.
However, since only the timing matters due to the six threads involved, JUnit-5 may have improved things here.
But perhaps the CI-VM is so slow that the error occurs less frequently there. However, I have seen the error several times under JUnit 4.

Does this FIFO Executor make global share interpreter cannot run in concurrently  and invalidate parameter like zeppelin.jdbc.concurrent.use ？ @Reamer

I am currently on vacation without a PC. Unfortunately, I don't know the implemention of the configuration. Can you send me code links where the configuration is used.

The configuration zeppelin.jdbc.concurrent.use is used in function isConcurrentExecution(). You can find it in codes above

authors: Reamer huage1994 kevinjmh


text:
[release] 0.47.2

Reviewer instructions

Check the changelog to ensure the version increment is consistent with https://semver.org/.

If a different version should be released, follow the instructions.

Review the changeset as a sanity check.
Approve the PR, if it pases muster.
Merge this branch (do not squash or rebase), but DO NOT DELETE THE BRANCH.
After the release has been published to github, delete this branch.

Release notes
0.47.2 - 2024-12-16

@bugarela I have repaired the release process. If you need a new release in Quint, here it is.

@bugarela I have repaired the release process. If you need a new release in Quint, here it is.

Thanks! I'll aim to update Quint before the holidays.

authors: bugarela coffeeinprogress konnov

repo: apalache-mc/apalache


text:
Added tests to readNumber and readStringSlowPath

Added four unit tests to functions readNumber and readStringSlowPath in IterImplForStreaming.java to increase branch coverage.
This should resolve #69 and resolve #73

Codecov Report

Merging #74 into lab3 will increase coverage by 0.31%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             lab3      #74      +/-   ##
==========================================
+ Coverage   68.31%   68.62%   +0.31%
==========================================
  Files         107      107
  Lines        7334     7334
  Branches     1388     1388
==========================================
+ Hits         5010     5033      +23
+ Misses       1868     1850      -18
+ Partials      456      451       -5

Impacted Files
Coverage Δ

...c/main/java/com/jsoniter/IterImplForStreaming.java
76.48% <0%> (+6.21%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d3167fe...99c193a. Read the comment docs.

authors: Jonagoo codecov-io

repo: apeinot/java

license: MIT


text:
Add CSS class based on provider name to SAML delegated login links

This will allow a deployer to style the link in login-providers.html (e.g. to add an organizational logo) using a CSS class based on the provider's name (e.g. "This Organization (Cool People)" would result in a CSS class called "This-Organization--Cool-People-").
Deployers will need to provide their own CSS to leverage this, either by modifiying cas.css in the overlay, or updating templates/layout.html to include a different CSS file.

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@rrenomeron I see you have made a number of changes here. Would you please merge with master once so we can revisit this properly?

Going to close this for now, due to inactivity. At any time if you wanted to resume work on this, please post back and we'd be happy to reopen and continue.

So I've rebased to a relatively recent commit and pre-compiled the search-and-replace pattern to address the concerns about performance.  I'd like to re-open, please, esp if there's a chance this could get into 5.2.0-GA.

Of course. Thank you for the follow-up.

Also, do you mind if I ask you to squash commits as well? I normally don't care, but given this dates back to a few months ago and is likely going to increment further, it would be good to consolidate.

Ugh, what a mess.  I think I just might start fresh with a new branch, with the size of the actual changes I think it will be less work than fighting with git (which I've already done too much of).  Will doing it this way require a new PR?

Yes it's likely much easier to close this one and start anew.

Going to close this for now.

authors: CLAassistant mmoayyed rrenomeron

repo: apereo/cas


text:
Re-add player list UI

We can create a placeholder player list UI while we wait for UI designs. It should mimic the old design pretty closely.

Nice work!

repo: apetavern/grubs


text:
TypeError when using Typed Arrays for data

Bug report
Codepen
https://codepen.io/asder/pen/xxwYaym
Explanation
I don't know if typed arrays are supported. But it seems they work fine (zooming - dragging etc.). Except when you pressing the reset zoom button. Where it gives the following error
apexcharts:6 Uncaught (in promise) TypeError: t[l].data.map is not a function
    at t.value (apexcharts:6)
    at t.value (apexcharts:6)
    at t.value (apexcharts:14)
    at t.create (apexcharts:6)
    at apexcharts:14
    at new Promise (<anonymous>)
    at t.value (apexcharts:14)
    at i.value (apexcharts:6)
    at apexcharts:6
    at Array.forEach (<anonymous>)

any ? :\


text:
Support Marker Comments

What would be involved in supporting marker comments in this extension? I would love to be able to use the section markers popularised by Xcode in my Typescript projects in Nova. In particular, getting a horizontal divider in the editor (similar or identical to Nova’s 'type and function separators'), and being able to browse the markers in the symbol list.
// MARK: - My Section Title
I’d be willing to propose a PR for this if anyone can suggest whether it’s actually possible within the extension API and roughly how to approach it.

I personally think this should actually live outside of this extension as a separate extension, since it's not a feature of the typescript language, and would apply to any language.

authors: apexskier danhalliday

repo: apexskier/nova-typescript


text:
Optimize signer wallet derivation by pre-computing the private key

Currently, the node derives the Airnode wallet from the mnemonic to be able to sign fulfillment data. This is a computationally heavy operation that gets repeated with each request/gateway API call, which bloats our compute time. Instead, we should pre-compute the private key from the mnemonic at deployment and use that to sign data. Derivation from mnemonic is only required for deriving sponsor wallets.
See the thread for more information.

Not exactly. As mentioned in the Slack post we don't want to modify the config file but instead, add an environment variable with the derived private key. Similar to how we're currently adding the gateway URL and passing it to the heartbeat.

We'd need to keep the existing functionality for local use however, right?
I don't think we need to though. I think we can do the same for the non-cloud version.

@kolenic-martin To test this all you need to do is try that an airnode-example deployment/removal works so this can be tested alongside something else.

@kolenic-martin Just to clarify, please, test both local and cloud versions (both AWS and GCP actually) because there are changes all over the place.

Hi @amarthadan, I've tested this as a part of another test, running Airnode (the coingecko example) both locally and on the two cloud providers. I did not note any issues and both the deployment and the request have been performed correctly.

authors: amarthadan kolenic-martin vponline


text:
bug: address & connector from useAccount are undefined

When I test our connection button with v3 of starknet-react, the value of account from useAccount is defined correctly, isConnected is true,  but address is undefined and connector is undefined.
Here is how I connect :
import { InjectedConnector } from "starknetkit/injected";
import { Connector } from "@starknet-react/core";

 const connectors = [
    new InjectedConnector({ options: { id: "argentX" } }),
    new InjectedConnector({ options: { id: "braavos" } }),
  ];
 <StarknetConfig
              chains={chains}
              provider={providers}
              connectors={connectors as Connector[]}
              autoConnect
            ></StarknetConfig>

import { Connector, useConnect } from "@starknet-react/core";

const { connectAsync } = useConnect();
const connectWallet = async (connector: Connector) => {
    await connectAsync({ connector });
}

I'm testing with
"@starknet-react/chains": "^3.0.0-beta.1",
"@starknet-react/core": "^3.0.0-beta.1",
"starknetkit": "^2.2.15",
"starknet": "^6.11.0",

Thank you for the report! I will check what's going on and fix it.

Hey @jrmncos if you are able to reproduce and can open a PR with a fix I'm happy to review it. Notice this is with the code in the develop branch.

Is someone working on it? I'm currently debugging this issue, might send a patch PR later if I figure what's going on.

I couldn't reproduce the issue. If you can and can submit a fix I'll be happy to review it!

I couldn't reproduce the issue. If you can and can submit a fix I'll be happy to review it!

I got the issue in my work project and this sandbox here for reference. I'm still investigating the issue, but so far I understand that the address is set when connected, but then reverts back to an empty string, while the connector is always undefined.
Tested with the following wallet versions:
Argent X: v5.17.8
Braavos: v3.69.2

I couldn't reproduce the issue. If you can and can submit a fix I'll be happy to review it!

I got the issue in my work project and this sandbox here for reference. I'm still investigating the issue, but so far I understand that the address is set when connected, but then reverts back to an empty string, while the connector is always undefined.
Tested with the following wallet versions: Argent X: v5.17.8 Braavos: v3.69.2
Chromium Engine Version 127.0.6533.100

That's super helpful thank you!

I'm currently debugging this issue. Can anyone help me?

hey @fracek can you help me?

Hey @irisdv @kseikyo , this issue has been fixed in the latest beta release. Please update the starknet-react and starknet packages to the latest versions. Here's the updated sandbox if it helps.
Also thank you @NGUYENDANGVU-coder  and @jrmncos  for offering to help with this issue. it's already been resolved now, so no further action is needed. : )

authors: NGUYENDANGVU-coder fracek irisdv jaipaljadeja kseikyo

repo: apibara/starknet-react


text:
ContentTest - Added "oneOf" check

ContentTest - Added "oneOf" check for "responseBody" & "responseHeader"

Linked to #313

nice!

@nicklloyd Are you ok with the "oneOf" option? or prefer a different term for the check?

LGTM! 🤘🏻

repo: apideck-libraries/portman


text:
Dependencies

All peer dependencies should be probably installed as dependencies for simple usage.

Maybe we can do eslint as a peer dependency and all the others as dependencies. This way, we could (probably?) only have eslint and eslint-config-apify in our projects' package.jsons and everything else will be hidden inside this package.
I think eslint is needed as a peer dependency because IDEs and other tools might be checking for the presence of eslint in package.json to enable eslint specific features. I've encountered this with jest, where Webstorm would not show test integration tools unless jest was declared, even though it was installed in node_modules.

I am not sure if it is possible. I think that even if you require a configuration, eslint still searches for the plugins inside the project (first level) node_modules. I've made a branch  feature/dependencies with such a package.json, and I'm not able to link it, but the issue could be related to npm link not to the package itself.

I published two packages with two different variants.

With the eslint in peerDependencies https://www.npmjs.com/package/@petrpatek/eslint-config-apify
without peerDependencies https://www.npmjs.com/package/@petrpatek/eslint-apify

authors: mnmkng petrpatek

repo: apifytech/eslint-config-apify


text:
SUGGESTION: Return error code 400 for required path parameter.

Given the following scenario:
paths:
  /example/{searchString}:
    get:
      description: "Searches for and retrieves the information for the search string specified."
      parameters:
      - name: "searchString"
        in: "path"
        description: "The string on which to search."
        required: true
        type: "string"

if i hit GET: /example/ without any searchString I get a 404.
Given that I dont have a path implemented in my swagger for /, this unlikely to be a case of 'NOT FOUND' but more likely an API consumer making a 'BAD REQUEST' to the /example/{searchString} endpoint.
As such would it not be better for the validator to return 400 rather than 404 if this is the case?
P.S. If the answer is submit a PR that's fine, will put it on my (long) todo list. Just thought i'd raise it as it is a use case that has come up and thought it might be useful for others.

The problem with this is that the swagger-tools middleware only runs for paths that get matched.  So if you make a request for a path that doesn't match, there is nothing we can do.  I don't want to assume that a request for /example was really intended to be a request for /example/{searchString}.  Assumptions are bad and once you make them, users will come out of the woodwork to complain.  Unfortunately, this is not only working as designed but also working properly from an HTTP perspective.  I don't see a way to make this better.
I'm going to close this but I'm more than willing to continue the conversation.

authors: maxcbc whitlockjc

license: mit