text:
[SCB-647] Update LICENSE for release 1.0.0-m2

Follow this checklist to help us incorporate your contribution quickly and easily:

[x] Make sure there is a JIRA issue filed for the change (usually before you start working on it).  Trivial changes like typos do not require a JIRA issue.  Your pull request should address just this issue, without pulling in other changes.
[x] Each commit in the pull request should have a meaningful subject line and body.
[x] Format the pull request title like [SCB-XXX] Fixes bug in ApproximateQuantiles, where you replace SCB-XXX with the appropriate JIRA issue.
[x] Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
[x] Run go build go test go fmt go vet to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

Coverage increased (+0.1%) to 74.744% when pulling f7b847a1529c87b03f3ef537918d00ffdd818e28 on yangbor:master into e729a5ee0faf8bd5db2a1fd2539fc7b62bb66b54 on apache:master.

Codecov Report

Merging #371 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #371   +/-   ##
=======================================
  Coverage   71.46%   71.46%
=======================================
  Files          25       25
  Lines        3690     3690
=======================================
  Hits         2637     2637
  Misses        884      884
  Partials      169      169

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e729a5e...28e39ee. Read the comment docs.

Can we merge it now?

authors: WillemJiang codecov-io coveralls yangbor


text:
Enhancement about the benchmark site

Our benchmark site is available right now: https://shardingsphere.apache.org/benchmark/
Great progress has been made in presentation than the previous one.
We have reviewed the site and found several issues need to be resolved:
For text:

[ ] All mysql change to MySQL.

[ ] Modules can be sharding-jdbc/sharding-proxy while can not be sharding/sharding proxy/shardingproxy/sharding_proxy/sharding jdbc/shardingjdbc/sharding_jdbc.

[ ] Explicit point out this(https://shardingsphere.apache.org/benchmark/#/mysql-vs-sharding) is a loss test.

[ ] There's no VS statement in:
https://shardingsphere.apache.org/benchmark/#/sharding-proxy-master-slave
https://shardingsphere.apache.org/benchmark/#/sharding-proxy-master-slave-sharding
https://shardingsphere.apache.org/benchmark/#/sharding-proxy-single-database-single-table
Compared with:
https://shardingsphere.apache.org/benchmark/#/shardingjdbc-vs-shardingproxy-encrypt
https://shardingsphere.apache.org/benchmark/#/shardingjdbc-vs-shardingproxy-sharding-encrypt
Please find a way to describe who vs who clearly.

For data:

[ ] The general order of the TPS in a compared performance is: MySQL > sharding-jdbc > sharding-proxy. But we can find several exceptions like the UPDATE case in https://shardingsphere.apache.org/benchmark/#/sharding-proxy-master-slave-sharding. I this case sharding-proxy > sharding-jdbc. We need to check the test environment and find the root cause. Of course, it does not meen there's no bugs in sharding-jdbc or sharding-proxy.

[ ] For the case SELECT in https://shardingsphere.apache.org/benchmark/#/mysql-vs-sharding, the TPS of sharding-proxy is much lower than MySQL. We need to find out whether the case is reasonable or not. Or the performance of sharding-proxy need to be improved.

Maybe we should move this issue to https://github.com/apache/incubator-shardingsphere-benchmark

Yes, you are right.
The issues module for incubator-shardingsphere-benchmark is not available for now, I have asked INFRA to enable it: https://issues.apache.org/jira/browse/INFRA-19288.
I'll transport this issue to benchmark repo as soon as INFRA-19288 is fixed.

source: gharchive/issue

repo: apache/incubator-shardingsphere

license_type: permissive

license_source: bigquery


text:
Query the column information of the table through SQL statement

I query the field infromation of the table with the "select COLUMN_NAME FROM information_schema.COLUMNS WHERE TABLE_NAME='goods'" statement,but result is null.
the "goods" is the source table of subtable.
I found through debug that the reason is that the information stored in the mysql database is
the subtable information ,so it cannot be obtained.
if I want query the column information of the table by source table names,what  to do .

duplicate with #1176

authors: terrymanu waterMeteors


text:
Optimize MySQL DDLStatement Visitor

Fixes #3914.
Optimize MySQL DDLStatement visitor:

Consider to simply visitCreateTable in MySQLDDLVisitor
Consider to simply visitAlterTable in MySQLDDLVisitor
Consider to simply createColumnDefinitionSegment in MySQLDDLVisitor

Changes proposed in this pull request:

Simply visitCreateTable in MySQLDDLVisitor
Simply visitAlterTable in MySQLDDLVisitor
Simply createColumnDefinitionSegment in MySQLDDLVisitor

Codecov Report

Merging #4209 into master will decrease coverage by 0.15%.
The diff coverage is 0%.

@@             Coverage Diff              @@
##             master    #4209      +/-   ##
============================================
- Coverage     60.85%   60.69%   -0.16%
  Complexity      352      352
============================================
  Files          1013     1013
  Lines         16922    16966      +44
  Branches       2983     2991       +8
============================================
  Hits          10298    10298
- Misses         5972     6016      +44
  Partials        652      652

Impacted Files
Coverage Δ
Complexity Δ

...dingsphere/sql/parser/visitor/MySQLDDLVisitor.java
0% <0%> (ø)
0 <0> (ø)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a0a91df...ae40ca2. Read the comment docs.

Pull Request Test Coverage Report for Build 9389

0 of 116 (0.0%)  changed or added relevant lines in 1 file are covered.
4 unchanged lines in 2 files lost coverage.
Overall coverage decreased (-0.2%) to 64.6%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

shardingsphere-sql-parser/shardingsphere-sql-parser-mysql/src/main/java/org/apache/shardingsphere/sql/parser/visitor/MySQLDDLVisitor.java
0
116
0.0%

Files with Coverage Reduction
New Missed Lines
%

shardingsphere-sql-parser/shardingsphere-sql-parser-mysql/src/main/java/org/apache/shardingsphere/sql/parser/visitor/MySQLDDLVisitor.java
1
0%

sharding-orchestration/sharding-orchestration-core/src/main/java/org/apache/shardingsphere/orchestration/internal/util/IpUtils.java
3
76.0%

Totals

Change from base Build 797:
-0.2%

Covered Lines:
10960

Relevant Lines:
16966

💛  - Coveralls

Hi @SteNicholas Thanks for this optimization. It seems this class becomes much better. Your contribution made this class become more clear, not as ugly as ever. Althought more optimization could be done, one point concerns me much is the name of some rules, like CreateDefinitionClause_.
Our instruction tells us CreateDefinitionClause is standard one, since we visited it in our Visitor.class, so i suggest to review any rule name ending with _ in MySQLDDLVisitor.java.
I'd like to see your another PR concerning this. Welcome your opinions. 😃  Good job!

source: gharchive/pull-request

authors: SteNicholas codecov-io coveralls tristaZero


text:
Not Found the image of version 2.1.1

Search before asking

[X] I had searched in the issues and found no similar issues.

Java Version
No response
Scala Version
2.11.x
StreamPark Version
2.1.1
Flink Version
1.16.2
deploy mode
None
What happened
The 2.1.0 version was found in the docker warehouse, but the 2.1.1 version image was not found. Using the v2.1.1 version image, the pod keeps restarting.
Error Exception
No response
Screenshots

Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!(您是否要贡献这个PR?)

Code of Conduct

[X] I agree to follow this project's Code of Conduct

I tried to use the Dockfile in deploy to compile the 2.1.1 image by myself. After the image is compiled, deploying with helm has the same effect as using the official V2.1.1 image, and it keeps restarting.

I tried to use the Dockfile in deploy to compile the 2.1.1 image by myself. After the image is compiled, deploying with helm has the same effect as using the official V2.1.1 image, and it keeps restarting.

I've encountered this issue before. In the startup script, if it uses 'start' instead of 'start_docker', the application will start in the background, causing the Pod to enter the 'completed' state. I'm currently looking into how to fix this better. If your startup command uses 'start', you can temporarily remove the '&'.

I tried to use the Dockfile in deploy to compile the 2.1.1 image by myself. After the image is compiled, deploying with helm has the same effect as using the official V2.1.1 image, and it keeps restarting.

I've encountered this issue before. In the startup script, if it uses 'start' instead of 'start_docker', the application will start in the background, causing the Pod to enter the 'completed' state. I'm currently looking into how to fix this better. If your startup command uses 'start', you can temporarily remove the '&'.

Thank you very much for your reply, but I still don't quite understand what you mean. I saw that the helm file is started directly like this. I don't quite understand what you mean by start_docker and removing '&'. Can you explain in detail, thank you !

I tried to use the Dockfile in deploy to compile the 2.1.1 image by myself. After the image is compiled, deploying with helm has the same effect as using the official V2.1.1 image, and it keeps restarting.

I've encountered this issue before. In the startup script, if it uses 'start' instead of 'start_docker', the application will start in the background, causing the Pod to enter the 'completed' state. I'm currently looking into how to fix this better. If your startup command uses 'start', you can temporarily remove the '&'.

Hello, I see that start() is used by default in the startup script, and I didn't find the '&' you said that needs to be removed.
shellcheck disable=SC2120
start() {
shellcheck disable=SC2006
local PID=$(get_pid)
if [ $PID -gt 0 ]; then
# shellcheck disable=SC2006
echo_r "StreamPark is already running pid: $PID , start aborted!"
exit 1
fi
Bugzilla 37848: only output this if we have a TTY
if [[ ${have_tty} -eq 1 ]]; then
echo_w "Using APP_BASE:   $APP_BASE"
echo_w "Using APP_HOME:   $APP_HOME"
if [[ "$1" = "debug" ]] ; then
echo_w "Using JAVA_HOME:   $JAVA_HOME"
else
echo_w "Using JRE_HOME:   $JRE_HOME"
fi
echo_w "Using APP_PID:   $APP_PID"
fi
local PROPER="${APP_CONF}/application.yml"
if [[ ! -f "$PROPER" ]] ; then
echo_r "ERROR: config file application.yml invalid or not found! ";
exit 1;
else
echo_g "Usage: config file: $PROPER ";
fi
shellcheck disable=SC2046
eval $(parse_yaml "${PROPER}" "conf_")
shellcheck disable=SC2001
shellcheck disable=SC2154
shellcheck disable=SC2155
local workspace=$(echo "$conf_streampark_workspace_local" | sed 's/#.*$//g')
if [[ ! -d $workspace ]]; then
echo_r "ERROR: streampark.workspace.local: "$workspace" is invalid path, Please reconfigure in application.yml"
echo_r "NOTE: "streampark.workspace.local" Do not set under APP_HOME($APP_HOME). Set it to a secure directory outside of APP_HOME.  "
exit 1;
fi
if [[ ! -w $workspace ]] || [[ ! -r $workspace ]]; then
echo_r "ERROR: streampark.workspace.local: "$workspace" Permission denied! "
exit 1;
fi
if [ "${HADOOP_HOME}"x == ""x ]; then
echo_y "WARN: HADOOP_HOME is undefined on your system env,please check it."
else
echo_w "Using HADOOP_HOME:   ${HADOOP_HOME}"
fi

classpath options:
1): java env (lib and jre/lib)
2): StreamPark
3): hadoop conf
shellcheck disable=SC2091
local APP_CLASSPATH=".:${JAVA_HOME}/lib:${JAVA_HOME}/jre/lib"
shellcheck disable=SC2206
shellcheck disable=SC2010
local JARS=$(ls "$APP_LIB"/.jar | grep -v "$APP_LIB/streampark-flink-shims_..jar$")
shellcheck disable=SC2128
for jar in $JARS;do
APP_CLASSPATH=$APP_CLASSPATH:$jar
done
if [[ -n "${HADOOP_CONF_DIR}" ]] && [[ -d "${HADOOP_CONF_DIR}" ]]; then
echo_w "Using HADOOP_CONF_DIR:   ${HADOOP_CONF_DIR}"
APP_CLASSPATH+=":${HADOOP_CONF_DIR}"
else
APP_CLASSPATH+=":${HADOOP_HOME}/etc/hadoop"
fi
shellcheck disable=SC2034
shellcheck disable=SC2006
local vmOption=$_RUNJAVA -cp "$APP_CLASSPATH" $PARAM_CLI --vmopt
local JAVA_OPTS="""
$vmOption
$DEFAULT_OPTS
$DEBUG_OPTS
"""
eval $NOHUP $_RUNJAVA $JAVA_OPTS
-classpath "$APP_CLASSPATH"
-Dapp.home="${APP_HOME}"
-Dlogging.config="${APP_CONF}/logback-spring.xml"
-Dspring.config.location="${PROPER}"
-Djava.io.tmpdir="$APP_TMPDIR"
$APP_MAIN >> "$APP_OUT" 2>&1 "&"
local PID=$!
local IS_NUMBER="^[0-9]+$"

# Add to pid file if successful start
if [[ ${PID} =~ ${IS_NUMBER} ]] && kill -0 $PID > /dev/null 2>&1 ; then
    echo $PID > "$APP_PID"
    # shellcheck disable=SC2006
    echo_g "StreamPark start successful. pid: $PID"
else
    echo_r "StreamPark start failed."
    exit 1
fi

}

@A-little-bit-of-data

After you have built the image (by executing ./build.sh), if you encounter an error message stating that the /opt/streampark_workspace directory is missing during runtime, you can create it and add the necessary access permissions. Then, test it locally first to ensure it works correctly (accessing http://localhost:10000/). You should be able to log in and see the project interface. This step is to verify if the compiled image has any issues.

Temporarily modify the Java startup parameters in the streampark-console/streampark-console-service/src/main/assembly/bin/streampark.sh file. The "&" at the end is used to start the program in the background. Remove the "&" to delete it.

  eval $NOHUP $_RUNJAVA $JAVA_OPTS \
    -classpath "$APP_CLASSPATH" \
    -Dapp.home="${APP_HOME}" \
    -Dlogging.config="${APP_CONF}/logback-spring.xml" \
    -Dspring.config.location="${PROPER}" \
    -Djava.io.tmpdir="$APP_TMPDIR" \
    $APP_MAIN >> "$APP_OUT" 2>&1 "&"

Recompile the project. Just like we tested locally, it requires the /opt/streampark_workspace directory. Add the following line to the Dockerfile: "RUN mkdir -p /opt/streampark_workspace". Then use the Dockerfile to build your image. After that, update the image in your Helm chart. Now you can successfully start it.

Additionally, it's worth mentioning that I am using the Kubernetes cluster provided by Docker. If you don't have an existing cluster, you can consider exploring other Helm chart deployment templates for various components in the central Helm chart repository. They can be used in conjunction with Streampark. However, I recommend deploying Streampark directly on the host machine and deploying other components that are not convenient to deploy locally using Docker or Kubernetes. If you encounter issues connecting Streampark on the host machine to components in different Kubernetes clusters, I highly recommend using https://alibaba.github.io/kt-connect/#/. ↗ It can effectively help you solve this problem.

3. streampark_workspace

Thank you very much for your detailed answer, following your explanation here, I have succeeded,

Currently, we only provide the latest version of the docker image, and the bug of starting in docker has been fixed.

authors: A-little-bit-of-data onlyotdn wolfboys

repo: apache/incubator-streampark

license_source: github-api


text:
[Improve][Flink-Kubernetes-V2] add exception throw

In some places where it might be null, I add an exception throw
①[KubernetesApplicationClientV2.scala]
②[KubernetesSesionClientV2.scala]
What changes were proposed in this pull request
fix: #2882 and #2884

Brief change log

Verifying this change

This change is a trivial rework / code cleanup without any test coverage.
(or)
This change is already covered by existing tests, such as (please describe tests).
(or)
This change added tests and can be verified as follows:

Does this pull request potentially affect one of the following parts

Dependencies (does it add or upgrade a dependency): (yes / no)

cc @Al-assad It is true that the cluster part may throw an exception for null, but I'm not sure if there is a risk of application being null

Whether it is necessary for DeployRequest to be as complete as SubRequest, it seems that DeployRequest is imperfect at present, and many configurable parameters have not been passed in

For ② I think it is OK, ① I am not sure if it is appropriate to make such a change

cc @Al-assad

Thank you for your submission, but the code modified in the commit do not require a value. Therefore they are all of type Option in the original code. I'm afraid I will reject this PR.

Thank you for your reply and review . If the value is NULL, the return is null. There is no need to return an exception

I seem to understand what you mean. None doesn't affect the overall operation of the program, but Exception does, so those arguments can be None

If so, I will shut this pr down


text:
Docker compose up exit code 217

Make sure these boxes are checked before submitting your issue - thank you!

[X] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[X] I have reproduced the issue with at least the latest released version of superset.
[X] I have checked the issue tracker for the same issue and I haven't found one similar.

Superset version
Lastest commit on master
Expected results
Docker compose working
Actual results
$ docker-compose up
superset_postgres_1 is up-to-date
superset_redis_1 is up-to-date
Creating superset_superset_1 ... done
Attaching to superset_postgres_1, superset_redis_1, superset_superset_1
superset_1  | + '[' 0 -ne 0 ']'
superset_1  | + '[' development = development ']'
superset_1  | + celery worker --app=superset.sql_lab:celery_app --pool=gevent -Ofair
superset_1  | + cd superset/assets/
superset_1  | + npm ci
postgres_1  | 2019-02-07 15:41:55.069 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres_1  | 2019-02-07 15:41:55.069 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres_1  | 2019-02-07 15:41:55.109 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1  | 2019-02-07 15:41:55.235 UTC [22] LOG:  database system was shut down at 2019-02-07 15:31:58 UTC
postgres_1  | 2019-02-07 15:41:55.249 UTC [1] LOG:  database system is ready to accept connections
postgres_1  | 2019-02-07 15:42:51.678 UTC [30] ERROR:  duplicate key value violates unique constraint "ab_user_username_key"
postgres_1  | 2019-02-07 15:42:51.678 UTC [30] DETAIL:  Key (username)=(admin) already exists.
postgres_1  | 2019-02-07 15:42:51.678 UTC [30] STATEMENT:  INSERT INTO ab_user (id, first_name, last_name, username, password, active, email, last_login, login_count, fail_login_count, created_on, changed_on, created_by_fk, changed_by_fk) VALUES (nextval('ab_user_id_seq'), 'admin', 'user', 'admin', 'pbkdf2:sha256:50000$fgf6eYlb$bb1e6c1f420de4d894f7a917a72712c0b846dd969f51afb7cf386f0848508422', true, 'admin@fab.org', NULL, NULL, NULL, '2019-02-07T15:42:51.677213'::timestamp, '2019-02-07T15:42:51.677242'::timestamp, NULL, NULL) RETURNING ab_user.id
redis_1     | 1:C 07 Feb 15:41:55.444 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis_1     |                 _._
redis_1     |            _.-``__ ''-._
redis_1     |       _.-``    `.  `_.  ''-._           Redis 3.2.12 (00000000/0) 64 bit
redis_1     |   .-`` .-```.  ```\/    _.,_ ''-._
redis_1     |  (    '      ,       .-`  | `,    )     Running in standalone mode
redis_1     |  |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
redis_1     |  |    `-._   `._    /     _.-'    |     PID: 1
redis_1     |   `-._    `-._  `-./  _.-'    _.-'
redis_1     |  |`-._`-._    `-.__.-'    _.-'_.-'|
redis_1     |  |    `-._`-._        _.-'_.-'    |           http://redis.io
redis_1     |   `-._    `-._`-.__.-'_.-'    _.-'
redis_1     |  |`-._`-._    `-.__.-'    _.-'_.-'|
redis_1     |  |    `-._`-._        _.-'_.-'    |
redis_1     |   `-._    `-._`-.__.-'_.-'    _.-'
redis_1     |       `-._    `-.__.-'    _.-'
redis_1     |           `-._        _.-'
redis_1     |               `-.__.-'
redis_1     |
redis_1     | 1:M 07 Feb 15:41:55.446 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
redis_1     | 1:M 07 Feb 15:41:55.446 # Server started, Redis version 3.2.12
redis_1     | 1:M 07 Feb 15:41:55.446 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
redis_1     | 1:M 07 Feb 15:41:55.446 * DB loaded from disk: 0.000 seconds
redis_1     | 1:M 07 Feb 15:41:55.446 * The server is now ready to accept connections on port 6379
superset_1  | npm WARN prepare removing existing node_modules/ before installation
superset_1  | npm ERR! path /home/superset/superset/assets/node_modules/core-js/fn/array/virtual
superset_1  | npm ERR! code ENOTEMPTY
superset_1  | npm ERR! errno -39
superset_1  | npm ERR! syscall rmdir
superset_1  | npm ERR! ENOTEMPTY: directory not empty, rmdir '/home/superset/superset/assets/node_modules/core-js/fn/array/virtual'
superset_1  |
superset_1  | npm ERR! A complete log of this run can be found in:
superset_1  | npm ERR!     /home/superset/.npm/_logs/2019-02-07T15_47_06_436Z-debug.log
postgres_1  | 2019-02-07 15:47:06.572 UTC [36] LOG:  unexpected EOF on client connection with an open transaction
superset_1  | npm WARN prepare removing existing node_modules/ before installation
superset_1  | npm ERR! path /home/superset/superset/assets/node_modules/core-js/fn/array/virtual
superset_1  | npm ERR! code ENOTEMPTY
superset_1  | npm ERR! errno -39
superset_1  | npm ERR! syscall rmdir
superset_1  | npm ERR! ENOTEMPTY: directory not empty, rmdir '/home/superset/superset/assets/node_modules/core-js/fn/array/virtual'
superset_1  |
superset_1  | npm ERR! A complete log of this run can be found in:
superset_1  | npm ERR!     /home/superset/.npm/_logs/2019-02-07T15_50_51_169Z-debug.log
postgres_1  | 2019-02-07 15:50:51.297 UTC [44] LOG:  unexpected EOF on client connection with an open transaction
superset_superset_1 exited with code 217
superset_1  | npm WARN prepare removing existing node_modules/ before installation
superset_1  | npm ERR! path /home/superset/superset/assets/node_modules/core-js/fn/array/virtual
superset_1  | npm ERR! code ENOTEMPTY
superset_1  | npm ERR! errno -39
superset_1  | npm ERR! syscall rmdir
superset_1  | npm ERR! ENOTEMPTY: directory not empty, rmdir '/home/superset/superset/assets/node_modules/core-js/fn/array/virtual'
superset_1  |
superset_1  | npm ERR! A complete log of this run can be found in:
superset_1  | npm ERR!     /home/superset/.npm/_logs/2019-02-07T15_54_35_445Z-debug.log
postgres_1  | 2019-02-07 15:54:35.595 UTC [52] LOG:  unexpected EOF on client connection with an open transaction
superset_superset_1 exited with code 217
superset_1  | npm WARN prepare removing existing node_modules/ before installation
superset_1  | npm ERR! path /home/superset/superset/assets/node_modules/core-js/fn/array/virtual
superset_1  | npm ERR! code ENOTEMPTY
superset_1  | npm ERR! errno -39
superset_1  | npm ERR! syscall rmdir
superset_1  | npm ERR! ENOTEMPTY: directory not empty, rmdir '/home/superset/superset/assets/node_modules/core-js/fn/array/virtual'
superset_1  |
superset_1  | npm ERR! A complete log of this run can be found in:
superset_1  | npm ERR!     /home/superset/.npm/_logs/2019-02-07T15_58_19_852Z-debug.log
postgres_1  | 2019-02-07 15:58:19.963 UTC [61] LOG:  unexpected EOF on client connection with an open transaction
superset_superset_1 exited with code 217
superset_1  | npm WARN prepare removing existing node_modules/ before installation
superset_1  | npm ERR! path /home/superset/superset/assets/node_modules/core-js/fn/array/virtual
superset_1  | npm ERR! code ENOTEMPTY
superset_1  | npm ERR! errno -39
superset_1  | npm ERR! syscall rmdir
superset_1  | npm ERR! ENOTEMPTY: directory not empty, rmdir '/home/superset/superset/assets/node_modules/core-js/fn/array/virtual'
superset_1  |
superset_1  | npm ERR! A complete log of this run can be found in:
superset_1  | npm ERR!     /home/superset/.npm/_logs/2019-02-07T16_02_04_350Z-debug.log
postgres_1  | 2019-02-07 16:02:04.473 UTC [69] LOG:  unexpected EOF on client connection with an open transaction
superset_superset_1 exited with code 217
superset_1  | npm WARN prepare removing existing node_modules/ before installation
superset_1  | npm ERR! path /home/superset/superset/assets/node_modules/core-js/fn/array/virtual
superset_1  | npm ERR! code ENOTEMPTY
superset_1  | npm ERR! errno -39
superset_1  | npm ERR! syscall rmdir
superset_1  | npm ERR! ENOTEMPTY: directory not empty, rmdir '/home/superset/superset/assets/node_modules/core-js/fn/array/virtual'
superset_1  |
superset_1  | npm ERR! A complete log of this run can be found in:
superset_1  | npm ERR!     /home/superset/.npm/_logs/2019-02-07T16_05_48_096Z-debug.log
postgres_1  | 2019-02-07 16:05:48.222 UTC [78] LOG:  unexpected EOF on client connection with an open transaction
superset_superset_1 exited with code 217
superset_1  | npm WARN prepare removing existing node_modules/ before installation

Note that the execution started on  2019-02-07 15:41:55.069 and the "screenshot" was taken on  2019-02-07 16:05:48.222 without any result. It seems that it is always repeating the:
superset_1  | npm WARN prepare removing existing node_modules/ before installation
superset_1  | npm ERR! path /home/superset/superset/assets/node_modules/core-js/fn/array/virtual
superset_1  | npm ERR! code ENOTEMPTY
superset_1  | npm ERR! errno -39
superset_1  | npm ERR! syscall rmdir
superset_1  | npm ERR! ENOTEMPTY: directory not empty, rmdir '/home/superset/superset/assets/node_modules/core-js/fn/array/virtual'
superset_1  |
superset_1  | npm ERR! A complete log of this run can be found in:
superset_1  | npm ERR!     /home/superset/.npm/_logs/2019-02-07T16_05_48_096Z-debug.log
postgres_1  | 2019-02-07 16:05:48.222 UTC [78] LOG:  unexpected EOF on client connection with an open transaction
superset_superset_1 exited with code 217

Steps to reproduce
I followed these steps: https://github.com/apache/incubator-superset/blob/master/docs/installation.rst
based on that commit 823555e

Don't know why and how but the error has disappeared :/

repo: apache/incubator-superset


text:
Error Whilst Adding Database

A clear and concise description of what the bug is.
When attempting to add a database to Superset on Ubuntu 18 I get the following error:
TypeError: Object of type int64 is not JSON serializable

This was caused here:
https://github.com/apache/incubator-superset/blob/d62c37b8958fe89aa69df8f1d582460abef548d1/superset/views/core.py#L1833
What is happening is that to test the connection, Superset is issuing the query, the result coming back is a numPy int64.
Expected results
I expected the connection to be added
Actual results
TypeError: Object of type int64 is not JSON serializable

How to reproduce the bug

Go to '...'
Click on '....'
Scroll down to '....'
See error

Environment
Ubuntu v. 18
(please complete the following information):

superset version: 0.999.0dev
python version: 3.6.8 :: Anaconda custom (64-bit)
node.js version: 10.16.0
npm version: 6.8.0

Checklist
Make sure these boxes are checked before submitting your issue - thank you!

[ ] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[ ] I have reproduced the issue with at least the latest released version of superset.
[ X] I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context
Add any other context about the problem here.
This can be fixed by modifying the line as follows:
return json_success(json.dumps(int(conn.scalar(select([1])))))

@john-bodley looks like this came from recent https://github.com/apache/incubator-superset/pull/7570

@cgivre how about return json_success(json.dumps(conn.scalar(select(['a']))))  or how about return json_success(json.dumps(int(conn.scalar(select([1])))))

@mistercrunch I think either of those would work.  I tried the int version that you listed there, and it worked for me.
I can submit a PR if you'd like tomorrow.

Surprised it didn't get caught by https://github.com/apache/incubator-superset/blob/36a219da7f81c0e65a259049896bb77fd66edbe2/tests/core_tests.py#L354
What db engine are you connecting to? From the test we know mysql/postgres/sqlite should work...

It was Drill.  I originally thought it was a problem with the sqlalchemy dialect.

Shouldn't the DBAPI driver return native types? What type is int64 here?

This is a very strange error because this works fine on my Mac.  It was only when I tried this on an Ubuntu machine that it caused problems.  The issue is that the int64 is a numpy data type.
@JohnOmernik, any ideas?

Would approve solutions suggested in my comment here: https://github.com/apache/incubator-superset/issues/7669#issuecomment-499756804

@mistercrunch  I'll submit a PR this weekend.

Sorry wouldn’t a simpler solution be:
conn.scalar(select([1]))
return json_success(‘“OK”’)

i.e., we really don’t care what the JSON response is but rather that the status code is 200. The try/except handles the failed connection case.

authors: cgivre john-bodley mistercrunch


text:
fix(sql-lab): relax column name restrictions

SUMMARY
Currently virtual datasources (=SQL queries) are required to consist of columns that start with a non-numeric character and don't contain special characters, including whitespace. This is unnecessary, as the majority of databases support special characters as long as they are quoted. For instance, querying the example dataset birth_france_by_region produces an error due to the numeric columns 2003-2014, despite the datasource working just fine on the majority of databases.
This PR relaxes the column name restrictions to only two rules:

__timestamp is reserved for the temporal column
anything ending in __\d, e.g. my_col__1, as that pattern is used to deduplicate column names.

BEFORE
Trying to use SELECT * FROM birth_france_by_region as a datasource:

AFTER
Nema problema!

TEST PLAN
Adjusted existing unit test
ADDITIONAL INFORMATION

[ ] Has associated issue:
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

🏷 polidea

🗑 🏷 polidea

authors: mistercrunch villebro


text:
feat: add modal to import dashboards

SUMMARY

Identical to https://github.com/apache/incubator-superset/pull/11923/files, but for charts.
BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

Tested importing dashboard to an existing database/chart, as well as a dashboard to a new database, works as expected.
Also added tests.
ADDITIONAL INFORMATION

[X] Has associated issue: https://github.com/apache/incubator-superset/issues/11167
[ ] Changes UI
[ ] Requires DB Migration.
[ ] Confirm DB Migration upgrade and downgrade tested.
[ ] Introduces new feature or API
[ ] Removes existing feature or API

Codecov Report

Merging #11924 (33ecffb) into PISQL-2 (b934e6e) will decrease coverage by 4.73%.
The diff coverage is 86.66%.

@@             Coverage Diff             @@
##           PISQL-2   #11924      +/-   ##
===========================================
- Coverage    68.08%   63.35%   -4.74%
===========================================
  Files          939      459     -480
  Lines        48362    28169   -20193
  Branches      4862        0    -4862
===========================================
- Hits         32927    17846   -15081
+ Misses       15332    10323    -5009
+ Partials       103        0     -103

Flag
Coverage Δ

cypress
?

javascript
?

python
63.35% <86.66%> (-2.23%)
:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files
Coverage Δ

superset/dashboards/commands/export.py
91.17% <0.00%> (ø)

...erset/dashboards/commands/importers/v1/__init__.py
94.69% <85.71%> (-0.60%)
:arrow_down:

superset/dashboards/api.py
84.27% <100.00%> (-2.07%)
:arrow_down:

...perset/dashboards/commands/importers/dispatcher.py
80.64% <100.00%> (+1.33%)
:arrow_up:

superset/dashboards/schemas.py
98.78% <100.00%> (ø)

superset/db_engines/hive.py
0.00% <0.00%> (-85.72%)
:arrow_down:

superset/db_engine_specs/hive.py
54.08% <0.00%> (-32.09%)
:arrow_down:

superset/db_engine_specs/presto.py
69.76% <0.00%> (-12.65%)
:arrow_down:

superset/dashboards/commands/importers/v0.py
80.51% <0.00%> (-12.34%)
:arrow_down:

superset/db_engine_specs/mysql.py
79.59% <0.00%> (-12.25%)
:arrow_down:

... and 523 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b934e6e...33ecffb. Read the comment docs.

authors: betodealmeida codecov-io


text:
Optimize presto SQL Lab query performance.

By stop polling when presto query already finished.
When user make queries to Presto via SQL Lab, presto will run the query
and then it can return all data back to superset in one shot.
However, the default implementation of superset has enabled a default
polling for presto to:

Get the fancy progress bar
Get the data back when the query finished.

However, the polling implementation of superset is not right.
I've done a profiling with a table of 1 billion rows, here're some data:

Total number of rows: 1.02 Billion
SQL Lab query limit: 1 million
Output Data: 1.5 GB
Superset memory consumed: about 10-20 GB
Time: 7 minutes to finish in Presto, takes additional 15 minutes for
superset to get and store data.

The problems with default issue is, even if presto has finished the
query (7 minutes with above profiling), superset still do lots of wasted
polling, in above profiling, superset sent about 540 polling in total,
and at half of the polling is not necessary.
Part of the simplied polling response:
{
  "infoUri": "http://10.65.204.39:8000/query.html?20180525_042715_03742_nza9u",
  "id": "20180525_042715_03742_nza9u",
  "nextUri": "http://10.65.204.39:8000/v1/statement/20180525_042715_03742_nza9u/11",
  "stats": {
    "state": "FINISHED",
    "queuedSplits": 21701,
    "progressPercentage": 35.98264191882267,
    "elapsedTimeMillis": 1029,
    "nodes": 116,
    "completedSplits": 15257,
    "scheduled": true,
    "wallTimeMillis": 2571904,
    "peakMemoryBytes": 0,
    "processedBytes": 40825519532,
    "processedRows": 47734066,
    "queuedTimeMillis": 0,
    "queued": false,
    "cpuTimeMillis": 849228,
    "rootStage": {
      "state": "FINISHED",
      "queuedSplits": 0,
      "nodes": 1,
      "totalSplits": 17,
      "processedBytes": 16829644,
      "processedRows": 11495,
      "completedSplits": 17,
      "stageId": "0",
      "done": true,
      "cpuTimeMillis": 69,
      "subStages": [
        {
          "state": "CANCELED",
          "queuedSplits": 21701,
          "nodes": 116,
          "totalSplits": 42384,
          "processedBytes": 40825519532,
          "processedRows": 47734066,
          "completedSplits": 15240,
          "stageId": "1",
          "done": true,
          "cpuTimeMillis": 849159,
          "subStages": [],
          "wallTimeMillis": 2570374,
          "userTimeMillis": 730020,
          "runningSplits": 5443
        }
      ],
      "wallTimeMillis": 1530,
      "userTimeMillis": 50,
      "runningSplits": 0
    },
    "totalSplits": 42401,
    "userTimeMillis": 730070,
    "runningSplits": 5443
  }
  }
}

Superset will terminate the polling when it finds that nextUri
becomes none, but actually, when ["stats"]["state"] == "FINISHED",
it means that presto has already finished the query and superset can stop
polling and get the data back.
After this simple optimization, we get a 2-5x performance boost for
Presto SQL Lab queries.

Codecov Report

Merging #5132 into master will decrease coverage by 0.02%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master    #5132      +/-   ##
==========================================
- Coverage   77.51%   77.48%   -0.03%
==========================================
  Files          44       44
  Lines        8734     8737       +3
==========================================
  Hits         6770     6770
- Misses       1964     1967       +3

Impacted Files
Coverage Δ

superset/db_engine_specs.py
53.71% <0%> (-0.27%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dc21e0d...a0f5802. Read the comment docs.

Oh wow. Thanks for digging in and improving this. @timifasubaa @john-bodley

Thanks @xiaohanyu for looking into this. For your example above to do know how much time is spent writing to and reading from S3?

Thanks for working on this. I'm curious what the baseline comparison for the 2-5x speedup was. How complex was the query that saw a 2-5x speedup.

Hi, @john-bodley , @timifasubaa ,
In general, the technical process of a SQL Lab presto query is as follows:

Step A: Superset send presto query
Step B: Presto accept query request and schedule to run this query
Step C: Presto run and finish the query
Step D: Superset polling presto for query status
Step E: Superset get the query result back (this include data transfer between presto and lumos, for big queries, this really take some time)
Step F: Superset parse the query result, load it to memory, and then send the data back to browser
Step G: Browser take some time to load the data

Step C, D, E, F occupies 90% time of the whole cycle, and Step C, E, F is not feabible for easy optimizations. My optimization focus on step E, and get rid of 80% of unnecessary pollings, which, as you can see, lead to a 2-5x improvement for end-to-end performance.

I've added some debug info to the default presto polling process, and here's a sample debugging output for a query with 1 million output rows:

In our environment, this query takes Presto 7 minutes to finish, and takes additional 15 minutes for superset to get the final data.
In Step D, it send about 540 queries just to polling the progress, and the polling will end when self._nextUri becomes None. However, the presto query has already finished long before self._nextUri becomes None. Actually, the presto query has finished when `stats.get('state') == 'FINISHED'.
So my patch just utilize this kind of private internal presto API, and removes the unnecessary polling queries.

Here's a benchmark report from our own production server:

I've also test the consistency of data before and after patch and it works to be the same data result.
One potential risk is, I'm not an expert of Presto, so the internal json response maybe changed in Presto, and this may be break.

Thanks @xiaohanyu for the detailed explanation. The reason I asked in @timifasubaa was looking into improvements to phases F and G per https://github.com/apache/incubator-superset/pull/4834.

authors: codecov-io john-bodley mistercrunch timifasubaa xiaohanyu


text:
[feature] Allow min/max value for the sparkline in time series table

Letting a user set a fixed y-min/max

@williaster @graceguo-supercat

Codecov Report

Merging #5603 into master will decrease coverage by 0.01%.
The diff coverage is 20%.

@@            Coverage Diff             @@
##           master    #5603      +/-   ##
==========================================
- Coverage   63.59%   63.57%   -0.02%
==========================================
  Files         359      359
  Lines       22799    22809      +10
  Branches     2533     2537       +4
==========================================
+ Hits        14499    14501       +2
- Misses       8285     8293       +8
  Partials       15       15

Impacted Files
Coverage Δ

superset/assets/src/visualizations/time_table.jsx
0% <0%> (ø)
:arrow_up:

...re/components/controls/TimeSeriesColumnControl.jsx
89.58% <66.66%> (-1.53%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d1ef81f...713b8f9. Read the comment docs.

Add an option to show bounds

@kristw this looks great! One thought about the reference lines/showing the bounds, do you think it'd be useful to label them? Something like the following but just the number for min / max

The tricky thing with this would be the padding to accommodate the label, so might not be worth it 🤔 would be nice to have better support for that tho.

Done

WOW! 🙉
this will also be really useful (possibly more so!) without setting ymin/max bounds too

authors: codecov-io kristw williaster


text:
Get SGX_ERROR_INVALID_PARAMETER when Calling sgx_get_quote()

I'm a newbee in rust and sgx.  When I'm writing some toy examples for sgx_get_quote() in the simulation mode, I meet some problems that I can't solve.
In my main.rs file, I define some extern C function:
extern "C" {
    pub fn sgx_init_quote(p_target_info: *mut sgx_target_info_t, p_gid: *mut sgx_epid_group_id_t) -> sgx_status_t;

    pub fn sgx_calc_quote_size(p_sig_rl: *const uint8_t, sig_rl_size: uint32_t, p_quote_size: *mut uint32_t) -> sgx_status_t;

    pub fn sgx_get_quote(p_report: *const sgx_report_t, quote_type: sgx_quote_sign_type_t,
                         p_spid: *const sgx_spid_t, p_nonce: *const sgx_quote_nonce_t, p_sig_rl: *const uint8_t,
                         sig_rl_size: uint32_t, p_qe_report: *mut sgx_report_t, p_quote: *mut sgx_quote_t,
                         quote_size: uint32_t) -> sgx_status_t;
}

Then I initialize some paramters to call these functions:
   let mut target_info = sgx_target_info_t::default();
    let mut gid = sgx_epid_group_id_t::default();
    let status = unsafe { sgx_init_quote(&mut target_info, &mut gid) };
    if status != sgx_status_t::SGX_SUCCESS {
        println!("[-] Get error when get_init_quote, {}", status.as_str());
        return;
    }
    println!("SGX_INIT_QUOTE SUCCESS!");

    let mut quote_size: u32 = 0;
    let status = unsafe { sgx_calc_quote_size(std::ptr::null(), 0, &mut quote_size) };
    if status != sgx_status_t::SGX_SUCCESS {
        println!("[-] Get error when sgx_calc_quote_size, {}", status.as_str());
        return;
    }
    println!("SGX_calc_quote_size SUCCESS!");

    let spid = sgx_spid_t::default();
    let mut report = sgx_report_t::default();
    let mut the_quote = vec![0u8; quote_size as usize].into_boxed_slice();

    let status = unsafe {
        sgx_get_quote(&report,
                      sgx_quote_sign_type_t::SGX_UNLINKABLE_SIGNATURE,
                      &spid,
                      std::ptr::null(),
                      std::ptr::null(),
                      0,
                      std::ptr::null_mut(),
                      the_quote.as_mut_ptr() as *mut sgx_quote_t,
                      quote_size,
        )
    };
    if status != sgx_status_t::SGX_SUCCESS {
        println!("[-] Get error when sgx_get_quote, {}", status.as_str());
        return;
    }
    println!("SGX_GET_QUOTE SUCCESS!");

The results show that I can call sgx_init_quote() and sgx_calc_quote_size() successfully. But when I call sgx_get_quote(), it returns internal error "SGX_ERROR_INVALID_PARAMETER". I'm really confused of it because I set it as default value.
The following is the result:

Please look closer into Intel's manual:
https://download.01.org/intel-sgx/sgx-linux/2.7.1/docs/Intel_SGX_Developer_Reference_Linux_2.7.1_Open_Source.pdf
sgx_get_quote is on page 163-165.
at the bottom of page 165, intel writes clearly:

Library: Requirements - libsgx_epid.so or libsgx_uae_service_sim.so (simulation)

these two requirements (two general linux shared object), as well as the name of "uae_service) both indicate that this function can only be called in the untrusted app. you cannot call it in the enclave.
you can look into the mutual-ra enclave sample to see how to generate a quote:
https://github.com/apache/incubator-teaclave-sgx-sdk/blob/master/samplecode/mutual-ra/enclave/src/lib.rs#L373

repo: apache/incubator-teaclave-sgx-sdk


text:
[BACKPORT] - Change TO monitoring.json to raw SQL

This is 3-10x faster, and this endpoint is one of the slowest.
(cherry picked from commit 8942ce09e78e0b4e89e02075537ae65434189fc5)

merged to 2.0.x
please close

repo: apache/incubator-trafficcontrol


text:
an issue when packing android apk

The error is here:
What went wrong: Could not resolve all files for configuration ':sdk:releaseCompileClasspath'. Could not find com.android.support:recyclerview-v7:25.3..
Searched in the following locations: file:/Users/mac/Library/Android/sdk/extras/m2repository/com/android/support/recyclerview-v7/25.3./recyclerview-v7-25.3..pom file:/Users/mac/Library/Android/sdk/extras/m2repository/com/android/support/recyclerview-v7/25.3./recyclerview-v7-25.3..jar file:/Users/mac/Library/Android/sdk/extras/google/m2repository/com/android/support/recyclerview-v7/25.3./recyclerview-v7-25.3..pom file:/Users/mac/Library/Android/sdk/extras/google/m2repository/com/android/support/recyclerview-v7/25.3./recyclerview-v7-25.3..jar file:/Users/mac/Library/Android/sdk/extras/android/m2repository/com/android/support/recyclerview-v7/25.3./recyclerview-v7-25.3..pom file:/Users/mac/Library/Android/sdk/extras/android/m2repository/com/android/support/recyclerview-v7/25.3./recyclerview-v7-25.3..jar
I tried many ways but still couldn't solve it..... Thanks for your help

I think this problems has nothing to do with Weex, it is about normal Android development.

authors: YorkShen nikojxie

repo: apache/incubator-weex


text:
KAFKA-12226: Prevent source task offset failure when producer is overwhelmed

Jira
When a task fails to commit offsets because all outstanding records haven't been ack'd by the broker yet, it's better to retry that same batch. Otherwise, the set of outstanding records can grow indefinitely and all subsequent offset commit attempts can fail. By retrying the same batch, it becomes possible to eventually commit offsets, even when the producer is unable to keep up with the throughput of the records provided to it by the task.
Two unit tests are added to verify this behavior.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@gharris1727 @tombentley @chia7712 anyone got a moment? 😃

Yeah, that's fair. I found https://github.com/apache/kafka/blob/2caef070d4199877337f59399563354e41b203f7/connect/runtime/src/test/java/org/apache/kafka/connect/runtime/distributed/DistributedHerderTest.java#L550-L639 which seems applicable here. I'll refactor the tests to use a similar approach. Thanks Greg!

The only consistent test failure for the last run appears unrelated to the changes here. See https://github.com/apache/kafka/pull/10140/checks?check_run_id=1917700907 and https://github.com/apache/kafka/pull/10077/checks?check_run_id=1877979511 for other instances of the same failure.
org.apache.kafka.connect.integration.InternalTopicsIntegrationTest.testCreateInternalTopicsWithDefaultSettings
org.opentest4j.AssertionFailedError: Condition not met within timeout 30000. Didn't find the topics [connect-storage-topic-connect-cluster-1, connect-config-topic-connect-cluster-1, connect-offset-topic-connect-cluster-1] ==> expected: <true> but was: <false>

Yeah, the unit tests for the worker classes in general can be a little gnarly. The boilerplate segments (for things like setting up the converter, transformation chain, task context, headers, topic tracking, offset buffering, status tracking, and performing metrics assertions) are in line with the other tests in this class, though. The only difference with the new unit test here is that we want to test task behavior under some pretty fine-grained circumstances, which is accomplished right now by setting up latches and awaiting them to ensure that the task (which is running on a separate thread) has reached certain points in its lifecycle and not gone any further. If you have suggestions for how to improve that I'm all ears!
I'm not sure what you're referring to with task restart--that's not tested for at the moment, and any tests for that would likely be dependent on the success (or lack thereof) of any prior offset commit attempts, which are already tested for. Can you clarify or provide a brief example?

I did not look at the tests closely yet but I thought sourceTask.stop(); (https://github.com/apache/kafka/pull/10112/files#diff-b8da514e331b4d8bff623626ee31a45048b753911193b564a9b65f02b97f53c8R922) was a key part of the test. Maybe it doesn't!
I hope to take another look some time next week.

Ah, gotcha! Yeah, SourceTask::stop isn't really a key part of the functional changes here or the tests added. The idea is just to prevent high-throughput source tasks whose producers are unable to keep up with them from entering a death spiral where they stop being able to commit offsets completely.
This is because right now, when an offset commit attempt fails because the current batch of source records wasn't flushed to Kafka in time, all backlogged records that were read from SourceTask::poll during that time get added to the batch, which just compounds the problem.
For example:

The source task generates 10000 records and the worker starts sending those to Kafka.
An offset commit is triggered on a separate thread and the worker waits for up to offset.flush.timeout.ms milliseconds for all of those records to be ack'd by the broker. In the meantime, on the main thread, the worker continues polling the source task for data.
The worker times out while waiting the original batch of 10000 records to be flushed; let's say that 5000 managed to be written successfully but 5000 remain unacknowledged. Additionally, during this time, the worker managed to poll an additional 1000 records from the task.

At this point, the current behavior is:

Abort the offset commit, and start a new batch consisting of the 5000 unacknowledged records from the previous batch and the 1000 records polled from the task during the failed offset commit attempt.
Continue adding records to that batch until the next offset commit attempt is triggered.

If the task is generating a steady throughput of 10000 records per offset commit attempt, and the worker's producer is only able to write 5000 of those before the offset commit attempt times out, the worker will never be able to successfully commit offsets for the task, even though there are plenty of records that have been sent to and ack'd by the broker.
The proposed behavior in the PR is:

Abort the offset commit, and keep the old batch of the 5000 unacknowledged records. Add the 1000 records polled during the failed offset commit attempt to a backlog.
Continue adding newly-polled records from the task to that backlog.
On the next offset commit attempt, only wait to flush out the records from the active batch (i.e., the 5000 unacknowledged records from step 4), and only write offsets for that batch.
If successful, use the backlog of records as the new batch of records. Otherwise, keep the same batch of records and continue adding newly-polled records to the backlog.

First of all, thanks for trying to fix this issue, @C0urante.
And thanks for your insight, @hachikuji. I agree that it seems like we should not have to block the offset commits until the full batch of records has been written to Kafka.
I suspect the current logic was written this way because it's the simplest thing to do, given that the source partition map and offset map in the source records are opaque, meaning we can't sort them and have to instead rely upon the order of the source records returned by the connector. And because the producer can make progress writing to some topic partitions while not making progress on others, it's possible that some records in a batch are written before earlier records in the same batch.
The bottom line is that we have to track offsets that can be committed using only the order of the records that were generated by the source task. The current logic simply blocks committing offsets until each "batch" of records is completely flushed. That way we can commit all of the offsets in the batch together, and let the offset writer rely upon ordering to use only the latest offset map for each partition map when we tell it to flush.
But, flushing offsets requires synchronization, and the current logic switches between the outstandingMessages and outstandingMessagesBacklog buffers to track the "batches" of records that have to complete for offset commits. It's really sort of a mess.
@hachikuji wrote:

The patch gets around the problem by relaxing offset.flush.timeout.ms a little bit. Rather than treating expiration of records as a fatal error, we continue to allow more time for outstandingMessages to be drained. This ensures that we do not have to wait for the messages from outstandingMessagesBacklog which are added while the flush is in progress.

That's my understanding, too. And maybe I don't grasp the subtleties of the fix, but it seems like the fix won't necessarily help when a producer is consistently slow. In such cases, the outstandingMessages will fill with the records sent to the producer since the previous commit offsets, and as soon as we start committing offsets all records then get sent to outstandingMessagesBacklog. If the producer writes records significantly slower than the source task generates them, then outstandingMessagesBacklog could be larger than outstandingMessages by the time the offsets for outstandingMessagesBacklog are finally committed, especially if we're blocking offset commits even longer with this change. So while we're able to eventually commit those first offsets, if the backlog is larger than it will likely take longer for the producer to flush those records than it took the producer to flush the first batch.
Fortunately, we do have back pressure to not let this get this too out of control: when the producer's buffer fills up, the worker source task's thread will block (up to max.block.ms) on calls to producer.send(...), and the worker source task will retry any sends that fail after that timeout. And since this same thread that calls poll(), the worker source task will eventually slow calls to poll().
But I think we can change how offsets are flushed such that we don't have to wait for the producer, and instead we can simply flush the latest offsets for records that have been successfully written at that point. We just need a different mechanism (other than the two "outstanding" lists and the flush-related flags) to track the offsets for the most recently written records.
One way to do that is to use a single concurrent queue that bookkeeps records in the same order as generated by the source task, but in a way that allows us to track which records have been acked and tolerates those records being acked in any order.
For example, we could replace the outstandingMessages and outstandingMessagesBacklog fields in WorkerSourceTask with something like this:
   private final Queue<SubmittedRecord> submittedRecords = new ConcurrentLinkedQueue<>();

An element is appended to this queue just before the record is sent to the producer, and the SubmittedRecord class allows us to track which of these records has been acknowledged:
    protected static class SubmittedRecord {
        private final SourceRecord record;
        private final AtomicBoolean acked = new AtomicBoolean();
        public SubmittedRecord(SourceRecord sourceRecord) {
            record = Objects.requireNonNull(sourceRecord);
        }
        public void acknowledge() {
            acked.set(true);
        }
        public boolean isAcknowledged() {
            return acked.get();
        }
        public SourceRecord record() {
            return record;
        }
    }

and where acknowledge() is called from the producer callback and the commitOffsets() method can safely call isAcknowledged() and record() from the commit thread. The sendRecords() method would add a SubmittedRecord to the end of the queue for each record that will be sent to the producer:
    private boolean sendRecords() {
        ...
        for (final SourceRecord preTransformRecord : toSend) {
            ...
            SubmittedRecord submittedRecord = new SubmittedRecord(record);
            if (!submittedRecords.offer(submittedRecord)) {
                // If a blocking queue, then retry using the existing mechanism in WorkerSourceTask
                log.warn("{} Failed to add record to buffer. Backing off before retrying", this);
                toSend = toSend.subList(processed, toSend.size());
                lastSendFailed = true;
                counter.retryRemaining();
                return false;
            }
            ...

and then have the producer callback call the SubmittedRecord.acknowledge() method:
            try {
                ...
                producer.send(
                    producerRecord,
                    (recordMetadata, e) -> {
                        if (e != null) {
                            ...
                        } else {
                            submittedRecord.acknowledge();
                            ...
                        }
                    });

This effectively replaces the outstandingMessages, outstandingMessagesBacklog and flushing flag, and it simplifying the logic in the sendRecords() to not have to know which of those to use.
Then here's the big change: in commitOffsets(), we can dequeue all acked records, then take the snapshot of offsets, and immediately flush offsets without waiting for the producer. And by using a concurrent queue, we don't even need to synchronize between the sendRecords() method adding to the back of the queue and the commitOffsets() pulling from the front of the queue.
    public boolean commitOffsets() {
        ...
        // Dequeue all submitted records that have been
        while (!submittedRecords.isEmpty()) {
            SubmittedRecord next = submittedRecords.peek();
            if (!next.isAcknowledged()) {
                // This record is not yet acknowledge, so we can't continue processing any more offsets
                break;
            }
            submittedRecords.poll();
            // The record is acknowledged, so add the offsets to the offset writer
            // Offsets are converted & serialized in the OffsetWriter
            SourceRecord record = next.record();
            offsetWriter.offset(record.sourcePartition(), record.sourceOffset());
        }
        ...
        synchronized (this) {
            boolean flushStarted = offsetWriter.beginFlush();
            if (!flushStarted) {
                ...
            }
        }

        // Now we can actually flush the offsets to user storage.
        Future<Void> flushFuture = offsetWriter.doFlush((error, result) -> {
            if (error != null) {
                log.error("{} Failed to flush offsets to storage: ", WorkerSourceTask.this, error);
            } else {
                log.trace("{} Finished flushing offsets to storage", WorkerSourceTask.this);
            }
        });
        ...
    }

I've shown the snippet above using a non-blocking queue of unlimited size. I think we could do this because the existing WorkerSourceTask logic already handles the possibility that the producer.send(...) blocks when its buffer is full, up to max.block.ms before throwing a retriable exception, and then retrying the send if needed. Since this happens on the same thread that calls SourceTask.poll(), this existing logic already has the backpressure that is based upon the producer setting and that prevents the source task getting too far ahead of the producer.
Alternatively, we could use a blocking queue, but this would require an additional worker configuration, which is not ideal and can't be backported.
@C0urante, WDYT?

My previous suggestion simply dequeues all completed records until an unacked record is found. This is really straightforward, but we could try to do better.
We could dequeue all records except those that have a source partition that has not been acked. For example, let's say we have enqueue 4 records when commitOffsets() is called:
1. SubmittedRecord{ SourceRecord{...partition=P1,offset=O1...}, acked=true}
2. SubmittedRecord{ SourceRecord{...partition=P1,offset=O2...}, acked=true}
3. SubmittedRecord{ SourceRecord{...partition=P1,offset=O3...}, acked=false}
4. SubmittedRecord{ SourceRecord{...partition=P2,offset=O1...}, acked=true}
5. SubmittedRecord{ SourceRecord{...partition=P2,offset=O2...}, acked=true}
6. SubmittedRecord{ SourceRecord{...partition=P1,offset=O4...}, acked=false}
7. SubmittedRecord{ SourceRecord{...partition=P2,offset=O3...}, acked=true}
8. SubmittedRecord{ SourceRecord{...partition=P2,offset=O4...}, acked=true}
9. SubmittedRecord{ SourceRecord{...partition=P2,offset=O5...}, acked=true}
10. SubmittedRecord{ SourceRecord{...partition=P2,offset=O6...}, acked=false}

This might happen if records 4, 5, and 7-10 were written to different topic partitions than records 1, 2, 3, and 6, and the producer is stuck on the latter partitions.
With the simplistic logic, we'd only dequeue record 1 and 2, we'd add the offset for these two records to the offset writer, and we'd flush offset partition=P1,offset=O2. But the connector has made significantly more progress on source partition partition=P2, and it would be good to also flush offset partition=P2,offset=O7.
We could do that by dequeuing acked records only if their source partition map does not match a previously unacked message. This way, we'd end up with the following remaining in the queue (using the same record numbers as before):
3. SubmittedRecord{ SourceRecord{...partition=P1,offset=O3...}, acked=false}
6. SubmittedRecord{ SourceRecord{...partition=P1,offset=O4...}, acked=false}
10. SubmittedRecord{ SourceRecord{...partition=P2,offset=O6...}, acked=false}

This minor change will dramatically improve the ability to commit offsets closer to what has actually be acked.

Also, with my proposed approach, the offsetWriter is only used (directly or indirectly) within the commitOffsets() method. That means we could actually do without the synchronized(this) block around the offsetWriter.beginFlush().

@rhauch Overall that looks good to me. It's an elegant solution to the tricky problem you noted about the opacity of task-provided source offsets w/r/t ordering.
I'm a little worried about offset commits taking longer and longer with the more sophisticated approach you proposed (where we would unconditionally iterate over every record in the batch, instead of only until the first unacknowledged record). It's true that there would be natural back pressure from the producer as its buffer.memory gets eaten up, but with the default of 32MB, it still seems possible for a large number of unacknowledged records to build up. If this does happen, then offset commits may end up exceeding the offset.flush.timeout.ms for the worker, which may cause issues with the current model where a single shared, worker-global thread is used for offset commits of all tasks.
If this is a valid concern and we'd like to take it into account for now, I can think of a couple ways to handle it off the top of my head:

Use the simpler approach that blocks offset commits across the board if a single record remains unacknowledged for a long period of time (which may realistically be a problem if a single partition out of many is unavailable for some reason).
Enable concurrent offset commits by multiple tasks.
Instead of a single dequeue per task, use a ConcurrentMap<Map<String, ?>, Queue<SubmittedRecord>> that stores a single dequeue per unique source partition. This would allow us to iterate over the bare minimum number of records for every single offset commit and not spend time, for example, on accumulated records for unavailable Kafka partitions. We'd still have to iterate over those records eventually if the Kafka partition came back online, but that iteration would only ever occur once, instead of once for every offset commit.

I think option 3 may be warranted, although it's still possible that offset commits take a long time if 32MB worth of records end up getting queued. Option 2 may be worth implementing or at least considering as a follow-up item to handle this case.
Thoughts?

@C0urante, thanks for the feedback on my suggestion. I like your option 3, because it does allow the iteration to stop on each source partition as soon as it encounters the first unacknowledged record in each queue. I also think that the behavior with the suggested approach and your option 3 is still a lot better than the current situation.
One question, though: you mention that it might be a problem if iterating over the submitted records takes longer than offset.flush.timeout.ms. But IIUC the offset.flush.timeout.ms would actually not be used anymore, as there actually are no timeouts as the offset commit thread doesn't block anymore. So, worst case, if task A has a ton of submitted records that have to be iterated over (e.g., fast producer and fast source task), it might slow the committing of offsets for other tasks. (Again, this is not any worse than the current behavior.) But your option 2 would help with this at the risk of using more threads, and so we may also want to consider this to help ensure that no slowly-proceeding producer of a task blocks other offset commits.
Of course, another option might be to incur the iteration on the worker source task thread. That would essentially move the use of the queue(s) to the worker source task thread, tho we still need to get the offsets to the offset commit thread and so would likely have to keep the synchronization blocks around the offset writer snapshot. On one hand, that's putting more work onto the worker source task thread and making the offset thread super straightforward (snapshot and write); on the other it's putting the onus on the worker source task thread.
Thoughts?

I also think that the behavior with the suggested approach and your option 3 is still a lot better than the current situation.

Agreed 👍

IIUC the offset.flush.timeout.ms would actually not be used anymore, as there actually are no timeouts as the offset commit thread doesn't block anymore.

That's mostly correct--we wouldn't be waiting on a blocking operation while iterating through the dequeue(s), although we might still choose to block on the actual write to the offset topic in the same way that we currently do just for the sake of metrics and allowing users to monitor the health of the connection between the Connect worker and the offsets topic. Not a huge deal though, and the point that we wouldn't be blocking on the task's producer is still valid.
I think the issue is less that we'd end up timing out and more that we'd end up violating the guarantee that's provided right now by the framework that each task gets to take up only offset.flush.timeout.ms milliseconds per offset commit attempt before aborting the attempt and yielding control to the next task. A dequeue-based approach may actually be worse than the current behavior in that regard if there's no check in place to ensure that iterating over the dequeue doesn't exceed the offset flush timeout. Probably worth the tradeoff, but we can probably satisfy both objectives with your suggestion:

another option might be to incur the iteration on the worker source task thread.

I think this'd be great, especially with the snapshotting logic you mention, which should basically eliminate any blocking between the two threads except to prevent race conditions while simple operations like clearing a hash map or assigning a new value to an instance variable take place.
One thing that gave me pause initially was the realization that we'd be double-iterating over every source record at this point: once to transform, convert, and dispatch the record to the producer, and then once to verify that it had been acknowledged while iterating over the dequeue it's in. But I can't imagine it'd make a serious difference with CPU utilization given that transformation, conversion, and dispatching to a producer are likely to be at least an order of magnitude more expensive than just checking a boolean flag and possibly inserting the record's offset into a hash map. And memory utilization should be very close to the existing approach, which already tracks every single unacknowledged record in the outstandingMessages and outstandingMessagesBacklog fields.
I think this buys us enough that my earlier-mentioned option 2 (multiple threads for offset commits) isn't called for, since the only blocking operation that would be performed during offset commit at this point is a write to the offsets topic. If the offsets topic is unavailable, it's likely that the impact would be the same across all tasks (unless the task is using a separate offsets topic, which will become possible once the changes for KIP-618 are merged), and even if not, things wouldn't be made any worse than they already are: the offset flush timeout would expire, and the next task in line would get its chance to commit offsets.
@rhauch If this is all agreeable I think we're ready to start implementing. Since you've provided a lot of the code yourself I'm happy to let you take on that work if you'd like; otherwise, I'll get started and see if I can have a new PR with these changes out by early next week.

@C0urante wrote:

@rhauch If this is all agreeable I think we're ready to start implementing. Since you've provided a lot of the code yourself I'm happy to let you take on that work if you'd like; otherwise, I'll get started and see if I can have a new PR with these changes out by early next week.

Sounds good to me! I'm looking forward to your new PR; please link here and ping me. Thanks!

@rhauch (and, if interested, @hachikuji) new PR is up: https://github.com/apache/kafka/pull/11323

authors: C0urante mimaison rhauch

repo: apache/kafka


text:
MINOR: Update LICENSE-binary file

Tested with:
$ ./gradlewAll clean releaseTarGz
$ tar xzf core/build/distributions/kafka_2.13-3.8.0-SNAPSHOT.tgz
$ cd kafka_2.13-3.8.0-SNAPSHOT/
$ for f in $(ls libs | grep -v "^kafka\|connect\|trogdor"); do if ! grep -q ${f%.*} LICENSE; then echo "${f%.*} is missing in license file"; fi; done

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

If you need to run this several times, you can try to omit the test suite, it will be faster :)
./gradlewAll clean releaseTarGz -x test

authors: jlprat mimaison


text:
KAFKA-15517: Improve MirrorMaker logging in case of authorization errors

Log the list of topics for which an authorization error has been received when try to describe configs, along with the cluster alias.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Hello, @mimaison
Please, take a look.


text:
MINOR: KAFKA-3989_follow_up_PR: update script to run from kafka root eg ./jm…

…h-benchmarks/jmh.sh

@ijuma - script updated  ./jmh-benchmarks/jmh.sh works now

@ijuma - my apologies for the oversight.  Confirmed the following all work with this update.

./jmh.sh
./jmh-benchmarks/jmh.sh
jmh-benchmarks/jmh.sh

LGTM. Merged to trunk and cherry-picked to 0.11.0.

authors: bbejeck guozhangwang


text:
Add note about segment.ms to log cleaner docs

We were bitten by missing the essential segment.ms configuration option when using log compaction. Add a note about that to the "Configuring The Log Cleaner" section of the documentation.

Sorry, my editor removed some redundant whitespace from some lines.

@hachikuji Thanks for the feedback, I've made some changes based on it.

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/kafka-pr-test-coverage/275/

@hachikuji Is it a problem that the default build step failed? I'm not adding tests since this is just a change to the docs.

authors: asfgit thijsc


text:
KAFKA-6474: [WIP] Rewrite tests to use new public TopologyTestDriver

Please do not merge yet. This is still work in progress.

Remove ProcessorTopologyTestDriver from TopologyTest
Fix ProcessorTopologyTest
Remove ProcessorTopologyTestDriver and InternalTopologyAccessor
Partially refactored StreamsBuilderTest but missing one test
Refactor KStreamBuilderTest
Refactor AbstractStreamTest
Further cleanup of AbstractStreamTest
Refactor GlobalKTableJoinsTest
Refactor InternalStreamsBuilderTest
Fix circular dependency in build.gradle
Refactor KGroupedStreamImplTest
Partial modifications to KGroupedTableImplTest
Refactor KGroupedTableImplTest
Refactor KStreamBranchTest
Refactor KStreamFilterTest
Refactor KStreamFlatMapTest KStreamFlatMapValuesTest
Refactor KStreamForeachTest
Refactor KStreamGlobalKTableJoinTest
Refactor KStreamGlobalKTableLeftJoinTest
Refactor KStreamImplTest
Refactor KStreamImplTest
Refactor KStreamKStreamJoinTest
Refactor KStreamKStreamLeftJoinTest
Refactor KStreamKTableJoinTest
Refactor KStreamKTableLeftJoinTest
Refactor KStreamMapTest and KStreamMapValuesTest
Refactor KStreamPeekTest and KStreamTransformTest
Refactor KStreamSelectKeyTest
Refactor KStreamTransformValuesTest
Refactor KStreamWindowAggregateTest
Add Depercation anotation to KStreamTestDriver and rollback failing tests in StreamsBuilderTest and KTableAggregateTest
Refactor KTableFilterTest
Refactor KTableForeachTest
Add getter for ProcessorTopology, and simplify tests in StreamsBuilderTest
Refactor KTableImplTest
Remove unused imports
Refactor KTableAggregateTest

\cc @bbejeck @vvcephei

@h314to Thanks a lot for this PR -- it's quite big -- I was wondering if we could break it down into multiple smaller PRs? This would make reviewing simpler.

@mjsax Thank you very much for your thorough review. Sorry for the large PR, but a lot of stuff depends on KStreamTestDriver, which trying to remove. I like your suggestion: I'll move all the straightforward stuff into a separate PR, and take care the remaining stuff subsequent PRs. I'll also be addressing your comments above.

@h314to , Thanks for the PR! One strategy you might consider for sending smaller PRs is not to try and remove the new driver, but just to stop using it one test at a time.
This way, you can have an arbitrary number of PRs where you whittle down the usage or the current driver in favor of TopologyTestDriver and then one last PR where you remove the now-unused classes.

@h314to Are you working on the second PR to extract the "straight forward changes"? Or did I miss that you opened a PR for that already?

In order to keep comment history, which might make further code review simpler, I removed the thorny parts from this PR, addressed your previous comments, and rebased to the current trunk. I hope that is ok. I'll be working on the remaining classes and will submit smaller PR with the remaining stuff when they are done.

There were some conflicts due to recent commits in trunk. I rebased and fixed them.

Most of the tests use the same properties except for the application-id. I'm wondering if we want to add a method StreamsTestUtils where you pass in the application id and get back the properties as configured in the tests.
Or since it looks like all the tests use the same setup and cleanup methods do we want an abstract base test and provide a method to define the application id?

@bbejeck good suggestion. I'll suggest filing a JIRA and address it in a different PR :)

@h314to thanks for your contribution! Please ping us when you have the second PR ready for review.

Thank you all for your helpful reviews! @vvcephei left some nice tips after this was merged (thanks!), so if that is ok I'd like to address his comments in a followup cleanup PR.

Hi Filipe,
Yeah, I think that's a good solution. Sorry I was too slow on my review!
Thanks,
-John
On Fri, Apr 27, 2018 at 7:47 AM, Filipe Agapito notifications@github.com
wrote:

Thank you all for your helpful reviews! @vvcephei
https://github.com/vvcephei left some nice tips after this was merged
(thanks!), so if that is ok I'd like to address his comments in a followup
cleanup PR.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/kafka/pull/4832#issuecomment-384959961, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAy1EwXuf-pSSMrtl-KlrIDwXRqbxyNIks5tsxN-gaJpZM4TJXJ_
.

Ah, oh well, nevermind then. I missed that nuance.
On Fri, Apr 27, 2018 at 10:26 AM, Filipe Agapito notifications@github.com
wrote:

@h314to commented on this pull request.
In streams/src/test/java/org/apache/kafka/streams/kstream/internals/
KStreamKStreamJoinTest.java
https://github.com/apache/kafka/pull/4832#discussion_r184721987:

@@ -327,17 +341,15 @@ public void testWindowing() {
assertEquals(1, copartitionGroups.size());
assertEquals(new HashSet<>(Arrays.asList(topic1, topic2)), copartitionGroups.iterator().next());

   driver.setUp(builder, stateDir);

   driver = new TopologyTestDriver(builder.build(), props, time);

While I was changing it I realised that, although it would be cleaner to
remove time and use advanceTimeMs instead, it is actually easier to use
time because the tests require the time to advance and recede.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/apache/kafka/pull/4832#discussion_r184721987, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAy1Eweeb-r26r6ub5VdmfzpBVF815Ntks5tsziMgaJpZM4TJXJ_
.

authors: guozhangwang h314to mjsax vvcephei


text:
MINOR: Enable ignored upgrade system tests - trunk

Removed ignore annotations from the upgrade tests.  This PR includes the following changes for updating the upgrade tests:

Uploaded new versions 0.10.2.2, 0.11.0.3, 1.0.2, 1.1.1, and 2.0.0 (in the associated scala versions) to kafka-packages
Update versions in version.py, Dockerfile, base.sh
Added new versions to StreamsUpgradeTest.test_upgrade_downgrade_brokers including version 2.0.0
Added new versions StreamsUpgradeTest.test_simple_upgrade_downgrade test excluding version 2.0.0

Version 2.0.0 is excluded from the streams upgrade/downgrade test as StreamsConfig needs an update for the new version, requiring a KIP. Once the community votes the KIP in, a minor follow-up PR can be pushed to add the 2.0.0 version to the upgrade test.

Fixed minor bug in kafka-run-class.sh for classpath in upgrade/downgrade tests across versions.

Follow on PRs for 0.10.2x, 0.11.0x, 1.0.x, 1.1.x, and 2.0.x will be pushed soon with the same updates required for the specific version.
For testing, the following https://jenkins.confluent.io/job/system-test-kafka-branch-builder/1942/ branch builder was executed and passed.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@guozhangwang @mjsax @vvcephei for reviews

updated this

LGTM, cc @mjsax or @vvcephei for another look before merging.

retest this please

kicked off new branch builder https://jenkins.confluent.io/job/system-test-kafka-branch-builder/1960/

updated this

rebased this to pick up changes for failing tests


text:
KAFKA-8443 Broker support for fetch from followers

Follow on to #6731, this PR adds broker-side support for KIP-392 (fetch from followers).
Changes:

All brokers will handle FetchRequest regardless of leadership
Leaders can compute a preferred replica to return to the client
New ReplicaSelector interface for determining the preferred replica
Incremental fetches will include partitions with no records if the preferred replica has been computed
Adds new JMX to expose the current preferred read replica of a partition in the consumer

Two new conditions were added for completing a delayed fetch. They both relate to communicating the high watermark to followers without waiting for a timeout:

For regular fetches, if the high watermark changes within a single fetch request
For incremental fetch sessions, if the follower's high watermark is lower than the leader

A new JMX attribute preferred-read-replica was added to the kafka.consumer:type=consumer-fetch-manager-metrics,client-id=some-consumer,topic=my-topic,partition=0 object. This was added to support the new system test which verifies that the fetch from follower behavior works end-to-end. This attribute could also be useful in the future when debugging problems with the consumer.

retest this please

retest this please

Retest this please

retest this please

retest this please

retest this please

retest this please

retest this please

retest this please

Submitted #7033 for previous failures.

authors: hachikuji mumrah


text:
KAFKA-8855; Collect and Expose Client's Name and Version in the Brokers (KIP-511 Part 1)

This PR implements the first part of KIP-511, namely:

update the ApiVersionsRequest/Response to the auto-generated ones;
bump the versions of ApiVersionsRequest/Response;
add ClientSoftwareName and ClientSoftwareVersions in the ApiVersionsRequest;
add validation for those two fields and handle error in the client;
implement the proposed fail-back mechanism.

More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Thanks, @dajac .  This looks pretty close to being ready to go.  I left a few comments.

@cmccabe Thanks for the review. I have addressed all your comments.

LGTM

retest this please

retest this please

Failing tests
JDK 8 and Scala 2.11

org.apache.kafka.connect.integration.ConnectWorkerIntegrationTest.testAddAndRemoveWorker
org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector
org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSinkConnector
kafka.api.SslAdminClientIntegrationTest.testCreateTopicsResponseMetadataAndConfig

JDK 11 and Scala 2.12

org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector
org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSinkConnector

JDK 11 and Scala 2.13

org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSourceConnector
org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testSinkConnector

They are unrelated to the changes made in this PR.

I ran tests locally and got another failing test:

ReassignPartitionsClusterTest. shouldPerformMultipleReassignmentOperationsOverVariousTopics

I wanted to confirm that the failing tests in CI are flaky ones.

Those tests pass for me locally.  Committing

hi my friend,i don't know why you have to "add ClientSoftwareName and ClientSoftwareVersions in the ApiVersionsRequest;"。 It seems that the software version number doesn't work.

authors: cmccabe dajac hxfghgh


text:
KAFKA-9605: Do not attempt to abort batches when txn manager is in fatal error

We detected a bug in soak where the producer batches shall be failed in sender loop before the produce response callback. This shall trigger an illegal state exception on the producer batch as it is already aborted.
The impact is not severe since sender is on its own thread but should be fixed to avoid unnecessary critical exception.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@bob-barrett @hachikuji Could you take a look when you got time?

ok to test

Thanks @bob-barrett ! Note that we got 1/2 green build already.

test this please

retest this please

authors: abbccdda hachikuji ijuma vvcephei


text:
KNOX-2717: upgrade shiro

(It is very important that you created an Apache Knox JIRA for this change and that the PR title/commit message includes the Apache Knox JIRA ID!)
What changes were proposed in this pull request?
https://issues.apache.org/jira/browse/KNOX-2717
security issue in shiro
How was this patch tested?
(Please explain how this patch was tested. For instance: running automated unit/integration tests, manual tests. Please write down your test steps as detailed as possible)
(If this patch involves UI changes, please attach a screen-shot; otherwise, remove this)
Please review Knox Contributing Process before opening a pull request.

My 2 cents:

this change contains 2 upgrades in fact:

shiro
log4j2
Please make them committed separately.

I'd love to see some details on the tests you were executing

@smolnar82 I reverted the log4j change - I added it because the CI didn't kick off for my initial commit. I'm relying on the CI build test the changes. I'm not a Knox user. I'm involved with the ASF Security team and am trying to get ASF projects to start looking at their transitive dependencies. The ASF get many requests from users and now even government agencies asking us why we don't have all the latest security fixes applied in our projects.

@pjfanning - I got it.
In this case, please give me some time to checkout this PR and execute some sanity checks (at least PAM/LDAP authentication on the Shiro side).

Verified basic authentication methods. Let's wait for a green TravisCI run and merge this one.

repo: apache/knox


text:
Bump Netty 4.1.93.Final

Why are the changes needed?

Upgrade Netty to the latest Arrow-compatible version: 4.1.93.Final
Currently, we can not upgrade to 4.1.94.Final or above because of https://github.com/netty/netty/pull/13408
java.lang.NoSuchMethodError: 'io.netty.buffer.PoolThreadCache io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.threadCache()'
	at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.newDirectBufferL(PooledByteBufAllocatorL.java:164)
	at io.netty.buffer.PooledByteBufAllocatorL$InnerAllocator.directBuffer(PooledByteBufAllocatorL.java:214)
	at io.netty.buffer.PooledByteBufAllocatorL.allocate(PooledByteBufAllocatorL.java:58)
	at org.apache.arrow.memory.NettyAllocationManager.<init>(NettyAllocationManager.java:77)
	at org.apache.arrow.memory.NettyAllocationManager.<init>(NettyAllocationManager.java:84)
	at org.apache.arrow.memory.NettyAllocationManager$1.create(NettyAllocationManager.java:34)
	at org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:354)
	at org.apache.arrow.memory.BaseAllocator.newAllocationManager(BaseAllocator.java:349)
	at org.apache.arrow.memory.BaseAllocator.bufferWithoutReservation(BaseAllocator.java:337)
	at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:315)
	at org.apache.arrow.memory.BaseAllocator.buffer(BaseAllocator.java:279)
	at org.apache.arrow.vector.BaseVariableWidthVector.allocateBytes(BaseVariableWidthVector.java:462)
	at org.apache.arrow.vector.BaseVariableWidthVector.allocateNew(BaseVariableWidthVector.java:420)
	at org.apache.arrow.vector.BaseVariableWidthVector.allocateNew(BaseVariableWidthVector.java:380)
	at org.apache.spark.sql.execution.arrow.ArrowWriter$.$anonfun$create$1(ArrowWriter.scala:42)

How was this patch tested?

[ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

[ ] Add screenshots for manual tests if appropriate

[x] Run test locally before make a pull request

Codecov Report

Merging #4992 (9dd8f9a) into master (c2e861b) will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #4992   +/-   ##
======================================
  Coverage    0.00%   0.00%
======================================
  Files         561     561
  Lines       30953   30953
  Branches     4063    4063
======================================
  Misses      30953   30953

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

@cxzl25 I see the CVE, looks acceptable. Also cc @bowenliang123

Thanks, merged to master

repo: apache/kyuubi


text:
LOG4J2-1631 Honor timezone in file name pattern

Given pattern "app-%d{yyyy-MM-dd}{timezone}.log.gz" treat second pair of curly braces as timezone option.
If timezone option is not set, then use default timezone as before.
If timezone option is set, but it's parsing fails, then use GMT+0.
(same DatePatternConverter behavior as in other places)

BTW: run mvn spotless:apply -pl log4j-core to fix the formatting. Due to our current configuration whoever touches a file, has to format it entirely.

I would like to highlight that Calendar.getInstance().getTimeZone() is different from TimeZone.getDefault().
So pattern "logs/app-%d{yyyy-MM-dd}.log.gz" has slight change.
Before this PR, timezone used was Calendar.getInstance().getTimeZone()
After this PR, for same pattern timezone will be TimeZone.getDefault().
Is such change acceptable?
From Calendar.java:
private static TimeZone defaultTimeZone(Locale l) {
    TimeZone defaultTZ = TimeZone.getDefault();
    String shortTZID = l.getUnicodeLocaleType("tz");
    return shortTZID != null ?
        TimeZoneNameUtility.convertLDMLShortID(shortTZID)
            .map(TimeZone::getTimeZone)
            .orElse(defaultTZ) :
        defaultTZ;
}

Where are you seeing getTimeZone() implemented like that? It certainly doesn't look like that in Java 8 from what I can see. It pretty much just uses Timezone.getDefault().

@ppkarwasz
I have a nagging wish to convert PatternProcessorTest to use java.time and use date formats ISO_ZONED_DATE_TIME, ISO_INSTANT.
Would that be useful?
(this would reduce some verbosity; it could be done as separate "migration" commit)

Where are you seeing getTimeZone() implemented like that?

In jdk-11.

If they changed getTimeZone to work differently in Java 11 then I don't think us directly using TimeZone.getDefault() matters since 2.x is based on Java 8 support.

Great.

@dnsmkl,
Thanks for the remark, I didn't know it worked like this in Java 11. Experimenting a little bit with this feature I conclude that:

you can add timezone data to a locale: e.g. Locale.forLanguageTag("pl-PL-u-tz-uschi"), which uses America/Chicago as timezone,
no sane OS sets a timezone data for the "usual" locales,
our date formatters always use TimeZone.getDefault() if no timezone is set explicitly,
our rotation logic should do the same (we don't want timestamps in Warsaw time, but rotation on Chicago time),
fortunately there are only 5 call sites of Calendar.getInstance() in Log4j2: the two that you replaced with the correct Calendar.getInstance(TimeZone), two irrelevant in CronExpression and one in AbsoluteDateFormat.

I have a nagging wish to convert PatternProcessorTest to use java.time and use date formats ISO_ZONED_DATE_TIME, ISO_INSTANT. Would that be useful? (this would reduce some verbosity; it could be done as separate "migration" commit)

You can do whatever you want in test classes.
In the main classes like PatternProcessor however, we strive to achieve a garbage-free code, so AFAIK we are stuck with Calendar. @rgoers, is it safe to assume that calls to PatternProcessor#getNextTime are synchronized and we can replaced the two Calendar.getInstance() calls with cached instances? It certainly seems so in the current code.

... could you approach the problem of "daylight saving time" too?

Implementation works just as you described.
Tests added.

I'll squash everything into 2 commits:

honor timezone option
Test migration to java.time

Ok?

What name should I put in the release notes?

Danas Mikelinskas

I'll squash everything into 2 commits:
1. honor timezone option

2. Test migration to java.time

Ok?

Yes, please rebase and squash.

@dnsmkl, thank you for your contribution.

authors: dnsmkl ppkarwasz rgoers


text:
Introduce IVFFlat to Lucene for ANN similarity search [LUCENE-9136]

Representation learning (RL) has been an established discipline in the machine learning space for decades but it draws tremendous attention lately with the emergence of deep learning. The central problem of RL is to determine an optimal representation of the input data. By embedding the data into a high dimensional vector, the vector retrieval (VR) method is then applied to search the relevant items.
With the rapid development of RL over the past few years, the technique has been used extensively in industry from online advertising to computer vision and speech recognition. There exist many open source implementations of VR algorithms, such as Facebook's FAISS and Microsoft's SPTAG, providing various choices for potential users. However, the aforementioned implementations are all written in C+, and no plan for supporting Java interface, making it hard to be integrated in Java projects or those who are not familier with C/C+  [[https://github.com/facebookresearch/faiss/issues/105]].
The algorithms for vector retrieval can be roughly classified into four categories,

Tree-base algorithms, such as KD-tree;
Hashing methods, such as LSH (Local Sensitive Hashing);
Product quantization based algorithms, such as IVFFlat;
Graph-base algorithms, such as HNSW, SSG, NSG;

where IVFFlat and HNSW are the most popular ones among all the VR algorithms.
IVFFlat is better for high-precision applications such as face recognition, while HNSW performs better in general scenarios including recommendation and personalized advertisement. The recall ratio of IVFFlat could be gradually increased by adjusting the query parameter (nprobe), while it's hard for HNSW to improve its accuracy. In theory, IVFFlat could achieve 100% recall ratio.
Recently, the implementation of HNSW (Hierarchical Navigable Small World, LUCENE-9004) for Lucene, has made great progress. The issue draws attention of those who are interested in Lucene or hope to use HNSW with Solr/Lucene.
As an alternative for solving ANN similarity search problems, IVFFlat is also very popular with many users and supporters. Compared with HNSW, IVFFlat has smaller index size but requires k-means clustering, while HNSW is faster in query (no training required) but requires extra storage for saving graphs [indexing 1M vectors|[https://github.com/facebookresearch/faiss/wiki/Indexing-1M-vectors]]. Another advantage is that IVFFlat can be faster and more accurate when enables GPU parallel computing (current not support in Java). Both algorithms have their merits and demerits. Since HNSW is now under development, it may be better to provide both implementations (HNSW && IVFFlat) for potential users who are faced with very different scenarios and want to more choices.
The latest branch is lucene-9136-ann-ivfflat](https://github.com/irvingzhang/lucene-solr/commits/jira/lucene-9136-ann-ivfflat)

Migrated from LUCENE-9136 by Xin-Chun Zhang, 3 votes, updated Dec 09 2020
Attachments: glove-100-angular.png, glove-25-angular.png, image-2020-03-07-01-22-06-132.png, image-2020-03-07-01-25-58-047.png, image-2020-03-07-01-27-12-859.png, sift-128-euclidean.png

Hi guys
Thanks for your great job on Lucene and specially this ANN search!
Any progress on this issue? We're trying to use vector search, but HNSW seems to take too much memory...

authors: Mahdi-Seeker asfimport

repo: apache/lucene


text:
More consistently use a SEQUENTIAL ReadAdvice for merging.

Merging IOContexts use a SEQUENTIAL ReadAdvice. However, some file formats hardcode IOContext.LOAD for some of their files, which silences the whole merging context, in particular the SEQUENTIAL ReadAdvice.
This PR switches file formats to
ioContext.toReadAdvice(ReadAdvice.RANDOM_PRELOAD) so that merges will use a SEQUENTIAL ReadAdvice while searches will use a RANDOM_PRELOAD ReadAdvice.
This is not a huge deal for RANDOM_PRELOAD, which is only used for very small files. However, this change becomes more relevant for the new RANDOM ReadAdvice as we would like merges to keep using a SEQUENTIAL ReadAdvice.

Looks fine now. How should we handle the DEFAULT context?
I think for the purpose here it can be seen as READ.

Looks fine now. How should we handle the DEFAULT context?

I'm looking into a (separate) change that renames it to WRITE and enforces that it's never used for reading. Let's see how it goes. :)

I'm looking into a (separate) change that renames it to WRITE and enforces that it's never used for reading.

I opened #13236.

Thanks.

authors: jpountz uschindler


text:
[MDEPLOY-280] Ability so skip specified classifiers from deploy

Add deploy mojo parameter that allows one to skip certain classifiers
from deploy.
https://issues.apache.org/jira/browse/MDEPLOY-280

Am leaving this PR here for ref, but this is not for merge.

repo: apache/maven-deploy-plugin