text:
Run R CGAN Example Fail

Example Code:
https://github.com/apache/incubator-mxnet/blob/master/example/gan/CGAN_mnist_R/CGAN_train.R
Description
I try to run CGAN example, but it seems shape inference got trouble with CPU context.
Environment info (Required)
> sessionInfo()
R version 3.4.4 (2018-03-15)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.4 LTS

Matrix products: default
BLAS: /usr/lib/openblas-base/libblas.so.3
LAPACK: /usr/lib/libopenblasp-r0.2.18.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets
[6] methods   base

other attached packages:
 [1] bindrcpp_0.2   readr_1.1.1    dplyr_0.7.4
 [4] imager_0.40.2  magrittr_1.5   plyr_1.8.4
 [7] reticulate_1.6 tensorflow_1.5 keras_2.1.4
[10] mxnet_1.2.0

loaded via a namespace (and not attached):
 [1] Rook_1.1-1         purrr_0.2.4
 [3] colorspace_1.3-2   htmltools_0.3.6
 [5] viridisLite_0.3.0  base64enc_0.1-3
 [7] yaml_2.1.18        XML_3.98-1.10
 [9] rlang_0.2.0        pillar_1.2.1
[11] glue_1.2.0         RColorBrewer_1.1-2
[13] jpeg_0.1-8         bindr_0.1.1
[15] stringr_1.3.0      munsell_0.4.3
[17] gtable_0.2.0       visNetwork_2.0.3
[19] htmlwidgets_1.0    codetools_0.2-15
[21] evaluate_0.10.1    knitr_1.20
[23] tfruns_1.3         DiagrammeR_1.0.0
[25] Rcpp_0.12.16       scales_0.5.0
[27] backports_1.1.2    jsonlite_1.5
[29] rgexf_0.15.3       gridExtra_2.3
[31] brew_1.0-6         bmp_0.3
[33] png_0.1-7          ggplot2_2.2.1
[35] hms_0.4.2          digest_0.6.15
[37] stringi_1.1.7      grid_3.4.4
[39] rprojroot_1.3-2    influenceR_0.1.0
[41] tools_3.4.4        lazyeval_0.2.1
[43] tibble_1.4.2       readbitmap_0.1-4
[45] whisker_0.3-2      zeallot_0.1.0
[47] tidyr_0.8.0        pkgconfig_2.0.1
[49] downloader_0.4     assertthat_0.2.0
[51] rmarkdown_1.9      rstudioapi_0.7
[53] viridis_0.5.0      R6_2.2.2
[55] igraph_1.2.1       compiler_3.4.4

Build info (Required if built from source)
Compiler (gcc/clang/mingw/visual studio):
gcc --version
gcc (Ubuntu 5.4.1-2ubuntu1~16.04) 5.4.1 20160904
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

MXNet commit hash:
git rev-parse HEAD
91c2daa8c602a0f1a8ff945c317e0859a032358d

Error Message:
Error:
> mx.exec.update.arg.arrays(exec_D, arg_param_ini_D, match.name=TRUE)
Error in exec$update.arg.arrays(arg.arrays, match.name, skip.null) :
  [21:13:11] src/operator/random/../elemwise_op_common.h:123: Check failed: assign(&dattr, (*vec)[i]) Incompatible attr in node  at 0-th output: expected [64,64,4,4], got [64,32,3,3]

Stack trace returned 10 entries:
[bt] (0) /usr/local/lib/R/site-library/mxnet/libs/libmxnet.so(dmlc::StackTrace[abi:cxx11]()+0x5b) [0x7f4f560898cb]
[bt] (1) /usr/local/lib/R/site-library/mxnet/libs/libmxnet.so(dmlc::LogMessageFatal::~LogMessageFatal()+0x28) [0x7f4f5608a408]
[bt] (2) /usr/local/lib/R/site-library/mxnet/libs/libmxnet.so(bool mxnet::op::ElemwiseAttr<nnvm::TShape, &mxnet::op::shape_is_none, &mxnet::op::shape_assign, true, &mxnet::op::shape_string[abi:cxx11], -1, -1>(nnvm::NodeAttrs const&, std::vector<nnvm::TShape, std::allocator<nnvm::TShape> >*, std::vector<nnvm::TShape, std::allocator<nnvm::TShape> >*, nnvm::TShape const&)::{lambda(std::vector<nnvm::TShape, std::allocator<nnvm::TShape> >*, unsigned long, char const*)#1}::operator()(std::vector<nnvm::TShape, std::allocator<nnvm::TSh

Minimum reproducible example
Sys.setenv("LD_LIBRARY_PATH"="/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64")

require("imager")
require("dplyr")
require("readr")
require("mxnet") # GCC>=5.3.0

G_iterator<- function(batch_size){

    batch<- 0
    batch_per_epoch<-5

    reset<- function(){
        batch<<- 0
    }

    iter.next<- function(){
        batch<<- batch+1
        if (batch>batch_per_epoch) {
            return(FALSE)
        } else {
            return(TRUE)
        }
    }

    value<- function(){
        set.seed(123+batch)
        digit<- mx.nd.array(sample(0:9, size = batch_size, replace = T))
        data<- mx.nd.one.hot(indices = digit, depth = 10)
        data<- mx.nd.reshape(data = data, shape = c(1,1,-1, batch_size))
        return(list(data=data, digit=digit))
    }

    return(list(reset=reset, iter.next=iter.next, value=value, batch_size=batch_size, batch=batch))
}

D_iterator<- function(batch_size){

    batch<- 0
    batch_per_epoch<-5

    reset<- function(){
        batch<<- 0
    }

    iter.next<- function(){
        batch<<- batch+1
        if (batch>batch_per_epoch) {
            return(FALSE)
        } else {
            return(TRUE)
        }
    }

    value<- function(){
        set.seed(123+batch)
        idx<- sample(length(train_label), size = batch_size, replace = T)
        data<- train_data[,,,idx, drop=F]
        label<- mx.nd.array(train_label[idx])
        digit<- mx.nd.one.hot(indices = label, depth = 10)

        return(list(data=mx.nd.array(data), digit=digit, label=label))
    }

    return(list(reset=reset, iter.next=iter.next, value=value, batch_size=batch_size, batch=batch))
}

random_dim<- 96
gen_features<- 96
dis_features<- 32
image_depth = 1
fix_gamma<- T
no_bias<- T
eps<- 1e-5 + 1e-12
batch_size<- 64

data <-  mx.symbol.Variable('data')
gen_rand <- mx.symbol.normal(loc=0, scale=1, shape=c(1, 1, random_dim, batch_size), name="gen_rand")

G_sym <- mx.symbol.concat(data = list(data, gen_rand), num.args = 2, name="gen_concat") %>%
    mx.symbol.Deconvolution( name='g1', kernel=c(4,4), num_filter=gen_features*4, no_bias=T) %>%
    mx.symbol.BatchNorm( name='gbn1', fix_gamma=fix_gamma, eps=eps) %>%
    mx.symbol.Activation( name='gact1', act_type='relu') %>%
    mx.symbol.Deconvolution( name='g2', kernel=c(3,3), stride=c(2,2), pad=c(1,1), num_filter=gen_features*2, no_bias=no_bias) %>%
    mx.symbol.BatchNorm(name='gbn2', fix_gamma=fix_gamma, eps=eps) %>%
    mx.symbol.Activation(name='gact2', act_type='relu') %>%
    mx.symbol.Deconvolution(name='g3', kernel=c(4,4), stride=c(2,2),
                            pad=c(1,1), num_filter=gen_features, no_bias=no_bias) %>%
    mx.symbol.BatchNorm( name='gbn3', fix_gamma=fix_gamma, eps=eps) %>%
    mx.symbol.Activation( name='gact3', act_type='relu') %>%
    mx.symbol.Deconvolution( name='g4', kernel=c(4,4), stride=c(2,2), pad=c(1,1), num_filter=image_depth, no_bias=no_bias) %>%
    mx.symbol.Activation( name='G_sym', act_type='tanh')

data = mx.symbol.Variable('data')
dis_digit = mx.symbol.Variable('digit')
label = mx.symbol.Variable('label')

dis_digit<- mx.symbol.Reshape(data=dis_digit, shape=c(1,1,10,batch_size), name="digit_reshape") %>%
            mx.symbol.broadcast_to(data=., shape=c(28,28,10, batch_size), name="digit_broadcast")

data_concat <- mx.symbol.concat(list(data, dis_digit), num.args = 2, dim = 1, name='dflat_concat')

D_sym <-  mx.symbol.Convolution(data=data_concat, name='d1', kernel=c(3,3),
                                stride=c(1,1), pad=c(0,0), num_filter=24, no_bias=no_bias) %>%
    mx.symbol.BatchNorm( name='dbn1', fix_gamma=fix_gamma, eps=eps) %>%
    mx.symbol.LeakyReLU( name='dact1', act_type='elu', slope=0.25) %>%
    mx.symbol.Pooling( name="pool1", pool_type="max", kernel=c(2,2), stride=c(2,2), pad=c(0,0)) %>%
    mx.symbol.Convolution( name='d2', kernel=c(3,3), stride=c(2,2), pad=c(0,0), num_filter=32, no_bias=no_bias) %>%
    mx.symbol.BatchNorm( name='dbn2', fix_gamma=fix_gamma, eps=eps) %>%
    mx.symbol.LeakyReLU( name='dact2', act_type='elu', slope=0.25) %>%
    mx.symbol.Convolution( name='d3', kernel=c(3,3), stride=c(1,1), pad=c(0,0), num_filter=64, no_bias=no_bias) %>%
    mx.symbol.BatchNorm( name='dbn3', fix_gamma=fix_gamma, eps=eps) %>%
    mx.symbol.LeakyReLU( name='dact3', act_type='elu', slope=0.25) %>%
    mx.symbol.Convolution( name='d3', kernel=c(4,4), stride=c(1,1), pad=c(0,0), num_filter=64, no_bias=no_bias) %>%
    mx.symbol.BatchNorm( name='dbn4', fix_gamma=fix_gamma, eps=eps) %>%
    mx.symbol.LeakyReLU( name='dact4', act_type='elu', slope=0.25) %>%
    mx.symbol.Flatten(name="dflat") %>%
    mx.symbol.FullyConnected(data = ., name="dfc", num_hidden=1, no_bias=F) %>%
    mx.symbol.LogisticRegressionOutput(data=., label=label, name='D_sym')

input_shape_G<- c(1, 1, 10, batch_size)
input_shape_D<- c(28, 28, 1, batch_size)

graph.viz(G_sym, type = "graph", direction = "TD")
graph.viz(D_sym, type = "graph", direction = "TD")

devices<- mx.cpu()

data_shape_G<- c(1, 1, 10, batch_size)
data_shape_D<- c(28, 28, 1, batch_size)
digit_shape_D<- c(10, batch_size)

mx.metric.binacc <- mx.metric.custom("binacc", function(label, pred) {
  res <- mean(label==round(pred))
  return(res)
})

mx.metric.logloss <- mx.metric.custom("logloss", function(label, pred) {
  # KL Divergence D(x)log(D(x)) -D(x)log(D(G(z)))
  #
  res <- mean(label*log(pred)+(1-label)*log(1-pred))
  return(res)
})

iter_G<- G_iterator(batch_size = batch_size)
iter_D<- D_iterator(batch_size = batch_size)

exec_G<- mx.simple.bind(symbol = G_sym, data=data_shape_G, ctx = devices, grad.req = "write")
exec_D<- mx.simple.bind(symbol = D_sym, data=data_shape_D, digit=digit_shape_D, ctx = devices, grad.req = "write")

### initialize parameters - To Do - personalise each layer
initializer<- mx.init.Xavier(rnd_type = "gaussian", factor_type = "avg", magnitude = 3)

arg_param_ini_G<- mx.init.create(initializer = initializer, shape.array = mx.symbol.infer.shape(G_sym, data=data_shape_G)$arg.shapes, ctx = mx.cpu())
# aux 参数是针对 batchNorm 来更新的
aux_param_ini_G<- mx.init.create(initializer = initializer, shape.array = mx.symbol.infer.shape(G_sym, data=data_shape_G)$aux.shapes, ctx = mx.cpu())

arg_param_ini_D<- mx.init.create(initializer = initializer, shape.array = mx.symbol.infer.shape(D_sym, data=data_shape_D, digit=digit_shape_D)$arg.shapes, ctx = mx.cpu())

aux_param_ini_D<- mx.init.create(initializer = initializer, shape.array = mx.symbol.infer.shape(D_sym, data=data_shape_D, digit=digit_shape_D)$aux.shapes, ctx = mx.cpu())

mx.exec.update.arg.arrays(exec_G, arg_param_ini_G, match.name=TRUE)
mx.exec.update.aux.arrays(exec_G, aux_param_ini_G, match.name=TRUE)

mx.exec.update.arg.arrays(exec_D, arg_param_ini_D, match.name=TRUE)

Steps to reproduce
(Paste the commands you ran that produced the error.)
critical command:
mx.exec.update.arg.arrays(exec_D, arg_param_ini_D, match.name=TRUE)

What have you tried to solve it?
I thought the input data shape is wrong,however, that is not the key problem:

exec_D$ref.arg.arrays %>% sapply(dim) %>% with(d1_weight)
[1]  3  3 11 24
arg_param_ini_D %>% sapply(dim) %>% with(d1_weight)
[1]  3  3 11 24

I am wrong at Discriminator pipeline operator , just copy original code, it works.

repo: apache/incubator-mxnet


text:
Very Large CPU RAM Memory Consumption (>1GB)

Description
Mxnet consumes nearly 2GB CPU RAM even when loading a relatively small model (e.g. Resnet-18) directed on GPU (ctx=mxnet.gpu()). From what I understand, there is no real need to use so much CPU memory when the model is running on GPU.
This issue is extremely prohibitive when trying to run multiple processes with mxnet on the same machine, and IMO gives it a significant disadvantage compared to other frameworks for being used in AI production systems.
Environment info
/usr/bin/python3.6 /home/ran-face/src/CameraResearch/workspace/mxnet_diagnose.py
----------Python Info----------
Version      : 3.6.7
Compiler     : GCC 8.2.0
Build        : ('default', 'Oct 22 2018 11:32:17')
Arch         : ('64bit', 'ELF')
------------Pip Info-----------
Version      : 9.0.1
Directory    : /usr/lib/python3/dist-packages/pip
----------MXNet Info-----------
Version      : 1.3.1
Directory    : /usr/local/lib/python3.6/dist-packages/mxnet
Commit Hash   : 19c501680183237d52a862e6ae1dc4ddc296305b
----------System Info----------
Platform     : Linux-4.15.0-45-generic-x86_64-with-Ubuntu-18.04-bionic
system       : Linux
node         : ranface-Lenovo-Y720-15IKB
release      : 4.15.0-45-generic
version      : #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019
----------Hardware Info----------
machine      : x86_64
processor    : x86_64
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              8
On-line CPU(s) list: 0-7
Thread(s) per core:  2
Core(s) per socket:  4
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               158
Model name:          Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
Stepping:            9
CPU MHz:             2411.711
CPU max MHz:         3800.0000
CPU min MHz:         800.0000
BogoMIPS:            5616.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            6144K
NUMA node0 CPU(s):   0-7
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp flush_l1d
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0694 sec, LOAD: 0.9665 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0841 sec, LOAD: 1.2019 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.0988 sec, LOAD: 0.9435 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0708 sec, LOAD: 1.2242 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0637 sec, LOAD: 1.3118 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0499 sec, LOAD: 0.3116 sec.

Process finished with exit code 0

Package used (Python/R/Scala/Julia):
I'm using Python3
Build info
mxnet installed using pip3
Steps to reproduce
(Paste the commands you ran that produced the error.)

Run the following code

import mxnet as mx
import time

if __name__ == '__main__':

    path='http://data.mxnet.io/models/imagenet/'
    [mx.test_utils.download(path+'resnet/18-layers/resnet-18-0000.params'),
     mx.test_utils.download(path+'resnet/18-layers/resnet-18-symbol.json'),
     mx.test_utils.download(path+'synset.txt')]

    ctx = mx.gpu()

    sym, arg_params, aux_params = mx.model.load_checkpoint('resnet-18', 0)
    mod = mx.mod.Module(symbol=sym, context=ctx, label_names=None)
    mod.bind(for_training=False, data_shapes=[('data', (1,3,224,224))],
             label_shapes=mod._label_shapes)
    mod.set_params(arg_params, aux_params, allow_missing=True)

    time.sleep(100)

check process memory (run top, shift M to sort processes by memory usage)
memory usage is about ~1.5-2GB RAM

@mxnet-label-bot Add [Performance, Memory]

@rvardimon thanks for raising the issue. Have you looked at the profiler output from mxnet , when you load the model params it initially loads on cpu so you may see a spike but then it should be released after set_params is called. if not module api is adding some overhead which should be looked at: @karan6181

I see roughly 2 GB of CPU memory usage when I ran the user script and also with the below minimum reproducible script using Gluon.

import mxnet as mx
import gluoncv
from time import sleep

net = gluoncv.model_zoo.get_model('cifar_resnet20_v1', pretrained=True, ctx=mx.gpu())
net.hybridize()

#print(net.summary(mx.nd.ones((1, 3,28,28), ctx=mx.gpu())))
sleep(20)

I tried using different MXNet version but the CPU memory usage keeps on increasing. Not Much difference in memory consumption if using with/without MKL.

# P3.16xLarge
# DLAMI V24

MXNet-cu92: 1.3.1: 2.15 GB
MXNet-cu92: 1.4.1: 2.3 GB
MXNet-cu92: 1.5.0: 2.55 GB
MXNet-cu92: 1.6.0(1.6.0b20190906): 2.9 GB

If I run the same above script on CPU context, then the memory usage is approx. 100 MB.

We ( I and @karan6181  ) looked at a bunch of things to understand where the overhead is coming from. We looked at Resource Request and attach op resources pass. We looked at object pool in threaded engine, and at turning off OPENMP and MKLDNN and checking, but the overhead is still there. We looked at the overhead caused by different arrays in the module api.  Overhead is not coming from any of these areas.
We also check that increase in memory consumption happens at the bind stage and probably coming from somewhere in the graph executor. Next step is to check this in 1.2.1 to see if the increase happened in 1.3.1.

19   1488.9 MiB    252.7 MiB           sym, arg_params, aux_params = mx.model.load_checkpoint(prefix, epoch)
20   1489.0 MiB      0.1 MiB           all_layers = sym.get_internals()
21   1489.1 MiB      0.1 MiB           sym = all_layers['fc1_output']
22   1489.2 MiB      0.1 MiB           self.model = mx.mod.Module(symbol=sym, context=ctx, label_names=None)
23   3331.3 MiB   1842.2 MiB           self.model.bind(data_shapes=[('data', (1, 3, 112, 112))])
24   3333.9 MiB      2.5 MiB           self.model.set_params(arg_params, aux_params)

I used mxnet-cu100mkl        1.4.1 in win10.

@anirudh2290  Hi did you discover any solution for this memory consumption issue ?
Best

source: gharchive/issue

authors: MyraBaba anirudh2290 ddavydenko jiangwch karan6181 rvardimon

license_type: permissive

license_source: github-api


text:
MXNet nightly cpu build (w/o MKLDNN) does not work with Horovod

Description
Installing Horovod with MXNet nightly CPU (w/o MKLDNN) failed. GPU build and CPU mkldnn build are both okay.
Error Message:
OSError: /usr/local/lib/python3.6/dist-packages/horovod/mxnet/mpi_lib.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN5mxnet10CopyFromToERKNS_7NDArrayEPS1_i
@TaoLv Is it also related to the change you made recently that fixed MKLDNN build?
@szha @leezu FYI

Hmm if you're using the latest nightly build, I'm afraid MKL-DNN is already enabled in it. Could please share the output of mx.runtime.Features()?

Tested with the latest nightly build and the issue was gone.

authors: TaoLv apeforest


text:
Periodic Loss Value when training with "step" learning rate policy

When training deep CNN, a common way is to use SGD with momentum with a "step" learning rate policy(e.g. learning rate set to be 0.1,0.01,0.001.. at different stages of training).But I encounter an unexpected phenomenon  when training with this strategy under MXNet.
That is the periodic training loss value

The above is the training loss at a fixed learning rate 0.01, where the loss is decreasing normally

However, at the second stage of training(with lr 0.001) , the loss goes up and down periodically, and the period is exactly an epoch
So I thought it might be the problem of data shuffling, but it cannot explain why it doesn't happen in the first stage.  Actually I used ImageRecordIter as the DataIter and reset it after every epoch, is there anything I missed or set mistakenly
train_iter = mx.io.ImageRecordIter(
	path_imgrec=recPath,
	data_shape=dataShape,
	batch_size=batchSize,
	last_batch_handle='discard',
	shuffle=True,
	rand_crop=True,
	rand_mirror=True)

The codes for training and loss evaluation
while True:
	train_iter.reset()
	for i,databatch in enumerate(train_iter):
                globalIter += 1
		mod.forward(databatch,is_train=True)
		mod.update_metric(metric,databatch.label)
		if globalIter % 100 == 0:
                    loss = metric.get()[1]
                    metric.reset()
                mod.backward()
                mod.update()

Actually the loss can converge, but it takes too long.
I've suffered from this problem for a long period of time, on different network and different datasets.
I didn't have this problem when using Caffe.  Is this due to the implementation difference?
Anyone can give some suggestions?

I found my training loss has the same problem:
When the training step into a new epoch, the loss goes down abruptly, then goes up gradually in this epoch.
Dose anyone know why it happens?

I have the same problem

authors: MDR-EX1000 cccorn denemmy


text:
mx.nd.array indexing broken in armv7 / raspberrypi / jessie 8.0 (5 dimensional tensor)

Description
test_ndarray_slice broken in armv7
multidimensional array indexing broken for 5 dimensions.
Environment info (Required)
armv7 / raspberrypi / jessie 8.0
Simple reproduction:
nosetests --verbose python/unittest/test_ndarray.py

Manual reproduction with a simpler example:
In [150]: a=mx.nd.array([[[[[0,1,2],[3,4,5]]]]])

In [151]: a[0]
Out[151]:

[[[[ 0.  1.  2.]
   [ 3.  4.  5.]]]]
<NDArray 1x1x2x3 @cpu(0)>

In [153]: a[0]
Out[153]:

[[[[ 0.  1.  2.]
   [ 3.  4.  5.]]]]
<NDArray 1x1x2x3 @cpu(0)>

In [154]: a[0,0]
Out[154]:

[[[  2.00000000e+00   3.00000000e+00   4.00000000e+00]
  [  5.00000000e+00   0.00000000e+00   5.53355948e-39]]]
<NDArray 1x2x3 @cpu(0)>

In [155]: a[0,0,0]
Out[155]:

[[  3.00000000e+00   4.00000000e+00   5.00000000e+00]
 [  0.00000000e+00   5.53355948e-39   9.62310147e+27]]
<NDArray 2x3 @cpu(0)>

In [156]: a[0,0,0,0]
Out[156]:

[ 4.  5.  0.]
<NDArray 3 @cpu(0)>

In [157]: a[0,0,0,0,0]
Out[157]:

[ 0.]
<NDArray 1 @cpu(0)>

In [158]:

Possibly due to int32 vs int64?

I'm looking at this issue.

I think the problem must be in array slicing inside the native lib. I have debugged the python code and the call to slice is exactly the same in mac and arm:
addressing a[0,0] calls
op.slice(handle, begin=[0,0], end=[1,1])

@larroy Yes, there is a high chance that something went wrong in the op's backend implementation on arm devices. Can you run gdb to find out the root cause on your device? Or you can printing statements in the code to output intermediate information.

@reminisce that's what I'm trying to do, thanks.

Found the bug:
in matrix_op-inl.h:472 we are addressing the parameter vector outside of bounds. If you slice a 5 dimensional tensor, the loop is accessing past the end, since the param.begin and param.end have only size 2.
When you access a[0,0] it translates to a slice(begin=(0,0), end(1,1)) for example.
This is broken, and what's funny is that is working in other platforms...
I'm working on a fix. Why is param.end not even used in this slice function?

Great catch! This looks like a serious bug. param.end was used in inferring output shape.

Right, we verified this with @KellenSunderland. The oshape is inferred and param.end is used in inline TShape GetSliceShape(const SliceParam& param, const TShape& dshape)
I will send a PR with a proposed fix shortly. Unfortunately this bug is not revealed unless the right garbage is on the uninitialized params beyond ndim.

authors: larroy piiswrong reminisce


text:
Fix windows setup doc using VS 2017

Description
Update the MXNet windows installation doc by adding the details about how to install MXNet from source on windows using VS 2017. Besides, also add more details about the installation with OpenCV, Cmake, CUDA, cuDNN. Should be helpful to solve issues #7945 , #8155 .
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.

[ ] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
[ ] Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
[ ] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

@sxjscience please take a look at this as well.

@aaronmarkham

Looks great!

Also, we have another place that shows the windows setup doc. http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-10363/1/install/windows_setup.html . @aaronmarkham Will these two be merged later? Should we revise only one page?

I think we should put 2017 as option 1

@piiswrong Put 2017 as option 1, and add one more detail about setting up the vs 2017 version suggested by @yajiedesign .

source: gharchive/pull-request

authors: cgraywang piiswrong sxjscience


text:
[MXNET-1358]Fit api tutorial

Description
bring back https://github.com/apache/incubator-mxnet/pull/14462
main contributor @piyushghai
I have updated the tutorial to the latest fit/estimator api
@ThomasDelteil @thomelane @aaronmarkham

@mxnet-label-bot add [pr-awaiting-review]

@aaronmarkham I have addressed your comments, could you take another look? thanks!
Another problem is the estimator API doc is not showing up on the website, not sure if this will fix it

any update on this ? mxnet 1.5 is released but no doc for the feature.

I have addressed commend @aaronmarkham @abhinavs95 could you take another look? Thanks

I have verified the documentation is successfully generated, tutorial is passing, nightly test are passing and unit test should be already checked by CI
ping @aaronmarkham for review on the docs. Thanks!

Update website API docs can be found at: http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-15353/10/api/python/gluon/contrib.html
Tutorial: http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-15353/10/tutorials/gluon/fit_api_tutorial.html

authors: anirudhacharya eric-haibin-lin roywei


text:
[NEMO-232] Implement InputWatermarkManager

JIRA: NEMO-232: Implement InputWatermarkManager
Major changes:

Tracks the minimum watermark among multiple input streams by creating InputWatermarkManager
Emits watermarks from OutputCollector to InputWatermarkManager. Then InputWatermarkManager emits the minimum input watermarks to next operators

Minor changes to note:

Create NextOperatorInfo to wrap edge index, OperatorVertex, and InputWatermarkManager.

Tests for the changes:

Create InputWatermarkManagerTest
Add testMultipleIncomingEdges to TaskExecutorTest

Other comments:
Closes #137

@johnyangk Thanks for the review! I've addressed your comments

repo: apache/incubator-nemo


text:
Saving a file downloaded with wget

Saving a file downloaded with wget
wget, as is coded, only displays downloaded files on console.
This patch allow saving files on  as per command syntax:
        wget [-o <local-path>] <url>

Tested on esp32-devkit.
               mount -t spiffs /dev/esp/partition/spiffs /var
               wget -o /var/test4.txt http://192.168.1.43:8000/partitions.csv

@koliqi could you remove the merge commit from the PR? and fix the check error here:
https://github.com/apache/incubator-nuttx-apps/runs/4515099082?check_suite_focus=true

let's track the change by: https://github.com/apache/incubator-nuttx-apps/pull/925

repo: apache/incubator-nuttx-apps


text:
cxd56: mkspk tool should be built when CONFIG_CXD56_BINARY is enabled

Right now there is a manual step build the mkspk tool which means we do not actually test this functionality in CI.

on it

https://github.com/apache/incubator-nuttx/pull/2145

authors: btashton jerpelea

repo: apache/incubator-nuttx


text:
arch: arm: z16: Author Gregory Nutt: update licenses to Apache

Summary
Gregory Nutt has submitted the SGA and we can migrate the licenses to Apache.
Impact
NONE
Testing
NONE

@xiaoxiang781216 comments?

@xiaoxiang781216 comments?

LGTM.

this one finished

@jerpelea - Shall we ignore the Tabs and mix case?

please ignore
they will be fixed on the cleanup phase for the whole nuttx


text:
health actions can reuse containers and may not flag an unhealthy invoker

There are scenarios for which the health actions will not correctly flag an unhealthy invoker:

health action reuses a previous container
health action is allocated a stem cell container

In either of these cases, if the invoker is actually unhealthy and cannot create a new container, the health action will provide a false indicator of healthiness - causing the invoker state to flop between health and unhealthy when in fact docker is already sick or the vm is sick.
The invoker should know that it is unhealthy in these cases, and could suspend its pings and generate an alert instead. This will cause new requests to be reassigned until the health is properly restored. In another way, the pings could carry various error counts: cloudant, kafka, docker and that could be used to take an invoker topic offline. If the errors are more explicitly communicated, I wonder if we still need the health action tests apart from an initial prologue/warm up of the invoker, or when the invoker topic doesn't have any messages already in queue.
Alternatively, the health actions must not be allowed to reuse a warm container since otherwise they're just misleading.
FYI @cbickel @sven-lange-last

With multiple active controllers and no sharding, a status but flip will cause multiple controllers to send health actions. Is that right @cbickel?

Yes, that's correct.

authors: cbickel rabbah

repo: apache/incubator-openwhisk


text:
Bump fetch.max.wait of Kafka-consumers to lower the load on Kafka in large systems.

Description
In systems with a lot of invokers, all Kafka-consumers are slowing down Kafka if they are ideling.
I did some tests with the fetch.max.wait.ms-parameter.
I produce messages to one topic. Each consumer has it’s own topic. So if there are 100 consumers, 99 of them are idling.
I used the command kafka-producer-perf-test.sh --topic invoker0 --num-records 100000 --throughput 999999999 --record-size 600 --producer-props bootstrap.servers=localhost:9072 for the test.

20ms
100ms
500ms
1000ms

1 consumers
39698.292973 records/sec
40225.261464 records/sec
39047.247169 records/sec

10 consumers
39936.102236 records/sec
39729.837108 records/sec
42625.745951 records/sec

50 consumers
25144.581343 records/sec
36324.010171 records/sec
39603.960396 records/sec

100 consumers
16860.563143 records/sec
28425.241615 records/sec
35435.861091 records/sec
36616.623947 records/sec

On the test-machine, a got these numbers. The only changes were made in the amount of consumers and topics and the parameter fetch.max.wait.
As you can see, the procuder can write much more messages to Kafka, if the idling consumers are having a higher fetch.max.wait.
As fetch.min.bytes is set to 1, there should be no latency-impact on executing activations.
Related issue and scope

[ ] I opened an issue to propose and discuss this change (#????)

My changes affect the following components

[ ] API
[ ] Controller
[x] Message Bus (e.g., Kafka)
[ ] Loadbalancer
[ ] Invoker
[ ] Intrinsic actions (e.g., sequences, conductors)
[ ] Data stores (e.g., CouchDB)
[ ] Tests
[ ] Deployment
[ ] CLI
[ ] General tooling
[ ] Documentation

Types of changes

[x] Bug fix (generally a non-breaking change which closes an issue).
[ ] Enhancement or new feature (adds new functionality).
[ ] Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

[x] I signed an Apache CLA.
[x] I reviewed the style guides and followed the recommendations (Travis CI will check :).
[ ] I added tests to cover my changes.
[ ] My changes require further changes to the documentation.
[ ] I updated the documentation where necessary.

Codecov Report

Merging #3971 into master will decrease coverage by 4.53%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #3971      +/-   ##
==========================================
- Coverage   85.29%   80.75%   -4.54%
==========================================
  Files         146      146
  Lines        7057     7057
  Branches      413      413
==========================================
- Hits         6019     5699     -320
- Misses       1038     1358     +320

Impacted Files
Coverage Δ

...ain/scala/whisk/core/invoker/InvokerReactive.scala
73.94% <100%> (+15.12%)
:arrow_up:

...core/database/cosmosdb/RxObservableImplicits.scala
0% <0%> (-100%)
:arrow_down:

...core/database/cosmosdb/CosmosDBArtifactStore.scala
0% <0%> (-95.1%)
:arrow_down:

...sk/core/database/cosmosdb/CosmosDBViewMapper.scala
0% <0%> (-92.6%)
:arrow_down:

...whisk/core/database/cosmosdb/CosmosDBSupport.scala
0% <0%> (-81.82%)
:arrow_down:

...abase/cosmosdb/CosmosDBArtifactStoreProvider.scala
0% <0%> (-58.83%)
:arrow_down:

...whisk/connector/kafka/KafkaProducerConnector.scala
75.67% <0%> (-16.22%)
:arrow_down:

...la/whisk/core/database/cosmosdb/CosmosDBUtil.scala
92% <0%> (-4%)
:arrow_down:

...rc/main/scala/whisk/common/ForcibleSemaphore.scala
88.46% <0%> (-3.85%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f365d1c...729fa0e. Read the comment docs.

authors: cbickel codecov-io


text:
Reverting d990b171f5d3f14a2a196363fb315594c46afab6

Motivation
There was a concern that the client will end up using 30x the memory if there is a GC pause or N/W issue. Hence changing the queue size back to 1K.
Modifications
Reverted commit d990b171f5d3f14a2a196363fb315594c46afab6
Result
Expect to see more ProducerQueueFull error/exceptions in case of GC pauses.

@rdhabalia @saandrews @msb-at-yahoo @merlimat

Separated from the discussion on whether it's good or not, please use a more descriptive log commit subject :). It can be difficult to understand the change from the git log.

repo: apache/incubator-pulsar


text:
Update CONTRIBUTING.md

A small update to the contributing guide to be more inline with the wiki

+1. Thanks!

+1

authors: abifet gdfm gyfora

repo: apache/incubator-samoa

license_source: bigquery


text:
if columns has name "source"  will report error When update version from 4.0.0 to 4.0.1.  like select/insert/update.

Error querying database.  Cause: java.lang.ClassCastException: org.antlr.v4.runtime.tree.ErrorNodeImpl cannot be cast to org.antlr.v4.runtime.ParserRuleContext
The error may exist in com/lumi/aiot/cloud/mapper/IotUserMappingMapper.java (best guess)
The error may involve com.lumi.aiot.cloud.mapper.IotUserMappingMapper.selectByExample-Inline
The error occurred while setting parameters
SQL: SELECT  id,user_id,source,value,country_code,create_time,update_time  FROM iot_user_mapping  WHERE       (  source = ? )
Cause: java.lang.ClassCastException: org.antlr.v4.runtime.tree.ErrorNodeImpl cannot be cast to org.antlr.v4.runtime.ParserRuleContext
When can fix this bug? Project Urgent

What is your version?

@aqara The same issue to #4546 .I will close this issue and discuss in #4546 .

authors: SteNicholas aqara terrymanu

repo: apache/incubator-shardingsphere


text:
Add some missing documents for v6

Prepare v6 release.

Coverage increased (+0.1%) to 12.29% when pulling 7c1a995bfaff9dcf62093dca15188c881b881bdf on v6-doc into 0c6a784e99354c694a2747a4bb911f035c2cf8da on master.

authors: coveralls wu-sheng

repo: apache/incubator-skywalking


text:
[Bug] checkFlinkEnv do not refresh the hdfs flink lib dir while there are changes in local flink lib dir,

Search before asking

[X] I had searched in the issues and found no similar issues.

Java Version
java 8
Scala Version
2.11.x
StreamPark Version
2.1.1
Flink Version
1.16.2
deploy mode
yarn-application
What happened
while we have configured a flink home before and use it to start a flink job with our jar using yarn-application mode.At the same time, this JAR will use a new connector. Since we are building a source code JAR package, we are uploading the connector to the local lib directory of Flink.In this case, when executing the Flink JAR, an error occurs indicating that the class for the connector cannot be found.
The issue arises because the EnvInitializer.checkFlinkEnv method uploads the Flink directory to the HDFS StreamPark workspace only on certain occasions, such as when starting or deploying the application. In the YARN application mode, the /streampark/flink/flink-1.16.2/lib directory is included in the classpath. However, since the checkFlinkEnv method checks whether the directory exists before uploading files (if (!fsOperator.exists(flinkHome))), any subsequent modifications to the local Flink lib directory will not be updated in HDFS.
We can only manually upload any added JARs to HDFS, which is not very user-friendly. This is because, without inspecting the source code, users may not be aware that uploading to this directory（/streampark/flink/flink-1.16.2/lib） is required for working nomally.
Error Exception
Caused by: java.lang.NoClassDefFoundError: com/ververica/cdc/connectors/postgres/PostgreSQLSource

if (!fsOperator.exists(flinkHome)) {
      log.info("{} is not exists,upload beginning....", flinkHome);
      fsOperator.upload(flinkLocalHome, flinkHome, false, true);
 }

Screenshots

Are you willing to submit PR?

[X] Yes I am willing to submit a PR!(您是否要贡献这个PR?)

Code of Conduct

[X] I agree to follow this project's Code of Conduct

Thank you for your report and look forward to your repair

Thank you for your report and look forward to your repair

Thank you for your response. I am honored to receive your reply.
My idea is that in the application mode, it mainly uses the /streampark/flink/flink-{version}/lib and /streampark/flink/flink-{version}/plugins directories on HDFS as yarn.provided.lib.dirs. In the production environment, there might be a need to update third-party libraries in flink/lib to support the execution of new job JARs. I'm not sure if our project usage is correct, and I appreciate your guidance if there are alternative approaches.
Therefore, when there are changes in the lib directory of the local Flink home, it is necessary to update the files in these two directories on the remote HDFS. While manually uploading to these directories can solve the problem, it generally poses challenges for Streampark deployers: they may not be familiar with directly manipulating HDFS directories, might not be aware that files need to be uploaded to these two directories, or may not have the necessary permissions on the machines to operate HDFS.

My idea is whether it is possible to add a logic inside the if (!fsOperator.exists(flinkHome)) statement in the EnvInitializer.checkFlinkEnv method. When the mode is yarn application, use the HdfsOperator.fileMd5 method to verify whether the files in the local and HDFS lib directories are consistent, and then trigger the fsOperator.upload logic again if necessary.Just like the pseudocode below。

if (!fsOperator.exists(flinkHome)) {
log.info("{} is not exists,upload beginning....", flinkHome);
fsOperator.upload(flinkLocalHome, flinkHome, false, true);
}
//here is the pseudocode,upload the differences to the remote directory
if (mode == YarnApplicationMode && !HdfsOperator.fileMd5(localLibDir).equals(HdfsOperator.fileMd5(hdfsLibDir))) {
// Files are not the same, trigger upload logic
fsOperator.upload(localLibDir, hdfsLibDir);
}

Alternatively, the web page could provide an option to manually upload and trigger the update of the Flink home's 'lib' directory.

Thanks for your feedback. The current design treats the local Flink as just a reference and mapping. It is not recommended to modify the local Flink files. If there are jobs running in yarn application mode and there are updates to the local Flink files, you would need to manually operate on HDFS to update the changed files into it.

authors: caicancai liuxinjie5211990 wolfboys

repo: apache/incubator-streampark


text:
Changed TinkerGraphSerializer to reuse existing Kryo instance

Let the tinker graph serializer reuse an existing kryo instance instead of creating a new one.
Tests assert that the same kryo instance is being used inside the Gryo reader/writer classes.
https://groups.google.com/forum/#!topic/gremlin-users/apFfQW3DkK4

thanks for doing this - about to merge to master.

authors: christianbellinaef spmallette

repo: apache/incubator-tinkerpop


text:
[feature request] Support for cuda 11

Currently, if I build tvm with cublas and CUDA 11, I'll encounter the following error:
/home/***/repo/incubator-tvm/src/runtime/contrib/cublas/cublas.cc: In function ‘void tvm::contrib::CallLtIgemm(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*, cublasLtHandle_t)’:
/home/***/repo/incubator-tvm/src/runtime/contrib/cublas/cublas.cc:175:63: error: cannot convert ‘cudaDataType_t’ to ‘cublasComputeType_t’
  175 |   CHECK_CUBLAS_ERROR(cublasLtMatmulDescCreate(&operationDesc, CUDA_R_32I));
      |                                                               ^~~~~~~~~~
      |                                                               |
      |                                                               cudaDataType_t

which is no longer compatible with CUDA 11 cublasLtMatmulDescCreate interface, the new API has three arguments:
cublasStatus_t cublasLtMatmulDescCreate(
	cublasLtMatmulDesc_t *matmulDesc,
	cudaDataType computeType);

it breaks the old api which only has two arguments.
I'm not sure if there are other breaking changes in CUDA 11 that would fail the compilation of tvm. I think a macro like #if CUDART_VERSION >= 11000 could solve the problem mentioned above.

Thanks @yzh119 for bringing it up, would you like to take a stab? cc @Laurawly @icemelon9

closed by #6213 thanks to @lanchongyizu

repo: apache/incubator-tvm


text:
[REFACTOR][IR] Add Node suffix to low-level IR nodes

This PR adds Node suffix to all the low-level IR containers(see RFC #4648) so that we have a consistent naming convention throughout the codebase.
Ideally we would also move the Node::make function into a Reference class. However, given that the reference classes will have the same name as the original Node class, we propose to wait for
a bit until the codebase stabilizes, before adding the reference class

Node suffix is added to each low-level IR node
Variable-> VarNode

cc @merrymercy @icemelon9 @yzhliu @ZihengJiang


text:
[INLONG-7829][Sort] Add mini-batch pre aggregate by partition when ingesting data into iceberg

ingesting data into iceberg
Prepare a Pull Request
(Change the title refer to the following example)

Title Example: [INLONG-XYZ][Component] Title of the pull request

(The following XYZ should be replaced by the actual GitHub Issue number)

Fixes #7829

Motivation
Explain here the context, and why you're making that change. What is the problem you're trying to solve?
Modifications
Describe the modifications you've done.
Verifying this change
(Please pick either of the following options)

[ ] This change is a trivial rework/code cleanup without any test coverage.

[ ] This change is already covered by existing tests, such as:
(please describe tests)

[ ] This change added tests and can be verified as follows:
(example:)

Added integration tests for end-to-end deployment with large payloads (10MB)
Extended integration test for recovery after broker failure

Documentation

Does this pull request introduce a new feature? (yes / no)
If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
If a feature is not applicable for documentation, explain why?
If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation

Please add Motivation and Modifications to show what changed. Thanks.

authors: dockerzhang thexiay

repo: apache/inlong


text:
The performance diff between the Session API and JDBC 请问使用原生接口和JDBC的性能是一样的么？

请问使用原生接口和JDBC的加载性能是一样的么？
使用JDBC测试，性能没有官网文档的好啊。

对于数据写入，原生接口能比JDBC快5-10倍吧。。
目前JDBC的接口中，写入模块里SQL解析耗时太久了。

十分感谢你的回答

repo: apache/iotdb


text:
Try to fix CQ IT

add if-block before check the CQ result, if after we succeed to create the CQ, the time has already been larger than the expected execution time, it means that there will be some time range won't be queried, in such case, we just return.
We change wait time from 5s to 10s to make sure that CQ has already finished.
Add a dummy insert before actual data being inserted, just for initializing data region firstly which is time-consuming.

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (1e7395a) 49.17% compared to head (68abeff) 49.18%.
Report is 3 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff            @@
##             master   #11769   +/-   ##
=========================================
  Coverage     49.17%   49.18%
- Complexity    25323    25327    +4
=========================================
  Files          2886     2886
  Lines        178678   178684    +6
  Branches      21340    21340
=========================================
+ Hits          87865    87882   +17
+ Misses        90813    90802   -11

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


text:
[UPDATE] Update spark version: Jetty update

Getting back on this: https://github.com/apache/james-project/pull/1957

Still many error
org.apache.james.webadmin.routes.AddressMappingRoutesTest.addAddressMappingShouldSupportOneTimeEncodedAddress

Hmmm... I understand the need to move out a project that is not being maintained anymore to one of its forks, but a bit trickier here actually.
There seems to be issues with encoded slash, and we have a lot of routes that can allow that and related tests, like this one: https://ci-builds.apache.org/job/james/job/ApacheJames/job/PR-1974/3/testReport/junit/org.apache.james.webadmin.routes/AliasRoutesTest$FilteringOtherRewriteRuleTypes/putUserForAliasWithEncodedSlashShouldReturnNoContent/
It feels like it interprets the encoded slash as an actual slash...
@chibenwa the code of that fork definitely raises issues on our project. This is trickier and would require caution. Maybe potentially investigating and someone could contribute to the fork if finding a solution? It would require time though.
WDYT?

WDYT?

This was opportunistic trial but if it do not work let's forget about this.

authors: Arsnael chibenwa

repo: apache/james-project


text:
KAFKA-12263: Change MockClient.RequestMatcher to RequestAssertion

As described in KAFKA-12263, this PR refactors MockClient.RequestMatcher to become MockClient.RequestAssertion using assertions in the implementations.

@hachikuji since you raised the JIRA perhaps you could review this to check it's what you were after?

@hachikuji any chance you could take a look, since you opened KAFKA-12263?

@hachikuji any chance you could review this?

repo: apache/kafka


text:
KAFKA-15415: On producer-batch retry, skip-backoff on a new leader

When producer-batch is being retried, new-leader is known for the partition Vs the leader used in last attempt, then it is worthwhile to retry immediately to this new leader. A partition-leader is considered to be newer, if the epoch has advanced.
Code change summary -

ProducerBatch now tracks latest leader-epoch on which attempts will be made, and change in leader(epoch) is tracked through method hasLeaderChanged.
RecordAccumulator decides if batch should immediately-retry(skip-backoff), or backoff, on detecting a leader change.
All classes have UTs & e2e test in SenderTest.java

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@kirktrue

@jsancio - can you weigh in about changes to org.apache.kafka.common needing a KIP?

I have removed the changes to public classes, so this shouldn't require a KIP, take a look.

Looking through CI failures -
https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14384/8/pipeline/10/
For JDK 11 - The 2 flaky runs, and are unrelated to Java clients.
https://ge.apache.org/s/q63xej2ew4w6u
For JDK 17 - Again flaky tests, unrelated to Java client.
https://ge.apache.org/s/dayodofvlspty/tests/overview?outcome=FAILED,FLAKY

Would it make sense to backport this to 3.6 as well?

@ijuma this is merged in 3.6 now, so should be available in 3.6.1.
https://github.com/apache/kafka/pull/14522

Thanks!

authors: ijuma msn-tldr


text:
MINOR: Fix connect development guide.

Fix some trivial misses.
@ewencp

LGTM, thanks for the cleanup!

authors: ewencp uchan-nos


text:
KAFKA-16410: kafka-leader-election / LeaderElectionCommand doesn't set exit code on error

https://issues.apache.org/jira/browse/KAFKA-16410
as title, set exit code on error in LeaderElectionCommand.java
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Can you change the line#291 (
https://github.com/apache/kafka/blob/trunk/tools/src/test/java/org/apache/kafka/tools/LeaderElectionCommandTest.java#L291) to use mainNoExit instead of main? The exit could break the CI if we forgot to set exit procedure.

Oops, that was something I missed, thanks for the reminder :smiley:

cherry-pick to 3.7. see https://github.com/apache/kafka/commit/3ea2acf60f2eea31d6ebbb667b8542defb1e6332


text:
KAFKA-3941: Delay eviction listener in InMemoryKeyValueLoggedStore after restoration

Also move the initialization that restores from changelog to inner stores.

Review by @norwood @dguy @mjsax @enothereska .

@guozhangwang The refactoring to the inner stores looks good. However, I'm having a hard time understanding how the restoration process originally failed and what exactly is currently fixing it. Perhaps additional comments around that code and/or an integration test that exposes the problem would help?

@enothereska I have updated the JIRA descrption: https://issues.apache.org/jira/browse/KAFKA-3941, please let me know if it looks easier to understand now.

Thanks, looks better. LGTM, if you think it's important to add a test feel free to do so. LGTM.

i tested this locally with an inmemory store and it seems to be working 🔂

authors: enothereska guozhangwang norwood


text:
KAFKA-17124: Fix flaky DumpLogSegmentsTest#testDumpRemoteLogMetadataNonZeroStartingOffset

his changes should fix the flakiness reported for DumpLogSegmentsTest#testDumpRemoteLogMetadataNonZeroStartingOffset.
I was not able to reproduce locally, but the issue was that the second segment was not created in time:
Missing required argument "[files]"

The fix consists of getting the log path directly from the rolled segment.

We were also creating the log twice, and that was producing this warning:
[2024-07-12 00:57:28,368] WARN [LocalLog partition=kafka-832386, dir=/tmp/kafka-2956913950351159820] Trying to roll a new log segment with start offset 0 =max(provided offset = Some(0), LEO = 0) while it already exists and is active with size 0. Size of time index: 873813, size of offset index: 1310720. (kafka.log.LocalLog:70)

This is also fixed.

@showuon last patch worked fine:
https://ci-builds.apache.org/job/Kafka/job/kafka-pr/job/PR-16580/16/#showFailuresLink
Now running the pipeline again to confirm.

OK, I'll start to review once the latest CI build has no failed testDumpRemoteLogMetadataNonZeroStartingOffset. Please remember to update the PR description to reflect your change, and change to "ready" state. Thanks.

@showuon @chia7712 the CI pipeline passed two times without failures on this test, so it is now ready for review. Thanks.

authors: fvaleri showuon


text:
KAFKA-17857 Move AbstractResetIntegrationTest and subclasses to tools

related to https://issues.apache.org/jira/browse/KAFKA-17857,
as title, move AbstractResetIntegrationTest and subclasses to tools module
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@brandboat could you please rebase code?


text:
KAFKA-18263; Group lock must be acquired when reverting static member rejoin

When a static member rejoins the group, the group state is rewritten to the partition in order to persist the change. If the write fails, the change is reverted. However, this is done without acquiring the group lock.
This is only try in the old group coordinator. The new one does not suffer from this issue.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Merged to trunk, 4.0 and 3.9.


text:
MINOR: Update possible errors in OffsetFetchResponse

Note: None of the use cases for offset fetch would lead to a TOPIC_AUTHORIZATION_FAILED error (fetching offset of an unauthorized partition would return an UNKNOWN_TOPIC_OR_PARTITION error). That is why it is being removed from the PARTITION_ERRORS list.

This is a simple one @hachikuji.

authors: ijuma vahidhashemian


text:
MINOR: Allow scripts to be symlinked

Scripts now support one level of symlinking, for example from /usr/bin to your Kafka install directory.
Previously the scripts would break if you didn't run them directly from the installation directory.
I installed this on a test system and ran every script from both the installation directory and as a symlink from /usr/bin/

@ewencp @mjsax do you mind taking a look at this?

Closing this PR due to inactivity.

authors: dignan mjsax


text:
MINOR: fix typo in heartbeat request protocol

s/generationid/generationId
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

@lambdaliu Thank you!

Still missing one substitution:
10:58:17 /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk11-scala2.12@2/core/src/test/scala/unit/kafka/server/KafkaApisTest.scala:596: value setGenerationid is not a member of org.apache.kafka.common.message.HeartbeatRequestData
10:58:17 possible cause: maybe a semicolon is missing before `value setGenerationid'?
10:58:17 Error occurred in an application involving default arguments.
10:58:17         .setGenerationid(1)
10:58:17          ^

authors: abbccdda hachikuji


text:
KAFKA-9487: Follow-up PR of Kafka-9445

More detailed description of your change,
if necessary. The PR title and PR message become
the squashed commit message, so use a separate
comment to ping reviewers.
Summary of testing strategy (including rationale)
for the feature or bug fix. Unit and/or integration
tests are expected for any behaviour change and
system tests should be considered for larger changes.
Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

Left few comments..

Thanks @guozhangwang and @vinothchandar . Addressed a few comments and have replied to some left.

Hey, thanks for the update @brary .
IIUC, the main motivation for this PR was some follow-up comments from @mjsax . It would be good to get a +1 from him before merging. Maybe after the other comments are addressed.

Thanks @vvcephei for the comments. I have made the changes suggested by you as well.
@mjsax  I guess everyone else has reviewed and their comments have been incorporated. It would be great if you can give one last look and if it's okay merge it.

@vinothchandar @vvcephei I would still propose to rename the new class to StoreQueryParameters -- it's not a big deal to rename even after feature freeze IHMO.
Thoughts?

StoreQueryParameters does sound better to me. but leave it you all. :)

@mjsax  I have changed the class to StoreQueryParameters and the comment you gave in earlier PR. There were few checkstyles which were failing in ValueAndTimestampSerializerTest unrelated to this PR. I have fixed those as well.

Thanks @mjsax . I'll take care of it tomorrow.
I don't see any test results, so I'll ask Jenkins to re-run it:
Retest this, please.

Unrelated test failures:
kafka.admin.DeleteConsumerGroupsTest.testDeleteCmdAllGroups
kafka.admin.DeleteConsumerGroupsTest.testDeleteCmdWithMixOfSuccessAndError
Proceeding to merge.

Cherry-picked onto 2.5 as 3e77458cfb0f7a7586f17777f0e775d91e4744e1

authors: brary mjsax vinothchandar vvcephei


text:
MINOR: Fix typo in Utils#toPositive

Committer Checklist (excluded from commit message)

[x] Verify design and implementation
[x] Verify test coverage and CI build status
[x] Verify documentation (including upgrade notes)

Ping @guozhangwang
Original typo was added in
https://github.com/apache/kafka/commit/3a9f4b833bb7e86dc759361c33f4321ab043db05

Ping @guozhangwang
Original typo was added in
https://github.com/apache/kafka/commit/3a9f4b833bb7e86dc759361c33f4321ab043db05

@hongshaoyang Thanks for your patch!

@hongshaoyang Thanks for your patch!


text:
[REST] Using ForbiddenException instead of NotAllowedException

:mag: Description
Issue References 🔗

Seems NotAllowedException is used for method not allowed, and currently, we use false constructor, the error message we expected would not be return to client end.
It only told:
{"message":"HTTP 405 Method Not Allowed"}

Because the message we used to build the NotAllowedException was treated as allowed method, not as message.

Describe Your Solution 🔧
We should use the ForbidenException instead, and then the error message we excepted can be visible in client end.
https://github.com/apache/kyuubi/blob/85dd5a52efa790195ed46b713c53938ffd31c5d9/kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/api.scala#L47-L51
Types of changes :bookmark:

[ ] Bugfix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Test Plan 🧪
Behavior Without This Pull Request :coffin:
Behavior With This Pull Request :tada:
Related Unit Tests

Checklist 📝

[x] This patch was not authored or co-authored using Generative Tooling

Be nice. Be informative.

Codecov Report
Attention: Patch coverage is 0% with 13 lines in your changes missing coverage. Please review.

Project coverage is 0.00%. Comparing base (85dd5a5) to head (4dd6fc1).
Report is 1 commits behind head on master.

Files with missing lines
Patch %
Lines

...rg/apache/kyuubi/server/api/v1/AdminResource.scala
0.00%
12 Missing :warning:

...ache/kyuubi/server/KyuubiRestFrontendService.scala
0.00%
1 Missing :warning:

Additional details and impacted files
@@          Coverage Diff           @@
##           master   #6750   +/-   ##
======================================
  Coverage    0.00%   0.00%
======================================
  Files         684     684
  Lines       42281   42281
  Branches     5768    5768
======================================
  Misses      42281   42281

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

Thanks, merged to master(1.10.0) and branch-1.9(1.9.3).

repo: apache/kyuubi


text:
generic Create android.yml

[A clear and concise description of what the pull request is for along with a reference to the associated issue IDs, if they exist.]
Checklist

Base your changes on 2.x branch if you are targeting Log4j 2; use main otherwise
./mvnw verify succeeds (if it fails due to code formatting issues reported by Spotless, simply run ./mvnw spotless:apply and retry)
Non-trivial changes contain an entry file in the src/changelog/.2.x.x directory
Tests for the changes are provided
Commits are signed (optional, but highly recommended)

@0xETHengineer,
I really don't understand the purpose of this PR. We don't use Gradle, not have a 2.0 branch.


text:
[MINSTALL-115] installAtEnd fails with multiple class realms

Setting installAtEnd causes no installs to occur when a multimodule project has multiple class realms

Unfortunately, this technique interferes with surefire/failsafe.  The user properties must contain String properties.  I could make the pojos into json encoding.
Or, what other places can a plugin save information across invocations?  It appears that plugin context is also tied to the ClassLoader.

Is this still in progress? Should we close it?

Since implementation is difficult without changes to core maven, I've updated this pull request to be documentation only.

This PR is obsolete now, as skip/installAtEnd should be normally and "as expected" supported. No need anymore to "consistently" (I guess it was means "same"?), one can now skip some, installAtEnd some and install the rest.

authors: chonton cstamas elharo

repo: apache/maven-install-plugin


text:
METRON-1180:  Make Stellar Shell accept zookeeper quorum as a CSV list and not require a port

Contributor Comments
ZK Quora do not require a port (it defaults to 2181) and it may be a comma separated list.  Allowing the REPL to validate the port only if provided and allowing multiple hosts.
Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our Development Guidelines for the complete guide to follow for contributions.
Please refer also to our Build Verification Guidelines for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:
For all changes:

[x] Is there a JIRA ticket associated with this PR? If not one needs to be created at Metron Jira.
[x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
[x] Has your PR been rebased against the latest commit within the target branch (typically master)?

For code changes:

[x] Have you included steps to reproduce the behavior or problem that is being changed or addressed?

[x] Have you included steps or a guide to how the change may be verified and tested manually?

[x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
mvn -q clean integration-test install && build_utils/verify_licenses.sh

[x] Have you written or updated unit tests and or integration tests to verify your changes?

[x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?

[ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

For documentation related changes:

[x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via site-book/target/site/index.html:
cd site-book
mvn site

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
It is also recommended that travis-ci is set up for your personal repository such that your branches are built there before submitting a pull request.

+1 by inspection.  This is good stuff @cestella!

+1 by inspection. Nice fix

authors: cestella mmiklavc ottobackwards

repo: apache/metron


text:
hw/sensor: Report read errors

Allow "sensor error listeners" to be registered with a sensor.  When a read from a sensor fails, all error listeners that match the read attempt are notified of the error.

@ccollins476ad This is a good idea, thanks for adding it. Any change of perhaps adding a test case, just as built in documentation of how it works? The test cases are usually the first place I look personally when working with new packages or features, and since it's part of the build process they generally are kept up to date with API changes. (Obviously just a convenience request, feel free to ignore it 😉)

@microbuilder, that is a good idea.  I have added a very basic unit test for reporting of sensor read errors.

repo: apache/mynewt-core


text:
MINIFI-469 - support OSX in travis automated builds

Thank you for submitting a contribution to Apache NiFi - MiNiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
For all changes:

[ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?

[ ] Does your PR title start with MINIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.

[ ] Has your PR been rebased against the latest commit within the target branch (typically master)?

[ ] Is your initial contribution a single, squashed commit?

For code changes:

[ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi-minifi folder?
[ ] Have you written or updated unit tests to verify your changes?
[ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
[ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under minifi-assembly?
[ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under minifi-assembly?

For documentation related changes:

[ ] Have you ensured that format looks appropriate for the output in which it is rendered?

Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.

hey @dtrodrigues,
thanks for taking the time to add this in.  those changes look good and travis appears happy with the build.  I'll get this merged.

authors: apiri dtrodrigues

repo: apache/nifi-minifi


text:
MINIFI-142 Upgrading NiFi dependencies to 1.1.0

MINIFI-142 Upgrading NiFi dependencies to 1.1.0 and adjusting NAR and ClassLoader code to reflect changes introduced in NiFi.

Hey @apiri I noticed changes to the License and Notice files were not added (I think we missed them for the 1.0.0 upgrade too). I went through and updated all the L&N files to be in line with for the 1.1.0 deps and create a PR on your branch here: https://github.com/apiri/nifi-minifi/pull/2
Let me know what you think!

Good catch. Will scope those out.  Thanks much!

@apiri It seems you left out NIFI-2818 (minimizing FS permissions needed) and NIFI-2909 (the class loader isolation). Is there a reason for that?
In the interest of keeping parity with NiFi I would prefer to have them included.

Thanks for pointing out, will include those.

okay, merged the relevant changes from the following:
NIFI-2909:

apache/nifi@d1d053725b72d91fbfca1a2e86691e9c9a1a3f2f
apache/nifi@2f9ec03242546dcae694ac0effa52d9185850af7

NIFI-2818:

apache/nifi@b9ef0fb847cd2678a62964698f17a5afb37684b9

Merged the changes to L&N from apiri/nifi-minifi#2.

+1
Visually verified code and did a contrib check build. Verified the resulting assembly by running a variety of different configs. Thanks @apiri I will merge it in.

authors: JPercivall apiri


text:
NIFI-7888 Add support for SAML authentication

This PR adds the ability to authenticate to NiFi via a SAML identity provider, similar to how OIDC authentication works.
In addition, there is an option to obtain group membership info from the assertions in a successful SAML AuthN response from the IDP. These groups are then passed along in the NiFiUser instance in order to be leveraged during authorization. Currently if using the file-based policy provider, then these groups would also have to exist in the configured user-group-provider in order to have created policies against them.
The integration with spring-security-saml is heavily based on the primary example application here:
https://github.com/vdenotaris/spring-boot-security-saml-sample
I've primarily tested against KeyCloak and the SSOCirlce IDP which is used by the example app above (https://www.ssocircle.com/en/).
High-level changes:

Add dependency on spring-security-saml2-core
Updated AccessResource with new SAML end-points
Updated Login/Logout filters to handle SAML scenario
Updated logout process to track a logout request using a cookie
Added database storage for cached SAML credential and user groups
Updated proxied requests when clustered to send IDP groups in a header
Updated X509 filter to process the IDP groups from the header if present
Updated admin guide
Fixed logout action on error page
Updated StandardManagedAuthorizer to combine groups from request with groups from lookup
Updated UserGroupProvider implementations with more efficient impl of getGroupByName
Added/updated unit tests

Thanks for the PR @bbende! Will review...

I've also been reviewing this. I was able to set up Google Suite as a SAML IDP and log in. I am still testing a few things such as signing validation.

@mcgilman @thenatog added some additional commits to address some of the review feedback and improve a few things I ran into while testing, here is a summary of the changes...

Refactored some of the DB operations to have "replace" methods instead of calling "delete" and "create" in separate transactions where one could succeed and the second could fail

Added properties for configuring the values of AuthnRequestsSigned and WantAssertionsSigned for the service provider metadata that is generated for nifi at /nifi-api/access/saml/metadata
nifi.security.user.saml.request.signing.enabled=false
nifi.security.user.saml.want.assertions.signed=true

Remove the property for specifying the signing key alias, it now inspects the keystore and finds the private key entry and gets the alias automatically, if more than one private key entry exists then an exception is thrown (nifi already assumes a single private key in the keystore)

Added a property for specifying an attribute to obtain the user identity from, if an attribute is not specified or if the attribute is not found in the response, then the Subject NameID is used by default
nifi.security.user.saml.identity.attribute.name=

@bbende Thanks again for the PR! My earlier concerns have been addressed and things look good. Was able to verify the functionality. I am a +1 once the comments from @thenatog are addressed.

Thanks @mcgilman  !
@thenatog I pushed up another commit that makes SHA-256 the default when no value is provided in nifi.properties for the signature or digest algorithms. Let me know if there is anything else needed to get this merged, thanks!

authors: bbende mcgilman thenatog

repo: apache/nifi


text:
Addressing issue prevent access to login page

NIFI-2271:

Fixing issue that was preventing an anonymous user from accessing the log in page.

Closing this PR for now to revert some proposed changes. Some items addressed here was unnecessary.


text:
NIFI-12372 MiNiFi C2 Encrypt Flow Configuration Properties when Transferring

Summary
NIFI-12372
Tracking
Please complete the following tracking steps prior to pull request creation.
Issue Tracking

[x] Apache NiFi Jira issue created

Pull Request Tracking

[x] Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
[x] Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

[x] Pull Request based on current revision of the main branch
[x] Pull Request refers to a feature branch with one commit containing changes

Verification
Please indicate the verification steps performed prior to pull request creation.
Build

[x] Build completed using mvn clean install -P contrib-check

[x] JDK 21

Licensing

[ ] New dependencies are compatible with the Apache License 2.0 according to the License Policy
[ ] New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

[ ] Documentation formatting appears as expected in rendered files

A flow incoming via C2 may contain unencrypted sensitive properties as the agents can have there our encryption key.
The goal is to check the properties in the incoming flow, and if there are unencrypted sensitive properties there, use the configured encryption key to encrypt them before persisting the flow.json.raw to disk.
The logic first tries to utilise the property descriptors sent with the flow to detect the sensitive property names. However the property descriptors can omitted. In this case the logic uses the runtime manifest to get the list of sensitive properties.

Thanks for the feature, this really useful. I have only one question.

Thanks your all the reviews. Addressed the comments, and pushed the most recent changes. Please re-review.

I addressed the second round of comments. Decided to make a bigger refactor: moved services to minifi-commons, created interfaces for those services to become consistent. Also removed the flow serialization and deserialization from those particular services. Performance was not an issue there as that code was not on a critical path, eg flow updates are expected maximum a few times a day, but most frequently we talk about weeks or months here. However having the serde in the services hurt SRP at some level, so moved the logic to an upper layer. Also now the flow serde logic is encapsulated in one class.
Rebased onto the latest main. Tested a flow with C2, unecrypted properties were encrypted correctly.
Please review the latest changes.


text:
Prevent cycle sending

Stop sending activation message to QueueManager itself again but recover MemoryQueue instead when cycle happens
Description

Related issue and scope

[x] I opened an issue to propose and discuss this change (#5250)

My changes affect the following components

[ ] API
[ ] Controller
[ ] Message Bus (e.g., Kafka)
[ ] Loadbalancer
[X] Scheduler
[ ] Invoker
[ ] Intrinsic actions (e.g., sequences, conductors)
[ ] Data stores (e.g., CouchDB)
[ ] Tests
[ ] Deployment
[ ] CLI
[ ] General tooling
[ ] Documentation

Types of changes

[X] Bug fix (generally a non-breaking change which closes an issue).
[ ] Enhancement or new feature (adds new functionality).
[ ] Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

[X] I signed an Apache CLA.
[X] I reviewed the style guides and followed the recommendations (Travis CI will check :).
[X] I added tests to cover my changes.
[ ] My changes require further changes to the documentation.
[ ] I updated the documentation where necessary.

I have an issue where the namespaceContainer metric still emits containers for a namespace even after no activations are being run in a long time for that namespace. The value just remains constant forever until I restart the scheduler in which it then correctly goes to 0 not emit for that namespace. My thought was something getting stuck in memory with the memory queue even after it should have been shut down since the metric is reported from that actor. And it's weird that it would still report that there are containers even if there are none in etcd for the namespace even if the memory queue wasn't properly shut down as I assume it would get updated with the correct value when still emitting unless it's stuck in a zombie state or something. Do you think this could be the same issue?

Codecov Report

Merging #5251 (b5f7aaf) into master (1a6c99d) will decrease coverage by 7.44%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master    #5251      +/-   ##
==========================================
- Coverage   70.59%   63.14%   -7.45%
==========================================
  Files         238      238
  Lines       14009    14047      +38
  Branches      567      569       +2
==========================================
- Hits         9889     8870    -1019
- Misses       4120     5177    +1057

Impacted Files
Coverage Δ

...in/scala/org/apache/openwhisk/common/Logging.scala
68.58% <0.00%> (-10.09%)
:arrow_down:

.../openwhisk/core/scheduler/queue/QueueManager.scala
7.05% <0.00%> (-0.95%)
:arrow_down:

...che/openwhisk/core/invoker/LogStoreCollector.scala
0.00% <0.00%> (-100.00%)
:arrow_down:

...core/database/cosmosdb/RxObservableImplicits.scala
0.00% <0.00%> (-100.00%)
:arrow_down:

...ore/database/cosmosdb/cache/CacheInvalidator.scala
0.00% <0.00%> (-100.00%)
:arrow_down:

...e/database/cosmosdb/cache/ChangeFeedConsumer.scala
0.00% <0.00%> (-100.00%)
:arrow_down:

...core/database/cosmosdb/CosmosDBArtifactStore.scala
0.00% <0.00%> (-95.85%)
:arrow_down:

...sk/core/database/cosmosdb/CosmosDBViewMapper.scala
0.00% <0.00%> (-93.90%)
:arrow_down:

...tabase/cosmosdb/cache/CacheInvalidatorConfig.scala
0.00% <0.00%> (-92.31%)
:arrow_down:

...a/org/apache/openwhisk/http/BasicHttpService.scala
3.17% <0.00%> (-87.31%)
:arrow_down:

... and 42 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1a6c99d...b5f7aaf. Read the comment docs.

LGTM

authors: bdoyle0182 codecov-commenter jiangpengcheng ningyougang

repo: apache/openwhisk


text:
HDDS-4947. Reuse compiled binaries in combined coverage calculation

What changes were proposed in this pull request?
Reuse binaries created by compile check instead of running a full build for coverage.  This saves ~12 minutes for non-PR builds (push, schedule).
https://issues.apache.org/jira/browse/HDDS-4947
How was this patch tested?
Regular CI after temporarily enabled coverage check in forks.
https://github.com/adoroszlai/hadoop-ozone/actions/runs/638883332
Verified that combined coverage is around the same as on master:

download coverage artifacts for both builds
unzip and open all/index.html

:rocket:

Thanks @elek for reviewing and merging this.

authors: adoroszlai elek

repo: apache/ozone


text:
HDDS-8281. Add IntelliJ code style scheme import recommendation in CONTRIBUTING.md

What changes were proposed in this pull request?
We have included an IntelliJ code style export since HDDS-3674 (thanks @captainzmc).
Prompted by https://github.com/apache/ozone/pull/4376/files#r1145395241, we should include a few lines of instructions in the markdown doc as well.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8281
How was this patch tested?
n/a

Thanks @smengcl for documenting this.

authors: adoroszlai smengcl


text:
HDDS-10932. Reduce number of watch requests by using CommitInfoProto from NotReplicatedException

What changes were proposed in this pull request?
We can use CommitInfoProto in NotReplicatedException introduced in RATIS-2089 to remove the need for watch MAJORITY_COMMITTED calls if NotReplicatedException is thrown from the DN Ratis leader.
This also requires DN Ratis server watch timeout configuration change hdds.ratis.raft.server.watch.timeout to be lower than the client watch timeout hdds.ratis.raft.client.rpc.watch.request.timeout so that NotReplicatedException will be thrown instead of TimeoutException.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10932
How was this patch tested?
Existing tests. Increased the client watch RPC timeout in TestBlockOutputStream to be strictly larger than the server watch timeout to reflect the current default configurations.
Clean CI: https://github.com/ivandika3/ozone/actions/runs/9707025051

Thanks @ivandika3 for the patch, @szetszwo for the review.

Thank you @szetszwo for the review and @adoroszlai for the merge.

Thanks @ivandika3 for the patch!

authors: adoroszlai ivandika3 smengcl


text:
Avoid division by 0 in shading function interpolation

I am seeing a case of division by 0f (producing a NaN) when trying to evaluate a gradient with a specific type 3 color function.
The gradient renders correctly with PDFbox, because the error affects only one point of the gradient (at x = 1). But when I call shading.eval(new float[] {1.0}) I get the wrong color (black) instead of a shade of blue. I did a small investigation and found this:
I believe pdfbox isn't complying with the following sentence in the spec for functionType 3 (see section 3.9 - page 175 of the PDF reference version 1.7). Please see the reference for context, as this sentence alone is rather cryptic

If the last bound, Boundsk−2, is equal to Domain1, then x′ is defined to
be Encode2i.

Here is a simple PDF that reproduces the error: gradient-bug.pdf
In that example, the type3 color function has 3 sub functions and the following parameters:
Bounds = [0.00188385, 1]
Domain = [0, 1]
Encode = [0, 1, 0, 1, 0, 1]
When trying to evaluate the interpolate function at x = 1.0 with the following parameters, we get a NaN

Let me know if I can help further

Thanks
https://issues.apache.org/jira/browse/PDFBOX-5593
(Will test and commit later)

I found a file where the change does make a difference in rendering
PDFJS-3096.pdf
(the "back" key now appears properly)

Thank you!

authors: Etienne-Gautier THausherr

repo: apache/pdfbox


text:
Cannot build the nodejs client following the Readme

System: Ubuntu-18.04
When I followed the readme to build the client, it threw following errors at the npm run build step:
make: Entering directory '/workspaces/pulsar-client-node/build'
  SOLINK_MODULE(target) Release/obj.target/Pulsar.node
g++: error: ../pkg/lib/libpulsarwithdeps.a: No such file or directory
make: *** [Pulsar.target.mk:157: Release/obj.target/Pulsar.node] Error 1
make: Leaving directory '/workspaces/pulsar-client-node/build'
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/opt/nodejs/17.6.0/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (node:events:527:28)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:291:12)
gyp ERR! System Linux 5.4.0-1090-azure
gyp ERR! command "/opt/nodejs/17.6.0/bin/node" "/opt/nodejs/17.6.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "build" "--module=/workspaces/pulsar-client-node/lib/binding/Pulsar.node" "--module_name=Pulsar" "--module_path=/workspaces/pulsar-client-node/lib/binding" "--napi_version=8" "--node_abi_napi=napi" "--napi_build_version=0" "--node_napi_label=node-v102"
gyp ERR! cwd /workspaces/pulsar-client-node
gyp ERR! node -v v17.6.0
gyp ERR! node-gyp -v v8.4.1
gyp ERR! not ok
node-pre-gyp ERR! build error
node-pre-gyp ERR! stack Error: Failed to execute '/opt/nodejs/17.6.0/bin/node /opt/nodejs/17.6.0/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js build --module=/workspaces/pulsar-client-node/lib/binding/Pulsar.node --module_name=Pulsar --module_path=/workspaces/pulsar-client-node/lib/binding --napi_version=8 --node_abi_napi=napi --napi_build_version=0 --node_napi_label=node-v102' (1)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (/workspaces/pulsar-client-node/node_modules/@mapbox/node-pre-gyp/lib/util/compile.js:89:23)
node-pre-gyp ERR! stack     at ChildProcess.emit (node:events:527:28)
node-pre-gyp ERR! stack     at maybeClose (node:internal/child_process:1090:16)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)
node-pre-gyp ERR! System Linux 5.4.0-1090-azure
node-pre-gyp ERR! command "/opt/nodejs/17.6.0/bin/node" "/workspaces/pulsar-client-node/node_modules/.bin/node-pre-gyp" "build"
node-pre-gyp ERR! cwd /workspaces/pulsar-client-node
node-pre-gyp ERR! node -v v17.6.0
node-pre-gyp ERR! node-pre-gyp -v v1.0.9
node-pre-gyp ERR! not ok

Cannot find the libpulsarwithdeps.a. I found that there is a script to build the libpulsarwithdeps.a on the macos system but not for the linux system. I think we may need to update the Readme.
/cc @shibd

For Mac: You need first execute pkg/mac/build-cpp-deps-lib.sh and pkg/mac/build-cpp-lib.sh.
For Linux: You need first execute build-support/install-cpp-client.sh.
I'll update the README later.

Update README: https://github.com/apache/pulsar-client-node/pull/239

authors: RobertIndie shibd

repo: apache/pulsar-client-node


text:
Documentation: Suggest adding documentation for supported bundle split algorithms

Is your enhancement request related to a problem? Please describe.
Add documentation for how to choose between load balancing algorithms
Describe the solution you'd like

supportedNamespaceBundleSplitAlgorithms supports two algorithms for namespace bundle split range_equally_divide,topic_count_equally_divide
Suggest adding some documentation around that to here https://pulsar.apache.org/docs/en/administration-load-balance/ including (1) documentation how each algorithm works and how each algorithm can be adjusted (any parameters to control operation) (2) documentation how and when to choose between the default range_equally_divide and topic_count_equally_divide

CC: @Jennifer88huang

@frankjkelly thanks for raising this up. Would you like to add docs accordingly? Then we can help review, thanks

@Anonymitaet I would love to but I really do not have the expertise to understand how these mechanisms work at a detailed level and to provide some guidance on how to choose between them.

@frankjkelly we have documented the usage in broker.conf as below:
# Supported algorithms name for namespace bundle split.
# "range_equally_divide" divides the bundle into two parts with the same hash range size.
# "topic_count_equally_divide" divides the bundle into two parts with the same topics count.
supportedNamespaceBundleSplitAlgorithms=range_equally_divide,topic_count_equally_divide

Does this work for you?

I have a few questions:

How does each algorithm work with other split bundle configurations?

Is each algorithm limited by other split bundle configurations?

Thanks @Anonymitaet exactly - it gets pretty interesting to understand how all those parameters will interact.
I'm particularly interested in how each algorithm behaves when we add brokers (we're hoping to implement Kubernetes autoscaling)- how quickly would those brokers be discovered and how  quickly some load would be redirected to those new brokers.
In my limited experience with range_equally_divide it's not until we hit the loadBalancerBrokerOverloadedThresholdPercentage and the broker is overloaded that some namespace bundles will be split and offloaded but at that point usually the load on the broker is pretty high so that the offload seems to be "expensive" or at least take a long time and then our Pulsar Clients start timing out.
For us we'd rather have a way that was a faster balancing (but perhaps less precisely load balanced) than one that was more accurately load balanced but takes longer to shed load. Again we're pretty new to this but would appreciate to hear perspectives or pointers to code. Thanks!

I have a few questions:

How does each algorithm work with other split bundle configurations?

Is each algorithm limited by other split bundle configurations?

@Anonymitaet each algorithm is affected by these configurations the same. range_equally_divide use hash range, topic_count_equally_divide is evenly distributed.

Thanks @codelipenghui this is super helpful!

authors: Anonymitaet congbobo184 frankjkelly

repo: apache/pulsar


text:
"Clusters can not be empty" error on tenant creation when authentication is enabled

Describe the bug
I’m trying to set-up authentication via JWT tokens in a standalone pulsar container, but I keep getting “HTTP 412 Precondition Failed Clusters can not be empty” error when try to create a tenant via pulsar-admin command, although it works correctly with disabled authentication. According to pulsar logs it “Successfully authorized admin1 as super-user" but then “[admin1] Failed to validate due to clusters are empty”. I do have a default “clusterName=standalone” in standalone.conf but it does not help.
To Reproduce
Steps to reproduce the behavior:

I use apachepulsar/pulsar:2.7.2 docker container with "bin/pulsar standalone -nfw -nss" command (function worker is disabled to avoid auth issues).
I followed instructions on https://pulsar.apache.org/docs/en/security-jwt documentation page to enable authentication via symmetric secret key:
2.1. I created a symmetric secret key to /pulsar/conf/my-secret.key.
2.2. I generated a JWT token for admin1 role.
2.3. client.conf channges:
authPlugin=org.apache.pulsar.client.impl.auth.AuthenticationToken
authParams=token:eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJhZG1pbjEifQ.M5THXy95jofensE7egEqwRtUGyQ1f6HtpU_si4cH_7c
2.4 standalone.conf changes:
authenticationEnabled=true
authenticationProviders=org.apache.pulsar.broker.authentication.AuthenticationProviderToken
authorizationEnabled=true
tokenSecretKey=file:///pulsar/conf/my-secret.key
superUserRoles=admin1
Container successfully starts.
Execute "bin/pulsar-admin tenants create trade_routing_sf"
Get error "Failed to validate due to clusters are empty”.

Expected behavior
Tenant is successfully created.
Desktop (please complete the following information):

OS: apachepulsar/pulsar:2.7.2 docker container

I found an example of working pulsar standalone docker container with JWT authentication enabled, but the problem is that, it has "anonymousUserRole=admin" in standalone.conf, which effectively means - no authentication.
After replacing it on "anonymousUserRole=anonymous" I got errors like "org.apache.pulsar.client.admin.PulsarAdminException$NotAuthorizedException" from function worker. After disabling it by " -nfw -nss" I again got "Clusters can not be empty" error.

I follow your operations in pulsar 2.8.1, doesn't appear this problem. Maybe you can try higher version ?

@Shoothzj I switched on 2.8.1 and reproduced the same issue.
Here is pulsar-auth-container.tar.gz archive with a small script (run_pulsar_container.sh) that reproduces the issue along with Dockerfile and changed config files.

@Shoothzj I switched on 2.8.1 and reproduced the same issue.
Here is pulsar-auth-container.tar.gz archive with a small script (run_pulsar_container.sh) that reproduces the issue along with Dockerfile and changed config files.

did you missed to config the brokerClientAuthenticationPlugin and brokerClientAuthenticationParameters  in standalone.conf
It should be config like
brokerClientAuthenticationPlugin=org.apache.pulsar.client.impl.auth.AuthenticationToken
brokerClientAuthenticationParameters={"token":"eyJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ0ZXN0LXVzZXIifQ.9OHgE9ZUDeBTZs7nSMEFIuGNEX18FLR3qvy8mqxSxXw"}

@Shoothzj, thanks a lot! I tried these parameters but for some reason they it did not work on 2.7.x but it does work on 2.8.0 and 2.8.1.
So, this means that JWT token authentication does not work on 2.7.x (at least in standalone mode).

authors: Shoothzj VadimMolodyh


text:
pulsar broker feature request

Pulsar cluster has three brokers: broker 1:6650, broker 2:6650 and broker 3:6650
Now I replace the original broker with three new broker nodes. The new brokers are broker 4:6650, broker 5:6650 and broker 6:6650
My server config the new broker (4 ~ 6),  but when I started server, I found that the old broker (1 ~ 3)  was connected either. Why? Should the broker be connected with what I configed ?

broker.conf  advertisedAddress whether there is a set value？

broker.conf advertisedAddress whether there is a set value？

new broke node config right value to advertisedAddress in broker.conf.

Closed as stale. Please open a new issue if it's still relevant in maintained versions.
This seems not a feature request. If you'd like to ask a question, you can go to https://github.com/apache/pulsar/discussions/categories/q-a.

authors: hepyu liudezhi2098 tisonkun


text:
Flaky-test: ZKSessionTest.testReacquireLeadershipAfterSessionLost

Search before asking

[ ] I searched in the issues and found nothing similar.

Example failure
ses below
Exception stacktrace
see below
Are you willing to submit a PR?

[ ] I'm willing to submit a PR!

see https://github.com/apache/pulsar/pull/18578/files

also this test fail in local environment very often.

also https://github.com/apache/pulsar/pull/18518/files

i dont know how to find the relate CI link.

another one  https://github.com/apache/pulsar/pull/18672/files

This looks like a possible return of #11032

recent failures:
java.util.concurrent.CompletionException: org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
	at java.base/java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:781)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$4(ZKMetadataStore.java:201)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$7(ZKMetadataStore.java:201)
	at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient$3$1.processResult(PulsarZooKeeperClient.java:489)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:714)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:553)
Caused by: org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$7(ZKMetadataStore.java:200)

Usage tip: To enable automatic navigation to failure message, open the following links with CTRL/CMD-click.
example failure 2023-11-01T16:28:13.4988707Z
example failure 2023-10-29T09:21:00.2103841Z

Full exception stacktrace

java.util.concurrent.CompletionException: org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:332)
	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:347)
	at java.base/java.util.concurrent.CompletableFuture$UniRun.tryFire(CompletableFuture.java:781)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$4(ZKMetadataStore.java:201)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$7(ZKMetadataStore.java:201)
	at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient$3$1.processResult(PulsarZooKeeperClient.java:489)
	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:714)
	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:553)
Caused by: org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$7(ZKMetadataStore.java:200)
	... 3 more
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:134)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)
	... 4 more

close for leave project any one can reopen this issue again.


text:
brokerWebServiceURL should not be required in proxy conf

It's a perfectly valid usecase to only want the proxy to be used for produce and consume traffic, and to only allow admin requests from inside the pulsar cluster. However, currently if URL based configuration is being used (i.e. zk is not configured), then brokerWebServiceURL is required.
Update docs(http://pulsar.apache.org/docs/en/administration-proxy/) when this is fixed to note that not all URLs need to be configured.

Closed as stale and no one seems to work on it. Please create a new issue if it's still relevant to the maintained versions.

authors: ivankelly tisonkun


text:
add protobuf/duration support for pulsar-sql

Motivation
add protobuf/duration support for pulsar-sql
Modifications
add protobuf/duration support in PulsarProtobufNativeRowDecoderFactory
return millis parsed from protobuf/duration  in PulsarProtobufNativeColumnDecoder
Does this pull request potentially affect one of the following parts:
If yes was chosen, please highlight the changes

Dependencies (does it add or upgrade a dependency): (yes)
The public API: (yes)
The schema: (no)
The default values of configurations: (no)
The wire protocol: (no)
The rest endpoints: (no)
The admin cli options: (no)
Anything that affects deployment: (no)

Documentation
Check the box below or label this PR directly (if you have committer privilege).
Need to update docs?

[ ] doc-required
(If you need help on updating docs, create a doc issue)

[x] no-need-doc
(Please explain why)

[ ] doc
(If this PR contains doc changes)

This pr will be closed