text:
TensorFlow ~80% slower than PyTorch on training Wide ResNet

cc @tfboyd
From https://github.com/tensorflow/tensorflow/issues/7187#issuecomment-295502315
On an AWS p2.xlarge, using the tensorflow/tensorflow:1.0.1-devel-gpu Docker image as a base, I see ~270 ms per epoch while training a WRN-16-4 without dropout on CIFAR-10.
Using a PyTorch implementation from https://github.com/xternalz/WideResNet-pytorch, I see instead ~150 ms per epoch for the same.
My implementation of Wide ResNets uses NCHW and fused batch norm. It does use feed_dict for data loading, but I've observed with nvidia-smi that my GPU utilization stays near 100%.

To reproduce:

Clone https://github.com/4Catalyzer/dl-papers, and go to that directory.
Check out the benchmark branch.
Build the Docker image, which is based on the Docker hub image above:

$ docker build -t dl-papers .

Run the Docker image using NVIDIA Docker:

$ nvidia-docker run --rm -it dl-papers /bin/bash

Run the TF WRN-16-4 training:

# python -m dl_papers.wide_resnet.train cifar10

Observe the logged batch timings, then kill the process.
In the same Docker container up the PyTorch Wide ResNet example:

# cd ..
# pip install http://download.pytorch.org/whl/cu80/torch-0.1.11.post5-cp27-none-linux_x86_64.whl
# pip install torchvision tensorboard_logger
# git clone https://github.com/xternalz/WideResNet-pytorch.git
# cd WideResNet-pytorch

Run PyTorch training:

# python train.py --dataset cifar10 --layers 16 --widen-factor 4 -p 1

Observe logged batch timings.

To rule out issues with my use of feed_dict, I've added a benchmark-constant branch to my repo that uses constant tensors for the batch inputs and outputs, and removes all the actual data loading logic.
On this branch, on the same AWS instance, I see batch times of ~246 ms, which is still over 60% slower than what I see with the PyTorch example.

I doubt I can be helpful on this one.  I am having a hard time following all of the code references, e.g. wrappers around everything.  I am not doubting it is correct but about about 30 minutes of scanning through the code I know it is beyond me to follow. This is a shot in the dark but if you are trying things to see if they impact the time, I would remove feed_dic completely.  You can decrease/decay the learning rate (even on a set schedule) via the optimizer combined with global_step, which is the way I normally see it done.  I am not saying that will fix your issue, that is just the only thing I see that sticks out to me.  Sorry.  Maybe someone else will take a look.  I know we are working on a wide&deep model to open source but it may not be your exact one.  I will try to check on that, it might also give me an idea of what might be causing you problems.  Thank you for sharing.

Would it help if I inlined the non-TF layer helpers into a single module? As you can probably tell, we pulled this out from a larger internal repo, where things were organized to optimize code reuse.
If I take out feed_dict entirely, I get ~241 ms. I suspect this is largely from eliding the conditional checks on training, though.

Yeah that did not save much.  :-(    I understand this may sound like I am deflecting the question but how do you know your model is exactly the same as the pytorch version?
Removing the wrappers and inlining seems painful, I would hate to have you do it and then not get time to dig in.  This might be a fun side project for me to look at in regards to checking out the timeline.  I am also asking internally about a wide&deep model as I would like to start from something where I have more experts to poke.  I will try to find time to ask around.  I sent an email but I suspect I will need to ask a few people.

I believe they're the same. The two models have the same number of parameters, and I can't see anything different on inspection.
Also, the performance delta I see here is broadly consistent with those reported for ResNet-50 and ResNet-56 in the paper linked in the OP on https://github.com/tensorflow/tensorflow/issues/7187.

One thing to keep in mind and it is something common in most of the code from the HK benchmark.   1) put your input pipeline on the CPU (I fixed this for another simpler CIFAR example and the person got more than a 5x improvement from ~700 imgs/sec to ~3800 imgs/sec on a K80.
https://github.com/tensorflow/models/issues/1264

Queues and following best practices for them.  Feed_dict is just for messing around it is very slow.  Next week we should release code what is even faster than queues but queues can still get a lot of performance.

I know some of the sample code and published model code did not follow those best practices.

Since it's slow even with a constant fake input and 100% GPU util, it's probably not a problem with feed_dict or input, right? Also I'm able to reproduce the 60% performance difference with my own TF code written fully independently from the above TF code.
Slow speed at high GPU util may be a result of using inefficient kernels. For example by setting cudnn.benchmark=False (disable autotuner) I can make the above pytorch code 2x slower, while still taking 100% GPU utilization -- maybe that's something to look at.

No idea, someone else will have to pick this up.  I know very little and even less about this model.  I am sure it will come back around when we focus on these types of models (and thus the ops and kernels they are using) in a couple weeks.

To clarify:

Even when using feed_dict for the data, I'm seeing 95%-100% GPU utilization, so I'm fairly confident that I'm not starving my GPU, even when using feed_dict (data augmentation/&c. are still happening in a separate thread)
When I remove all placeholders and just use constant fake training data (i.e. to remove all GPU memory or whatever pressure from moving data around), I still see TF as ~60% slower than the PyTorch model that's doing actual fitting
I intentionally wrote this Wide ResNet model from scratch; the ResNet models in tensorflow/models have a number of problems, from inefficiency for the non-TF-Slim versions (using NHWC and not using fused batch norm) to actually implementing the model incorrectly in the TF-Slim versions (not dropping biases on convolutions before batch norms and not using beta and gamma on the batch norm operations, plus the same inefficiencies as above)

I know this version also has issues but at a glance if I comment this in and the other filter out.
https://github.com/tensorflow/models/blob/master/resnet/resnet_model.py#L87
Yes, I can ask someone on the team, but asking you might save me time and get this moving.
Will that give me the same model in general we are discussing.  It is much easier me to "package" this up and get people to look at it than code with a lot of extra "wrappers".  I can then get it cleaned up so the community can have a nice version to use.  I am pretty sure they team is working on a version of this to go with Estimators and it would be great to flush out any issues sooner rather than later.
I hope I was clear.  I am not super experienced with the models.  I know a few things and try to help with simple problems.  I saw feed_dict and wanted to make sure you did not have a simple problem.  The danger is trying to help (I hope to stop in the future) is I end up in a situation where I cannot help directly and people get unfriendly.
Thank you for helping me try to help you.

Here are the results on a GTX 1080 (which I also use as my main display card):
Not apples-to-apples due to TF using a constant.  I used git checkout benchmark-constant with the following commands on Ubuntu 14.04 (a custom internal google build) with CUDA 8.0 and 5.1 cuDNN as one would expect:

python -m dl_papers.wide_resnet.train cifar10
python train.py --dataset cifar10 --layers 16 --widen-factor 4 -p 1

.069 vs .055.  I am running TF 1.1rc2 compiled with CUDA and compute 6.1 as well as avx (which I doubt matters in this instance).
TensorFlow with the constant
[2017-04-20 00:15:48,235] INFO in train: epoch 0, batch 176: 0.068s
[2017-04-20 00:15:48,303] INFO in train: epoch 0, batch 177: 0.068s
[2017-04-20 00:15:48,372] INFO in train: epoch 0, batch 178: 0.068s
[2017-04-20 00:15:48,440] INFO in train: epoch 0, batch 179: 0.068s
[2017-04-20 00:15:48,510] INFO in train: epoch 0, batch 180: 0.069s
[2017-04-20 00:15:48,581] INFO in train: epoch 0, batch 181: 0.071s
[2017-04-20 00:15:48,649] INFO in train: epoch 0, batch 182: 0.068s
PyTorch with real data
Epoch: [0][83/391]	Time 0.055 (0.069)	Loss 1.5680 (1.8102)	Prec@1 42.188 (31.343)
Epoch: [0][84/391]	Time 0.054 (0.069)	Loss 1.6633 (1.8084)	Prec@1 39.062 (31.434)
Epoch: [0][85/391]	Time 0.053 (0.069)	Loss 1.6345 (1.8064)	Prec@1 39.844 (31.532)
Epoch: [0][86/391]	Time 0.056 (0.069)	Loss 1.4338 (1.8021)	Prec@1 46.875 (31.708)
Epoch: [0][87/391]	Time 0.056 (0.069)	Loss 1.4923 (1.7986)	Prec@1 43.750 (31.845)
Epoch: [0][88/391]	Time 0.057 (0.068)	Loss 1.6836 (1.7973)	Prec@1 45.312 (31.996)
Toby

If I turn on WINOGRAD I get another small bump still using the same branch.
export TF_ENABLE_WINOGRAD_NONFUSED=1;python -m dl_papers.wide_resnet.train cifar10
[2017-04-20 00:37:44,189] INFO in train: epoch 0, batch 192: 0.062s
[2017-04-20 00:37:44,251] INFO in train: epoch 0, batch 193: 0.062s
[2017-04-20 00:37:44,313] INFO in train: epoch 0, batch 194: 0.062s
[2017-04-20 00:37:44,375] INFO in train: epoch 0, batch 195: 0.062s
[2017-04-20 00:37:44,438] INFO in train: epoch 0, batch 196: 0.062s
[2017-04-20 00:37:44,499] INFO in train: epoch 0, batch 197: 0.062s
[2017-04-20 00:37:44,561] INFO in train: epoch 0, batch 198: 0.062s
[2017-04-20 00:37:44,623] INFO in train: epoch 0, batch 199: 0.062s
[2017-04-20 00:37:44,688] INFO in train: epoch 0, batch 200: 0.065s
[2017-04-20 00:37:44,750] INFO in train: epoch 0, batch 201: 0.062s

On the K80 (AWS p2.xlarge).  TF 1.1rc2+ (built from head earlier today) CUDA 8.0, avx2, cuDNN 5.1 and compute 3.7.
WINOGRAD did not help much on K80.
export TF_ENABLE_WINOGRAD_NONFUSED=1;python -m dl_papers.wide_resnet.train cifar10
[2017-04-20 07:52:13,740] INFO in train: epoch 0, batch 91: 0.168s
[2017-04-20 07:52:13,909] INFO in train: epoch 0, batch 92: 0.169s
[2017-04-20 07:52:14,079] INFO in train: epoch 0, batch 93: 0.170s
[2017-04-20 07:52:14,248] INFO in train: epoch 0, batch 94: 0.169s
[2017-04-20 07:52:14,417] INFO in train: epoch 0, batch 95: 0.169s
[2017-04-20 07:52:14,587] INFO in train: epoch 0, batch 96: 0.169s
[2017-04-20 07:52:14,756] INFO in train: epoch 0, batch 97: 0.169s
[2017-04-20 07:52:14,925] INFO in train: epoch 0, batch 98: 0.169s
[2017-04-20 07:52:15,094] INFO in train: epoch 0, batch 99: 0.169s
without WINOGRAD
[2017-04-20 07:53:38,301] INFO in train: epoch 0, batch 40: 0.169s
[2017-04-20 07:53:38,470] INFO in train: epoch 0, batch 41: 0.168s
[2017-04-20 07:53:38,639] INFO in train: epoch 0, batch 42: 0.169s
[2017-04-20 07:53:38,808] INFO in train: epoch 0, batch 43: 0.169s
[2017-04-20 07:53:38,977] INFO in train: epoch 0, batch 44: 0.169s
[2017-04-20 07:53:39,147] INFO in train: epoch 0, batch 45: 0.169s
[2017-04-20 07:53:39,316] INFO in train: epoch 0, batch 46: 0.169s
[2017-04-20 07:53:39,485] INFO in train: epoch 0, batch 47: 0.169s
[2017-04-20 07:53:39,654] INFO in train: epoch 0, batch 48: 0.169s

And now with real data on K80 WINOGRAD on and the rest the same, but WINOGRAD did very little on K80.
[2017-04-20 08:03:47,636] INFO in train: epoch 0, batch 383: 0.174s
98%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉  | 49152/50000 [01:07<00:01, 739.85ex/s][2017-04-20 08:03:47,810] INFO in train: epoch 0, batch 384: 0.173s
99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 49280/50000 [01:07<00:00, 739.13ex/s][2017-04-20 08:03:47,983] INFO in train: epoch 0, batch 385: 0.173s
99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌ | 49408/50000 [01:07<00:00, 739.33ex/s][2017-04-20 08:03:48,156] INFO in train: epoch 0, batch 386: 0.173s
99%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊ | 49536/50000 [01:07<00:00, 739.21ex/s][2017-04-20 08:03:48,330] INFO in train: epoch 0, batch 387: 0.173s
99%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏| 49664/50000 [01:07<00:00, 738.45ex/s][2017-04-20 08:03:48,504] INFO in train: epoch 0, batch 388: 0.173s
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍| 49792/50000 [01:07<00:00, 737.94ex/s][2017-04-20 08:03:48,675] INFO in train: epoch 0, batch 389: 0.172s
I like your graphic with the progress bar.  Kind of fun but not so much fun to cut and paste.

Let me know if I did something wrong.  I am going to bed.

Actually with  TF_ENABLE_WINOGRAD_NONFUSED=1, I saw 5% difference between pytorch and tensorflow. I'm on Tesla M40.

I managed to get a set of traces from pytorch and tensorflow about what convolution algorithm and shapes they use:
pytorch:
Input: 128 3 32 32 0 Weight: 16 3 3 3 0 pad: 1 1 0 Stride: 1 1 0 -> Algo: 1
Input: 128 16 32 32 0 Weight: 160 16 3 3 0 pad: 1 1 0 Stride: 1 1 0 -> Algo: 6
Input: 128 160 32 32 0 Weight: 160 160 3 3 0 pad: 1 1 0 Stride: 1 1 0 -> Algo: 6
Input: 128 16 32 32 0 Weight: 160 16 1 1 0 pad: 0 0 0 Stride: 1 1 0 -> Algo: 1
Input: 128 160 32 32 0 Weight: 320 160 3 3 0 pad: 1 1 0 Stride: 2 2 0 -> Algo: 1
Input: 128 320 16 16 0 Weight: 320 320 3 3 0 pad: 1 1 0 Stride: 1 1 0 -> Algo: 7
Input: 128 160 32 32 0 Weight: 320 160 1 1 0 pad: 0 0 0 Stride: 2 2 0 -> Algo: 1
Input: 128 320 16 16 0 Weight: 640 320 1 1 0 pad: 0 0 0 Stride: 2 2 0 -> Algo: 1
Input: 128 320 16 16 0 Weight: 640 320 3 3 0 pad: 1 1 0 Stride: 2 2 0 -> Algo: 1
Input: 128 640 8 8 0 Weight: 640 640 3 3 0 pad: 1 1 0 Stride: 1 1 0 -> Algo: 7

bd Input: 128 640 8 8 0 Weight: 640 640 3 3 0 pad: 1 1 0 Stride: 1 1 0 -> Algo: 5
bf Input: 128 640 8 8 0 Weight: 640 640 3 3 0 pad: 1 1 0 Stride: 1 1 0 -> Algo: 5

bd Input: 128 320 16 16 0 Weight: 640 320 3 3 0 pad: 1 1 0 Stride: 2 2 0 -> Algo: 0
bf Input: 128 320 16 16 0 Weight: 640 320 3 3 0 pad: 1 1 0 Stride: 2 2 0 -> Algo: 1

bd Input: 128 320 16 16 0 Weight: 640 320 1 1 0 pad: 0 0 0 Stride: 2 2 0 -> Algo: 0
bf Input: 128 320 16 16 0 Weight: 640 320 1 1 0 pad: 0 0 0 Stride: 2 2 0 -> Algo: 3

Input: 128 320 16 16 0 Weight: 320 320 3 3 0 pad: 1 1 0 Stride: 1 1 0 -> Algo: 5
Input: 128 320 16 16 0 Weight: 320 320 3 3 0 pad: 1 1 0 Stride: 1 1 0 -> Algo: 5

Input: 128 160 32 32 0 Weight: 320 160 1 1 0 pad: 0 0 0 Stride: 2 2 0 -> Algo: 0
Input: 128 160 32 32 0 Weight: 320 160 1 1 0 pad: 0 0 0 Stride: 2 2 0 -> Algo: 0

Input: 128 160 32 32 0 Weight: 320 160 3 3 0 pad: 1 1 0 Stride: 2 2 0 -> Algo: 0
Input: 128 160 32 32 0 Weight: 320 160 3 3 0 pad: 1 1 0 Stride: 2 2 0 -> Algo: 3

Input: 128 160 32 32 0 Weight: 160 160 3 3 0 pad: 1 1 0 Stride: 1 1 0 -> Algo: 4
Input: 128 160 32 32 0 Weight: 160 160 3 3 0 pad: 1 1 0 Stride: 1 1 0 -> Algo: 5

bd Input: 128 16 32 32 0 Weight: 160 16 1 1 0 pad: 0 0 0 Stride: 1 1 0 -> Algo: 1
bf Input: 128 16 32 32 0 Weight: 160 16 1 1 0 pad: 0 0 0 Stride: 1 1 0 -> Algo: 3

bd Input: 128 16 32 32 0 Weight: 160 16 3 3 0 pad: 1 1 0 Stride: 1 1 0 -> Algo: 4
bf Input: 128 16 32 32 0 Weight: 160 16 3 3 0 pad: 1 1 0 Stride: 1 1 0 -> Algo: 3

bf Input: 128 3 32 32 0 Weight: 16 3 3 3 0 pad: 1 1 0 Stride: 1 1 0 -> Algo: 0

TensorFlow:
Conv accepts: 128, 3, (32, 32), 16, (3, 3), (1, 1), (2, 2), 1, 0 -> (1, 0)
Conv accepts: 128, 16, (32, 32), 160, (3, 3), (1, 1), (2, 2), 1, 0 -> (6, 0)
Conv accepts: 128, 160, (32, 32), 160, (3, 3), (1, 1), (2, 2), 1, 0 -> (6, 0)
Conv accepts: 128, 16, (32, 32), 160, (1, 1), (1, 1), (0, 0), 1, 0 -> (1, 0)
Conv accepts: 128, 160, (33, 33), 320, (3, 3), (2, 2), (1, 1), 1, 0 -> (1, 0)
Conv accepts: 128, 320, (16, 16), 320, (3, 3), (1, 1), (2, 2), 1, 0 -> (7, 0)
Conv accepts: 128, 160, (32, 32), 320, (1, 1), (2, 2), (0, 0), 1, 0 -> (1, 0)
Conv accepts: 128, 320, (16, 16), 640, (1, 1), (2, 2), (0, 0), 1, 0 -> (1, 0)
Conv accepts: 128, 320, (17, 17), 640, (3, 3), (2, 2), (1, 1), 1, 0 -> (1, 0)
Conv accepts: 128, 640, (8, 8), 640, (3, 3), (1, 1), (2, 2), 1, 0 -> (7, 0)

ConvBwdData accepts: 128, 640, (8, 8), 640, (3, 3), (1, 1), (2, 2), 1, 0 -> (5, 0)
ConvBwdFilter accepts: 128, 640, (8, 8), 640, (3, 3), (1, 1), (2, 2), 1, 0 -> (5, 0)

ConvBwdData accepts: 128, 320, (17, 17), 640, (3, 3), (2, 2), (1, 1), 1, 0 -> (0, 0)
ConvBwdFilter accepts: 128, 320, (17, 17), 640, (3, 3), (2, 2), (1, 1), 1, 0 -> (1, 0)

ConvBwdData accepts: 128, 320, (16, 16), 640, (1, 1), (2, 2), (0, 0), 1, 0 -> (0, 0)
ConvBwdFilter accepts: 128, 320, (16, 16), 640, (1, 1), (2, 2), (0, 0), 1, 0 -> (3, 0)

ConvBwdData accepts: 128, 320, (16, 16), 320, (3, 3), (1, 1), (2, 2), 1, 0 -> (5, 0)
ConvBwdFilter accepts: 128, 320, (16, 16), 320, (3, 3), (1, 1), (2, 2), 1, 0 -> (5, 0)

ConvBwdData accepts: 128, 160, (32, 32), 320, (1, 1), (2, 2), (0, 0), 1, 0 -> (0, 0)
ConvBwdFilter accepts: 128, 160, (32, 32), 320, (1, 1), (2, 2), (0, 0), 1, 0 -> (0, 0)

ConvBwdData accepts: 128, 160, (33, 33), 320, (3, 3), (2, 2), (1, 1), 1, 0 -> (0, 0)
ConvBwdFilter accepts: 128, 160, (33, 33), 320, (3, 3), (2, 2), (1, 1), 1, 0 -> (3, 0)

ConvBwdData accepts: 128, 160, (32, 32), 160, (3, 3), (1, 1), (2, 2), 1, 0 -> (4, 0)
ConvBwdFilter accepts: 128, 160, (32, 32), 160, (3, 3), (1, 1), (2, 2), 1, 0 -> (5, 0)

ConvBwdData accepts: 128, 16, (32, 32), 160, (1, 1), (1, 1), (0, 0), 1, 0 -> (1, 0)
ConvBwdFilter accepts: 128, 16, (32, 32), 160, (1, 1), (1, 1), (0, 0), 1, 0 -> (3, 0)

ConvBwdData accepts: 128, 16, (32, 32), 160, (3, 3), (1, 1), (2, 2), 1, 0 -> (4, 0)
ConvBwdFilter accepts: 128, 16, (32, 32), 160, (3, 3), (1, 1), (2, 2), 1, 0 -> (3, 0)

ConvBwdFilter accepts: 128, 3, (32, 32), 16, (3, 3), (1, 1), (2, 2), 1, 0 -> (0, 0)
# batch size, input chan, (input shape), out chan, (kernel shape), (stride), (padding), _, _

With TF_ENABLE_WINOGRAD_NONFUSED=1, tensorflow always chooses the same algorithm as pytorch. But the log shows that tensorflow sometimes calls cudnn with irregular input shapes and paddings when stride=2.

When it comes to 5% difference if you are not doing 10 plus tests (stop and start the process) and then on multiple machines it could easily just be an anomaly.  Also don't just look at every 10, you need to average it all together correctly.  Not doubting the 5% and it could be real but I have been excited about perf improvements and they were just luck of the machine.  I have also noticed that if you get a p2.8xlarge and just use 1 GPU it is faster than the p2.xlarge.  It could be the CPU difference, it could be a lot of things.  Just interesting and not really worth a deep dive.
@ppwwyyxx is this is a bug?  I am not sure I follow your sentence.  Is TF calling the wrong thing with or without WINOGRAD?  Is it wrong and a problem?  I got confused between your use of the word always and then sometimes.  Can you let me know which version of TF and general hardware you were using when you were getting the 60%-80% difference and the actual numbers even if just estimates?

But the log shows that tensorflow sometimes calls cudnn with irregular input shapes and paddings when stride=2.

If you are doing actual research, I would increase the batch-size (PyTorch as well) unless you think it will harm accuracy.  I did not check memory usage but if you want to train as fast as possible that is sometimes overlooked.
Here are also some other flags we use in the benchmarks.  They did not seem to have an impact on K80 but I did not extensively.
config = tf.ConfigProto()
config.allow_soft_placement = True
config.intra_op_parallelism_threads = 1
config.inter_op_parallelism_threads = 0  <-- let the system figure it out
config.gpu_options.force_gpu_compatible = True  <-- requires master head from the past few days
This env variable might be of interest.
os.environ['TF_AUTOTUNE_THRESHOLD']
It defaults to 1 if I understand the underlying code so there is likely no need to set it but it might be interesting.

I posted an issue before about TF logging the conv algorithm in use (pytorch has an option for this): https://github.com/tensorflow/tensorflow/issues/8941
This probably isn't a huge issues, but one issues is that TF will have to transpose the weights on each forward / backward pass: https://github.com/tensorflow/tensorflow/issues/8287 (and https://github.com/tensorflow/tensorflow/issues/7187#issuecomment-285191193).
@Yangqing also had some comments on workspace size (TF_CUDNN_WORKSPACE_LIMIT_IN_MB?): https://github.com/tensorflow/tensorflow/issues/7187#issuecomment-290209116 that I don't fully understand.

@cancan101  Thank you Alex those are some items I can ask about and thank you for linking them in one chunk.  I would 100% build TF 1.1 (not the branch just build from master) from source.  I have not gone back to see when the algorithm picking (if that is what closed the gap) got better but my guess is toward the end of March.  That is a wild guess based on not understanding the code I am reading the the diff history.

I would also add that the following env var don't seem documented:

TF_AUTOTUNE_THRESHOLD
TF_ENABLE_WINOGRAD_NONFUSED
TF_CUDNN_WORKSPACE_LIMIT_IN_MB

@tfboyd
Sorry about that progress bar. It works a lot better in actual use when we're not logging out timings after every batch. 😛
I'm working on using a modified version of the data pipeline from the tutorial (modified to use 32x32 and NCHW) to get a more apples-to-apples comparison. Though, is feed_dict expected to be slower even if GPU utilization is in the 95%-100% range?
I'll take a look at TF 1.1 RCs and master as well. We've been holding off on that upgrade because of some TensorBoard regressions, but we weren't aware that there was a potential performance boost.

@taion   I thought your progress bars were cool, they are kind of fun when logging that often.  It was  1am when I ran it and my terminal was like a black and white explosion of moving stuff.  My first thought was that my scripts need to be much cooler.
Yeah that TensorBoard thing got me as well.  I don't want to talk about it.  :-)
Apologies again.  When I said apples-to-apples, I meant in the first post I was running TF with a constant and PyTorch with real data.  Turns out the results were similar.  I honestly do not know if the queues will help over feed_dict with the GPU at 100%.  I do know that every time I see it I assume it is a problem and it is good practice to just not use the feed_dict approach.  I hope I am being clear that I don't know in this instance.  I would do it just so I knew it was not a problem.  If I was faster with TF, I would "hack" it in and try it.
Thank you for all the feedback Jimmy, you are a nice person.  It is really hard to work through this stuff with "strangers" and trying to get on the same page.  You likely know way more than I do but I have access to some code and happen to be connected to recent changes.
@cancan101 Yup.  Once we are sure WINOGRAD is working in all instances it will just be the default.  The AUTOTUNE is not really needed but you could play with it.  TF_CUDNN_WORKSPACE_LIMIT_IN_MB is not something I have used.  I will include them in the perf guide update next week and the ones we use in the benchmark is embedded in the code.  I shared them with this group because all of you were curious and I felt knowledgeable enough (more than me in many ways) to turn them on or off if you have issues, e.g. things don't converge or things "explode".
This is great feedback.  Yes, I type too much.

The queue-based data loading from the tutorial appears to be comparable to what I had before with feed_dict.
Using the same setup as above (TF 1.0.1 from the Docker image on a p2.xlarge).
Queue (https://github.com/4Catalyzer/dl-papers/tree/benchmark-queue):
[2017-04-20 15:16:30,871] INFO in train: epoch 0, batch 200: 0.268s
[2017-04-20 15:16:31,141] INFO in train: epoch 0, batch 201: 0.269s
[2017-04-20 15:16:31,416] INFO in train: epoch 0, batch 202: 0.275s
[2017-04-20 15:16:31,691] INFO in train: epoch 0, batch 203: 0.274s
[2017-04-20 15:16:31,966] INFO in train: epoch 0, batch 204: 0.275s
[2017-04-20 15:16:32,239] INFO in train: epoch 0, batch 205: 0.273s
[2017-04-20 15:16:32,513] INFO in train: epoch 0, batch 206: 0.274s
[2017-04-20 15:16:32,791] INFO in train: epoch 0, batch 207: 0.277s
[2017-04-20 15:16:33,066] INFO in train: epoch 0, batch 208: 0.275s
[2017-04-20 15:16:33,335] INFO in train: epoch 0, batch 209: 0.269s

feed_dict (https://github.com/4Catalyzer/dl-papers/tree/benchmark-feed-data-only):
[2017-04-20 15:18:00,192] INFO in train: epoch 0, batch 200: 0.275s
[2017-04-20 15:18:00,470] INFO in train: epoch 0, batch 201: 0.278s
[2017-04-20 15:18:00,751] INFO in train: epoch 0, batch 202: 0.281s
[2017-04-20 15:18:01,029] INFO in train: epoch 0, batch 203: 0.277s
[2017-04-20 15:18:01,309] INFO in train: epoch 0, batch 204: 0.280s
[2017-04-20 15:18:01,583] INFO in train: epoch 0, batch 205: 0.274s
[2017-04-20 15:18:01,859] INFO in train: epoch 0, batch 206: 0.276s
[2017-04-20 15:18:02,138] INFO in train: epoch 0, batch 207: 0.279s
[2017-04-20 15:18:02,414] INFO in train: epoch 0, batch 208: 0.275s
[2017-04-20 15:18:02,690] INFO in train: epoch 0, batch 209: 0.276s

This doesn't say anything on TF v PyTorch, but it's consistent with what we saw earlier in there not being a huge performance gap between feed_dict and queues.

Turns out I don't need to re-build TF after all. In fact it looks like all I need to do is to set TF_ENABLE_WINOGRAD_NONFUSED=1.
I get comparable numbers to what you report above when using the 1.0.1 Docker image, and when installing 1.1.0rc2 from pip.
Against https://github.com/4Catalyzer/dl-papers/tree/benchmark-queue as above:
[2017-04-20 15:25:47,770] INFO in train: epoch 0, batch 200: 0.162s
[2017-04-20 15:25:47,934] INFO in train: epoch 0, batch 201: 0.164s
[2017-04-20 15:25:48,100] INFO in train: epoch 0, batch 202: 0.165s
[2017-04-20 15:25:48,266] INFO in train: epoch 0, batch 203: 0.166s
[2017-04-20 15:25:48,431] INFO in train: epoch 0, batch 204: 0.164s
[2017-04-20 15:25:48,594] INFO in train: epoch 0, batch 205: 0.164s
[2017-04-20 15:25:48,761] INFO in train: epoch 0, batch 206: 0.167s
[2017-04-20 15:25:48,926] INFO in train: epoch 0, batch 207: 0.164s
[2017-04-20 15:25:49,089] INFO in train: epoch 0, batch 208: 0.163s
[2017-04-20 15:25:49,254] INFO in train: epoch 0, batch 209: 0.165s

I can see the comment in the source code regarding testing, but this seems to make a significant difference in performance when using common model types that heavily use 3x3 convolutions.
How can we get this added to the TF documentation?

@tfboyd is there an open ticket tracking the progress on (ie making this work):

Once we are sure WINOGRAD is working in all instances it will just be the default.

@tfboyd Currently I don't have other machines to test on, but I ran both several times. TF is always within 5% slower. It's 1.94it/s for TF and around 1.99it/s for pytorch (without extra arguments to change the widen factor, etc). Before enabling the NONFUSED algorithm (cudnn algo No.7, as you can see in the log), TF is about 1.25it/s.
I'm using Tesla M40, E5-2680v3, tensorflow nightly downloaded yesterday, cuda8.0, cudnn5.1.
I modified my TF code to use constant input and the results are the same. I'm looking at 10-step average time, but the number is quite stable after the warm-up.
To clarify a bit, my log shows that

although TF and pytorch use different methods to choose cudnn algo, they chose the same algo for all the convolutions in this model.
But in some convolution, they invoke cudnn with different shapes. I think that's because TF padding convention is different from others.
According to the doc, for Conv32x32 kernel3 stride2 padSAME, TF will pad 0 at left/top and 1 at right/bottom. If I'm not mistaken, cudnn pad in the opposite way. This explains this line from TF VLOG:

Conv accepts: 128, 160, (33, 33), 320, (3, 3), (2, 2), (1, 1), 1, 0 -> (1, 0)

where TF pre-pad the image to become (33,33) and then call cudnn code here. I'm just wondering could that be a possible performance issue, because 33 doesn't look like a good number for cache.

@ppwwyyxx that is good data.  I am not super worried about 5% although I would like to close the gap.
I will try to followup on this and get a bunch of the items mentioned above documented.

I'd like to highlight that I think the biggest gap here is in DX. We started migrating from Theano to TensorFlow earlier this year. This is partly my frustration speaking, but the process looked something like this:

Write naive model implementations, which end up using NHWC and unfused batch norm.
See documentation that suggests NCHW is faster for convolutions (this predates the perf guide). Refactor models to support both NCHW and NHWC, because we still need to support CPU inference. Observe that the models run ~6x slower. Give up on this for a bit.
See documentation that fused batch norm is faster. Switch back from tf.layers.batch_normalization to tf.contrib.layers.batch_norm. This shows a performance improvement.
Try NCHW again, and now see a performance improvement.
a. Realize earlier issue was due to https://github.com/tensorflow/tensorflow/issues/7551
See the performance guide. Convert input pipeline to use queues instead of feed_dict. See negligible speedup.
Compare to straightforward PyTorch impl, and observe that the PyTorch impl runs much faster.
Learn through this issue that we need to enable an undocumented environment flag to access the fastest cuDNN mode for 3x3 convolutions.

As a developer, this is a really suboptimal experience.
My impression is that the modal TF example or published code is at something like our step (1) above, in that it uses NHWC, uses unfused batch norm, and doesn't enable the non-fused Winograd convolution. Correspondingly, performance is quite far from optimal.
By contrast, though with a smaller sample size, the PyTorch examples I've seen generally seem to do "the right thing" performance-wise, and seem to run quickly out-of-the-box. (Also, the status of the built-in PyTorch layer API makes separate PyTorch examples far more consistent in terms of how the code reads).
I'm very grateful for your help in tracking down these issues, but I really wish the out-of-the-box experience were better, and that it didn't take so much work to get to this point.

I feel your pain.
On the plus side TF 1.1 gives you very similar performance of the "secret flag" that I should not have shared.  One step at a time.  I do not want to argue your points as I do understand the frustration.  On the feed_dict issue, even if it was not helpful to remove in this instance it is not normally a good practice for scaling.  Thank you for the feedback.  I consider this item closed and I will try to followup on the requests and items in the various comments.

@tfboyd
Can you double-check the timings you got against master without TF_ENABLE_WINOGRAD_NONFUSED=1?
I just built TF by hand against master, with compute 3.7 and all the CPU stuff enabled, and without setting TF_ENABLE_WINOGRAD_NONFUSED=1, I still see:
[2017-04-20 20:27:27,056] INFO in train: epoch 0, batch 200: 0.284s
[2017-04-20 20:27:27,340] INFO in train: epoch 0, batch 201: 0.284s
[2017-04-20 20:27:27,616] INFO in train: epoch 0, batch 202: 0.276s
[2017-04-20 20:27:27,898] INFO in train: epoch 0, batch 203: 0.282s
[2017-04-20 20:27:28,175] INFO in train: epoch 0, batch 204: 0.277s

With TF_ENABLE_WINOGRAD_NONFUSED=1, I see as expected:
[2017-04-20 20:29:37,525] INFO in train: epoch 0, batch 200: 0.170s
[2017-04-20 20:29:37,695] INFO in train: epoch 0, batch 201: 0.169s
[2017-04-20 20:29:37,861] INFO in train: epoch 0, batch 202: 0.166s
[2017-04-20 20:29:38,028] INFO in train: epoch 0, batch 203: 0.166s
[2017-04-20 20:29:38,195] INFO in train: epoch 0, batch 204: 0.167s

Naively this makes sense to me. The non-fused Winograd operation should still be the fastest for 3x3 convolutions, so it'd be odd to see a performance gain without enabling it.

yeah I will.  It was late at night and it is possible I exported the variable and did not clean it.  I will include the sha-hash as well.

@taion  Hey, I want to do a short write up for tensorflow-discuss covering your model and the flags we used to improve performance.  I know you are frustrated by the churn and I want to provide others with these tips so they can get improved performance while we continue to bring the feature into the core.  I did not want to reference your Github without your permission, I would like to reference your github username as well if that tis ok with you.   Obviously I need to work out if WINOGRAD is still needed as a flat in TF 1.1.  I ran out of time today and I need to spin up my AWS instance again and install everything.  Not a big deal just takes time.

Thanks again for your help.
Feel free to reference our repo. We'd been meaning to open source some of our paper repros anyway, since there didn't seem to be many TF examples in the wild that supported flexible data_format and used fused batch norm.
I personally don't mind churn at all; if anything I'd be quite happy to see e.g. defaults updated so that the fast way and the default way were the same.

Feel free to me and/or the repo, I mean.

Cool and thank you.

@taion   I am finally back to doing a write up and I hope to send it this week but I suspect that means next week.  Also I confirmed what you saw.  Regardless of version, even the latest head of master, you need WINOGRAD on for the boost.  I am sure we have an internal bug to make that on by default or on when it makes sense, but I am going to start obnoxiously asking about it.

Thanks, looking forward to it. We've since done some of that benchmarking on Pascal GPUs and verified that enabling non-fused Winograd wasn't as big a deal here, but it'd be great to see that massive speedup on 3x3 convolutions by default on Kepler. 3x3 convolutions are extremely common, and most cloud vendors are still only providing Kepler hardware.

authors: cancan101 ppwwyyxx taion tfboyd


text:
Fix a small compilation failed bug #20545

Fixed: #20545
Line 86: details.name += ":" + string(profiling_string); should be details.name += ":" + std::string(profiling_string);

I signed it!

Nagging Assignee @martinwicke: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Hi Nandi Wang (@slobber),
We are working on identifying redundant development and duplicate pull requests. We have found there is a pull request: https://github.com/tensorflow/tensorflow/pull/20330 which might be duplicate to this one and already merged into main stream. So maybe this pull request could be closed.
Thank you very much for your time!

That looks like a correct duplicate detection, I'll close this PR. Thanks!

source: gharchive/pull-request

authors: FancyCoder0 martinwicke slobber tensorflowbutler

repo: tensorflow/tensorflow

license_type: permissive

license_source: github-api


text:
Remove duplicated arg description in doc string

sequence_length description is repeated in bidirectional_dynamic_rnn documentation. It looks like it split the dtype description into two sections. I believe that dtype is required if either initial state is not provided (like bidirectional_rnn) and that these changes should reflect that.

Can one of the admins verify this patch?

LGTM.

Jenkins, test this please.

Jenkins, test this please.

authors: drpngx nschuc sotojuan tensorflow-jenkins


text:
[TF-TRT] Removing tensor name from error message to clean non conversion report

@bixia1 : for review
Simplifies the error message generated when a node is converted during segmentation

@bixia1 any update ?

Would you please simplify the PR description by moving some detail to a conversation block? Other than this, the PR looks fine to me.

The old detail logging can be retrieved using export TF_CPP_VMODULE="segment=1,convert_nodes=2"
Before:
- Identity -> ##x
		- [Count: ##x] Failed to convert input <tensor_name> to a TRT_TensorOrWeights: Unsupported tensorflow data type bool

- ConcatV2 -> ##x
        - [Count: ##x] Failed to convert input <tensor_name> to a TRT_TensorOrWeights: Input tensor rank is unknown.
		- [Count: ##x] Failed to convert input <tensor_name> to a TRT_TensorOrWeights: Input tensor with shape [0] is an empty tensor, which is not supported by TRT

- Split -> ##x
		- [Count: ##x] Failed to convert input <tensor_name> to a TRT_TensorOrWeights: Input tensor rank is unknown.

- Reshape -> ##x
		- [Count: ##x] Failed to convert input B<tensor_name> to a TRT_TensorOrWeights: Unsupported tensorflow data type int64

After:
- Identity -> ##x
		- [Count: ##x] Failed to convert at least one input to a TRT_TensorOrWeights: Unsupported tensorflow data type bool

- ConcatV2 -> ##x
		- [Count: ##x] Failed to convert at least one input to a TRT_TensorOrWeights: Input tensor rank is unknown.
		- [Count: ##x] Failed to convert at least one input to a TRT_TensorOrWeights: Input tensor with shape [0] is an empty tensor, which is not supported by TRT

- Split -> ##x
		- [Count: ##x] Failed to convert at least one input to a TRT_TensorOrWeights: Input tensor rank is unknown.

- Reshape -> ##x
		- [Count: ##x] Failed to convert at least one input to a TRT_TensorOrWeights: Unsupported tensorflow data type int64
		- [Count: ##x] Failed to convert at least one input to a TRT_TensorOrWeights: Input tensor rank is unknown.

@bixia1 done

@DEKHTIARJonathan  would you please rebase?

@DEKHTIARJonathan Can you please check @bixia1's comments and keep us posted ? Thanks!

@bixia1 : rebase done

authors: DEKHTIARJonathan bixia1 gbaned


text:
Basic setup steps for contributing

There was no instructions on how to get set up and running tests.
(e.g. I found a bug in tensorflow and just want to fix it)

Edit: TODO:

[ ] Setup instructions for non-MacOS users
[ ] System tests for setup instructions?
[ ] bin/setup?
[ ] bin/test? Would test all targets by default?

Can one of the admins verify this patch?

@wolffg feel free to chime in.
That'll work for Mac, but we have other instructions for other OS. Maybe it's best to point to the general source building instructions.

@wolffg ping?

I started working on a draft https://github.com/Dorian/doma/blob/master/_drafts/2017-04-16-contributing-to-tensorflow.md
But ultimately a bin/setup and bin/test would be my goal (even if not integrated in the main repo)

I'm going to setup an automated process with a MacOS virtual image to test my instruction steps, might take a bit more time than I thought but it's all good :)

Hi Dorian, it turns out as part of a huge documentation day on Friday, a change is in the works to provide a full contributions list, which apparently includes this as well.  So I'm going to close this for now, and perhaps you can add to that one when it gets pushed (hopefully tomorrow?)

https://github.com/tensorflow/tensorflow/pull/9604/commits/a8cf54a732d2385fa7b97495c73a91e4515e8d60 is the pending file.

@vrv Nice! Thanks!
I also found a hacky way to get things working quickly:
> docker run -it tensorflow/magenta
> cd /tensorflow/tensorflow
> python examples/learn/iris.py
> bazel test python

authors: Dorian drpngx martinwicke tensorflow-jenkins vrv


text:
Update README.md for graph transform

Do not recommend to useremove_nodes(op=Identity, op=CheckNumerics) in the README. It breaks the control flow based operations.

Can one of the admins verify this patch?

Thanks for sending the PR @raingo. @petewarden mind taking a look? Thanks!

Thanks for the PR! Can you give some more background on the specific issues you're hitting? Control flow is generally only needed for Variable assignment, which generally doesn't happen during inference, so it would be good to know more about what's going wrong in your case.

@petewarden For example, for object detection, there is a tf.image.non_max_suppression step per each object class. I used tf.map_fnto iterate through each class. This iteration is necessary in the deploy time.

@petewarden any modifications we would need here?

Sorry for the slow response on this. Can we address this by leaving the code example but adding documentation along the lines of "this may cause problems if you have a model with control-flow based operations"? I'd hate people to miss out on the optimizations in other cases.
I think the real solution would be to warn or otherwise avoid removing ops that are significant for control flows, but I'm not sure how to do that yet.

ping for @raingo !

@petewarden PTAL. Just added a note about the control flow ops.

authors: drpngx petewarden raingo tensorflow-jenkins vrv yifeif


text:
splitting some tests for parallel speedups.

Have split the presubmit Cortex-M tests into bluepill and stm32f4 jobs as well as splitting the Makefile x86 presubmit into two jobs. This roughly halves the time for ci.yml to run (from around an hour to roughly 30 minutes). Unfortunately, it's all for naught right now because the xtensa hifimini test added to xtensa.xml last week is a pig that takes around 45 minutes to run.
Note that these times are approximate because there is a 20%+ time variance depending on github's weather. Also note there are occasional pathological runtimes where a job can take twice as long as normal to run for no apparent reason. During testing this PR I had a bazel (presubmit) that took over an hour when the normal run time is ~32 minutes.
Also note that CI on this is going to fail because we are changing some required test names. The following changes will need to be implemented in our branch protections job list:
Makefile x86 (presubmit) is now Makefile x86 One (presubmit) and Makefile x86 Two (presubmit)
Cortex-M (presubmit) is now Cortex-M Bluepill (presubmit) and Cortex-M stm32f4 (presubmit)
BUG=https://issuetracker.google.com/issues/266654467

List of test runtimes at bug.

Have created a new PR covering this issue: https://github.com/tensorflow/tflite-micro/pull/1746

repo: tensorflow/tflite-micro

license_source: bigquery


text:
Remote Server Refactor

Motivation and Context
Why is this change required? What problem does it solve?:
Refactor of the remote server protocol and code. This sets us up to allow for multiple remote storage locations in the near future.
Description
Describe your changes in detail:

New RPCs

PushFindDataOrigin: Preceeds calls to PushData. Ask remote server where it should push to, and receive instructions on what format/info should be attached to Push (URI, compression, location (GRPC, S3, MINIO - eventually).
FetchFindDataOrigin: Ask remote server where some data with digest can be fetched from. Returns all necessary info to access it (location, uri, authorization, etc.)
PushBeginContext: Tell server your about to start sending many requests to PushData and that it should open necessary file containers.
PushEndcontext: Tell server you've finished uploading content and that it can close and forget about opened file containers.

Fetch/Push Data RPCs send one piece of data at a time.

concurrent.futures.ThreadPoolExecutor is used in order to parallelize requests (lots of time is spent waiting on the network). This is a fairly performant solution for the time being.

Removed ability to limit download to max_num_bytes. This was not compatible with implementation details and can be solved by using the new fetch_data_samples() function introduced in #210

Screenshots (if appropriate):
Types of changes
What types of changes does your code introduce? Put an x in all the boxes that apply:

[x] Documentation update
[x] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[x] Breaking change (fix or feature that would cause existing functionality to change)

Is this PR ready for review, or a work in progress?

[x] Ready for review
[ ] Work in progress

How Has This Been Tested?
Put an x in the boxes that apply:

[x] Current tests cover modifications made
[ ] New tests have been added to the test suite
[x] Modifications were made to existing tests to support these changes
[ ] Tests may be needed, but they are not included when the PR was proposed
[ ] I don't know. Help!

Checklist:

[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] I have read the CONTRIBUTING document.
[x] I have signed (or will sign when prompted) the tensorwork CLA.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

@lantiga can you please review when able?

repo: tensorwerk/hangar-py


text:
Dependency Conflict When Installing tt-smi in Environment with Pybuda Already Installed

Description
When installing current tt-smi version within the same python environment where pybuda v0.18.1-rc is already installed a pip dependency conflict occur.
Reproduce
To reproduce follow the following steps:

Scp over zip version of the release.
Unzip zip files.
Create the python env

python3 -m venv python_env
source python_env/bin/activate
pip install -U pip

Pip install the whl files within the environment
Then install tt-smi using instruction within the tt-smi repository README.

Observed behavior
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
flax 0.6.0 requires rich~=11.1, but you have rich 13.7.0 which is incompatible.
pybuda 0.1.240703+dev.gs.c2d0102 requires requests==2.28.2, but you have requests 2.32.0 which is incompatible.

Expected behavior
No pip dependencies errors and a successful install of both pybuda and tt-smi within the same python virtual environment

For our next release lets update these versions to see if it causes any issues.

Should be resolved in the latest main.

source: gharchive/issue

authors: jserbedzijaTT milank94 staylorTT

repo: tenstorrent/tt-buda


text:
Remove spurious docs jobs / workflows / executions

Issue #8068
Moved docs-latest-public.yaml into all-post-commit-workflows.yaml'
Removed the doc step from fast-dispatch-build-and-unit-tests.yaml and also from the secret steps hidden within slow and fast dispatch tests (run_pre_post_commit_regressions_fast_dispatch.shandrun_pre_post_commit_regressions_slow_dispatch.sh` )

I would recommend running both post commit and the docs workflows separately and ensuring they work

Build Docs - https://github.com/tenstorrent/tt-metal/actions/runs/8941441478
All Commits - https://github.com/tenstorrent/tt-metal/actions/runs/8941437613

let's not submit this until the post-commit time is back to normal; starting docs step after tests increases post-commit time by quite a bit

Many many thanks for the review!
I have updated to use the build runner for docs and removed the gate in all_commit tests on the unit-tests.
Here are the successful runs of all-commit and build-docs workflows:

https://github.com/tenstorrent/tt-metal/actions/runs/8950827437
https://github.com/tenstorrent/tt-metal/actions/runs/8950828950

You can see that the build of docs take 6 mins.

I am going to test the standalone workflow after the merge since it is a new one and it does not yet appear in the list of available workflows.

authors: dimitri-tenstorrent tt-rkim

repo: tenstorrent/tt-metal


text:
autosave feature

Hello
Do you think it should be possible to add an "auto-save" function. For example the file I am working on is automatically saved  every 10 minutes ?

Hi
That is a good idea, will add it to the plan. Should be easy to implement.
Thanks a lot!
Cheers

authors: shantee tentone

repo: tentone/nunuStudio

license: MIT


text:
README assets

Done


text:
NvStrapsReBar.exe displays "UEFI DXE driver status: Unconfigured (0x0000000000000028)" all the time

System

Motherboard: ASUS P9X79 PRO
BIOS Version: 4801 (modded)
GPU: nVidia GTX 1660 SUPER
[ X] CSM is turned off. Make sure to confirm this in the BIOS and not with GPU-Z or similar since it can be inaccurate
[ X] 4G decoding is enabled. Make sure to confirm this in the BIOS and not with GPU-Z or similar since it can be inaccurate
[ X] UEFIPatch is applied (see Using UEFIPatch for more information). On some motherboards DSDT Patching is also needed
[ X] I have read Common issues (and fixes)

Description
I've managed to boot with 4G enabled in my ASUS P9X79 PRO mainboard with this patch and GPU-Z confirms that it's enabled. I'm not sure if I also need to use this. (DSDT Patching)

Also I've this in my device manager:

But when execute the NvStrapsReBar.exe it always says "UEFI DXE driver status: Unconfigured (0x0000000000000028)". I've tried my best selecting the P option to set the "Target BAR size" to 256MiB and then S to save the settings. But after reboot my system it says "UEFI DXE driver status: Unconfigured (0x0000000000000028)" again. So no changes and Resizable BAR still disabled. I've tried many times and there is no way to get it working.
What's wrong? Do I need any other UEFI patch than the one referenced before?
This is the BIOS that I'm using:
P9X79PRO.zip

Have you tried option O (Enable override for BAR size mask for PCIe ReBAR capability)?

No. Should I use it? I didn't know what it was for, so I decided not to touch it.

Have you tried option O (Enable override for BAR size mask for PCIe ReBAR capability)?

What I did was to uncomment the line "Extend MMIOH limit to fix Above 4G Decoding (X79)" in patches.txt. And with this now my system boots fine with no issues and GPU-Z (I don't know any other procedure to check it) shows that "Above 4G Decode" is enabled. But the issue is that it seems that the NvStrapsReBar.exe has no effect. With every reboot it displays again the same message "UEFI DXE driver status: Unconfigured (0x0000000000000028)"

I don't know what any other thing should I test or what other info provide here to help. The NvStrapsReBar.exe itself doesn't provide too much info or any clues, so I don't know what's wrong in my setup. However you can take a look at my current BIOS that I've attached into the first message. The NvStrapsReBar.ffs is included as you can confirm by using UEFITool.

This is what I get when using the W option:
DXE Driver configuration:
        isDirty:           0
        OptionFlags:       0x08
        nPciBarSize:       8
        nGPUSelectorCount: 1
                GPUSelector1:  deviceID:            21C4
                GPUSelector1:  subsysVendorID:      FFFF
                GPUSelector1:  subsysDeviceID:      FFFF
                GPUSelector1:  bus:                 FF
                GPUSelector1:  device:              FF
                GPUSelector1:  function:            FF
                GPUSelector1:  barSizeSelector:     2
                GPUSelector1:  overridebarSizeMask: 0

        nGPUConfigCount:   1
                GPUConfig1:    deviceID:        21C4
                GPUConfig1:    subsysVendorID:  1458
                GPUConfig1:    subsysDeviceID:  407D
                GPUConfig1:    bus:             01
                GPUConfig1:    device:          00
                GPUConfig1:    function:        0
                GPUConfig1:    BAR0 base:       0xFA00'0000
                GPUConfig1:    BAR0 top:        0xFAFF'FFFF

        nBridgeCount:      1
                BridgeConfig1: vendorID:        8086
                BridgeConfig1: deviceID:        3C04
                BridgeConfig1: bus:             00
                BridgeConfig1: device:          02
                BridgeConfig1: function:        0
                BridgeConfig1: secondary bus:   01

I don't know if it helps.

I've also noticed that the NVIDIA Control Panel doesn't display any info (yes or no, enable or diable) about "Resizable BAR" into the System Info dialog. I suppose that this is normal because it's not officially supported on this model.

Have you tried option O (Enable override for BAR size mask for PCIe ReBAR capability)?

Now I've tried it and there is no changes. Using thisssssssssssss option doesn't seems to have any effect on the obtained results. May be that @terminatorul could create a debug version wich provides some additional info to know what's happening in my case (ASUS P9X79).

Finally solved by installing the 0.3 version of NvStrapsReBar. It seems that the 0.4 release is not working fine.

Hi, can you share your final modded bios for P9X79 Pro?
Thanks!

authors: FirstEver-eu fidoboy

repo: terminatorul/NvStrapsReBar


text:
added create query parameter for update document

closes #186
Signed-off-by: NeelParihar neel@terminusdb.com

Great! Did you test it against the server?

Yes it is working as expected 👍

authors: Neelterminusdb spl

repo: terminusdb/terminusdb-client-js


text:
🛑 El Loro de Oro is down

In 02f2798, El Loro de Oro (https://lorodeoro.la-razon.com) was down:

HTTP code: 503
Response time: 67 ms

Resolved: El Loro de Oro is back up in be948cd.

repo: terorero/monitor


text:
fix: Ensure resource exists so that destroy command works properly

Description
Ensure that ressource exist before referencing it so that destroy command works properly
Motivation and Context

Fixes https://github.com/terraform-aws-modules/terraform-aws-autoscaling/issues/215
Breaking Changes

No breaking change
How Has This Been Tested?

[ ] I have updated at least one of the examples/* to demonstrate and validate my change(s)
[ ] I have tested and validated these changes using one or more of the provided examples/* projects

[x] I have executed pre-commit run -a on my pull request

This is a provider related issue and not a module issue:

https://github.com/hashicorp/terraform/pull/32207
https://github.com/hashicorp/terraform/pull/32208


text:
docs: Variable default_cache_behavior is required if distribution is enabled

Description
It is unclear if/that default default_cache_behavior is required is a distribution is enabled. Attempting to use the module without generates a bunch of errors.
If your request is for a new feature, please use the Feature request template.

[X ] ✋ I have searched the open/closed issues and my issue is not listed.

This is a continuation of https://github.com/terraform-aws-modules/terraform-aws-cloudfront/issues/8 with a Pull Request to update the README.
Versions

Module version [Required]: 3.2.1

Terraform version: 1.3.7

Provider version(s): 5.26.0

Reproduction Code [Required]
module "cloudfront" {
  source  = "terraform-aws-modules/cloudfront/aws"
  version = "3.2.1"

  comment               = "My CloudFront distribution"

  origin = {
    puppy = {
      domain_name = module.puppy_origin.dns_name
      custom_origin_config = {
        http_port              = 80
        https_port             = 443
        origin_protocol_policy = "http-only"
        origin_ssl_protocols   = ["TLSv1", "TLSv1.1", "TLSv1.2"]
      }
    }

    kitty = {
      domain_name = module.kitty_origin.dns_name
      custom_origin_config = {
        http_port              = 80
        https_port             = 443
        origin_protocol_policy = "http-only"
        origin_ssl_protocols   = ["TLSv1", "TLSv1.1", "TLSv1.2"]
      }
    }
  }

  # default_cache_behavior = {
  #   target_origin_id           = "puppy"
  #   viewer_protocol_policy     = "allow-all"

  #   allowed_methods = ["GET", "HEAD", "OPTIONS"]
  #   cached_methods  = ["GET", "HEAD"]
  #   compress        = true
  #   query_string    = true
  # }

  ordered_cache_behavior = [
    {
      path_pattern           = "/puppy/*"
      target_origin_id       = "puppy"
      viewer_protocol_policy = "allow-all"

      allowed_methods = ["GET", "HEAD", "OPTIONS"]
      cached_methods  = ["GET", "HEAD"]
      compress        = true
      query_string    = true
    },
    {
      path_pattern           = "/kitty/*"
      target_origin_id       = "kitty"
      viewer_protocol_policy = "allow-all"

      allowed_methods = ["GET", "HEAD", "OPTIONS"]
      cached_methods  = ["GET", "HEAD"]
      compress        = true
      query_string    = true
    }
  ]
}

Errors (truncated):
.....................
.....................
╷
│ Error: Invalid function argument
│
│   on .terraform/modules/cloudfront/main.tf line 165, in resource "aws_cloudfront_distribution" "this":
│  165:         for_each = lookup(i.value, "lambda_function_association", [])
│     ├────────────────
│     │ i.value is null
│
│ Invalid value for "inputMap" parameter: argument must not be null.
╵
╷
│ Error: Invalid function argument
│
│   on .terraform/modules/cloudfront/main.tf line 176, in resource "aws_cloudfront_distribution" "this":
│  176:         for_each = lookup(i.value, "function_association", [])
│     ├────────────────
│     │ i.value is null
│
│ Invalid value for "inputMap" parameter: argument must not be null.

PR still open

Why this was marked as completed? The error shown in the description, still appears when you don't specify any default_cache_behavior

authors: kjenney matiasbertani


text:
fix: Use a version for  to avoid GitHub API rate limiting on CI workflows

Description

Update pre-commit workflow CI checks to use pinned version for tflint to avoid GitHub API rate limiting and use latest action versions
Update pre-commit versions to latest

Motivation and Context

Avoid failing CI checks due to GitHub API rate limiting

Breaking Changes

No

How Has This Been Tested?

[ ] I have updated at least one of the examples/* to demonstrate and validate my change(s)
[ ] I have tested and validated these changes using one or more of the provided examples/* projects
[x] I have executed pre-commit run -a on my pull request

This PR is included in version 4.0.1 :tada:

authors: antonbabenko bryantbiggs

repo: terraform-aws-modules/terraform-aws-elb


text:
Missing dependency between domain and access policy resources

While troubleshooting a domain permission error, I found out the missing dependency was the root cause.
https://github.com/terraform-aws-modules/terraform-aws-opensearch/blob/f4792509f652f8c085cdcdf3c9b576d6588bc8ca/main.tf#L266
Problem is exhibited when creating a cluster from scratch with the variables set as following:
create_access_policy = true

access_policy_statements = {
  iam = {
    principals = [
      {
        type        = "AWS"
        identifiers = ["*"]
      }
    ]
    actions = ["es:ESHttp*"]
  }
}

My best guess is aws_opensearch_domain_policy.this[0] is changed while aws_opensearch_domain.this[0] is still being changed.

what exactly is the issue or the error that you are receiving?

Terraform code:
`...
enable_access_policy = true
Access policy
access_policy_statements = [
{
effect = "Allow"
  principals = [{
    type        = "*"
    identifiers = ["*"]
  }]

  actions = ["es:*"]

  condition = [{
    test     = "IpAddress"
    variable = "aws:SourceIp"
    values   = ["127.0.0.1/32"]
  }]
}

]
...Terraform apply output:Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:

create

Terraform will perform the following actions:
module.opensearch.module.opensearch.aws_opensearch_domain_policy.this[0] will be created

resource "aws_opensearch_domain_policy" "this" {

access_policies = jsonencode(
{
+ Statement = [
+ {
+ Action    = "es:"
+ Effect    = "Allow"
+ Principal = ""
+ Resource  = "arn:aws:es:eu-central-1:111076863103:domain/opensearch-domain/*"
},
]
+ Version   = "2012-10-17"
}
)
domain_name     = "opensearch-domain"
id              = (known after apply)
}

Plan: 1 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.
Enter a value: yes
module.opensearch.module.opensearch.aws_opensearch_domain_policy.this[0]: Creating...
╷
│ Error: updating OpenSearch Domain Policy (): ValidationException: Apply a restrictive access policy to your domain
│
│   with module.opensearch.module.opensearch.aws_opensearch_domain_policy.this[0],
│   on .terraform/modules/opensearch.opensearch/main.tf line 263, in resource "aws_opensearch_domain_policy" "this":
│  263: resource "aws_opensearch_domain_policy" "this" {`
The problem is in local create_access_policy variable rule. It is require access_policy_override_policy_documents and access_policy_source_policy_documents variables.
Thanks!

authors: bryantbiggs ohmer oleksiikhoriev


text:
Use a template_file datasource rather than ${replace()} for user-data

I guess the template data source didn't exist when this was first written -- rather than implement our own very simple templating system lets just use it now.

lgtm

authors: antonbabenko ashb


text:
chore: update CODEOWNERS

Update CODEOWNERS per team update

Rather than individuals, please provide a GitHub team to use for CODEOWNERS.
Thanks!

That may have to come at a later time. I can add a TODO if it helps.

authors: anguillanneuf rogerthatdev


text:
fix: remove replica pwd validation config

Address the issue in the BUG https://github.com/terraform-google-modules/terraform-google-sql-db/issues/430.
We are aware that password validation policy cannot be enabled on a read-replica, however this change provides users with an option to separately enable PWD validation policy on a master and promoted replica instances.

/gcbrun

/gcbrun

authors: anuhyapolisetti bharathkkb g-awmalik


text:
Update contributor content

Description
Mainly updates the contribute-module.md file.
The easiest way to review this might be to view the rendered markdown and click through the includes to view the content that will be embedded there.
Includes these changes:

Create include files (inc-xxx.md) to embed reusable content
Update contribute-module.md to provide specific steps and use includes.
Update implementation-guidelines.md

Small editorial fixes for consistency and style
Change details about commit messages and versioning
Add includes

Reorganize local-dev-setup and use includes
Small editorial changes to tests.md. Also add includes.
Also customizes the Docsify theme (the documentation site generator)
Also removes custom heading anchor IDs (:id=)

Check the relevant boxes:

[x] Documentation update

Checklist

If relevant, a test for the change has been added or updated as part of this PR.
[x] If relevant, documentation for the change has been added or updated as part of this PR.

Merge

Merge using "Squash and merge".
Make sure to use a relevant conventional commit message that is based on the PR contents. The commit message determines whether a new version of the modules needs to be released, and if so, which semver number to use).

As discussed in scrum, merging for now and we can continue to iterate from there.

authors: SirSpidey vburckhardt

repo: terraform-ibm-modules/documentation


text:
chore(deps): update common-dev-assets digest to 910eeda

This PR contains the following updates:

Package
Update
Change

common-dev-assets
digest
7d105a4 -> 910eeda

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, click this checkbox.

This PR has been generated by Renovate Bot.

:tada: This PR is included in version 5.6.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

repo: terraform-ibm-modules/terraform-ibm-cos


text:
update to COS 8.14.0 to pull in auth policy scope changes

Description
update to COS 8.14.0 to pull in auth policy scope changes
Release required?

[ ] No release
[ ] Patch release (x.x.X)
[ ] Minor release (x.X.x)
[ ] Major release (X.x.x)

Release notes content

Run the pipeline
If the CI pipeline doesn't run when you create the PR, the PR requires a user with GitHub collaborators access to run the pipeline.
Run the CI pipeline when the PR is ready for review and you expect tests to pass. Add a comment to the PR with the following text:
/run pipeline

Checklist for reviewers

[ ] If relevant, a test for the change is included or updated with this PR.
[ ] If relevant, documentation for the change is included or updated with this PR.

For mergers

Use a conventional commit message to set the release level. Follow the guidelines.
Include information that users need to know about the PR in the commit message. The commit message becomes part of the GitHub release notes.
Use the Squash and merge option.

/run pipeline

/run pipeline

Need to wait for https://github.com/terraform-ibm-modules/terraform-ibm-cos/pull/765

/run pipeline

As expected, upgrade test failed with:
        	Messages:   	Resource(s) identified to be destroyed
        	            	Name: policy
        	            	Address: module.cos[0].ibm_iam_authorization_policy.policy[0]
        	            	Actions: [create delete]

Since the code is going to do a create before destroy, there will be no disruption to services, so proceeding to skip upgrade test

/run pipeline

:tada: This PR is included in version 1.14.0 :tada:
The release is available on:

GitHub release
v1.14.0

Your semantic-release bot :package::rocket:

authors: ocofaigh terraform-ibm-modules-ops


text:
chore(deps): update common-dev-assets digest to 1d683d9

This PR contains the following updates:

Package
Update
Change

common-dev-assets
digest
1a4e4e6 -> 1d683d9

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

:tada: This PR is included in version 1.1.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


text:
fix(deps): bump KMS with code change

Description
Update KMS all inclusive which removes key ring force delete because it is deprecated by the provider
Release required?

[x] No release
[ ] Patch release (x.x.X)
[ ] Minor release (x.X.x)
[ ] Major release (X.x.x)

Release notes content

Run the pipeline
If the CI pipeline doesn't run when you create the PR, the PR requires a user with GitHub collaborators access to run the pipeline.
Run the CI pipeline when the PR is ready for review and you expect tests to pass. Add a comment to the PR with the following text:
/run pipeline

Checklist for reviewers

[ ] If relevant, a test for the change is included or updated with this PR.
[ ] If relevant, documentation for the change is included or updated with this PR.

For mergers

Use a conventional commit message to set the release level. Follow the guidelines.
Include information that users need to know about the PR in the commit message. The commit message becomes part of the GitHub release notes.
Use the Squash and merge option.

The expectation is that this PR will need to skip upgrade test because of the key ring force delete state change. The changes are in example code, not the main module, so no release is required.

/run pipeline

/run pipeline

/run pipeline

/run pipeline

:tada: This PR is included in version 1.8.11 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

authors: shemau terraform-ibm-modules-ops


text:
chore(deps): update ci dependencies

This PR contains the following updates:

Package
Type
Update
Change

github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper
require
patch
v1.41.1 -> v1.41.2

terraform-ibm-modules/common-pipeline-assets
action
patch
v1.22.4 -> v1.22.5

Release Notes

terraform-ibm-modules/ibmcloud-terratest-wrapper (github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper)
v1.41.2
Compare Source
Bug Fixes

deps: update gomod (#887) (dbc4073)

terraform-ibm-modules/common-pipeline-assets (terraform-ibm-modules/common-pipeline-assets)
v1.22.5
Compare Source
Bug Fixes

removed support for passing MZ_INGESTION_KEY env var to the pipeline (#734) (cb9de3b)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

ℹ Artifact update notice
File name: tests/go.mod
In order to perform the update(s) described in the table above, Renovate ran the go get command, which resulted in the following additional change(s):

2 additional dependencies were updated

Details:

Package
Change

github.com/IBM/go-sdk-core/v5
v5.18.0 -> v5.18.1

github.com/IBM/platform-services-go-sdk
v0.69.3 -> v0.70.0

/run pipeline

:tada: This PR is included in version 1.3.10 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


text:
chore(deps): update common-dev-assets digest to 9211b0b

This PR contains the following updates:

Package
Update
Change

common-dev-assets
digest
9d04954 -> 9211b0b

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, click this checkbox.

This PR has been generated by Renovate Bot.

:tada: This PR is included in version 1.0.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


text:
chore(deps): update ci dependencies

This PR contains the following updates:

Package
Type
Update
Change

common-dev-assets

digest
fedf06c -> 3ece40d

github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper
require
patch
v1.21.5 -> v1.21.7

Release Notes

terraform-ibm-modules/ibmcloud-terratest-wrapper (github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper)
v1.21.7
Compare Source
Bug Fixes

clear the plan file path only if SkipTestSetup is enabled (#647) (169b0d0)

v1.21.6
Compare Source
Bug Fixes

deps: update gomod (#646) (19f21bf)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

/run pipeline

:tada: This PR is included in version 1.2.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


text:
fix(deps): update terraform terraform-ibm-modules/cos/ibm to v8.14.4

This PR contains the following updates:

Package
Type
Update
Change

terraform-ibm-modules/cos/ibm (source)
module
patch
8.14.3 -> 8.14.4

Release Notes

terraform-ibm-modules/terraform-ibm-cos (terraform-ibm-modules/cos/ibm)
v8.14.4
Compare Source
Bug Fixes

deps: update terraform ibm to latest for the deployable architecture solution (#778) (19ca0d7)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

/run pipeline

/run pipeline

:tada: This PR is included in version 1.7.10 :tada:
The release is available on:

GitHub release
v1.7.10

Your semantic-release bot :package::rocket:


text:
Sidecar implementation

Hi,
do you plan on implementing sidecar configuration and collectors?
API endpoints:
/sidecar
/sidecars
/sidecars/configurations
/sidecar/collectors
/sidecar/configurations

[ ] /sidecar/collectors #14

I have released v0.2.0, which supports the Sidecar Collector.
https://terraform-provider-graylog.github.io/#/resources/collector

Sorry but I'll rename graylog_collector to graylog_sidecar_collector.
#16

I have released v0.3.0, which renames graylog_collector to graylog_sidecar_collector and supports the Sidecar Configuration.
https://terraform-provider-graylog.github.io/#/resources/sidecar_configuration

I close this issue because Sidecar Collector and Sidecar Configuration have been supported.
If you have any problems, please comment or create new issues.
Thanks.

Hi sorry for the delayed answer,
I have tried it and it works perfectly for collector and configuration creation/import
one more endpoint that would be awesome to add is /sidecars/{sidecarId}
so that it would be possible to assign collector and configuration to a sidecar
API example:
"assignments": [
    {
      "collector_id": "5ddd3a4dc9e77d0012150861",
      "configuration_id": "5ebd333c48e0fb0012a68280"
    }
  ]

📝

https://docs.graylog.org/en/3.2/pages/sidecar.html
https://docs.graylog.org/en/3.2/pages/configuration/rest_api.html#creating-and-using-access-token

How to setup sidecar with Docker
# docker-compose.yaml
  sidecar:
    image: debian:10.4
    networks:
    - graylog
    tty: true
    stdin_open: true
    command: ["tail", "-f", "/dev/null"]

apt-get update
apt-get install wget
wget https://packages.graylog2.org/repo/packages/graylog-sidecar-repository_1-2_all.deb
dpkg -i graylog-sidecar-repository_1-2_all.deb
apt-get update
apt-get install graylog-sidecar
apt-get install vim

# server_url: http://graylog:9000/api/
# server_api_token: generate api token
vim /etc/graylog/sidecar/sidecar.yml  # update

graylog-sidecar  # Run sidecar

Assign configurations to sidecars
PUT /sidecars/configurations
{
  "nodes": [
    {
      "node_id": "10517347-8fad-4f25-835f-95d1943fa338",
      "assignments": [
        {
          "collector_id": "5ec661032ab79c0012267d29",
          "configuration_id": "5ec709642ab79c001226edf9"
        }
      ]
    }
  ]
}

response status code: 202
Get sidecar and assignments

#23

In my understanding, we have to manage all nodes's assignments in one Terraform resource due to the Graylog API's limitation.
There is no API to manage only one's Graylog Sidecar's assignments.
We have to use the API PUT /sidecars/configurations to manage Sidecar's assignments, and we can't specify Sidecar's ID.
Example
resource "graylog_sidecar" "test" {

}

Document: https://github.com/terraform-provider-graylog/terraform-provider-graylog.github.io/pull/10

@roiterorh
I have released v0.4.0-0 as pre release version.
About document, please see

https://github.com/terraform-provider-graylog/terraform-provider-graylog.github.io/pull/10
Data Source: sidecar
Resource: sidecars

These document isn't published to https://terraform-provider-graylog.github.io/ yet because these resources are pre release.
Please check these resources work well.
If there is no problem, I'll release v0.4.0 and the document.
Thank you.

I have released v0.4.0 and document, so I close this issue.

authors: roiterorh suzuki-shunsuke


text:
SW-2304 Common web app component to select jpeg/png files

added this to the opt-in page just for testing, I WILL remove this prior to check-in
this should pop the camera on mobile

Confirmed it provides an option to load a photo vs browse files on mobile.
This one is good to go.

Works great! Tested Vercel on desktop and mobile. Code is clean and easy to follow

Thanks @charlie-tf ! I reverted one refactor to simplify things - and renamed a file since your last review. If you can take a look again lmk.

@constanzauanini I got withdrawals working with photo uploads.
Had to refactor some code (see PR description).
Outplant is yet to be supported, will work on that next. Same for formatting the success message.
The bulk withdraw quantity setting is to be done - as per plan will leave that for you.
Will merge this in and happy to follow up if you have comments.

repo: terraware/terraware-web


text:
Fix c_align in kontrol.sensact.optical_lever

Reference
https://klog.icrr.u-tokyo.ac.jp/osl/?r=17056
https://gwdoc.icrr.u-tokyo.ac.jp/cgi-bin/private/DocDB/ShowDocument?docid=12874

1465704

repo: terrencetec/kontrol


text:
Can't get the carId .  already logging by the tesla token!

Is there an existing issue for this?

[X] I have searched the existing issues

What happened?
no data

Expected Behavior
No response
Steps To Reproduce
No response
Relevant log output
SELECT
  sum(GREATEST(start_rated_range_km - end_rated_range_km, 0) * car.efficiency * 1000) /
  convert_km(sum(distance)::numeric, 'km') as "consumption_km"
FROM drives
JOIN cars car ON car.id = car_id
WHERE start_date BETWEEN '2024-01-30T16:00:00Z' AND '2024-01-31T06:29:14.495Z' AND car_id =

No request and response collected yet. Hit refresh button

Screenshots

Additional data
No response
Type of installation
Docker
Version
v1.28.2

Possibly same issue as https://github.com/teslamate-org/teslamate/issues/3629.
A fix for the API endpoint change in the works, https://github.com/teslamate-org/teslamate/pull/3630.

ty

repo: teslamate-org/teslamate


text:
Accuracy and speed tests

Related to discussion in: https://github.com/tesseract-ocr/tesseract/issues/707#issuecomment-823491996.
@stweil,
Can you contribute these images and their GT and release them under Apache 2.0 license?
Then, can you make a new daily CI test (for Ubuntu at least) that will include these images?
UNLV images could also be used as part of an accuracy test with CI.

All images and the ground truth are already available online under the free license Public Domain Mark 1.0.
The script for the evaluation is also available online (do5.sh here, currently used local filenames), and so is the list of GT files.
The URLs for a GT filename follow this pattern:
41989179X_0036.txt is at https://digi.bib.uni-mannheim.de/fileadmin/digi/41989179X/gt/41989179X_0036.txt, the image is at https://digi.bib.uni-mannheim.de/fileadmin/digi/41989179X/max/41989179X_0036.jpg.

According to this report, there was a huge regression in accuracy and speed in version 3.03 compared to 3.02.02.
We need to evaluate Tesseract weekly using this dataset, in order to make sure there are no regression in performance. The legacy engine and the CRNN engine should be tested separately. This is very importent TODO.

authors: amitdo stweil

repo: tesseract-ocr/tesseract


text:
"make install" ends with some errors

I wanted to update my working tesseract (version from August) by pulling and compiling the latest version but make install ends with some errors:
$ sudo make install
...
Makefile:333: recipe for target 'install-dataDATA' failed
make[3]: *** [install-dataDATA] Error 1
make[3]: Leaving directory '/work/usr/local/src/tesseract/tessdata/configs'
Makefile:403: recipe for target 'install-am' failed
make[2]: *** [install-am] Error 2
make[2]: Leaving directory '/work/usr/local/src/tesseract/tessdata/configs'
Makefile:461: recipe for target 'install-recursive' failed
make[1]: *** [install-recursive] Error 1
make[1]: Leaving directory '/work/usr/local/src/tesseract/tessdata'
Makefile:483: recipe for target 'install-recursive' failed
make: *** [install-recursive] Error 1

$ tesseract --version

tesseract: error while loading shared libraries: libtesseract.so.4: cannot open shared object file: No such file or directory

Full output see http://dpaste.com/1AMP4MA.txt
Please can you help?

I think it was due to a version conflict on my debian 8 with a packaged-tesseract. After removal of the packaged tesseract, it works (but make install errors remain).

install-dataDATA
This indicates problem with configuration. How did you configured  build?

this is my standard procedure:
make clean
git pull
./autogen.sh
./configure --enable-debug
make
sudo make install

Extract from your build protocol:
make[3]: Entering directory '/work/usr/local/src/tesseract/tessdata/configs'
make[3]: Nothing to be done for 'install-exec-am'.
 /bin/mkdir -p '/usr/local/share/tessdata/configs'
 /usr/bin/install -c -m 644 inter makebox box.train unlv ambigs.train api_config kannada box.train.stderr quiet logfile digits hocr tsv linebox pdf rebox strokewidth bigram txt '/usr/local/share/tessdata/configs'
/usr/bin/install: „inter“ und „/usr/local/share/tessdata/configs/inter“ sind die gleiche Datei

So you are building in /work/usr/local/src/tesseract and want to install to /usr/local. The installer says that /work/usr/local/src/tesseract/tessdata/configs/inter and /usr/local/share/tessdata/configs/inter are the same file and fails therefore. Do you have a symbolic link somewhere (either in your build directory or in the installation directory) which connects both directory hierarchies?

Yeah, you are right. I had indeed such a link when I build the last version !
I will now re-compile and report here, and close, if it solved.
Nevertheless, perhaps the "make" could be improved to catch (capture?) such a user mistake.

Well, make did catch that mistake and said that the installation failed because for some files source and destination were identical ("sind die gleiche Datei").

I reported that issue, and close it now. Issue solved by removing my softlink for tessdata.

root@scube014:~/qingy-fork-master/src/libraries# make clean
Making clean in libraries
/bin/bash: line 20: cd: libraries: No such file or directory
make: *** [clean-recursive] Error 1
pls find this problem .

make install
if [ -L /usr/include/sound ]; then
rm -f /usr/include/sound;
ln -sf /media/ss/DATA/SOFTWARE/Rt-Linux-HDaudio-5.18/alsa-driver-RTv5.18/alsa/include/sound /usr/include/sound;
else
rm -rf /usr/include/sound;
install -d -m 755 -g root -o root /usr/include/sound;
for f in include/sound/.h; do
install -m 644 -g root -o root $f /usr/include/sound;
done
fi
install: cannot stat 'include/sound/.h': No such file or directory
Makefile:188: recipe for target 'install-headers' failed
make: *** [install-headers] Error 1

similar problem with my alsa make install
make install
if [ -L /usr/include/sound ]; then
rm -f /usr/include/sound;
ln -sf /media/ss/DATA/SOFTWARE/Rt-Linux-HDaudio-5.18/alsa-driver-RTv5.18/alsa/include/sound /usr/include/sound;
else
rm -rf /usr/include/sound;
install -d -m 755 -g root -o root /usr/include/sound;
for f in include/sound/.h; do
install -m 644 -g root -o root $f /usr/include/sound;
done
fi
install: cannot stat 'include/sound/.h': No such file or directory
Makefile:188: recipe for target 'install-headers' failed
make: *** [install-headers] Error 1

I got the same error.
just preceded sudo to make install
then boom

@vee-varun : can you be more specific? What kind of error you get, what version of code you used, of etc?

I have same errors.
Not "make install", but "sudo make install" in terminal.
Then i have solved.

I would like to add that I had the same problem on my system

OS: Manjaro 20.2.1 Nibia
Kernel: x86_64 Linux 5.10.2-2-MANJARO

When installing bitcoin-0.21;
To solve it I had to install the new version of boost $ pamac install boost=1.75.0-1

I would like to add that I had the same problem on my system

OS: Manjaro 20.2.1 Nibia
Kernel: x86_64 Linux 5.10.2-2-MANJARO

When installing bitcoin-0.21;
To solve it I had to install the new version of boost $ pamac install boost=1.75.0-1


text:
unittest: Disable build rules for tests which still fail to build

This fixes warnings from autogen.sh.
Signed-off-by: Stefan Weil sw@weilnetz.de

Don't apply. It is bad.

Fixed now, tested and ready to get merged.


text:
[WIP] Span batch implementation

Everything is in progress
Code structure and naming may be changed

Please change the base branch to tip/span-batch because initial implementation of span batch(pcw109550/span-batch-datastructure) got merged to it.

@ImTei Please backmerge https://github.com/testinprod-io/optimism/tree/tip/span-batch since pcw109550/span-batch-remove-tx-data-header got merged(https://github.com/testinprod-io/optimism/pull/9) to tip/span-batch branch.

Can we place all batch related files to batch directory? op-node/rollup/derive/batch/*.

repo: testinprod-io/optimism


text:
Clarify the scope of stdin/stdout for dynamically linked modules

See https://github.com/bep/javywazeroissue/blob/58a6147e45096a2dbfc4d106002898c8f960b677/lib.go#L43
I'm pasting the relevant lines here for discussion:
buff := &bytes.Buffer{}
buff.WriteString(`{ "n": 2, "bar": "baz" }`)

_, err = r.InstantiateModule(ctx, compiledQuickJS, wazero.NewModuleConfig().WithStdout(buff).WithStderr(os.Stderr).WithStdin(buff).WithName("javy_quickjs_provider_v2"))
if err != nil {
	return "", err
}

_, err = r.InstantiateModule(ctx, compiledFoo, wazero.NewModuleConfig().WithName("foo"))
if err != nil {
	return "", err
}

fmt.Println(buff.String()) // Output: {"foo":3,"newBar":"baz!"}

The above works and is more or less the "hello world" example from https://github.com/bytecodealliance/javy.
To me, the foo module is the running instance, yet it uses stdin/stdout from the dynamically linked module, which I find surprising, and makes it impossible to init multiple JS modules (either different modules, or multiple instances of the same) in the same runtime.
I have implemented a proof of concept on this for inclusion in Hugo, with a simple RPC dispatcher over stdin/stdout, and it mostly works great. But for slower running scripts (e.g. Katex), I would want to have multiple instances of the script.[^1] And with the above, I need to create a new runtime per instance (with a shared compilation cache). This works, but is resource heavy:
BenchmarkKatexStartStop/PoolSize1-10         	      61	  18518154 ns/op	12242519 B/op	   18054 allocs/op
BenchmarkKatexStartStop/PoolSize8-10         	       9	 117701093 ns/op	97889447 B/op	  144385 allocs/op
BenchmarkKatexStartStop/PoolSize16-10        	       5	 222097958 ns/op	195748028 B/op	  288710 allocs/op

Ideally I would want to do something ala:
_, err = r.InstantiateModule(ctx, compiledQuickJS, wazero.NewModuleConfig().WithName("javy_quickjs_provider_v2"))
if err != nil {
	return "", err
}

_, err = r.InstantiateModule(ctx, compiledFoo, wazero.NewModuleConfig().WithStdout(buff1).WithStderr(os.Stderr).WithStdin(buff1).WithName("foo1"))
if err != nil {
	return "", err
}
_, err = r.InstantiateModule(ctx, compiledFoo, wazero.NewModuleConfig().WithStdout(buff2).WithStderr(os.Stderr).WithStdin(buff2).WithName("foo2"))
if err != nil {
	return "", err
}
// ...

[^1]: Yes, I have looked at https://github.com/suborbital/javy and https://github.com/F21/javy-wazero, but the bytecodealliance/Shopify fork seems to be the active one, and I don't think the suborbital fork would help with the main problem above.

Looking at my own writing above, I see that I need to do a little investigation to see how this is wired up at the Javy side, but I'll keep this open and I welcome any tips.

Closing this for now, I'm not totally sure my conclusion above is correct, and if this is a real issue, I'm almost certain it's javy's.

I have done some investigating, and I'm guess the current behaviour is as expected with the current API: The executing function gets the fs/stdin/etc. from the module it's invoked.
What could possibly work and be worth while would be something ala:
RegisterModule(name, compiled)

Which when requested as an imported, and not found in the regular instance map, would be instansiated with most of the config inherited from the importer.

repo: tetratelabs/wazero


text:
Propagates error return Close to public API

Before, we only handled errors internally and swallowed them when re-exporting to the public API.

ps we probably should have a convenience Runtime.Close to close all the modules in the right order.


text:
mobile api: need initial sync group state event

Is your feature request related to a problem? Please describe.
Yes, I'm implementing mobile file sync and it seems there is no initial event when a file is marked as pending.
Describe the solution you'd like
There should be an event fired when the first file in a group is marked as pending. This will notify any listener that an item is beginning to be processed.

It looks like this is already in there: https://github.com/textileio/go-textile/blob/master/mobile/cafes.go#L138 - you sure you don't see it?

Yea the event is already there, I see it. But I’m talking about is a method to call when the os gives me upload progress for a single  cafe request. For that single cafe request, the os will tell me like “456 bytes of 1254 total”. I want to be able to pass that information into the go api, have go incorporate that into the size complete calculation, and fire an updated status event. Make sense?

Cool not surprised. I probably had some other big that made me think it wasn’t there. I’ll verify and close this

authors: asutula sanderpick

repo: textileio/go-textile


text:
Flag locally created logs as direct

Fixes #373, and a better alternative to #372.
The approach here is to simply take advantage of the existing metadata store on the logstore to set a flag as to whether a given log is "managed?" by the local peer. Directly "managed" logs are those logs created or directly added by the local peer. Otherwise, unmanaged logs are those from other peers in a given thread. Related, "owned" logs are those logs for which the peer has the private key. There can only be one "owned" log, versus potentially many "managed" logs.
This should support multiple use cases:

The local peer is the client peer. One "owned" log is used to interact with a given thread (single-tenant, single-writer setup)
The local peer is (also) a relay peer. Maybe multiple "managed" logs on behalf of other clients, and zero or one "owned" log to also participate in the thread on its own (multi-tenant, multi-writer setup)
The local peer is (also) a proxy-writer. Maybe multiple "managed" logs on behalf of other clients, and one "owned" log that is written to on behalf of other 'agents' (multi-tenant, single-writer setup)

Ok, @sanderpick, this ended up touching a lot more places than I had initially anticipated, but it should now cover all our bases. Checks for all keys types have been added, and errors thrown for any mismatches. One 'own' log per thread is enforced, but multiple 'managed' logs are possible. There are also tests for the expected behavior, and peers can even replicate managed logs which is important for our JS use-cases.

Thank you @sanderpick!

repo: textileio/go-threads


text:
Non-English letters not recognized

Describe the bug
Some logic only works with English alphabet. It doesn't recognize other uppercased letters.
Text
Ők Cornel West és Robert F Kennedy Jr. Ők elvehetik a szavazatokat Joe Biden-től és Donald Trump-tól.

Actual Result
Ők Cornel West és Robert F Kennedy Jr. Ők elvehetik a szavazatokat Joe Biden-től és Donald Trump-tól.

URL: https://sentence-splitter.netlify.app/#Ők Cornel West és Robert F Kennedy Jr. Ők elvehetik a szavazatokat Joe Biden-től és Donald Trump-tól.

Expected Result
Two sentences should be recognized in the text:

Ők Cornel West és Robert F Kennedy Jr.
Ők elvehetik a szavazatokat Joe Biden-től és Donald Trump-tól.

fixed in https://github.com/textlint-rule/sentence-splitter/releases/tag/v4.4.0

authors: Thyrst azu

repo: textlint-rule/sentence-splitter


text:
Remove tf.reset_default_graph()

This call is unnecessary and can have unexpected consequences for the user.

Looks good, as long as tests pass:)

authors: OKapple aseemw

repo: tf-coreml/tf-coreml


text:
add auto crouch feature

automatically crouches in the air when jumping

amazing useless feature

^

Bunnyhop doesn't work for me since I use +crouchjump.
However, if bunnyhop crouched (and uncrouched) automatically, this would be nice.

license: WTFPL


text:
Support older versions of GHC

Is it possible to support versions of GHC other than 8.10? I'm not sure. It seems like HLint is tightly coupled to that version, but I haven't dug into it.
https://twitter.com/Iceland_jack/status/1264928483840491520

When using HLint as a library, if you can produce a compatible parse tree exogenously, then you can pass it through the HLint API. If you can't though, not all is lost - you can have HLint do it's own parse. This of course means that you for example, you can with build compiler 8.6.5 link HLint with ghc-lib and if you allow it to do it's own parse, then it can do lint analysis. Not being familiar with the plugin concept I can't tell you if this information is helpful or not but I hope you can make sense of it!

Thanks @shayne-fletcher! It sounds like the way forward is to do what hlint-source-plugin does. Or rather, also do that. When using GHC 8.10, everything should work the way it does now. But when using any other version (older or newer) HLint should re-parse the module itself and use that.

Yes, check out hlint's stack.yaml and .cabal - do as it does with respect to ghc-lib and ghc-lib-parser-ex flags for linking, a little cpp in your plugin on ghc version and you should be all set.

Alternatively, you could have hlint link ghc-lib unconditionally and allow it to always do it's parse. Might be simpler. Whatever works before the boss fires you! 😁

authors: shayne-fletcher tfausak

repo: tfausak/splint

license: ISC


text:
XDG_CONFIG_HOME ignored by default on macOS

Describe the bug
Despite having support for XDG_CONFIG_HOME in actions like organize list (as well as having support for finding configs in that location via find_config), the only way to use a config in that location is to explicitly set the ORGANIZER_CONFIG environment variable. It feels silly to have support built into the tool for this convention without supporting it for the most important use case.
Screenshots
❯ organize show
Error: Config problem
Cannot find config "/Users/fontenot/Library/Application Support/organize/config.yaml".
❯ organize list
┏━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Config ┃ Path                                         ┃
┡━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ config │ /Users/fontenot/.config/organize/config.yaml │
└────────┴──────────────────────────────────────────────┘

Environment (please complete the following information):

OS: macOS 14.3
Output of organize --version: organize v3.2.2

Thanks a lot for reporting. Yes this is silly and will be fixed in the next release 👍

Thanks for the quick turnaround on this!

authors: gfontenot tfeldmann

repo: tfeldmann/organize

license: mit


text:
Closing SwiftySettings ?

Hi Tomasz,
I tried to integrate SwifySettings to my app and made a few changes to improve UI (will submit a PR).
In my UI, the user accesses the settings from an app menu, then they lan on the SwiftySettings controller...
Then I did not find a way to bring them back to the app (dismiss the settings controller).
How would you achieve this?

Hi,
I am not sure what kind of navigation stack you are using, but if you have a sliding menu or tab bar it shouldn't be a problem to dismiss settings view controller. Otherwise, you would have to hook up a custom button to SwiftySettingsController which would dismiss the controller. Unfortunatelly this is not  currently supported out of box, but you may achieve something similar by embedding SwiftySettingsViewController in another ViewController which provides such a button.

Hi @jmcazaux,
I used this code:
(viewControllers.first as? UINavigationController)?.navigationBar.items?.first?.rightBarButtonItem = UIBarButtonItem.init(barButtonSystemItem: .done, target: self, action: #selector(dismissVC))

@hintoz Wish i found your answer couple of hours ago ! I was trying to add the button to leftBarButtonItem with no hope. Not it worked when I added to the rightBarButtonItem

@hintoz @iballan
I'm new to iOS programming, where should I put this line to dismiss the setting view?
(viewControllers.first as? UINavigationController)?.navigationBar.items?.first?.rightBarButtonItem = UIBarButtonItem.init(barButtonSystemItem: .done, target: self, action: #selector(dismissVC))

@xzhub In the viewDidLoad method for example.

Thanks, now it works! For newbies like me, please note that you have to add the dismissVC function in ViewController as well.
class ViewController: SwiftySettingsViewController {

    var storage = Storage()

    override func viewDidLoad() {
        super.viewDidLoad()
        // Do any additional setup after loading the view, typically from a nib.
        loadSettingsTopDown()
        (viewControllers.first as? UINavigationController)?.navigationBar.items?.first?.rightBarButtonItem = UIBarButtonItem.init(barButtonSystemItem: .done, target: self, action: #selector(dismissVC))
    }

    @IBAction func dismissVC(_ sender: Any) {
        dismiss(animated: true, completion: nil)
    }

... ...

authors: hintoz iballan jmcazaux tgebarowski xzhub

repo: tgebarowski/SwiftySettings