text:
Placeholder modification behavior

I am using Tensorflow 1.0.1, on Mac OS Sierra 10.12.4, with CPU in a jupyter notebook (but the behavior I am describing appeared in a regular python script too).
The problem I am having is with placeholders. I have been using tensorflow for only 1 month so I was not really comfortable with using them and I made a mistake that took me a very long time to debug.
Here is the type of code I wrote:
import numpy as np
import tensorflow as tf
x_ = tf.placeholder(tf.float64, [5])
x_ = tf.where(tf.is_nan(x_), tf.zeros_like(x_), x_)
a = tf.constant(1, shape=[5])
a = tf.cast(a, tf.float64)
c = a + x_
init_op = tf.group(
    tf.global_variables_initializer(),
    tf.local_variables_initializer())
sess = tf.Session()
sess.run(init_op)
x = np.arange(5, dtype=float)
x[3] = np.nan
sess.run(c, feed_dict={x_: x})

To this, the output was: array([  1.,   2.,   3.,  nan,   5.]), which is not what I expected since I had replaced all the nan with the tf.where. I understood only later that it was because tensorflow was feeding the placeholder in the latest position it appeared (maybe not exactly how it's implemented, but empirically, that is what happens).
Therefore a good (in the sense of what I intended to do) code is:
import numpy as np
import tensorflow as tf
x_ = tf.placeholder(tf.float64, [5])
r = x_
r = tf.where(tf.is_nan(r), tf.zeros_like(r), r)
a = tf.constant(1, shape=[5])
a = tf.cast(a, tf.float64)
c = a + r
init_op = tf.group(
    tf.global_variables_initializer(),
    tf.local_variables_initializer())
sess = tf.Session()
sess.run(init_op)
x = np.arange(5, dtype=float)
x[3] = np.nan
sess.run(c, feed_dict={x_: x})

This indeed outputs array([ 1.,  2.,  3.,  1.,  5.]). However, to me it's a bit weird that you would need to throw in an extra variable to achieve this purpose and even weirder that I didn't receive any error when modifying my placeholder.
My suggestions are:

Throw an error or a warning when reallocating a placeholder.
State clearly that placeholders are not meant to be modified (I looked for it in the docs and didn't find it, maybe I didn't look well enough).
Allow placeholders to be modified (might be offputting for someone who understands it well).

Hi @zaccharieramzi, I would argue this is just how Python works. In your code, you're not actually reallocating or modifying your original placeholder, but reassigning the x_ variable, and then feeding the new x_ value (which is the where op):
x_ = tf.placeholder(tf.float64, [5])
x_ = tf.where(tf.is_nan(x_), tf.zeros_like(x_), x_)
...
sess.run(c, feed_dict=x_: x)

This is analogous to doing something like:
x_ = 1
x_ = x_ * 2
print(x_) # prints 2

In general, I think it's not a bad idea to use a new variable for each TensorFlow op to avoid bugs like these. Note you can write it in the same number of lines:
x_ = tf.placeholder(tf.float64, [5])
r = tf.where(tf.is_nan(x_), tf.zeros_like(x_), x_)

I'm gonna close this for now as I don't think it's a TensorFlow bug, but feel free to comment further.

I agree with your explanation of what I am doing (wrong choice of words on my end).
I also agree with your analogy. However, it's because it's analogous that I made this mistake.
I am not saying that it's a bug, but I think it can be a bit puzzling the first time you look at it, and that is why I was suggesting to throw a warning when reassigning a placeholder because it is going to lead to an unwanted behavior.
I also agree with your general comment on naming ops.

True, I agree it's not really a bug per-se. Even so, I don't think we should change the default assignment operator to print a warning (in fact, I don't think it's even possible to do so in Python). Thank you for reporting though, hopefully if others run into the same problem they will find this thread.

authors: skye zaccharieramzi


text:
ci_build updates

fix domain in pip metadata
ci_build: add user inside docker to sudo group
ci_build: detect python path within container

@jendap The files involved in this PR overlap with my PR https://github.com/tensorflow/tensorflow/pull/1051 and is smaller. I can include your changes in my PR to avoid potential conflict and make it easier to merge everything to r0.7. How does that sound to you?

Are all changes in this PR included in #1051? Because I don't see the sudo change in #1051.

@martinwicke I didn't include it in 1051 because I didn't hear back. Please merge this to master and cherry-pick it to r0.7 separately.

There are conflicts. I'll wait for jan to resolve.

Conflict resolved.

I manually pulled them to my fork of the release branch and tests of them on Jenkins has been okay so far. We can just merge them into r0.7 and then master when the release is done.

The coordinator test failure is unrelated?

@martinwicke That's unrelated. I forgot to add the TF_BUILD_SERIAL_TESTS=1 to the build configuration, which is required for GPU tests because concurrent running of the GPU tests is known to cause OOM errors. We can consider adding TF_BUILD_SERIAL_TESTS=1 as an automatic default for GPU to ci_parameterized_build.sh.

Included in #1096. Closing.

source: gharchive/pull-request

authors: caisq jendap martinwicke

repo: tensorflow/tensorflow

license_type: permissive

license_source: github-api


text:
Fix #27994: compat.as_bytes should support bytearray

I made these changes manually in a local venv and was then able to write to a GFile handle from a fastavro Writer, which writes out bytearrays.

@erwa please sign CLA

I'm speaking with my legal team and just need to figure out a few details before signing the CLA. Hope to get back to you later this week.

@googlebot I signed it!

@rthadur , I've signed the CLA. Could you help review the patch?

Thanks for the review, @mihaimaruseac. I took a look at some of the failed CI builds and the failures look unrelated. Can you merge this PR?

There's some infrastructure changes going on at the moment. Will merge when resolved.

Thanks, Mihai.

Can you solve conflict?

Rebased on master and resolved conflicts.

authors: erwa mihaimaruseac rthadur


text:
Update doc for categorical hinge

Hey, this is just a minor doc update.
Please feel free to suggest improvements.
Btw. what am I doing wrong that my older commits always appear when I create a new pull request?
I just create a new branch on my tf fork.

#29760


text:
Real system exit()

sys.exit(1) works  definitely

Is there a bug that this fixes?

@mihaimaruseac Sorry I don't get it completely your comment. You mean if there was a bug which causes this modification of replacing exit() by sys.exit() ?

I just saw that all python-exits in TensorFlow except for control_flow_grad.py - because it refers to its own exit-function - are real sys.exit() and not just exit().
For me, it makes sense because it is a safe exit as well as is more convenient for python2.7.

Sorry if it not fits

We no longer support python2.7 and would prefer one line fixes to be real bug fixes

So close the request better?

authors: Anselmoo mihaimaruseac


text:
[issue4339]Add adjust_gamma for image

Add "Gamma Correction" Image Adjustment issue4339 according to scikit-image adjust_gamma
Change list:

Add adjust_gamma in image_ops.py
Add corresponding tests in image_ops_test.py

test_adjust_gamma_one
test_adjust_gamma_zero
test_adjust_gamma_less_one
test_adjust_gamma_greater_one

Add dtype range and limits for numpy dtype

Compare to scikit-image:

adjust_gamma
Test Gamma Correction
dtype range
dtype limits

An explanation of Gamma Correction in Wikipedia

Can one of the admins verify this patch?

@girving @vrv Please check!
BTW, would we support the pow of uint8, uint16 and tf.int32 / tf.float32 in the future? ?

Vijay, please review or reassign if you know someone more appropriate.

@vrv  could u review the PR, or reassign others @jhseu

Hmm, the other good reviewers for this change are out-of-office right now and shlens will only have time to review probably next week. I'm hesitant to do the review because of my lack of context on image ops. Thanks for your patience.

Can one of the admins verify this patch?

A mistake of deleting branch. Restore it now

@jhseu @vrv @girving Can one of you review the PR?

@shlens friendly ping :)

Jenkins, test this please.

Jenkins, test this please.

Jenkins, test this please.

@shlens Do you know how to get the error log of Linux GPU FAILURE? I don't know why it fail, but I remember it succeeded last time. :(

Jenkins, test this please.

@shlens Does changes approved means that the PR can be merged now?

Jenkins, test this please.

authors: DjangoPeng drpngx jhseu shlens tensorflow-jenkins


text:
Fix duration computation in MicroProfiler.

Prior to this change, if the BeginEvent occurred at a tick that was larger than EndEvent, we would incorrectly compute the duration for that event.
The only requirement that we have is that no more than 2^32-1 ticks should have passed between a call to BeginEvent and EndEvent
The discussion on https://github.com/tensorflow/tensorflow/pull/49144 alerted us to this bug.
Note that we do not have any unit tests for MicroProfiler at this time.
Manually confirmed that the profiling info for this command:
make -f tensorflow/lite/micro/tools/make/Makefile TARGET=xtensa TARGET_ARCH=fusion_f1 OPTIMIZED_KERNEL_DIR=xtensa XTENSA_CORE=F1_190305_swupgrade run_keyword_benchmark -j8

is identical with and without the current change.

@njeffrie confirmed that this PR is unnecessary. If GetCurrentTimeTicks overflows as a two's complement then end-start will always give a positive number within the int32_t range.


text:
The following upstream commits caused a 15% performance degradation o…

…n the resnet50 model for ROCm:
4193058f98378f6f67db0cc67fb68332f8396d41: increase row tiling to 64
37ddd35d129280e80a847f09264118c7a0cbd894: enable tree reduction by default
The changes were "undone" for ROCm by guarding the previous implementation details with the TENSORFLOW_USE_ROCM macro.
The commit which increased row tiling to 64 made substantial changes to the reduce_unnested.hlo unit test to accomodate the change so the test fails on ROCm with this commit.
Therefore, the reduce_unnested.hlo unit test has been disabled.

@cheshire I am not familiar with how to measure an XLA kernel such as fusion_235 with and without tree reduction.  Please let me know how I can make similar measurements on ROCm.  Also, I am not familiar with what is meant by "67% of roofline".

"roofline" is the best-possible performance. This is determined by
bandwidth of the GPU (e.g. 650GB/s for Titan V), and the size of the
input/output buffers.
As an example, if we are reducing an array of size 650GB on Titan V, and
the operation took one second, we can't possibly do better (100% roofline).
If it takes 2 seconds, that's 50% roofline, etc.

I am not familiar with how to measure an XLA kernel such as fusion_235
with and without tree reduction
You can set env XLA_FLAG=--xla_gpu_deterministic_reductions=false ...

On Mon, Aug 30, 2021 at 2:43 PM rsanthanam-amd @.***>
wrote:

@cheshire https://github.com/cheshire I am not familiar with how to
measure an XLA kernel such as fusion_235 with and without tree reduction.
Please let me know how I can make similar measurements on ROCm. Also, I am
not familiar with what is meant by "67% of roofline".
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/pull/51703#issuecomment-908721250,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACVGH63UZP3ZQ2CES5UIBDT7P3QHANCNFSM5C4K3ERA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

@cheshire Sorry, I have should been more specific.  I am not sure how to measure the XLA kernel performance as you did (471ms vs. 476ms etc.).

Hi,
You could use https://www.tensorflow.org/tensorboard, where the trace view
should tell you how much time each kernel takes (the kernel name will
correspond to the fusion name).
Alternatively, given that you work for AMD, I'm sure that AMD has profilers
and tools to give you the trace view which would show you each kernel on
the timeline, and that you can get access to that expertise better than I
can.
On Tue, Aug 31, 2021 at 4:07 AM rsanthanam-amd @.***>
wrote:

@cheshire https://github.com/cheshire Sorry, I have should been more
specific. I am not sure how to measure the XLA kernel performance as you
did (471ms vs. 476ms etc.).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/pull/51703#issuecomment-909134865,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACVGHYAD67WEPRXGC3E36TT7SZXBANCNFSM5C4K3ERA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

@cheshire I measured a large difference in execution time for fusion_235 on ROCm:
| Configuration | fusion_235 execution time |
| tree reduction disabled and row tiling size = 16 | 4.3 ms |
| tree reduction enabled and row tiling size = 64 | 17.6 ms |

OK but could you debug on what is actually going on? Why is it slow? Is it spilling registers? How far are these numbers from theoretical roofline?

@cheshire I am currently in debugging mode.
By inspecting the HLO, I noticed that on CUDA, XLA uses NHWC for fp16 but on ROCm, it uses NCHW for fp16.
I'm wondering if at a high level, that could explain the performance difference on ROCm and the lack of a performance difference on CUDA.

What %% of the overall benchmark runtime is spent in convolutions?
On Wed, Sep 1, 2021 at 4:29 AM rsanthanam-amd @.***>
wrote:

@cheshire https://github.com/cheshire I am currently in debugging mode.
By inspecting the HLO, I noticed that on CUDA, XLA uses NHWC for fp16 but
on ROCm, it uses NCHW for fp16.
I'm wondering if at a high level, that could explain the performance
difference on ROCm and the lack of a performance difference on CUDA.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/pull/51703#issuecomment-910195483,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACVGH7QO527CGIM6TGJCATT7YFBLANCNFSM5C4K3ERA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

@cheshire
Tree reduction cracks a single reduction op into 2 reduction ops to avoid the need for atomics.
How does this guarantee deterministic output?

Because at most one block writes to one output location.
On Thu, Sep 2, 2021 at 5:53 AM rsanthanam-amd @.***>
wrote:

@cheshire https://github.com/cheshire
Tree reduction cracks a single reduction op into 2 reduction ops to avoid
the need for atomics.
How does this guarantee deterministic output?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/tensorflow/pull/51703#issuecomment-911647191,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACVGH5IV2I4IVK7PWNJ6I3T75XUTANCNFSM5C4K3ERA
.
Triage notifications on the go with GitHub Mobile for iOS
https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675
or Android
https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

@rsanthanam-amd Can you please resolve conflicts? Thanks!

@gbaned Done.

@gbaned This is not awaiting response.
@rsanthanam-amd Sorry for the confusion. Deterministic output is a requirement, we can't conditionally disable it for AMD. Do you think you could try to figure out where exactly the slowdown is coming from?

@cheshire I am still working on that.

authors: cheshire gbaned rsanthanam-amd


text:
Support read data from ViewFs.

Support read data from ViewFs.

Can one of the admins verify this patch?

@gunan @jhseu Our Hadoop cluster need this feature. thanks.

How is the viewfs cluster name being picked up? The change seems to ignore that component.

For example, how would you distinguish between:
viewfs://clusterx/path/to/foo vs. viewfs://clustery/path/to/foo

Jenkins, test this please

authors: jhseu tensorflow-jenkins wangyum


text:
[FEATURE] added op: nonMaxSuppression

FEATURE added op: nonMaxSuppression
I am looking to convert a mobilenet SSD model using the tfjs converter tool, but as pointed out in tensorflow/tfjs/issues/188, there are some OPs missing. As a starting point I added the op for Non-Maximum Suppression, which is used in some SSD models such as mobilenet SSD.
Now the question is, there are 3 versions of NonMaxSuppression in tensorflow.
v1: iouThreshold is assigned as an attr
v2: iouThreshold is now an input argument
v3: scoreThreshold is now an additional argument
This PR basically adds v3, how does tfjs deal with compatibility to models using other op versions? Is it desired to provide an op for every version?
Another question:
How does tfjs deal with node attributes?

This change is

Thank you for the contribution and apologies for the delayed review!

authors: dsmilkov justadudewhohacks

repo: tensorflow/tfjs-core

license_source: bigquery


text:
#11247: Remove in-place flag in binary operations

Ticket : #11247
Work Done:

Removed the in-place flags in binary operations

Checklist

[x] All Post commit CI
[ ] Model Perf Regression

Status: Will add Test unit test and then merge.

What is the current status of inplace now for binary op?

What I see:
new function ttnn.add_ and new tests for it.
What PR says: "remove inpalce flag in binaru operations"
@mouliraj-mcw  please always add in the description all new registered functions.
Also there should be added any changes to the api.
Unfortunately right now it is impossible to track changes.

What I see: new function ttnn.add_ and new tests for it. What PR says: "remove inpalce flag in binaru operations" @mouliraj-mcw please always add in the description all new registered functions. Also there should be added any changes to the api. Unfortunately right now it is impossible to track changes.

Hi @dmakoviichuk-tt .

The ttnn.add_ and other in-place operations were already available. I updated the structure by removing the in-place flag, and since it's an in-place version, I added the required unit tests.
Let me Update the Description Clearly. Thankyou.

authors: dmakoviichuk-tt mouliraj-mcw

repo: tenstorrent/tt-metal


text:
Add sweeps for unary_backward ops

Ticket
https://github.com/tenstorrent/tt-metal/issues/11512
Problem description
Missing sweeps for unary_backward ops:

clamp_bw
hardtanh_bw
mul_bw
softplus_bw
threshold_bw
div_bw

What's changed
Added aforementioned sweeps to sweep_framework and to ttnn-run-sweeps.yaml.
Modified parameters for div and clamp.
All nightly sweeps pass: https://github.com/tenstorrent/tt-metal/actions/runs/11231447592/workflow
Checklist

[x] ttnn sweeps CI passes
[x] Post commit CI passes
[x] Blackhole Post commit (if applicable)
[x] Model regression CI testing passes (if applicable)
[x] Device performance regression CI testing passes (if applicable)
[x] New/Existing tests provide coverage for changes

Please run these 6 tests in the sweeps pipeline before merging.

https://github.com/tenstorrent/tt-metal/actions/runs/11237054907 (clamp)
https://github.com/tenstorrent/tt-metal/actions/runs/11237048678 (div)
https://github.com/tenstorrent/tt-metal/actions/runs/11237035731 (threshold_bw)
https://github.com/tenstorrent/tt-metal/actions/runs/11237040369 (div_bw)
https://github.com/tenstorrent/tt-metal/actions/runs/11237031621 (softplus_bw)
https://github.com/tenstorrent/tt-metal/actions/runs/11237022442 (mul_bw)
https://github.com/tenstorrent/tt-metal/actions/runs/11237013960 (hardtanh_bw)
https://github.com/tenstorrent/tt-metal/actions/runs/11236985314 (clamp_bw)


text:
TT-Fabric Bringup Initial Check-in

Ticket
https://github.com/tenstorrent/tt-metal/issues/13844
https://github.com/tenstorrent/tt-metal/issues/13845
Problem description
Combined work for initial bringup of TT-FABRIC, by @aagarwalTT @ubcheema @aliuTT
What's changed
In first check-in of TT-FABRIC, we have only added new components and tests for it. There should be no functional changes to existing runtime/FD/ops etc.
Still need to rebase and run CI, but from top of my head there are some known issues:

@ubcheema currently, I've added the eth chan noc encoding map to the generated_bank_to_noc_coord_mapping.h, which is removed on latest main. Need to figure out where to load this. Commit d9f211596 needs special attention.
we should add the passing tests to the respective CIs
some commented out code, can be removed

Checklist

[ ] Post commit CI passes
[ ] Blackhole Post commit (if applicable)
[ ] Model regression CI testing passes (if applicable)
[ ] Device performance regression CI testing passes (if applicable)
[ ] (For models and ops writers) Full new models tests passes
[ ] New/Existing tests provide coverage for changes

@Allan, the eth channel to noc x/y mapping is fixed. We can add the mapping look up table to tt_faric header, instead of generating it on the fly.
What do you think?

authors: aliuTT ubcheema


text:
#9735: fix issues with including reflect library

Ticket
#9735
Problem description
When users include <reflect> library with files that are used in both ttnn_lib and tt_eager targets, there were compilation errors complaining that it wasn't found.
What's changed

Since we currently cross-link tt_eager target with ttnn_lib target, we needed to make the reflect::reflect library by changing the link dependency to PUBLIC
We also needed to update link dependency for tt_dnn target

Checklist

[ ] Post commit CI passes
[x] Model regression CI testing passes (if applicable)
[x] New/Existing tests provide coverage for changes

Thanks @vtangTT @TT-billteng for the help!


text:
Add export --internal to access metrics

This PR makes it so that we store pipeline metrics in the database. They are marked as "internal", which means that they are not exported by default. Instead, we can use export --internal to do so. The implementation uses a #internal meta extractor as a marker, but this extractor can only be accessed from code and not user-written pipelines.
It also fixes a bug where export --live | where <filter> discarded the filter.
Example
export --internal |
where source == true |
summarize bytes=sum(output.approx_bytes) by timestamp resolution 1s |
sort timestamp desc

{
  "timestamp": "2023-11-10T11:15:54.000000",
  "bytes": 79927223
}
{
  "timestamp": "2023-11-10T11:15:53.000000",
  "bytes": 81788928
}
{
  "timestamp": "2023-11-10T11:15:52.000000",
  "bytes": 80740352
}
{
  "timestamp": "2023-11-10T11:15:51.000000",
  "bytes": 75497472
}
{
  "timestamp": "2023-11-10T11:15:50.000000",
  "bytes": 75497472
}
{
  "timestamp": "2023-11-10T11:15:49.000000",
  "bytes": 76546048
}
{
  "timestamp": "2023-11-10T11:15:48.000000",
  "bytes": 78643200
}

Tasklist

[x] Where to tick import source in pipeline manager?
[ ] export --live --internal does not work yet
[ ] Write docs for metrics format.

I still have to write the docs for the actual metrics events, but besides that, this should be ready for review.

repo: tenzir/tenzir


text:
build: upgrade glam 0.25 → 0.27

Obsoletes #70.

I was just updating my PR.

Happy to close mine, since you got there first! 😊

authors: ErichDonGubler waywardmonkeys

repo: teoxoy/encase


text:
Release 5.3.0 including English ver #54

Please review.#54

発生している差分は、以下であることを確認しました。
・[英語版] 5.3.0（英語版）の追加
・[日本語版] ビルド環境が変更されたことが起因で生じたもの

authors: btkoremurakn itouyun


text:
Split lyapunov module into another crate

Lyapunov analysis is not a part of ode solver/

Lyapunov analysis (including CLV) are separated into examples/ directory

source: gharchive/issue

repo: termoshtt/ndarray-odeint

license: mit


text:
🛑 Ventas Inmediatas is down

In 065b159, Ventas Inmediatas (https://ventasinmediatas.transturin.com) was down:

HTTP code: 502
Response time: 392 ms

Resolved: Ventas Inmediatas is back up in ee9970d.

repo: terorero/monitor

license: MIT


text:
feat: Add logic and vars to opt out of acm / https listener if desired

Description
Add three variables, bool true by default so as not to conflict with existing expected functionality:
create_acm_certificate
create_https_listener
atlantis_url_ssl
Motivation and Context
There are scenarios where one may wish to have this be generated for testing and the requirement of an acm certificate and https listener is not necessary. For instance, security enforced by IP ingress restriction in a test/nonprod.
Of course, in a real deployment it makes sense to force that functionality
(hence leaving these things to bool true by default so that only a user who intends to not have ssl and knows what they're doing would toggle it to false using the variable)

When applying this, they'd need to add the following in order to get a no https LB listener
atlantis_url_ssl = false
create_acm_certificate=false
create_https_listener=false

Breaking Changes
None that I can see.
How Has This Been Tested?

[ x] I have updated at least one of the examples/* to demonstrate and validate my change(s)
[ x] I have tested and validated these changes using one or more of the provided examples/* projects

[ x] I have executed pre-commit run -a on my pull request

ran docker version
only issue was interpolation syntax, which I fixed in a commit

additionally, I guess they could all be one bool... just depends on how piecewise you'd want that functionality to be in the case of only setting some, but not all of these


text:
feat: Add cloudwatch_log_group_arn to outputs

PR o'clock
Description

Problem

Currently we cannot apply some resources which depends on eks module generated cloudwatch logs arn, like followings.

How to solve?

add output cloudwatch_log_group_arn

module "eks" {
  source = "terraform-aws-modules/eks/aws"
  ...
}

resource "aws_lambda_permission" "log" {
  ...
  source_arn    = module.eks.cloudwatch_log_group_arn // <- we cannot pass cloudwatch arn in the same terraform apply.
}

To work-around, we need to define data resource. But in this case, we need to execute terraform apply twice to get cloudwatch resource.
data "aws_cloudwatch_log_group" "log" {
  name = module.eks.cloudwatch_log_group_name[0]
}

Checklist

[x] CI tests are passing
[x] README.md has been updated after any changes to variables and outputs. See https://github.com/terraform-aws-modules/terraform-aws-eks/#doc-generation

ci failed on master branch, and fix it in https://github.com/terraform-aws-modules/terraform-aws-eks/pull/1073

Thanks @krrrr38 for your contribution.

than you to maintain this module 🙏

repo: terraform-aws-modules/terraform-aws-eks


text:
Can you import an existing VPN?

We use this module and also have manual assets, can I use this module with the older assets by importing them? I'm not sure how to map them?

I guess you have figured it out @runtman, but if you still struggle with importing the resources I would be happy to help you out.

Hey, yes I figured it out.
From memory it was something like
terraform import module.foo.aws_vpn_connection.default {CONNECTIONID}

Great 👍

authors: miguelaferreira runtman


text:
Question about logical operators

Sorry if my question is explained or resolved in any doc, I've trying to do an aws resource tag check and I've the following scenario

I've specific tags that should be set to governance and cost control purposes
The tags can be set in tag_all OR directly in the resources tags block

And here lies my problem or lack of understanding the tool :)
How to do the OR logic?
My scenarios are:
Feature: Governance and Cost Compliance Check
  Scenario Outline: Ensure that specific tags are defined
    Given I have resource that supports tags_all defined
    When it has tags_all
    Then it must contain tags_all
    Then it must contain "<tags>"
    And its value must not be null

    Examples:
      | tags                |
      | tag01              |
      | tag02              |

  Scenario Outline: Ensure that specific tags are defined
    Given I have resource that supports tags defined
    When it has tags
    Then it must contain tags
    Then it must contain "<tags>"
    And its value must not be null

    Examples:
      | tags                |
      | tag01              |
      | tag02              |

Ps.: the tag keys are just examples :)

Hi Ricardo,
You actually did the OR logic buy using When it has tags_all and When it has tags. Except using the same Scenario name as it must be unique per Scenario, all looks ok actually.
What is the problem you are having ?

hmm, is it possible to share plan output in json ? I couldn't still understand the exact problem :)

This is just a example I'm using localstack and creating a basic SQS
The pretty output:
  + resource "aws_sqs_queue" "terraform_queue" {
      + arn                               = (known after apply)
      + content_based_deduplication       = false
      + deduplication_scope               = (known after apply)
      + delay_seconds                     = 90
      + fifo_queue                        = false
      + fifo_throughput_limit             = (known after apply)
      + id                                = (known after apply)
      + kms_data_key_reuse_period_seconds = (known after apply)
      + max_message_size                  = 2048
      + message_retention_seconds         = 86400
      + name                              = "terraform-example-queue"
      + name_prefix                       = (known after apply)
      + policy                            = (known after apply)
      + receive_wait_time_seconds         = 10
      + tags                              = {
          + "custom1" = "meat"
          + "custom2" = "potato"
        }
      + tags_all                          = {
          + "custom1" = "meat"
          + "custom2" = "potato"
          + "tag1"    = "popcorn"
          + "tag2"    = "bacon"
          + "tag3"    = "cheese"
        }
      + url                               = (known after apply)
      + visibility_timeout_seconds        = 30
    }

This is what happen when I execute the tf_compliance:
terraform-compliance v1.3.32 initiated
. Converting terraform plan file.
> Features      : ~/doc/lab-terraform-compliance/
> Plan File     : ~/doc/lab-terraform-compliance/plan.out.json

> Running tests.

Feature: Governance and Cost Compliance Check  # ~/doc/lab-terraform-compliance/governance.feature

    Scenario Outline: Ensure that specific tags are defined in resources
        Given I have resource that supports tags defined
        When it has tags
        Then it must contain tags
        Then it must contain "<tags>"
        And its value must not be null

    Examples:
        | tags |
                Failure: aws_sqs_queue.terraform_queue (resource that supports tags) does not have tag1 property.
        | tag1 |
          Failure:
                Failure: aws_sqs_queue.terraform_queue (resource that supports tags) does not have tag2 property.
        | tag2 |
          Failure:
                Failure: aws_sqs_queue.terraform_queue (resource that supports tags) does not have tag3 property.
        | tag3 |
          Failure:

    Scenario Outline: Ensure that specific tags are defined in providers default tags
        Given I have resource that supports tags_all defined
        When it has tags_all
        Then it must contain tags_all
        Then it must contain "<tags>"
        And its value must not be null

    Examples:
        | tags |
        | tag1 |
        | tag2 |
        | tag3 |

1 features (0 passed, 1 failed)
6 scenarios (3 passed, 3 failed)
30 steps (24 passed, 3 failed, 3 skipped)
Run 1647970914 finished within a moment

Hi, It was a gigantic lack of knowledge of how tags and tags_all works in terraform plan.
So whatever I define in tags blocks inside a resource will automatically apears inside the tags_all key, but the inverse is not true.
If I define the Compliance tag at tag block, the tf-complice will check tags_all and give a OK!
It was explicit right here in this part of output:
          "tags":{
            "custom1":"meat",
            "custom2":"potato"
          },
          "tags_all":{
            "custom1":"meat",
            "custom2":"potato",
            "tag1":"popcorn",
            "tag2":"bacon",
            "tag3":"cheese"
          },

Thanks for the support and sorry about this inconvenience.

No worries at all, sorry just reading your message. I am glad you sorted it out. :tada:

authors: eerkunt ricardo-f

repo: terraform-compliance/cli


text:
chore(deps): update ci dependencies

This PR contains the following updates:

Package
Type
Update
Change

common-dev-assets

digest
d5bdf01 -> b015de0

github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper
require
patch
v1.25.4 -> v1.25.5

Release Notes

terraform-ibm-modules/ibmcloud-terratest-wrapper (github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper)
v1.25.5
Compare Source
Bug Fixes

deps: update gomod (#711) (9a2a01d)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline


text:
chore(deps): update ci dependencies

This PR contains the following updates:

Package
Type
Update
Change

common-dev-assets

digest
540953e -> e1289a9

github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper
require
patch
v1.26.1 -> v1.26.2

Release Notes

terraform-ibm-modules/ibmcloud-terratest-wrapper (github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper)
v1.26.2
Compare Source
Bug Fixes

deps: update module github.com/go-openapi/strfmt to v0.22.0 (#731) (4aa1e01)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

:tada: This PR is included in version 1.1.6 :tada:
The release is available on:

GitHub release
v1.1.6

Your semantic-release bot :package::rocket:


text:
chore(deps): update common-dev-assets digest to 0f1f11e

This PR contains the following updates:

Package
Update
Change

common-dev-assets
digest
ffeb38a -> 0f1f11e

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline


text:
chore(deps): update common-dev-assets digest to ba8da70

This PR contains the following updates:

Package
Update
Change

common-dev-assets
digest
9afd4ef -> ba8da70

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

:tada: This PR is included in version 2.1.0 :tada:
The release is available on:

GitHub release
v2.1.0

Your semantic-release bot :package::rocket:


text:
chore(deps): update ci dependencies

This PR contains the following updates:

Package
Type
Update
Change

common-dev-assets

digest
a958729 -> 20ea4bb

github.com/gruntwork-io/terratest
require
patch
v0.46.0 -> v0.46.1

github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper
require
patch
v1.23.8 -> v1.23.9

Release Notes

gruntwork-io/terratest (github.com/gruntwork-io/terratest)
v0.46.1
Compare Source

Description

Updated dependency golang.org/x/net from 0.8.0 to 0.17.0

Related links

https://github.com/gruntwork-io/terratest/pull/1356

terraform-ibm-modules/ibmcloud-terratest-wrapper (github.com/terraform-ibm-modules/ibmcloud-terratest-wrapper)
v1.23.9
Compare Source
Bug Fixes

deps: update gomod (#680) (f4dbcb9)

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

:tada: This PR is included in version 3.8.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


text:
chore(deps): update common-dev-assets digest to 939f4a6

This PR contains the following updates:

Package
Update
Change

common-dev-assets
digest
a08e843 -> 939f4a6

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

:tada: This PR is included in version 2.1.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


text:
chore(deps): update common-dev-assets digest to eec8f4a

This PR contains the following updates:

Package
Update
Change

common-dev-assets
digest
f16f7a2 -> eec8f4a

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, click this checkbox.

This PR has been generated by Renovate Bot.

:tada: This PR is included in version 1.0.2 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


text:
update the provider version

Description

https://github.com/terraform-ibm-modules/terraform-ibm-vpe-gateway/issues/472
Release required?

[ ] No release
[ ] Patch release (x.x.X)
[ ] Minor release (x.X.x)
[ ] Major release (X.x.x)

Release notes content

Run the pipeline
If the CI pipeline doesn't run when you create the PR, the PR requires a user with GitHub collaborators access to run the pipeline.
Run the CI pipeline when the PR is ready for review and you expect tests to pass. Add a comment to the PR with the following text:
/run pipeline

Checklist for reviewers

[ ] If relevant, a test for the change is included or updated with this PR.
[ ] If relevant, documentation for the change is included or updated with this PR.

For mergers

Use a conventional commit message to set the release level. Follow the guidelines.
Include information that users need to know about the PR in the commit message. The commit message becomes part of the GitHub release notes.
Use the Squash and merge option.

/run pipeline

:tada: This PR is included in version 4.1.1 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


text:
Support deploying some of the watson services in jp-tok region.

Description
Use case: some customers want to deploy some of the watson services in jp-tok region.
Issue: currently, the location field is set to allow only the regions that are supported by ALL watson service.
2 approaches:

In the location field - add all regions supported by at least one watson service. Add some static checks in the terraform logic to error out if the users selected to deploy a service in a region that is not supported
Or, add an optional region input for each watson service

The cleanest way is to use this feature https://github.com/hashicorp/terraform/issues/25609 at plan time. Unfortunatly terraform v1.9 is not yet available and we don't know when it will be avaliable in Schematics.
Let's do this with a check block https://developer.hashicorp.com/terraform/language/checks, do you agree?

@andreainnocenti - here is an example of the way it is typically done with current version of terraform. https://github.com/terraform-ibm-modules/terraform-ibm-base-ocp-vpc/blob/05e504dbd0e3057171616ec98bfb9f96d7d45813/main.tf#L22
check block assertion cannot be applied here as they won't block execution (just surface a warning)

Before going ahead and implement, we must first understand if the watsonx platform works with services deployed in different regions, and which ones can be deployed in different regions.

Service
Regions

Watson Machine Learning
jp-tok,eu-gb,eu-de,us-south

Watson Studio
jp-tok,eu-gb,eu-de,us-south

watsonx.governance
eu-de,us-south

watsonx Assistant
au-syd,jp-tok,eu-gb,eu-de,us-south,us-east

Watson Discovery
au-syd,jp-tok,eu-gb,eu-de,us-south,us-east

the final decision was to allow the regions jp-tok,eu-gb,eu-de,us-south and add a check on watsonx.governance
  lifecycle {
    precondition {
      condition     = contains(["eu-de", "us-south"], var.location)
      error_message = "Watwatsonx.governance is only available in eu-de and us-south region."
    }
  }

in this way the plan fails

authors: andreainnocenti mkrudele vburckhardt


text:
Redis caching system implementation

Tegola implementation for Redis caching system

Move this over to the tegola repo as issue #300.

authors: gdey peldhose

repo: terranodo/tegola-osm


text:
Adding the MacOS to CI

What is the change?
Adding the MacOS to CI.
Why is the change being made?
I recently found out that some of our users are Mac people.
Notes

I can not immediately get our unit tests working in Mac in parallel (something about our test temp directory logic breaks). So, this PR only runs the unit tests in series.
The Mac Unit Tests take 14 minutes to run. Is that too long?

Checklist

[X] This PR has only one purpose or idea.
[X] Tests have been added/updated to verify any new/changed code.

[X] The code style follows good practices.
[X] The commit message(s) follow good practices.

[X] The release notes have been updated if necessary.
[X] The documentation is still up-to-date in the doc folder.
[X] The dependencies are still up-to-date in pyproject.toml.

I've got an old macbook air from grad school lying around if you ever need more mac-specific testing.

I borrowed my wife's Mac. But I'm sure I'd love a second tester. Up to you!

repo: terrapower/armi


text:
Alphabetize entries in some source files

We have a handful of source files that have lists of tables or methods that aren't
trivially short but make sense to keep grouped together. They weren't ordered in a
way that made them easy to navigate or that made it obvious exactly where to add
new entries. Update them to sort the relevant parts alphabetically.
No functional changes here, just shuffling of names.

Current dependencies on/for this PR:

main

PR #546

PR #547

PR #548

PR #549

PR #550

PR #551

PR #552

PR #553

Graphite Merge Job
Current status: ⏳ Queued to merge
This pull request is currently queued to be merged as part of a stack.
This comment was auto-generated by Graphite.

Job Reference: xGqpbvjWJBhk791ZI7p8

Graphite Merge Job
Current status: ⏳ Queued to merge
This pull request is currently queued to be merged as part of a stack.
This comment was auto-generated by Graphite.

Job Reference: cjUYbgw8NcVK1ZvvabkK

repo: terraware/terraware-server


text:
SW-1951 Add seedling batch CSV upload API

Add a set of endpoints under /api/v1/nursery/batches/uploads to import seedling
batches from CSV files.
The API is similar to the CSV upload APIs for species and accessions, but in the
case of seedling batches, there are no warning conditions that require user
confirmation. There is thus no need for endpoints to resolve problems or cancel
an upload that is awaiting confirmation.

Current dependencies on/for this PR:

main

PR #761

PR #762   👈

This comment was auto-generated by Graphite.

This may need a slight change depending on the outcome of SW-2060 but we can do that as a followup if needed.

Graphite Merge Job
Current status: ⏳ Queued to merge
This pull request is currently queued to be merged as part of a stack.
This comment was auto-generated by Graphite.

Job Reference: eiHYvN0UCPVMZk8mLgrI


text:
SW-1890,1891 Notifications for Nursery Workflows

Generate notification events with daily task

create lookup method in BatchStore

Add event listeners

Listen to event and insert application notification
Add notification type to database
Listen to event and send emails

Create email templates

Local Testing (performed by running the service locally and setting the "daily" task cron to every 5 minutes):

"SW-1980,1981" won't be detected as a list of two issue numbers in the SW project by Jira's GitHub integration. You'd want to pick just one of the two to include in the title and put the other one (with SW- prefix) in the body of the PR description.

authors: charlie-tf sgrimm


text:
Symbolizer GUI not fully visible

The GUI for editing a Symbolizer is not fully visible. The lower part is cut off.
With zoom 100%:

With zoom 90% (see the prev. missing field Stroke-Opacity):

Tested with Firefox 61.

Looks like the footer overlaps the lower part.

The symbolizer editing should be moved to some kind of window ui anyway.

Solved with introduction of SymobilizerWindow.

authors: KaiVolland chrismayer jansule

repo: terrestris/geostyler


text:
发布到自建typecho不渲染公式，用markdown形式发布格式没有渲染。

我现在在做发布测试
用html格式发布效果：https://blog.xuezhao.space/index.php/archives/3/
用markdown格式发布效果：https://blog.xuezhao.space/index.php/archives/6/
两种情况公式都不渲染，用markdown发布的结果是源码。
这是我的配置：

还有一个错误：

1、typecho 公式问题，请参照此文章 https://blog.terwer.space/static/20240227143900-be07wt1 或者 https://blog.terwergreen.com/typecho-platform-configuration-guide-1htgmj.html 解决
2、发布格式暂时只能支持 html 模式。
3、错误可以先强制删除再试试，常规发布点击强制删除或者点击后面的强制接触关联。

authors: dengxuezhao terwer

repo: terwer/siyuan-plugin-publisher


text:
建议支持统一管理所有文章

每个文章都去引入这个挂件没有必要，相关配置统一管理就好

类似我写的这种使用方式，你的这个挂件适配的平台更多，但是也更复杂

统一管理已经实现，只是没有放开页面。
目前是浏览器插件版本会展示统一管理入口。挂件受限于使用方式所以这块屏蔽了。
关于统一有什么思路吗，挂件好像默认必须每个文章引用，单独的入口目前还不知道怎么支持。

类似下面的代码，挂件放在父文件，这个时候就可以拿到这个文件的 id，然后查询这个文件夹下的所有文档块就能拿到所有的文档了
https://github.com/mohuishou/siyuan-sync-github/blob/main/src/api/siyuan.ts#L90

类似下面的代码，挂件放在父文件，这个时候就可以拿到这个文件的 id，然后查询这个文件夹下的所有文档块就能拿到所有的文档了
https://github.com/mohuishou/siyuan-sync-github/blob/main/src/api/siyuan.ts#L90

感谢，我大致明白你的意思了，很好，我会在下版本加上这个功能。

authors: mohuishou terwer

repo: terwer/src-sy-post-publisher


text:
Ensure TesselSeeker.prototype.stop always returns the same value.

And a bunch of other tests

Looks good! Will merge.


text:
RFC: Use SPDX license identifier instead of long license plate and clean comments in file headers

From @egorpugin:
// SPDX-License-Identifier: Apache-2.0

https://spdx.org/licenses/
Linux uses them already.
https://github.com/torvalds/linux/blob/master/arch/x86/kernel/cpuid.c#L1

I created this issue from @egorpugin's  project list item to allow broader discussion.
Using SPDX license identifiers would reduce the source code lines by removing redundant information from the comments in the file headers (1 line instead of 10 lines).
We could start with the public header files in include/tesseract, use it for new files and then extend the changes to the remaining files as well.

I think it can be automated in case if we remove other info in headers (date, author etc.)

Note that the planning "Tesseract next" now includes additional columns for bug fixes and new features for release 5. If needed we could also create a separate project planning "Tesseract future" for release 6.

I think it can be automated in case if we remove other info in headers

Should such additional information be removed? Do we have to preserve it? I don't know the answer and would therefore start with removing the license text only.

Author probably should present in git history but it can't be true in all cases.
Copyright is used usually.
For example, I have it https://github.com/SoftwareNetwork/sw/blob/b0.4.3/src/sw/manager/database.cpp#L1

Author: ???

Didn't you remove such lines in the past?
IMO, Author and Copyright should be kept for legal reason.

Yes, I remove History and Created in files where I make other changes.

My point was that you removed developers names AFAIR. I hope all of them are at least mentioned in AUTHORS.
I think most if not all the removed named were HP employees. Maybe also some Googlers.

You are right, I did not remember that I removed authors in a few cases. I now found the 3 commits b5498c70fa0b171cb952e04c5d9176a09c70963b (new file content), d960a50c12c4b991d0a86ff4c1fb9b05fd580aae and 6d170a15ec7ca0950fc69734ed586ffe6465f9ca (uncertain author). So all of them where very special cases.

So do we agree to

replace the license text by SPDX license identifier
remove File:, History: and Created:
keep the rest of the header comments

?

IANAL, but IMO, It's not a good idea to remove people names from files. We don't have to credit contributors, but once a name appears in a file, it should not be removed without explicit permission from that person. Also, if a contributor wants to have a credit in AUTHORS we must give him that credit, unless his contribution is tiny like a one line typo fix.
https://en.wikipedia.org/wiki/Moral_rights

@stweil
Agreed.
We can revisit authors later.
On heavily modified code new authors should be added following current approach, but I don't like big list of authors in code.

I think these changes should go into next v6 (main) branch.
Let v5 keep old file headers.

https://lwn.net/Articles/739183/
https://www.linuxfoundation.org/blog/solving-license-compliance-at-the-source-adding-spdx-license-ids/
https://reuse.software/
https://spdx.dev/
https://github.com/torvalds/linux/blob/master/LICENSES/preferred/BSD-3-Clause (One example for a license).

Let v5 keep old file headers.

I never had noticed SPDX before your comment in the planning list, but now saw that it already used rather often. Obviously many users (mostly companies?) need some reliable and parsable license information. Therefore I think it would be good to add at least the SPDX-License-Identifier to the public header files like it is done in pull request #3689.

Sure.

authors: amitdo egorpugin stweil

repo: tesseract-ocr/tesseract


text:
contributing count_chars.py utility

As discussed in https://github.com/tesseract-ocr/tesstrain/pull/235#discussion_r1517037975_ .
Would a pull request from myself be sufficient here, or would it be preferable to add some language about this contribution being under the Apache-2.0 license?

Thank you! It would be great if you could add a statement about the license either as commit comment or in the header of the code.

Sounds good, I've updated the language to make the licensing here more explicit.

authors: cmroughan stweil

repo: tesseract-ocr/tesstrain


text:
Avoid loading preflight checks in web server mode.

This is a temporary workaround to allow running the app in web server mode when no kubeconfig file/env var has been provided.
For preflight, the clientsholder needs to be fully and correctly initialized in order to run the checks, otherwise it won't have any TestEnvironment to get the containers/operators under test.
For web server mode, there's no need to have a valid clientsholder yet, as the kubeconfig file is provided via web form, and it can be a different one on each run. The clientsholder is then created with the clientsholder.GetNewClientsHolder().
This workaround should probably be removed/refactored when PR #1684 is merged.

Closing PR in favour of https://github.com/test-network-function/cnf-certification-test/pull/1696


text:
Look at FreeCodeCamp: JS Test User Submitted Code

How does this button trigger a code run in the FreeCodeCamp site?
https://github.com/freeCodeCamp/freeCodeCamp/blob/main/client/src/templates/Challenges/components/tool-panel.tsx#L78-L82
      <Button
        aria-label='Run the tests use shortcut Ctrl+enter'
        block={true}
        bsStyle='primary'
        onClick={handleRunTests}
      >

https://www.freecodecamp.org/learn/javascript-algorithms-and-data-structures/basic-javascript/comment-your-javascript-code

Just curious how they do it. Is it secure?

repo: testbrew/testbrew


text:
Add support for WithCreateContainerParametersModifier

This PR adds a new builder method WithCreateContainerParametersModifier() which corresponds to withCreateContainerCmdModifier() from the Java implementation.
The purpose of this method is to gain access to low-level configuration on the Docker.DotNet client, for instance to configure Ulimits, without having to expose all of these low-level configurations through the testcontainer builder.
Merging this PR closes testcontainers#481.

I won't be able to do the review until the beginning of next week - sorry, but thanks already for your pull request and contribution.

@HofmeisterAn Yes of course I should add a test. I looked around for a short while but I could not determine the right location for it. It has been added now. I just need clarification on a couple of your comments now

I see your point with immutability. I am not that well versed in the dotnet
collection types, but how about an IReadOnlyList<T> or
IImmutableList<T>? Or do you think that gets too specific?
man. 4. jul. 2022 18.28 skrev Andre Hofmeister @.***>:

@.**** commented on this pull request.
In
src/Testcontainers/Configurations/Containers/ITestcontainersConfiguration.cs
https://github.com/testcontainers/testcontainers-dotnet/pull/503#discussion_r913138775
:

@@ -105,5 +106,14 @@ public interface ITestcontainersConfiguration : IDockerResourceConfiguration

 /// This callback will be executed after starting the container, but before executing the wait strategies.

 /// </remarks>

 Func<ITestcontainersContainer, CancellationToken, Task> StartupCallback { get; }

///

/// Get the modifier callbacks for .

///

///

/// These actions will be executed after the remaining Testcontainer configuration has been applied on the Docker

/// client, but just before creating the container.

///

IList<Action> ParameterModifiers { get; }

The correct collection type here is IList since it expresses exactly the
contract intended with ParameterModifiers
Immutable is a contract too, and this won't work well with IList.
But this is your project after all.
It's a community project. My personal preferences don't matter 😃. I
think both points are valid, there is just no type that covers both in
.NET. Usually, I try to reduce changes to a minimal, but let's stick to
IList and add a comment?
—
Reply to this email directly, view it on GitHub
https://github.com/testcontainers/testcontainers-dotnet/pull/503#discussion_r913138775,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFBM6ZWCYLNTSZ4ENMVIEYLVSMGKBANCNFSM52OZC4CQ
.
You are receiving this because you authored the thread.Message ID:
@.***
com>

I see your point with immutability. I am not that well versed in the dotnet collection types, but how about an IReadOnlyList<T> or IImmutableList<T>?

Me neither. I checked IImmutableList<T> too, but that's not part of netstandard and IReadOnlyCollection<T> is just an IEnumerable<T>. But I missed IReadOnlyList<T>. That sounds right 👏.

@HofmeisterAn There you go. Also added some unit tests for BuildConfiguration to clarify expectations of the different method implementations. That way we can catch it if someone breaks contract.
I have never used XUnit before, so pardon me if the class structure is not in line with best practices. Let me know if I should use a combination of the static outer class and sealed inner classes (I have seen that in other tests), and I can refactor it.

@HofmeisterAn There we go - squashed into a single commit. Thank you for the collaboration so far 🚀

authors: HofmeisterAn Xitric

repo: testcontainers/testcontainers-dotnet


text:
Java Debug - Handshake failed

I'm trying to debug an integration issue in my Java stack.
I've registered and exposed the debug port, but when I try and connect to from my IDE I get handshake failed - connection prematurely closed. The debugger is listening on 50900. I guess this is an ambassador problem as when I run it via docker run debug works fine. Connecting to an illegal port states connection refused
Are there any config options around proxying? I'm thinking that when I get around to testing my service to service comms over HTTP (the issue I'm trying to debug) that they will fail to find each other on their defined linked service configuration. But I guess that's a second issue :)
 new DockerComposeContainer(new File("src/test/resources/docker-compose.yml"))
                    .withExposedService("rabbit_1", 5672)
                    .withExposedService("fdengine_1", 8090)
                    .withExposedService("fdengine_1", 50900)
                    .withExposedService("fdsearch_1", 8091);

$ dps

CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS              PORTS                                                 NAMES
8e143ced1dbe        flockdata/fd-search           "java -jar /opt/flock"   8 minutes ago       Up 7 minutes        8081/tcp, 9201/tcp, 9301/tcp                          tbddtq_fdsearch_1
bf25816f9c9b        flockdata/fd-engine           "java -jar /opt/flock"   8 minutes ago       Up 7 minutes        7474/tcp, 8080/tcp, 50900/tcp                         tbddtq_fdengine_1
62442c970ac5        rabbitmq:3-management         "/docker-entrypoint.s"   8 minutes ago       Up 8 minutes        4369/tcp, 5671-5672/tcp, 15671-15672/tcp, 25672/tcp   tbddtq_rabbit_1
1e0dba8dd6cf        richnorth/ambassador:latest   "./ambassadorize"        8 minutes ago       Up 8 minutes        0.0.0.0:33196->5672/tcp                               tender_hopper
16ff57531cb3        richnorth/ambassador:latest   "./ambassadorize"        8 minutes ago       Up 8 minutes        0.0.0.0:33195->8090/tcp                               mad_stonebraker
c795a19c89a4        richnorth/ambassador:latest   "./ambassadorize"        8 minutes ago       Up 8 minutes        0.0.0.0:33194->8091/tcp                               desperate_hypatia
1bb21b567a82        richnorth/ambassador:latest   "./ambassadorize"        8 minutes ago       Up 8 minutes        0.0.0.0:33193->50900/tcp                              nauseous_kowalevski

Hi Mike,
Sorry, that sounds odd - I'll have to try and reproduce. I've not tried an ambassador container with a Java debugger.. yet. Just to check:

this is regular Java debugger JDWP - correct?
could you provide the relevant bit of the server-side JVM config properties so that I can copy exactly? e.g. I'm guessing it's something like -agentlib:jdwp=transport=dt_socket,server=y,address=50900,suspend=n
could you also provide the connection parameters you're using from the debugger side (e.g. IDE settings or transcript of using jdb to try and connect)?
finally, are you using docker directly on a linux machine, or docker-machine on OS X, etc?

The ambassador container is a pretty basic HAProxy instance (source) which just sets up a TCP proxy for a single port. The container logs for the ambassador container might reveal some useful information on the TCP session events.
The only thing that I can think of right now is that the address property in the JDWP settings may need to match those that the client is using to connect. I vaguely recall seeing that in distant (pre-testcontainers) days. That's possibly not the cause though, so if not this will take a bit more digging...
Richard

Groan! Command args being passed via docker were not in the correct order (despite working when executed from the a $prompt).
Thanks for looking over my shoulder. FWIW, this works
CMD ["java",\
     "-Xdebug","-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=50900",\
     "-jar",\
     "/opt/flockdata/engine.jar",\
     "${JAVA_OPTS}"]

PBKAC

i use
ENTRYPOINT ["java", "-Xdebug", "-Xrunjdwp:server=y,transport=dt_socket,suspend=n,address=50900", "-Djava.security.egd=file:/dev/./urandom", "-jar","/app.jar", "${JAVA_OPTS}"]
and have the same problem..

or: ENTRYPOINT ["java", "-Xdebug", "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=50900", "-Djava.security.egd=file:/dev/./urandom", "-jar","/app.jar", "${JAVA_OPTS}"]

I switched to using docker-compose.yml and use env vars instead.
_JAVA_OPTIONS: "-Xms256m -Xmx512m -Xdebug -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=61000"
My compose script

authors: Sergey80 monowai rnorth

repo: testcontainers/testcontainers-java


text:
Incorrect Auth encoding

I think there is potential bug when parsing/decoding a custom auth config (e.g. for gcr).
We have create a config.json which looks like this:
{
  "auths": {
    "eu.gcr.io": {
      "auth": "BASE64-ENCODED-JSON"
    }
  }
}

The decoded auth contains a json key file:
_json_key:{
  "type": "service_account",
  "project_id": "XXX",
  "private_key_id": "XXX",
  "private_key": "XXX",
  "client_email": "XXX",
  "client_id": "XXX",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/...",
  "client_x509_cert_url": "https://www.googleapis.com/..."
}⏎

In RegistryAuthLocator#findExistingAuthConfig the decoded string is splitted and checked if there are two parts (username & password). This fails for the above json, such that you are not able to pull from a private gcr registry.
Docker java has similar logic, but limits the amount of splits to two, which results in having _json_key as username and the complete json key file definition as password. I have implemented the same splitting in testcontainers and tested it in our environment (using gitlab ci with dind) and pulling from our private gcr registry works just fine.
If you are okay with that change I can provide a PR for that.

Hi @atomfrede, thanks for raising this - seems worth fixing. If you could please raise a PR I'd really appreciate it!
Thanks

Sure will do.

authors: atomfrede rnorth


text:
Organisation? Roadmap?

I've started working on a version of this for .NET. Would be good to start to put together documentation around what TC delivers right now and then future work as well.
Time to turn it into a github organisation? @rnorth

Yup, cool! Will try and set this up over the weekend.

OK, this is in progress:

Github org is set up: https://github.com/testcontainers/
testcontainers.org is purchased and set up; basic DNS and mail, no website. We need this at a minimum to use the org.testcontainers prefix in maven central.
will migrate this project over to the organisation next

Please note that I've switched ownership of the repo and renamed it to testcontainers-java. New URL is: https://github.com/testcontainers/testcontainers-java

How about testcontainers.io ?

Happy to go half way with the costs in regards to all of this btw :)

Cheers! If you want to chip in a :beer: sometime would be more than generous enough ;)
I went for testcontainers.org basically because it was readily available, plus org.* is more common in java package naming than io.* so looks less jarring.
Next steps:

rename all the source packages and POM groupIds
release via bintray and maven central (probably v0.9.6)
update docs and add roadmap
when you're ready, set up a testcontainers-dotnet (name?) repo under the org

authors: rnorth swissarmykirpan


text:
feat: expose watchdog feature of testcontainers

Right now you have to do the following in your Cargo.toml to activate the watchdog feature from the testcontainers crate:
[dev-dependencies]
testcontainers = { version = "0.20", features = ["watchdog"] }
testcontainers-modules = "0.8"

I think it would be preferable to be able to do this:
[dev-dependencies]
testcontainers-modules = { version = "0.8", features = ["watchdog"] }

Thank you for the contribution! 👍
Totally makes sense to expose the feature


text:
some example show calling wasm written with go office

wasm source example with go office.
tests call go office wasm binnay with wazero.

Hi, and sorry we missed this. Is this the example you are interested in? https://github.com/wuhuizuo/go-wasm-go
It seems you want to have an example where wazero loads a wasm binary that calls a go function. Correct?
This example calls a host function: https://github.com/tetratelabs/wazero/blob/main/examples/host_func_test.go#L27
Another more simpler is this, but note the text format is incomplete, so changing the source will not likely work, yet. See #59 for progress: https://github.com/tetratelabs/wazero/blob/main/examples/simple_test.go

嗨，很抱歉我们错过了这个。这是您感兴趣的示例吗？https://github.com/wuhuizuo/go-wasm-go
似乎您想要一个示例，其中 wazero 加载调用 go 函数的 wasm 二进制文件。正确的？
这个例子调用了一个主机函数：https ://github.com/tetratelabs/wazero/blob/main/examples/host_func_test.go#L27
另一个更简单的方法是，但请注意文本格式不完整，因此更改源不太可能起作用。进度见#59 ： https ://github.com/tetratelabs/wazero/blob/main/examples/simple_test.go

I mean wazero load WASM file which was compiled with office golang compile, and call the functions.

Hi, and sorry we missed this. Is this the example you are interested in? https://github.com/wuhuizuo/go-wasm-go
It seems you want to have an example where wazero loads a wasm binary that calls a go function. Correct?
This example calls a host function: https://github.com/tetratelabs/wazero/blob/main/examples/host_func_test.go#L27
Another more simpler is this, but note the text format is incomplete, so changing the source will not likely work, yet. See #59 for progress: https://github.com/tetratelabs/wazero/blob/main/examples/simple_test.go

I mean wazero load WASM file which was compiled with office golang compile, and call the functions.

maybe you can look at this which builds on @pkedy's work to use wazero in wapc-go. It is a fairly complete usage pattern including memory https://github.com/wapc/wapc-go/pull/15
I'm closing the issue because I hope this answered the question!

authors: codefromthecrypt wuhuizuo

repo: tetratelabs/wazero


text:
feat(org): switch docs to be community

just a first pass of the basics to migrate this to textileio/community

oop, i didn't mean to re-request your review @carsonfarmer! thanks for the first one :D

repo: textileio/docs


text:
Completely remove SendFil

Had previously removed it from some apis, but not all. Now it's gone gone.

closing in favor of https://github.com/textileio/powergate/pull/684

repo: textileio/powergate


text:
Proposal: preset and config to be extendable

Summary
Make textlint preset and config more casual.
Purpose
textlint has three abstruction of collection rules.

Plugin(not recommented that use as preset)
Config - https://github.com/textlint/textlint/blob/master/docs/configuring.md#sharable-configuration
Preset - https://github.com/textlint/textlint/blob/master/docs/rule-preset.md

Config is similar with Preset.
My implementation of these:

Config is come from ESLint.

Config is a shareable .textlintrc
It has rules and rulesConfig
The use create own Config.

Preset is come from Babel.

Preset is a collection of rules
It has rules and rulesConfig
The developer create own Preset of rules.

JSCS has same terms, but Preset is a alias of Config.

Creating a Custom Preset or Config · jscs-dev/node-jscs Wiki

Question: How to use multiple presets and configs?
Currently, The user can use multiple presets like that
{
    "rules": {
        "preset-a": true,
        "preset-b": true
    }
}

But, it has a issue about "duplicated errors".
I exaplain this issue in #209
On the other hand, The use can't use multiple config.
textlint only support --config option.
textlint should provide a solution for these issues:

duplicated of rules in preset
config is not overrided by user

My idea:
textlit will support "extends" like ESLint and resolve these issue on the above.

Extending Configuration Files

Images of textlintrc
Pattern A that same with ESLint
{
    // preset override config
    "extends": [
        "config",
        "preset"
    ],

    // rules override extends
    "rules": {
        // Override any settings from the "parent" configuration
        "eqeqeq": "warn"
    }
}

Pattern B: rules allow to set Array
"rules": ["config", "preset", {

}]

Pattern C: programatic
"rules": Object.assign({}, config, preset, {

})

We need to other idea!

extends will be hard to maintain.
ESLint describESLint describe how extends is difficult.
e how extends is difficult.

ESLint's new config system, Part 1: Background - ESLint - Pluggable JavaScript Linter

We need to visit again in the future.
Current, you can use .texltintrc.js as  escape hatch.

repo: textlint/textlint


text:
rc-loader alternative?

We want to get an alternative of MoOx/rc-loader.
rc module is fragile (I think)

for...in
https://github.com/dominictarr/rc/blob/d6ac3a3ac1555b25964e963fb949cd1d30b32c0f/lib/utils.js#L24

RangeError: Maximum call stack size exceeded ?
Alternative

spalger/rcloader

Related #87

Other motivation: improve error message.
Current
{
  "rules": {
    "max-ten": {
      "max": 3
    },
    "spellcheck-tech-word": true,
    "no-mix-dearu-desumasu": true,
    "spellcheck-tech-word": true
  }
}

Load avove file via rc-loader, and throw following error.
/Users/azu/.ghq/github.com/KengoTODA/what-is-maven/node_modules/rc-loader/dist/parser.js:31
    throw new Error("Cannot parse runtime configuration as YAML, JSON or JS" + (file ? "from '" + file + "'" : ""));
    ^

Error: Cannot parse runtime configuration as YAML, JSON or JS

What is meaning?

Is https://github.com/davidtheclark/cosmiconfig better?

cosmiconfig is always async.


text:
2021/7/16 放送予定内容

[ ] connpassサイトを作る
[ ] 以下コメントで放送予定のURLを貼っていく

Getting started with Kafka Connector for Azure Cosmos DB using Docker

Welcome to the Engineering@Microsoft Blog

Separating the signal from the noise

Symbol loading improvements to PIX Timing Captures

New File IO-related counters in the Timing Capture Metrics View

Code Scanning C++ with GitHub Actions

Azure Form Recognizer now processes IDs in Azure Goverment

Java on Visual Studio Code Update - June 2021

API versioning extension with ASP.NET Core OData 8

Up & Running w/ OData in ASP.NET 6

.NET July 2021 Updates - 5.0.8 and 3.1.17

Azure DevOps Server 2020.1.1 RC now available

Visual Studio 2022 Preview 2 is out!

Using Cosmos DB as an ASP.NET session state and caching provider

Announcing .NET MAUI Preview 6

Announcing Entity Framework Core 6.0 Preview 6: Configure Conventions

ASP.NET Core updates in .NET 6 Preview 6

Windows Terminal Preview 1.10 Release

.NET Framework July 2021 Security and Quality Rollup

Build and Debug C++ with WSL 2 Distributions and Visual Studio 2022

repo: tfsugjp/devblogradio


text:
Allow admin-instance resource config

See https://github.com/tfwright/live_admin/pull/33
Problems:

right now default config must be defined at the app level, which makes it awkward to use a different set of default configs for different admin UIs (if more than 1 is desired for whatever reason).

since overrides are likely to be desired for most resources, the router config is large/noisy (see this thread)

Proposed solution: deprecate app level default config, and add support for passing a module, which specifies all configuration, including resources:
defmodule MyApp.AdminA do
  use LiveAdmin,
    ecto_repo: MyApp.RepoA,
    create_with: {MyApp.AdminContext, :default_create, []},
    update_with: :schema_function_name,
    resources: :get_resources

    def get_resources, do: [MyApp.SchemaA]
end

defmodule MyApp.AdminB do
  use LiveAdmin,
    ecto_repo: MyApp.RepoB,
    resources: [MyApp.SchemaB]
end

Alternatives:

extract resource level config into separate macro that would be called as part of a block, similar to how Phoenix routes work:

live_admin "/admin_url", title: "MyAdmin" do
  admin_resource "/a", MyApp.SchemaA, create_with: :create
  admin_resource "/b", MyApp.SchemaB, create_with: :something
end

It might be that these are two separate problems with config that need to be addressed separately.

@tfwright - thanks a lot for your work here. Just discovered live_admin and it fit our needs perfectly. Specifically, I stumbled upon this issue when looking for multi-repo support. However, I realised, accidentally, that this is already supported out of the box.
I am documenting our use case and how we got it working so someone else looking for the same thing doesn't spend too long on this.
We have 3 apps each with a separate repo. Let's call them AppA.Repo, AppB.Repo and AppC.Repo. Of these, AppA is the parent app to which AppB and AppC are dependencies.
To get live_admin working on all of these with their schemas, all we had to do was the following:
In AppA's config.exs
config :app_a,
  ecto_repos: [AppC.Repo, AppB.Repo, AppA.Repo]

The reason AppA.Repo is listed last is because some of its migrations depend on AppB and AppC and this way migrations run in the correct order when you run mix ecto.migrate.
Then, in AppAWeb's router.ex,
live_admin("/admin",
     resources: [
       AppA.SomeSchema,
       AppB.SomeSchema,
       AppC.SomeSchema
     ]
)

This works because (and I didn't know this previously), AppA.Repo.all(%AppB.SomeSchema{}) just works as long as AppB.Repo is included in ecto_repos for AppA.
Ignore if this is known behaviour but it certainly surprised me and Kaffy's autodetection only picks up AppA schemas.

Definitely not expected behavior! I haven't tried to use multiple Ecto repos myself, so I'm not sure, but LiveAdmin currently executes all queries in the context of whatever repo is set in its config. I am surprised that it would work with any other repos.

Here's a simple example that demos this behaviour: https://github.com/tejpochiraju/multi_repo_live_admin

Hm, just a guess but I think the key implementation detail there is that the same db is being used. I think that means that even though LiveAdmin is still using the "wrong" repo when querying whatever schemas are "supposed" to be on the other repo, since the db is the same the queries end up being directed to the same place. My expectation would be if you alter the AppB.Repo config to point at a different file with the AppB schema tables, and remove those tables from AppA.Repo file, then it will no longer work.

You are right. With separate DBs (SQLite3 or Postgres), I get the following error:
[error] GenServer #PID<0.851.0> terminating
** (Postgrex.Error) ERROR 42P01 (undefined_table) relation "users" does not exist

Nevertheless, this is still useful behaviour if possibly undocumented on the Ecto side of things (I can't seem to find a reference). Before I stumbled upon this approach, I was considering my options.

You said it was "awkward" to configure multiple admin UIs. Does this mean it's possible, how would one go about this?
Your suggested approach of passing an MFA during the router-level config would work.
Replace calls to repo/0 with a new repo/1 that infers the right Repo from the resource being queried. This seemed the simplest to me. What are the pitfalls here?

authors: tejpochiraju tfwright


text:
Issue with implementing filtering on nested collection

Hi there,
I'm having an issue with trying to implement a filter on a nested collection. I have the following ViewModel set up:
`public class MailoutViewModel
{
private int MailoutTotal => Mailouts.Count();
private int ResponseTotal => Responses.Count();
    public int ParticipantId { get; set; }
    public bool HaveAnyMailouts => MailoutTotal > 0;
    public bool HaveAnyResponses => ResponseTotal > 0;
    public int NoOfDaysSinceActive
    {
        get
        {
            var active = Responses.FirstOrDefault(x => x.Type.Code == "Y2");
            if (active != null)
            {
                return (int)(DateTime.Now - active.DataReceived).TotalDays;
            }
            return 0;
        }
}

    public IEnumerable<MailComm> Mailouts { get; set; }
    public IEnumerable<Response> Responses { get; set; }
    public IEnumerable<CustomItem> CustomItems { get; set; }
    public IEnumerable<ConsentItem> ConsentItems { get; set; }

}`
I'm using this so that I can query several tables to check if a participant has a mailout due e.g. if they have a particular response but no letter has been sent then we can send them a specific letter.
I've implemented as such:
`
var searchFilterTest = new QueryBuilderFilterRule()
{
Condition = "and",
Rules = new List()
{
new QueryBuilderFilterRule()
{
Condition = "and",
Field = "HaveAnyMailouts",
Id = "HaveAnyMailouts",
Operator = "equal",
Type = "boolean",
Value = new [] { "true" }
},
new QueryBuilderFilterRule()
{
Condition = "and",
Field = "HaveAnyResponses",
Id = "HaveAnyResponses",
Operator = "equal",
Type = "boolean",
Value = new [] { "true" }
},
new QueryBuilderFilterRule()
{
Condition = "and",
Id = "Responses.Ambiguous",
Field = "Responses.Ambiguous",
Input = "NA",
Operator = "equal",
Type = "boolean",
Value = new[] {"false"}
}
}
};
        var mailouts = _participantManager.GetAll().Where(x => x.Responses.Any()).Select(x => new MailoutViewModel()
        {
            ParticipantId = x.ParticipantId,
            Mailouts = x.Mailouts,
            Responses = x.Responses,
            CustomItems = x.CustomItems,
            ConsentItems = x.Consent
        });

var result = mailouts.BuildQuery(searchFilterTest).ToList();
`
However, upon building the query, I'm getting a System.ArgumentNullException: 'Value cannot be null.
Parameter name: property'
Here is my Response class:
`public class Response : BaseEntity
{
public Response()
{
ResponseItems = new List();
}
    [Key, DatabaseGenerated(DatabaseGeneratedOption.Identity)]
    public int ResponseId { get; set; }
    public int ParticipantId { get; set; }
    public int TypeId { get; set; }
    [Index]
    public DateTime DataReceived { get; set; }
    [Index]
    public bool Ambiguous { get; set; }

    #region Navigation properties

    public virtual Participant Participant { get; set; }
    public virtual ResponseType Type { get; set; }
    public virtual OptOut OptOut { get; set; }
    public virtual ICollection<ResponseItem> ResponseItems { get; set; }
    #endregion
}`

I've been following your example within your Unit Test but it doesn't seem to work (also it's worth noting that in your unit test, you are using FilterRule which has been marked as deprecated!).
Can someone offer any advice please?

I think I've finally solved it...
In my MailoutViewModel, I was using IEnumerable instead of List. Because of using an interface, it could not find the property within the list.
Thanks anyway :-) this is a great library and I'm looking forward to using it in my project.

In my MailoutViewModel, I was using IEnumerable instead of List. Because of using an interface, it could not find the property within the list.

Thanks @harpsicord86 , that was my issue as well! Thank you for commenting your solution and not leave the thread without an answer

repo: tghamm/dynamic-linq-query-builder


text:
js friendly queries

Hi to everybody.
I don't know if this has discussed before,  but i want to know if this is possible.
I write an angularjs app, and i want to pass some filters as js object on post, and make a layer to tranform these filters to a bookshelf query.
In sequelize i was before, this was easy because it has a mongodb like query system , and just passing an object its easy to build a query.
How can i do something like that on bookshelf?
For example, in sequelize you can do things like this:
{
  where: {
    somenumber: { $gt: 42 }
    $or: [
     { someOthernumber: { $in: [1,2 ] },
     { name: { $like: 'jan%' }
   ]
  }
}

and its easy to build.
I dont know if im clear enough, if you want to know more please tell me.
Thanks a lot.

There is also a "where" function in bookshelf.
On Fri, Feb 19, 2016, 6:49 AM Antony Agrios notifications@github.com
wrote:

Hi to everybody.
I don't know if this has discussed before, but i want to know if this is
possible.
I write an angularjs app, and i want to pass some filters as js object on
post, and make a layer to tranform these filters to a bookshelf query.
In sequelize i was before, this was easy because it has a mongodb like
query system , and just passing an object its easy to build a query.
How can i do something like that on bookshelf?
For example. where in sequelize is something like this:
where:{
something: value
}
and its easy to build.
I dont know if im clear enough, if you want to know more please tell me.
Thanks a lot.
—
Reply to this email directly or view it on GitHub
https://github.com/tgriesser/bookshelf/issues/1134.

No doc? How to use the where function?

??? Look at the website
On Fri, Feb 19, 2016, 2:50 PM YangLe notifications@github.com wrote:

No doc? How to use the where function?
—
Reply to this email directly or view it on GitHub
https://github.com/tgriesser/bookshelf/issues/1134#issuecomment-186040271
.

Oh, sorry. I was responding via email and the message only included this example (maybe you edited your original?):
where:{
  something: value
}

Look at the knex.js docs for info on this, but you can do:
{
  where: {
    somenumber: { $gt: 42 }
    $or: [
     { someOthernumber: { $in: [1,2 ] },
     { name: { $like: 'jan%' }
   ]
  }
}

Model.query(function(query) {
  query
    .where('somenumber', '>' 42)
    .orWhereIn('someOtherNumber', [1, 2])
    .or('name', 'like', 'jan%')
}).fetch().then(models =>

The actual question is how i can implemet this.I mean its easier to loop my filters and graduaĺly build an object and then return it from a helper function. Has anyone has an idea how i can abstract things in bookshelf?I dont want to have all my logic in a function.

Please close this issue, as i see i can pass an object to query function, and i think this is what i was looking for.
Thank you all for your answers!

authors: kozer rhys-vdw smilingpoplar

repo: tgriesser/bookshelf


text:
Default pool2 configuration often leads to errors

I recently ran into hair-pulling errors using knex with google cloud SQL. The issue was that GCS was closing idle connections in the pool, however by default pool2 was not testing the connections effectively before trying to use them.
There are two possible solutions, and I would suggest the default config does one of the following:
Option 1 - don't keep idle connections around.
  client: 'mysql',
  pool: {
    min: 0
  }
  ...
}

Option 2 - effectively test connections (may be database specific)
  client: 'mysql',
  pool: {
    min: 2,
    max: 10,
    ping: function (conn, cb) { conn.query('SELECT 1', cb); }  }
  ...
}

@acgourley I was also pulling my hair out over this issue but luckily I found this issue.  Thanks for the tips (and hopefully knex can provide better defaults to handle this).

Would be great if you could include the errors you were getting on this issue so that other people with the same problem can find this easily by searching for the error message :grin:

@acgourley  Thank you very much

The main reason I didn't include errors is that they are vague - basically node complains about ECONNRESET but that could have many other causes. I'll add it to the title, though!

Thanks for posting this. Very helpful! Curious if you've had any issues with the first request being slow? I tried both of the configurations above and while I'm not seeing the ECONNRESET anymore, after an idle period, the first request to the server is slow -- subsequent requests are fast.

@jamesdixon - are you using a first generation on demand  instance?  If so the instance might be shutting down so the slowness is in it starting back up.

@seansullivan I'm actually on Azure. The application runs on an Azure App Service (formerly Azure Websites) and the database sits on an Azure VM. The App Service is set to "Always On," which should prevent it from being suspended and then having to initialize when a request comes in. However, I experienced the same issue when using Amazon RDS, so I'm not sure if there is something I need to configure within Postgres itself.

I haven't really noticed it being slow at first. I'm not sure why that would even be the case, once you set the pooling to 0 it should be making a new connection every time and so the client should be doing the same thing on call 1 as call 2. If you quickly bounce your client app, is the first request still slow? If not, it's probably the server.

@acgourley I switched pooling back to 0 and the issue is still there, but further investigation reveals that it's something to do with Azure.
On another note, are you successfully able to use pooling or do you have it off as well? If the latter, has it affected your application performance at all?

If pooling works for you, and if your host allows you to keep connections
open a while, great. The concern would be if your host closes connections
quickly anyway which effectively turns makes queries even more expensive as
they first test the connection, realize it's down, and then have to make a
new connection anyway. If you have control over your database server, you
can probably set things up to support this well. If you're using a hosted
SQL solution with minimum control, maybe a pool of zero is better. I
personally am using Google Cloud SQL and have kept the pool at 0.
On Sun, Feb 7, 2016 at 10:12 PM, James Dixon notifications@github.com
wrote:

@acgourley https://github.com/acgourley I switched pooling back to 0
and the issue is still there, but further investigation reveals that it's
something to do with Azure.
On another note, are you successfully able to use pooling or do you have
it off as well? If the latter, has it affected your application performance
at all?
—
Reply to this email directly or view it on GitHub
https://github.com/tgriesser/knex/issues/975#issuecomment-181213296.

Thanks for the feedback, @acgourley. I'm actually using my own Postgres instance hosted on an Azure VM. That said, it seems that whenever I attempt to use pooling, I run into the ECONNRESET error. I believe I'll be sticking with a pool of 0 until I can get that resolved.

I think the fixes in 0.12 should help with this, please open another ticket if you have any errors.

@jamesdixon did you ever resolve this? I'm having a similar issue "too many connections" with knex in azure

I'm on 0.14.6 and am experiencing this issue on Google Cloud SQL (Postgres).
Here's a screenshot from my error logs.

I'm curious if the Transaction query already complete error is related? I can't reproduce locally, no matter how long I run it.

authors: ErisDS acgourley jamesdixon jeffijoe jimmanu smlynch techniq tgriesser vivekduddempudi

repo: tgriesser/knex


text:
Add test for enum on mssql

Mssql does not have a enum data type. We can use check constraints.

@caiquecastro Emmm, adjusted test is failing now. Were you planning to adjust mssql implementation as well?

Yeah! I'm checking how can i do it. Does the function receive any parameters about the column name?

authors: caiquecastro kibertoad


text:
⚠ DDoS-Relay EU-C-nbg1-4 has degraded performance

In 02b6e48, DDoS-Relay EU-C-nbg1-4 (http://128.140.77.59:42069/;csv) experienced degraded performance:

HTTP code: 200
Response time: 221 ms

Resolved: DDoS-Relay EU-C-nbg1-4 performance has improved in ca56611 after 19 minutes.

repo: tgstation-operations/status


text:
Receiver: improve uniformity of load distribution across replicas

https://github.com/thanos-io/thanos/issues/3488 describes an issue from a Thanos user that we have also seen in production at Red Hat. At its core, the load distribution for the ingestion of series in a Thanos receive hashring is not guaranteed to be uniform, and in some cases can be very uneven.
Background
The fact that one replica sees consistently higher load than the others is most likely due to some inherent lumpiness in the time series being sent. Thanos decides which replica should ingest data by hashing the name and label-value pairs of a time series and picking a corresponding replica from the ring to handle this hash. It seems that the data being sent simply has more time series that hash to one replica. There is no guarantee that data will be distributed uniformly across replicas, however, for a good hash function, the greater the number of time series and the more random the names and labels of those time series, statistically, we should coverage towards a uniform distribution.
Example
Unfortunately, there is no bug here. Take for example the case where a Prometheus server only produces a single time series and remote writes a million samples/second. We would rightfully expect super high load on a single hashring replica. This is essentially an extreme version of the case we see here.
Proposal
I suggest we improve the randomness of replica selection and thus drive load distribution towards uniformity even in the degenerate single-time-series case. One way to achieve this would be by adding a random value to the hash that changes for every request. It would be important that replicas forward this metadata along with the request so that once it is set, the random value is fixed and replicas can agree on who should ultimately ingest a sample. This would help solve load distribution for lumpy data.
WDTY @thanos-io/thanos-maintainers
Note: this proposal is adapted from my comment in https://github.com/thanos-io/thanos/issues/3488#issuecomment-778141503

I like it, but would be nice to understand tradeoffs and challenges. One of those would be an increased share-ability of data across receive nodes: We will distribute more having potentially hundreds of TSDB instances per receiver in case of 100 tenants, right?

I don't think there is really any negative trade-off here. We already expect every receive node in a hashring for a tenant to have a TSDB open for that tenant. This is because it is almost 0% likely that 0 time series would get hashed to one node for any serious load other than degenerate cases like the extreme example posed above, which are be normal. So, we would not expect the number of TSDBs per receiver to change at all. What would change would be the predictability of where a time series would end up. This could make unit testing trickier, but could be avoided by passing a hard-coded value for the "random" value.

One drawback that does occur to me is that when querying for a single time series, queries will need to load data from all receivers in the hashring. Normally, all samples for a time series would be on one single replica, or three replicas would contain all samples of replication-factor=3. Now, all replicas will likely contain some samples from a single time series though none of the replicas will hold the entire time series. This means more work joining the series at query time but only before data is uploaded and deduplicated in object storage.

We have a rather naive distribution hash right now. I feel like experimenting with a variety of hashes might be a higher win. Would it be possible for you to record the label-sets of your active series and use that dataset as a basis for experimentation before we think about introducing randomness?

By experimenting with hashes do you mean experimenting with different hashing functions? Or with adding more data to the hash?
I see two orthogonal issues here:

the hash's distribution; maybe xxhash is fast but not well distributed compared to other hashes?
the underlying lumpiness of the data; how many different time series are we sending? And specifically how many time series with heavy load are we sending?

In the first case, if we assume the xxhash has a decent distribution, then non-uniform load has is most likely due to lack of scale. The larger the set size, the less likely it is to see high variability in the distribution of load. Adding extra static data to a hash to try to shift around some non-uniform load will be just a likely to produce bad results as not adding the data.
The second case, which is what this issue is concerned with, is about addressing cases where the load caused by a handful time series is disproportionately greater than the load of all other 10000+ time series. In these cases, even if we have a perfectly uniform distribution of time series, where every replica gets exactly the same number of series, we can end up with very unequal distribution of load if the few heavy series end up on a single machine. This is not unlikely because the set size of high load series is small, and thus high deviation from the mean is not so unlikely.
One trick we can do is simulate having an infinite number of time series and thus improving the likelihood of an equal distribution of series across replicas. Essentially, instead of rolling the dice one time for each series, we roll it every single time we get a request containing that series. I think this is better than finding the perfect hashing function, i.e. optimizing case 1, as it addresses the case of having too few time series to get a good distribution, solves the case of having lots of time series but only a few heavy series, and even solves extreme case of only sending one single time series to the cluster.

Can you explain what a single time-series that is 10000x more expensive than others looks like? 10000x more samples per second, or what kind of other characteristics? Even if outliers like that exist it sounds to me that they should be breaching some limit and Thanos simply doesn't accept them (we don't have limiting mechanisms like that today but I think it would make sense to have them).

yes exactly

10000x more samples per second
this is one of the examples I was describing. Thanos has no abuse limitation mechanisms, but even if it did, this is not necessarily abuse. What if it's legitimate traffic? Essentially, thanos receive can in many cases have bad load balancing.

Yes, bwplotka comment is correct but I think it's not directly related to this issue: if a hashring handles 100 tenants, then the each replica in the hashring will likely have 100 TSDBs, but that is already the case today independent of this proposal. For any significant load, we will already be in this scenario. There are ways to fix this, e.g. tiered time series routing, but that is an orthogonal problem to this issue. Even if the we end up with only three nodes in a hashring of 100 nodes handling the time series for a tenant, there is still the possibility of bad load balancing across those nodes.
What IS a problem is that samples for a single time series will be distributed across all nodes, meaning increased overhead. I also tried to capture that in my earlier comment: https://github.com/thanos-io/thanos/issues/3793#issuecomment-778615327

Actually I was thinking about the 10000x more samples per second case, and since the chunks get mmapped immediately when 120 samples are full, this shouldn't actually have the effect described. Do you have a dataset that can be shared that reproduces the effect you are aiming to solve? I think if we had that we could be much more systematic about solving it than hypothesizing about problems and solutions.

An example of a user report can be found in #3488, as I mentioned at the top of the issue. Note that this is by no means just a question of TSDB load but importantly also a question of distribution of load in the HTTP and request processing layers in Thanos receive.

I'm not actually seeing any data validating the theory, the only data in that thread is memory usage. Your theory could be right, but it could just as well be a memory leak or something entirely different. The number of active series and samples per second on each receiver would be helpful to support the theory plus profiles of both processes.

@heartTorres would it be possible for you to supply us with this additional data?

Still valid.

Connected to https://github.com/thanos-io/thanos/pull/3390/files (finding better sharding)

Still valid.

authors: brancz bwplotka kakkoyun squat yeya24

repo: thanos-io/thanos


text:
Support for /api/v1/targets/metadata

the current issue is that Prometheus supports the /api/v1/targets/metadata endpoint, while Thanos lacks compatibility with this feature.

+1


text:
🛑 Diamond Residential is down

In 0972e81, Diamond Residential (https://diamondresidential.com) was down:

HTTP code: 0
Response time: 0 ms

Resolved: Diamond Residential is back up in 4498268 after 16 minutes.

repo: thatsenam/uptime


text:
Route [voyager.hooks] not defined

Laravel Version: 5.4.*
Voyager Version: "^1.0"
PHP Version: 5.6.34
Database Driver & Version: 10.1.16-MariaDB

Description:
InvalidArgumentException
Route [voyager.hooks] not defined.

Steps To Reproduce:
When I go to 127.0.0.1/admin, I get a login screen.
After I enter the credentials and login, this exception is thrown.

Dupe of #1779

repo: the-control-group/voyager


text:
BREAD, Multiple Images can't delete one image

Laravel Version: 5.5.14
Voyager Version: 1.0.3
PHP Version: 7.0.22
Database Driver & Version: MySQL 5.6.37

Description:
BREAD, Multiple Images can't delete one image
Steps To Reproduce:

Services entity has a multiple images field (in Bread)
Add images
Save & View (so large)
Delete one image
Can't touch

Fixed in https://github.com/the-control-group/voyager/pull/3240

authors: emptynick nguyentrongthanh