text:
[MLIR/XLA] add a pass to outline kLoop/kInput fusion pattern in xla_hlo dialect

This pass implements the basic kLoop/kInput fusion in fully dynamic shape scenario. Similar to XLA, this pass also supports fusion pattern having multiple outputs if all the shape of outputs are consistent. To this end, we also add a simple shape constrain analysis. This could be built on the shape dialect once it is ready.

It seems like you create a new function call for each of these "fusion".
Have you considered a lighter-weight region-based approach?
I'm thinking about the equivalent of lhlo.fusion operation? https://github.com/tensorflow/tensorflow/blob/master/tensorflow/compiler/mlir/xla/ir/lhlo_ops.td#L388

It seems like you create a new function call for each of these "fusion".
Have you considered a lighter-weight region-based approach?
I'm thinking about the equivalent of lhlo.fusion operation? https://github.com/tensorflow/tensorflow/blob/master/tensorflow/compiler/mlir/xla/ir/lhlo_ops.td#L388

Thanks for your reply!
I agree that region style is more lightweight for fusion itself. Here I use a function style because this style  is easier to play with other existing passes (e.g. hlo-legalize-to-lhlo) .  I will change to region style after the general structure review of this pass is finished.

I looked at the code and left some stylistic comments.
What is the plan for this code moving forward? What do you aim to build and where does it fit in?

We plan to add E2E dynamic shape fusion&codegen support based on MLIR.
This PR mainly covers the fusion pass. As to the codegen part, we are already working on it and later another PR may be submit.
Thanks.

We plan to add E2E dynamic shape fusion&codegen support based on MLIR.
This PR mainly covers the fusion pass. As to the codegen part, we are already working on it and later another PR may be submit.

This is fantastic!
Can you send RFCs and conduct discussions about the plan ahead of time? We should likely coordinate a bit more to make sure your work align with our.
We will also try to surface a better roadmap on our side to give visibility to everything that we're doing.
At the moment IREE is also doing some dynamic codegen and they are very interested in better alignment as well.

@sherhut Thanks for your suggestion. I have done some refine work.

@joker-eph  Here are some documents:
https://groups.google.com/a/tensorflow.org/forum/#!topic/mlir/_X48poNcbDI
https://drive.google.com/open?id=1ZDzXluB2uVc35r1fBNK5jW6rY8s82pc_

@joker-eph @sherhut Could you help to have a look?

The corresponding codegen PR for kLoop/kInput fusion is as following:
https://github.com/tensorflow/tensorflow/pull/38186/files
https://reviews.llvm.org/D77447
Just refer them here to ensure the reviewers can have a full-picture of what we are working on.

@wyzero Can you please check @joker-eph's comments and resolve conflicts?. Thanks!

Sorry for the delay. I will fix it soon.

For interface-based fusibility check, it will be implemented in next commit soon.

@joker-eph Could you help to take a look?

@joker-eph Hi Medhi, could you take another round of review for this PR? It has been pending for more than one month:) If possible, we would like to close  it ASAP.

Hi Mehdi, could you take another round of review for this PR? It has been pending for more than one month:) If possible, we would like to close it ASAP.

@alibaba-common it has been in review with a few round that happened during this time, it wasn't "pending" as waiting for being reviewed. Most of the time I believe has been spent waiting for update on the author side to address comments. I invite you to take another look at the delays between the pushes/updates to the PR and my reviews.

Hi Mehdi, could you take another round of review for this PR? It has been pending for more than one month:) If possible, we would like to close it ASAP.

@alibaba-common it has been in review with a few round that happened during this time, it wasn't "pending" as waiting for being reviewed. Most of the time I believe has been spent waiting for update on the author side to address comments. I invite you to take another look at the delays between the pushes/updates to the PR and my reviews.
It also does not help that some comments/questions are left unanswered.

FYI the sanity build log is complaining about formatting:
FAIL: buildifier found errors and/or warnings in above BUILD files.
buildifier suggested the following changes:
tensorflow/compiler/mlir/xla/BUILD:
153d152
<         "ir/infer_fusibility_op_interface.h.inc",
154a154
>         "ir/infer_fusibility_op_interface.h.inc",
369d368
<     hdrs = ["transforms/cycle_detector.h"],
370a370
>     hdrs = ["transforms/cycle_detector.h"],
381d380
<         ":hlo",
382a382
>         ":hlo",
384c384
<         "//tensorflow/compiler/jit/graphcycles:graphcycles",
---
>         "//tensorflow/compiler/jit/graphcycles",
593d592
<         ":xla_canonicalize_inc_gen",
594a594
>         ":xla_canonicalize_inc_gen",
893d892
<         ":xla_hlo_fusion",
900a900
>         ":xla_hlo_fusion",
exit status 1
Please fix manually or run buildifier <file> to auto-fix.

I suspect you should also have access to the windows build log: https://source.cloud.google.com/results/invocations/dadc78f3-a639-40d5-86bd-301ad8866b1f/log

T:\tmp\bigvaudl\execroot\org_tensorflow\bazel-out\x64_windows-opt\bin\tensorflow\compiler\mlir\xla\ir\hlo_ops.h.inc(4285) : error C4716: 'mlir::xla_hlo::XorOp::inferEffectiveWorkloadSize': must return a value
T:\tmp\bigvaudl\execroot\org_tensorflow\bazel-out\x64_windows-opt\bin\tensorflow\compiler\mlir\xla\ir\hlo_ops.h.inc(3936) : error C4716: 'mlir::xla_hlo::SubOp::inferEffectiveWorkloadSize': must return a value
T:\tmp\bigvaudl\execroot\org_tensorflow\bazel-out\x64_windows-opt\bin\tensorflow\compiler\mlir\xla\ir\hlo_ops.h.inc(1752) : error C4716: 'mlir::xla_hlo::Expm1Op::inferEffectiveWorkloadSize': must return a value
T:\tmp\bigvaudl\execroot\org_tensorflow\bazel-out\x64_windows-opt\bin\tensorflow\compiler\mlir\xla\ir\hlo_ops.h.inc(3878) : error C4716: 'mlir::xla_hlo::SqrtOp::inferEffectiveWorkloadSize': must return a value
T:\tmp\bigvaudl\execroot\org_tensorflow\bazel-out\x64_windows-opt\bin\tensorflow\compiler\mlir\xla\ir\hlo_ops.h.inc(3744) : error C4716: 'mlir::xla_hlo::SinOp::inferEffectiveWorkloadSize': must return a value
...

FYI the sanity build log is complaining about formatting:
FAIL: buildifier found errors and/or warnings in above BUILD files.
buildifier suggested the following changes:
tensorflow/compiler/mlir/xla/BUILD:
153d152
<         "ir/infer_fusibility_op_interface.h.inc",
154a154
>         "ir/infer_fusibility_op_interface.h.inc",
369d368
<     hdrs = ["transforms/cycle_detector.h"],
370a370
>     hdrs = ["transforms/cycle_detector.h"],
381d380
<         ":hlo",
382a382
>         ":hlo",
384c384
<         "//tensorflow/compiler/jit/graphcycles:graphcycles",
---
>         "//tensorflow/compiler/jit/graphcycles",
593d592
<         ":xla_canonicalize_inc_gen",
594a594
>         ":xla_canonicalize_inc_gen",
893d892
<         ":xla_hlo_fusion",
900a900
>         ":xla_hlo_fusion",
exit status 1
Please fix manually or run buildifier <file> to auto-fix.

Thanks for your help. It seems that there is  no difference for most of lines (except the line for "//tensorflow/compiler/jit/graphcycles:graphcycles").  It that true?

Thanks for your help. It seems that there is no difference for most of lines (except the line for "//tensorflow/compiler/jit/graphcycles:graphcycles"). It that true?

The other lines are pointing at issues in ordering: these should be alphabetically sorted I think.

(FYI: there is an issue with our import script I had to fix as well, but I need an internal review and I won't get it today I think)

@joker-eph  I have fixed the above problems and rebase to master as well.

It fails with:
tensorflow/compiler/mlir/xla/ir/infer_fusibility_op_interface.h.inc:89:61: error: use of undeclared identifier 'lhs'
        assert(lhs < op.getOperation()->getNumOperands() && lhs >= 0 &&
                                                            ^

Our internal linter also forced me to add a header guard in cycle_detector.h, add newlines at end of source files, and replace the using namespace directive with more targeted using.

Also fixed other issues:

single-argument constructors must be marked explicit to avoid unintentional implicit conversions
private field 'planner_' is not used
function 'mlir::GraphCycles::IsReachable' has a definition with different parameter names
function 'mlir::GraphCycles::InsertEdge' has a definition with different parameter names
function 'mlir::GraphCycles::RemoveEdge' has a definition with different parameter names
function 'mlir::GraphCycles::HasEdge' has a definition with different parameter names

single-argument constructors must be marked explicit to avoid unintentional implicit conversions

Thanks! I will fix it soon.

@joker-eph Done! Thanks!

@jpienaar  Thanks for your suggestions.  I have done some refine.

@joker-eph Thanks for your suggestions. I will add some tests for cycle_detector.cc.  I have one question though. Do we need to use llvm testing framework or just use tensorflow testing framework? And since these codes are in TF repo. now, it seems the latter one is better?

Please follow MLIR best practices.

Okay. Do you mean use FileCheck (like the tests in xla_hlo_fusion.mlir) ?

MLIR/LLVM is using gtest for testing datastructures, and FileCheck for IR test
(Sorry i am terse but I am on my phone)

@joker-eph  Done!

@joker-eph  Done!

FYI: there is an issue with our import script I had to fix as well,

FYI: my fix got reverted because it broke on windows and so this is blocking the integration of this PR, sorry for the delay I need to give another try to fix the script.

I havent looked into details of this PR yet, but wanted to leave a high-level comment

Having the OpInterfaces is useful. Please consider starting a discussion on MLIR discourse to get more info on how this could be moved into MLIR.
Within IREE, the approach being taken is to convert from XLA-HLO to Linalg on tensors and do fusion there. Its still WIP and we can always convert an xla_hlo.fusion op to a linalg.generic op, but the advantage with fusion on linalg is much lesser surface area of ops. So makes it easier to reason about which ops can be fused and which cant be. Building up that infrastructure in core will benefit more users in the long run. But it is perfectly valid to do fusion at xla_hlo level as done here, and to some extent the use of OpInterface mitigates considering all pair-wise combinations. So not a comment on value of this approach (which is also very valuable).

:+1: to what @MaheshRavishankar said - the op interface for making it possible to write a nice generic fusion pass would be a great addition to core. I'd have to do some more research but we may not be able to use this pass directly as we have different semantics around what fusions can be in certain cases, but reusing the op pass and the definitions on the HLO ops would dramatically simplify our version and decouple the pass from the HLO ops themselves.

@joker-eph  @MaheshRavishankar @benvanik
Thanks for your nice comments.
Is there any other work you suggest us to do to complete this PR merge? We want to ensure important code change from our side can be synced-up with community in a timely manner to avoid big divergence.
In the long run, we do consider implementing the fusion pass in lower-level dialect such as Linalg mentioned before. Currently we choose HLO dialect since lots of our previous experiences on XLA can be reused in an efficient way. With the evolution of MLIR I do expect more and more exciting work can be implemented on top of the MLIR infra.
Thanks

@mihaimaruseac this integration was fixed by the revert from b/154887010 ; can you ping here when it is fixed?

Assinging to me and will look for both. thanks for the bisect

@mihaimaruseac @joker-eph  Hi,  Is there any update about the integration ^v^? Thanks!

@mihaimaruseac @joker-eph Hi! Any update? Thanks!

@mihaimaruseac @joker-eph Hi! Any update? Thanks!

@mihaimaruseac @joker-eph
Hi folks, any feedbacks and comments are highly appreciated from your side:)

Hi. Apologies for the delay.
The sync is broken due to a conflict between tooling not supporting a pattern that would allow some shortcuts during integration (the path @joker-eph wanted to take) and the requirement to do multiple manual changes internally since usage pattern does not work (the part that slipped between the cracks and we didn't have an assignee between me and @joker-eph)

@mihaimaruseac @joker-eph  Hi, thanks for your feedback. Is there any plan to fix the problem and continue the integration?

We are blocked on @joker-eph from what I recall

@joker-eph  Hi, could you help to take a look? Thanks!

We hit some internal both technical and organizational issue on this, to the point where I'm not sure we'll be efficient at accepting contributions here.
Instead I am considering migrating/rewriting this PR to https://github.com/llvm/llvm-project right now. Let me know if you're interested in this, otherwise I'll tackle this probably later this week.

@wyzero Can you please check @joker-eph's comments and keep us posted. Thanks!

We're in the process of fixing it, last week I reproduced the issue (that exists only on Windows) and narrowed it down to the fact that our Kokoro CI is using an older version of the tool on Windows compared to Mac/Linux.

@joker-eph Thanks for your update.  It would be very nice if this PR could be merged into TF repo. first since our internal repo. also use this PR and this could reduce divergence. Initially this pass was designed for the HLO dialect since many of our previous work are based on XLA.  Thanks to your nice comment, we extended the pass to use OpInterfaced to make it more general.  In the future,  we would also like to collaborate with community to make this pass even more general and finally merge it to upstream LLVM repo.

Sorry for the delay, as mentioned before this ran into a corner case of our automation. The infrastructure that performs the import/export between our internal repo and GitHub is now updated (it used an outdated prebuilt binary of some tool on Windows), if you rebase on master I should be able to integrate it.

@jpienaar Thanks for your comments. I have changed accordingly.

@joker-eph Thanks for your update. I have rebased to  the latest master.

@joker-eph  I checked the log of failed uts. As shown as below, it seems that this error is not related to what I changed? Any suggestions? Thanks!
==================== Test output for //tensorflow/python/kernel_tests:metrics_test (shard 12 of 20):
Traceback (most recent call last):
File "/Volumes/BuildData/tmpfs/tmp/bazel/execroot/org_tensorflow/bazel-out/darwin-opt/bin/tensorflow/python/kernel_tests/metrics_test.runfiles/org_tensorflow/tensorflow/python/kernel_tests/metrics_test.py", line 36, in
from tensorflow.python.ops import metrics
File "/Volumes/BuildData/tmpfs/tmp/bazel/execroot/org_tensorflow/bazel-out/darwin-opt/bin/tensorflow/python/kernel_tests/metrics_test.runfiles/org_tensorflow/tensorflow/python/ops/metrics.py", line 23, in
from tensorflow.python.ops.metrics_impl import *
File "/Volumes/BuildData/tmpfs/tmp/bazel/execroot/org_tensorflow/bazel-out/darwin-opt/bin/tensorflow/python/kernel_tests/metrics_test.runfiles/org_tensorflow/tensorflow/python/ops/metrics_impl.py", line 30, in
from tensorflow.python.ops import nn
File "/Volumes/BuildData/tmpfs/tmp/bazel/execroot/org_tensorflow/bazel-out/darwin-opt/bin/tensorflow/python/kernel_tests/metrics_test.runfiles/org_tensorflow/tensorflow/python/ops/nn.py", line 28, in
from tensorflow.python.ops import ctc_ops as _ctc_ops
File "/Volumes/BuildData/tmpfs/tmp/bazel/execroot/org_tensorflow/bazel-out/darwin-opt/bin/tensorflow/python/kernel_tests/metrics_test.runfiles/org_tensorflow/tensorflow/python/ops/ctc_ops.py", line 39, in
from tensorflow.python.ops import linalg_ops
File "/Volumes/BuildData/tmpfs/tmp/bazel/execroot/org_tensorflow/bazel-out/darwin-opt/bin/tensorflow/python/kernel_tests/metrics_test.runfiles/org_tensorflow/tensorflow/python/ops/linalg_ops.py", line 29, in
from tensorflow.python.ops import map_fn
File "/Volumes/BuildData/tmpfs/tmp/bazel/execroot/org_tensorflow/bazel-out/darwin-opt/bin/tensorflow/python/kernel_tests/metrics_test.runfiles/org_tensorflow/tensorflow/python/ops/map_fn.py", line 25, in
from tensorflow.python.autograph.core import ag_ctx as autograph_ctx
File "/Volumes/BuildData/tmpfs/tmp/bazel/execroot/org_tensorflow/bazel-out/darwin-opt/bin/tensorflow/python/kernel_tests/metrics_test.runfiles/org_tensorflow/tensorflow/python/autograph/init.py", line 39, in
from tensorflow.python.autograph.impl.api import AutoGraphError
File "/Volumes/BuildData/tmpfs/tmp/bazel/execroot/org_tensorflow/bazel-out/darwin-opt/bin/tensorflow/python/kernel_tests/metrics_test.runfiles/org_tensorflow/tensorflow/python/autograph/impl/api.py", line 32, in
from tensorflow.python.autograph.impl import conversion
File "/Volumes/BuildData/tmpfs/tmp/bazel/execroot/org_tensorflow/bazel-out/darwin-opt/bin/tensorflow/python/kernel_tests/metrics_test.runfiles/org_tensorflow/tensorflow/python/autograph/impl/conversion.py", line 35, in
from tensorflow.python.autograph.converters import lists
File "/Volumes/BuildData/tmpfs/tmp/bazel/execroot/org_tensorflow/bazel-out/darwin-opt/bin/tensorflow/python/kernel_tests/metrics_test.runfiles/org_tensorflow/tensorflow/python/autograph/converters/lists.py", line 1
^

SyntaxError: I/O error while reading

Closed in https://github.com/tensorflow/tensorflow/commit/9fc38527dabf8d08ca2a608dda0c052524eec058

authors: MaheshRavishankar alibaba-common benvanik gbaned joker-eph mihaimaruseac wyzero yangjunpro


text:
Cache the reordered weight tensors to avoid redundant reorders for MklEinsum

This PR adds caching support for the additional weight reorders. The weight reorders are required for some backend kernels for efficient gemm operations, for e.g. ACL gemm kernels need the weights in a blocked format. We tested Bert Sentiment Analysis model inference on AWS c7g platform with oneDNN backend. At TFServing level we observed
(i) ~20% improvement for latency for the optimized reorder kernels, and
(ii)  ~3x improvement for the unoptimized reorder kernels

Not ready for merge yet.
ARM CI build passed but it's not testing with TF_ONEDNN_ASSUME_FROZEN_WEIGHTS=1
I see few unit test failures when I tested locally with the above flag. Will fix and update the PR.

On behalf of @TensorFlow-MKL, The PR looks good to me.

@mdfaijul Thank you for the quick review!

Thanks for the review, @mdfaijul . Will address your comments.

Hi @snadampal Any update on this PR? Please. Thank you!

Hi @gbaned , I will close it for now, will reopen it with more details. Thanks for checking.

source: gharchive/pull-request

authors: gbaned mdfaijul penpornk snadampal

repo: tensorflow/tensorflow

license_type: permissive

license_source: github-api


text:
Mistral7b hanging on N150

On latest main, Mistral7B is hanging after 2 iterations on N150.
This has been replicated in aus-glx-01 and aus-glx-02.
On aus-wh-01 it passes.
This can be replicated with the mistral7b demo code, on single layer. The hang occurs at ttnn.to_torch after model run.
We're currently trying to identify what introduced the regression.

We've tried in two galaxy hosts (single chip) and on a T3000 (single chip) so far.
These are the findings when running the mistral7b demo code (120 iterations):

T3000 works fine, no hangs on latest main.

aus-glx-01 and aus-glx-08 both hang after 5 iterations even at 1 layer.

Tried clean build on localdev, no luck
Tried deleting pytest cache and ttnn cache and moving the cache to localdev. No luck

One weird issue is that we were seeing in older commits (from ~1 week ago) pytest segfault in the galaxy machines, but no problem on T3000.
Tested several different commits. The process of building was:

git checkout
git submodule update --recursive
make nuke
make build

The default env flags active are:
export ARCH_NAME=wormhole_b0
export TT_METAL_HOME=/localdev/${USER}/tt-metal
export PYTHONPATH=/localdev/${USER}/tt-metal
export TT_METAL_ENV=dev
export PYTHONPATH=${PYTHONPATH}:${TT_METAL_HOME}
export CONFIG=release
export TTNN_CONFIG_OVERRIDES='{"enable_fast_runtime_mode": true}'

Additionally, we also regenerate the TTNN weight cache files on localdev, but the hang still happens on the aus-glx machines.

On latest main with clean environment the issue seems now concentrated to aus-glx-01.
aus-glx-08 is not showing hangs, with/without program cache for the full model, on latest clean main.
On aus-glx-01 the hang is on the final dense matmul, which leads to me believe it might be specific to this host.
It is configured like this:
self.compute_kernel_config = ttnn.WormholeComputeKernelConfig(
            math_fidelity=ttnn.MathFidelity.HiFi4,
            math_approx_mode=False,
            fp32_dest_acc_en=True,
            packer_l1_acc=True,
        )

...

  self.output_program_config = ttnn.operations.matmul.create_matmul_1d_systolic_array_program_config(
      input_shape_a=ttnn.Shape([1, 1, 32, 4096]),
      input_shape_b=self.output_weight.shape,
      core_grid=args.max_grid_size,
      fp32_dst=args.get_compute_kernel_config().fp32_dest_acc_en,
  )

...
 output = ttnn.linear(
     x,
     self.output_weight,
     program_config=self.output_program_config,
     compute_kernel_config=self.args.get_compute_kernel_config(),
)

x.shape = (1,1,32,4096)
weight.shape = (1,1,4096,32000)
Dispatch is using eth cores, so max_core_grid should be (8,8).
I've isolated the above matmul in a unit test, using same program config and dimensions but it passes the test.

Changing from P0 to P1 since we can use other hosts succesfully.

are we blocked by this? @mtairum

This is a non-blocker. I'll lower to P2.

@mtairum is this still a problem? Are we seeing a different behaviour on different machines? (it could be di/dt problem, that's why we are interested)

No. On the latest PR for the updated mistral attention (here https://github.com/tenstorrent/tt-metal/pull/8251) we also tested all skipped tests and re-enabled them. All are passing.
As soon as we merge that to main I'll close this issue.

Now merged. https://github.com/tenstorrent/tt-metal/pull/8251

source: gharchive/issue

authors: mtairum uaydonat

repo: tenstorrent/tt-metal


text:
Height Shader

when I open Height Shader,a hole appeared.

Hello
Thanks a lot for reporting this situation.
This might be caused by the API dropping out some of the requests due to timeout, connection issues, etc
A solution for this problem could be a mechanism to re-request these failed tiles.
Thanks a lot!

Hello
Thanks a lot for reporting this situation.
This might be caused by the API dropping out some of the requests due to timeout, connection issues, etc
A solution for this problem could be a mechanism to re-request these failed tiles.
Thanks a lot!

Thank you for your reply

repo: tentone/geo-three

license: MIT


text:
Add retry logic to CentOS desktop install

Adds YUM cache clean and retries to the Linux GUI installation process in
order to work around intermittent checksum validation errors with the
OpenLogic repo.

Will try to run it today to see if code works as intended

authors: BobLj dbergel


text:
To/separa seccion boot

Sección máxima verosimilitud y bootstrap paramétrico en dos secciones

Correcto, mejor separar la parte de bootstrap paramétrico

authors: felipegonzalez tereom

repo: tereom/fundamentos


text:
Fix timezone for dateFormat

Timezone gap is occurred.
libraryItem["creationDate"] = self.dateFormatter.string(from: asset.creationDate!)

asset.creationDate 2017-01-10 03:09:58 +0000
libraryItem["creationDate"] 2017-01-10T12:09:58.110Z
expected
libraryItem["creationDate"] 2017-01-10T03:09:58.110Z
or
libraryItem["creationDate"] 2017-01-10T12:09:58.110+09:00
related https://github.com/terikon/cordova-plugin-photo-library/pull/21

Fixed, thanks

authors: fiahfy viskin

repo: terikon/cordova-plugin-photo-library

license: mit

license_source: bigquery


text:
woqlPrinter: const next1 reassigned

Got this error using the client in a new project with vite:

next1 can't be reassigned if it's a constant, making it a variable should fix the issue
https://github.com/terminusdb/terminusdb-client-js/blob/8e064e1d8688a5ca8b949754e5d264f5f68aeb60/lib/query/woqlPrinter.js#L418-L421

fixed in version v10.0.25

authors: Francesca-Bit fidelthomet

repo: terminusdb/terminusdb-client-js


text:
Support @class and @constructor jsdoc attributes

http://usejsdoc.org/tags-class.html
@class and @constructor behave the same as @this, but unlike @this, which needs an explicit type name, these two attributes define the 'this' parameter as the current function without having to implicitly name it.
Only the default syntax without explicit type/name is supported.

Should this really refer to the constructor function in this case? The way I understand it, it should be pointing at an instance of that constructor (i.e. an object derived from its prototype property).

Doesn't the parseType take care of that?
/** @param {Foo} f */
var method = function(f) {
  // f is instance of 'Foo'
}

/** @class */
var Foo = function() {
  // 'this' is instance of 'Foo'
}

.. or then I'm completely failing at jsdoc. Will look whether I'll figure out how the jsdoc type definitions work with self defined types.

I'm going by your tests, which assert the type of this to be fn(), which looks suspicious.

Oh right. That one.
I admit that confused me as well, but figured that was some acorn-specific detail that I didn't understand.
Did more testing. Apparently somewhere there is code that compares the case of the function. 'abc' -> fn(), 'Abc' -> Abc. Fixed tests.

Ah, I see what was going on now -- because the constructors weren't capitalized, the parseType didn't treat them as constructors. I've pushed a somewhat robust implementation in d45e911, and merged your tests in 1ac35c058b2

authors: Rantanen marijnh

repo: ternjs/tern


text:
RegExp prototype def update to include overridden 'toString' method

This PR updates 'RegExp' prototype definition to include 'toString' method.
Reference : https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp/toString
@marijnh Can you please have a look and merge if the change looks fine.

toString is already in Object.prototype, so I've avoided adding it to the various prototypes that derive from that, since that doesn't really add any new information.

Thanks a lot @marijnh for review. That was really quick :+1:
The problem what I have is, when I create a RegExp object and try to provide hints on the instance var upon typing '.' , the hint list doesn't include 'toString'. Ideally, as 'toString' is overridden in 'RegExp' , that kind of make 'toString' an explicit behavior in case of 'RegExp' instead of an inherited behavior. So, the immediate hint list from Tern should include the 'toString' method after inferring the member to be of type 'RegExp' I guess. Do let me know your thoughts on this.

That is indeed the case, but I don't think it's a big enough problem to start duplicating toString definitions for.

authors: marijnh swmitra


text:
Fix Cannot read property 'prev' of null

Not sure if this is the correct way to do this, but it fixed the bug that I was seeing locally with auto-completion and all that.
Should be a fix for: #891

That doesn't really address the underlying issue, though. I'd prefer to see an example of how to reproduce it so that I can properly fix it.

This issue can be reproduced with atom-ternjs package. Probably @jfelsinger can provide a concrete example?

FIXED: https://github.com/lyjstudy/tern
rmScope will destroy node.scope, so we cache it

Solved via #905

authors: guiguan jfelsinger lyjstudy marijnh


text:
Multiple Regions: Support provider=

Is your request related to a new offering from AWS?
Is this functionality available in the [AWS provider for Terraform]

Yes ✅:  multiple providers: https://www.terraform.io/language/providers/configuration#alias-multiple-provider-configurations

Is your request related to a problem? Please describe.

Not really. I would like to spin up 2 EKS clusters, 1 in 1 region & the other in another region.

Describe the solution you'd like.
module "eks" {
  provider = aws.secondary
  source                          = "terraform-aws-modules/eks/aws"

Describe alternatives you've considered.

I copied the module into another folder & run TF from that folder on an EC2 in that region

This is already possible today using:
# will use default AWS provider
module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  ...
}

module "eks_oregon" {
  source  = "terraform-aws-modules/eks/aws"

  providers = {
    aws = aws.uswest2
  }
}

module "eks_virginia" {
  source  = "terraform-aws-modules/eks/aws"

  providers = {
    aws = aws.useast1
  }
}

Nice! I looked in the docs & didn't see this option! Thank you

authors: bryantbiggs darrenfurr

repo: terraform-aws-modules/terraform-aws-eks


text:
Not logged into the server error when running kubectl get nodes after provisioning cluster using the module

Description
Just recently upgraded the module version to latest and provisioned a cluster. After running update kubeconfig I'm receiving error below
error: You must be logged in to the server (Unauthorized)
I'm  using same credentials I used to provision the cluster. Seems like some changes in current version.

Please provide the details required on the issue template

Ok adding this enable_cluster_creator_admin_permissions = true fixed. I guess earlier version it was default setting up creator as admin. Seems like this behavior is changed in new version

authors: bryantbiggs rprakashg-redhat


text:
Use amazon-manged AmazonEKS_CNI_Policy, don't create it

Versions
terraform: v1.5.4
terraform-aws-modules/vpc/aws: v5.1.1
eks: v19.15.4
iam-role-for-service-accounts-eks: v5.28.0
Is your request related to a problem? Please describe.
Currently the module creates its own policy and a role for VPC-CNI addon.
Creating a new policy for IPv6 makes sense, but for IPv4 there is already an amazon managed one -  AmazonEKS_CNI_Policy
Describe the solution you'd like.
Attach amazon-provided CNI policy for IPv4 instead of creating new one.
Describe alternatives you've considered.
Creating role via separate role module and attaching policy like
  role_policy_arns = {
    policy = "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
  }

Additional context
N/A

What is the benefit of this approach?

Well, I'm not sure what is the benefits of the current approach. Can you explain this as well?
For example, eks module doesn't create new policies, but reuses what Amazon provides.
Some things that come to my mind immediately:

policy deduplication. Why to create duplicates? It's just common sense.
no need to keep it up to date. If/when amazon updates their AmazonEKS_CNI_Policy (not sure if it ever happened but...) for new AWS EKS release you don't have to keep your copy up to date. This is more practical sense.

There are no tangible benefits to change this at this time

Wow, closed. I expected some kind of discussion here.

There wasn't much provided to discuss. There isn't anything that prevents someone from using the managed policy with this sub-module, therefore the way its written provides the most value (some organizations do not accept the use of managed policies since the permissions can change without their knowledge). If you wish to have a discussion, please provide some useful facts and data to back up the topic of discussion

authors: bryantbiggs ptimofee

repo: terraform-aws-modules/terraform-aws-iam


text:
this_rds_cluster_master_username output is required to be sensitive for unknown reason

Description
After Terraform upgrade v0.14 -> v0.15, terraform plan started to fail with
Error: Output refers to sensitive values
│
│   on .terraform/modules/xxx/outputs.tf line 49:
│   49: output "this_rds_cluster_master_username" {
│
│ Expressions used in outputs can only refer to sensitive values if the sensitive attribute is true.

Initially, the issue occurred with terraform-aws-modules/rds-aurora/aws v2.29.0, but upgrade to v4.1.0 didn't help.
Looks like there was a pull request with a similar issue https://github.com/terraform-aws-modules/terraform-aws-rds-aurora/pull/176 but it was closed.
As it was mentioned in that pull request discussion, master_username field which is used by this_rds_cluster_master_username output is not marked as sensitive in aws_rds_cluster resource but nevertheless, the error occurs.
Versions

Terraform: v0.15.0
Provider(s):
provider registry.terraform.io/hashicorp/aws v3.37.0
provider registry.terraform.io/hashicorp/local v2.1.0
provider registry.terraform.io/hashicorp/random v3.1.0
provider registry.terraform.io/hashicorp/template v2.2.0
Module:
terraform-aws-modules/rds-aurora/aws v2.29.0 or v4.1.0

Reproduction
Steps to reproduce the behavior:

Use terraform-aws-modules/rds-aurora/aws module with Terraform v0.14
Update Terraform to v0.15
Run terraform plan

Expected behavior
terraform plan finishes successfully.
Actual behavior
terraform plan fails with an error:
Error: Output refers to sensitive values
│
│   on .terraform/modules/xxx/outputs.tf line 49:
│   49: output "this_rds_cluster_master_username" {
│
│ Expressions used in outputs can only refer to sensitive values if the sensitive attribute is true.

There seems to be a breaking change in terraform .15 that requires all sensitive values to have the sensitive flag.

Yeah, I got that too. I simply add sensitive   = true in the output "this_rds_cluster_master_username" on .terraform/modules/aurora/outputs.tf, and it works through.

@JimBeam2019 thats a non ideal solution because typically you shouldn't commit what is in the .terraform folders.


text:
feat: create route tables per AZ for intra subnets

#Description
This change implements a per-AZ route table for the intra subnets that get created as a part of the module.
It uses the same naming scheme to keep feature parity with the other subnets.

Your MR seems to be review, you push lot of bad/not purpose "formating" changes.
Could you keep only changes related to routes tables per AZ ?
And what your issue motivating this change ? I don't undertand why you done this.

6 months later, not sure why some of the formatting (terraform fmt) were included and this clearly didn't receive the time initially. ☮

authors: gillg iamnande

repo: terraform-aws-modules/terraform-aws-vpc


text:
Remove web_server_ipv4_cidr_block from composer v2

Composer v2 does not support web_server_ipv4_cidr_block
fixes #51

Thanks for the PR! 🚀
Unfortunately it looks like some of our CI checks failed. See the Contributing Guide for details.

⚠check_documentation
The documentation needs to be regenerated. Please run make generate_docs.

Checking for documentation generation
diff -r '--exclude=.terraform' '--exclude=.kitchen' '--exclude=.git' '--exclude=autogen' '--exclude=*.tfvars' /workspace/modules/create_environment_v2/README.md /tmp/tmp.dgHbRHNzoF/generate_docs/workspace/modules/create_environment_v2/README.md
49d48
< | web\_server\_ipv4\_cidr | The CIDR block from which IP range in tenant project will be reserved for the web server. | `string` | `null` | no |
Error: Documentation generation has not been run, please run the
'make docker_generate_docs' command and commit the above changes.

Note: non breaking change since the provider does not allow for the creation of composer v2 with this set, so its unlikely that removing this var will break anyone.
https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/composer_environment#web_server_ipv4_cidr_block

authors: bharathkkb comment-bot-dev majduk


text:
fix: add backend.groups.description variable to serverless_neg

It seems description variable is being used in module like below.
# modules/serverless_neg
resource "google_compute_backend_service" "default" {
...
  dynamic "backend" {
    for_each = toset(each.value["groups"])
    content {
      description = lookup(backend.value, "description", null)
      group       = backend.value["group"]

    }
  }
...
}

But there's no description in variables.tf, and documentation.

@Sonins plz update template here and follow this doc to generate the code and submit.

@imrannayer Just fixed it, can you check?

/gcbrun

/gcbrun

authors: Sonins imrannayer


text:
migrating tim repos to pull cra config from a file

Description

Release required?

[ ] No release
[ ] Patch release (x.x.X)
[ ] Minor release (x.X.x)
[ ] Major release (X.x.x)

Release notes content

Run the pipeline
If the CI pipeline doesn't run when you create the PR, the PR requires a user with GitHub collaborators access to run the pipeline.
Run the CI pipeline when the PR is ready for review and you expect tests to pass. Add a comment to the PR with the following text:
/run pipeline

Checklist for reviewers

[ ] If relevant, a test for the change is included or updated with this PR.
[ ] If relevant, documentation for the change is included or updated with this PR.

For mergers

Use a conventional commit message to set the release level. Follow the guidelines.
Include information that users need to know about the PR in the commit message. The commit message becomes part of the GitHub release notes.
Use the Squash and merge option.

/run pipeline

/run pipeline

:tada: This PR is included in version 1.17.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:

repo: terraform-ibm-modules/terraform-ibm-cbr


text:
chore(deps): update common-dev-assets digest to de9d7f6

This PR contains the following updates:

Package
Update
Change

common-dev-assets
digest
bab77a8 -> de9d7f6

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

:tada: This PR is included in version 2.0.2 :tada:
The release is available on:

GitHub release
v2.0.2

Your semantic-release bot :package::rocket:


text:
chore(deps): update common-dev-assets digest to 9b8a0b5

This PR contains the following updates:

Package
Update
Change

common-dev-assets
digest
8d95c46 -> 9b8a0b5

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

:tada: This PR is included in version 1.0.4 :tada:
The release is available on:

GitHub release
v1.0.4

Your semantic-release bot :package::rocket:

repo: terraform-ibm-modules/terraform-ibm-hpc


text:
chore(deps): update common-dev-assets digest to d366e67

This PR contains the following updates:

Package
Update
Change

common-dev-assets
digest
6f24289 -> d366e67

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

:tada: This PR is included in version 1.2.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


text:
First

Added Landing Zone Module code

Closing in favor of https://github.com/terraform-ibm-modules/terraform-ibm-landing-zone/pull/2

authors: jvallexm vburckhardt


text:
Add outputs surfacing service endpoints and ips for each vpe

Description
There is currently no outputs in the module making it difficult to consume in a broader solution.
Add:

List of service endpoint
For instance

output "vpe_service_endpoint" {
  description = "Vpe Service Endpoint"
  value = [for vpe_pg in ibm_is_virtual_endpoint_gateway.vpe : vpe_pg.service_endpoints[0]]
  //value = { for k, vpe in ibm_is_virtual_endpoint_gateway.vpe : k => vpe.service_endpoints[0] }
}

List of IPs for each VPE. https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/is_virtual_endpoint_gateway#ips

New or affected modules

By submitting this issue, you agree to follow our Code of Conduct

It seems like the service_endpoints is not an attribute

@Aayush-Abhyarthi - Example is coming from customer. It is mainly to give an idea on what customer is asking for as output. Implementation details are left to us.

authors: Aayush-Abhyarthi vburckhardt


text:
chore(deps): update common-dev-assets digest to 5f5fd35

This PR contains the following updates:

Package
Update
Change

common-dev-assets
digest
d63f9de -> 5f5fd35

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.
👻 Immortal: This PR will be recreated if closed unmerged. Get config help if that's undesired.

[ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Renovate Bot.

/run pipeline

:tada: This PR is included in version 3.0.6 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


text:
Remove specialization of ThRZAssembly and RZAssembly

The current implementations of ThRZAssembly and RZAssembly implement methods like radialOuter,
radialInner, etc., which reach down into their corresponding ThRZBlock's to get the outer_radius  and
inner_radius dimensions.
This specialization seems outdated with respect to the implementation of grids and should be reassessed.
It seems that maintaining this separate set of assemblies that is different than just the base Assembly class
introduces more framework complexities. #264 introduced changes where the specialization of HexAssembly
and CarestianAssembly were removed and the classes only exist now as placeholders for their "type".

@john-science this might be something we can tackle pretty easily. @sombrereau - do you still use these class definitions?

So, @sombrereau points out that both RZAssembly and ThRZAssembly are still heavily used. So we can clean up the internal implementation of these things, but we need to keep the API and end results of these classes the same.

Actually, as @drewj-tp and I have been talking lately about how to make ARMI more flexible, I think I want to go in the opposite direction this ticket suggests.
I think we want to make better use of our class heirarchies:

Block

HexBlock(Block)
CartesianBlock(Block)
ThetaRZBlock(Block)

PinnedBlock(Block)

PinnedHexBlock(PinnedBlock, HexBlock)
PinnedCartesianBlock(PinnedBlock, CartesianBlock)

FastPinnedHexBlock(PinnedHexBlock)
Etc

The idea being that right now ARMI's HexBlock is actually a "fast-reactor, pin-type, hex-shaped block". And ARMI forcing all this extra stuff on people without letting them just have a "PinnedHexBlock" to use for the thermal reactor is very limiting.

Actually, as @drewj-tp and I have been talking lately about how to make ARMI more flexible, I think I want to go in the opposite direction this ticket suggests.
I think we want to make better use of our class heirarchies:

Block

HexBlock(Block)
CartesianBlock(Block)
ThetaRZBlock(Block)

PinnedBlock(Block)

PinnedHexBlock(PinnedBlock, HexBlock)
PinnedCartesianBlock(PinnedBlock, CartesianBlock)

FastPinnedHexBlock(PinnedHexBlock)

Etc

The idea being that right now ARMI's HexBlock is actually a "fast-reactor, pin-type, hex-shaped block". And ARMI forcing all this extra stuff on people without letting them just have a "PinnedHexBlock" to use for the thermal reactor is very limiting.

I'm not sure I like the multiple inheritance structure here. I think that if we want ARMI to be abstract we should make the minimum viable abstract type and then provide an open source plugin that extends the data model as an example for how one may create an application and build what is needed. Going down the path of pre making these inheritances will be messy quickly and implies that the framework knows or needs to know and handle a bunch of situations. I like the idea of the framework allowing for applications to extend the data model on initialization and just providing example documentation.
Thoughts?

authors: jakehader john-science

repo: terrapower/armi


text:
getCircleInnerDiameter() on holed components

Description
Add a getCircleInnerDiameter() method to all Holed components. This is in response to #755 .
Also filled out the relevant component parameter definitions.

Checklist

[x] The code is understandable and maintainable to people beyond the author.
[x] Tests have been added/updated to verify that the new or changed code works.
[x] There is no commented out code in this PR.
[x] The commit message follows good practices.
[x] All docstrings are still up-to-date with these changes.

Looks good to me! :shipit: 🚀

@keckler  black needs to be run.
(I would do it myself, but it's in your repo. Sorry.)

I'm not sure what's up with that. On my machine, black says everything is fine:

And looking at the changes, I don't see anything amiss from a formatting perspective.
Maybe re-trigger the tests?

@keckler your feature branch is 1 commit behind armi::main. I wonder if that could be an issue?

Hm, my click version is behind. Let's try that.

Okay, it was click.
I just pushed up a new reformatted version.
Sorry about all the hassle!

This should be good to merge now.

authors: albeanth john-science keckler


text:
chore(master): release 0.3.1

:robot: I have created a release beep boop
0.3.1 (2024-06-09)
Bug Fixes

bug fix (93a2fd1)

This PR was generated with Release Please. See documentation.

:robot: Release is at https://github.com/terrasible/terraform-cloudflare-rulesets/releases/tag/v0.3.1 :sunflower:


text:
SW-5763 Include project name with list document responses, delete unused function in DocumentStore

Add nullable projectName field to ExistingDocumentModel and change the two fetch methods in the DocumentStore that are used in the "list documents" API call to use the DSL Context instead of DAO, and plumb in the Project Name field
Removes an unused function in the DocumentStore - fetchProjectId (only usage removed here)

#2240  👈
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @nickgraz and the rest of your teammates on  Graphite

Merge activity

Aug 6, 5:09 PM EDT: @nickgraz started a stack merge that includes this pull request via Graphite.

repo: terraware/terraware-server


text:
Fix github workflow deprecations in terraware-web

With this PR, I don't see any of the deprecation warnings anymore.
Compare new build vs old build.

Use newer versions of github actions that support newer Nodejs versions
Replace set-output with GITHUB_OUTPUT workflow (see references)
Remove usage of satackey/action-docker-layer-caching, use github action buildx support cache-from and cache-to tags for docker layers caching. Tested by resubmitting a build and saw cached docker layers being used.

References:

https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
https://depot.dev/blog/docker-layer-caching-in-github-actions

Thanks for including the reference links, looks good 🚢

Thanks @joshbuchea 👍

Looks like I did this incorrectly. Will take this off my plate for now.

Reopening this to look into.

Updated this PR with one more change as per instructions here https://github.com/aws-actions/configure-aws-credentials#notice-node12-deprecation-warning

repo: terraware/terraware-web


text:
Force UTF8 encoding when reading locale csv

The locale CSV is encoded in UTF8. The current code only worked on systems where UTF8 was the default file encoding (as FileReader always uses the system default). The code did not work properly on systems with a different default encoding, e.g. file.encoding=ANSI_X3.4-1968.
This PR fixes this problem by reading the locale CSV with UTF8 encoding.

:+1:

authors: buehner dnlkoch


text:
Support Node 0.12.0 and io.js 1.0.4

Except for tessel repl, which hangs on Linux and tries to run invalid Lua on Windows. (No regression on 0.10)
Tested on

Linux (Ubuntu 14.10) Node 0.10.25
Linux Node 0.12.0
Linux io.js 1.0.4
Windows 7 Node 0.12

WFM on mac (thanks), didn't work on windows, but could be a driver issue. No compile issues on windows though.

repo: tessel/cli


text:
Pre-compiled module is missing: grpc@1.6.6.

Hi,
I'm trying to use the tessel-av Microphone module, but I keep getting this error:
Pre-compiled module is missing: grpc@1.6.6.
Please could someone advise me on what to do.
Thanks,
Jake

@jakecahill I'm wondering if that just means that a tessel 2 compatible version isn't available yet. I have a similar issue

That's strange, tessel-av doesn't depend on that module.
Either way, I've built the pre-compiled modules and pushed them to the package server:

@rwaldron how do you precompile a module so if I come across this issue again I can help fix said issue?

To do it manually is kind of a pain in the ass, but like this:

Clone https://github.com/tessel/t2-compiler/
Follow the installation and setup instructions
Run either ./compile-docker.sh module@X.X.X or ./compile-vagrant.sh module@X.X.X
Open t2-compiler/out and unzip the resulting .tgz into $HOME/.tessel/binaries/

When you t2 run ... the bundler process will look for the precompiled modules in $HOME/.tessel/binaries/

@rwaldron then how do you push them to the public?

There is no way to do that without providing credentials to the Tessel Project's AWS account

@rwaldron ok Thanks for the info. I have found a few other packages that aren't in the tessel account file: colony-compiler, colors, colorsafeconsole, effess, hardware-resolve, humanize, my-local-ip, nomnom, semver, structured-clone, temp, and usb.
Thanks for working to make this project grrrrrrrrreat!

@rwaldron Thanks for the info

authors: KyleWMiller jakecahill rwaldron


text:
Word-Level OCR

Hi there,
Currently I had an idea of Word-Level training, basically it focuses on word-level recognition rather than the line-level & character-level that Tesseract is currently using.
The idea is simple:

First, you train a new language model using a Word List that contains a list of the words of a certain language. Thus, generating an image for each word, and train on it.
Latter-on, when you want to recognize the text of a scanned text image, Tesseract will segment the image into lines, then segment those lines into words, and finally use your trained model to recognize those words.

Now my question is, why such idea will or won't work?
@theraysmith @amitdo @Shreeshrii @stweil @kba

This is called "word spotting". See https://sci-hub.tw/10.1016/j.patcog.2017.02.023 for a 2017 survey on word spotting and related technologies.
The issues here are only for bugs in the software, best to use the mailing list for such questions.

segment those lines into words

How do you suggest to segment the words?

and finally use your trained model to recognize those words.

Again, which method to use here?
Language model is only small part of the recognition process.

@amitdo

For recognition, TPP-PHOCNet is the state-of-the-art wordspotting method, and it's open sourced!
For line segmentation, Tesseract will do, but if you want something fancy, there is ocroseg from Ocropus3.
The problem is the word segmentation, and I was hoping you'd suggest some.

Though I wonder after finding a word detection/ segmentation method, can Tesseract still be used to recognize the segmented words?
@kba Some latest and interesting word-spotting research:
Deep Learning for Word Spotting
Exploring Architectures for CNN-Based Word Spotting
Attribute CNNs for Word Spotting in Handwritten Documents

Improving OCR for an Under-Resourced Script
Using Unsupervised Word-Spotting

https://www.cs.tau.ac.il/~wolf/papers/magicocr.pdf

@amitdo is there any method available that can segment lines into words? perhaps even in github?
@kba Thanks for the survey, it was interesting.

https://github.com/tesseract-ocr/tesseract/blob/master/src/textord/wordseg.cpp
This is used for the legacy engine.

@amitdo it was very hard to find a word segmentation tool, but I just found an interesting repo, it's called Image2code also have a look at their paper.
Whats interesting about their project is that they are using a dynamic white-space threshold, which means that their tool calculates for each line separately the average space between each word within it.
That means that their tool is suitable for both printed and handwriting word segmentation.
amitdo, can you test Image2code word segmentation?

A few points:

I don't know how many hours a month is spent by Ray to develop Tesseract these days.
Although Tesseract is very popular software, we don't have a lot of
outside contributors.
It's already quite hard to provide support for Tesseract.

My conclusion is that we should focus on small improvements to the existing CNN-LSTM engine.
That's my personal opinion.

I was not talking about word-spotting, I was talking about Word Detection.
I think I found what I was looking for:
https://github.com/gaxler/dataset_agnostic_segmentation
http://www.cs.tau.ac.il/~wolf/papers/dataset-agnostic-word.pdf
@amitdo @kba what do you think about my ocr pipe line below?
Training

Create a huge .txt word-list using Crunch.
Generate synthesized words from that .txt wordlist ,only words & no white-space, by kraken linegen.
Train a new language model using those word chunks. Tesseract or Calamari.

Recognition

Segment pages into lines, using ocropus3 ocrseg.
Segment lines into words, using Agnostic Segmentation & heatmaps.
Use the trained Word-level language model to recognize the segmented word images.

Nice find, but my previous answer still applies.

@christophered

Create a huge .txt word-list using Crunch.

Given how small a fraction of possible permutations of letters are words of natural languages, I would not recommend that. For brute-forcing passwords sure, but not for OCR.

Generate synthesized words from that .txt wordlist ,only words & no white-space, by kraken linegen.

Not sure whether training an engine on words only makes sense because internally they will train on/predict sequences of glyphs. Also skeptical whether word segmentation and recognition of individual words beats one-step recognition of a line with implicit word segmnetation. IMHO it would make more sense to use a word-level language model (or even a spell checker with a custom dict) in post-processing of the results.
But I haven't measured and wouldn't want to discourage you, best of luck and feel free to share your results. BTW, since you've been trying out a lot of software wrt OCR/segmentation etc, PR to https://github.com/kba/awesome-ocr are very welcome :-)

Off the top of my head: ocropy encodes bboxes in PNG convertible to hOCR, kraken generates JSON, tesseract API with line iterator with some coding... All would be fairly straightforward to serialize into a list of x0,y0,x1,y1 coordinates.

authors: amitdo christophered kba

repo: tesseract-ocr/tesseract


text:
CMake currently does not support Tesseract builds for non Intel hosts (Android cross-build error)

Host OS: Linux xubuntu64 4.15.0-33-generic #36-Ubuntu SMP Wed Aug 15 16:00:05 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux; CMake GUI 3.10.2; Android toolchain - android-ndk-r15c; gcc version 7.3.0; Tesseract rev. ac7ea4322a23af46fa6ddccf23497bd06f1e2d38.
CMake GUI changes: CMAKE_PREFIX_PATH:PATH="/home/bleach/Lib/leptonica/build_arm" specified.
Build error received:

make output
/home/bleach/Lib/tesseract/source/src/arch/dotproductavx.cpp:20:2: error: Implementation only for AVX capable architectures
#error Implementation only for AVX capable architectures
 ^
In file included from /home/bleach/Lib/tesseract/source/src/arch/dotproductavx.cpp:23:
In file included from /home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/immintrin.h:28:
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:47:5: error: use of undeclared identifier '__builtin_ia32_emms'; did you mean
      '__builtin_isless'?
    __builtin_ia32_emms();
    ^
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:47:5: note: '__builtin_isless' declared here
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:47:25: error: too few arguments to function call, expected 2, have 0
    __builtin_ia32_emms();
                        ^
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:64:19: error: use of undeclared identifier '__builtin_ia32_vec_init_v2si'
    return (__m64)__builtin_ia32_vec_init_v2si(__i, 0);
                  ^
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:81:12: error: use of undeclared identifier '__builtin_ia32_vec_ext_v2si'
    return __builtin_ia32_vec_ext_v2si((__v2si)__m, 0);
           ^
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:143:19: error: use of undeclared identifier '__builtin_ia32_packsswb'
    return (__m64)__builtin_ia32_packsswb((__v4hi)__m1, (__v4hi)__m2);
                  ^
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:173:19: error: use of undeclared identifier '__builtin_ia32_packssdw'
    return (__m64)__builtin_ia32_packssdw((__v2si)__m1, (__v2si)__m2);
                  ^
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:203:19: error: use of undeclared identifier '__builtin_ia32_packuswb'
    return (__m64)__builtin_ia32_packuswb((__v4hi)__m1, (__v4hi)__m2);
                  ^
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:230:19: error: use of undeclared identifier '__builtin_ia32_punpckhbw'
    return (__m64)__builtin_ia32_punpckhbw((__v8qi)__m1, (__v8qi)__m2);
                  ^
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:253:19: error: use of undeclared identifier '__builtin_ia32_punpckhwd'
    return (__m64)__builtin_ia32_punpckhwd((__v4hi)__m1, (__v4hi)__m2);
                  ^
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:274:19: error: use of undeclared identifier '__builtin_ia32_punpckhdq'
    return (__m64)__builtin_ia32_punpckhdq((__v2si)__m1, (__v2si)__m2);
                  ^
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:301:19: error: use of undeclared identifier '__builtin_ia32_punpcklbw'
    return (__m64)__builtin_ia32_punpcklbw((__v8qi)__m1, (__v8qi)__m2);
                  ^
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:324:19: error: use of undeclared identifier '__builtin_ia32_punpcklwd'
    return (__m64)__builtin_ia32_punpcklwd((__v4hi)__m1, (__v4hi)__m2);
                  ^
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:345:19: error: use of undeclared identifier '__builtin_ia32_punpckldq'
    return (__m64)__builtin_ia32_punpckldq((__v2si)__m1, (__v2si)__m2);
                  ^
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:366:19: error: use of undeclared identifier '__builtin_ia32_paddb'; did you mean
      '__builtin_arm_qadd'?
    return (__m64)__builtin_ia32_paddb((__v8qi)__m1, (__v8qi)__m2);
                  ^
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:366:19: note: '__builtin_arm_qadd' declared here
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:366:40: error: cannot initialize a parameter of type 'int' with an rvalue of type
      '__v8qi' (vector of 8 'char' values)
    return (__m64)__builtin_ia32_paddb((__v8qi)__m1, (__v8qi)__m2);
                                       ^~~~~~~~~~~~
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:387:19: error: use of undeclared identifier '__builtin_ia32_paddw'; did you mean
      '__builtin_arm_qadd'?
    return (__m64)__builtin_ia32_paddw((__v4hi)__m1, (__v4hi)__m2);
                  ^
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:366:19: note: '__builtin_arm_qadd' declared here
    return (__m64)__builtin_ia32_paddb((__v8qi)__m1, (__v8qi)__m2);
                  ^
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:387:40: error: cannot initialize a parameter of type 'int' with an rvalue of type
      '__v4hi' (vector of 4 'short' values)
    return (__m64)__builtin_ia32_paddw((__v4hi)__m1, (__v4hi)__m2);
                                       ^~~~~~~~~~~~
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:408:19: error: use of undeclared identifier '__builtin_ia32_paddd'; did you mean
      '__builtin_arm_qadd'?
    return (__m64)__builtin_ia32_paddd((__v2si)__m1, (__v2si)__m2);
                  ^
/home/bleach/Lib/android-ndk-r15c/toolchains/llvm/prebuilt/linux-x86_64/lib64/clang/5.0.300080/include/mmintrin.h:366:19: note: '__builtin_arm_qadd' declared here
    return (__m64)__builtin_ia32_paddb((__v8qi)__m1, (__v8qi)__m2);
                  ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
CMakeFiles/libtesseract.dir/build.make:86: recipe for target 'CMakeFiles/libtesseract.dir/src/arch/dotproductavx.cpp.o' failed
make[2]: *** [CMakeFiles/libtesseract.dir/src/arch/dotproductavx.cpp.o] Error 1
CMakeFiles/Makefile2:104: recipe for target 'CMakeFiles/libtesseract.dir/all' failed
make[1]: *** [CMakeFiles/libtesseract.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

Please advise how to quickly remove AVX(2) support from the code or Cmake config. I tried

my changes
index ef5d1b47..3afc4275 100644
@@ -117,14 +117,14 @@ AM_CONDITIONAL([AVX2_OPT], false)
 AM_CONDITIONAL([SSE41_OPT], false)
 AM_CONDITIONAL([MARCH_NATIVE_OPT], false)

-AX_CHECK_COMPILE_FLAG([-mavx], [avx=true], [avx=false])
+AX_CHECK_COMPILE_FLAG([-mavx], [avx=false], [avx=false])
 if $avx; then
-    AM_CONDITIONAL([AVX_OPT], true)
+    AM_CONDITIONAL([AVX_OPT], false)
 fi

-AX_CHECK_COMPILE_FLAG([-mavx2], [avx2=true], [avx2=false])
+AX_CHECK_COMPILE_FLAG([-mavx2], [avx2=false], [avx2=false])
 if $avx2; then
-    AM_CONDITIONAL([AVX2_OPT], true)
+    AM_CONDITIONAL([AVX2_OPT], false)
 fi

 AX_CHECK_COMPILE_FLAG([-msse4.1], [sse41=true], [sse41=false])

---------------------------- src/arch/simddetect.h ----------------------------
index def66896..c137e118 100644
@@ -31,12 +31,12 @@ extern DotProductFunction DotProduct;
 class SIMDDetect {
  public:
   // Returns true if AVX is available on this system.
-  static inline bool IsAVXAvailable() { return detector.avx_available_; }
+  static inline bool IsAVXAvailable() { return false; }
   // Returns true if AVX2 (integer support) is available on this system.
-  static inline bool IsAVX2Available() { return detector.avx2_available_; }
+  static inline bool IsAVX2Available() { return false; }
   // Returns true if AVX512 Foundation (float) is available on this system.
   static inline bool IsAVX512FAvailable() {
-    return detector.avx512F_available_;
+    return false;
   }
   // Returns true if AVX512 integer is available on this system.
   static inline bool IsAVX512BWAvailable() {

, but it did not help.

Is this a duplicate of issue #2334? All errors shown in your make output are in files which are not part of the Tesseract sources. Those files obviously are for the Intel architecture, so they should not be included by ARM toolchains. This looks like a wrong build setting.

@stweil: this is cmake related related issue while 2334 is autotools related, so from perspective of build toots it is not duplicate.
I am not sure if our current cmake configuration will work for android build without modification, but just setting CMAKE_PREFIX_PATH seems to be not sufficient...

Thank you, @zdenop. Yes, the current CMake has fixed unconditional settings for Intel builds and cannot support cross builds for other architectures like ARM. I update the summary.

Sorry, I forgot to mention that host os is running under VirtualBox, which in turn is running under Win10. CPU Intel Xeon X.

@Bleach665 : try current master - there is detection of available features in cmake.

@zdenop.
I do not have at fingertips Linux, therefore I try to build in Windows 10.
I configure Tesseract via CMake with Android 16 toolchain and MinGW makefile. MinGW was taken from Qt distrib.
Leptonica configured in such way was build successfully. Tesseract CMake configuration performed well (with minimal changes CMakeList.txt), but on build I receive errors:
In file included from E:\Lib_prebuild\tesseract\source\src\ccmain\adaptions.cpp:26:
E:\Lib_prebuild\tesseract\source\src\ccmain/tesseractclass.h:30:10: fatal error: 'allheaders.h' file not found
#include "allheaders.h"             // for pixDestroy, pixGetWidth, pixGetHe...

Seems like missing include path for leptonica. How did you run
configuration?
Dňa po 22. 4. 2019, 12:24 Bleach665 notifications@github.com napísal(a):

@zdenop https://github.com/zdenop.
I do not have at fingertips Linux, therefore I try to build in Windows 10.
I configure Tesseract via CMake with Android 16 toolchain and MinGW
makefile. MinGW was taken from Qt distrib.
Leptonica configured in such way was build successfully. Tesseract CMake
configuration performed well (with minimal changes CMakeList.txt), but on
build I receive errors:
In file included from E:\Lib_prebuild\tesseract\source\src\ccmain\adaptions.cpp:26:
E:\Lib_prebuild\tesseract\source\src\ccmain/tesseractclass.h:30:10: fatal error: 'allheaders.h' file not found
#include "allheaders.h"             // for pixDestroy, pixGetWidth, pixGetHe...
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/tesseract/issues/2337#issuecomment-485385842,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEMFTFQOEN6Z4ZQUCANNMTPRWG4NANCNFSM4G7RCFGQ
.

Just run path-to\mingw32-make.exe in CMake build directory.

Make does not start configuration....
Dňa po 22. 4. 2019, 13:24 Bleach665 notifications@github.com napísal(a):

Just run path-to\mingw32-make.exe in CMake build directory.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tesseract-ocr/tesseract/issues/2337#issuecomment-485395900,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAEMFTAD2DNNGIJHXBSZXHDPRWN7DANCNFSM4G7RCFGQ
.

Oh, I did not read carefully. I wrote a little of text, maybe it would helpfully to draw up instruction in future.
CMake GUI was used. After 'configure' button was pressed next parameters were specified:
Specify the generator for this project - MinGW Makefile
Specify toolchain file for cross-compiling was selected

On the next page toolchain file was specified.

In my case CMAKE_SH was defined, because msys64\usr was in my PATH envinronment variable. So we undefine it to perform success configure.
After that we need to specify CMAKE_MAKE_PROGRAM with mingw32-make.exe path. MinGW was taken from Qt distribution.

Configure button pressed again.
Next, let's point out the path to Leptonica - Leptonica_DIR must point to Leptonica install CMake folder.
Configure, Generate.
Open command prompt, go to CMake configure output folder. Type path-to-mingw\mingw32-make.exe install.
Done.

In xubuntux64 I receive same error:
[ 32%] Building CXX object CMakeFiles/libtesseract.dir/src/ccutil/elst2.cpp.o
[ 32%] Building CXX object CMakeFiles/libtesseract.dir/src/ccutil/errcode.cpp.o
[ 33%] Building CXX object CMakeFiles/libtesseract.dir/src/ccutil/fileio.cpp.o
/home/bleach/Lib/Tesseract/source/src/ccutil/fileio.cpp:23:10: fatal error: 'glob.h' file not found
#include <glob.h>
         ^~~~~~~~
1 error generated.
CMakeFiles/libtesseract.dir/build.make:1958: recipe for target 'CMakeFiles/libtesseract.dir/src/ccutil/fileio.cpp.o' failed
make[2]: *** [CMakeFiles/libtesseract.dir/src/ccutil/fileio.cpp.o] Error 1
CMakeFiles/Makefile2:104: recipe for target 'CMakeFiles/libtesseract.dir/all' failed
make[1]: *** [CMakeFiles/libtesseract.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

Configure button pressed again.
Next, let's point out the path to Leptonica - Leptonica_DIR must point to Leptonica install CMake folder.
Configure, Generate.
Open command prompt, go to CMake configure output folder. Type path-to-mingw\mingw32-make.exe install.
Done.
Are you building leptonica after tesseract?
You need to build it before.

Leptonica was build before Tesseract. As far as I remember in CMake config Leptonica find_package is with REUARED flag, so Leptonica_DIR should point to right dir (cmake dir in our install dir), if not then we get an error while configuring.
And maybe that's important, TARGET_ARCHITECTURE was set to armeabi-v7a.

@Bleach665 : glob.h is provided by os (linux  or android) so this is not tesseract issue anymore. Seems like you need to setup include path for cmake, so compiler finds all needed system header files.

@zdenop . Yes,

if(ANDROID_TOOLCHAIN)
include_directories(${ANDROID_TOOLCHAIN}/sysroot/usr/include)
endif()

fix it.
But after fix we have another error:
[ 19%] Building CXX object CMakeFiles/libtesseract.dir/src/ccutil/errcode.cpp.o
[ 20%] Building CXX object CMakeFiles/libtesseract.dir/src/ccutil/fileio.cpp.o
E:\Lib_prebuild\tesseract\source\src\ccutil\fileio.cpp:121:7: error: use of undeclared identifier 'glob'
  if (glob(pattern, 0, nullptr, &pglob) == 0) {
      ^
E:\Lib_prebuild\tesseract\source\src\ccutil\fileio.cpp:125:5: error: use of undeclared identifier 'globfree'
    globfree(&pglob);
    ^
2 errors generated.
CMakeFiles\libtesseract.dir\build.make:1088: recipe for target 'CMakeFiles/libtesseract.dir/src/ccutil/fileio.cpp.o' failed
mingw32-make[2]: *** [CMakeFiles/libtesseract.dir/src/ccutil/fileio.cpp.o] Error 1
CMakeFiles\Makefile2:71: recipe for target 'CMakeFiles/libtesseract.dir/all' failed
mingw32-make[1]: *** [CMakeFiles/libtesseract.dir/all] Error 2
Makefile:128: recipe for target 'all' failed
mingw32-make: *** [all] Error 2

Sorry, but this is not tesseract related. You should go to some android forum.

Ok.
Only for testing i directly define __ANDROID_API_FUTURE__:
add_compile_definitions(__ANDROID_API_FUTURE__) added to CMake config file.
So, with all of the above described changes we have successful CMake configuration and build for Android cross build in Widows using android NDK toolchain and MinGW32.
Patch:
0001-fix-Android-cross-build-in-Windows-and-Linux.zip

thanks

This breaks the normal Linux build:
CMake Error at CMakeLists.txt:151 (find_package):
  Could not find a package configuration file provided by "Leptonica"
  (requested version 1.74) with any of the following names:

    LeptonicaConfig.cmake
    leptonica-config.cmake

  Add the installation prefix of "Leptonica" to CMAKE_PREFIX_PATH or set
  "Leptonica_DIR" to a directory containing one of the above files.  If
  "Leptonica" provides a separate development package or SDK, be sure it has
  been installed.

This is because
-    if (NOT Leptonica_DIR AND NOT MSVC)
+    if (NOT Leptonica_DIR AND NOT MSVC AND COMMAND pkg_check_modules)

checks for pkg_check_modules but this function is not defined until you require PkgConfig hence creating a chicken-egg problem.

thanks for report. should be fixed.

Commit 0fe9290 actually fixed cmake based build on macOS, which I could not make work otherwise (missing -lleptonica although libleptonica present in /usr/local/lib). d5f23a6 makes it fail again.

0fe9290 works on MacOS 10.14.4
git clone https://github.com/tesseract-ocr/tesseract.git target/tesseract
cd tesseract
git reset --hard 0fe929010a09c99ec8bf0c25ad89d2872454ffd0
mkdir build && cd build
cmake ..
make -j8

...
[100%] Built target tesseract

7a5b9b8 fails
git clone https://github.com/tesseract-ocr/tesseract.git target/tesseract
cd tesseract
mkdir build && cd build
cmake ..

...
Configuring tesseract version 5.0.0-alpha-45-g7a5b...
-- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)
CMake Error at CMakeLists.txt:149 (message):
  Missing required pkg-config

-- Configuring incomplete, errors occurred!

Although which pkgconf outputs /usr/local/bin/pkgconf - but this error is fixable by specifying the pgkconf path explicitly:
cmake .. -D PKG_CONFIG_EXECUTABLE=/usr/local/bin/pkgconf
make -j8

...
[ 99%] Linking CXX shared library libtesseract.dylib
ld: library not found for -lleptonica
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [libtesseract.5.0.0.dylib] Error 1
make[1]: *** [CMakeFiles/libtesseract.dir/all] Error 2
make: *** [all] Error 2

Altough Leptonica is present in /usr/local/lib:
ls /usr/local/lib/liblept*

/usr/local/lib/liblept.dylib -> libleptonica.1.79.0.dylib
/usr/local/lib/libleptonica.1.79.0.dylib
/usr/local/lib/libleptonica.5.3.0.dylib -> libleptonica.1.79.0.dylib
/usr/local/lib/libleptonica.dylib -> libleptonica.5.3.0.dylib

Thanks for the quick reply.

How did you install leptonica? Leptonica cmake build also requires pkg-config.

Leptonica seems to be able compile even without pkgconfig:
git clone https://github.com/DanBloomberg/leptonica.git
cd leptonica && mkdir build && cd build
cmake ..

...
-- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)
...
-- Generating done

Makefile is created and make works just fine.
My approach is a bit unusual, as I am compiling everything from source, as most users seem to install it using brew or port.

@Bleach665: can you please test the latest code if it works for you?

@zdenop. I will be able to do it from the middle of next week.

@zdenop. I try perform cross build in Windows 10, mingw32, CMake 3.13.4, android ndk r16b (a few commits ago build with this configuration was successful) and get CMake configuration error:

CMake gui output:
fatal: not a git repository (or any of the parent directories): .git

Configuring tesseract version 5.0.0-alpha...
CMake Error at CMakeLists.txt:165 (add_subdirectory):
  add_subdirectory given source ".cppan" which is not an existing directory.

Could NOT find LibArchive (missing: LibArchive_LIBRARY LibArchive_INCLUDE_DIR)
No support for auto-detection of the target instruction set/extension

General configuration for Tesseract 5.0.0-alpha
--------------------------------------------------------
Build type:
Compiler: Clang
Used standard: C++14
CXX compiler options:  -isystem E:/Lib_prebuild/android-ndk-r16b/sysroot/usr/include/arm-linux-androideabi -D__ANDROID_API__=14 -g -DANDROID -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -fno-integrated-as -mthumb -Wa,--noexecstack -Wformat -Werror=format-security   --std=c++11
Linker options: -Wl,--exclude-libs,libgcc.a -Wl,--exclude-libs,libatomic.a --sysroot E:/Lib_prebuild/android-ndk-r16b/platforms/android-14/arch-arm -Wl,--build-id -Wl,--warn-shared-textrel -Wl,--fatal-warnings -Wl,--fix-cortex-a8 -Wl,--no-undefined -Wl,-z,noexecstack -Qunused-arguments -Wl,-z,relro -Wl,-z,now -Wl,--gc-sections -Wl,-z,nocopyreloc
Install directory: E:/Lib_prebuild/tesseract/build_arm7
Architecture flags:
Vector unit list:
AVX_OPT: OFF
AVX2_OPT: OFF
SSE41_OPT: OFF
MARCH_NATIVE_OPT: OFF
sim_flags:

Build with cppan [CPPAN_BUILD]: OFF
Build with openmp support [OPENMP_BUILD]: OFF
Disable disable graphics (ScrollView) [GRAPHICS_DISABLED]: OFF
Enable embedded build [EMBEDDED]: OFF
Disable the legacy OCR engine [DISABLED_LEGACY_ENGINE]: OFF
Build training tools [BUILD_TRAINING_TOOLS]: ON
Build tests [BUILD_TESTS]: OFF
--------------------------------------------------------

Configuring incomplete, errors occurred!
See also "E:/Lib_prebuild/tesseract/prebuild_arm7/CMakeFiles/CMakeOutput.log".
See also "E:/Lib_prebuild/tesseract/prebuild_arm7/CMakeFiles/CMakeError.log".

CMakeError.log
It look like problem is with using OpenCL.

@Bleach665 can you please do check once more time?

@zdenop.
Tested on
Winsows 10 x64, CMake gui 3.13.4, MinGW 530_32 (from Qt lib), Android NDK r16b toolchain, rev 17c8ac29a261c352b9868c3d716fa293f6910cf5.
Tesseract was configured with CMake GUI and Android toolchain and built with MinGW.
Build errors:
[ 99%] Built target libtesseract
[100%] Linking CXX executable bin\tesseract
E:/Lib_prebuild/android-ndk-r16b/toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin\ld: error: cannot find -lpvt.cppan.demo.gif
E:/Lib_prebuild/android-ndk-r16b/toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin\ld: error: cannot find -lpvt.cppan.demo.jpeg
E:/Lib_prebuild/android-ndk-r16b/toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin\ld: error: cannot find -lpvt.cppan.demo.png
E:/Lib_prebuild/android-ndk-r16b/toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin\ld: error: cannot find -lpvt.cppan.demo.tiff
E:/Lib_prebuild/android-ndk-r16b/toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin\ld: error: cannot find -lpvt.cppan.demo.webmproject.webp
E:/Lib_prebuild/android-ndk-r16b/toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/lib/gcc/arm-linux-androideabi/4.9.x/../../../../arm-linux-androideabi/bin\ld: error: cannot find -lpvt.cppan.demo.openjpeg.openjp2
clang++.exe: error: linker command failed with exit code 1 (use -v to see invocation)
CMakeFiles\tesseract.dir\build.make:87: recipe for target 'bin/tesseract' failed
mingw32-make.exe[2]: *** [bin/tesseract] Error 1
CMakeFiles\Makefile2:108: recipe for target 'CMakeFiles/tesseract.dir/all' failed
mingw32-make.exe[1]: *** [CMakeFiles/tesseract.dir/all] Error 2
Makefile:128: recipe for target 'all' failed
mingw32-make.exe: *** [all] Error 2

CMake changes:
BUILD_TRAINING_TOOLS:BOOL=0
OPENMP_BUILD:BOOL=0
CMAKE_INSTALL_PREFIX:PATH=E:/Lib_prebuild/tesseract/build_arm7
Leptonica_DIR:PATH=E:/Lib_prebuild/leptonica/build_arm7/cmake
CPPAN_BUILD:BOOL=0
TARGET_ARCHITECTURE:STRING=armeabi-v7a
CMAKE_MAKE_PROGRAM:FILEPATH=C:/Lib/Qt/Tools/mingw530_32/bin/mingw32-make.exe

Can you please make a clean build (maybe including git cloning).
Because cannot find -lpvt.cppan.demo.tiff indicate you run cppan build...

Build was clean and repo cloned from scratch.

than please provide full log from configuration.

Configuring tesseract version 5.0.0-alpha-152-g17c8a...
Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)
Could NOT find LibArchive (missing: LibArchive_LIBRARY LibArchive_INCLUDE_DIR)
No support for auto-detection of the target instruction set/extension

General configuration for Tesseract 5.0.0-alpha-152-g17c8a
--------------------------------------------------------
Build type:
Compiler: Clang
Used standard: C++17
CXX compiler options:  -isystem E:/Lib_prebuild/android-ndk-r16b/sysroot/usr/include/arm-linux-androideabi -D__ANDROID_API__=14 -g -DANDROID -ffunction-sections -funwind-tables -fstack-protector-strong -no-canonical-prefixes -march=armv7-a -mfloat-abi=softfp -mfpu=vfpv3-d16 -fno-integrated-as -mthumb -Wa,--noexecstack -Wformat -Werror=format-security
Linker options: -Wl,--exclude-libs,libgcc.a -Wl,--exclude-libs,libatomic.a --sysroot E:/Lib_prebuild/android-ndk-r16b/platforms/android-14/arch-arm -Wl,--build-id -Wl,--warn-shared-textrel -Wl,--fatal-warnings -Wl,--fix-cortex-a8 -Wl,--no-undefined -Wl,-z,noexecstack -Qunused-arguments -Wl,-z,relro -Wl,-z,now -Wl,--gc-sections -Wl,-z,nocopyreloc
Install directory: O:/21/tesseract_arm_inst
Architecture flags:
Vector unit list:
AVX_OPT: OFF
AVX2_OPT: OFF
SSE41_OPT: OFF
MARCH_NATIVE_OPT: OFF
sim_flags:

Build with cppan [CPPAN_BUILD]: OFF
Build with openmp support [OPENMP_BUILD]: OFF
Disable disable graphics (ScrollView) [GRAPHICS_DISABLED]: OFF
Enable embedded build [EMBEDDED]: OFF
Disable the legacy OCR engine [DISABLED_LEGACY_ENGINE]: OFF
Build training tools [BUILD_TRAINING_TOOLS]: OFF
Build tests [BUILD_TESTS]: OFF
--------------------------------------------------------

Configuring done
Generating done

I downloaded the latest ndk (r20) and it provide make tool \android-ndk-r20\prebuilt\windows-x86_64\bin\make.exe ), so there is not need to use Qt mingw make file. IMO it is better to use tools from one source instead of multiple sources/providers.
Your output is reconfigure output from cmake-gui!  Full/clean configure seems like this:

ANDROID_PLATFORM not set. Defaulting to minimum supported version
16.
Check for working C compiler: F:/0Android/android-ndk-r20/toolchains/llvm/prebuilt/windows-x86_64/bin/clang.exe
Check for working C compiler: F:/0Android/android-ndk-r20/toolchains/llvm/prebuilt/windows-x86_64/bin/clang.exe -- works
Detecting C compiler ABI info
Detecting C compiler ABI info - done
Detecting C compile features
Detecting C compile features - done
Check for working CXX compiler: F:/0Android/android-ndk-r20/toolchains/llvm/prebuilt/windows-x86_64/bin/clang++.exe
Check for working CXX compiler: F:/0Android/android-ndk-r20/toolchains/llvm/prebuilt/windows-x86_64/bin/clang++.exe -- works
Detecting CXX compiler ABI info
Detecting CXX compiler ABI info - done

And I guess problem is that you first (clean) run is started as cppan build and then you try to reconfigure at non cppan build, but is does not work (cmake uses cached results during reconfiguration!)
So you either remove .cppan dir (and probably cppan tool from path) or you correcting initialize cmake variables before running initial cmake configuration:

authors: Bleach665 Shreeshrii gorankarlic ismail stweil zdenop


text:
Using openssl doesn't work on a Mac, and is overkill on Linux

Hi!  Thanks for this training example; it's really helpful!
I found that the Makefile line which sorts the list of training examples breaks on a Mac, presumably because it doesn't have named pipes.  But it is also somewhat of an overkill: sort -R --random-source only reads in 16 bytes from the random source to initialise the sorting, so sort -R is probably adequate here.  shuf reads in much more randomness than this, using the random-source for every step of the permutation, but it will fail on a Mac as the shuf command doesn't exist there.
Best wishes,
Julian

Maybe I should reactivate my Python script to sort and shuffle the data. Obviously that would not only be much faster but also improve the portability. @wrznr, what do you think?

that would not only be much faster but also improve the portability.

+1

@juliangilbey, did you see my report in the Wiki? Maybe it also helps. And of course you are invited to share your results there, too.

Thanks - that's a nice report!
I wouldn't go to the effort of using python; I'd just replace
find $(GROUND_TRUTH_DIR) -name '*.lstmf' | sort | \
	  sort -R --random-source=<(openssl enc -aes-256-ctr -pass pass:"$(RANDOM_SEED)" -nosalt </dev/zero 2>/dev/null) > "$@"

(and I just noticed: there's an extra unnecessary sort in that pipeline!) with
find $(GROUND_TRUTH_DIR) -name '*.lstmf' | sort -R > "$@"

The only difference between this and the original version is that sort -R reads from /dev/urandom if it can, and if not, it does something else reasonably intelligent.  The original version uses the --random-source to get a 16-byte seed, and then uses some form of md5 hashing to generate later randomness.  So it will still be at least as good as the Python version.  (And MacOS supports sort -R.)
Best wishes,
Julian

Just a quick clarification: sort -R reads its seed from /dev/urandom if it can, and if not, it does something else reasonably intelligent to get a random seed.

We want reproducible training results. Therefore we can only use pseudo random sequences. Using /dev/urandom is therefore not possible.

Ah, I missed that!
If all you then want is a reproducible result, you could have a 16+ byte file called seed.txt (eg "this is the reproducible seed file; do not modify"), and call --random-source=seed.txt.  That would work without the hassle of calling openssl or python.  You could even use one of the files already present for that, but its content might change at some point.
Best wishes,
Julian

Does that work for Linux, too? I had a similar solution before, but with less than 16 bytes because I did not know that.

I would assume so, as both MacOS and Linux use GNU sort (from coreutils) as their implementation of /usr/bin/sort.

I am pretty sure that the BSD version of coreutils (which is the default on MacOS) does not offer -R.

After a little research, I fear that we have to go with Python if we want to be portable. shuf and sort -R are not available on MacOS (you have to install coreutils via brew or macports). Python is always present. I think the PR proposed by @stweil is sufficient.

I think that it may depend on what version of MacOS you are running; macOS Mojave (macOS 10.14) allows sort -R --random-source, and that's the BSD version of sort.  (I've just checked that on a fresh install of macOS, without brew or macports.)  Of course, the Python solution works too, but it assumes that python3 is installed, which it is not by default on a Mac :-(

If we are talking about portability, we should consider MacOS versions which are still officially supported. Concerning Python3, cf. https://github.com/tesseract-ocr/tesstrain/pull/86#discussion_r320618926

@juliangilbey, Python 3 is required anyway for generating the line boxes, so that is not a new dependency.

True, true!

PR #86 is merged now, so this issue should be fixed.

authors: juliangilbey kba stweil wrznr

repo: tesseract-ocr/tesstrain


text:
Inconsistency in root level element in JUnit format

It appears that there are several different implementations of the JUnit xml format. The root element is either a <testsuites> that contains a collection of <testsuite> elements, or a single <testsuite>.
As per https://github.com/bryanbcook/azdevops-testplan-extension/issues/16, there appears to be two conflicting schemas:

https://github.com/testmoapp/junitxml - defines <testsuites> as a root node. This is the format used by the testparser project
https://github.com/windyroad/JUnit-Schema/blob/master/JUnit.xsd - referenced by IBM's website defines an xml schema that supports either <testsuites> or <testsuite> as the root node.

There is also an issue in this repo for #40 which doesn't have any details.

@ASaiAnudeep Thanks for merging the PR. How do we create a release for 0.1.17?

repo: test-results-reporter/parser


text:
Test library via TravisCI

I've seen that there is a PR (#12) by @gianarb on integrating a CircleCI config.
I'd like to propose integrating TravisCI since they are able to run docker inside the VM the test runs in.
Also, most open-source projects currently use Travis.
What CI would you rather like to support?

Just one is fine, I picked circleci because I think it is what the other projects like the java implementation uses. But I don't have a blocking one :) The best we can get in place is good

I think we can keep trying to make circleci to work @mraerino I don't think we need travis for now

authors: gianarb mraerino

repo: testcontainers/testcontainer-go