text:
Adding formatters for text calls that can be used to format the outpu…

…t if the value changes.
Changing Text nodes to always be created with empty text.
Loosening the type for text() to allow for numbers and booleans.
Adding formatters variable arguments to text().

LGTM

This takes care of the text() part of #42. It seems to make sense to have some sort of equivalent for attributes, but I'm not sure if the attr() function is the right way to do it.

I'm not sure if the attr() function is the right way to do it

Agreed. I don't like the disparity it'd create between the normal elementOpen and elementOpenStart-attr.

If you pull, https://github.com/google/incremental-dom/commit/e1520c7b340dbb3f5b89ee30a06a736d5fa8195e fixed it.

Awesome, I missed that push.

authors: cramforce jridgewell sparhami

repo: google/incremental-dom


text:
GPU kernels get stopped after 2 seconds on Pixel 4

Describe the bug
I tried to run batch mode benchmarking on MobileNetV2. With tracy, I know that it actually takes ~60ms to finish one inference on Pixel 4 on GPU. If I benchmark the model with size=32, it reports ~60 ms. However, I found something weird when setting the size larger. I found that a launch of GPU kernel always gets stopped after 2 seconds on Pixel 4. The launch will always end at 2 seconds if I set the size >= 64.
This only happens on Pixel 4 (or andoird). For desktop, it can take more than 2 seconds to finish the work.

For CPU, it can also takes more than 2 seconds to finish the work.

To Reproduce
$ build-tracy/iree/tools/iree-translate --iree-mlir-to-vm-bytecode-module --iree-hal-target-backends=vulkan-spirv ~/mobilenet_v2.mlir -o a.vmfb --iree-hal-benchmark-dispatch-repeat-count=5
$ adb push a.vmfb /data/local/tmp
$ adb shell LD_LIBRARY_PATH=/data/local/tmp/ taskset 80 /data/local/tmp/iree-benchmark-module \
  --flagfile=/data/local/tmp/mobilenet_v2_flagfile \
  --module_file=/data/local/tmp/a.vmfb \
  --driver=vulkan \
  --batch_size=5

It can be reproduced by any models. I simplified the command to default config and verified that it can be repro with above instructions. By default it runs in ~1 sec. Making it 5 still make the program end in 2 seconds.

Interesting. Probably due to some emulated timeline semaphore weirdness. We don't have native timeline semaphore support on Pixel4 so enumerated timeline semaphore is chosen there.

You could use emulated timeline semaphores on non-Android to investigate that theory.
My first thought was some other Vulkan ICD timeout. If a particular dispatch takes too long, we may need to split it up into several dispatches.

+1 to what @ScottTodd said. You can force timeline semaphore on by --vulkan_force_timeline_semaphore_emulation.

I tried it on desktop. Looks like it is not the issue of emulated timeline semaphores. It takes 4 seconds to finish the work.

Are you getting a VK_ERROR_DEVICE_LOST return if trying to call vulkan after the 2s? OS usually has a watchdog that will reset the device if it thinks the GPU is hung.

Is the issue just that the work is finishing faster than you expect? you aren't inserting barriers between the dispatches (which is fine), which means all will run concurrently, and you may just be completing it fast :)
Try running on CPU with more than 1 thread enabled and you should see the same - 2 threads should run 2x faster, etc.

I get this error if I run with --benchmark_repetitions=2
F ../iree/hal/vulkan/emulated_semaphore.cc:140] Check failed: outstanding_semaphores_.empty() Destroying an emulated timeline semaphore without first waiting on outstanding signals
Aborted

Is the issue just that the work is finishing faster than you expect? you aren't inserting barriers between the dispatches (which is fine), which means all will run concurrently, and you may just be completing it fast :)
Try running on CPU with more than 1 thread enabled and you should see the same - 2 threads should run 2x faster, etc.

No, I think the issue is not the work is finishing faster than I expect. It should converge to a number when batch_size gets larger. But it always stops after 2 seconds.

No. So far I can only repro it on Pixel 4.

This looks like a Pixel 4 specific issue. Maybe it only happens on my Pixel 4...
Below is the result from S10, which looks good to me.

hi, I think it is the adreno driver that stops too long  tasks. There may be an internal watchdog inside the driver.
For practical uses, do not encode very large command, split into several small commands as a workaround.

Thanks @nihui for the insights! Now we know 2s is the kill threshold for Pixel 4 or Qualcomm SoC. :) Yes, this is just duplicating dispatches in a command buffer for benchmarking purposes, so that we can hide overhead. It is not meant for production usage.

Thanks, I will keep this in mind when setting benchmarking configurations!

source: gharchive/issue

authors: ScottTodd ThomasRaoux antiagainst benvanik hanhanW nihui

repo: google/iree

license_type: permissive

license_source: github-api


text:
[codegen] try to support dynamic shape in llvm-ir backend

First of all ,this PR want to address more discussions of this problem
This problem is previously discussed in discord and addressed in https://github.com/google/iree/issues/1265 and https://github.com/google/iree/issues/2013
This PR try to use a somewhat simplified way to solve this problem
1 first we process push_constant in compiler side, put it as the last argument of the llvm function
2 in LLVMJITCommandProcessor::DispatchInline, I use the passed push_constant to compose a memref and pass it to jit function
On my test, most dynamic shape case can passed , but it should still be some cases I don't encounter. Let's discuss more ^_^
@stellaraccident  @antiagainst @benvanik

Thanks @qqsun8819 for working on this, for the runtime part adding an extra push_constant argument to the invocation function will definitely work.
On the compiler side one major thing that is missing here is to either [1] Update the memref descriptor shape field or [2] rewrite all dim ops in the IR to use load values from push_constant.
Some details below:
When we lower Memref (e.g memref<?x?xf32>) type to  LLVM  it will be the following struct:
!llvm<"{ float*, float*, i64, [2 x i64], [2 x i64] }">

The third field is the shape information which what need to be updated by push_constant values.
The information in this field determine loop bounds, as we lower  from std dialect to LLVMIR dialect  memref shape is queried via dim op, for the static case it will be just a constant and the dim op will get canonized into a hostiled constant e.g
%0 = dim %2, 0 : memref<2x2xf32>

====>
%c2 = constant 2 : index

"For this reason the runtime works for static shapes by leaving the shape field empty knowing that IR will never query them ;)"
For the dynamic case this will actually query the memref descriptor :
%4 = llvm.extractvalue %0[3, 0] : !llvm<"{ float*, float*, i64, [2 x i64], [2 x i64] }">

In the dynamic shapes case the IR will look like the following assume we are operating on a memref<?x?xf32>
%1 = hal.interface.load.constant offset = 0 : index
%2 = hal.interface.load.constant offset = 0 : index
%3 = iree.placeholder for "interface buffer" {binding = @legacy_io::@arg0} : memref<?x?xf32>
%4 = shapex.make_ranked_shape %1, %2 : (index, index) -> !shapex.ranked_shape<[?,?]>
%5 = shapex.tie_shape %3, %4 : memref<?x?xf32>, !shapex.ranked_shape<[?,?]>
%6 = dim %5, 0 : memref<?x?xf32>

SSA value %6 is the problematic one. to deal with it we need to do one of the following:
[1] Lower (tie_shape+dim) -> (llvm.load + llvm.insert_value)
[2] Rewrite (tie_shape + dim) -> load from push_constant memref<4xi32>
If I am not missing anything this isn't handled in the PR however seems like dynamic_add test case is passing.
Can you please post the print-ir-after-all,prit-ir-before-all dump to the dynamic_add test and we can look how this is getting lowered to LLVMIR ?

I ran the test with just adding (ResolveShapeOps).
I can see dim is replaced with load constant (which will be doing the load) case [2].
// *** IR Dump Before mlir::iree_compiler::(anonymous namespace)::ResolveShapeOpsPass ***
func @main_ex_dispatch_0() {
  %c4 = constant 4 : index
  %c0 = constant 0 : index
  %c1 = constant 1 : index
  %0 = iree.placeholder for "interface buffer" {binding = @legacy_io::@ret0} : memref<?x4xf32>
  %1 = hal.interface.load.constant offset = 0 : index
  %2 = hal.interface.load.constant offset = 1 : index
  %3 = iree.placeholder for "interface buffer" {binding = @legacy_io::@arg1} : memref<?x4xf32>
  %4 = iree.placeholder for "interface buffer" {binding = @legacy_io::@arg3} : memref<?x4xf32>
  %5 = shapex.make_ranked_shape %1 : (index) -> !shapex.ranked_shape<[?,4]>
  %6 = shapex.make_ranked_shape %2 : (index) -> !shapex.ranked_shape<[?,4]>
  %7 = shapex.tie_shape %4, %5 : memref<?x4xf32>, !shapex.ranked_shape<[?,4]>
  %8 = shapex.tie_shape %3, %6 : memref<?x4xf32>, !shapex.ranked_shape<[?,4]>
  %9 = dim %7, 0 : memref<?x4xf32>
  br ^bb1(%c0 : index)
^bb1(%10: index):  // 2 preds: ^bb0, ^bb4
  %11 = cmpi "slt", %10, %9 : index
  cond_br %11, ^bb2(%c0 : index), ^bb5
^bb2(%12: index):  // 2 preds: ^bb1, ^bb3
  %13 = cmpi "slt", %12, %c4 : index
  cond_br %13, ^bb3, ^bb4
^bb3:  // pred: ^bb2
  %14 = load %7[%10, %12] : memref<?x4xf32>
  %15 = load %8[%10, %12] : memref<?x4xf32>
  %16 = addf %14, %15 : f32
  store %16, %0[%10, %12] : memref<?x4xf32>
  %17 = addi %12, %c1 : index
  br ^bb2(%17 : index)
^bb4:  // pred: ^bb2
  %18 = addi %10, %c1 : index
  br ^bb1(%18 : index)
^bb5:  // pred: ^bb1
  return
}

// *** IR Dump After mlir::iree_compiler::(anonymous namespace)::ResolveShapeOpsPass ***
func @main_ex_dispatch_0() {
  %c4 = constant 4 : index
  %c0 = constant 0 : index
  %c1 = constant 1 : index
  %0 = iree.placeholder for "interface buffer" {binding = @legacy_io::@ret0} : memref<?x4xf32>
  %1 = hal.interface.load.constant offset = 0 : index
  %2 = iree.placeholder for "interface buffer" {binding = @legacy_io::@arg1} : memref<?x4xf32>
  %3 = iree.placeholder for "interface buffer" {binding = @legacy_io::@arg3} : memref<?x4xf32>
  br ^bb1(%c0 : index)
^bb1(%4: index):  // 2 preds: ^bb0, ^bb4
  %5 = cmpi "slt", %4, %1 : index
  cond_br %5, ^bb2(%c0 : index), ^bb5
^bb2(%6: index):  // 2 preds: ^bb1, ^bb3
  %7 = cmpi "slt", %6, %c4 : index
  cond_br %7, ^bb3, ^bb4
^bb3:  // pred: ^bb2
  %8 = load %3[%4, %6] : memref<?x4xf32>
  %9 = load %2[%4, %6] : memref<?x4xf32>
  %10 = addf %8, %9 : f32
  store %10, %0[%4, %6] : memref<?x4xf32>
  %11 = addi %6, %c1 : index
  br ^bb2(%11 : index)
^bb4:  // pred: ^bb2
  %12 = addi %4, %c1 : index
  br ^bb1(%12 : index)
^bb5:  // pred: ^bb1
  return
}

Now I understand what is happening ResoveShapeOp is doing part of [2] this PR is doing the load part.
Nice, This should work as long as there is no linalg named op with dynamic shapes memref here.
So we can add dynamically illegal op (dim of memref of dynamic dim) and dynamically illegal dialect (any linalg op operating on dynamic shapes).
Which will make the failure more debug friendly (failed to legalize instead of runtime error :))
Which we can do in a separate PR!

thanks review for me !@antiagainst @antiagainst
I'll update the pr according to your comments ^_^

@asaadaldien @antiagainst pr updated, take a look again ^_^

@asaadaldien @antiagainst pr updated, take a look again ^_^

Thanks @qqsun8819 for pushing forward on this! LGTM!

LGTM! Thanks @qqsun8819, This will work for the single outermost dimension dynamic case. As @antiagainst mentioned we still need to update the LLVM lowering and update the memref descriptors.

source: gharchive/pull-request

authors: antiagainst asaadaldien qqsun8819


text:
Updated validations.py python script.

Fixed the behavior of validate_user function in validations.py.
Closes: #1
Fixes #<issue_number_goes_here>

It's a good idea to open an issue first for discussion.

[ ] Tests pass
[ ] Appropriate changes to README are included in PR

@googlebot I signed it!

repo: google/it-cert-automation-practice


text:
Updated validations.py python script

Fixes #<issue_number_goes_here>

It's a good idea to open an issue first for discussion.

[ ] Tests pass
[ ] Appropriate changes to README are included in PR

@googlebot I signed it!

@googlebot I signed it!


text:
Closes: #1

Updated validations.py python script.
Fixed the behavior of validate_user function in validations.py.
Fixed the conditionals.
Fixes #<issue_number_goes_here>

It's a good idea to open an issue first for discussion.

[ ] Tests pass
[ ] Appropriate changes to README are included in PR

All good.


text:
Closes: #1

Updated validations.py python script.
Fixed the behavior of validate_user function in validations.py.
Fixes #<issue_number_goes_here>

It's a good idea to open an issue first for discussion.

[ ] Tests pass
[ ] Appropriate changes to README are included in PR

@googlebot I signed it!


text:
Closes: #1

Updated validations.py python script.
Fixed the behavior of validate_user function in validations.py.
Fixes #<issue_number_goes_here>

It's a good idea to open an issue first for discussion.

[ ] Tests pass
[ ] Appropriate changes to README are included in PR

@googlebot I signed it!

@googlebot I signed it!

@googlebot I fixed it.


text:
loops.Scope() arguments to iterate over

The issue here is the parameters one can mention to the range function in Python which is normally: range(start, stop, step).
The example below works
with loops.Scope() as s:

    s.arr = np.zeros(shape = (10, 10), dtype = np.float64)

    for i in s.range(x.shape[0]):

        for j in s.range(x.shape[0]):
            print(i,j)

whereas if one changes the start value of j to be i:
with loops.Scope() as s:

    s.arr = np.zeros(shape = (10, 10), dtype = np.float64)

    for i in s.range(x.shape[0]):

        for j in s.range(i, x.shape[0]):
            print(i,j)

we get the following error:
---------------------------------------------------------------------------
ConcretizationTypeError                   Traceback (most recent call last)
/home/zchandani/q_infinity/jax_comparisons.ipynb Cell 10' in <cell line: 1>()
      [3](vscode-notebook-cell://ssh-remote%2Blambda.lab.rigetti.com/home/zchandani/q_infinity/jax_comparisons.ipynb#ch0000015vscode-remote?line=2)[ s.arr = np.zeros(shape = (10, 10), dtype = np.float64)
      ]()[5](vscode-notebook-cell://ssh-remote%2Blambda.lab.rigetti.com/home/zchandani/q_infinity/jax_comparisons.ipynb#ch0000015vscode-remote?line=4)[ for i in s.range(x.shape[0]):
----> ]()[7](vscode-notebook-cell://ssh-remote%2Blambda.lab.rigetti.com/home/zchandani/q_infinity/jax_comparisons.ipynb#ch0000015vscode-remote?line=6)[     for j in s.range(i, x.shape[0]):
      ]()[8](vscode-notebook-cell://ssh-remote%2Blambda.lab.rigetti.com/home/zchandani/q_infinity/jax_comparisons.ipynb#ch0000015vscode-remote?line=7)[         print(i,j)

File ~/.cache/pypoetry/virtualenvs/q-infinity-O1jTU4hC-py3.9/lib/python3.9/site-packages/jax/experimental/loops.py:201, in Scope.range(self, first, second, third)
    ]()[199](file:///home/zchandani/.cache/pypoetry/virtualenvs/q-infinity-O1jTU4hC-py3.9/lib/python3.9/site-packages/jax/experimental/loops.py?line=198)[ step = 1
    ]()[200](file:///home/zchandani/.cache/pypoetry/virtualenvs/q-infinity-O1jTU4hC-py3.9/lib/python3.9/site-packages/jax/experimental/loops.py?line=199)[ if second is not None:
--> ]()[201](file:///home/zchandani/.cache/pypoetry/virtualenvs/q-infinity-O1jTU4hC-py3.9/lib/python3.9/site-packages/jax/experimental/loops.py?line=200)[   start = int(first)
    ]()[202](file:///home/zchandani/.cache/pypoetry/virtualenvs/q-infinity-O1jTU4hC-py3.9/lib/python3.9/site-packages/jax/experimental/loops.py?line=201)[   stop = int(second)
    ]()[203](file:///home/zchandani/.cache/pypoetry/virtualenvs/q-infinity-O1jTU4hC-py3.9/lib/python3.9/site-packages/jax/experimental/loops.py?line=202)[ else:

File ~/.cache/pypoetry/virtualenvs/q-infinity-O1jTU4hC-py3.9/lib/python3.9/site-packages/jax/core.py:583, in Tracer.__int__(self)
--> ]()[583](file:///home/zchandani/.cache/pypoetry/virtualenvs/q-infinity-O1jTU4hC-py3.9/lib/python3.9/site-packages/jax/core.py?line=582)[ def __int__(self): return self.aval._int(self)

File ~/.cache/pypoetry/virtualenvs/q-infinity-O1jTU4hC-py3.9/lib/python3.9/site-packages/jax/core.py:1123, in concretization_function_error.<locals>.error(self, arg)
   ]()[1122](file:///home/zchandani/.cache/pypoetry/virtualenvs/q-infinity-O1jTU4hC-py3.9/lib/python3.9/site-packages/jax/core.py?line=1121)[ def error(self, arg):
-> ]()[1123](file:///home/zchandani/.cache/pypoetry/virtualenvs/q-infinity-O1jTU4hC-py3.9/lib/python3.9/site-packages/jax/core.py?line=1122)[   raise ConcretizationTypeError(arg, fname_context)

ConcretizationTypeError: Abstract tracer value encountered where concrete value is expected: Traced<ShapedArray(int32[]):JaxprTrace(level=1/0)>
The problem arose with the `int` function. If trying to convert the data type of a value, try using `x.astype(int)` or `jnp.array(x, int)` instead.

See https://jax.readthedocs.io/en/latest/errors.html#jax.errors.ConcretizationTypeError]()

A workaround for this is:
with loops.Scope() as s:

    s.arr = np.zeros(shape = (10, 10), dtype = np.float64)
    u = -1
    for i in s.range(x.shape[0]):
        u+=1
        for j in s.range(u, x.shape[0]):
            print(i,j)

but curious to see if there is a better way of doing this...
Thanks.

jax.experimental.loops is being deprecated
https://github.com/google/jax/releases/tag/jax-v0.3.6

authors: YouJiacheng zohimchandani

repo: google/jax


text:
Error when creating test for new implementation

Hello, I am trying to add an implementation for numpy's histogramdd.  My implementation is currently in my fork here.  I wanted to create a test for my histogramdd implementation but I am running into an error that I'm having a hard time understanding.
My test implementation is also in my fork here which basically compares my implementations result of a test case with numpy's result.
The test case args are np.array([[2,3,1], [4,6,2], [5,5,1]]), (5, 4, 3), None, None, None, None and the result for my implementation is
(DeviceArray([[[1., 0., 0.],
              [0., 0., 0.],
              [0., 0., 0.],
              [0., 0., 0.]],

             [[0., 0., 0.],
              [0., 0., 0.],
              [0., 0., 0.],
              [0., 0., 0.]],

             [[0., 0., 0.],
              [0., 0., 0.],
              [0., 0., 0.],
              [0., 0., 0.]],

             [[0., 0., 0.],
              [0., 0., 0.],
              [0., 0., 0.],
              [0., 0., 1.]],

             [[0., 0., 0.],
              [0., 0., 0.],
              [1., 0., 0.],
              [0., 0., 0.]]], dtype=float32), [DeviceArray([2. , 2.6, 3.2, 3.8, 4.4, 5. ], dtype=float32), DeviceArray([3.  , 3.75, 4.5 , 5.25, 6.  ], dtype=float32), DeviceArray([1.       , 1.3333333, 1.6666667, 2.       ], dtype=float32)])

and the result from numpy is
(array([[[1., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.]],

       [[0., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.]],

       [[0., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.]],

       [[0., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.],
        [0., 0., 1.]],

       [[0., 0., 0.],
        [0., 0., 0.],
        [1., 0., 0.],
        [0., 0., 0.]]]), [array([2. , 2.6, 3.2, 3.8, 4.4, 5. ]), array([3.  , 3.75, 4.5 , 5.25, 6.  ]), array([1.        , 1.33333333, 1.66666667, 2.        ])])

Which appear to match however when I run my test case I get the error
Traceback (most recent call last):
  File "tests/lax_numpy_test.py", line 4492, in <module>
    absltest.main(testLoader=jtu.JaxTestLoader())
  File "/home/alexminnaar/jax-venv/lib/python3.8/site-packages/absl/testing/absltest.py", line 1999, in main
    _run_in_app(run_tests, args, kwargs)
  File "/home/alexminnaar/jax-venv/lib/python3.8/site-packages/absl/testing/absltest.py", line 2116, in _run_in_app
    app.run(main=main_function)
  File "/home/alexminnaar/jax-venv/lib/python3.8/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/home/alexminnaar/jax-venv/lib/python3.8/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/home/alexminnaar/jax-venv/lib/python3.8/site-packages/absl/testing/absltest.py", line 2114, in main_function
    function(argv, args, kwargs)
  File "/home/alexminnaar/jax-venv/lib/python3.8/site-packages/absl/testing/absltest.py", line 2430, in run_tests
    result = _run_and_get_tests_result(
  File "/home/alexminnaar/jax-venv/lib/python3.8/site-packages/absl/testing/absltest.py", line 2401, in _run_and_get_tests_result
    test_program = unittest.TestProgram(*args, **kwargs)
  File "/usr/lib/python3.8/unittest/main.py", line 101, in __init__
    self.runTests()
  File "/usr/lib/python3.8/unittest/main.py", line 271, in runTests
    self.result = testRunner.run(self.test)
  File "/home/alexminnaar/jax-venv/lib/python3.8/site-packages/absl/testing/_pretty_print_reporter.py", line 87, in run
    return super(TextTestRunner, self).run(test)
  File "/usr/lib/python3.8/unittest/runner.py", line 176, in run
    test(result)
  File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.8/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/lib/python3.8/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.8/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/lib/python3.8/unittest/case.py", line 736, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.8/unittest/case.py", line 676, in run
    self._callTestMethod(testMethod)
  File "/usr/lib/python3.8/unittest/case.py", line 633, in _callTestMethod
    method()
  File "/home/alexminnaar/jax-venv/lib/python3.8/site-packages/absl/testing/parameterized.py", line 284, in bound_param_test
    return test_method(self, *testcase_params)
  File "tests/lax_numpy_test.py", line 2310, in testHistogramdd
    self._CompileAndCheck(jnp.histogramdd, args_maker)
  File "/home/alexminnaar/jax/jax/test_util.py", line 855, in _CompileAndCheck
    monitored_ans = cfun(*args)
  File "/home/alexminnaar/jax/jax/test_util.py", line 837, in wrapped_fun
    return fun(*args)
  File "/home/alexminnaar/jax/jax/numpy/lax_numpy.py", line 912, in histogramdd
    bin_edges = histogram_bin_edges(sample[:, i], bins[i], range, weights)
  File "/home/alexminnaar/jax/jax/numpy/lax_numpy.py", line 879, in histogram_bin_edges
    return linspace(range[0], range[1], bins + 1, dtype=dtype)
  File "/home/alexminnaar/jax/jax/numpy/lax_numpy.py", line 2528, in linspace
    if num < 0:
jax.traceback_util.FilteredStackTrace: jax.core.ConcretizationTypeError: Abstract tracer value encountered where concrete value is expected.

The problem arose with the `bool` function.

While tracing the function wrapped_fun at /home/alexminnaar/jax/jax/test_util.py:835, this value became a tracer due to JAX operations on these lines:

  operation cy:int32[] = add b:int32[] cx:int32[]
    from line tests/lax_numpy_test.py:2310 (testHistogramdd)

  operation da:bool[] = lt cy:int32[] cz:int32[]
    from line tests/lax_numpy_test.py:2310 (testHistogramdd)

You can use transformation parameters such as `static_argnums` for `jit` to avoid tracing particular arguments of transformed functions.

See https://jax.readthedocs.io/en/latest/faq.html#abstract-tracer-value-encountered-where-concrete-value-is-expected-error for more information.

Encountered tracer value: Traced<ShapedArray(bool[])>with<DynamicJaxprTrace(level=0/1)>

The stack trace above excludes JAX-internal frames.
The following is the original exception that occurred, unmodified.

--------------------

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/alexminnaar/jax-venv/lib/python3.8/site-packages/absl/testing/parameterized.py", line 284, in bound_param_test
    return test_method(self, *testcase_params)
  File "tests/lax_numpy_test.py", line 2310, in testHistogramdd
    self._CompileAndCheck(jnp.histogramdd, args_maker)
  File "/home/alexminnaar/jax/jax/test_util.py", line 855, in _CompileAndCheck
    monitored_ans = cfun(*args)
  File "/home/alexminnaar/jax/jax/traceback_util.py", line 137, in reraise_with_filtered_traceback
    return fun(*args, **kwargs)
  File "/home/alexminnaar/jax/jax/api.py", line 208, in f_jitted
    out = xla.xla_call(
  File "/home/alexminnaar/jax/jax/core.py", line 1144, in bind
    return call_bind(self, fun, *args, **params)
  File "/home/alexminnaar/jax/jax/core.py", line 1135, in call_bind
    outs = primitive.process(top_trace, fun, tracers, params)
  File "/home/alexminnaar/jax/jax/core.py", line 1147, in process
    return trace.process_call(self, fun, tracers, params)
  File "/home/alexminnaar/jax/jax/core.py", line 577, in process_call
    return primitive.impl(f, *tracers, **params)
  File "/home/alexminnaar/jax/jax/interpreters/xla.py", line 529, in _xla_call_impl
    compiled_fun = _xla_callable(fun, device, backend, name, donated_invars,
  File "/home/alexminnaar/jax/jax/linear_util.py", line 234, in memoized_fun
    ans = call(fun, *args)
  File "/home/alexminnaar/jax/jax/interpreters/xla.py", line 595, in _xla_callable
    jaxpr, out_avals, consts = pe.trace_to_jaxpr_final(fun, abstract_args)
  File "/home/alexminnaar/jax/jax/interpreters/partial_eval.py", line 1023, in trace_to_jaxpr_final
    jaxpr, out_avals, consts = trace_to_subjaxpr_dynamic(fun, main, in_avals)
  File "/home/alexminnaar/jax/jax/interpreters/partial_eval.py", line 1004, in trace_to_subjaxpr_dynamic
    ans = fun.call_wrapped(*in_tracers)
  File "/home/alexminnaar/jax/jax/linear_util.py", line 151, in call_wrapped
    ans = self.f(*args, **dict(self.params, **kwargs))
  File "/home/alexminnaar/jax/jax/test_util.py", line 837, in wrapped_fun
    return fun(*args)
  File "/home/alexminnaar/jax/jax/numpy/lax_numpy.py", line 912, in histogramdd
    bin_edges = histogram_bin_edges(sample[:, i], bins[i], range, weights)
  File "/home/alexminnaar/jax/jax/numpy/lax_numpy.py", line 879, in histogram_bin_edges
    return linspace(range[0], range[1], bins + 1, dtype=dtype)
  File "/home/alexminnaar/jax/jax/numpy/lax_numpy.py", line 2528, in linspace
    if num < 0:
  File "/home/alexminnaar/jax/jax/core.py", line 507, in __bool__
    def __bool__(self): return self.aval._bool(self)
  File "/home/alexminnaar/jax/jax/core.py", line 864, in error
    raise_concretization_error(arg, fname_context)
  File "/home/alexminnaar/jax/jax/core.py", line 853, in raise_concretization_error
    raise ConcretizationTypeError(msg)
jax.core.ConcretizationTypeError: Abstract tracer value encountered where concrete value is expected.

The problem arose with the `bool` function.

While tracing the function wrapped_fun at /home/alexminnaar/jax/jax/test_util.py:835, this value became a tracer due to JAX operations on these lines:

  operation cy:int32[] = add b:int32[] cx:int32[]
    from line tests/lax_numpy_test.py:2310 (testHistogramdd)

  operation da:bool[] = lt cy:int32[] cz:int32[]
    from line tests/lax_numpy_test.py:2310 (testHistogramdd)

You can use transformation parameters such as `static_argnums` for `jit` to avoid tracing particular arguments of transformed functions.

See https://jax.readthedocs.io/en/latest/faq.html#abstract-tracer-value-encountered-where-concrete-value-is-expected-error for more information.

Encountered tracer value: Traced<ShapedArray(bool[])>with<DynamicJaxprTrace(level=0/1)>

----------------------------------------------------------------------
Ran 1 test in 0.641s

FAILED (errors=1)

I can't quite understand the error.  Am I missing something in my test implementation?

In JAX's JIT compilation model, the shapes of arrays cannot be dependent on data values, unless those values are static. In your function, the bins value determines the shape of the output, so it must be a static parameter in the computation.
There are various ways to make certain a parameter is static, but in the test suite, this is generally done by leaving them out of args_maker and passing them to the function in question within a closure. You can see an example of that here, in the test for the 1D histogram: https://github.com/google/jax/blob/e0af77fbca88ec882ef33c18926f8382f7178aed/tests/lax_numpy_test.py#L2286-L2300
Notice that bins and density are treated as static arguments, i.e. they're not parameters passed to the jitted function. If bins were passed to the function by args_maker in this test, it would result in the same error you're seeing in your test.
Does that make sense?

Thanks @jakevdp I made that change and it looks like it fixed it.  I am curious though, in the testHistogram example you posted, doesn't the shape value also determine the shape of the output? Why does that not produce the error when put in args_maker?

Yes, it's fine for the shape of input arrays to determine the shape of outputs within JIT-compiled computations. Where problems arise is when the values within input arrays determine the shape of outputs.
There's a bit of relevant discussion of this here: https://jax.readthedocs.io/en/latest/notebooks/Common_Gotchas_in_JAX.html#python-control-flow-+-JIT

Interesting.  Thanks again @jakevdp

authors: alexminnaar jakevdp


text:
TensorFlow can find GPU but JAX can't?

Sorry if this has already been asked, but I'm running into a problem where JAX can't find my GPU:
venv/lib/python3.7/site-packages/jax/lib/xla_bridge.py:130: UserWarning: No GPU/TPU found, falling back to CPU.
However, TensorFlow has no problem finding the GPU. When I run tf.config.list_physical_devices('GPU'), I get the following output:
2020-12-28 20:31:33.309894: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5646344172e0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-12-28 20:31:33.309918: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
2020-12-28 20:31:33.311146: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:d8:00.0 name: Tesla V100-PCIE-32GB computeCapability: 7.0
coreClock: 1.38GHz coreCount: 80 deviceMemorySize: 31.75GiB deviceMemoryBandwidth: 836.37GiB/s
2020-12-28 20:31:33.311171: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-12-28 20:31:33.311201: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-12-28 20:31:33.311211: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-12-28 20:31:33.311220: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-12-28 20:31:33.311229: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-12-28 20:31:33.311238: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-12-28 20:31:33.311248: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-12-28 20:31:33.313354: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2020-12-28 20:31:33.313382: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-12-28 20:31:33.850156: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-12-28 20:31:33.850186: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0
2020-12-28 20:31:33.850192: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N
2020-12-28 20:31:33.852686: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 30132 MB memory) -> physical GPU (device: 0, name: Tesla V100-PCIE-32GB, pci bus id: 0000:d8:00.0, compute capability: 7.0)

I'm using tensorflow-gpu version 2.3.0 and JAX version 0.1.73.

I wondered if the issue is that I don't have jaxlib installed with GPU support. I tried the README instructions i.e. pip install --upgrade jax jaxlib==0.1.52+cuda101 -f https://storage.googleapis.com/jax-releases/jax_releases.html
But this return the following error:
ERROR: Requested jaxlib==0.1.52+cuda101 from https://storage.googleapis.com/jax-releases/cuda101/jaxlib-0.1.52%2Bcuda101-cp37-none-manylinux2010_x86_64.whl has different version in metadata: '0.1.52'

I had no trouble installing a different version of jaxlib e.g.
Looking in links: https://storage.googleapis.com/jax-releases/jax_releases.html
Collecting jaxlib==0.1.56+cuda101
  Downloading https://storage.googleapis.com/jax-releases/cuda101/jaxlib-0.1.56%2Bcuda101-cp37-none-manylinux2010_x86_64.whl (145.1 MB)
     |████████████████████████████████| 145.1 MB 42.4 MB/s
Requirement already satisfied: scipy in ./dopamine_venv/lib/python3.7/site-packages (from jaxlib==0.1.56+cuda101) (1.4.1)
Requirement already satisfied: numpy>=1.12 in ./dopamine_venv/lib/python3.7/site-packages (from jaxlib==0.1.56+cuda101) (1.18.5)
Requirement already satisfied: absl-py in ./dopamine_venv/lib/python3.7/site-packages (from jaxlib==0.1.56+cuda101) (0.11.0)
Requirement already satisfied: six in ./dopamine_venv/lib/python3.7/site-packages (from absl-py->jaxlib==0.1.56+cuda101) (1.15.0)
Installing collected packages: jaxlib
  Attempting uninstall: jaxlib
    Found existing installation: jaxlib 0.1.52
    Uninstalling jaxlib-0.1.52:
      Successfully uninstalled jaxlib-0.1.52
Successfully installed jaxlib-0.1.56+cuda101

What's wrong with jaxlib 0.1.52?

This now causes a new import error:
  File "/n/home02/rschaeffer/PehlevanLab-Dopamine/dopamine_venv/lib/python3.7/site-packages/jaxlib/xla_client.py", line 39, in <module>
    from . import xla_extension as _xla
ImportError: /lib64/libm.so.6: version `GLIBC_2.23' not found (required by /n/home02/rschaeffer/PehlevanLab-Dopamine/dopamine_venv/lib/python3.7/site-packages/jaxlib/xla_extension.so)

This issue suggests the error might be my OS:
Right now, my machine is running
LSB Version:    :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID: CentOS
Description:    CentOS Linux release 7.8.2003 (Core)
Release:        7.8.2003
Codename:       Core

I found that running
pip install pip==20.3.3
pip install --upgrade jax jaxlib==0.1.52+cuda101 -f https://storage.googleapis.com/jax-releases/jax_releases.html

reproduces the different version in metadata error, but using older versions of pip do not result in this error. For example, this installs correctly:
pip install pip==20.2.4
pip install --upgrade jax jaxlib==0.1.52+cuda101 -f https://storage.googleapis.com/jax-releases/jax_releases.html

I suspect that the pip 20.3 dependency resolver places stricter requirements on the contents of package metadata.

@jakevdp thanks for discovering that! I was able to install jaxlib==0.1.56+cuda101 without downgrading pip, but now I'm getting the error
ImportError: /lib64/libm.so.6: version `GLIBC_2.23' not found```

Any idea what causes that or how to fix it?

The GLIBC issue sounds like it's related to this: https://github.com/google/jax/issues/854

Yeah I commented that above. I was more asking whether you know of a solution other than to change the OS?

I don't know of any workaround, but others might.

authors: RylanSchaeffer jakevdp


text:
named_call ignores some transformations

named_call is a call primitive that adds metadata to the HLO for use in profiling and debugging. In implementation, named_call avoids abstracting its inputs like jit does to support use-cases like:
@named_call('foo')
def f(x):
  if x > 0:
    ...

If x were abstracted (like with jit), we couldn't use it in a Python if statement.
This approach has a downside: non-dynamic traces (vmap, ad, etc.) can entirely miss the named_call and thereby not update its name if the transformation does not take in the appropriate tracers.
Take the following example:
@jax.vmap
@functools.partial(jax.named_call, name='foo')
def f(x):
  return jnp.square(x)

print(jax.make_jaxpr(f)(jnp.ones(5)))

This results in the following jaxpr:
{ lambda  ; a.
  let b = named_call[ call_jaxpr={ lambda  ; a.
                                   let b = integer_pow[ y=2 ] a
                                   in (b,) }
                      name=foo ] a
  in (b,) }

Note that the name param in the named_call is 'foo' not 'vmap(foo)'.  Here's another example:
@jax.grad
@functools.partial(jax.named_call, name='foo')
def f(x):
  return jnp.square(x).sum()

print(jax.make_jaxpr(f)(jnp.ones(5)))

We get this jaxpr:
{ lambda  ; a.
  let _ _ b = named_call[ call_jaxpr={ lambda  ; a.
                                       let b = integer_pow[ y=2 ] a
                                           c = integer_pow[ y=1 ] a
                                           d = mul 2.0 c
                                           e = reduce_sum[ axes=(0,) ] b
                                       in (e, *, d) }
                          name=foo ] a
      c = named_call[ call_jaxpr={ lambda  ; a b.
                                   let c = broadcast_in_dim[ broadcast_dimensions=(  )
                                                             shape=(5,) ] b
                                       d = mul c a
                                   in (d,) }
                      name=transpose(foo) ] b 1.0
  in (c,) }

In this example, the named_call is "unzipped" into two -- the first is for the primals pass (part of jvp) and the second is for the transpose. Note that we do not update foo in the jvp pass with any annotation.
Overall, this limitation can lead to misleading annotations in jaxpr and HLO where transformations can be dropped.

I believe that this could be addressed if only post_process_call had the power to modify the call primitive's params. Conceptually speaking the post-process point is where we'd catch this otherwise.
So taking on this issue may mainly boil down to changing the post_process_call convention, and then using it for the fix.

authors: froystig sharadmv


text:
jax.tree_util.Partial is not hash-stable in jax>=0.2.22

Following #8101 , jax.tree_util.Partial.func is no more hash-stable:
jax==0.2.21
>>> import jax
>>> from jax.tree_util import Partial
>>> from functools import partial
>>> fun = partial(print, 1)
>>> hash(fun)
278539187
>>> hash(Partial(fun, 2).func)
278539187
>>> hash(Partial(fun, 2).func)
278539187

latest release
>>> from jax.tree_util import Partial
>>> from functools import partial
>>> fun = partial(print, 1)
>>> hash(fun)
8749850128694
>>> hash(Partial(fun, 2).func)
8750211514691
>>> hash(Partial(fun, 2).func)
8750211514664

# instead, if the function wrapped is just a standard function, everything works fine
>>> hash(Partial(print, 1).func)
56249262
>>> hash(Partial(print, 1).func)
56249262

This is annoying because in some code in NetKet, we were wrapping user-passed functions into a tree_util.Partial in order to simplify some code, but now this triggers recompilation if the user's supplied function is wrapped in a partial or not.
cc @hawkinsp who committed the changes

#9430 fixes the problem identified in this bug. Note that .func will still not return the original object: instead it returns a wrapper object that has the same hash and compares as equal to the original function. If we wanted to avoid that, we'd need to reimplement Partial without subclassing functools.partial.

That's great, thanks and solves our issue!
I mainly care about the hash stability so that I can use that without triggering recompilation. The fact that the object is not identical does not seem a problem to me.

We released jax 0.3.0 which includes this fix.

authors: PhilipVinc hawkinsp


text:
Added support for loading JSON/YAML files for topology (with .yaml, yml and .json extensions)

Fixes https://github.com/google/kne/issues/93.
File formats are detected based on extensions .yaml, .yml and .json. If no match is found, files are assumed to be .pb.txt (i.e. no check for extension when loading .pb.txt, as was the case before).

Will need to re-create PR using an org account.

repo: google/kne


text:
Fix use of ln on OS X [phab:D895]

Created by zarko at 2016-05-16 23:48:24:
This change unbreaks builds after clean (and possibly some
other scenarios, but I was reliably able to get it to trip
using bazel clean):
$ bazel clean --expunge
$ bazel build //kythe/go/storage/leveldb/...
INFO: Found 2 targets...
...
INFO: Elapsed time: 67.263s, Critical Path: 29.49s
$ bazel clean
INFO: Starting clean (this may take a while). Consider using --expunge_async if the clean takes more than several minutes.
$ bazel build //kythe/go/storage/leveldb/...
INFO: Found 2 targets...
ERROR: /private/var/tmp/.../2be6994bf77f75bac8f1433eb6e9dc33/external/go_levigo/BUILD:9:1: error executing shell command: 'set -e
export GOROOT=$PWD/external/local_goroot
export CC=$PWD/external/local_config_cc/clang
export CGO_CFLAGS="-DLEVELDB_PLATFORM_POSIX -DOS_MACOSX -isystem $PWD/third_party/leveldb/include -isys...' failed: bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
ln: /var/folders/ft/bv0xnfpx2bqbqc1zp24gc4n8001p27/T//gopath/src/github.com/jmhodges/levigo/go_levigo: File exists
INFO: Elapsed time: 24.082s, Critical Path: 20.79s

schroederc accepted this revision at 2016-05-17 04:01:26

zarko closed via commit 432309a547339c86f43dda8058fb450608c57f89 at 2016-05-17 17:48:48

repo: google/kythe


text:
Simplify treatment of extractor arguments. [phab:D1572]

Created by fromberger at 2017-04-24 21:01:44:
Pack all the arguments into a single dict rather than having some be
special. This is a little easier to read, and ensures the order is consistent.

fromberger wrote at 2017-04-25 06:05:53:
I'm going to just fold this into #1994 instead.

fromberger abandoned revision at 2017-04-25 06:05:53


text:
Update sample web ui dependencies [phab:D2077]

Created by schroederc at 2017-12-05 18:28:52:

fromberger accepted this revision at 2017-12-05 18:29:56

schroederc closed via commit 0434e53d0bc9bbdf2c3c1b3a75135bb256ec78fd at 2017-12-05 18:33:00


text:
Test python manylinux package

Creating a proper manylinux python package is challenging for us due to rust's magika depending on ort, and ort requiring GLIBC >= 2.31 for pre-built binaries while the latest manylinux is 2_28. See https://github.com/google/magika/pull/747.
For now, we implemented a hack (see https://github.com/google/magika/pull/748) that should work well enough: we generate a generic "linux" python package (with an old ubuntu github runner), and then we patch it so that it looks like a manylinux package. This should work because the rust magika's binary only depend on very few ubiquitous libraries, and we don't expect problems in most cases.
That being said, more tests are needed to check how frequently this does not work.

This approach seems to break build on ubuntu 20.04: https://github.com/google/magika/actions/runs/11274566487/job/31353998130#step:5:1898

The actual requirement seems to be glibc >= 2.35 (ubuntu 22.04), see https://github.com/pykeio/ort/pull/293.

Can we link against an old version of ort crate so that we don't need such a high version of glibc?

Can we link against an old version of ort crate so that we don't need such a high version of glibc?

That could be an option, but that would mean:

Quite some work depending on how old we need to go. We currently use a pre-release so there are breaking changes between almost every version.
We'll lose features, some we asked like telemetry, and some that probably make our life much easier that if we were still using version 1 (I think we had compilation problems or difficulties before we started using version 2).

Ok, good points, linking against old crate could result in even more headaches. Let's put this plan aside for the moment.
Thought about two other things:

is the manylinux_x_y thingy generic? as in: 1) can we compile a manylinux compatible package with arbitrary x and y, or we are stuck with the specific manylinux_2_xx that are already prepared from the manylinux project? 2) do you know if python/pip package installers would check which libc version the system has and compare it with x and y?
is there any way in the rust binary to detect, at start, which libc version is available on the system, and if the version is too old, cleanly error out without crashing? if yes, then one option could be: we have the rust client for reasonably recent clients; for old clients, we detect problems in the rust binary, and then fallback to the python implementation (which I could re-add and make compatible with the rust one).

Thoughts? /cc @invernizzi

Also, https://github.com/pypa/auditwheel seems very useful

@invernizzi my understanding from @ia0 is that we can't do that due to problems with the ort crate, which we depend on. To my understanding, ort crate has dependencies on a too-high version of glibc, which is too high for the currently available manylinux options.
BTW, as of now I don't think we have any way to have a magika rust binary that works on ubuntu 20.04 (which is still supported until next year), so this seems a problem bigger than the mere python packaging.

Understood.
I put together a repo anyway to test this, as it can be useful once manylinux is compatible.  However, I haven't run in the same issue. Maybe I'm missing something.
Here it is.
It depends on the same ort version (ort-sys v2.0.0-rc.6), though it doesn't actually use it for anything.
The guide on how to run it is on that repo. Salient bits:
Downloaded ort-sys v2.0.0-rc.6
[...]
🍹 Building a mixed python/rust project
🔗 Found bin bindings
📡 Using build options bindings from pyproject.toml
🎯 Found 1 Cargo targets in `Cargo.toml`: hello
    Finished `release` profile [optimized] target(s) in 0.09s
📦 Built wheel to /rust/hello/target/wheels/hello-0.1.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Installing python wheel
Processing ./rust/hello/target/wheels/hello-0.1.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Installing collected packages: hello
Successfully installed hello-0.1.0
Testing Python import
Hello from hello!  <- the python code works
Auditing the wheel

hello-0.1.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl is
consistent with the following platform tag: "manylinux_2_17_x86_64".

The wheel references external versioned symbols in these
system-provided shared libraries: libgcc_s.so.1 with versions
{'GCC_3.3', 'GCC_4.2.0', 'GCC_3.0'}, libpthread.so.0 with versions
{'GLIBC_2.2.5'}, libc.so.6 with versions {'GLIBC_2.2.5',
'GLIBC_2.3.4', 'GLIBC_2.16', 'GLIBC_2.14', 'GLIBC_2.3'}

This constrains the platform tag to "manylinux_2_17_x86_64". In order
to achieve a more compatible tag, you would need to recompile a new
wheel from source on a system with earlier versions of these
libraries, such as a recent manylinux image.
Running the executable
Hello, world!  <- the rust binary works.

After that, it can be installed in Ubuntu 20.04 (see the README.md in that repo). Salient bits:
Processing /workspace/rust/hello/target/wheels/hello-0.1.0-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Installing collected packages: hello
Successfully installed hello-0.1.0
Testing Python import
Hello from hello!
Running the executable
Hello, world!
Dropping into an interpreter
root@2913ecd93d74:/#

It seems it's working out fine

The issue might only present itself when ort is actually used - I haven't tested that. I'll leave it to the great @ia0 to tell me what it is I'm not seeing here

The difference is probably that you need to add the download-binaries feature of ort. Since compiling ONNX Runtime from source is the proper solution. So the problem is not the ort crate itself, just the prebuilt binaries it provides for convenience.

@invernizzi nice!

So the problem is not the ort crate itself, just the prebuilt binaries it provides for convenience.

@ia0 I see... but is there any big challenge in compiling from source then? And it makes sense that the issue is with built-in binaries... I would expect it to be relatively rare to truly use glibc features that are so cutting edge...

I don't know, it just needs to be done, and confirm that there are no other problems on that new step.

The issue might only present itself when ort is actually used - I haven't tested that. I'll leave it to the great @ia0 to tell me what it is I'm not seeing here

@invernizzi, interesting; this looks very promising. For context, we detected the problems when we tried to create the python package on a ubuntu 20.04 machine: something related to ort complains about "undefined reference to symbol XYZ in glibc"... which makes sense, as it seems some prebuilt binaries rely on a newer version of glibc. The difference from your setup seems to be: you built the package on a recent ubuntu, and then you tried to install and run on a 20.04 machine... which could work for us! The concern I have is: are we sure it would actually work? It seems that for now we are relaying on prebuilt binaries... that apparently rely on symbols not present in old glibc... can it be that we don't see problems because this setup relies on lazy loading of the dynamically linked dependencies? Maybe we could try to run the binary with LD_BIND_NOW=1 to disable lazy loading (https://man7.org/linux/man-pages/man8/ld.so.8.html)...

I don't know, it just needs to be done, and confirm that there are no other problems on that new step.

ACK. So, it could really be that ort per-se doesn't truly rely on newer glibc, and it could be this problem goes away. I think this is the most reasonable next step. @ia0 please go for it when you have some time and let us know whether there are unexpected challenges! Thank you!

Ok so I probably have something working (still need to wait some confirmation/cleanup workflow runs).
Apparently, the v2 of ort expects ONNX Runtime 1.19.2. Using 1.18 or older will result in an error at runtime when trying to use inference (no error at linking and no error if not running inference).
There are linking problems when using the latest ort 2.0.0-rc.6 which are fixed at main and will soon be released as 2.0.0-rc.7.
In #747 I have an example workflow building a wheel for linux x86_64 using manylinux 2_28. You can download and test the wheel in this workflow run.
I'm still running 2 experiments in that PR:

Make sure I can run the compiled binary in the container: https://github.com/google/magika/actions/runs/11407210939/job/31742704136?pr=747
Final cleanup with macos and windows: https://github.com/google/magika/actions/runs/11407484168/job/31743595565?pr=747

This looks like good news!
Approved https://github.com/google/magika/pull/760.
I've seen you are playing with a maturin workflow; I guess the plan would be that, once all is fixed, we could just integrate whatever you came up with within the build package workflow (which currently lives in https://github.com/google/magika/blob/main/.github/workflows/python-build-and-upload-packages.yml and relies on https://github.com/google/magika/blob/main/python/scripts/build_python_package.py)?

I'm not sure how well we'll be able to integrate with the existing workflow. The 2 difficult points are:

We need to use a cache because building ONNX Runtime takes 40 minutes. This shouldn't be too much of an issue, just copy paste the cache action to your workflow.
We need to compile under a manylinux container. That's probably more of an issue because I just rely on the maturin action to choose the right container and build everything under it. You would probably need to spawn the container yourself and run the python script under it.

The alternative is to just use Maturin and run the version swap logic there. Unless there is something else custom that we need to do? I just checked the Maturin output and it seems to be correct (including the version even though I didn't swap anything, so maybe it's now fixed in Maturin). Steps to reproduce:

Download the wheel produced by the Maturin workflow: example run
Unzip it, install it, test it.

( unzip wheels-linux-x86_64.zip && python3 -m venv venv && venv/bin/pip install *.whl && venv/bin/magika --version && venv/bin/python3 -c 'import magika; print(magika.__version__)' )

Make sure the output is

magika 0.1.0-rc.1 standard_v2_1
0.6.0-rc1

This is what we currently do the build package script (https://github.com/google/magika/blob/main/python/scripts/build_python_package.py):

remove the README to avoid issues with maturin
do the version swap
build the package (with uv build, which uses maturin under the hood)
for the linux build:

check that the dynamic dependencies of the rust magika binary are within the accepted manylinux ones.
patch the wheel to make it look like a manylinux package

For the looks of it, most of the complexity should go away if we can have a legit manylinux package from maturin, so, using maturin workflow (and get rid of the existing build script) seems very doable? The catch: in the current build gh workflow, we actually do a bit more than building: we build it, then I try to install it, and I've run some tests. But again, it looks like we could do all this within the maturin workflow you are working on.
I propose: let's finish whatever last tests you are doing, then we discuss how to proceed. But both options seem reasonable, I don't see anything blocking here. Having an actual legit manylinux package would be really good news!

In this workflow, you can get all the wheels generated by Maturin: https://github.com/google/magika/actions/runs/11437454207.
(Note that the cache reduces the runtime from 40 minutes to 2 minutes.)

Ah right, the README is because of sdist, let me add that to the workflow and see if I can add the same tests.

#781 was the last problem we were running into; now it seems that we have everything to publish an -rc to pypi. Closing this for now.

authors: ia0 invernizzi reyammer

repo: google/magika


text:
Restrict viewing to X axis only

Sorry to raise this as an issue but I couldn't see a better way to get in touch.
Is there a way to restrict a scene to only allow the viewer to pan horizontally, along the X axis, rather than in full 360 space? We have a 360 image, so I don't want to use the flat image example, rather, we don't want the user to be able to scroll up and see a ceiling, or down and see a floor!

You should be able to set up your [RectilinearViewLimiter](https://www.marzipano.net/reference/global.html#RectilinearViewLimiter) to limit the pitch min and max both to 0.

You should be able to set up your RectilinearViewLimiter to limit the pitch min and max both to 0.

Amazing - problem solved, thankyou!

how can i use this?

authors: campmdabt ismaildkc unofficialmatt

repo: google/marzipano

license_source: bigquery


text:
RuntimeError: Unable to initialize runfiles: ERROR: external/bazel_tools/tools/cpp/runfiles/runfiles.cc(120): cannot find runfiles (argv0="")

OS Platform and Distribution
window 7
Compiler version
No response
Programming Language and version
Python
Installed using virtualenv? pip? Conda?(if python)
No response
MediaPipe version
0.9.2
Bazel version
No response
XCode and Tulsi versions (if iOS)
No response
Android SDK and NDK versions (if android)
No response
Android AAR (if android)
None
OpenCV version (if running on desktop)
No response
Describe the problem
detector = vision.ObjectDetector.create_from_options(options)

Complete Logs
Traceback (most recent call last):
  File "D:/AI/mediaPipeExample/object_detector.py", line 17, in <module>
    detector = vision.ObjectDetector.create_from_options(options)
  File "D:\AI\mediaPipeExample\venv\lib\site-packages\mediapipe\tasks\python\vision\object_detector.py", line 207, in create_from_options
    return cls(
  File "D:\AI\mediaPipeExample\venv\lib\site-packages\mediapipe\tasks\python\vision\core\base_vision_task_api.py", line 66, in __init__
    self._runner = _TaskRunner.create(graph_config, packet_callback)
RuntimeError: Unable to initialize runfiles: ERROR: external/bazel_tools/tools/cpp/runfiles/runfiles.cc(120): cannot find runfiles (argv0="")

Hi @RTL8710,
Could you let us know the complete steps you have followed, standalone code to reproduce and look into the issue share the possible solution in the root cause.

object_detector.ipynb

python version and mediapipe version:

code

`# STEP 1: Import the necessary modules.
import numpy as np
import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision
IMAGE_FILE = 'cat_and_dog.jpg'
import cv2
from google.colab.patches import cv2_imshow
STEP 2: Create an ObjectDetector object.
base_options = python.BaseOptions(model_asset_path='efficientdet_lite2_uint8.tflite')
options = vision.ObjectDetectorOptions(base_options=base_options,
score_threshold=0.5)
detector = vision.ObjectDetector.create_from_options(options)
detector =  vision.ObjectDetector.create_from_model_path("efficientdet_lite2_uint8.tflite")
STEP 3: Load the input image.
image = mp.Image.create_from_file(IMAGE_FILE)
STEP 4: Detect objects in the input image.
detection_result = detector.detect(image)
mp_drawing = mp.solutions.drawing_utils
STEP 5: Process the detection result. In this case, visualize it.
image_copy = np.copy(image.numpy_view())
annotated_image = visualize(, detection_result)
mp_drawing.draw_detection(image_copy, detection_result)
rgb_annotated_image = cv2.cvtColor(image_copy, cv2.COLOR_BGR2RGB)
cv2.imshow('face_mesh_annotated_image', rgb_annotated_image)
`

@RTL8710,
Thank you for bringing up to our notice, this issue is known to us and we are working towards fix. As workaround, you can  load the file manually and use model_asset_buffer: https://github.com/google/mediapipe/blob/master/mediapipe/tasks/python/core/base_options.py#L47

tks The problem has been resolved！

May I ask you a new question: How to efficiently and succinctly convert the output of the model into the structure of the C language by porting the mediapipe model (object_dection) to the arm embedded system?

@RTL8710,
Good to hear that issue has been resolved. However, this issue seems to be related to a different error. To keep the context of the issue as relevant to the original issue as possible, may I suggest you close this issue and open another one?
This helps community and us to search the issues better in concise manner. thank you

ok

@RTL8710,
Thank you for the confirmation, Closing this as resolved. Please raise new issue with another query mentioned here.

@RTL8710, Thank you for bringing up to our notice, this issue is known to us and we are working towards fix. As workaround, you can load the file manually and use model_asset_buffer: https://github.com/google/mediapipe/blob/master/mediapipe/tasks/python/core/base_options.py#L47

thanks your reply, but i try it as you, but a error as follow:

@RTL8710, Thank you for bringing up to our notice, this issue is known to us and we are working towards fix. As workaround, you can load the file manually and use model_asset_buffer: https://github.com/google/mediapipe/blob/master/mediapipe/tasks/python/core/base_options.py#L47

thanks your reply, but i try it as you, but a error as follow:

the code as follow:
`#!/usr/bin/env python
-- coding: utf-8 --
'''
@Project ：mediapipe
@File ：mediapipe_task.py
@Author ：xiaolin_peter
@email：zheng.xiaolin@tslsmart.com
@Date ：2023/3/5 15:28
'''
STEP 1: Import the necessary modules.
import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision
from baseOptions import BaseOptions
STEP 2: Create an GestureRecognizer object.
base_options = python.BaseOptions(model_asset_buffer='gesture_recognizer.task')
options = vision.GestureRecognizerOptions(base_options=base_options)
recognizer = vision.GestureRecognizer.create_from_options(options)
images = []
results = []
IMAGE_FILENAMES = ["images/img.png"]
for image_file_name in IMAGE_FILENAMES:
STEP 3: Load the input image.
image = mp.Image.create_from_file(image_file_name)
STEP 4: Recognize gestures in the input image.
recognition_result = recognizer.recognize(image)
STEP 5: Process the result. In this case, visualize it.
images.append(image)
top_gesture = recognition_result.gestures[0][0]
hand_landmarks = recognition_result.hand_landmarks
results.append((top_gesture, hand_landmarks))
print(hand_landmarks)
`

@RTL8710, Thank you for bringing up to our notice, this issue is known to us and we are working towards fix. As workaround, you can load the file manually and use model_asset_buffer: https://github.com/google/mediapipe/blob/master/mediapipe/tasks/python/core/base_options.py#L47

thanks your reply, but i try it as you, but a error as follow:

the code as follow: `#!/usr/bin/env python
-- coding: utf-8 --
''' @project ：mediapipe @file ：mediapipe_task.py @author ：xiaolin_peter @email：zheng.xiaolin@tslsmart.com @Date ：2023/3/5 15:28 '''
STEP 1: Import the necessary modules.
import mediapipe as mp from mediapipe.tasks import python from mediapipe.tasks.python import vision from baseOptions import BaseOptions
STEP 2: Create an GestureRecognizer object.
base_options = python.BaseOptions(model_asset_buffer='gesture_recognizer.task') options = vision.GestureRecognizerOptions(base_options=base_options) recognizer = vision.GestureRecognizer.create_from_options(options)
images = [] results = [] IMAGE_FILENAMES = ["images/img.png"] for image_file_name in IMAGE_FILENAMES:
STEP 3: Load the input image.
image = mp.Image.create_from_file(image_file_name)
STEP 4: Recognize gestures in the input image.
recognition_result = recognizer.recognize(image)
STEP 5: Process the result. In this case, visualize it.
images.append(image) top_gesture = recognition_result.gestures[0][0] hand_landmarks = recognition_result.hand_landmarks results.append((top_gesture, hand_landmarks)) print(hand_landmarks) `

finally,  my python version is 3.8 and mediapipe version 0.9.2.1

authors: RTL8710 kuaashish xiaolinpeter

repo: google/mediapipe


text:
How to build face_landmarks.tflite from mediapipe graph?

OS Platform and Distribution
macOS Ventura 13.2.1
Compiler version
No response
Programming Language and version
TFLite
Installed using virtualenv? pip? Conda?(if python)
No response
MediaPipe version
Newest from source with branch master
Bazel version
5.2.0
XCode and Tulsi versions (if iOS)
No response
Android SDK and NDK versions (if android)
No response
Android AAR (if android)
None
OpenCV version (if running on desktop)
No response
Describe the problem
I just want to build model .tflite from the graph of face landmark.
Please help me or give me the instruction how to do that. Really appreciate for your helps

Complete Logs
(Don't know how to build .tflite from .pbtxt of face_landmark)

Hello @nguyennyssf
TfLite models are artifacts which cannot be created from MediaPipe graphs. They can only be used in MediaPipe graph like here in face landmarks. These can be downloaded from the Model cards page/documentation page of the corresponding solution. For face landmark, the model can be found in this section.

Yes, thanks for your response @ayushgdev. In fact that I really really need to build an ios framework use the latest model of facemesh. I cannot build it on macOS Ventura with XCode 14.3 (macOS Ventura only supports XCode 14 and so I cannot downgrade version).
Really need help or could you give me an example of facemesh xcodeproj? I cannot install Tusi app as well due to macOS version.

@nguyencse We are using MacOS Ventura 13.4 with Xcode 14.1. You can build the legacy solutions in Tulsi with this version setup. However, iOS APIs are getting major upgrades since we are ending support for these MediaPipe Legacy Solutions, and upgrading the others.
We are yet to release the iOS MediaPipe Task APIs and we would request you to wait for sometime. These APIs will be much simpler to use.

ok thanks @ayushgdev , I built a temp framework of my own in ios and waiting for your task api. Thank you

authors: ayushgdev nguyencse nguyennyssf


text:
Mozc does not seem to work under VanillaOS

Description
When using the linux distribution VanillaOS, mozc doesn't seem to work properly.
Steps to reproduce
Steps to reproduce the behavior:

install mozc via abroot pkg add ibus-mozc; abroot pkg apply
Enable mozc in gnome configuration panel
Attempt to type

Expected behavior
Mozc starts correctly and allows input
Actual behavior
The following dialogs appear:
"Failed to get current config values."
"Cannot start conversion engine. Please restart your computer."
Screenshots
n/a
Version or commit-id
2.28.4715.102
Environment

OS:  VanillaOS Orchid
IMF: ibus
Related Applications: all

Investigations

Whether this issue happens on both Ibus and other IMF (e.g. Fcitx, uim).

no

Whether this issue happens on other IMEs (e.g. Anthy, SKK).

does not appear to happen in Anthy

What applications this issue happens on (e.g. Chromium, gedit).

all of them

What applications this issue does not happen on (e.g. Chromium, gedit).

none found

(optional) What versions or commit-ids this issue did not happen on
(e.g. Mozc-2.28.4960.100+24.11.oss).

2.28.4715.102

Additional context
It appears that it may be related to some sort of IPC setup, see the following strace excerpt:
......
[pid  6566] 01:25:17 openat(AT_FDCWD, "/etc/passwd", O_RDONLY|O_CLOEXEC) = 6
[pid  6566] 01:25:17 newfstatat(6, "", {st_mode=S_IFREG|0644, st_size=2670, ...}, AT_EMPTY_PATH) = 0
[pid  6566] 01:25:17 openat(AT_FDCWD, "/dev/shm/sem.fc32c9369a38", O_RDWR|O_NOFOLLOW|O_CLOEXEC) = -1 ENOENT (No such file or directory)
strace: Process 6569 attached
[pid  6566] 01:25:17 openat(AT_FDCWD, "/proc/self/task/6569/comm", O_RDWR) = 6
[libprotobuf ERROR google/protobuf/message_lite.cc:133] Can't parse message of type "mozc.commands.Input" because it is missing required fields: type
[libprotobuf ERROR google/protobuf/message_lite.cc:133] Can't parse message of type "mozc.commands.Input" because it is missing required fields: type
......

From what I know about VanillaOS, there's two major differences that could cause this. Either mozc doesn't like that /usr is a link to /.system/usr, or there's some file in the root FS it's expecting read/write access to but can only get read access.
The relevant bug report in VanillaOS is here, many of us would be happy to help diagnose further! https://github.com/Vanilla-OS/live-iso/issues/68

Hi 0chroma,
Thank you for the report.

2.28.4715.102 is an old version more than 2 years ago.
If protobuf is linked as a shared library, we highly recommend to link protobuf as a static library.

Could you please try the recent version with statically linked protobuf?
FYI. the log files are available at ~/.config/mozc/ (or ~/.mozc)

Was there any update on this?  I'm converting this to discussion as it seems there is no immediate action item in the Mozc side.

authors: 0chroma hiroyuki-komatsu yukawa

repo: google/mozc


text:
Compilation error for fenv_t while building for aarch64-linux-gnu.

While building ortools for aarch64-linux-gnu getting below error:
In file included from ./ortools/util/fp_utils.cc:14:0:
./ortools/util/fp_utils.h: In member function 'void operations_research::ScopedFloatingPointEnv::EnableExceptions(int)':
./ortools/util/fp_utils.h:87:11: error: '**struct fenv_t**' has no member named '**__control_word**'
     fenv_.__control_word &= ~excepts;
           ^
./ortools/util/fp_utils.h:89:11: error: 'struct fenv_t' has no member named '**__mxcsr**'
     fenv_.__mxcsr &= ~(excepts << 7);
           ^
make: *** [objs/util/fp_utils.o] Error 1

This works with g++ compiler, i am not understanding why?
Please help me if there are any clue.

hi,
According to https://android.googlesource.com/platform/bionic/+/afe835d/libc/arch-arm64/include/machine/fenv.h#78 i would say fenv struct is like on Apple
so in https://github.com/google/or-tools/blob/master/ortools/util/fp_utils.h#L85
We should use something like
#if defined(__APPLE__) || defined(__arm__)
    fenv_.__control &= ~excepts;
#else  // Linux
    fenv_.__control_word &= ~excepts;
#endif

Will be better to merge this, no ?

authors: Mizux UpsilonAudio dextran52

repo: google/or-tools


text:
Update renovate.json

Get lockfile maintenance to run on the same schedule as everything else so all the PRs come out on the desired schedule

Thanks!


text:
Upload_it notebook extension: open the URL returned by the upload server

Also enable authentication of xhr requests.

Thank you for the review! Pulling now without waiting for another round of review, but feel free to add more comments if you have any, and I will follow up later.

repo: google/prog-edu-assistant


text:
Multiple Rows for same project _id, region and quota_metric in Notification Table

While creating the Notification Table, multiple rows are inserted for same projectId, region and quota_metric with different added_at times. This leads to confusion when alerts are raised which show similar rows without added_at column.

Fixed by PR #83

authors: bgood shadowshot-x

repo: google/quota-monitoring-solution


text:
[Pygen]: Fix minor issues

Add fix for randcase TODO.
Fixed other minor issues related to RV32B and illegal_instr_test.

@googlebot I consent.

repo: google/riscv-dv


text:
dash real time 2s dash manifest doesn't play under shaka

What version of Shaka Player are you using: 2.2.6
Can you reproduce the issue with our latest release version: yes
Can you reproduce the issue with the latest code from master: yes
Are you using the demo app or your own custom app: both
If custom app, can you reproduce the issue using our demo app: yes
What browser and OS are you using: google chrome / windows 10
What are the manifest and license server URIs:
(you can send the URIs to shaka-player-issues@google.com instead, but please use GitHub and the template for the rest)
What did you do?
hello, i'm R&D and architect OTT senior engineer, we are about to deploy via two proprietary players
a real time dash machine. from the beginning of this project ( 9 month ) we decided to declare shaka as enginnering reference player. more, we are about to start a poc with exoplayer.
What did you expect to happen?
for the moment we can't play our manifest with shaka. think it's a small compromise to choose with you
What actually happened?
shaka tries to synchronize adaptations in loop and never starts a live

@caldrix, we received your email with a manifest attached, but we need to have a manifest URL so that we can access the segments as well.  An MPD as an email attachment is not playable, so we cannot use it to reproduce and work on your issue.
Can you please send us an accessible manifest URL?

Hi joey,

as explained i work for a confidential company. i'm working for enable you a remote access bur it may take time.

can you please review the logs and m4s sended. may be if i send you TS files, you can "simulate" the stream ?

let me know.

regards

Message du 05/12/17 22:56
De : "Joey Parrish"
A : "google/shaka-player"
Copie à : "caldrix" , "Mention"
Objet : Re: [google/shaka-player] dash real time 2s dash manifest doesn't play under shaka (#1165)
@caldrix, we received your email with a manifest attached, but we need to have a manifest URL so that we can access the segments as well. An MPD as an email attachment is not playable, so we cannot use it to reproduce and work on your issue.
Can you please send us an accessible manifest URL?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/google/shaka-player","title":"google/shaka-player","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/google/shaka-player"}},"updates":{"snippets":[{"icon":"PERSON","message":"@joeyparrish in #1165: @caldrix, we received your email with a manifest attached, but we need to have a manifest URL so that we can access the segments as well. An MPD as an email attachment is not playable, so we cannot use it to reproduce and work on your issue.\r\n\r\nCan you please send us an accessible manifest URL?"}],"action":{"name":"View Issue","url":"https://github.com/google/shaka-player/issues/1165#issuecomment-349454855"}}}

hello,

i can open you an access via vpn.

thanks to confirm you ip adress.

regards

Message du 05/12/17 22:56
De : "Joey Parrish"
A : "google/shaka-player"
Copie à : "caldrix" , "Mention"
Objet : Re: [google/shaka-player] dash real time 2s dash manifest doesn't play under shaka (#1165)
@caldrix, we received your email with a manifest attached, but we need to have a manifest URL so that we can access the segments as well. An MPD as an email attachment is not playable, so we cannot use it to reproduce and work on your issue.
Can you please send us an accessible manifest URL?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/google/shaka-player","title":"google/shaka-player","subtitle":"GitHub repository","main_image_url":"https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88-11e6-95fc-7290892c7bb5.png","avatar_image_url":"https://cloud.githubusercontent.com/assets/143418/15842166/7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/google/shaka-player"}},"updates":{"snippets":[{"icon":"PERSON","message":"@joeyparrish in #1165: @caldrix, we received your email with a manifest attached, but we need to have a manifest URL so that we can access the segments as well. An MPD as an email attachment is not playable, so we cannot use it to reproduce and work on your issue.\r\n\r\nCan you please send us an accessible manifest URL?"}],"action":{"name":"View Issue","url":"https://github.com/google/shaka-player/issues/1165#issuecomment-349454855"}}}

@caldrix, we sent you IP addresses privately and we haven't heard back.  If you can't provide access to the content, we can't reproduce the bug.  If we can't reproduce the bug, we can't work on a fix.
We would love to help you, but we will need more information.

We received the manifest URI privately, and we will try to reproduce the issue.  Thank you!

We are unable to access the manifest at the URI you provided.  The connection to your server times out.
In your original report, you mentioned Shaka Player v2.2.6.  Can you try again with the latest release, v2.2.8?  https://v2-2-8-dot-shaka-player-demo.appspot.com/demo/
We have fixed many bugs since v2.2.6, including ones related to live streams, drift, and startup in v2.2.7.
If the issue is still present in v2.2.8, can you please try the nightly build?  https://nightly-dot-shaka-player-demo.appspot.com/demo/
Finally, if all else fails, can you please try a few different versions from our index page?  https://index-dot-shaka-player-demo.appspot.com/  If you can find a version that did work, it would really help us narrow down our search.
Although we can't access your content yet, these steps could help us track down the issue in the mean time.  Thanks!

authors: caldrix joeyparrish

repo: google/shaka-player


text:
Shaka v1.6.4 problem in playing in FireFox

Hi, colleagues.
We're testing Shaka v.1.6.4 playing encrypted live content in FireFox 50. We faced a problem: several HD manifests are not playing in FireFox browser in shaka. Browser devtools show, that networking stops after receiving init-chanks. Looks like shaka receives init-chanks and stops. The same manifest is playing in Chrome without problems. Also we have manifests with identical parameters, which are playing in both Chrome and Firefox without problems. Do you have any ideas about this issue?

@MegaLabs1, can you share a link to a test manifest that reproduces the Firefox problem?

cpeterso,
here are links to manifests that reproduce the problem in FireFox:
https://staging-play.megafon.tv/out/u/v1-video-11.mpd?scheme=https
https://staging-play.megafon.tv/out/u/v1-video-14.mpd?scheme=https

{
  "type": "geo"
}

This is all those URLs give me.

Please, try out these:
https://staging-playvip.megafon.tv/out/u/v1-video-11.mpd?scheme=https
https://staging-playvip.megafon.tv/out/u/v1-video-14.mpd?scheme=https

@MegaLabs1: What is the custom license server I can use for the Shaka Player Demo page?

@cpeterso, you can use https://staging-widevine.megafon.tv:8081/cenc/getlicense/inventosmegalabs as the custom license server

Sorry for getting involved so late.  Testing the same content in Shaka Player v2, the content looks broken.
Your availabilityStartTime parameter keeps changing on manifest updates.  This does not make sense.  availabilityStartTime is the time when the broadcast began.  Timestamps are reckoned relative to this time.
Quick repro using the latest Shaka Player:
http://shaka-player-demo.appspot.com/demo/?asset=https://staging-playvip.megafon.tv/out/u/v1-video-11.mpd?scheme=https;license=https://staging-widevine.megafon.tv:8081/cenc/getlicense/inventosmegalabs

@joeyparrish, sorry for long time response. I've tested another manifest that plays in FireFox correctly. There is also availabilityStartTime changing during manifest updates (in +- 5 seconds interval), but this doesn't affect on playing.

Changing availabilityStartTime is not valid, therefore this is broken content.  Shaka v1 went through a lot of effort to fix certain bad content.  This is why your other manifest works in v1.6.4.
We no longer try to guess what the manifest should say.  Now, we play based on what it does say.
We are a very small team.  We do not have the resources to support broken content, and we will not be putting any more effort into Shaka v1.
Check to see if your encoder has an update.  Perhaps they have already fixed that bug.  If not, please file a ticket with the encoder vendor.  Feel free to reference this discussion or add me to the conversation to help explain to the encoder vendor why this is broken.

Closing due to inactivity.

authors: MegaLabs1 TheModMaker cpeterso joeyparrish sandersaares


text:
sys/netbsd: adding ipc and system management syscalls

Syscalls added:
Manual: msgctl
compat_14_msgctl
compat_50___msgctl13
Manual: semctl
compat_14___semctl
compat_50_____semctl13
____semctl50
Manual: shmctl
compat_14_shmctl
compat_50___shmctl13
rasctl: manual
swapctl: manual
modctl: manual

Hi @ais2397. So far no one with NetBSD expertise has reacted. Overall your PR looks fine to me, is it in a ready-to-be-merged state?

Hi @ais2397. So far no one with NetBSD expertise has reacted. Overall your PR looks fine to me, is it in a ready-to-be-merged state?

Yes

Great, then I'm finally merging this :)

repo: google/syzkaller


text:
tools/create-openbsd-vmm-worker: allow dt

Must happen before cranking securelevel. This makes it possible to fuzz btrace
if we ever want that but more importantly: I have a pending request of running
btrace on the workers in order to investigate a recent performance regression.

cc @blackgnezdo

authors: dvyukov mptre


text:
Updated and simplified macros

syn updated to latest version
quote updated to latest version
proc-macro-2  updated to latest version
Performance improvements

@googlebot I signed it!

clippy check failed - it seems like new version of quote! macro produces more output. I will do some refactoring tomorrow to make clippy happy.

repo: google/tarpc

license: mit


text:
Is this repo still supported and is there an example that involves excel data?

I used handsontable on my web app
Tho that allows me to copy paste data from excel its tedious to write code that works
Im wondering if there's an example of transmat working with excel

Hi, although developments have stalled, the concept of the library still stands. An example of transferring data to Excel could be achieved by using the text/html mimetype and a HTML table (<table>.....</table>) as the contents. Other possible alternatives could be text/csv, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet, but this would require a bit more research.
Would love to hear from you what eventually worked!

authors: jtangelder simkimsia

repo: google/transmat


text:
Update index.rst

Fixing user docs rendering issue. https://github.com/google/turbinia/issues/1156

Thanks for fixing this!
Fixes: https://github.com/google/turbinia/issues/1156

authors: aarontp hed-g

repo: google/turbinia


text:
Publish .aar in a standard artifact repository

Has the uribeacon library been published in a standard artifact repository (like jcenter)?  If not, would you all be interested in doing that?  It would be helpful when building with maven/gradle to point to that instead of having to throw the binary into the source tree.

I think that for now, the .aar are published here: https://github.com/google/uribeacon/releases
If jcenter is a good standard repo, that'd be great to publish it here too.

The purpose of this is so that we can reference it in our gradle files.  For an example see:
https://github.com/cco3/physical-web/commit/b8b90ea2a04289ddd29430a76999de9c3dcbcf9a
Without publishing to something like jcenter, we are stuck putting the binaries in our source tree.

repo: google/uribeacon


text:
Editor: remove duplicate icons

Context

See issue #8555 for details.
This PR kept some of the icons because currently there is no alternatives for following icons in design-system/icons.
story-editor/src/icons/index.js:
export { ReactComponent as TurningLine } from './turning_line.svg';
export { ReactComponent as Union } from './union.svg';
export { ReactComponent as Muted } from './volume_off.svg';

Summary

Relevant Technical Choices

To-do

User-facing changes

Testing Instructions

QA

[ ] This is a non-user-facing change and requires no QA

This PR can be tested by following these steps:

UAT

[ ] UAT should use the same steps as above.

This PR can be tested by following these steps:

Reviews
Does this PR have a security-related impact?

Does this PR change what data or activity we track or use?

Does this PR have a legal-related impact?

Checklist

[x] This PR addresses an existing issue and I have linked this PR to it in ZenHub
[x] I have tested this code to the best of my abilities
[x] I have verified accessibility to the best of my abilities (docs)
[x] I have verified i18n and l10n (translation, right-to-left layout) to the best of my abilities
[x] This PR contains automated tests (unit, integration, and/or e2e) to verify the code works as intended (docs)
[x] I have added documentation where necessary
[x] I have added a matching Type: XYZ label to the PR

Fixes #8555

So this one is a little tricker than it appears.
This Branch	Staging environment

See the difference in the icon? Can you go through and ensure that updating to the design system icons doesn't cause a visual regression?
It would be really helpful too if you could document them in the PR description of where each of these is appearing in the editor (bonus points for screenshots like I did above in a table) - it'll make it a lot easier for review and QA.
Thanks!

Thanks for capturing that! Yeah I did notice the difference when I was doing the change, but I thought that was expected since we are changing the design.
I don't see we have any prop options to control the size of the dots, don't we?
Thanks!

@g-sqiao Looks like something went wrong when rebasing. There are now some unrelated/old commits from main in the PR. You might need to rebase to get those away.

@g-sqiao Looks like something went wrong when rebasing. There are now some unrelated/old commits from main in the PR. You might need to rebase to get those away.
Done

I don't see we have any prop options to control the size of the dots, don't we?
I think we'll just need to use our best judgment (increase the size of the icon) and ask UX for review.

Please also add before vs. after screenshots for each replaced icon.

I just checked the source code for the new design-system icon. Apparently we can't fix the problem by changing the icon size since it's controlled by its path.
<svg width="32" height="32" viewBox="0 0 32 32" fill="none" xmlns="http://www.w3.org/2000/svg">
<path fill-rule="evenodd" clip-rule="evenodd" d="M12 16C12 16.5523 11.5523 17 11 17C10.4477 17 10 16.5523 10 16C10 15.4477 10.4477 15 11 15C11.5523 15 12 15.4477 12 16ZM17 16C17 16.5523 16.5523 17 16 17C15.4477 17 15 16.5523 15 16C15 15.4477 15.4477 15 16 15C16.5523 15 17 15.4477 17 16ZM21 17C21.5523 17 22 16.5523 22 16C22 15.4477 21.5523 15 21 15C20.4477 15 20 15.4477 20 16C20 16.5523 20.4477 17 21 17Z" fill="currentColor"/>
</svg>

Also its wrapper Button has fixed size 28x28.

If you see the previous icon (more_horiz.svg) we used, it's even smaller
<svg width="28" height="28" viewBox="0 0 28 28" fill="none" xmlns="http://www.w3.org/2000/svg">
<path fill-rule="evenodd" clip-rule="evenodd" d="M10 14C10 12.9 9.1 12 8 12C6.9 12 6 12.9 6 14C6 15.1 6.9 16 8 16C9.1 16 10 15.1 10 14ZM12 14C12 15.1 12.9 16 14 16C15.1 16 16 15.1 16 14C16 12.9 15.1 12 14 12C12.9 12 12 12.9 12 14ZM20 16C18.9 16 18 15.1 18 14C18 12.9 18.9 12 20 12C21.1 12 22 12.9 22 14C22 15.1 21.1 16 20 16Z" fill="currentColor"/>
</svg>

I'd suggest to redo the design, but before that we can roll back to the old one. WDYT or any better solution?
@choumx @BrittanyIRL

Combined the changes in #8649 with this PR to make issue tracking easier.
Basically we got ride of the unused ArrowDown button (discussion) and simplified the More button (discussion)

Thank you so much @BrittanyIRL ! Your screenshots made it so easy to find where to visually see the change!
Here are the screenshots for the changes I made (in a way as you suggested).

New
Staging

New
Figma

Thank you so much @BrittanyIRL ! Your screenshots made it so easy to find where to visually see the change!

Hey no problem :)
There's still other icons in here to update though, I thought maybe you could try solving them given the additional context. They will also need to have QA directions added, I edited your PR description to show you a pattern to use for helping others identify where your changes are.

Since @g-sqiao rolled off, I'm going to close this and move this ticket to pea pod's backlog. I'll take care of it some time soon - it's really not hurting anything so there's no massive hurry.

authors: BrittanyIRL choumx g-sqiao swissspidy

repo: google/web-stories-wp


text:
Layer panel updates

Context
Layer panel is getting a shiny new coat of paint. Small updates ahead of larger panel ux updates.
Summary
Layer panel height updates:

Default height is now 25%
Changing the height of the panel is persisted across pages
There is no maximum height
Collapsing the panel will keep the panel closed across pages
Any size change is persisted in local storage

Relevant Technical Choices
n/a
To-do

[x] Layer panel was hidden from screen readers in #4432. Waiting for this discussion to decide whether to update accessibility. Decision: don't do anything for now

User-facing changes

Testing Instructions

[ ] This is a non-user-facing change and requires no QA

This PR can be tested by following these steps:

Load editor
Play with layers panel. The following should all happen:

Default layers panel height to 25% of the total screen height
Unless the user adjusts the height manually, the panel will stay this default height regardless of the layer count
If the user changes the height of the palette, it stays their height as they go through their workflow
If the user toggles the layers panel to expand/collapse using the arrow at the top of the panel this is also respected as the height through their workflow.
The height and collapsed state will persist when the page is refreshed

Reviews
Does this PR have a security-related impact?
no
Does this PR change what data or activity we track or use?
no
Does this PR have a legal-related impact?
no
Checklist

[x] This PR addresses an existing issue and I have linked this PR to it in ZenHub
[x] I have tested this code to the best of my abilities
[x] I have verified accessibility to the best of my abilities (docs)
[x] I have verified i18n and l10n (translation, right-to-left layout) to the best of my abilities
[x] This code is covered by automated tests (unit, integration, and/or e2e) to verify it works as intended (docs)
[x] I have added documentation where necessary
[x] I have added a matching Type: XYZ label to the PR

Fixes #8931

@samwhale Looks like the layer panel is no longer hidden from screen readers, which causes these aXe failures in the e2e tests being surfaced

Nice catch! The layer panel should remain hidden - I put the aria-hidden in the wrong place there.


text:
Java 17 support

Java 17 removed the sun/nio/ch/SocketChannelImpl constructor used here[1] and now throws java.lang.NoSuchMethodError: <init>. Can support be added to use an available constructor?
[1] https://github.com/google/zetasql/blob/master/zetasql/local_service/local_service_jni.cc#L117-L120

This should be fixed in 2022.04.1, please give it a try (thanks @apilloud!)

Confirmed this works in https://github.com/apache/beam/pull/17434

authors: apilloud kileys matthewcbrown

repo: google/zetasql


text:
Copy over java_library.baseline to protoannotations testdata

Copy over the existing java_library.baseline (baseline file for legacy GAPIC yaml only generation) into the java_library_no_gapic_config.baseline file that will be generated by https://github.com/googleapis/gapic-generator/pull/2451/.
Also update the annotated library.proto file to reflect the recent changes in the un-annotated library.proto.  (Note: We should stop needing to copy over changes from the un-annotated library.proto to the annotated.proto once there's a toggle for turning off parsing proto annotations.)

PTAL

Codecov Report

Merging #2462 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master    #2462   +/-   ##
=========================================
  Coverage     86.54%   86.54%
  Complexity     5291     5291
=========================================
  Files           458      458
  Lines         21079    21079
  Branches       2300     2300
=========================================
  Hits          18243    18243
  Misses         2012     2012
  Partials        824      824

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f2bd441...63f9644. Read the comment docs.

authors: andreamlin codecov-io

repo: googleapis/gapic-generator


text:
feat(mtls): support for GOOGLE_API_USE_CLIENT_CERTIFICATE/GOOGLE_API_USE_MTLS_ENDPOINT

Implementation of logic for loading client certificates.
TODO: add unit tests. ✅
Refs: https://github.com/googleapis/gapic-generator-typescript/pull/900
CC: @arithmetic1728

@summer-ji-eng tests added.

@summer-ji-eng @alexander-fenster

To unblock @bcoe, we could merge this PR and refactor by moving to _detectClientCertificate and _mtlsServicePath to credentials.ts in a separate PR.

I think it would be nice to support mTLS in the REST transport, but starting in gRPC lets us support most of the APIs that are initially launching with mTLS support, with the exception of Compute.
if someone is using the REST transport for Webpack purposes, mTLS would need a different implementation than we have for gRPC -- it's probably actually not a good idea, because your client side code would have a cert/key.

I like this, because now the fs code is all inside src/grpc.ts which is already Node.js-dependent, so the webpack clients should not care at all (they don't go into this codepath).

authors: alexander-fenster bcoe

repo: googleapis/gax-nodejs


text:
Play Developer Reporting API - Unable to get hourly aggregated crashRate and in realtime

I am trying to use Play Developer Reporting API to get crashRate for my app.
It says that the only supported aggregation  is DAILY. Is there anyway I can get HOURLY aggregation for retrieving crashRate.
Also the latency for the API is 1-2 days. Can we get the data in realtime or with low latency?
Is it possible with the current API or is there any other API that can help me get in realtime?

Hi @Sid11,
I've opened an internal issue for the API team for these feature requests. Googlers see b/232914641 and b/229301918.

Hey @parthea ,
Thanks for the update
Any ETA on when these feature request might be out for use?

Hi @Sid11,
I've opened the feature request however this work is not planned and there is no ETA. Only DAILY aggregates are available in both API and UI. In addition, the API does not support low latency data currently, but you can go to Google Play Console UI to see low latency data. Unfortunately, currently there are no other APIs that can get realtime data.

@parthea can you please point us to feature request which has been open with API team for tracking ?

The internal bugs are b/195754883 and b/229301918. I don't believe these are available publicly.

repo: googleapis/google-api-python-client


text:
Javadoc builder is more strict than the javadoc tests

[ ] f6b93fa0305dacb1f40c88aca7c811016bf86ee0 fixed some documentation errors that were not caught by the javadoc tests. The tests should ensure that the full documentation site can build and not block a release
[ ] Update to maven-javadoc-plugin 3.0+ and set doclint configuraiton option

It turns out we were running the javadoc tests in Java 7 which was allowing javadoc to build with stuff caught in Java 8. We build our docs with Java 8 so it was failing at doc building time.

repo: googleapis/google-cloud-java


text:
Spanner: PDML should have its own timeout settings with a default of 2 hours

Partitioned DML statements could be long running and should have its own timeout settings with a higher default value.

This will be solved by creating an rpc call with a custom client context.


text:
chore: turn off fail-fast behavior to prevent cancellation of jobs

As seen in https://github.com/googleapis/java-bigtable/pull/1497, unit tests (11) gets cancelled if unit tests (7) fails. This PR prevents workflows from getting cancelled at the first failed job.

Running into a flaky test with a Instance and cluster metadata write requests' and limit 'Instance and cluster metadata write requests per day' of service message. Will try again in a few.

repo: googleapis/java-bigtable


text:
docs: update classification sample to use v2 model

Fixes #898

Hi Neenu, could you possibly provide some clarity on why the Kokoro Test failed? The new imports were all added in the previous commit, ex: https://github.com/googleapis/java-language/blob/main/proto-google-cloud-language-v1/src/main/java/com/google/cloud/language/v1/ClassificationModelOptions.java

Hi @wizeng23 ,
Yes, looks like the kokoro check is failing because the feature has not been released yet.
Triggering a release now. Once the samples pom is updated with the new version and the kokoro check passes with the new version, this PR can be merged.

repo: googleapis/java-language